feat(router): verified cause → fix, bounded triage for unknown causes (#3157, #3158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-24 23:07:41 +00:00
1 parent 00100f1598
commit 9457595888
11 files changed
+260 -7

No files matched your search

+16
View File
@@ -1,5 +1,21 @@
# Changelog
## [Unreleased]
**Asking gstack to fix a failure no longer always opens /investigate (#3157, #3158).** The router sent every bug report straight to the full /investigate workflow, even when the user had named the cause or one look at the failing code showed it. Now the router checks first. When the user names a cause and a narrow fix, it makes one read-only check of the evidence. When that confirms the cause, it makes the fix and runs the focused test. When the cause is unknown, it makes one bounded, read-only triage pass over the supplied evidence and the directly implicated test and code, with no reproduction, hypothesis testing or wider search, and fixes the failure directly only when that pass establishes the root cause. Anything still unclear, including a cause the check contradicts, goes to /investigate before any edit, so the root-cause bar holds. A failing test or stack trace no longer counts as proof of its cause.
### Changed
- **`SKILL.md.tmpl`** (root router): the bug-report rule points to a new **Failures** block that carries the two paths. Its text comes from `generateFailureRouting()` in `scripts/resolvers/preamble/generate-test-failure-triage.ts`.
- **`bin/gstack-skill-start`** and `CLAUDE.md`: the routing section gstack offers for a project's CLAUDE.md now reads "Bugs/errors → invoke /investigate, unless one read-only check confirms the cause and a narrow fix (then fix it and run the focused test)", so it agrees with the router.
- **`/ship` Test Failure Ownership Triage**, "Investigate and fix now": "Switch to /investigate mindset" becomes one read-only pass for the root cause, then `/investigate` if it is still unclear.
### Tests
- New paid eval `test/skill-e2e-failure-routing.test.ts` (gate, `rule`): four fixture repos, two per issue. `router-failure-verified-cause` and `router-failure-triage-resolves` must fix the bug without /investigate and leave the suite green. `router-failure-contradicted-cause` (a user-supplied cause the code contradicts) and `router-failure-triage-unclear` (an intermittent CI-only failure) must invoke /investigate, and an edit before it is a contract violation. Every case needs one to four read-only calls before it routes. `GSTACK_ROUTER_ARM=before` runs the same cases against the base branch's router.
- `test/skill-validation.test.ts` checks that the rendered router carries the rule and no longer sends bug reports straight to /investigate, that both paths keep the /investigate fallback, that the shipped CLAUDE.md routing line matches this repo's own, and that /ship's fix-now step uses the bounded pass.
### Gate integrity
- No threshold, budget, timeout, trial count or existing case kind changed. The ship skill's parity size stays under its cap (313,277 of 313,281 bytes). The new eval's negative controls are the two /investigate cases, and on the base router the triage cases go red.
## [1.91.186.0] - 2026-10-24
**Approving an autoplan run no longer fails one time in ten because its run id looks like a card number.** Today's daily flaky-fix pass found `test/capy-captain-flow.test.ts` failing once and passing on the serial retry in 7 of the 57 Free Tests runs from 2026-10-21 to 2026-10-23, one of them a push to main (39878113535). Every failure had the same assertion: `gstack-approvals import` refused the approval row with "field ask: pii.cc". The default autoplan run id is `run-YYYYMMDDHHMMSS`, the import writes it into the row's ask and source, and the credit-card rule saw those 14 digits as a card number whenever they passed the Luhn check, which a timestamp does about one time in ten. A real user's approval import failed the same way.
+1 -1
View File
@@ -544,7 +544,7 @@ Key routing rules:
- Architecture → invoke /plan-eng-review
- Design system/plan review → invoke /design-consultation or /plan-design-review
- Full review pipeline → invoke /autoplan
- Bugs/errors → invoke /investigate
- Bugs/errors → invoke /investigate, unless one read-only check confirms the cause and a narrow fix (then fix it and run the focused test)
- QA/testing site behavior → invoke /qa or /qa-only
- Code review/diff check → invoke /review
- Visual polish → invoke /design-review
+6 -1
View File
@@ -189,7 +189,7 @@ quality gates that produce better results than answering inline.
- User asks to review design of a plan → invoke `/plan-design-review`
- User asks about developer experience of a plan, API/CLI/SDK design → invoke `/plan-devex-review`
- User wants all reviews done automatically, "review everything" → invoke `/autoplan`
- User reports a bug, error, broken behavior, "why is this broken", "this doesn't work", "wtf", "something's wrong" → invoke `/investigate`
- User reports a bug, error, failing test or broken behavior, "why is this broken", "this doesn't work", "wtf", "something's wrong" → check the cause by **Failures** below, then fix it or invoke `/investigate`
- User asks to test the site, find bugs, QA, "does this work", "check the deploy" → invoke `/qa`
- User asks to just report bugs without fixing → invoke `/qa-only`
- User asks to review code, check the diff, pre-landing review, "look at my changes" → invoke `/review`
@@ -244,6 +244,11 @@ Generic “second opinion”, “outside review”, or “cross-model review”
- Voice trigger "quality check", "test the app", "run QA" → invoke `/qa`
- Voice trigger "bug report", "just check for bugs" → invoke `/qa-only`
**Failures: check the cause, then route.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, invoke `/investigate` before any further tracing or edit.
- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause, scope or intended behavior is still unclear, invoke `/investigate` before any broader diagnosis or edit.
**Route only to skills in your available-skills list.** A rule above whose skill is
not listed there is turned off on this install; skip it, never try to invoke it.
+6 -1
View File
@@ -65,7 +65,7 @@ quality gates that produce better results than answering inline.
- User asks to review design of a plan → invoke `/plan-design-review`
- User asks about developer experience of a plan, API/CLI/SDK design → invoke `/plan-devex-review`
- User wants all reviews done automatically, "review everything" → invoke `/autoplan`
- User reports a bug, error, broken behavior, "why is this broken", "this doesn't work", "wtf", "something's wrong" → invoke `/investigate`
- User reports a bug, error, failing test or broken behavior, "why is this broken", "this doesn't work", "wtf", "something's wrong" → check the cause by **Failures** below, then fix it or invoke `/investigate`
- User asks to test the site, find bugs, QA, "does this work", "check the deploy" → invoke `/qa`
- User asks to just report bugs without fixing → invoke `/qa-only`
- User asks to review code, check the diff, pre-landing review, "look at my changes" → invoke `/review`
@@ -99,6 +99,11 @@ quality gates that produce better results than answering inline.
- User asks to tune question sensitivity, "stop asking me that" → invoke `/plan-tune`
- User asks for code quality dashboard, "health check" → invoke `/health`{{VOICE_TRIGGER_ROUTING}}
**Failures: check the cause, then route.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, invoke `/investigate` before any further tracing or edit.
- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause, scope or intended behavior is still unclear, invoke `/investigate` before any broader diagnosis or edit.
**Route only to skills in your available-skills list.** A rule above whose skill is
not listed there is turned off on this install; skip it, never try to invoke it.
+1 -1
View File
@@ -604,7 +604,7 @@ Key routing rules:
- Architecture → invoke /plan-eng-review
- Design system/plan review → invoke /design-consultation or /plan-design-review
- Full review pipeline → invoke /autoplan
- Bugs/errors → invoke /investigate
- Bugs/errors → invoke /investigate, unless one read-only check confirms the cause and a narrow fix (then fix it and run the focused test)
- QA/testing site behavior → invoke /qa or /qa-only
- Code review/diff check → invoke /review
- Visual polish → invoke /design-review
@@ -1,6 +1,13 @@
import { FREE_TEXT_WRITE_RULE, freeTextFileBash, freeTextFileUse } from '../free-text-file';
export function generateFailureRouting(): string {
return `**Failures: check the cause, then route.** A failing test, error or stack trace shows where a failure surfaces, not its root cause.
- **Cause supplied** (the user names a specific cause and a narrow fix): make one read-only check of the evidence it rests on, such as the named file, test or log. If that confirms the cause, make the fix and run the focused test that covers it. If it does not, invoke \`/investigate\` before any further tracing or edit.
- **Cause unknown:** make one read-only triage pass over the supplied evidence and the directly implicated test and code. Do not reproduce, test hypotheses or search wider. If that pass establishes the root cause and a narrow fix, make the fix and run the focused test that covers it. If the cause, scope or intended behavior is still unclear, invoke \`/investigate\` before any broader diagnosis or edit.`;
}
export function generateTestFailureTriage(): string {
return `## Test Failure Ownership Triage
@@ -48,7 +55,7 @@ Ask with AskUserQuestion in the AskUserQuestion Format. List each failure the sa
### Step T4: Execute the chosen action
**If "Investigate and fix now":**
- Switch to /investigate mindset: root cause first, then minimal fix.
- One read-only pass for the root cause; if unclear, invoke \`/investigate\` first.
- Fix the pre-existing failure.
- Commit the fix separately from the branch's changes: \`git commit -m "fix: pre-existing test failure in <test-file>"\`
- Continue with the workflow.
+1 -1
View File
@@ -319,7 +319,7 @@ Ask with AskUserQuestion in the AskUserQuestion Format. List each failure the sa
### Step T4: Execute the chosen action
**If "Investigate and fix now":**
- Switch to /investigate mindset: root cause first, then minimal fix.
- One read-only pass for the root cause; if unclear, invoke `/investigate` first.
- Fix the pre-existing failure.
- Commit the fix separately from the branch's changes: `git commit -m "fix: pre-existing test failure in <test-file>"`
- Continue with the workflow.
@@ -12,7 +12,7 @@ Key routing rules:
- Architecture → invoke /plan-eng-review
- Design system/plan review → invoke /design-consultation or /plan-design-review
- Full review pipeline → invoke /autoplan
- Bugs/errors → invoke /investigate
- Bugs/errors → invoke /investigate, unless one read-only check confirms the cause and a narrow fix (then fix it and run the focused test)
- QA/testing site behavior → invoke /qa or /qa-only
- Code review/diff check → invoke /review
- Visual polish → invoke /design-review
+12
View File
@@ -1125,6 +1125,10 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts"
],
'router-failure-verified-cause': ['SKILL.md.tmpl', 'SKILL.md', 'scripts/resolvers/preamble/generate-test-failure-triage.ts', 'investigate/SKILL.md.tmpl', 'lib/bun-isolation.ts', 'test/skill-e2e-failure-routing.test.ts'],
'router-failure-contradicted-cause': ['SKILL.md.tmpl', 'SKILL.md', 'scripts/resolvers/preamble/generate-test-failure-triage.ts', 'investigate/SKILL.md.tmpl', 'lib/bun-isolation.ts', 'test/skill-e2e-failure-routing.test.ts'],
'router-failure-triage-resolves': ['SKILL.md.tmpl', 'SKILL.md', 'scripts/resolvers/preamble/generate-test-failure-triage.ts', 'investigate/SKILL.md.tmpl', 'lib/bun-isolation.ts', 'test/skill-e2e-failure-routing.test.ts'],
'router-failure-triage-unclear': ['SKILL.md.tmpl', 'SKILL.md', 'scripts/resolvers/preamble/generate-test-failure-triage.ts', 'investigate/SKILL.md.tmpl', 'lib/bun-isolation.ts', 'test/skill-e2e-failure-routing.test.ts'],
};
/**
@@ -1525,6 +1529,10 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
'overlay-harness-claude-dedicated-tools-vs-bash-sonnet': 'periodic',
'overlay-harness-claude-dedicated-tools-vs-bash-opus-5-5': 'periodic',
'journey-negatives': 'periodic',
'router-failure-verified-cause': 'gate',
'router-failure-contradicted-cause': 'gate',
'router-failure-triage-resolves': 'gate',
'router-failure-triage-unclear': 'gate',
};
/**
@@ -1890,6 +1898,10 @@ export const E2E_KINDS: Record<string, 'rule' | 'behavior' | 'judge'> = {
'overlay-harness-claude-dedicated-tools-vs-bash-sonnet': 'rule',
'overlay-harness-claude-dedicated-tools-vs-bash-opus-5-5': 'rule',
'journey-negatives': 'rule',
'router-failure-verified-cause': 'rule',
'router-failure-contradicted-cause': 'rule',
'router-failure-triage-resolves': 'rule',
'router-failure-triage-unclear': 'rule',
'review/SKILL.md workflow': 'judge',
'setup-browser-cookies/SKILL.md workflow': 'judge',
'browse/SKILL.md reference': 'judge',
+175
View File
@@ -0,0 +1,175 @@
/**
* Router failure routing (#3157, #3158). The gstack router checks a reported
* failure's cause before it routes: a user-supplied cause that one read-only
* check confirms is fixed directly; an unknown cause gets one bounded triage
* pass; anything still unverified goes to /investigate before any edit.
*
* Four fixtures, two per issue. The direct cases (verified cause, triage that
* resolves it) must fix the bug without /investigate and leave the suite
* green. The fallback cases (contradicted cause, unclear intermittent failure)
* are the negative controls: an edit before /investigate is a contract
* violation. Every case needs one to four read-only calls before it routes. The agent sees the rendered router section, sliced from
* SKILL.md; GSTACK_ROUTER_ARM=before slices the base branch's router instead.
*/
import { afterAll, expect } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { runSkillTest } from './helpers/session-runner';
import type { SkillTestResult } from './helpers/session-runner';
import { expectContract } from './helpers/eval-store';
import {
ROOT, runId, describeIfSelected, testConcurrentIfSelected, logCost, recordE2E,
createEvalCollector, finalizeEvalCollector,
} from './helpers/e2e-helpers';
import { CAPTURE_MS } from './helpers/eval-budgets';
import { extractSkillHead, sliceBetween } from './helpers/skill-fixture';
import { bunIsolationArgs } from '../lib/bun-isolation';
const collector = createEvalCollector('e2e-failure-routing');
const ARM = process.env.GSTACK_ROUTER_ARM === 'before' ? 'before' : 'current';
const EDIT_TOOLS = new Set(['Edit', 'Write', 'MultiEdit', 'NotebookEdit']);
const READ_TOOLS = new Set(['Read', 'Grep', 'Glob', 'Bash']);
const SHELL_WRITE = /\b(?:sed|perl)\b[^|;&]*\s-[a-z]*i|(?<![=0-9&>])>>?\s*(?!&|\/dev\/null)\S|\btee\b|\b(?:mv|cp|rm)\s|\bgit\s+(?:apply|checkout|restore)\b/;
function isEdit(call: { tool: string; input: any }): boolean {
return EDIT_TOOLS.has(call.tool) || (call.tool === 'Bash' && SHELL_WRITE.test(String(call.input?.command ?? '')));
}
interface Fixture {
name: string;
route: 'direct' | 'investigate';
files: Record<string, string>;
request: string;
}
const FIXTURES: Fixture[] = [
{
name: 'router-failure-verified-cause',
route: 'direct',
files: {
'src/slug.js': "export function slugify(title) {\n return title.trim().toLowerCase().replace(/ /, '-');\n}\n",
'test/slug.test.js': "import { expect, test } from 'bun:test';\nimport { slugify } from '../src/slug.js';\n\ntest('slugify joins every word with a hyphen', () => {\n expect(slugify('Hello Big World')).toBe('hello-big-world');\n});\n",
},
request: "test/slug.test.js fails: slugify('Hello Big World') returns 'hello-big world'. The cause is the regex in src/slug.js: / / has no g flag, so only the first space is replaced. Fix it by making it / /g.",
},
{
name: 'router-failure-contradicted-cause',
route: 'investigate',
files: {
'src/price.js': "import { discountRate } from './rates.js';\n\nexport function applyDiscount(price, code) {\n return Math.round(price * (1 - discountRate(code)));\n}\n",
'src/rates.js': "import rates from '../config/rates.json' with { type: 'json' };\n\nexport function discountRate(code) {\n return rates[code] ?? 0;\n}\n",
'config/rates.json': '{\n "SAVE10": 10,\n "SAVE25": 0.25\n}\n',
'test/price.test.js': "import { expect, test } from 'bun:test';\nimport { applyDiscount } from '../src/price.js';\n\ntest('SAVE10 takes ten percent off', () => {\n expect(applyDiscount(200, 'SAVE10')).toBe(180);\n});\n",
},
request: "test/price.test.js fails: applyDiscount(200, 'SAVE10') returns -1800 instead of 180. It's a floating-point rounding problem in src/price.js; wrap the result in Math.round() and it will pass.",
},
{
name: 'router-failure-triage-resolves',
route: 'direct',
files: {
'src/initials.js': "export function initials(fullName) {\n const [first, last] = fullName.split(',');\n return (first[0] + last[0]).toUpperCase();\n}\n",
'test/initials.test.js': "import { expect, test } from 'bun:test';\nimport { initials } from '../src/initials.js';\n\ntest('initials of a first and last name', () => {\n expect(initials('Ada Lovelace')).toBe('AL');\n});\n",
},
request: "bun test fails on test/initials.test.js with: TypeError: undefined is not an object (evaluating 'last[0]') at initials (src/initials.js:3). Can you fix it?",
},
{
name: 'router-failure-triage-unclear',
route: 'investigate',
files: {
'src/queue.js': "const pending = [];\nlet timer = null;\n\nexport function enqueue(job, sink) {\n pending.push(job);\n if (!timer) timer = setTimeout(() => flush(sink), Number(process.env.FLUSH_MS ?? 5));\n}\n\nexport function flush(sink) {\n timer = null;\n sink.push(...pending.splice(0));\n}\n",
'src/metrics.js': "import { enqueue } from './queue.js';\n\nexport function track(event, sink) {\n if (event.sampled !== false) enqueue({ type: 'metric', event }, sink);\n}\n",
'test/queue.test.js': "import { expect, test } from 'bun:test';\nimport { enqueue } from '../src/queue.js';\n\ntest('three jobs reach the sink', async () => {\n const sink = [];\n enqueue(1, sink); enqueue(2, sink); enqueue(3, sink);\n await new Promise(r => setTimeout(r, 10));\n expect(sink.length).toBe(3);\n});\n",
'test/metrics.test.js': "import { expect, test } from 'bun:test';\nimport { track } from '../src/metrics.js';\n\ntest('sampled events are tracked', async () => {\n const sink = [];\n track({ name: 'signup' }, sink);\n await new Promise(r => setTimeout(r, 10));\n expect(sink.length).toBe(1);\n});\n",
},
request: "test/queue.test.js fails about one CI run in five with 'expected 3, received 2'. It has never failed on my machine. Can you fix it?",
},
];
function routerSection(): string {
const skill = ARM === 'before'
? spawnSync('git', ['show', `${process.env.EVALS_BASE || 'origin/main'}:SKILL.md`], { cwd: ROOT, encoding: 'utf8', timeout: 10_000 }).stdout
: fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf8');
return sliceBetween(skill, '## Route first', 'Best-effort, record') + sliceBetween(skill, '**Routing rules', 'If the user opts out');
}
function makeRepo(fixture: Fixture): string {
const repo = fs.mkdtempSync(path.join(os.tmpdir(), 'failure-routing-'));
fs.writeFileSync(path.join(repo, 'package.json'), JSON.stringify({ name: 'fixture', type: 'module', scripts: { test: 'bun test' } }, null, 2));
for (const [file, body] of Object.entries(fixture.files)) {
fs.mkdirSync(path.dirname(path.join(repo, file)), { recursive: true });
fs.writeFileSync(path.join(repo, file), body);
}
for (const skill of ['investigate', 'qa', 'review', 'ship']) {
fs.mkdirSync(path.join(repo, '.claude', 'skills', skill), { recursive: true });
fs.writeFileSync(path.join(repo, '.claude', 'skills', skill, 'SKILL.md'), extractSkillHead(path.join(ROOT, skill, 'SKILL.md')));
}
const git = (...args: string[]) => spawnSync('git', ['-c', 'user.email=test@example.invalid', '-c', 'user.name=Test', ...args], { cwd: repo, timeout: 10_000 });
git('init', '-q', '-b', 'main');
git('add', '.');
git('commit', '-qm', 'initial');
return repo;
}
function sourceSnapshot(repo: string, fixture: Fixture): string {
return Object.keys(fixture.files).filter(f => !f.startsWith('test/')).map(f => fs.readFileSync(path.join(repo, f), 'utf8')).join('\0');
}
function routing(result: SkillTestResult) {
const calls = result.toolCalls;
const investigateAt = calls.findIndex(c => c.tool === 'Skill' && String(c.input?.skill ?? '').includes('investigate'));
const firstEditAt = calls.findIndex(isEdit);
const before = investigateAt >= 0 ? calls.slice(0, investigateAt) : firstEditAt >= 0 ? calls.slice(0, firstEditAt) : calls;
return {
investigated: investigateAt >= 0,
editedFirst: firstEditAt >= 0 && (investigateAt < 0 || firstEditAt < investigateAt),
readsBeforeRoute: before.filter(c => READ_TOOLS.has(c.tool) && !isEdit(c)).length,
sequence: calls.map(c => c.tool === 'Skill' ? `Skill(${c.input?.skill})` : c.tool).join(' → '),
};
}
describeIfSelected('Router failure routing (#3157, #3158)', FIXTURES.map(f => f.name), () => {
const dirs: string[] = [];
afterAll(() => {
for (const dir of dirs) fs.rmSync(dir, { recursive: true, force: true });
finalizeEvalCollector(collector);
});
for (const fixture of FIXTURES) {
testConcurrentIfSelected(fixture.name, async () => {
const repo = makeRepo(fixture);
dirs.push(repo);
const sourceBefore = sourceSnapshot(repo, fixture);
const result = await runSkillTest({
prompt: `The user invoked the gstack router skill (/gstack) in this repository. The router's instructions:\n\n${routerSection()}\n\nThe user's request: "${fixture.request}"`,
workingDirectory: repo,
maxTurns: 6,
tools: ['Skill', 'Read', 'Grep', 'Glob', 'Edit', 'Bash'],
allowedTools: ['Skill', 'Read', 'Grep', 'Glob', 'Edit', 'Bash'],
timeout: CAPTURE_MS,
testName: fixture.name,
runId,
});
logCost(`router ${fixture.name} (${ARM})`, result);
const r = routing(result);
const suite = spawnSync(process.execPath, [...bunIsolationArgs(), 'test'], { cwd: repo, encoding: 'utf8', timeout: 60_000 });
const green = suite.status === 0;
const sourceChanged = sourceSnapshot(repo, fixture) !== sourceBefore;
const summary = `arm=${ARM} route=${r.investigated ? 'investigate' : 'direct'} reads=${r.readsBeforeRoute} green=${green} tools: ${r.sequence}`;
const bounded = r.readsBeforeRoute >= 1 && r.readsBeforeRoute <= 4;
const passed = bounded && (fixture.route === 'direct' ? !r.investigated && green : r.investigated && !r.editedFirst);
recordE2E(collector, fixture.name, 'Router failure routing', result, { passed, output: summary });
if (fixture.route === 'investigate') {
expectContract(!r.editedFirst && (r.investigated || !sourceChanged), `edited before /investigate on an unverified cause: ${summary}`, { collector, name: fixture.name });
expect(r.investigated, summary).toBe(true);
} else {
expect(r.investigated, summary).toBe(false);
expect(green, `suite still red after the direct fix: ${summary}\n${suite.stdout}${suite.stderr}`).toBe(true);
}
expect(r.readsBeforeRoute, `no read-only check before routing: ${summary}`).toBeGreaterThanOrEqual(1);
expect(r.readsBeforeRoute, `triage was not bounded: ${summary}`).toBeLessThanOrEqual(4);
}, CAPTURE_MS);
}
});
+33
View File
@@ -7,6 +7,8 @@ import * as os from 'os';
import * as path from 'path';
import { usePrivateStateRoot } from './helpers/private-state-root';
import { expectMentions } from './helpers/prompt-structure';
import { generateFailureRouting } from '../scripts/resolvers/preamble/generate-test-failure-triage';
import { readShippedSkillRouting } from './helpers/shipped-skill-routing';
const ROOT = path.resolve(import.meta.dir, '..');
@@ -2043,6 +2045,37 @@ describe('Test failure triage in ship skill', () => {
});
});
describe('Failure routing: verified cause → fix, unknown cause → bounded triage (#3157, #3158)', () => {
const rule = generateFailureRouting();
const unknownCause = rule.slice(rule.indexOf('**Cause unknown:** ') + '**Cause unknown:** '.length);
test('the router sends bug reports through the failure rule, not straight to /investigate', () => {
const router = fs.readFileSync(path.join(ROOT, 'SKILL.md'), 'utf-8');
expect(router).toContain(rule);
const bugRule = router.split('\n').find(line => line.startsWith('- User reports a bug'));
expect(bugRule).toContain('**Failures**');
expect(bugRule).not.toMatch(/→ invoke `\/investigate`$/);
});
test('the rule keeps the root-cause bar on both paths', () => {
expect(rule).toMatch(/\*\*Cause supplied\*\*[^\n]*one read-only check[^\n]*If it does not, invoke `\/investigate` before any further tracing or edit/);
expect(unknownCause).toMatch(/^make one read-only triage pass[^\n]*Do not reproduce, test hypotheses or search wider[^\n]*still unclear, invoke `\/investigate` before any broader diagnosis or edit\.$/);
});
test('the routing gstack offers for CLAUDE.md, and this repo\'s own, check the cause before /investigate', () => {
const bugRule = readShippedSkillRouting().rules.find(line => line.startsWith('- Bugs/errors'));
expect(bugRule).toMatch(/→ invoke \/investigate, unless one read-only check confirms the cause/);
expect(fs.readFileSync(path.join(ROOT, 'CLAUDE.md'), 'utf-8').split('\n')).toContain(bugRule!);
});
test('/ship fixes a pre-existing failure through the same bounded triage', () => {
const tests = fs.readFileSync(path.join(ROOT, 'ship', 'sections', 'tests.md'), 'utf-8');
const fixNow = tests.slice(tests.indexOf('**If "Investigate and fix now":**'), tests.indexOf('**If "Add as P0 TODO":**'));
expect(fixNow).toContain('- One read-only pass for the root cause; if unclear, invoke `/investigate` first.');
expect(fixNow).not.toContain('/investigate mindset');
});
});
describe('no compiled binaries in git', () => {
// Tracked files enumerated once and reused by both assertions. git ls-files -z
// + split is ~ms; the previous xargs-per-file shell loops blew past 5s on CI.