From b0f1e65e0d9c8a55c74adf3fff4b21126d6ec23c Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 16 Oct 2026 16:24:55 -0700 Subject: [PATCH] =?UTF-8?q?v1.91.108.0=20fix(evals):=20census=20repair=20?= =?UTF-8?q?=E2=80=94=20the=20late-input=20and=20simplification=20fixtures?= =?UTF-8?q?=20stop=20inviting=20their=20own=20failures=20(#3153)=20(#3200)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(qa-callers): the caller scheduling contract no longer calls a probe read-only discovery (#3153) Census 38212265208 (main @ e23f7b807, 2026-10-12) failed ship-exploratory-late-input: "probe preceded resource read: exploratory / system-functional / system-functional". The native events show capture 001 issued in the same provider response as the Reads of ship/sections/plan-completion.md and ship/sections/review-army.md, before the Step 9 section (which holds the "never batch a probe with a Read" STOP) returned and before any QA section was read. The Oct 7 capture (mtb1007a) is the same class. Diagnosis: fixture defect, not product prose. /ship's prose never names the QA capture helper (only qa/sections/exploratory.md does); the fixture prompt supplies it up front, and the fixture's appended scheduling contract (7697fb2, 2026-10-06) told the session to batch "Reads and read-only discovery" while its never-group list omitted Read-result dependencies. The case passed on Oct 3 and Oct 5, before that contract, and failed on Oct 7 and Oct 12 after it. Repair: the contract defines read-only discovery (captures, guard starts/runs and diagnostic-client runs are probes, never discovery), which removes the fixture's own invitation to batch them. It deliberately does not restate the product's Step 9 STOP: the case exists to detect that STOP being lost. The detector (callerProbeReadErrors) is unchanged. Test: the Oct 12 native tool order is replayed inline (no new fixture file, so the paid-runner global inputs stay untouched) with a reordered negative control, and the actual runner callback's contract is checked. Fails on 47d639a at the discovery-definition assertion; passes after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(review-army): the simplification case drops its false premise and stages only its section (#3153) Census 38212265208's pass-rate gate flagged review-army-simplification at 159 s of its 180 s session budget (88.1%, above the 85% cap; main-only panel 142/91/146 s). The three Oct 12 transcripts show where the time went: - the prompt claimed "the diff is over 100 lines", but the fixture diff is 68; every trial reconciled that (scope tool, specialist stats, force-include reasoning, coverage-gap prose in the report and final message); - each session read the whole extracted review-SKILL.md and the 18.7 KB checklist, and grepped for a "## Step 4.5" heading the carved fixture no longer has; - the report was unbounded (30-45 s of output composition); trial 3 also dispatched a subagent and re-verified its findings. Repair (fixture work only, budget unchanged at 180 s / 20 turns): the case uses the same stageReviewArmySession/reviewArmyScope staging as perf-n-plus-one and consensus (recorded scope, stats, learnings and diff; only review-army.md and simplification.md staged), selects the specialist with the real /review --simplification force flag, applies the checklist in-session as the original prompt asked, and bounds the report to 30 lines with a brief final acknowledgement. The planted-structure and disavowed-frame assertions are unchanged. Test: review-army-budget.test.ts drives the actual registered callback with a mocked runner (pass control plus two negative controls); it fails on 47d639a ("Expected to contain: DIFF_LINES: 68") and passes after. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(evals): record the #3153 census diagnoses (review-army-simplification headroom, ship-exploratory-late-input) (#3153) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore(release): tentative v1.91.108.0 stamp and CHANGELOG for the #3153 census repair (#3153) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 18 +++++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- docs/evals/headroom-diagnosis-2026-10.md | 36 ++++++++++++++ package.json | 2 +- test/helpers/qa-callers-fixture.ts | 2 +- test/qa-exploratory-callers.test.ts | 47 +++++++++++++++++++ test/review-army-budget.test.ts | 60 ++++++++++++++++++++++++ test/skill-e2e-review-army.test.ts | 26 +++++----- 9 files changed, 180 insertions(+), 15 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e7c0852ec..ff94efc08 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,23 @@ # Changelog +## [1.91.108.0] - 2026-10-16 + +**The weekly census's two #3153 findings were in the test fixtures, not in /ship or /review.** The Oct 12 census went red on `ship-exploratory-late-input` and flagged `review-army-simplification` over its session headroom cap. Both are diagnosed from the run's own transcripts (`docs/evals/headroom-diagnosis-2026-10.md`), and each fixture repair lands with a free test that failed first. No skill prose, budget, threshold or detector changed. Reported by the weekly periodic census bot (#3153). + +### What this means for you + +- **`ship-exploratory-late-input` no longer gets invited to probe before its QA methods are read.** The fixture's scheduling contract told the session to batch "Reads and read-only discovery" together, and the session counted a QA capture as discovery. It went out in the same response as the Step 9 section Read, before any QA section had returned. The contract now says a capture, a deadline guard start or run, or a diagnostic run is a probe, never discovery. The ordering rule itself stays in /ship's own Step 9 STOP, so the case still catches that STOP being lost. +- **`review-army-simplification` stops spending its budget on a false premise.** Its prompt said the diff was "over 100 lines" when the fixture is 68, so every run reconciled the mismatch, read the whole review skill and checklist, and wrote an unbounded report (142, 91 and 146 s of a 180 s budget on Oct 12; 159 s pooled). It now stages only the Review Army section and the simplification checklist with recorded scope, stats and diff, as the other Review Army cases do. It selects the specialist with the real `/review --simplification` flag and caps the report at 30 lines. The budget and assertions are unchanged. + +### If you… + +- **…maintain gstack's eval census:** the next census on this revision (or the paid reruns of the two cases) decides both; until then #3153 stays open. + +### For contributors + +- `test/qa-exploratory-callers.test.ts` replays census 38212265208's native tool order through `callerProbeReadErrors`, with a reordered negative control, and checks the actual runner callback's discovery definition. +- `test/review-army-budget.test.ts` drives the registered simplification callback with a mocked runner: one pass control and two negative controls. It checks that the premise matches the staged 68-line diff, which files are staged, and the report bound. + ## [1.91.107.0] - 2026-10-16 **The GBRA-40 field list is fixed: install and check agree on one Bun, `rec` approves an autoplan gate in one word, and pregate never sits silent.** Eighteen Capy threads upgraded to v1.91.101.0 and reported 21 problems (`GBRA-40`, October 14). Each gstack item was reproduced with a test that failed first; the plan went through `/autoplan` (two gate revisions, Claude and GPT-6.1 Sol voices per phase) and two independent code reviews. Two items are Capy's (skills not mounted on some crew threads, Drive copies not re-synced) and are handed off, not worked around. diff --git a/VERSION b/VERSION index ba334a60f..c5f649685 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.107.0 +1.91.108.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 694e2eaca..e1465c17c 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.107.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.108.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/docs/evals/headroom-diagnosis-2026-10.md b/docs/evals/headroom-diagnosis-2026-10.md index 64977e41c..8b7a0da7a 100644 --- a/docs/evals/headroom-diagnosis-2026-10.md +++ b/docs/evals/headroom-diagnosis-2026-10.md @@ -96,3 +96,39 @@ declared its target mid-line ("… by the user. Review target (fixed): With the target-field fix the same capture is credited at D5 (521 s). Replay: `bun test test/detector-corpus-plan-eng-multi-finding-batching.test.ts`. No budget or kind change. + +## review-army-simplification: 159 s of 180 s in the Oct 12 census (#3153) + +The pass-rate gate pooled 159 s of the 180 s session budget (88.1%, over the +85% cap); census 38212265208's own panel ran 142, 91 and 146 s. Phases from the +three transcripts (paid slices 25-27): + +| Phase (38212265208) | t1 | t2 | t3 | +|---|---|---|---| +| Read the extracted review-SKILL.md, checklist, specialist; search for a "## Step 4.5" heading the carved fixture no longer has | 17 s | 15 s | 36 s | +| Scope and stats commands, analysis, and reconciling the prompt's "over 100 lines" with the 68-line diff | 64 s | 29 s | 13 s, then a subagent (29 s) and re-verification (21 s) | +| Compose the unbounded report | 45 s | 33 s | 31 s | +| Final message (each restates the findings and the premise mismatch) | 11 s | 8 s | 11 s | + +The fixture, not the skill, carried the extra work. The case now uses the +staging its siblings use (`stageReviewArmySession`: recorded scope, stats, +learnings and diff; only `review-army.md` and `simplification.md` staged), +selects the specialist with the real `/review --simplification` force flag, +applies the checklist in-session and bounds the report to 30 lines. Budget and +assertions are unchanged. Free check: the simplification scenarios in +`test/review-army-budget.test.ts`. + +## ship-exploratory-late-input: a probe batched with its Reads (#3153) + +Not a headroom case; recorded here because the cause is also fixture work. +Census 38212265208 failed it with "probe preceded resource read". Capture 001 +went out in the same provider response as the Reads of the caller's Step 8 and +Step 9 sections, before any QA section was read (the Oct 7 capture `mtb1007a` +batched it with the exploratory Read). /ship's prose never names the QA capture +helper; the fixture prompt supplies it, and the fixture's scheduling contract +(added 2026-10-06) invited batching "Reads and read-only discovery" without +listing Read results among the dependencies a probe must wait for. The case +passed on Oct 3 and Oct 5, before that contract. The contract now classifies +captures, guard starts and runs as probes, never discovery; the ordering rule +itself stays in /ship's Step 9 STOP, which the case still has to follow unaided. +Free check: the census 38212265208 replay in `test/qa-exploratory-callers.test.ts`. diff --git a/package.json b/package.json index 05047cd10..5f1aff495 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.107", + "version": "1.91.108", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/test/helpers/qa-callers-fixture.ts b/test/helpers/qa-callers-fixture.ts index b2269aab7..20540b0a5 100644 --- a/test/helpers/qa-callers-fixture.ts +++ b/test/helpers/qa-callers-fixture.ts @@ -700,7 +700,7 @@ export function qaCallerSessionOptions(fixture: QaCallerFixture, runId: string): prompt: `Load gstack's /${fixture.caller} supplied parent phase from caller-${fixture.caller}.md and resume it on the selected working-tree diff against origin/main. This excerpt comes from ${fixture.runtime}/${fixture.caller}/SKILL.md; resolve installed-relative references there, not from the excerpt file or product directory. That path identifies the asset base, not another entrypoint: do not read or invoke the full parent SKILL.md or rerun its preamble. Earlier preamble/branch/base setup is complete; use the existing local origin/main ref without fetch. Earlier-phase asset locators are ${fixture.runtime}/review/checklist.md and ${fixture.runtime}/qa/templates/functional-report-template.md. Read those files directly when referenced; recursive Glob does not follow the installed asset symlinks. Cross-project learnings are configured off in this owned fixture. ${fixture.reviewStart ? `The actual review-start helper already captured REVIEW_START=${fixture.reviewStart} for this unchanged core pass; retain that token. ` : ''} This fixture evaluates only the supplied parent phase, not later publication stages. The fixture owner recorded these read-only observations when this phase began; they are current until an input changes, so use them instead of re-running those commands, and recheck freshness before completion outputs:\n${fixture.startObservations}\n Read README.md for the project contract and commands. Use diagnostic-client commands such as \`bun scripts/probe.ts \` for exploratory discoveries and their checkpoint evidence. A required \`bun run test\` is separate suite verification: report it as verification, never as a diagnostic observation or checkpoint anchor/target. Use the production evidence helper to publish each diagnostic checkpoint as \`reports/exploration-NNN.json\`, not inside a nested directory; do not transcribe its observed payload. ${fixture.caseId === 'ship-exploratory-plan-checks' ? 'The previously discovered plan is PLAN.md.' : 'No plan file was found.'} There is no remote service and no release publication is authorized. There is no interactive approver; do not invent answers or permission. Keep normal parent decision gates. Before the completion report and each completed:true review record, read HANDOFF.md and reports/HANDOFF.md if present for any concurrent collaborator update, await the results, and compare evidence with current inputs. If a probe's snapshot differs from an earlier probe's, read reports/HANDOFF.md first; it names any collaborator change. A gstack-review-log completed:true record is a completion, not preliminary bookkeeping; a later handoff read cannot validate an earlier completion.\n\nDeadline bookkeeping additionally permits \`bun ${fixture.runtime}/bin/gstack-qa-deadline start ${fixture.cwd}/reports/deadline.json SECONDS [EARLIER_UTC]\`, \`bun ${fixture.runtime}/bin/gstack-qa-deadline status ${fixture.cwd}/reports/deadline.json\`, and \`bun ${fixture.runtime}/bin/gstack-qa-deadline run ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These are closed literal forms: SECONDS must be positive and at most 300, EARLIER_UTC is the optional caller absolute deadline: use the section clock's Hard deadline UTC, never its Runner entry UTC, reserve-start time or a clock-read time. The child is only the existing diagnostic client with zero or one literal argument. Resolve these exact helper and state paths; do not use variables, another helper, another state file, nested wrappers, scripts, operators or substitutions. Only this helper may create or change reports/deadline.json and its .qa-deadline- temporary files; never use Write/Edit/MultiEdit on those paths. Record the full outer run command in checkpoints and evidence; keep the unchanged child JSON as observed, separate from prefixed guard diagnostics. A completed expired guard-run is not a probe or a pass: retain its unused checkpoint, report not-run coverage and do not restart the deadline. Keep the 12-probe smoke limit. Required suites and explicit plan checks are outside the bounded smoke budget, not permission to reset it.\n\nFunctional evidence uses the same production helper and existing diagnostic client: \`bun ${fixture.runtime}/bin/gstack-qa-evidence capture ${fixture.cwd}/reports NNN --public --deadline ${fixture.cwd}/reports/deadline.json -- bun scripts/probe.ts [literal]\`. These diagnostic receipts are declared public/synthetic, so --public is approved; a fresh three-digit ID is required each time. Explicit plan probes outside the smoke budget may replace --deadline with --timeout-ms 10000; this does not reset or bypass the smoke deadline. After the first capture, put causal intent in the next capture itself by adding \`--after PREV --hypothesis 'causal hypothesis'\` before \`--\` (PREV = the most recent complete capture ID); the helper publishes reports/exploration-NNN.json linking capture PREV's observation to that probe before running it, so one call is both checkpoint and probe. The separate form stays valid: \`bun ${fixture.runtime}/bin/gstack-qa-evidence checkpoint ${fixture.cwd}/reports NNN CAPTURE_ID 'full prior capture command' 'causal hypothesis' 'full next capture command'\`, or Write only capture, observationCommand, hypothesis and nextCommand to reports/intent.json and run checkpoint REPORT_ROOT NNN intent.json. Quote arguments literally. Materialize is supported when evidence.json is required. Sources stay inside reports. Decide to execute the next probe before publishing its checkpoint, then await successful publication and dispatch that exact probe. If you defer an optional idea or stop exploration, do not publish a checkpoint for it; describe it as not run in Markdown. An unused checkpoint requires an actual authenticated expired-capture result; nearing the deadline or choosing to stop is not enough. A complete capture can truthfully retain a nonzero domain exit (including expected rejection or unavailable dependency); it is not automatically a pass. Wrapper-failed, interrupted or incomplete captures are not observations or passes; an authenticated expired capture can retain an unused checkpoint as not-run evidence. No new native child command or shell authority is granted.\n\nThe supported Bash interface is one literal documented native command or one exact generated workflow shell block with main substituted for and PLAN.md for ; even read-only commands must not be chained except for the exact DIFF_BASE preface below. Native CLI commands accept no argument or one literal argument: an unquoted shell-safe word, single-quoted text, or double-quoted text without expansion or escapes; no multiline arguments or shell composition. Inventory commands are pwd, bun --version, and ls with an optional combined -l/-a flag, optional -- separator, and literal path operands using the same quoting rules; operands beginning with - require --. Listing never permits other options, glob expansion, substitution, redirection or composition. The supported Git forms are git status --short, git status --porcelain, git branch --show-current, git rev-parse HEAD, git rev-parse --short HEAD, git merge-base origin/main HEAD, the plan-completion section's git log origin/main..HEAD --oneline, git diff (optional origin/main base and at most one output mode: --stat, --numstat, --name-only, or --name-status, before or after the base; optional literal pathspec arguments after --), git ls-files, and git ls-files --others --exclude-standard. Diff pathspecs use the same literal argument syntax as the CLI; quote globs so Git, not the shell, interprets them. The only variable-base form is DIFF_BASE=$(git merge-base origin/main HEAD) && git diff with exactly the double-quoted "$DIFF_BASE" base, the same optional output mode and literal pathspecs. That exact preface recomputes the base in the same command; other assignments and variable expansions are forbidden; the only additional command substitutions are the two installed bookkeeping fields described below. No other Git options, configuration overrides, --output, --no-index, --ext-diff, --textconv, external helpers or mutations are authorized. Exact generated workflow blocks remain allowed. Review bookkeeping permits the installed gstack-review-log start commands and single-quoted JSON records for review/adversarial-review (the installed quote-spliced double-quoted $(date -u +%Y-%m-%dT%H:%M:%SZ) is permitted only as the complete top-level timestamp value, and $(git rev-parse --short HEAD) only as the complete top-level commit value; no other substitutions, field placements or shell evaluation), gstack-review-read, gstack-specialist-stats, plus date -u +%Y-%m-%dT%H:%M:%SZ; use an original start token for a started attempt and completed:false/converged:false without a token for an unstarted one. These installed helpers are shell scripts: run them by exact installed path, never through bun or another interpreter. A review record fills this installed template, keeping its keys and adding none: \`${callerReviewRecordTemplate(fixture)}\`. A review record's status is clean only for a completed review with zero unresolved defects, otherwise issues_found${fixture.caller === 'ship' ? ' (unavailable for missing dispatched reviewer output)' : ''}; a review stopped at a gate records completed:false, never a receipt status such as blocked. Bookkeeping has an owned state directory and private Git object store; the real product, index, refs and config must stay unchanged. Outside providers are unavailable in this isolated fixture; the native reviewer is still required.${fixture.child ? ` Subagent dispatch uses a deterministic child transport instead of Agent/Task: when the workflow dispatches ${fixture.caller === 'review' ? 'the native adversarial reviewer (Step 4.8)' : 'the plan completion audit (Step 8)'}, complete its dispatch prerequisites, do not invoke Agent or Task, and Read ${fixture.child} as that child's complete final output; apply the parent's normal handling to it.` : ''} Other interpreters, shell composition outside those generated blocks or the exact DIFF_BASE preface, global setup and external operations are not authorized. Use Read/Glob/Grep for discovery instead of cat/head/tail; use native Write/Edit tools for any authorized file output. Pass these same command and write boundaries to any child; generated shell fragments remain verbatim apart from the declared substitutions.\n\nWrite the phase report to reports/review.md. Keep it compact (aim under 300 words): retain each required heading and field, link to reports/evidence.json and the checkpoints for details already recorded there, and do not restate the diff, the evidence rows or the review record. Also save reports/receipt.json with this machine-readable result shape: {\"status\":\"pass|fail|blocked|inconclusive\",\"probes\":[\"observed diagnostic receipt ids\"],\"remaining\":[\"names of incomplete contracts\"]}. The status is the overall supplied phase gate, not whether some probes passed: copy it from the verdict.status that materialize prints and writes to reports/evidence.json, and if that verdict is not pass, rerun what it lists as open before finishing when a pass is required. The probes array contains the exact id values from the captured child JSON (probe-...), never the helper's three-digit capture IDs or QA_EVIDENCE.id. Capture IDs select stored observations for checkpoint/materialize; child diagnostic IDs identify the actual probes in this final receipt. Read the child JSON's id for each probe used as evidence; do not derive it from a filename or execution counter. remaining names unresolved required contracts or gates in this phase. Pass requires no remaining required contracts or gates. Optional unavailable providers and later stages outside this excerpt are not required remainder. This format does not establish that any work succeeded. Stop at the end of this phase, or at its first unresolved approval gate.`, appendSystemPrompt: `Caller execution scheduling (fixture contract): This session has at most 25 assistant turns, including required verification and final artifacts. The command boundary applies to each Bash call, not to the number of independent tool calls in an assistant turn. -After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. For example, after the first clock read, one response can Read the caller excerpt, README.md, every product file and the referenced checklist and templates. Wait for their results before decisions that depend on them. +After required clock and approval prerequisites settle, issue independent source Reads and read-only discovery together as separate native tool calls once their paths and inputs are known. Read-only discovery is Read, Glob, Grep and the declared inventory, Git and review-read forms; a capture, a deadline guard start or run, or a diagnostic-client run is a probe, never discovery. For example, after the first clock read, one response can Read the caller excerpt, README.md, every product file and the referenced checklist and templates. Wait for their results before decisions that depend on them. The completion reserve is for required verification, affected-input revalidation and artifacts, not an earlier deadline. Keep completing required work within the actual remaining deadline; reserve entry alone is not a reason to stop. Use each native probe's snapshot to distinguish current from superseded evidence before deciding which checks still need revalidation. Never group diagnostic probes, checkpoint publication with its next probe (a capture carrying --after is one probe, not a group), or any action with the clock/status/approval result it needs. Shell composition remains forbidden outside the declared forms. Preserve every required Read, probe, verification, freshness check, approval and report field; the turn limit does not authorize skipping work or reporting incomplete work as passed.`, workingDirectory: fixture.cwd, diff --git a/test/qa-exploratory-callers.test.ts b/test/qa-exploratory-callers.test.ts index 395cc3ac2..d66a9b8c6 100644 --- a/test/qa-exploratory-callers.test.ts +++ b/test/qa-exploratory-callers.test.ts @@ -108,6 +108,53 @@ describe('caller native-event observer controls', () => { ['exploratory', 'system-functional'].map(id => [id, ready.find(tool => String(tool.input.file_path).endsWith(`/qa/sections/${id}.md`))] as const))).toEqual([]); }); + test('census 38212265208: a probe batched with the caller section Read is flagged, and the scheduling contract never calls a probe discovery', async () => { + // Native order of census 38212265208 (gate-census-6, ship-exploratory-late-input): provider + // message, tool, input; outputs abbreviated. Capture 001 shares a response with the Step 9 section Read. + const R = '/home/runner/.cache/gstack-paid-shard-iHlpIM/tmp/qc-cALQYz'; + const capture = (id: string, extra = '') => `bun ${R}/host/runtime/bin/gstack-qa-evidence capture ${R}/product/reports ${id} --public --deadline ${R}/product/reports/deadline.json ${extra}-- bun scripts/probe.ts`; + const replay: Array<[string, string, Record, string]> = [ + ['YvqEApde', 'Read', { file_path: `${R}/host/runtime/ship/sections/plan-completion.md` }, '## Step 8: Plan Completion Audit'], + ['YvqEApde', 'Read', { file_path: `${R}/host/runtime/ship/sections/review-army.md` }, '## Step 9: Pre-Landing Review'], + ['YvqEApde', 'Bash', { command: `${capture('001')} 3` }, '{"id":"probe-b5ae2d9c-af87-4e8a-9aab-76972f826079","charter":"happy","input":"3"}'], + ['w9Gt4kmL', 'Read', { file_path: `${R}/host/runtime/qa/sections/exploratory.md` }, '# Shared exploratory QA'], + ['w9Gt4kmL', 'Read', { file_path: `${R}/host/runtime/bin/gstack-qa-evidence` }, '#!/usr/bin/env bun'], + ['w9Gt4kmL', 'Bash', { command: `${R}/host/runtime/bin/gstack-review-read` }, 'NO_REVIEWS'], + ['w9Gt4kmL', 'Bash', { command: `${R}/host/runtime/bin/gstack-specialist-stats` }, 'SPECIALIST_STATS: 0 reviews analyzed'], + ['cba4VKcQ', 'Bash', { command: `${capture('002', "--after 001 --hypothesis 'upper bound 9 should still print 18' ")} 9` }, '{"id":"probe-baf59bfd-ac54-4a4a-8bec-2cd416154bbf","charter":"plan:nine","input":"9"}'], + ['cba4VKcQ', 'Read', { file_path: `${R}/host/runtime/qa/sections/scope.md` }, '### Select the surface before setup'], + ['cba4VKcQ', 'Read', { file_path: `${R}/host/runtime/qa/sections/system-functional.md` }, '# Functional QA with repository-native tools'], + ]; + const replayTools = (rows: typeof replay) => callerTools(rows.flatMap(([message, name, input, output], i) => + nativeCall(`toolu_${i}`, name, input, output).map((event: any) => event.type === 'assistant' + ? { ...event, session_id: 'census-38212265208', message: { ...event.message, id: `msg_${message}` } } : event))); + const probeErrors = (tools: ReturnType) => tools.filter(tool => / capture /.test(String(tool.input.command))) + .flatMap(probe => callerProbeReadErrors(probe, ['exploratory', 'system-functional'] + .map(id => [id, tools.find(tool => String(tool.input.file_path).endsWith(`/qa/sections/${id}.md`))] as const))); + const tools = replayTools(replay); + const section = tools.find(tool => String(tool.input.file_path).endsWith('/ship/sections/review-army.md'))!; + expect(tools.find(tool => / capture /.test(String(tool.input.command)))!.messageId).toBe(section.messageId); + expect(probeErrors(tools)).toEqual([ + 'probe preceded resource read: exploratory', 'probe preceded resource read: system-functional', + 'probe preceded resource read: system-functional', + ]); + const isProbe = ([, , input]: (typeof replay)[number]) => / capture /.test(input.command ?? ''); + expect(probeErrors(replayTools([...replay.filter(row => !isProbe(row)), + ...replay.filter(isProbe).map(([message, ...rest]) => [`${message}-after-reads`, ...rest] as (typeof replay)[number])]))).toEqual([]); + + const fixture = createQaCallerFixture('ship-exploratory-late-input'); + try { + await runQaCaller(fixture, 'free-read-probe-contract', async options => { + const contract = options.appendSystemPrompt!; + const discovery = contract.match(/Read-only discovery is [^.]*;([^.]*)\./); + expect(discovery?.[1]).toMatch(/capture/); + expect(discovery?.[1]).toMatch(/guard start/); + expect(discovery?.[1]).toMatch(/never discovery/); + return { exitReason: 'error_max_turns', transcript: [] } as unknown as SkillTestResult; + }); + } finally { await fixture.close(); fs.rmSync(fixture.root, { recursive: true, force: true }); } + }); + test.each(['pending-result', 'same-event', 'same-message-id'])('method Read %s cannot authorize a dependent probe', ordering => { const observed = evidence(); const events = observed.result.transcript as any[]; diff --git a/test/review-army-budget.test.ts b/test/review-army-budget.test.ts index 30eb4ac1f..b5458f5c7 100644 --- a/test/review-army-budget.test.ts +++ b/test/review-army-budget.test.ts @@ -2,6 +2,7 @@ import { expect, test } from 'bun:test'; import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; +import { spawnSync } from 'node:child_process'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -158,3 +159,62 @@ afterAll(async () => { } }, 10_000); } + +// review-army-simplification headroom (#3153): the session's premise matches its staged +// diff and the fixture stages only the Review Army section and its one checklist. +test.each(['structure-finding', 'no-structure-finding', 'disavowed-frame']) + ('review-army-simplification registered session contract: %s', scenario => { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'simplification-contract-')); + const facts = path.join(dir, 'facts.json'); + const script = path.join(dir, 'callback.test.ts'); + fs.writeFileSync(script, ` +import {describe,mock,test} from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import {spawnSync} from 'node:child_process'; +const root=${JSON.stringify(ROOT)},scenario=${JSON.stringify(scenario)},facts=${JSON.stringify(facts)}; +const actual=await import(path.join(root,'test/helpers/session-runner.ts')); +let capturedRun={}; +mock.module(path.join(root,'test/helpers/e2e-helpers.ts'),()=>({ + ROOT:root,runId:'simplification-contract', + describeIfSelected:(name,ids,body)=>{if(ids.includes('review-army-simplification'))describe(name,body);}, + testConcurrentIfSelected:(id,body,timeout)=>{if(id==='review-army-simplification')test(id,body,timeout);}, + logCost(){},createEvalCollector:()=>null,finalizeEvalCollector:async()=>{}, + recordE2E:(_collector,_name,_suite,result)=>fs.writeFileSync(facts,JSON.stringify({...capturedRun,exitReason:result.exitReason})), +})); +mock.module(path.join(root,'test/helpers/session-runner.ts'),()=>({...actual,runSkillTest:async options=>{ + const cwd=options.workingDirectory; + const shortstat=spawnSync('git',['diff','--shortstat','main...HEAD'],{cwd,encoding:'utf8',timeout:5000}).stdout; + const staged=spawnSync('git',['ls-files','--others','--exclude-standard'],{cwd,encoding:'utf8',timeout:5000}).stdout.split('\\n').filter(Boolean).sort(); + capturedRun={prompt:options.prompt,timeout:options.timeout,maxTurns:options.maxTurns,staged, + diffLines:[...shortstat.matchAll(/(\\d+) (?:insertion|deletion)/g)].reduce((sum,m)=>sum+Number(m[1]),0)}; + fs.writeFileSync(path.join(cwd,'review-output.md'),{ + 'structure-finding':'{"category":"native","path":"date_utils.js","line":7} DateFormatter re-implements Intl.DateTimeFormat.', + 'no-structure-finding':'Simplification: nothing to report.', + 'disavowed-frame':'{"category":"native"} Lean already. Ship.', + }[scenario]); + return {exitReason:'success',browseErrors:[],toolCalls:[],transcript:[],output:'Saved.'}; +}})); +await import(path.join(root,'test/skill-e2e-review-army.test.ts')); +`); + try { + const child = spawnSync(process.execPath, ['test', script], { + cwd: dir, encoding: 'utf8', timeout: 20_000, + env: {PATH: process.env.PATH ?? '', HOME: dir, TMPDIR: dir, NO_COLOR: '1'}, + }); + expect(child.status, child.stdout + child.stderr).toBe(scenario === 'structure-finding' ? 0 : 1); + expect(fs.existsSync(facts), child.stdout + child.stderr).toBe(true); + const observed = JSON.parse(fs.readFileSync(facts, 'utf8')); + expect(observed.timeout).toBe(180_000); + expect(observed.maxTurns).toBe(20); + expect(observed.diffLines).toBe(68); + expect(observed.prompt).toContain(`DIFF_LINES: ${observed.diffLines}`); + expect(observed.prompt).not.toMatch(/over 100 lines/i); + expect(observed.prompt).toContain('/review --simplification'); + expect(observed.staged).toEqual(['review-army.md', 'review-specialists/simplification.md']); + expect(observed.prompt).toContain('at most 30 lines'); + expect(observed.prompt).toContain('brief acknowledgement'); + } finally { + fs.rmSync(dir, {recursive: true, force: true}); + } + }); diff --git a/test/skill-e2e-review-army.test.ts b/test/skill-e2e-review-army.test.ts index e31410dde..6e7e905e0 100644 --- a/test/skill-e2e-review-army.test.ts +++ b/test/skill-e2e-review-army.test.ts @@ -752,6 +752,7 @@ Write ${dir}/review-output.md with only the selection line, the SPECIALIST REVIE describeIfSelected('Review Army: Simplification activation', ['review-army-simplification'], () => { let dir: string; + let observations: string; beforeAll(() => { const repo = setupRepo('army-simplification'); @@ -769,24 +770,27 @@ describeIfSelected('Review Army: Simplification activation', ['review-army-simpl repo.run('git', ['add', '.']); repo.run('git', ['commit', '-m', 'add date utils']); - copyReviewFiles(dir); + observations = stageReviewArmySession(dir, `SCOPE_FRONTEND=false +SCOPE_BACKEND=true +SCOPE_PROMPTS=false +SCOPE_TESTS=false +SCOPE_DOCS=false +SCOPE_CONFIG=false +SCOPE_MIGRATIONS=false +SCOPE_API=false +SCOPE_AUTH=false`, ['simplification'], ['simplification']); }); afterAll(() => { try { fs.rmSync(dir, { recursive: true, force: true }); } catch {} }); testConcurrentIfSelected('review-army-simplification', async () => { const result = await runSkillTest({ - prompt: `You are in a git repo on a feature branch that adds a JS utility file. -Read review-SKILL.md for instructions. Also read review-checklist.md. -The specialist checklists are in review-specialists/ (testing.md, simplification.md, etc.). + prompt: `You are the /review parent on branch feature/date-utils, a JS utility change. The caller invoked /review --simplification. +${reviewArmyScope(observations, 'its Step 4.6 merge')} +Selection: --simplification force-includes the Simplification specialist; the recorded DIFF_LINES is below its size gate, so only the flag selects it. Dispatch no other specialist. +The checklist for this capture is review-specialists/simplification.md. Instead of launching a subagent, read it and apply it yourself against the recorded diff. -Skip the preamble, lake intro, telemetry sections. -Run Step 4.5 (Review Army) only. -The base branch is main. The diff is over 100 lines, so the Simplification specialist should activate. - -For the specialist dispatch, read review-specialists/simplification.md and apply it against the diff. - -Write your findings to ${dir}/review-output.md`, +Write ${dir}/review-output.md with only the selection line and the SPECIALIST REVIEW block with its PR Quality Score and Simplification footer, in at most 30 lines. After saving the report, finish with a brief acknowledgement rather than repeating the findings in the final response.`, workingDirectory: dir, maxTurns: 20, timeout: 180_000,