docs(designs): fix wave 3 plan revision 2 (gate rev 1 approved: rec)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-17 19:47:02 +00:00
1 parent 0f64a54ad8
commit 1aa060b1e4
2 files changed
+80 -235

No files matched your search

@@ -1,175 +0,0 @@
<!-- gstack-autoplan-scope: --no-ui --developer-tool -->
# Fix wave 3 (October 17, 2026): PR eval cost (W2) and paid-gate reliability round 2 (W3)
## Implementation plan
### Context
This is plan revision 2. Revision 1 was reviewed by six voices (CEO, DX, Eng; Claude Opus 5.5 native, GPT-6.1 Sol outside): 67 findings, 1 Critical, 8 user challenges. Gate rev 1 recommendations folded in here: d3b (split the W2 PR into A1, A2, A3), d4b (the exercised-path map ships in shadow mode), d5b/d8b (keep superseded/open semantics; the repair is a rerun requirement), d6b/d7b (a tested keyless pregate path before `remote_gate` is declared); d1a and d2a keep the direction. Every other finding is a build obligation, folded into the item it names.
Owner request: Garry approved GSTA-41's triage on 2026-10-17 with "Accept all recommendations", which includes "Autoplan W2 and W3 together now" (reports/triage-2026-10-17.md in the GStack Minder drive). Coordinator: GStack Minder (GSTA-36) owns the merge queue, version slots and every merge; this wave opens PRs and sends READY reports unstamped. Builder: GSTA-45.
Base: origin/main (v1.91.125.0 at revision time). Live owners this wave must not collide with: GSTA-46 (W5: lib/pregate/import-graph.ts, lib/pregate/checks/lanes.ts, lib/regen.ts, test/pregate-selection.test.ts, test/helpers/pregate-fixture.ts), GSTA-33 (W4, #3253: the duration-record workflow, windows-free-tests.yml, scripts/test-pr-profile.ts, docs; it does not touch scripts/test-free-shards.ts, per GStack Minder), GSTA-44 (#3220 brain-sync). A1 and A2 also edit scripts/test-pr-profile.ts, so whichever of #3253 and A1 lands second merges main. Out of scope: W4, W5, W6-W9.
Evidence base. Read-only investigations on 521fd42 (no paid spend):
- **W2:** 225 evals.yml pull_request runs with a paid-plan manifest over 10 days (2026-10-07 to 2026-10-17), every manifest, trial-outcomes artifact and verdict line, each run's changed files replayed through today's selector (`computePaidCaseSelection`), counterfactual replays of each fix, and a prototype exercised-path map built from 541 PR runs' recorded commands.
- **W3:** every red verdict in PR gate runs since #3231 (`bun run eval:pass-rates --reds --workflow evals.yml --runs 60`): 21 reds in 17 runs (38969924945 to 39011101113), plus two reds GStack Minder forwarded from #3252. Each rejected tool call was replayed through the real detector on main.
Thesis: **spend the PR gate's money only where a diff can change the outcome, and make a red verdict mean the diff broke something.**
### Release promises
- **P1. Paid selection never under-selects what it can see, and narrowing is earned with evidence.** The shebang blind spot closes now; the exercised-path map runs in shadow, so map drift is measured (MAP_GAP, census) before any narrowing is activated by a later decision. The cost metric tracks dollars and cases per run, not the fallback label.
- **P2. A gstack contributor on a small machine can finish pregate,** with or without a Ubicloud key: under 8 cores the local tier goes to a declared remote gate that runs on Ubicloud when a key exists and as parallel local shards when it does not, and prints a receipt bound to the tree it actually tested.
- **P3. Every open red cause found in the window is fixed at its cause or kept as a named negative control.** No threshold, trial count, case kind or quarantine changes.
- **P4. `/ship` slot allocation does not wait on serial GitHub calls,** and a fetch that times out says so.
### Premise table
| premise | claim | receipt | verdict |
|---|---|---|---|
| P-a | 42.5% of PR runs fall back to full selection | test:health 7d 68/160 at triage, 75/177 on 2026-10-17; 10-day replay 95/225 recorded, 91/225 under today's selector | true |
| P-b | touchfile mappings for bin/, lib/, hosts/, make-pdf/ cut the fallback | no file under bin/, lib/, hosts/, make-pdf/, scripts/ or browse/ was an unknown file in any of the 225 runs; the derivation (scripts/pr-dependencies.ts) already places them | **false as stated** |
| P-c | cutting the fallback rate cuts spend | fixing every narrowable trigger takes the rate 40.4% -> 14.2% on replay but predicted spend only -2.5%: every run that leaves full-fallback lands in `dependents`, which selected all 108 gate cases and 24 judges in 100 of 105 such runs | **false** |
| P-d | cost by mode | recorded known cost per run: full-fallback $91.60 (n=61), dependents $80.09 (median $91.06, n=35), pr $14.87 (median $0, n=59); fallback + dependents carry 90.5% of known spend | true |
| P-e | the derived closure reaches almost every case | 91 bin/, 103 lib/, 130 browse/ files reach at least 100 of 108 gate cases; hub edges include remediation prose `/gstack-upgrade` read as a call to bin/gstack-upgrade, string and doc-comment mentions, and real hubs (gstack-config, gstack-skill-start, gstack-pregate) | true |
| P-f | the derivation under-selects shebang files | `Bun.Transpiler.scanImports` throws on a leading `#!` and the error is swallowed; `computePaidCaseSelection(['lib/fs-utils.ts'])` selects 0 cases though bin/gstack-redact-prepush and three other bins import it; 51 extensionless bins and 66 .ts/.js files affected | true (latent; no observed miss) |
| P-g | lockfile-only changes restore every case | bun.lock and package.json are PR_FULL_GATE_FILES; 0 runs in the window fell back on them alone; dependabot runs are skipped by evals.yml (`github.actor != 'dependabot[bot]'`, lines 53/181/253); every runtime package's importer reaches all cases or is global | true; narrowing buys 0 runs |
| P-h | gstack's pregate has no remote gate | .gstack/pregate.json has no `remote_gate`; lanes.ts:207 needs `remote_gate` or a non-local test_backend; GSTA-35's 798-file local lane passed 3,550 s on 4 cores | true |
| P-i | gstack-next-version fetches serially | bin/gstack-next-version:90 `GH_API_CONCURRENCY = 10`, workers call `runCommand` -> `spawnSync` (:111); 8 stubbed PRs at 0.5 s took 4,160 ms (TODOS.md fix-wave-2 entry) | true |
| P-j | createQAFunctionalFixture lacks git maintenance config | test/helpers/qa-functional-fixture.ts:279-280 sets `gc.auto 0` and `maintenance.auto false` since 0bf9d50 (#3190); qa-callers-fixture.ts:594-595 too | **false (already fixed)**; the TODOS entry is stale |
| P-k | the webhook-report red was `bun test` / `git status` | both are declared (qa-functional-observer.ts:301); the two denied calls were `cat package.json probe.ts .gitignore` and `cat src/*.ts test/*.ts` | **false**; genuine model miss |
| P-l | autoplan-answer-invalidation runs near 5 s | the eng-only override test (:168, no explicit timeout) took 4,137 ms on 4 cores; file 7.8-9.4 s | true |
unverified=0
### Already done
Six of the 21 window reds are fixed on main with regressions: ship-docsync-store x2 (f592d67, c71d764), qa-only-no-fix (f592d67), ship-exploratory-plan-checks 38981523623 (f592d67), tpa-absent-linux 38988927989 (73f1394), qa-functional-webhook-fix 38974669950 (0bf9d50, maintenance config). Main already fixed 4 of the 95 recorded fallback runs (#3231 detector-corpora fixtures, #3125 and #3121 .gstack/* classification).
### Surface check
Verified by source: `selectPrProfile`, `FREE_ONLY_PR_FILES`, `PR_FULL_GATE_FILES`, `DERIVABLE_PREFIXES` (scripts/test-pr-profile.ts); `SELECTION_DATA_MODULES`, `refs` (:178), the global walk (:242), `targetsOf` (:250) (scripts/pr-dependencies.ts); `remote_gate`, the `PREGATE_REMOTE:` contract, `PREGATE_TREE`/`PREGATE_FILES_FILE`/`PREGATE_SPEND_CAP_USD`, exported `parseRemoteReport`/`remoteReportProblem` (lib/pregate/checks/lanes.ts:142-155, read-only here); `ubi-runner.sh run --src --setup --env --pull` and scripts/ubicloud/test-free.sh; `test-free-shards.ts` flags (no file-subset flag today; mode conflicts at :597-601); `gstack-docs-candidate` top-level `--help` only (:91) and `compare` schema check (:123). New surfaces are named as new.
### Binding decisions
- Every merge is GStack Minder's; READY reports go unstamped; never stamp before a slot call (CREW.md 6).
- No threshold, trial count, case kind or quarantine changes (EVAL_POLICY; AGENTS.md Validation discipline step 6). A detector change keeps or adds negative controls. Each PR body carries a gate-edit table (file, old vs new acceptance, negative control) and a `## Behavior changes` list.
- Do not edit files GSTA-46 (W5) owns; lib/pregate/** is read-only in this wave.
- Paid PR gates are pre-approved (Garry, 2026-10-15).
- Activating the exercised-path map (changing what a PR selects) is a later decision with its shadow data, not part of this wave.
### Numbers
- W2 replay: today 91/225 full-fallback (40.4%), mean 79.6 gate cases per run. Sound classification fixes alone: 36/225 full-fallback (16.0%) but no spend change, because those runs land in dependents mode, which still selects about 105 cases.
- Exercised-path map, replayed over the 225 runs: applied to every changed file it misses **0 of the 47 recorded gate reds** and cuts predicted spend **6.5%**; on the recorded basis ($6,811 trial-sum per 10 days, $101.66 per merged PR) that is about $440-600 per 10 days. Applied only to unmapped files it cuts 15% but **misses 5 reds** (rejected). Making free-fixtures-data.ts non-global without the map misses 3 #3231 reds, so that change waits for map activation.
- The floor: 46 files (32 bin, 11 lib, 3 other) reach 100-106 of 108 cases under any sound map, because every generated SKILL.md runs the preamble bins (gstack-skill-start alone closes over 73 files). Below that needs a cheaper contract for preamble-hub diffs (deferred ask D1).
- W3: of the 17 red runs, 4 are already green on main (38969924945, 38974669950, 38981523623, 38988927989; 6 reds fixed). W3 fixes the cause in 7 more (38973329985, 38986120559, 38990111230, 38993076767, 39002269281, 39004466422, 39011101113) and the detector misread in 38980005793. Still red after W3: 38976253774 (U, undetermined hang), 38980005793 (T, timeout tail), 38983306684 and 38984977679 (D3 model misses), 39006896632 (Q3 model miss), and 39001107574 (C) unless the W3-9 hardening changes behaviour: 4 + 7 + 6 = 17.
### Wave items
#### W2-A1: selection correctness and safe classification (PR A1; edits global touchfiles, so a full ~$93 gate)
**A1-1. Close the shebang blind spot.** In scripts/pr-dependencies.ts, strip a leading `#!` line before `scanImports` at all three sites (`refs` :178, the global walk :242, `targetsOf` :250) and treat bun/node-shebang extensionless files as code. Any other scan error keeps failing closed (selects everything) at all three sites. This broadens selection: 34 files flip from no-consumer to consumed; test/helpers/bootstrap-retention.ts joins the global set through scripts/test-paid-shards.ts. Free tests: `lib/fs-utils.ts` selects the cases of bin/gstack-redact-prepush; a shebang fixture's import is found; an injected non-shebang scan error still selects everything.
**A1-2. Safe classification fixes** (none narrows a case a diff can reach; they change labels and fallback reasons).
- FREE_ONLY_PR_FILES gains .github/dependabot.yml, .osv-scanner.toml, setup.ps1, .gitlab-ci.yml, .github/ISSUE_TEMPLATE/**, .gstack/ship-policy.example.json, each with its one-line reason (readers: osv-scanner.yml and free tests; windows-setup-e2e.yml; free gstack-doctor.test.ts:363). test/paid-free-boundary.test.ts:117 swaps its sample unknown file to one that stays unmapped.
- capy/ joins REPO_PATH and DERIVABLE_PREFIXES, so capy/support.env derives to bin/gstack-install-verdict.sh and bin/gstack-doctor-check.sh.
- A full-gate file is never also labelled "real unknown dependency" (fixes wrong fix text for evals.yml, test-setup.ts, .gitattributes).
- Not in A1: `setup` stays a full-gate file (its real executors include bin/gstack-upgrade:131, bin/gstack-capy-install:404-405 and bin/dev-setup:93, which today's matcher cannot see, and moving it saves no money); free-fixtures-data.ts stays global until map activation; generator inputs stay full (D2).
**A1-3. Measure what the money follows.** scripts/test-health-report.ts adds `pr-cases-per-run` (mean selected gate cases and judges per PR run) and `pr-known-cost-per-run` beside `pr-fallback-rate`, and the summary leads with cost per merged PR. The fallback text in docs/TESTING_INTERNALS.md#pr-paid-lane-fallback is corrected (it says bin/ and lib/ changes take the full gate because they are unmapped; they are derived).
**A1-4. Lockfile decision: keep package.json and bun.lock full.** No narrowing is built: no run fell back on them alone, dependabot never reaches the paid lane, and every runtime package's importer reaches all cases. Documented in docs/TESTING_INTERNALS.md#pr-paid-lane-fallback with the evidence.
#### W2-A2: exercised-path map in shadow mode (PR A2, stacked on A1; full gate)
**A2-1. The map.** A new generated, committed `test/helpers/exercised-commands.json` holds per gate case the union of commands observed in recorded trials, with `generatedAt`, `observedThrough`, the source run ids and per-case observation counts. It is registered as a SELECTION_DATA_MODULE with map-diff selection like touchfiles-data.ts (a refresh selects only cases whose set changed, failing closed on error), so a refresh never pays the full gate. Refreshes are union-only: an observation never removes an edge, and missing or empty artifacts never erase consumers; removal is a reviewed edit.
**A2-2. The consumer function** (scripts/pr-dependencies.ts, new `exercisedConsumers`), the union of: (1) direct touchfile matches; (2) the case's test file and helper import closure; (3) executable shell in the case's loaded skill (template and generated SKILL.md) and in bin/, lib/, scripts/: command-position `gstack-*` names and repo paths, `.`/`source` edges, `./setup` / `$ROOT/setup` exec edges, quoted path literals, path.join pairs, data-file paths, `bun run` scripts, compiled outfiles (`browse/dist/browse`, `design/dist/design`, `make-pdf/dist/pdf`, `$B`, `$D`, find-browse) mapped to their source trees, and shard perTest attribution; comments, echo/printf/die messages, case patterns, bare `VAR=gstack-x` strings and slash-command skill names are not edges, but command substitutions and executable or expanding heredocs are kept, and unparsed shell keeps its conservative edges; (4) observed bins from A2-1. The edge classes are ported from the measured prototype (exercised.ts), and a parity test pins that the shipped function equals or is a reviewed superset of the prototype's map at a pinned commit. PR_FULL_GATE_FILES and GLOBAL_TOUCHFILES stay full, and the closure stops at them.
**A2-3. Shadow, not selection.** `selectPrProfile` keeps today's selection. The paid-plan manifest gains a `shadow` block: the cases the map would select, the cases it would drop, the per-file provenance ("selects N cases: touchfile (…), import (…), skill bash via bin/gstack-x (…), observed (…)") and the predicted cost difference; the plan's coverage summary prints one line of it. test:health reports the shadow saving and shadow misses (a red on a case the shadow would have dropped) per week.
**A2-4. Gap detection.** The paid report job runs the same bin matcher over each executed case's commands and emits `MAP_GAP <case> <bin>` against static-plus-observed consumers (never the observed set alone) as its own non-blocking job result, separate from case verdicts and from the INFRA/INCOMPLETE re-dispatch rule, with the doc anchor docs/TESTING_INTERNALS.md#exercised-path-map. The weekly gate census emits the union-refreshed map as an artifact (built from its own artifacts, no REST downloads: the prototype hit the installation's 403 artifact limit) and prints the refresh command (`bun run test:exercised-map --from-run <id>`), which opens the refresh as an ordinary PR. In shadow mode the census reports gaps and the six PTY cases that record no commands; it does not block. PTY command capture is a named prerequisite for activation, not part of this wave.
Free tests: a synthetic fixture tree with one case per edge class (including the stripped-message negatives and the kept command substitution), the five variant-c miss shapes as negative controls, union-only refresh (disjoint branches, empty artifacts), map-diff selection of a refresh, and shadow output that never changes `e2e`/`judges`. The 47-red 0-miss replay is a pinned PR-body receipt reproducible by script at the PR commit, not a free test (historical file lists rot on renames).
#### W2-A3: small-machine pregate and next-version (PR A3; disjoint from A1/A2)
**A3-1. `scripts/ubicloud/pregate-remote.sh` (new) and `--files-from` (scripts/test-free-shards.ts).**
- Inputs: `PREGATE_FILES_FILE`, `PREGATE_TREE`, `PREGATE_SPEND_CAP_USD` from `gstack-pregate --stage tests --remote`. Run bare (as pregate's `run:` line shows), it exits 2 before any work with "PREGATE_FILES_FILE is not set; run: gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands".
- Before any upload it recomputes the tree with bin/gstack-wtree; on mismatch it prints both tree prefixes, "nothing uploaded, nothing spent" and the rerun command, and exits 3.
- **Keyed path (UBICLOUD_API_KEY set):** refuses before VM create when the worst case (wall timeout × the price of the size actually used, honouring UBI_SIZE) exceeds the cap, printing estimate and cap; wraps scripts/ubicloud/test-free.sh (setup-free-suite.sh, xvfb-run, GSTACK_EXPECT_BINARIES=1, the free retry policy) with `--files-from` and a wall timeout derived from the cap; recomputes gstack-wtree on the VM after unpack and refuses on mismatch; verifies the planned set equals the file list (digest) before running; prints `PREGATE_REMOTE: tree=<VM tree> tests=<n> failed=<n> spent=<usd>` from minutes used. Infra failures (create, setup, SSH drop) exit 4 with the ubi-runner message, the VM name and its `down` line or teardown confirmation, the log path and spent= for minutes billed; one retry after a quota message, then stop. Phase timings (create, setup, run) are logged; the target is 10 minutes from `--remote` to a cleared obligation.
- **Keyless path:** runs the same files locally through `test-free-shards.ts --files-from` in parallel shards, prints "no UBICLOUD_API_KEY: running N files locally in parallel shards", and the same honest `PREGATE_REMOTE:` line for the tree tested (recomputed before and after; a moved tree fails), with `spent=0`.
- `--files-from <path>`: one repo-relative path per line, blank lines ignored; rejects before planning unknown paths (first 5 plus a count, "not in the free planned set", fix: use paths from `--list`), an empty list and malformed lines; conflicts with `--quick`, `--windows-only`, `--list` and the `--ci-*` modes like the parser's existing conflicts.
- .gstack/pregate.json declares `"remote_gate": "bash scripts/ubicloud/pregate-remote.sh"` in the same PR (effective once on main). docs/pregate.md#remote-gate gets a gstack example and CONTRIBUTING.md "Which command do I run?" a small-machine pregate row with keyed and keyless commands. The READY report carries a one-paragraph crew rule for GStack Minder's CREW.md.
- Free tests: the output contract against a stub ubi-runner through the exported `parseRemoteReport`/`remoteReportProblem`; bare run, tree mismatch before and after, zero tests, failures, cap refusal with no VM created, create/setup failure; `--files-from` exact planning and every rejection message; the keyless path on a two-file list. One real Ubicloud run of the 798-file selection is recorded in the PR body.
**A3-2. Concurrent next-version fetches.** bin/gstack-next-version's workers use async `spawn` bounded at GH_API_CONCURRENCY, each with its 15 s timeout. Results and warnings are sorted by PR number. A timeout keeps its cause: "PR #N: VERSION fetch timed out after 15 s; claim unknown, the slot may collide; rerun, then ask the coordinator", and the JSON gains `unknown_claims`; the fork/path hint stays for not-found failures only. Free tests with a delayed fake `gh` on PATH: 8 PRs at 0.5 s finish in under 2 s with the same claims as before; a randomized-delay run is deterministic; a timed-out fetch prints the timeout text and counts in `unknown_claims`.
#### W3: paid-gate reliability, round 2 (PR B; prompt and fixture edits select many cases; ~$93 gate plus focused cases)
Each item names the cause class: (1) fixture/detector defect, (2) skill or prompt defect, (4) genuine model miss.
**W3-1. Docsync fault actor uses the production candidate protocol (1; 38990111230; 79 of 198 docsync captures).** test/helpers/docsync-fault-actor.ts `prepare` (:81-110) writes `docsCandidate()` records that the installed `gstack-docs-candidate compare` rejects (bin/gstack-docs-candidate:123), while ship/sections/documentation.md:70 requires running compare. Migrate prepare, dispatch observations, inspect, seeded prompt comparisons and legacy checks together to the production schema and hash representation: prepare runs the installed `snapshot` under a bounded timeout and prints its exact argv so the post-child rerun is literal, publishes state only after a valid candidate exists, and never overwrites a saved attempt. Free tests: the full adapter sequence (prepare, installed compare, inspect) for unchanged inputs, a doc edit, an index change and a deletion; a snapshot failure before and after candidate creation.
**W3-2. gstack-docs-candidate help and errors (1).** `snapshot --help` and `compare --help` print usage and exit 0 before any side effect; test/helpers/docsync-observer.ts:272-274 admits `<candidate> (snapshot|compare) --help` with nothing else. The compare reject names the path, the missing or wrong fields (schema_version, base, hashes) and the fix (`gstack-docs-candidate snapshot --out <path> --audit-id <id> --mode edit --base <rev>`), keeping exit 1. Free regression: the captured 38990111230 ndjson (redacted) replays to `[]`; `snapshot --help --out x.json`, `compare --help x.json` and an in-repo `--out` stay rejected; corpus rows pinning the old reject text are updated.
**W3-3. Docsync model misses become negative controls (4; 38983306684, 38984977679).** The captured Bash `grep -n -i …` and `cat …; cat …` join the docsync detector corpus as `refuse` rows. No detector change.
**W3-4. qa-callers prompt stops teaching helper syntax (2; 38973329985, 38993076767).** test/helpers/qa-callers-fixture.ts:700 hands over full capture/checkpoint syntax but only "Materialize is supported…"; both reds skipped qa/sections/exploratory.md and system-functional.md then ran a bare `materialize`, while all 83 passing sessions read both. The prompt declares only fixture bindings (evidence tool path, reports dir, deadline file, `--public` approved, `--timeout-ms 10000` for plan probes) and points to the section for the forms. The read requirement stays. Free tests: the two captured transcripts still fail `validateCallerEvidence`; the prompt contains no `gstack-qa-evidence capture` grammar.
**W3-5. State-machine scope in the generated scope block (2; 39002269281).** scripts/resolvers/review-army.ts:75 names `gstack-state-matrix detect --base <base>` in prose and small diffs keep only State Machines (:66), so agents run an undeclared command. The block (:27) gains `source <(… gstack-state-matrix detect --base <base> 2>/dev/null)` with its exit status checked: on failure it prints `SCOPE_STATE_MACHINE=unknown` and the State Machines specialist is included conservatively, with the detector's stderr kept. :75 reads "if SCOPE_STATE_MACHINE is true or unknown (printed by the scope block)", and its "pipe output to `check -`" step becomes a declared block form. Regenerate. test/review-workflow-clarity.test.ts pins both; a free test runs the block with a failing detector stub and sees `unknown`; the captured standalone call stays outside the caller interface.
**W3-6. Webhook fix runs re-verify the repaired invariant within the budget (2; 38986120559).** Stage 2 of the fix-run prompt (test/helpers/qa-functional-eval.ts:33) asks only for original, happy, cancel and dependency, while the verifier (qa-functional-evidence.ts:224) re-runs all 8 scenarios. The prompt states the budget split up front: at most 5 pre-repair captures, then after the final source edit every scenario of the repaired invariant (duplicate, partial, concurrent-ab, concurrent-ba) plus happy, cancel and dependency, 12 in total; if the allowance cannot establish the defect, report incomplete. `QA_WEBHOOK_REQUIRED_SCENARIOS` is unchanged (no verdict tightening); the post-repair list feeds the probe nudge instead. Free tests: the captured status-only repair applied to a fresh fixture fails `verifyQANativeRegression` on partial; an effect-based repair passes; a pre-repair-only scenario set plus a complete post-repair set exercise the nudge.
**W3-7. Lettered consent options when Aside is absent (2; 39011101113).** scripts/resolvers/third-party-actions.ts:54 fixes letters only for the detected case, and the agent numbered 1/2/3. The absent branch offers "B) gstack's own visible browser, C) manual instructions, D) defer (omit A)". Regenerate. Free test: the contract letters the absent branch; the captured numbered answer is a documented fail example. The test's assertion is unchanged.
**W3-8. Tier-1 reality rows are never n/a (2; 39004466422).** The agent wrote `REALITY: deferred-asks n/a …`, `gstack-plan-reality check` returned verdict=repair, and the repair loop ran past the 300 s capture. scripts/resolvers/plan-reality.ts:59 says tier-1 rows are never n/a and, with no linked issue, deferred-asks is `pass` with a receipt showing none. Regenerate. Free test: the captured line through `check` (pinned message) and a skill-text assertion.
**W3-9. Exploratory stop rule ties to the per-rule probes (2 hardening of a 4; 38983306684, 39001107574).** Neither cli-report red probed a non-digit amount; 32 of the other 34 CLI sessions did, and qa/sections/system-functional.md:33 already requires one probe per input rule. The shared owner, scripts/resolvers/qa.ts, states one rule for both exploratory templates: before stopping, each documented input rule has its own capture breaking only that rule; at the 12-probe cap, uncovered rules go into limits. Regenerate both host outputs. Text assertion that the final rule is single and coherent; behaviour is proven by the paid cases.
**W3-10. Shim-dom detector resolves literal assignments (1; 38980005793).** test/skill-e2e-design.test.ts:1248 looks for the literal `dom/run1`, but the agent ran the generated block's `"$_REPORT/dom/$_RUN"` with `_RUN="run1"` assigned in the same command. The check resolves that command's own literal `_RUN=`/`_REPORT=` assignments before matching. Free regression: the captured command passes; `scan --changed` and a scan over another directory still fail.
**W3-11. PTY harness keeps the last non-empty screen (diagnostic; 38976253774).** autoplan-guard-pty stalled at 489 s (normal 71-93 s) with empty journal and screen dumps. test/helpers/pty/ keeps the last non-empty `visibleText()` for the failure dump. No verdict change.
**W3-12. Explicit timeout for test/autoplan-answer-invalidation.test.ts.** Each test gets an explicit 30 s timeout (measured 4.1 s for the eng-only test on 4 cores; 7.8-9.4 s per file).
**W3-13. Close the stale createQAFunctionalFixture TODO** (already fixed by 0bf9d50) in TODOS.md, with the citation.
**W3-14. Superseded captures are rerun before completion (2; 39053431488 on #3252).** qa-functional-webhook-fix failed "missing exact sanitized evidence" because the agent labelled a pre-repair duplicate capture `superseded` and never reran it; the product (qa/sections/exploratory.md:88, lib/qa-evidence.ts:346-348) keeps such a row open, the detector (qa-functional-evidence.ts:160-166) requires a rerun, and the fixture prompt's schema never mentions superseded. The production semantics stay: the fixture prompt adds "after your final repair, rerun every pre-repair capture your verdict relies on; a superseded row stays open until its command reran on the repaired source", and scripts/resolvers/qa.ts adds the same completion step for fix runs (rerun superseded captures, or report them as an open remainder), including snapshotless clients. The detector and its negative test (test/qa-functional-evidence.test.ts:376-380) are unchanged. Free tests: the captured pre-repair duplicate labelled superseded and not rerun fails (negative control); the prompt states the rule.
**W3-15. Windows budget for test/review-army-budget.test.ts (free; windows-free-tests run 39053431430, shard 5).** The nested `bun test` finished in 5.58 s and exited 1 because Bun's default 5 s hook timeout fired on the real Red Team `beforeAll` (8 git spawns plus skill-section copies, test/skill-e2e-review-army.test.ts:643-667), slow on windows-latest (Windows seed 4,143 ms for the file vs 1,879 ms on Linux). The nested argv at :57 and :137 gains `--timeout 30000`; spawn timeouts move to 40 s and outer test timeouts to 45 s, including the consensus test that has none. Free check: the nested script with an injected 6 s `beforeAll` still prints the expected lifecycle; the Windows proof is the PR's windows-free-tests run.
Not repaired, kept visible: Q3 webhook-report (4; the two denied `cat` calls become negative controls in the qa-functional observer tests) and T qa-quick (one 300 s timeout against passing runs of 164-272 s; trimming what Quick tier loads is a product call, filed in TODOS).
### Deferred asks
- **D1. Preamble-hub diffs.** 46 hub files reach the whole gate under any sound map. A cheaper contract (a free contract suite for the preamble bins plus a 2-3 case paid canary for hub-only diffs) trades PR-time coverage for cost; Garry's call after A1-3's metrics and A2's shadow data. Recommendation: later.
- **D2. Generator-input narrowing** (resolvers/index.ts, preamble/**, gen-skill-docs.ts through committed outputs; 29 runs leave full-fallback on replay, mostly into dependents-all). Waits until every host output a paid case loads is shown committed and freshness-gated.
- **D3. Map activation** (A2 selects instead of shadowing; free-fixtures-data.ts leaves the global set with it). Needs PTY command capture for the six blind cases, two weeks of shadow data with zero shadow misses, and Garry's call on the measured saving.
- **D4. Run-count levers** (225 paid runs for 68 PRs; #3226 ran 13 full gates): readiness-gated or reuse-across-push dispatch, analysed read-only next wave.
- **D5. Red attribution in the paid report** (does the diff reach a red case directly, or only through a hub or the fallback). Report-only; next wave.
- **D6. Nested child output pollutes free-runner failure attribution** (scripts/lib/shard-engine.ts); filed in TODOS.
### Validation
Per AGENTS.md Validation discipline: one current failure list in `.context/fix-wave-3.md`; focused free tests per item during repair; then the cheap checks (typecheck, `bun test test/parity-suite.test.ts`, the SKILL.md size guard per CREW.md 11, skill-docs freshness, `test/spawnsync-timeout-tripwire.test.ts`, credential scanning); then the selected quality judges for changed prompts, reusing unchanged passing judge evidence by prompt-input identity; then PR B's focused paid cases once each at their EVAL_POLICY kind (ship-docsync-stale-after, ship-docsync-missing-asset, ship-exploratory-small-cli, ship-exploratory-plan-checks, qa-functional-webhook-fix, tpa-absent-linux, plan-mode-no-op, qa-functional-cli-report, design-review-detector-shim-dom); then each PR's gate; then `bun run test` once per PR on the frozen tree via Ubicloud. Paid evals never retry; a red is diagnosed from its transcript against the diff. Executed, reused and remaining required cases are reconciled before READY.
### Delivery shape
Four PRs, opened with Capy's PR tool, READY unstamped to GStack Minder:
- **A1:** scripts/pr-dependencies.ts (shebang), scripts/test-pr-profile.ts (classification), scripts/test-health-report.ts, docs/TESTING_INTERNALS.md, their tests.
- **A2 (stacked on A1):** scripts/pr-dependencies.ts (`exercisedConsumers`), scripts/test-pr-profile.ts (shadow block), test/helpers/exercised-commands.json, its refresh script and package script, scripts/lib/paid-report.ts (MAP_GAP), the census job in .github/workflows/evals-periodic.yml, test:health shadow lines, docs/TESTING_INTERNALS.md#exercised-path-map, their tests.
- **A3:** scripts/ubicloud/pregate-remote.sh, scripts/test-free-shards.ts (`--files-from` only), .gstack/pregate.json, bin/gstack-next-version, docs/pregate.md (example only), CONTRIBUTING.md (one row), their tests.
- **B (W3):** test/helpers/docsync-fault-actor.ts, docsync-fault-eval.ts, docsync-fixture.ts, docsync-observer.ts, bin/gstack-docs-candidate, test/helpers/qa-callers-fixture.ts, scripts/resolvers/review-army.ts, third-party-actions.ts, plan-reality.ts, qa.ts, test/helpers/qa-functional-eval.ts, test/qa-functional-evidence.test.ts, test/skill-e2e-design.test.ts, test/helpers/pty/*, test/autoplan-answer-invalidation.test.ts, test/review-army-budget.test.ts, detector corpora, regenerated SKILL.md and sections.
- Shared: TODOS.md (separate hunks) and docs/TESTING_INTERNALS.md (A1 and A2, stacked). Whichever lands second merges main. Only B regenerates skills.
- A3 changes every sub-8-core lane's pregate once merged; its READY report carries the crew rule and its PR body lists it under Behavior changes.
### Out of scope
W4 (seed recorder), W5 (pregate lanes, lib/pregate/**), W6-W9, map activation (D3), generator-input narrowing (D2), a lockfile narrowing rule, and any change to EVAL_POLICY, case kinds, trials, thresholds or quarantine.
## Review record
Revision 1, autoplan run run1-20261017181219 (CEO, DX, Eng; Claude Opus 5.5 native, GPT-6.1 Sol outside): CEO 13 + 3 findings, DX 16 + 9 (1 Critical: the scope detector's output used without its exit status, now W3-5), Eng 15 + 11. Gate rev 1 recommendations d3b, d4b, d5b, d6b, d7b, d8b changed this revision; d1a and d2a kept the direction. All other findings are build obligations folded into the items above.
+80 -60
View File
@@ -5,28 +5,30 @@
### Context
This is plan revision 2. Revision 1 was reviewed by six voices (CEO, DX, Eng; Claude Opus 5.5 native, GPT-6.1 Sol outside): 67 findings, 1 Critical, 8 user challenges. Gate rev 1 recommendations folded in here: d3b (split the W2 PR into A1, A2, A3), d4b (the exercised-path map ships in shadow mode), d5b/d8b (keep superseded/open semantics; the repair is a rerun requirement), d6b/d7b (a tested keyless pregate path before `remote_gate` is declared); d1a and d2a keep the direction. Every other finding is a build obligation, folded into the item it names.
Owner request: Garry approved GSTA-41's triage on 2026-10-17 with "Accept all recommendations", which includes "Autoplan W2 and W3 together now" (reports/triage-2026-10-17.md in the GStack Minder drive). Coordinator: GStack Minder (GSTA-36) owns the merge queue, version slots and every merge; this wave opens PRs and sends READY reports unstamped. Builder: GSTA-45.
Base: origin/main 521fd42 (v1.91.122.0). Queue above main: #3240 (.123) and #3242 (.124), both GSTA-24, both in lib/cso, cso/ and scripts/cso-eval*. Live owners this wave must not collide with: GSTA-46 (W5: lib/pregate/import-graph.ts, lib/pregate/checks/lanes.ts, lib/regen.ts, test/pregate-selection.test.ts, test/helpers/pregate-fixture.ts), GSTA-33 (W4 duration-seed recorder: scripts/free-test-durations*.json and free-tests.yml), GSTA-44 (#3220 brain-sync). Out of scope: W4, W5, W6-W9.
Base: origin/main (v1.91.125.0 at revision time). Live owners this wave must not collide with: GSTA-46 (W5: lib/pregate/import-graph.ts, lib/pregate/checks/lanes.ts, lib/regen.ts, test/pregate-selection.test.ts, test/helpers/pregate-fixture.ts), GSTA-33 (W4, #3253: the duration-record workflow, windows-free-tests.yml, scripts/test-pr-profile.ts, docs; it does not touch scripts/test-free-shards.ts, per GStack Minder), GSTA-44 (#3220 brain-sync). A1 and A2 also edit scripts/test-pr-profile.ts, so whichever of #3253 and A1 lands second merges main. Out of scope: W4, W5, W6-W9.
Evidence base. Two read-only investigations on 521fd42 (no paid spend):
- **W2:** 225 evals.yml pull_request runs with a paid-plan manifest over 10 days (2026-10-07 to 2026-10-17), every manifest, trial-outcomes artifact and verdict line, each run's changed files replayed through today's selector (`computePaidCaseSelection`, the replay tool's entry point), plus counterfactual replays of each proposed fix.
- **W3:** every red verdict in PR gate runs since #3231 (`bun run eval:pass-rates --reds --workflow evals.yml --runs 60`): 21 reds in 17 runs (38969924945 to 39011101113). Each rejected tool call was replayed through the real detector on main.
Evidence base. Read-only investigations on 521fd42 (no paid spend):
- **W2:** 225 evals.yml pull_request runs with a paid-plan manifest over 10 days (2026-10-07 to 2026-10-17), every manifest, trial-outcomes artifact and verdict line, each run's changed files replayed through today's selector (`computePaidCaseSelection`), counterfactual replays of each fix, and a prototype exercised-path map built from 541 PR runs' recorded commands.
- **W3:** every red verdict in PR gate runs since #3231 (`bun run eval:pass-rates --reds --workflow evals.yml --runs 60`): 21 reds in 17 runs (38969924945 to 39011101113), plus two reds GStack Minder forwarded from #3252. Each rejected tool call was replayed through the real detector on main.
Thesis: **spend the PR gate's money only where a diff can change the outcome, and make a red verdict mean the diff broke something.**
### Release promises
- **P1. A PR's paid selection follows what its diff can actually reach.** Selection never under-selects (the shebang defect is closed first), and the cost metric tracks dollars and cases per run, not the fallback label.
- **P2. A gstack contributor on a small machine can finish pregate.** Under 8 cores, the local test tier routes to a declared remote gate that prints a tree-bound receipt instead of running for an hour.
- **P1. Paid selection never under-selects what it can see, and narrowing is earned with evidence.** The shebang blind spot closes now; the exercised-path map runs in shadow, so map drift is measured (MAP_GAP, census) before any narrowing is activated by a later decision. The cost metric tracks dollars and cases per run, not the fallback label.
- **P2. A gstack contributor on a small machine can finish pregate,** with or without a Ubicloud key: under 8 cores the local tier goes to a declared remote gate that runs on Ubicloud when a key exists and as parallel local shards when it does not, and prints a receipt bound to the tree it actually tested.
- **P3. Every open red cause found in the window is fixed at its cause or kept as a named negative control.** No threshold, trial count, case kind or quarantine changes.
- **P4. `/ship` slot allocation does not wait on serial GitHub calls.**
- **P4. `/ship` slot allocation does not wait on serial GitHub calls,** and a fetch that times out says so.
### Premise table
| premise | claim | receipt | verdict |
|---|---|---|---|
| P-a | 42.5% of PR runs fall back to full selection | test:health 7d 68/160; 10-day replay 95/225 recorded, 91/225 under today's selector | true |
| P-a | 42.5% of PR runs fall back to full selection | test:health 7d 68/160 at triage, 75/177 on 2026-10-17; 10-day replay 95/225 recorded, 91/225 under today's selector | true |
| P-b | touchfile mappings for bin/, lib/, hosts/, make-pdf/ cut the fallback | no file under bin/, lib/, hosts/, make-pdf/, scripts/ or browse/ was an unknown file in any of the 225 runs; the derivation (scripts/pr-dependencies.ts) already places them | **false as stated** |
| P-c | cutting the fallback rate cuts spend | fixing every narrowable trigger takes the rate 40.4% -> 14.2% on replay but predicted spend only -2.5%: every run that leaves full-fallback lands in `dependents`, which selected all 108 gate cases and 24 judges in 100 of 105 such runs | **false** |
| P-d | cost by mode | recorded known cost per run: full-fallback $91.60 (n=61), dependents $80.09 (median $91.06, n=35), pr $14.87 (median $0, n=59); fallback + dependents carry 90.5% of known spend | true |
@@ -47,109 +49,127 @@ Six of the 21 window reds are fixed on main with regressions: ship-docsync-store
### Surface check
Verified on 521fd42 by source: `selectPrProfile`, `FREE_ONLY_PR_FILES`, `PR_FULL_GATE_FILES`, `DERIVABLE_PREFIXES` (scripts/test-pr-profile.ts); `SELECTION_DATA_MODULES`, `refs`, the global walk (scripts/pr-dependencies.ts); `remote_gate`, `PREGATE_REMOTE:` contract, `PREGATE_TREE`/`PREGATE_FILES_FILE` (lib/pregate/checks/lanes.ts, docs/pregate.md#remote-gate); `ubi-runner.sh run --src --setup --env --pull`; `test-free-shards.ts` flags (no file-subset flag today); `gstack-docs-candidate` top-level `--help` only (:91) and `compare` schema check (:123). New surfaces are named as new.
Verified by source: `selectPrProfile`, `FREE_ONLY_PR_FILES`, `PR_FULL_GATE_FILES`, `DERIVABLE_PREFIXES` (scripts/test-pr-profile.ts); `SELECTION_DATA_MODULES`, `refs` (:178), the global walk (:242), `targetsOf` (:250) (scripts/pr-dependencies.ts); `remote_gate`, the `PREGATE_REMOTE:` contract, `PREGATE_TREE`/`PREGATE_FILES_FILE`/`PREGATE_SPEND_CAP_USD`, exported `parseRemoteReport`/`remoteReportProblem` (lib/pregate/checks/lanes.ts:142-155, read-only here); `ubi-runner.sh run --src --setup --env --pull` and scripts/ubicloud/test-free.sh; `test-free-shards.ts` flags (no file-subset flag today; mode conflicts at :597-601); `gstack-docs-candidate` top-level `--help` only (:91) and `compare` schema check (:123). New surfaces are named as new.
### Binding decisions
- Every merge is GStack Minder's; READY reports go unstamped; never stamp before a slot call (CREW.md 6).
- No threshold, trial count, case kind or quarantine changes (EVAL_POLICY; AGENTS.md Validation discipline step 6). A detector change must keep or add negative controls.
- Do not edit files GSTA-46 (W5) or GSTA-33 (W4) own; lib/pregate/** stays untouched by this wave.
- No threshold, trial count, case kind or quarantine changes (EVAL_POLICY; AGENTS.md Validation discipline step 6). A detector change keeps or adds negative controls. Each PR body carries a gate-edit table (file, old vs new acceptance, negative control) and a `## Behavior changes` list.
- Do not edit files GSTA-46 (W5) owns; lib/pregate/** is read-only in this wave.
- Paid PR gates are pre-approved (Garry, 2026-10-15).
- Activating the exercised-path map (changing what a PR selects) is a later decision with its shadow data, not part of this wave.
### Numbers
- W2 replay: today 91/225 full-fallback (40.4%), mean 79.6 gate cases per run. Sound classification fixes alone (W2-2): 36/225 full-fallback (16.0%) but no spend change, because those runs land in dependents mode that still selects about 105 cases.
- W2-3 measured by replay (541 PR runs of observed commands: 12,127 case records, 235,795 commands): the exercised-path map applied to every changed file replays with **0 misses against all 47 recorded gate reds** and cuts predicted spend **6.5%** (scaled $14,751 -> $13,795 per 10 days, about $35k a year; mean gate cases 79.6 -> 74.5). Applied only to unmapped files, as first drafted, it cuts 15% but **misses 5 reds**, because touchfiles name hub bins for only some of the cases that run them; that variant is rejected. The classification fixes without the map cost slightly more than today and miss 3 reds once free-fixtures-data.ts stops being global, so that one change ships only with the map.
- The floor: 46 files (32 bin, 11 lib, 3 other) still reach 100-106 of 108 cases under any map, because every generated SKILL.md runs the preamble bins (gstack-skill-start alone closes over 73 files). Going below that needs a cheaper contract for preamble-hub diffs, not a finer map (deferred ask D1).
- W3: of 17 red runs in the window, 6 are already green on main; W3 fixes the cause in a further 7 (38973329985, 38986120559, 38990111230, 38993076767's small-cli, 39002269281, 39004466422, 39011101113) and the detector misread in 38980005793's shim-dom; 5 runs stay red: genuine model misses (D3 in 38983306684 and 38984977679, Q3 in 39006896632), one timeout tail (T, 38980005793) and one undetermined hang (U, 38976253774). 39001107574 (C) turns green only if the W3-9 hardening changes behaviour.
- W2 replay: today 91/225 full-fallback (40.4%), mean 79.6 gate cases per run. Sound classification fixes alone: 36/225 full-fallback (16.0%) but no spend change, because those runs land in dependents mode, which still selects about 105 cases.
- Exercised-path map, replayed over the 225 runs: applied to every changed file it misses **0 of the 47 recorded gate reds** and cuts predicted spend **6.5%**; on the recorded basis ($6,811 trial-sum per 10 days, $101.66 per merged PR) that is about $440-600 per 10 days. Applied only to unmapped files it cuts 15% but **misses 5 reds** (rejected). Making free-fixtures-data.ts non-global without the map misses 3 #3231 reds, so that change waits for map activation.
- The floor: 46 files (32 bin, 11 lib, 3 other) reach 100-106 of 108 cases under any sound map, because every generated SKILL.md runs the preamble bins (gstack-skill-start alone closes over 73 files). Below that needs a cheaper contract for preamble-hub diffs (deferred ask D1).
- W3: of the 17 red runs, 4 are already green on main (38969924945, 38974669950, 38981523623, 38988927989; 6 reds fixed). W3 fixes the cause in 7 more (38973329985, 38986120559, 38990111230, 38993076767, 39002269281, 39004466422, 39011101113) and the detector misread in 38980005793. Still red after W3: 38976253774 (U, undetermined hang), 38980005793 (T, timeout tail), 38983306684 and 38984977679 (D3 model misses), 39006896632 (Q3 model miss), and 39001107574 (C) unless the W3-9 hardening changes behaviour: 4 + 7 + 6 = 17.
### Wave items
#### W2: PR eval cost (one PR, ~$93 full gate; it edits global touchfiles, so every case runs)
#### W2-A1: selection correctness and safe classification (PR A1; edits global touchfiles, so a full ~$93 gate)
**W2-1. Close the shebang under-selection first (correctness).** In scripts/pr-dependencies.ts, strip a leading `#!` line before `scanImports` in `refs`, the global walk and `targetsOf`, and treat bun/node-shebang extensionless files as code. This broadens selection (34 files flip from no-consumer to consumed; test/helpers/bootstrap-retention.ts joins the global set through scripts/test-paid-shards.ts). Free tests: `lib/fs-utils.ts` selects the cases of bin/gstack-redact-prepush; a shebang fixture's import is found; an error other than the shebang still fails closed (selects everything), never silently.
**A1-1. Close the shebang blind spot.** In scripts/pr-dependencies.ts, strip a leading `#!` line before `scanImports` at all three sites (`refs` :178, the global walk :242, `targetsOf` :250) and treat bun/node-shebang extensionless files as code. Any other scan error keeps failing closed (selects everything) at all three sites. This broadens selection: 34 files flip from no-consumer to consumed; test/helpers/bootstrap-retention.ts joins the global set through scripts/test-paid-shards.ts. Free tests: `lib/fs-utils.ts` selects the cases of bin/gstack-redact-prepush; a shebang fixture's import is found; an injected non-shebang scan error still selects everything.
**W2-2. Sound classification fixes (scripts/test-pr-profile.ts, scripts/pr-dependencies.ts).**
**A1-2. Safe classification fixes** (none narrows a case a diff can reach; they change labels and fallback reasons).
- FREE_ONLY_PR_FILES gains .github/dependabot.yml, .osv-scanner.toml, setup.ps1, .gitlab-ci.yml, .github/ISSUE_TEMPLATE/**, .gstack/ship-policy.example.json, each with its one-line reason (readers: osv-scanner.yml and free tests; windows-setup-e2e.yml; free gstack-doctor.test.ts:363). test/paid-free-boundary.test.ts:117 swaps its sample unknown file to one that stays unmapped.
- capy/ joins REPO_PATH and DERIVABLE_PREFIXES, so capy/support.env derives to bin/gstack-install-verdict.sh and bin/gstack-doctor-check.sh.
- test/helpers/free-fixtures-data.ts stops being global by the import walk: it joins the map-diff path that touchfiles-data.ts uses (diff FREE_FIXTURES old vs new, select the cases whose closure holds a reclassified fixture, fail closed on any error). It lands only with W2-3: on replay it otherwise drops three #3231 reds whose cases really run the changed bin/gstack-skill-start and gstack-skill-end.
- `setup` leaves PR_FULL_GATE_FILES and is touchfiled to the cases that execute it (gstack-upgrade-happy-path, skillmd-setup-discovery and the install/overlay fixtures that spawn `join(src, 'setup')`). CI never runs ./setup (evals.yml runs `bun run build` and the register-gstack-skills composite). A new free completeness scanner fails when a paid test or helper spawns `setup` without being in that list.
- Skill-name mentions (`/gstack-x` after whitespace, a quote or a parenthesis) stop counting as bin invocations; path forms (`bin/gstack-x`, `$(dirname "$0")/gstack-x`) still count.
- A full-gate file is never also labelled "real unknown dependency" (cosmetic; fixes wrong fix text for evals.yml, test-setup.ts, .gitattributes).
- Generator inputs (scripts/resolvers/index.ts, gen-skill-docs.ts, preamble.ts, preamble/**) stay in PR_FULL_GATE_FILES in this wave. Selecting them through committed generated output is sound only if every host output a paid case loads is committed and freshness-gated; that is not verified, so it is deferred (TODOS entry with the 29-run replay effect).
- A full-gate file is never also labelled "real unknown dependency" (fixes wrong fix text for evals.yml, test-setup.ts, .gitattributes).
- Not in A1: `setup` stays a full-gate file (its real executors include bin/gstack-upgrade:131, bin/gstack-capy-install:404-405 and bin/dev-setup:93, which today's matcher cannot see, and moving it saves no money); free-fixtures-data.ts stays global until map activation; generator inputs stay full (D2).
**W2-3. Exercised-path consumers for every changed file (the spend lever, about 6.5%).** Today a derived file's consumers are every case whose reference closure reaches it, through prose, strings and slash-command names as well as code. Replace that with the union of: (1) cases whose direct touchfiles match; (2) cases whose test file or helper import closure holds the file; (3) cases whose loaded skill (template and generated SKILL.md) has executable bash blocks naming a bin, compiled binary (`$B`, `$D`, browse/design/make-pdf dist outputs mapped to their source trees) or sourced script whose code closure holds the file, with comments, heredocs, echo/printf/die messages, case patterns and bare `VAR=gstack-x` strings stripped; (4) cases observed invoking such a bin, from a committed, generated `test/helpers/exercised-commands.json`. The union applies to every changed non-global file in pr and dependents modes, never only to unmapped files. PR_FULL_GATE_FILES and GLOBAL_TOUCHFILES stay full, and the closure stops at them.
Guards, because observed sets are a moving lower bound (a typical run shows half its case's union, 23 cases gained a bin in the window's last third, six PTY cases record no commands):
- **Per-run, in the paid report job (non-blocking):** every case's recorded Bash commands go through the same bin matcher; a bin outside the case's recorded set is printed as `MAP_GAP <case> <bin>` in the report and uploaded as an artifact. It never turns a PR red.
- **Periodic census (blocking there):** the scheduled full run rebuilds the observed sets from its own artifacts (no API downloads; the build hit the installation's 403 artifact limit), fails on any `MAP_GAP` and on a PTY case that records zero commands while its skill's bash blocks name bins, and prints the refresh command.
- **Freshness:** selection refuses a map older than 14 days or missing a declared case, and falls back to today's derivation (never to a narrower one), saying so in the plan reasons.
Free tests: the 47 recorded reds replay with 0 misses (captured changed-file lists and case ids as a fixture); the five variant-c misses are negative controls; a stripped prose mention (`/gstack-upgrade` in remediation text) no longer reaches bin/gstack-upgrade while `bin/gstack-x` and `$(dirname "$0")/gstack-x` still do; a stale map falls back.
**A1-3. Measure what the money follows.** scripts/test-health-report.ts adds `pr-cases-per-run` (mean selected gate cases and judges per PR run) and `pr-known-cost-per-run` beside `pr-fallback-rate`, and the summary leads with cost per merged PR. The fallback text in docs/TESTING_INTERNALS.md#pr-paid-lane-fallback is corrected (it says bin/ and lib/ changes take the full gate because they are unmapped; they are derived).
**W2-4. Measure what the money follows.** scripts/test-health-report.ts adds `pr-cases-per-run` (mean selected gate cases and judges per PR run) and `pr-known-cost-per-run` beside `pr-fallback-rate`; the CEO metric moves to cost per merged PR, which test:health already reports.
**A1-4. Lockfile decision: keep package.json and bun.lock full.** No narrowing is built: no run fell back on them alone, dependabot never reaches the paid lane, and every runtime package's importer reaches all cases. Documented in docs/TESTING_INTERNALS.md#pr-paid-lane-fallback with the evidence.
**W2-5. Lockfile decision: keep package.json and bun.lock full.** No narrowing is built: no run fell back on them alone, dependabot never reaches the paid lane, and every runtime package's importer reaches all cases. Documented in docs/TESTING_INTERNALS.md#pr-paid-lane-fallback with the evidence, so the next PR that selects 132 on a lockfile change is understood rather than re-litigated.
#### W2-A2: exercised-path map in shadow mode (PR A2, stacked on A1; full gate)
**W2-6. A remote gate for gstack's own pregate (P2).** New `scripts/ubicloud/pregate-remote.sh`: reads `PREGATE_FILES_FILE` and `PREGATE_TREE`, recomputes the tree with bin/gstack-wtree before upload and refuses on mismatch, runs the selected files on one Ubicloud VM through the free shard runner in parallel (new `--files-from <path>` flag on scripts/test-free-shards.ts, restricting the planned set to those files and nothing else), recomputes the tree after, and prints `PREGATE_REMOTE: tree=<tree> tests=<n> failed=<n> spent=<usd>` (spent from VM minutes at the standard-16 rate). With no `UBICLOUD_API_KEY` it exits non-zero with a clear message and the obligation stays uncleared; it never fakes a pass. .gstack/pregate.json declares `"remote_gate": "bash scripts/ubicloud/pregate-remote.sh"` (takes effect once on main, since pregate reads the base branch's policy). No lib/pregate change. Free tests: `--files-from` plans exactly the listed files and rejects unknown paths; the script's output contract against a stub ubi-runner, including tree mismatch, zero tests and non-zero failures. One real Ubicloud run of the 798-file selection is recorded in the PR body.
**A2-1. The map.** A new generated, committed `test/helpers/exercised-commands.json` holds per gate case the union of commands observed in recorded trials, with `generatedAt`, `observedThrough`, the source run ids and per-case observation counts. It is registered as a SELECTION_DATA_MODULE with map-diff selection like touchfiles-data.ts (a refresh selects only cases whose set changed, failing closed on error), so a refresh never pays the full gate. Refreshes are union-only: an observation never removes an edge, and missing or empty artifacts never erase consumers; removal is a reviewed edit.
**W2-7. Concurrent next-version fetches (P4).** bin/gstack-next-version's workers use async `spawn` (bounded at GH_API_CONCURRENCY, each with its 15 s timeout and the same ok/stdout/stderr result), keeping result order deterministic (sorted by PR number). Free test with a delayed fake `gh` on PATH: 8 PRs at 0.5 s each finish in under 2 s and return the same claims as the serial path; a timed-out fetch yields the same warning as today.
**A2-2. The consumer function** (scripts/pr-dependencies.ts, new `exercisedConsumers`), the union of: (1) direct touchfile matches; (2) the case's test file and helper import closure; (3) executable shell in the case's loaded skill (template and generated SKILL.md) and in bin/, lib/, scripts/: command-position `gstack-*` names and repo paths, `.`/`source` edges, `./setup` / `$ROOT/setup` exec edges, quoted path literals, path.join pairs, data-file paths, `bun run` scripts, compiled outfiles (`browse/dist/browse`, `design/dist/design`, `make-pdf/dist/pdf`, `$B`, `$D`, find-browse) mapped to their source trees, and shard perTest attribution; comments, echo/printf/die messages, case patterns, bare `VAR=gstack-x` strings and slash-command skill names are not edges, but command substitutions and executable or expanding heredocs are kept, and unparsed shell keeps its conservative edges; (4) observed bins from A2-1. The edge classes are ported from the measured prototype (exercised.ts), and a parity test pins that the shipped function equals or is a reviewed superset of the prototype's map at a pinned commit. PR_FULL_GATE_FILES and GLOBAL_TOUCHFILES stay full, and the closure stops at them.
#### W3: paid-gate reliability, round 2 (one PR; prompt and fixture edits select many cases; ~$93 gate plus case panels)
**A2-3. Shadow, not selection.** `selectPrProfile` keeps today's selection. The paid-plan manifest gains a `shadow` block: the cases the map would select, the cases it would drop, the per-file provenance ("selects N cases: touchfile (…), import (…), skill bash via bin/gstack-x (…), observed (…)") and the predicted cost difference; the plan's coverage summary prints one line of it. test:health reports the shadow saving and shadow misses (a red on a case the shadow would have dropped) per week.
Each item names the cause class from the investigation: (1) fixture/detector defect, (2) skill or prompt defect, (4) genuine model miss.
**A2-4. Gap detection.** The paid report job runs the same bin matcher over each executed case's commands and emits `MAP_GAP <case> <bin>` against static-plus-observed consumers (never the observed set alone) as its own non-blocking job result, separate from case verdicts and from the INFRA/INCOMPLETE re-dispatch rule, with the doc anchor docs/TESTING_INTERNALS.md#exercised-path-map. The weekly gate census emits the union-refreshed map as an artifact (built from its own artifacts, no REST downloads: the prototype hit the installation's 403 artifact limit) and prints the refresh command (`bun run test:exercised-map --from-run <id>`), which opens the refresh as an ordinary PR. In shadow mode the census reports gaps and the six PTY cases that record no commands; it does not block. PTY command capture is a named prerequisite for activation, not part of this wave.
**W3-1. Docsync fault actor writes real candidates (1; 38990111230 and 79 of 198 docsync captures).** test/helpers/docsync-fault-actor.ts `prepare` (:81-110) writes `docsCandidate()` records that the installed `gstack-docs-candidate compare` rejects as "not a gstack-docs-candidate record" (bin/gstack-docs-candidate:123), while ship/sections/documentation.md:70 requires running compare. `prepare` runs the installed `snapshot` and prints its exact argv so the post-child rerun is literal; docsync-fault-eval.ts reads the helper schema. Free test: actor `prepare` then installed `compare` exits 0 with all-false change flags.
Free tests: a synthetic fixture tree with one case per edge class (including the stripped-message negatives and the kept command substitution), the five variant-c miss shapes as negative controls, union-only refresh (disjoint branches, empty artifacts), map-diff selection of a refresh, and shadow output that never changes `e2e`/`judges`. The 47-red 0-miss replay is a pinned PR-body receipt reproducible by script at the PR commit, not a free test (historical file lists rot on renames).
**W3-2. Per-subcommand help in gstack-docs-candidate (1).** `snapshot --help` and `compare --help` print usage and exit 0 before any side effect; test/helpers/docsync-observer.ts:272-274 admits `<candidate> (snapshot|compare) --help` with nothing else. Free regression: the captured 38990111230 ndjson (redacted) replays to `[]`; negative controls `snapshot --help --out x.json`, `compare --help x.json` and an in-repo `--out` stay rejected.
#### W2-A3: small-machine pregate and next-version (PR A3; disjoint from A1/A2)
**A3-1. `scripts/ubicloud/pregate-remote.sh` (new) and `--files-from` (scripts/test-free-shards.ts).**
- Inputs: `PREGATE_FILES_FILE`, `PREGATE_TREE`, `PREGATE_SPEND_CAP_USD` from `gstack-pregate --stage tests --remote`. Run bare (as pregate's `run:` line shows), it exits 2 before any work with "PREGATE_FILES_FILE is not set; run: gstack-pregate --stage tests --remote --spend-cap <usd> --allow-repo-commands".
- Before any upload it recomputes the tree with bin/gstack-wtree; on mismatch it prints both tree prefixes, "nothing uploaded, nothing spent" and the rerun command, and exits 3.
- **Keyed path (UBICLOUD_API_KEY set):** refuses before VM create when the worst case (wall timeout × the price of the size actually used, honouring UBI_SIZE) exceeds the cap, printing estimate and cap; wraps scripts/ubicloud/test-free.sh (setup-free-suite.sh, xvfb-run, GSTACK_EXPECT_BINARIES=1, the free retry policy) with `--files-from` and a wall timeout derived from the cap; recomputes gstack-wtree on the VM after unpack and refuses on mismatch; verifies the planned set equals the file list (digest) before running; prints `PREGATE_REMOTE: tree=<VM tree> tests=<n> failed=<n> spent=<usd>` from minutes used. Infra failures (create, setup, SSH drop) exit 4 with the ubi-runner message, the VM name and its `down` line or teardown confirmation, the log path and spent= for minutes billed; one retry after a quota message, then stop. Phase timings (create, setup, run) are logged; the target is 10 minutes from `--remote` to a cleared obligation.
- **Keyless path:** runs the same files locally through `test-free-shards.ts --files-from` in parallel shards, prints "no UBICLOUD_API_KEY: running N files locally in parallel shards", and the same honest `PREGATE_REMOTE:` line for the tree tested (recomputed before and after; a moved tree fails), with `spent=0`.
- `--files-from <path>`: one repo-relative path per line, blank lines ignored; rejects before planning unknown paths (first 5 plus a count, "not in the free planned set", fix: use paths from `--list`), an empty list and malformed lines; conflicts with `--quick`, `--windows-only`, `--list` and the `--ci-*` modes like the parser's existing conflicts.
- .gstack/pregate.json declares `"remote_gate": "bash scripts/ubicloud/pregate-remote.sh"` in the same PR (effective once on main). docs/pregate.md#remote-gate gets a gstack example and CONTRIBUTING.md "Which command do I run?" a small-machine pregate row with keyed and keyless commands. The READY report carries a one-paragraph crew rule for GStack Minder's CREW.md.
- Free tests: the output contract against a stub ubi-runner through the exported `parseRemoteReport`/`remoteReportProblem`; bare run, tree mismatch before and after, zero tests, failures, cap refusal with no VM created, create/setup failure; `--files-from` exact planning and every rejection message; the keyless path on a two-file list. One real Ubicloud run of the 798-file selection is recorded in the PR body.
**A3-2. Concurrent next-version fetches.** bin/gstack-next-version's workers use async `spawn` bounded at GH_API_CONCURRENCY, each with its 15 s timeout. Results and warnings are sorted by PR number. A timeout keeps its cause: "PR #N: VERSION fetch timed out after 15 s; claim unknown, the slot may collide; rerun, then ask the coordinator", and the JSON gains `unknown_claims`; the fork/path hint stays for not-found failures only. Free tests with a delayed fake `gh` on PATH: 8 PRs at 0.5 s finish in under 2 s with the same claims as before; a randomized-delay run is deterministic; a timed-out fetch prints the timeout text and counts in `unknown_claims`.
#### W3: paid-gate reliability, round 2 (PR B; prompt and fixture edits select many cases; ~$93 gate plus focused cases)
Each item names the cause class: (1) fixture/detector defect, (2) skill or prompt defect, (4) genuine model miss.
**W3-1. Docsync fault actor uses the production candidate protocol (1; 38990111230; 79 of 198 docsync captures).** test/helpers/docsync-fault-actor.ts `prepare` (:81-110) writes `docsCandidate()` records that the installed `gstack-docs-candidate compare` rejects (bin/gstack-docs-candidate:123), while ship/sections/documentation.md:70 requires running compare. Migrate prepare, dispatch observations, inspect, seeded prompt comparisons and legacy checks together to the production schema and hash representation: prepare runs the installed `snapshot` under a bounded timeout and prints its exact argv so the post-child rerun is literal, publishes state only after a valid candidate exists, and never overwrites a saved attempt. Free tests: the full adapter sequence (prepare, installed compare, inspect) for unchanged inputs, a doc edit, an index change and a deletion; a snapshot failure before and after candidate creation.
**W3-2. gstack-docs-candidate help and errors (1).** `snapshot --help` and `compare --help` print usage and exit 0 before any side effect; test/helpers/docsync-observer.ts:272-274 admits `<candidate> (snapshot|compare) --help` with nothing else. The compare reject names the path, the missing or wrong fields (schema_version, base, hashes) and the fix (`gstack-docs-candidate snapshot --out <path> --audit-id <id> --mode edit --base <rev>`), keeping exit 1. Free regression: the captured 38990111230 ndjson (redacted) replays to `[]`; `snapshot --help --out x.json`, `compare --help x.json` and an in-repo `--out` stay rejected; corpus rows pinning the old reject text are updated.
**W3-3. Docsync model misses become negative controls (4; 38983306684, 38984977679).** The captured Bash `grep -n -i …` and `cat …; cat …` join the docsync detector corpus as `refuse` rows. No detector change.
**W3-4. qa-callers prompt stops teaching helper syntax (2; 38973329985, 38993076767).** test/helpers/qa-callers-fixture.ts:700 hands over full capture/checkpoint syntax but only "Materialize is supported…", and both reds skipped qa/sections/exploratory.md and system-functional.md then ran a bare `materialize`; all 83 passing sessions read both sections. The prompt declares only the fixture bindings (evidence tool path, reports dir, deadline file, `--public` approved, `--timeout-ms 10000` for plan probes) and points to the section for the forms. The read requirement stays. Free tests: the two captured transcripts still fail `validateCallerEvidence` (negative control); the prompt contains no `gstack-qa-evidence capture` grammar.
**W3-4. qa-callers prompt stops teaching helper syntax (2; 38973329985, 38993076767).** test/helpers/qa-callers-fixture.ts:700 hands over full capture/checkpoint syntax but only "Materialize is supported…"; both reds skipped qa/sections/exploratory.md and system-functional.md then ran a bare `materialize`, while all 83 passing sessions read both. The prompt declares only fixture bindings (evidence tool path, reports dir, deadline file, `--public` approved, `--timeout-ms 10000` for plan probes) and points to the section for the forms. The read requirement stays. Free tests: the two captured transcripts still fail `validateCallerEvidence`; the prompt contains no `gstack-qa-evidence capture` grammar.
**W3-5. State-machine scope in the generated scope block (2; 39002269281).** #3235 added specialist 9 as inline prose naming `gstack-state-matrix detect --base <base>` (scripts/resolvers/review-army.ts:75), and small diffs skip every specialist except State Machines (:66), so agents run a command the block and the caller interface do not declare. The block (:27) adds `source <(… gstack-state-matrix detect --base <base> 2>/dev/null) || true` before `set | grep '^SCOPE_'`; :75 reads "if SCOPE_STATE_MACHINE=true (printed by the scope block)". Regenerate. test/review-workflow-clarity.test.ts requires SCOPE_STATE_MACHINE from the block; the captured standalone call stays outside the interface.
**W3-5. State-machine scope in the generated scope block (2; 39002269281).** scripts/resolvers/review-army.ts:75 names `gstack-state-matrix detect --base <base>` in prose and small diffs keep only State Machines (:66), so agents run an undeclared command. The block (:27) gains `source <(… gstack-state-matrix detect --base <base> 2>/dev/null)` with its exit status checked: on failure it prints `SCOPE_STATE_MACHINE=unknown` and the State Machines specialist is included conservatively, with the detector's stderr kept. :75 reads "if SCOPE_STATE_MACHINE is true or unknown (printed by the scope block)", and its "pipe output to `check -`" step becomes a declared block form. Regenerate. test/review-workflow-clarity.test.ts pins both; a free test runs the block with a failing detector stub and sees `unknown`; the captured standalone call stays outside the caller interface.
**W3-6. Webhook fix runs re-verify the whole repaired invariant (2; 38986120559).** The fix-run prompt's stage 2 (test/helpers/qa-functional-eval.ts:33) asks only for the original, happy, cancel and dependency probes, while the verifier (qa-functional-evidence.ts:224) re-runs all 8 scenarios. Stage 2 for webhook re-runs every scenario of the repaired invariant (duplicate, partial, concurrent-ab, concurrent-ba) plus happy, cancel and dependency, inside the 12-probe budget; `QA_WEBHOOK_REQUIRED_SCENARIOS.qa` mirrors it. Free test: the captured status-only repair applied to a fresh fixture fails `verifyQANativeRegression` on partial; an effect-based repair passes.
**W3-6. Webhook fix runs re-verify the repaired invariant within the budget (2; 38986120559).** Stage 2 of the fix-run prompt (test/helpers/qa-functional-eval.ts:33) asks only for original, happy, cancel and dependency, while the verifier (qa-functional-evidence.ts:224) re-runs all 8 scenarios. The prompt states the budget split up front: at most 5 pre-repair captures, then after the final source edit every scenario of the repaired invariant (duplicate, partial, concurrent-ab, concurrent-ba) plus happy, cancel and dependency, 12 in total; if the allowance cannot establish the defect, report incomplete. `QA_WEBHOOK_REQUIRED_SCENARIOS` is unchanged (no verdict tightening); the post-repair list feeds the probe nudge instead. Free tests: the captured status-only repair applied to a fresh fixture fails `verifyQANativeRegression` on partial; an effect-based repair passes; a pre-repair-only scenario set plus a complete post-repair set exercise the nudge.
**W3-7. Lettered consent options when Aside is absent (2; 39011101113).** scripts/resolvers/third-party-actions.ts:54 fixes letters only for the detected case, and the agent numbered 1/2/3. The absent branch offers "B) gstack's own visible browser, C) manual instructions, D) defer (omit A)". Regenerate. Free test: the contract text letters the absent branch; the captured numbered answer is a documented fail example. The test's assertion is unchanged.
**W3-7. Lettered consent options when Aside is absent (2; 39011101113).** scripts/resolvers/third-party-actions.ts:54 fixes letters only for the detected case, and the agent numbered 1/2/3. The absent branch offers "B) gstack's own visible browser, C) manual instructions, D) defer (omit A)". Regenerate. Free test: the contract letters the absent branch; the captured numbered answer is a documented fail example. The test's assertion is unchanged.
**W3-8. Tier-1 reality rows are never n/a (2; 39004466422).** The agent wrote `REALITY: deferred-asks n/a …`, `gstack-plan-reality check` returned verdict=repair, and the repair loop ran past the 300 s capture. scripts/resolvers/plan-reality.ts:59 says tier-1 rows are never n/a, and with no linked issue deferred-asks is `pass` with a receipt showing none. Regenerate. Free test: the captured line through `check` (pinned message) and a skill-text assertion.
**W3-8. Tier-1 reality rows are never n/a (2; 39004466422).** The agent wrote `REALITY: deferred-asks n/a …`, `gstack-plan-reality check` returned verdict=repair, and the repair loop ran past the 300 s capture. scripts/resolvers/plan-reality.ts:59 says tier-1 rows are never n/a and, with no linked issue, deferred-asks is `pass` with a receipt showing none. Regenerate. Free test: the captured line through `check` (pinned message) and a skill-text assertion.
**W3-9. Exploratory stop rule ties to the per-rule probes (2 hardening of a 4; 38983306684, 39001107574).** Neither cli-report red probed a non-digit amount; all 32 other CLI sessions did, and qa/sections/system-functional.md:33 already requires one probe per input rule. The exploratory stop rule (qa and qa-only exploratory templates) requires each documented input rule to have its own capture breaking only that rule before stopping. Text assertion only; behaviour is proven by the paid cases.
**W3-9. Exploratory stop rule ties to the per-rule probes (2 hardening of a 4; 38983306684, 39001107574).** Neither cli-report red probed a non-digit amount; 32 of the other 34 CLI sessions did, and qa/sections/system-functional.md:33 already requires one probe per input rule. The shared owner, scripts/resolvers/qa.ts, states one rule for both exploratory templates: before stopping, each documented input rule has its own capture breaking only that rule; at the 12-probe cap, uncovered rules go into limits. Regenerate both host outputs. Text assertion that the final rule is single and coherent; behaviour is proven by the paid cases.
**W3-10. Shim-dom detector resolves literal assignments (1; 38980005793).** test/skill-e2e-design.test.ts:1248 looks for the literal `dom/run1`, but the agent ran the generated block's `"$_REPORT/dom/$_RUN"` with `_RUN="run1"` assigned in the same command. The check resolves the command's own literal `_RUN=`/`_REPORT=` assignments before matching. This is a gate edit and is listed in the PR body. Free regression: the captured command passes; `scan --changed` and a scan over another directory still fail.
**W3-10. Shim-dom detector resolves literal assignments (1; 38980005793).** test/skill-e2e-design.test.ts:1248 looks for the literal `dom/run1`, but the agent ran the generated block's `"$_REPORT/dom/$_RUN"` with `_RUN="run1"` assigned in the same command. The check resolves that command's own literal `_RUN=`/`_REPORT=` assignments before matching. Free regression: the captured command passes; `scan --changed` and a scan over another directory still fail.
**W3-11. PTY harness keeps the last non-empty screen (diagnostic; 38976253774).** autoplan-guard-pty stalled at 489 s (normal 71-93 s) with empty journal and screen dumps, so the cause is undetermined. test/helpers/pty/ keeps the last non-empty `visibleText()` for the failure dump. No verdict change.
**W3-11. PTY harness keeps the last non-empty screen (diagnostic; 38976253774).** autoplan-guard-pty stalled at 489 s (normal 71-93 s) with empty journal and screen dumps. test/helpers/pty/ keeps the last non-empty `visibleText()` for the failure dump. No verdict change.
**W3-12. Explicit timeout for test/autoplan-answer-invalidation.test.ts.** Each test in the file gets an explicit per-test timeout sized from measured 4-core durations (30 s), matching the repo's other subprocess-heavy autoplan tests.
**W3-12. Explicit timeout for test/autoplan-answer-invalidation.test.ts.** Each test gets an explicit 30 s timeout (measured 4.1 s for the eng-only test on 4 cores; 7.8-9.4 s per file).
**W3-13. Close the stale createQAFunctionalFixture TODO.** TODOS.md's fix-wave-2 entry is already fixed by 0bf9d50; the entry is removed with that citation.
**W3-13. Close the stale createQAFunctionalFixture TODO** (already fixed by 0bf9d50) in TODOS.md, with the citation.
**W3-14. One superseded rule for qa fix runs (2; 39053431488 on #3252).** qa-functional-webhook-fix failed with "missing exact sanitized evidence" because the agent labelled a pre-repair duplicate capture `superseded` without rerunning it. qa/sections/exploratory.md:88 says to classify a capture taken before an input change `superseded`; lib/qa-evidence.ts:346-348 keeps such a row open; the fixture prompt's schema (qa-functional-eval.ts) never mentions superseded, and the detector (qa-functional-evidence.ts:160-166) accepts it only after a rerun, pinned on purpose by test/qa-functional-evidence.test.ts:376-380. Fix where the actor reads it: the fixture prompt states "a capture taken before your repair that you do not rerun keeps the label of what it observed and discloses in limits that it predates the repair; label superseded only a pre-repair capture whose same command you reran on the repaired source"; exploratory.md.tmpl (qa and qa-only) scopes its sentence the same way: rerun, else keep the observed label, with the open-superseded path kept for the helper's older-snapshot case. The detector and its negative test stay unchanged. W3-6's rerun rule also reruns duplicate, which would have closed this row. Free regression: the captured pre-repair duplicate labelled superseded and not rerun fails (negative control); prompt-text assertion for the rule.
**W3-14. Superseded captures are rerun before completion (2; 39053431488 on #3252).** qa-functional-webhook-fix failed "missing exact sanitized evidence" because the agent labelled a pre-repair duplicate capture `superseded` and never reran it; the product (qa/sections/exploratory.md:88, lib/qa-evidence.ts:346-348) keeps such a row open, the detector (qa-functional-evidence.ts:160-166) requires a rerun, and the fixture prompt's schema never mentions superseded. The production semantics stay: the fixture prompt adds "after your final repair, rerun every pre-repair capture your verdict relies on; a superseded row stays open until its command reran on the repaired source", and scripts/resolvers/qa.ts adds the same completion step for fix runs (rerun superseded captures, or report them as an open remainder), including snapshotless clients. The detector and its negative test (test/qa-functional-evidence.test.ts:376-380) are unchanged. Free tests: the captured pre-repair duplicate labelled superseded and not rerun fails (negative control); the prompt states the rule.
**W3-15. Windows budget for test/review-army-budget.test.ts (free; windows-free-tests run 39053431430, shard 5).** The nested `bun test` did not hit its 8 s spawn timeout: it finished in 5.58 s and exited 1 because Bun's default 5 s hook timeout fired on the real Red Team describe's `beforeAll` (8 git spawns plus skill-section copies; test/skill-e2e-review-army.test.ts:643-667), slow on windows-latest (Windows seed 4,143 ms for the file vs 1,879 ms on Linux). The nested argv at :57 and :137 gains `--timeout 30000` (the asserted registrations pass explicit timeouts, so the contract under test is unchanged); spawn timeouts move to 40 s and the outer test timeouts to 45 s, including the consensus test that has none today. Not chosen: a `beforeAll` timeout in the paid e2e file (selects paid cases) or Windows curation (that list is for structural incompatibility). Free check: the nested script with an injected 6 s `beforeAll` still prints the expected lifecycle; the Windows proof is the PR's windows-free-tests run.
**W3-15. Windows budget for test/review-army-budget.test.ts (free; windows-free-tests run 39053431430, shard 5).** The nested `bun test` finished in 5.58 s and exited 1 because Bun's default 5 s hook timeout fired on the real Red Team `beforeAll` (8 git spawns plus skill-section copies, test/skill-e2e-review-army.test.ts:643-667), slow on windows-latest (Windows seed 4,143 ms for the file vs 1,879 ms on Linux). The nested argv at :57 and :137 gains `--timeout 30000`; spawn timeouts move to 40 s and outer test timeouts to 45 s, including the consensus test that has none. Free check: the nested script with an injected 6 s `beforeAll` still prints the expected lifecycle; the Windows proof is the PR's windows-free-tests run.
Not repaired, kept visible: Q3 webhook-report (4; the two denied `cat` calls become negative controls in the qa-functional observer tests) and T qa-quick (one 300 s timeout with passing runs at 164-272 s; trimming what Quick tier loads is a product call, filed in TODOS).
Not repaired, kept visible: Q3 webhook-report (4; the two denied `cat` calls become negative controls in the qa-functional observer tests) and T qa-quick (one 300 s timeout against passing runs of 164-272 s; trimming what Quick tier loads is a product call, filed in TODOS).
### Deferred asks
- **D1. Preamble-hub diffs (cost below the floor).** 46 hub files reach the whole gate under any sound map. A cheaper contract (a free contract suite for the preamble bins plus a 2-3 case paid canary for hub-only diffs, the rest deferred to the census) trades PR-time coverage for cost. Not built here; it needs Garry's call on that trade after W2-3's real numbers land. Recommendation: later.
- **D2. Generator-input narrowing** (scripts/resolvers/index.ts, preamble/**, gen-skill-docs.ts selected through committed outputs: 29 runs leave full-fallback on replay, mostly into dependents-all). Deferred until every host output a paid case loads is shown committed and freshness-gated.
- **D3. Nested child output pollutes free-runner failure attribution** (review-army-budget's child `(fail)` line was attributed to its temp script, so the flaky retry was skipped). scripts/lib/shard-engine.ts is outside this wave; filed in TODOS.
- **D1. Preamble-hub diffs.** 46 hub files reach the whole gate under any sound map. A cheaper contract (a free contract suite for the preamble bins plus a 2-3 case paid canary for hub-only diffs) trades PR-time coverage for cost; Garry's call after A1-3's metrics and A2's shadow data. Recommendation: later.
- **D2. Generator-input narrowing** (resolvers/index.ts, preamble/**, gen-skill-docs.ts through committed outputs; 29 runs leave full-fallback on replay, mostly into dependents-all). Waits until every host output a paid case loads is shown committed and freshness-gated.
- **D3. Map activation** (A2 selects instead of shadowing; free-fixtures-data.ts leaves the global set with it). Needs PTY command capture for the six blind cases, two weeks of shadow data with zero shadow misses, and Garry's call on the measured saving.
- **D4. Run-count levers** (225 paid runs for 68 PRs; #3226 ran 13 full gates): readiness-gated or reuse-across-push dispatch, analysed read-only next wave.
- **D5. Red attribution in the paid report** (does the diff reach a red case directly, or only through a hub or the fallback). Report-only; next wave.
- **D6. Nested child output pollutes free-runner failure attribution** (scripts/lib/shard-engine.ts); filed in TODOS.
### Validation
Per AGENTS.md Validation discipline: one current failure list in `.context/fix-wave-3.md`; focused free tests per item during repair; `bun test test/parity-suite.test.ts`, the SKILL.md size guard (CREW.md 11), skill-docs freshness, `test/spawnsync-timeout-tripwire.test.ts` and credential scanning before any paid run; W3's affected paid cases (ship-docsync-stale-after, ship-exploratory-small-cli, ship-exploratory-plan-checks, qa-functional-webhook-fix, tpa-absent-linux, plan-mode-no-op, qa-functional-cli-report, design-review-detector-shim-dom) run once each at their EVAL_POLICY kind through `bun run eval:bg:pr` before the full gate; then the PR gate; then `bun run test` once on the frozen tree via Ubicloud. Paid evals never retry; a red is diagnosed from its transcript against the diff.
Per AGENTS.md Validation discipline: one current failure list in `.context/fix-wave-3.md`; focused free tests per item during repair; then the cheap checks (typecheck, `bun test test/parity-suite.test.ts`, the SKILL.md size guard per CREW.md 11, skill-docs freshness, `test/spawnsync-timeout-tripwire.test.ts`, credential scanning); then the selected quality judges for changed prompts, reusing unchanged passing judge evidence by prompt-input identity; then PR B's focused paid cases once each at their EVAL_POLICY kind (ship-docsync-stale-after, ship-docsync-missing-asset, ship-exploratory-small-cli, ship-exploratory-plan-checks, qa-functional-webhook-fix, tpa-absent-linux, plan-mode-no-op, qa-functional-cli-report, design-review-detector-shim-dom); then each PR's gate; then `bun run test` once per PR on the frozen tree via Ubicloud. Paid evals never retry; a red is diagnosed from its transcript against the diff. Executed, reused and remaining required cases are reconciled before READY.
### Delivery shape
Two PRs with disjoint files, opened with Capy's PR tool, READY unstamped to GStack Minder:
- **PR A (W2):** scripts/pr-dependencies.ts, scripts/test-pr-profile.ts, test/helpers/test-selection.ts (map-diff for free-fixtures-data), test/helpers/exercised-commands.json with its refresh script, the MAP_GAP report step (scripts/lib/paid-report.ts, a global touchfile) and the periodic census guard (.github/workflows/evals-periodic.yml), scripts/test-health-report.ts, scripts/ubicloud/pregate-remote.sh, scripts/test-free-shards.ts (`--files-from` only), .gstack/pregate.json, bin/gstack-next-version, their tests, docs/TESTING_INTERNALS.md, its own TODOS.md hunks.
- **PR B (W3):** test/helpers/docsync-fault-actor.ts, docsync-fault-eval.ts, docsync-observer.ts, bin/gstack-docs-candidate, test/helpers/qa-callers-fixture.ts, scripts/resolvers/review-army.ts, third-party-actions.ts, plan-reality.ts, the qa and qa-only exploratory templates, test/helpers/qa-functional-eval.ts, qa-functional-evidence.ts, test/qa-functional-evidence.test.ts, test/skill-e2e-design.test.ts, test/helpers/pty/*, test/autoplan-answer-invalidation.test.ts, test/review-army-budget.test.ts, detector corpora, regenerated SKILL.md and sections, its own TODOS.md hunks.
- Shared file: TODOS.md only, in separate hunks; whichever lands second merges main. PR A touches no skill template, so only PR B regenerates.
- scripts/test-free-shards.ts: confirm with GStack Minder that W4 (GSTA-33) is not editing it; otherwise PR A waits for W4's version.
Four PRs, opened with Capy's PR tool, READY unstamped to GStack Minder:
- **A1:** scripts/pr-dependencies.ts (shebang), scripts/test-pr-profile.ts (classification), scripts/test-health-report.ts, docs/TESTING_INTERNALS.md, their tests.
- **A2 (stacked on A1):** scripts/pr-dependencies.ts (`exercisedConsumers`), scripts/test-pr-profile.ts (shadow block), test/helpers/exercised-commands.json, its refresh script and package script, scripts/lib/paid-report.ts (MAP_GAP), the census job in .github/workflows/evals-periodic.yml, test:health shadow lines, docs/TESTING_INTERNALS.md#exercised-path-map, their tests.
- **A3:** scripts/ubicloud/pregate-remote.sh, scripts/test-free-shards.ts (`--files-from` only), .gstack/pregate.json, bin/gstack-next-version, docs/pregate.md (example only), CONTRIBUTING.md (one row), their tests.
- **B (W3):** test/helpers/docsync-fault-actor.ts, docsync-fault-eval.ts, docsync-fixture.ts, docsync-observer.ts, bin/gstack-docs-candidate, test/helpers/qa-callers-fixture.ts, scripts/resolvers/review-army.ts, third-party-actions.ts, plan-reality.ts, qa.ts, test/helpers/qa-functional-eval.ts, test/qa-functional-evidence.test.ts, test/skill-e2e-design.test.ts, test/helpers/pty/*, test/autoplan-answer-invalidation.test.ts, test/review-army-budget.test.ts, detector corpora, regenerated SKILL.md and sections.
- Shared: TODOS.md (separate hunks) and docs/TESTING_INTERNALS.md (A1 and A2, stacked). Whichever lands second merges main. Only B regenerates skills.
- A3 changes every sub-8-core lane's pregate once merged; its READY report carries the crew rule and its PR body lists it under Behavior changes.
### Out of scope
W4 (seed recorder), W5 (pregate lanes, lib/pregate/**), W6-W9, generator-input narrowing (deferred with its replay effect), a lockfile narrowing rule, and any change to EVAL_POLICY, case kinds, trials, thresholds or quarantine.
W4 (seed recorder), W5 (pregate lanes, lib/pregate/**), W6-W9, map activation (D3), generator-input narrowing (D2), a lockfile narrowing rule, and any change to EVAL_POLICY, case kinds, trials, thresholds or quarantine.
## Review record
Revision 1, autoplan run run1-20261017181219 (CEO, DX, Eng; Claude Opus 5.5 native, GPT-6.1 Sol outside): CEO 13 + 3 findings, DX 16 + 9 (1 Critical: the scope detector's output used without its exit status, now W3-5), Eng 15 + 11. Gate rev 1 recommendations d3b, d4b, d5b, d6b, d7b, d8b changed this revision; d1a and d2a kept the direction. All other findings are build obligations folded into the items above.