gstack/docs/test-audit-2026-10.md

Test, eval and CI audit — October 2026

This document records the evidence behind the October 2026 test, eval and CI wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest, and the commands that reproduce every number. Metrics that need post-merge traffic are measured with bun run test:health --since-days 7 (merge + 7 days) and bun run test:health --since-days 28 (after the fourth scheduled census).

EVAL_POLICY v2 decision memo (D1)

Why the census is red

The weekly census is red by arithmetic, not by a handful of broken tests. Over 22 censuses with trial outcomes the periodic lane recorded 76 failed trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in 2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3% for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of the time needs every case at or below about 0.12% failure per run, which a live agent judged by one trial does not reach. Fixing the top five chronic cases leaves the lane red.

Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make detectors deterministic over structured artifacts (W3), classify INFRA honestly (W2d, owned by the reliability follow-ups), and make quarantine reachable for detector, harness and model-latency failures. v2 targets the third lever.

test:health reports the census as three separate numbers, so a policy change cannot read as a reliability gain: the raw trial failure rate per kind and failure class, P(lane red) from pooled per-case rates under the recorded panels, and the same evidence re-aggregated with every case judged on one trial. Both red probabilities treat every observed failure as nondeterminism, so they are upper bounds for known-good inputs.

False-blocking targets. Periodic lane: at most 1.0 expected failed trial per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5 (P(red) ≤ 39%). Neither target may be reached by a threshold, retry or rejudge change. A miss opens a P1 TODOS entry with the per-kind, per-failure-class breakdown from test:health.

What v2 changes

• Series identity (W4a, CEO-27, ENG-3). A case's pass-rate series is keyed by the bytes it owns plus HARNESS_VERSION: its paid test file, its fixtures, and the prompt files of the skills it names (templates, sections, manifests and the generated SKILL.md the model reads). The series key adds the model id, the pinned Claude Code CLI version and the policy version. Shared helpers no longer start new series. Every trial also records the full consumed-input fingerprint (all touchfiles, globals included) as series_fingerprint, plus harness_version, as provenance. (caseSeriesIdentitiesV2 in scripts/eval-trial-series.ts.)
• Harness guard. scripts/harness-pins/<path>.json pins the blob of every shared execution-harness file (scripts/harness-version.json keeps the version and patterns): GLOBAL_TOUCHFILES plus the session runners, the PTY harness, the e2e gate and the LLM judge (42 files). A free test (test/harness-version.test.ts) fails when one changes without a decision. bun run scripts/bump-harness-version.ts --bump "<reason>" raises HARNESS_VERSION and starts a new series for every case; --non-behavioral "<reason>" records the new blobs as compatible with no reset. Each pin is its own file so concurrent PRs that touch different files merge cleanly; on a conflict, take the higher version, then rerun.
• Weekly history (W4b). Weeks for quarantine expiry and drift are evals-periodic.yml runs on main: scheduled runs and main dispatches. Trials from branch census runs in the same window pool into a series main has also run (option (b) below, approved 2026-10-05); a branch identity main has not run is dropped and never becomes the current series.
• Kinds (W4c). No kind changes; see the evidence below.
• Pre-v2 evidence does not qualify. Readers segment strictly by policy_version: only v2 trials count toward the quarantine entry rule; v1 trials stay visible as display-only. Quarantine is not seeded at merge; a case enters only after it reaches entry.minTrials qualifying v2 trials, and test:health shows qualifying-vs-required counts with "insufficient evidence" distinct from healthy (DX-12).

W4c: the named "stochastic" cases are deterministic in the record

Recorded census and PR trials (all 95 recorded revisions, October 2026):

Case	Kind	Passed / trials	Failures
plan-ceo-finding-floor	rule	26/27	1 timeout (608 s)
plan-devex-finding-floor	rule	27/27	—
plan-eng-finding-floor	rule	9/9	—
plan-design-finding-floor	rule	9/9	—
plan-ceo-review-plan-mode	rule	34/34	—
plan-devex-review-plan-mode	rule	27/27	—
plan-eng-review-plan-mode	rule	9/9	—

The comments calling plan-*-finding-floor "stochastic ask-first" are stale: the record shows these cases behave like rules. AGENTS.md forbids changing a kind without pass-rate evidence, and a 3-trial panel would triple their cost for no measurable gain, so they stay rule. plan-design-review-plan-mode (17/22) is handled by its detector repair, not by a kind change.

Backtest (ENG-3, DX-12)

bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir> replays identities with today's registry over historical trees (cases whose test file did not exist at a revision are absent there).

Weekly main revisions (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28). A case qualifies at a week when one identity holds long enough to collect 10 trials within the last 10 weekly runs (rule and judge cases record one trial per run, behavior cases a panel of three):

Identity	Qualifying at the latest week	Mean qualifying share, last 8 weeks	Median identity changes per case
v1 (own touchfiles minus globals)	0/195	0.3%	15
v2 (case-owned bytes)	0/195	0.7%	14
v2, reset on every pinned-harness change	0/195	0.1%	14
v2 without the generated SKILL.md (prototype)	0/217	2.0%	7

Recorded trials re-keyed by the v2 identity of the revision that produced them (blocking cases with at least 10 trials on one series):

Trial source	Recorded (v1) identity	v2 identity
22 census runs (5,665 trials)	52/210	67/210
Census + PR lane, 95 revisions (14,898 trials)	107/210	134/210

Harness churn. The pinned harness changed in 24 of 28 consecutive-week transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04) touched a pinned file. About half of PRs therefore record a bump-harness-version decision, and only --bump decisions reset series.

What the backtest shows

1. v2 makes series honest and longer when trials are dense: on the recorded census runs, 67 blocking cases reach 10 trials on one series versus 52 under v1, and 134 versus 107 with PR-lane trials included.
2. Under the approved weekly history (main scheduled runs plus main dispatches, about one run per week), no blocking case reaches 10 qualifying trials at any recent week. A rule case needs ten consecutive weekly runs on unchanged owned bytes, and a case's own prompt files change about every two weeks (the generated SKILL.md carries the shared preamble). Excluding the generated SKILL.md halves identity churn but still qualifies at most 2% of cases, and it would let preamble and resolver prompt changes pool unless every such edit went through the harness guard.
3. So v2 alone does not make quarantine reachable. Reaching it needs one of: (a) keep the history as approved and accept that quarantine stays rare, so census redness is fixed at source (W3) and tracked by test:health; (b) pool trials from every v2 census run whose series key matches, including branch dispatches, while weeks for expiry still count from main only — the recorded backtest's 67/210, at the cost of pooling branch code whose unbumped harness edits match; or (c) extra census dispatches on an unchanged main revision (about $175 recorded each). Garry approved (b) on 2026-10-05, and this PR implements it: pooled branch trials count only toward a series a main run also has.

Alternatives considered

• Gating vs trended. The native CEO voice proposed keeping live-agent cases trended (reported, never blocking) and gating only on deterministic detectors. v2 keeps the census blocking because quarantine and the pass-rate gate already provide a trended path per case; the false-blocking targets above are the check on that choice. If the 28-day check misses them, trended reporting for the remaining stochastic cases is the next proposal.
• Adaptive sequential panels (one trial, escalate to three on a failure, about 1.04× cost across ~190 cases) would be the strongest lever for a green census. They are not adopted: an escalation triggered by a failure is a retry, which Garry's rule forbids.
• Buy vs build. Hosted eval platforms (Braintrust, LangSmith, Weights & Biases Weave) store per-case pass-rate history and cost natively, but gstack's verdicts are produced inside GitHub Actions from Claude Code sessions with pre-registered panels and receipts, and the history is a few thousand JSONL lines per week that gh already serves. A platform would add an external dependency and a second source of truth for verdicts without removing the in-repo policy. Building on the existing artifacts stays the choice; revisit if history outgrows artifact retention (90 days).

Rollout and rollback (ENG-14, DX-12)

The reader-compatibility commit lands first: readers score only trials under their own EVAL_POLICY.version, show older-policy trials as display-only, and ignore newer-policy trials with a printed count. The v2 writer (identity stamping and version: 2) lands last. The supported rollback is reverting the v2 writer commits; a whole-PR revert is not claimed safe.

On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3 failures), the pre-PR reader at 2db0b3a pools the v2 records as the current v1 series and fails its gate:

pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
  FAILING       rule      gate      9/12 [46.8%–91.1%]  ...  plan-ceo-review-plan-mode  (baseline reset)
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
exit=1

The compatibility reader ignores the v2 records and keeps the v1 series:

pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
  ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
  PASSING       rule      gate      10/10 [72.2%–100%]  ...  plan-ceo-review-plan-mode
exit=0

Wave evidence

PR paid lane selection replay (CEO-23, ENG-18)

The replay sends each recorded PR push's changed files through today's selector and dependency graph:

gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
bun run crew/scripts/replay-pr-selection.ts --runs runs.json

Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):

Measure	Recorded	Replayed
Full-fallback pushes	165 of 168 (98.2%)	164 of 168 (97.6%)
Misses (a failed case that depends on the diff and is no longer selected)	—	0
Failed cases that no longer exist (counted apart)	—	6

113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on their first 300 files. The rate barely moves because the replay uses each push's cumulative branch diff, and the wave branches of that week touch shared inputs that correctly select every case: scripts/ in 156 pushes, .github/ in 125 (mostly the eval workflows), bin/ in 120, make-pdf/ in 90, test/helpers/ in 86, design/ in 67, lib/ in 58, hosts/ in 56 and tsconfig*.json in 37. The selector removes the single-class fallbacks (duration seeds, free-only workflows, scripts/ubicloud/**, free tests and free fixtures; run 36497566037 now selects the pr profile) and cannot narrow a branch that changes bin/ or lib/; narrowing those needs touchfile mappings for bin/, lib/, hosts/ and make-pdf/. The post-merge test:health check measures the fallback rate on ordinary PRs.

Paid slices and concurrency (ENG-17)

Live census with EVALS_ALL=1 (periodic: --list --slice-budget S --jobs 2; gate census: --emit-plan <file> --slice-budget S --jobs 2 --skip-judges):

Lane	Before (540 s)	After (420 s)
Periodic slices / peak shard processes	26 / 52	33 / 66 (max-parallel 26 → 36)
Weekly gate census slices / peak	12 / 24	16 / 32 (max-parallel 16 → 20)
CI job ceiling	One value for every slice: 173 min periodic, 254 min gate census	Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min

The PR lane (evals.yml) uses the same 420-second budget and per-slice ceilings.

Free suite

• The Windows lane runs on six windows-latest jobs packed by Windows-measured durations: about 4.6 minutes wall in the free-suite branch's validation runs, down from 8.1 minutes on one runner.
• Unseeded files pack at the 99th percentile (was the 75th), and the seed growth ratchet (test/free-seed-ratchet.test.ts) keeps files over 60 seconds on a shrink-only allowlist.
• The bash+zsh portability tests (#2669) run in CI now that the free-suite job installs zsh.
• Native Windows and Dia qualification campaigns moved out of the PR workflow into the dispatch-only native-qualification.yml.

Prose pins (D2)

D2 is approved: the prompt-byte contract in test-value-bar.md covers machine-read tokens only, and sentence pins in 55 free test files became structural checks through test/helpers/prompt-structure.ts. A sentence pin here is a 40-or-more-character English string literal, or a prose regex of five or more words, asserted against SKILL.md, template or section text in a free test.

Measure	main (2db0b3a)	This wave
Sentence pins	2,332 in 217 files	1,190 in 209 files
Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis)	8	4

The 4 remaining shouting pins are enum or marker tokens (PLAN MODE EXCEPTION — ALWAYS RUN, [CRITICAL], ALWAYS-ON CLI output), not emphasis. Most remaining sentence pins sit in QA caller, detector-helper and workflow files that other work in flight owns.

Post-merge follow-up (2026-10-15)

The "post-merge audit metrics" TODO, from a no-spend run of bun run test:health --since-days 7 (window 2026-10-08T22:14Z to 2026-10-15T22:14Z). The 2026-10-12 figures are the weekly health run in issue #3154 (run 38234556073), kept as dated history.

Metric	2026-10-12	2026-10-15	Target	Verdict
PR full-fallback rate	45.0% (58/129 runs)	45.9% (39/85 runs)	none numeric; the replay above names the cause	open: branches touching bin/, lib/, hosts/ or make-pdf/ still select every case
Recorded PR-lane spend per merged PR	$75.09 (46 merged)	$73.52 (33 merged)	down at least 50% from the pre-change baseline	unavailable: baseline not recorded (below)
Cancelled slice-minutes per week	1,939 (44 runs)	1,545 (33 runs)	none numeric	reported
PR push-to-verdict	p50 12.0 min, p95 1,160.7 min (122 runs)	p50 11.4 min, p95 107.9 min (80 runs)	3–5 minutes for typical PR checks (CONTRIBUTING.md)	missed
Free-suite flaky-pass rate	13.0% (6/46 main runs)	6.1% (2/33 main runs; terminal-agent-publication-lock 2/33, over the 5% per-file check)	under 5% of main runs	missed; fix wave 2 repairs the five ledger files

Spend baseline. The pre-change window was recomputed with the exported collectHealth at now = 2026-10-04T00:00Z, seven days, through a client that drops runs created and PRs merged after that instant (listWorkflowRuns has no upper bound of its own). The window holds 12 merged PRs but no trial-outcomes-pr-* artifacts and no paid-plan manifest with prCoverage: the recorder that feeds this metric shipped with the audit, so the PR lanes before it wrote nothing it can read (their other artifacts are retained until 2026-11-02 and later). Baseline not recorded; verdict unavailable. The P1 in TODOS.md is opened on the absolute figures.

Census reliability (not yet due). Due after the fourth scheduled census after the audit merge (2026-10-26). Interim: periodic 2.00 expected failed trials per run (6/538 over 3 runs; target at most 1.0), gate census 0.33 (1/308 over 3 runs; target at most 0.5). The TODO keeps its due date; the census repair stays with #3153 and #3200.