# Test, eval and CI audit — October 2026

This document records the evidence behind the October 2026 test, eval and CI
wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest,
and the commands that reproduce every number. Metrics that need post-merge
traffic are measured with `bun run test:health --since-days 7` (merge + 7 days)
and `bun run test:health --since-days 28` (after the fourth scheduled census).

## EVAL_POLICY v2 decision memo (D1)

### Why the census is red

The weekly census is red by arithmetic, not by a handful of broken tests.
Over 22 censuses with trial outcomes the periodic lane recorded 76 failed
trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in
2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3%
for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green
rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of
the time needs every case at or below about 0.12% failure per run, which a
live agent judged by one trial does not reach. Fixing the top five chronic
cases leaves the lane red.

Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make
detectors deterministic over structured artifacts (W3), classify INFRA
honestly (W2d, owned by the reliability follow-ups), and make quarantine
reachable for detector, harness and model-latency failures. v2 targets the
third lever.

`test:health` reports the census as three separate numbers, so a policy
change cannot read as a reliability gain: the raw trial failure rate per kind
and failure class, P(lane red) from pooled per-case rates under the recorded
panels, and the same evidence re-aggregated with every case judged on one
trial. Both red probabilities treat every observed failure as
nondeterminism, so they are upper bounds for known-good inputs.

**False-blocking targets.** Periodic lane: at most 1.0 expected failed trial
per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5
(P(red) ≤ 39%). Neither target may be reached by a threshold, retry or
rejudge change. A miss opens a P1 TODOS entry with the per-kind,
per-failure-class breakdown from `test:health`.

### What v2 changes

- **Series identity (W4a, CEO-27, ENG-3).** A case's pass-rate series is keyed
  by the bytes it owns plus `HARNESS_VERSION`: its paid test file, its
  fixtures, and the prompt files of the skills it names (templates, sections,
  manifests and the generated `SKILL.md` the model reads). The series key adds
  the model id, the pinned Claude Code CLI version and the policy version.
  Shared helpers no longer start new series. Every trial also records the
  full consumed-input fingerprint (all touchfiles, globals included) as
  `series_fingerprint`, plus `harness_version`, as provenance.
  (`caseSeriesIdentitiesV2` in `scripts/eval-trial-series.ts`.)
- **Harness guard.** `scripts/harness-version.json` pins the blob of every
  shared execution-harness file: `GLOBAL_TOUCHFILES` plus the session runners,
  the PTY harness, the e2e gate and the LLM judge (36 files). A free test
  (`test/harness-version.test.ts`) fails when one changes without a decision.
  `bun run scripts/bump-harness-version.ts --bump "<reason>"` raises
  `HARNESS_VERSION` and starts a new series for every case;
  `--non-behavioral "<reason>"` records the new blobs as compatible with no
  reset. Entries are one line per file so concurrent PRs that touch different
  files merge cleanly; on a conflict, take the higher version, then rerun.
- **Weekly history (W4b).** Weeks for quarantine expiry and drift are
  `evals-periodic.yml` runs on `main`: scheduled runs and main dispatches.
  Trials from branch census runs in the same window pool into a series `main`
  has also run (option (b) below, approved 2026-10-05); a branch identity
  `main` has not run is dropped and never becomes the current series.
- **Kinds (W4c).** No kind changes; see the evidence below.
- **Pre-v2 evidence does not qualify.** Readers segment strictly by
  `policy_version`: only v2 trials count toward the quarantine entry rule; v1
  trials stay visible as display-only. Quarantine is not seeded at merge; a
  case enters only after it reaches `entry.minTrials` qualifying v2 trials,
  and `test:health` shows qualifying-vs-required counts with "insufficient
  evidence" distinct from healthy (DX-12).

### W4c: the named "stochastic" cases are deterministic in the record

Recorded census and PR trials (all 95 recorded revisions, October 2026):

| Case | Kind | Passed / trials | Failures |
|---|---|---|---|
| plan-ceo-finding-floor | rule | 26/27 | 1 timeout (608 s) |
| plan-devex-finding-floor | rule | 27/27 | — |
| plan-eng-finding-floor | rule | 9/9 | — |
| plan-design-finding-floor | rule | 9/9 | — |
| plan-ceo-review-plan-mode | rule | 34/34 | — |
| plan-devex-review-plan-mode | rule | 27/27 | — |
| plan-eng-review-plan-mode | rule | 9/9 | — |

The comments calling `plan-*-finding-floor` "stochastic ask-first" are stale:
the record shows these cases behave like rules. AGENTS.md forbids changing a
kind without pass-rate evidence, and a 3-trial panel would triple their cost
for no measurable gain, so they stay `rule`. `plan-design-review-plan-mode`
(17/22) is handled by its detector repair, not by a kind change.

### Backtest (ENG-3, DX-12)

`bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir>`
replays identities with today's registry over historical trees (cases whose
test file did not exist at a revision are absent there).

**Weekly main revisions** (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28).
A case qualifies at a week when one identity holds long enough to collect 10
trials within the last 10 weekly runs (rule and judge cases record one trial
per run, behavior cases a panel of three):

| Identity | Qualifying at the latest week | Mean qualifying share, last 8 weeks | Median identity changes per case |
|---|---|---|---|
| v1 (own touchfiles minus globals) | 0/195 | 0.3% | 15 |
| v2 (case-owned bytes) | 0/195 | 0.7% | 14 |
| v2, reset on every pinned-harness change | 0/195 | 0.1% | 14 |
| v2 without the generated `SKILL.md` (prototype) | 0/217 | 2.0% | 7 |

**Recorded trials** re-keyed by the v2 identity of the revision that produced
them (blocking cases with at least 10 trials on one series):

| Trial source | Recorded (v1) identity | v2 identity |
|---|---|---|
| 22 census runs (5,665 trials) | 52/210 | 67/210 |
| Census + PR lane, 95 revisions (14,898 trials) | 107/210 | 134/210 |

**Harness churn.** The pinned harness changed in 24 of 28 consecutive-week
transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04)
touched a pinned file. About half of PRs therefore record a
`bump-harness-version` decision, and only `--bump` decisions reset series.

### What the backtest shows

1. v2 makes series honest and longer when trials are dense: on the recorded
   census runs, 67 blocking cases reach 10 trials on one series versus 52
   under v1, and 134 versus 107 with PR-lane trials included.
2. Under the approved weekly history (main scheduled runs plus main
   dispatches, about one run per week), no blocking case reaches 10
   qualifying trials at any recent week. A rule case needs ten consecutive
   weekly runs on unchanged owned bytes, and a case's own prompt files
   change about every two weeks (the generated `SKILL.md` carries the shared
   preamble). Excluding the generated `SKILL.md` halves identity churn but
   still qualifies at most 2% of cases, and it would let preamble and resolver
   prompt changes pool unless every such edit went through the harness guard.
3. So v2 alone does not make quarantine reachable. Reaching it needs one of:
   (a) keep the history as approved and accept that quarantine stays rare, so
   census redness is fixed at source (W3) and tracked by `test:health`;
   (b) pool trials from every v2 census run whose series key matches,
   including branch dispatches, while weeks for expiry still count from main
   only — the recorded backtest's 67/210, at the cost of pooling branch code
   whose unbumped harness edits match; or (c) extra census dispatches on an
   unchanged main revision (about $175 recorded each). Garry approved (b) on
   2026-10-05, and this PR implements it: pooled branch trials count only
   toward a series a `main` run also has.

### Alternatives considered

- **Gating vs trended.** The native CEO voice proposed keeping live-agent
  cases trended (reported, never blocking) and gating only on deterministic
  detectors. v2 keeps the census blocking because quarantine and the
  pass-rate gate already provide a trended path per case; the
  false-blocking targets above are the check on that choice. If the 28-day
  check misses them, trended reporting for the remaining stochastic cases is
  the next proposal.
- **Adaptive sequential panels** (one trial, escalate to three on a failure,
  about 1.04× cost across ~190 cases) would be the strongest lever for a
  green census. They are not adopted: an escalation triggered by a failure is
  a retry, which Garry's rule forbids.
- **Buy vs build.** Hosted eval platforms (Braintrust, LangSmith, Weights &
  Biases Weave) store per-case pass-rate history and cost natively, but
  gstack's verdicts are produced inside GitHub Actions from Claude Code
  sessions with pre-registered panels and receipts, and the history is a few
  thousand JSONL lines per week that `gh` already serves. A platform would add
  an external dependency and a second source of truth for verdicts without
  removing the in-repo policy. Building on the existing artifacts stays the
  choice; revisit if history outgrows artifact retention (90 days).

### Rollout and rollback (ENG-14, DX-12)

The reader-compatibility commit lands first: readers score only trials under
their own `EVAL_POLICY.version`, show older-policy trials as display-only,
and ignore newer-policy trials with a printed count. The v2 writer (identity
stamping and `version: 2`) lands last. The supported rollback is reverting
the v2 writer commits; a whole-PR revert is not claimed safe.

On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3
failures), the pre-PR reader at 2db0b3a pools the v2 records as the current
v1 series and fails its gate:

```
pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
  FAILING       rule      gate      9/12 [46.8%–91.1%]  ...  plan-ceo-review-plan-mode  (baseline reset)
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
exit=1
```

The compatibility reader ignores the v2 records and keeps the v1 series:

```
pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
  ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
  PASSING       rule      gate      10/10 [72.2%–100%]  ...  plan-ceo-review-plan-mode
exit=0
```

## Wave evidence

### PR paid lane selection replay (CEO-23, ENG-18)

The replay sends each recorded PR push's changed files through today's
selector and dependency graph:

```bash
gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
bun run scripts/replay-pr-selection.ts --runs runs.json
```

Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):

| Measure | Recorded | Replayed |
|---|---:|---:|
| Full-fallback pushes | 165 of 168 (98.2%) | 164 of 168 (97.6%) |
| Misses (a failed case that depends on the diff and is no longer selected) | — | 0 |
| Failed cases that no longer exist (counted apart) | — | 6 |

113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on
their first 300 files. The rate barely moves because the replay uses each push's
cumulative branch diff, and the wave branches of that week touch shared inputs
that correctly select every case: `scripts/` in 156 pushes, `.github/` in 125
(mostly the eval workflows), `bin/` in 120, `make-pdf/` in 90, `test/helpers/` in
86, `design/` in 67, `lib/` in 58, `hosts/` in 56 and `tsconfig*.json` in 37. The
selector removes the single-class fallbacks (duration seeds, free-only workflows,
`scripts/ubicloud/**`, free tests and free fixtures; run 36497566037 now selects
the `pr` profile) and cannot narrow a branch that changes `bin/` or `lib/`;
narrowing those needs touchfile mappings for `bin/`, `lib/`, `hosts/` and
`make-pdf/`. The post-merge `test:health` check measures the fallback rate on
ordinary PRs.

### Paid slices and concurrency (ENG-17)

Live census with `EVALS_ALL=1` (periodic: `--list --slice-budget S --jobs 2`;
gate census: `--emit-plan <file> --slice-budget S --jobs 2 --skip-judges`):

| Lane | Before (540 s) | After (420 s) |
|---|---|---|
| Periodic slices / peak shard processes | 26 / 52 | 33 / 66 (max-parallel 26 → 36) |
| Weekly gate census slices / peak | 12 / 24 | 16 / 32 (max-parallel 16 → 20) |
| CI job ceiling | One value for every slice: 173 min periodic, 254 min gate census | Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min |

The PR lane (`evals.yml`) uses the same 420-second budget and per-slice ceilings.

### Free suite

- The Windows lane runs on six `windows-latest` jobs packed by Windows-measured
  durations: about 4.6 minutes wall in the free-suite branch's validation runs,
  down from 8.1 minutes on one runner.
- Unseeded files pack at the 99th percentile (was the 75th), and the seed growth
  ratchet (`test/free-seed-ratchet.test.ts`) keeps files over 60 seconds on a
  shrink-only allowlist.
- The bash+zsh portability tests (#2669) run in CI now that the free-suite job
  installs zsh.
- Native Windows and Dia qualification campaigns moved out of the PR workflow
  into the dispatch-only `native-qualification.yml`.

### Prose pins (D2)

D2 is approved: the prompt-byte contract in
[test-value-bar.md](test-value-bar.md) covers machine-read tokens only, and
sentence pins in 55 free test files became structural checks through
`test/helpers/prompt-structure.ts`. A sentence pin here is a 40-or-more-character
English string literal, or a prose regex of five or more words, asserted against
SKILL.md, template or section text in a free test.

| Measure | main (2db0b3a) | This wave |
|---|---:|---:|
| Sentence pins | 2,332 in 217 files | 1,190 in 209 files |
| Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis) | 8 | 4 |

The 4 remaining shouting pins are enum or marker tokens (`PLAN MODE EXCEPTION —
ALWAYS RUN`, `[CRITICAL]`, `ALWAYS-ON` CLI output), not emphasis. Most remaining
sentence pins sit in QA caller, detector-helper and workflow files that other
work in flight owns.

## Post-merge follow-up (2026-10-15)

The "post-merge audit metrics" TODO, from a no-spend run of
`bun run test:health --since-days 7` (window 2026-10-08T22:14Z to
2026-10-15T22:14Z). The 2026-10-12 figures are the weekly health run in issue
#3154 (run 38234556073), kept as dated history.

| Metric | 2026-10-12 | 2026-10-15 | Target | Verdict |
|---|---:|---:|---|---|
| PR full-fallback rate | 45.0% (58/129 runs) | 45.9% (39/85 runs) | none numeric; the replay above names the cause | open: branches touching `bin/`, `lib/`, `hosts/` or `make-pdf/` still select every case |
| Recorded PR-lane spend per merged PR | $75.09 (46 merged) | $73.52 (33 merged) | down at least 50% from the pre-change baseline | unavailable: baseline not recorded (below) |
| Cancelled slice-minutes per week | 1,939 (44 runs) | 1,545 (33 runs) | none numeric | reported |
| PR push-to-verdict | p50 12.0 min, p95 1,160.7 min (122 runs) | p50 11.4 min, p95 107.9 min (80 runs) | 3–5 minutes for typical PR checks (CONTRIBUTING.md) | missed |
| Free-suite flaky-pass rate | 13.0% (6/46 main runs) | 6.1% (2/33 main runs; `terminal-agent-publication-lock` 2/33, over the 5% per-file check) | under 5% of main runs | missed; fix wave 2 repairs the five ledger files |

**Spend baseline.** The pre-change window was recomputed with the exported
`collectHealth` at `now = 2026-10-04T00:00Z`, seven days, through a client that
drops runs created and PRs merged after that instant (`listWorkflowRuns` has no
upper bound of its own). The window holds 12 merged PRs but no
`trial-outcomes-pr-*` artifacts and no paid-plan manifest with `prCoverage`:
the recorder that feeds this metric shipped with the audit, so the PR lanes
before it wrote nothing it can read (their other artifacts are retained until
2026-11-02 and later). Baseline not recorded; verdict unavailable. The P1 in
TODOS.md is opened on the absolute figures.

**Census reliability (not yet due).** Due after the fourth scheduled census
after the audit merge (2026-10-26). Interim: periodic 2.00 expected failed
trials per run (6/538 over 3 runs; target at most 1.0), gate census 0.33 (1/308
over 3 runs; target at most 0.5). The TODO keeps its due date; the census
repair stays with #3153 and #3200.