mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-17 13:44:32 +01:00
* test(review): one file per review and harness pin, so independent repins merge cleanly (GST-1084)
The reviewer-brief pins move from the table in test/review-packets-removed.test.ts to
test/fixtures/review-pins/{files,replays,brief-bodies}/*.json, and the harness pins from
scripts/harness-version.json to scripts/harness-pins/<path>.json. Every entry is ported
unchanged. New tests prove a stale pin still fails and that two repins of different pins
merge cleanly with a real git merge-tree.
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix: pre-landing review fixes (GST-1084)
- unadmitted() and the specialist coverage use Object.hasOwn, so an inherited name such as toString is not an admitted replay
- MUST_PIN names every non-specialist reviewer source; deleting one of those pins now fails
- writeManifest writes the pins before the manifest head; pin parse errors name the file
- mergePinEdits removes its repository when setup throws, and the merge tests clean up every merge they made
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: name the changelog fragment for the branch (GST-1084)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs: the harness guard pins one file per harness file (GST-1084)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(ship-release): GST-1031 re-pin route reads per-file harness pins (GST-1084)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* changelog: GST-1031 route reads per-file harness pins
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* docs(test-audit): harness guard pins 42 files
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
* fix(harness-version): a pin ahead of the manifest version counts as drift (GST-1084)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
---------
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
308 lines
17 KiB
Markdown
308 lines
17 KiB
Markdown
# Test, eval and CI audit — October 2026
|
||
|
||
This document records the evidence behind the October 2026 test, eval and CI
|
||
wave: the EVAL_POLICY v2 decision memo (D1) with its pre-registered backtest,
|
||
and the commands that reproduce every number. Metrics that need post-merge
|
||
traffic are measured with `bun run test:health --since-days 7` (merge + 7 days)
|
||
and `bun run test:health --since-days 28` (after the fourth scheduled census).
|
||
|
||
## EVAL_POLICY v2 decision memo (D1)
|
||
|
||
### Why the census is red
|
||
|
||
The weekly census is red by arithmetic, not by a handful of broken tests.
|
||
Over 22 censuses with trial outcomes the periodic lane recorded 76 failed
|
||
trials in 3,550 (about 3.5 failed trials per run) and the gate census 27 in
|
||
2,241 (about 1.2 per run). With independent failures, P(green) is e^-3.5 ≈ 3%
|
||
for the periodic lane and e^-1.2 ≈ 30% for the gate census; observed green
|
||
rates were 0/22 and 8/22. Holding a ~190-case single-trial lane green 80% of
|
||
the time needs every case at or below about 0.12% failure per run, which a
|
||
live agent judged by one trial does not reach. Fixing the top five chronic
|
||
cases leaves the lane red.
|
||
|
||
Three levers stay inside the no-retry, no-threshold, no-rejudge rule: make
|
||
detectors deterministic over structured artifacts (W3), classify INFRA
|
||
honestly (W2d, owned by the reliability follow-ups), and make quarantine
|
||
reachable for detector, harness and model-latency failures. v2 targets the
|
||
third lever.
|
||
|
||
`test:health` reports the census as three separate numbers, so a policy
|
||
change cannot read as a reliability gain: the raw trial failure rate per kind
|
||
and failure class, P(lane red) from pooled per-case rates under the recorded
|
||
panels, and the same evidence re-aggregated with every case judged on one
|
||
trial. Both red probabilities treat every observed failure as
|
||
nondeterminism, so they are upper bounds for known-good inputs.
|
||
|
||
**False-blocking targets.** Periodic lane: at most 1.0 expected failed trial
|
||
per census (P(red) ≤ 63%) at the 28-day check; gate census: at most 0.5
|
||
(P(red) ≤ 39%). Neither target may be reached by a threshold, retry or
|
||
rejudge change. A miss opens a P1 TODOS entry with the per-kind,
|
||
per-failure-class breakdown from `test:health`.
|
||
|
||
### What v2 changes
|
||
|
||
- **Series identity (W4a, CEO-27, ENG-3).** A case's pass-rate series is keyed
|
||
by the bytes it owns plus `HARNESS_VERSION`: its paid test file, its
|
||
fixtures, and the prompt files of the skills it names (templates, sections,
|
||
manifests and the generated `SKILL.md` the model reads). The series key adds
|
||
the model id, the pinned Claude Code CLI version and the policy version.
|
||
Shared helpers no longer start new series. Every trial also records the
|
||
full consumed-input fingerprint (all touchfiles, globals included) as
|
||
`series_fingerprint`, plus `harness_version`, as provenance.
|
||
(`caseSeriesIdentitiesV2` in `scripts/eval-trial-series.ts`.)
|
||
- **Harness guard.** `scripts/harness-pins/<path>.json` pins the blob of every
|
||
shared execution-harness file (`scripts/harness-version.json` keeps the version and patterns): `GLOBAL_TOUCHFILES` plus the session runners,
|
||
the PTY harness, the e2e gate and the LLM judge (42 files). A free test
|
||
(`test/harness-version.test.ts`) fails when one changes without a decision.
|
||
`bun run scripts/bump-harness-version.ts --bump "<reason>"` raises
|
||
`HARNESS_VERSION` and starts a new series for every case;
|
||
`--non-behavioral "<reason>"` records the new blobs as compatible with no
|
||
reset. Each pin is its own file so concurrent PRs that touch different
|
||
files merge cleanly; on a conflict, take the higher version, then rerun.
|
||
- **Weekly history (W4b).** Weeks for quarantine expiry and drift are
|
||
`evals-periodic.yml` runs on `main`: scheduled runs and main dispatches.
|
||
Trials from branch census runs in the same window pool into a series `main`
|
||
has also run (option (b) below, approved 2026-10-05); a branch identity
|
||
`main` has not run is dropped and never becomes the current series.
|
||
- **Kinds (W4c).** No kind changes; see the evidence below.
|
||
- **Pre-v2 evidence does not qualify.** Readers segment strictly by
|
||
`policy_version`: only v2 trials count toward the quarantine entry rule; v1
|
||
trials stay visible as display-only. Quarantine is not seeded at merge; a
|
||
case enters only after it reaches `entry.minTrials` qualifying v2 trials,
|
||
and `test:health` shows qualifying-vs-required counts with "insufficient
|
||
evidence" distinct from healthy (DX-12).
|
||
|
||
### W4c: the named "stochastic" cases are deterministic in the record
|
||
|
||
Recorded census and PR trials (all 95 recorded revisions, October 2026):
|
||
|
||
| Case | Kind | Passed / trials | Failures |
|
||
|---|---|---|---|
|
||
| plan-ceo-finding-floor | rule | 26/27 | 1 timeout (608 s) |
|
||
| plan-devex-finding-floor | rule | 27/27 | — |
|
||
| plan-eng-finding-floor | rule | 9/9 | — |
|
||
| plan-design-finding-floor | rule | 9/9 | — |
|
||
| plan-ceo-review-plan-mode | rule | 34/34 | — |
|
||
| plan-devex-review-plan-mode | rule | 27/27 | — |
|
||
| plan-eng-review-plan-mode | rule | 9/9 | — |
|
||
|
||
The comments calling `plan-*-finding-floor` "stochastic ask-first" are stale:
|
||
the record shows these cases behave like rules. AGENTS.md forbids changing a
|
||
kind without pass-rate evidence, and a 3-trial panel would triple their cost
|
||
for no measurable gain, so they stay `rule`. `plan-design-review-plan-mode`
|
||
(17/22) is handled by its detector repair, not by a kind change.
|
||
|
||
### Backtest (ENG-3, DX-12)
|
||
|
||
`bun run scripts/eval-trial-series.ts --backtest --weeks 30 --records <dir>`
|
||
replays identities with today's registry over historical trees (cases whose
|
||
test file did not exist at a revision are absent there).
|
||
|
||
**Weekly main revisions** (29 Mondays, 06:00 UTC, 2026-03-16 to 2026-09-28).
|
||
A case qualifies at a week when one identity holds long enough to collect 10
|
||
trials within the last 10 weekly runs (rule and judge cases record one trial
|
||
per run, behavior cases a panel of three):
|
||
|
||
| Identity | Qualifying at the latest week | Mean qualifying share, last 8 weeks | Median identity changes per case |
|
||
|---|---|---|---|
|
||
| v1 (own touchfiles minus globals) | 0/195 | 0.3% | 15 |
|
||
| v2 (case-owned bytes) | 0/195 | 0.7% | 14 |
|
||
| v2, reset on every pinned-harness change | 0/195 | 0.1% | 14 |
|
||
| v2 without the generated `SKILL.md` (prototype) | 0/217 | 2.0% | 7 |
|
||
|
||
**Recorded trials** re-keyed by the v2 identity of the revision that produced
|
||
them (blocking cases with at least 10 trials on one series):
|
||
|
||
| Trial source | Recorded (v1) identity | v2 identity |
|
||
|---|---|---|
|
||
| 22 census runs (5,665 trials) | 52/210 | 67/210 |
|
||
| Census + PR lane, 95 revisions (14,898 trials) | 107/210 | 134/210 |
|
||
|
||
**Harness churn.** The pinned harness changed in 24 of 28 consecutive-week
|
||
transitions, and 33 of the last 60 main commits (2026-08-04 to 2026-10-04)
|
||
touched a pinned file. About half of PRs therefore record a
|
||
`bump-harness-version` decision, and only `--bump` decisions reset series.
|
||
|
||
### What the backtest shows
|
||
|
||
1. v2 makes series honest and longer when trials are dense: on the recorded
|
||
census runs, 67 blocking cases reach 10 trials on one series versus 52
|
||
under v1, and 134 versus 107 with PR-lane trials included.
|
||
2. Under the approved weekly history (main scheduled runs plus main
|
||
dispatches, about one run per week), no blocking case reaches 10
|
||
qualifying trials at any recent week. A rule case needs ten consecutive
|
||
weekly runs on unchanged owned bytes, and a case's own prompt files
|
||
change about every two weeks (the generated `SKILL.md` carries the shared
|
||
preamble). Excluding the generated `SKILL.md` halves identity churn but
|
||
still qualifies at most 2% of cases, and it would let preamble and resolver
|
||
prompt changes pool unless every such edit went through the harness guard.
|
||
3. So v2 alone does not make quarantine reachable. Reaching it needs one of:
|
||
(a) keep the history as approved and accept that quarantine stays rare, so
|
||
census redness is fixed at source (W3) and tracked by `test:health`;
|
||
(b) pool trials from every v2 census run whose series key matches,
|
||
including branch dispatches, while weeks for expiry still count from main
|
||
only — the recorded backtest's 67/210, at the cost of pooling branch code
|
||
whose unbumped harness edits match; or (c) extra census dispatches on an
|
||
unchanged main revision (about $175 recorded each). Garry approved (b) on
|
||
2026-10-05, and this PR implements it: pooled branch trials count only
|
||
toward a series a `main` run also has.
|
||
|
||
### Alternatives considered
|
||
|
||
- **Gating vs trended.** The native CEO voice proposed keeping live-agent
|
||
cases trended (reported, never blocking) and gating only on deterministic
|
||
detectors. v2 keeps the census blocking because quarantine and the
|
||
pass-rate gate already provide a trended path per case; the
|
||
false-blocking targets above are the check on that choice. If the 28-day
|
||
check misses them, trended reporting for the remaining stochastic cases is
|
||
the next proposal.
|
||
- **Adaptive sequential panels** (one trial, escalate to three on a failure,
|
||
about 1.04× cost across ~190 cases) would be the strongest lever for a
|
||
green census. They are not adopted: an escalation triggered by a failure is
|
||
a retry, which Garry's rule forbids.
|
||
- **Buy vs build.** Hosted eval platforms (Braintrust, LangSmith, Weights &
|
||
Biases Weave) store per-case pass-rate history and cost natively, but
|
||
gstack's verdicts are produced inside GitHub Actions from Claude Code
|
||
sessions with pre-registered panels and receipts, and the history is a few
|
||
thousand JSONL lines per week that `gh` already serves. A platform would add
|
||
an external dependency and a second source of truth for verdicts without
|
||
removing the in-repo policy. Building on the existing artifacts stays the
|
||
choice; revisit if history outgrows artifact retention (90 days).
|
||
|
||
### Rollout and rollback (ENG-14, DX-12)
|
||
|
||
The reader-compatibility commit lands first: readers score only trials under
|
||
their own `EVAL_POLICY.version`, show older-policy trials as display-only,
|
||
and ignore newer-policy trials with a printed count. The v2 writer (identity
|
||
stamping and `version: 2`) lands last. The supported rollback is reverting
|
||
the v2 writer commits; a whole-PR revert is not claimed safe.
|
||
|
||
On a mixed v1/v2 fixture (10 passing v1 trials, 12 v2 trials with 3
|
||
failures), the pre-PR reader at 2db0b3a pools the v2 records as the current
|
||
v1 series and fails its gate:
|
||
|
||
```
|
||
pass-rates: policy v1, 22 post-policy trial(s), 0 pre-policy (display only)
|
||
FAILING rule gate 9/12 [46.8%–91.1%] ... plan-ceo-review-plan-mode (baseline reset)
|
||
ACTION REQUIRED (2): [drift] ... [rule-as-behavior] ...
|
||
exit=1
|
||
```
|
||
|
||
The compatibility reader ignores the v2 records and keeps the v1 series:
|
||
|
||
```
|
||
pass-rates: policy v1, 10 post-policy trial(s), 0 pre-policy (display only)
|
||
ignored 12 trial(s) recorded under a policy newer than v1: this checkout predates them. Fix: ...
|
||
PASSING rule gate 10/10 [72.2%–100%] ... plan-ceo-review-plan-mode
|
||
exit=0
|
||
```
|
||
|
||
## Wave evidence
|
||
|
||
### PR paid lane selection replay (CEO-23, ENG-18)
|
||
|
||
The replay sends each recorded PR push's changed files through today's
|
||
selector and dependency graph:
|
||
|
||
```bash
|
||
gh run list -R garrytan/gstack --workflow evals.yml --json databaseId,headSha,event,createdAt,conclusion -L 200 > runs.json
|
||
bun run scripts/replay-pr-selection.ts --runs runs.json
|
||
```
|
||
|
||
Over 168 PR pushes with plan manifests (2026-09-25 to 2026-10-04):
|
||
|
||
| Measure | Recorded | Replayed |
|
||
|---|---:|---:|
|
||
| Full-fallback pushes | 165 of 168 (98.2%) | 164 of 168 (97.6%) |
|
||
| Misses (a failed case that depends on the diff and is no longer selected) | — | 0 |
|
||
| Failed cases that no longer exist (counted apart) | — | 6 |
|
||
|
||
113 of the 168 diffs exceed the compare API's 300-file cap and are replayed on
|
||
their first 300 files. The rate barely moves because the replay uses each push's
|
||
cumulative branch diff, and the wave branches of that week touch shared inputs
|
||
that correctly select every case: `scripts/` in 156 pushes, `.github/` in 125
|
||
(mostly the eval workflows), `bin/` in 120, `make-pdf/` in 90, `test/helpers/` in
|
||
86, `design/` in 67, `lib/` in 58, `hosts/` in 56 and `tsconfig*.json` in 37. The
|
||
selector removes the single-class fallbacks (duration seeds, free-only workflows,
|
||
`scripts/ubicloud/**`, free tests and free fixtures; run 36497566037 now selects
|
||
the `pr` profile) and cannot narrow a branch that changes `bin/` or `lib/`;
|
||
narrowing those needs touchfile mappings for `bin/`, `lib/`, `hosts/` and
|
||
`make-pdf/`. The post-merge `test:health` check measures the fallback rate on
|
||
ordinary PRs.
|
||
|
||
### Paid slices and concurrency (ENG-17)
|
||
|
||
Live census with `EVALS_ALL=1` (periodic: `--list --slice-budget S --jobs 2`;
|
||
gate census: `--emit-plan <file> --slice-budget S --jobs 2 --skip-judges`):
|
||
|
||
| Lane | Before (540 s) | After (420 s) |
|
||
|---|---|---|
|
||
| Periodic slices / peak shard processes | 26 / 52 | 33 / 66 (max-parallel 26 → 36) |
|
||
| Weekly gate census slices / peak | 12 / 24 | 16 / 32 (max-parallel 16 → 20) |
|
||
| CI job ceiling | One value for every slice: 173 min periodic, 254 min gate census | Per slice: ordinary 50 min (30-minute shard wall + 20), periodic outliers 52 and 73 min, the overlay slice 173 min (5 serialized overlay wrappers); gate census 34–82 min |
|
||
|
||
The PR lane (`evals.yml`) uses the same 420-second budget and per-slice ceilings.
|
||
|
||
### Free suite
|
||
|
||
- The Windows lane runs on six `windows-latest` jobs packed by Windows-measured
|
||
durations: about 4.6 minutes wall in the free-suite branch's validation runs,
|
||
down from 8.1 minutes on one runner.
|
||
- Unseeded files pack at the 99th percentile (was the 75th), and the seed growth
|
||
ratchet (`test/free-seed-ratchet.test.ts`) keeps files over 60 seconds on a
|
||
shrink-only allowlist.
|
||
- The bash+zsh portability tests (#2669) run in CI now that the free-suite job
|
||
installs zsh.
|
||
- Native Windows and Dia qualification campaigns moved out of the PR workflow
|
||
into the dispatch-only `native-qualification.yml`.
|
||
|
||
### Prose pins (D2)
|
||
|
||
D2 is approved: the prompt-byte contract in
|
||
[test-value-bar.md](test-value-bar.md) covers machine-read tokens only, and
|
||
sentence pins in 55 free test files became structural checks through
|
||
`test/helpers/prompt-structure.ts`. A sentence pin here is a 40-or-more-character
|
||
English string literal, or a prose regex of five or more words, asserted against
|
||
SKILL.md, template or section text in a free test.
|
||
|
||
| Measure | main (2db0b3a) | This wave |
|
||
|---|---:|---:|
|
||
| Sentence pins | 2,332 in 217 files | 1,190 in 209 files |
|
||
| Shouting pins (NEVER/MUST/ALWAYS/CRITICAL emphasis) | 8 | 4 |
|
||
|
||
The 4 remaining shouting pins are enum or marker tokens (`PLAN MODE EXCEPTION —
|
||
ALWAYS RUN`, `[CRITICAL]`, `ALWAYS-ON` CLI output), not emphasis. Most remaining
|
||
sentence pins sit in QA caller, detector-helper and workflow files that other
|
||
work in flight owns.
|
||
|
||
## Post-merge follow-up (2026-10-15)
|
||
|
||
The "post-merge audit metrics" TODO, from a no-spend run of
|
||
`bun run test:health --since-days 7` (window 2026-10-08T22:14Z to
|
||
2026-10-15T22:14Z). The 2026-10-12 figures are the weekly health run in issue
|
||
#3154 (run 38234556073), kept as dated history.
|
||
|
||
| Metric | 2026-10-12 | 2026-10-15 | Target | Verdict |
|
||
|---|---:|---:|---|---|
|
||
| PR full-fallback rate | 45.0% (58/129 runs) | 45.9% (39/85 runs) | none numeric; the replay above names the cause | open: branches touching `bin/`, `lib/`, `hosts/` or `make-pdf/` still select every case |
|
||
| Recorded PR-lane spend per merged PR | $75.09 (46 merged) | $73.52 (33 merged) | down at least 50% from the pre-change baseline | unavailable: baseline not recorded (below) |
|
||
| Cancelled slice-minutes per week | 1,939 (44 runs) | 1,545 (33 runs) | none numeric | reported |
|
||
| PR push-to-verdict | p50 12.0 min, p95 1,160.7 min (122 runs) | p50 11.4 min, p95 107.9 min (80 runs) | 3–5 minutes for typical PR checks (CONTRIBUTING.md) | missed |
|
||
| Free-suite flaky-pass rate | 13.0% (6/46 main runs) | 6.1% (2/33 main runs; `terminal-agent-publication-lock` 2/33, over the 5% per-file check) | under 5% of main runs | missed; fix wave 2 repairs the five ledger files |
|
||
|
||
**Spend baseline.** The pre-change window was recomputed with the exported
|
||
`collectHealth` at `now = 2026-10-04T00:00Z`, seven days, through a client that
|
||
drops runs created and PRs merged after that instant (`listWorkflowRuns` has no
|
||
upper bound of its own). The window holds 12 merged PRs but no
|
||
`trial-outcomes-pr-*` artifacts and no paid-plan manifest with `prCoverage`:
|
||
the recorder that feeds this metric shipped with the audit, so the PR lanes
|
||
before it wrote nothing it can read (their other artifacts are retained until
|
||
2026-11-02 and later). Baseline not recorded; verdict unavailable. The P1 in
|
||
TODOS.md is opened on the absolute figures.
|
||
|
||
**Census reliability (not yet due).** Due after the fourth scheduled census
|
||
after the audit merge (2026-10-26). Interim: periodic 2.00 expected failed
|
||
trials per run (6/538 over 3 runs; target at most 1.0), gate census 0.33 (1/308
|
||
over 3 runs; target at most 0.5). The TODO keeps its due date; the census
|
||
repair stays with #3153 and #3200.
|