diff --git a/docs/designs/SHIP_MEASURE_FIX_LOOP_2026_10.md b/docs/designs/SHIP_MEASURE_FIX_LOOP_2026_10.md new file mode 100644 index 000000000..dba1346d2 --- /dev/null +++ b/docs/designs/SHIP_MEASURE_FIX_LOOP_2026_10.md @@ -0,0 +1,733 @@ + +# /ship measure-then-fix loop for a red paid eval (issue #3034, October 2026) + +Status: plan, approved for the next wave by Garry ("yeah ok build it in the next wave"); reviewed by /autoplan (CEO, DX, Eng; design skipped, no UI); not implemented. One PR. + +## Implementation plan + +### What changes, in one paragraph + +Today `/ship` stops at the first red paid eval (`ship/sections/tests.md.tmpl`, Step 6.3: "If any eval fails: Show failures and available costs, then **STOP**"), and whoever is driving improvises, usually by fixing something, pushing and rerunning the whole ~130-case gate (~$90, ~12 minutes) to learn one bit about one case. On PR #3033 that happened six times, with a different single red each time. After this PR, `/ship` runs a fixed loop on each red case alone: classify the red, prove any detector or fixture fix for free first, check without any model spend that a trial really selects the case, show the cost, measure the case 10 times in parallel on throwaway Ubicloud VMs inside the CI image, fix at the cause, measure again, and only then run the gate. The loop stops patching after 3 rounds on one case and rethinks the test instead; it caps gate runs per ship; and a case it cannot fix ends as a named red that stops the ship with a decision, never a quiet pass. Every measurement trial is labelled diagnostic and can never become a verdict or enter pass-rate history. The PR body gets one table row per red case and a line with the gate's predicted all-green chance. + +### What success means + +The loop exists to cut three numbers per ship, and the PR-body summary reports all three: full-gate executions, total paid spend (measurement plus gates), and wall time from the first red to a trustworthy lane verdict. It does not make an unchanged stochastic case more reliable: measuring only tells us which cases need a fix. The gate-level problem (about 130 verdicts, each a little noisy, so an all-green gate is unlikely even when every case is healthy: 26.8% at a 99% per-verdict rate, per `docs/TESTING_INTERNALS.md`) is shown on every ship and its root fix is named under "Next, not in this PR". + +### Base and evidence + +Base: origin/main `55f8151` (v1.91.30.0, #3033). The pieces the loop stands on already exist: + +- `scripts/test-paid-shards.ts --case [--trials N] [--list]` runs one E2E case through the CI panel runner (`runCaseDiagnosis()` in `scripts/lib/paid-report.ts`), and every red report line prints an "after a repair: ..." command (`afterRepairCommand()`), which for a standalone judge is a `bun test` with `EVALS_JUDGE_SELECTION_JSON` and an exact name filter. +- Failed trial records carry `failure_class`, `failure_cause`, `failure_cause_evidence` and `failure_detail` (`test/helpers/eval-store.ts`, `failureCauseOf()`), and `bun run eval:pass-rates --run --case ` prints one red's values, history and transcript pointer; `eval:pass-rates --reds` prints the latest census's all-green probability. +- Detector replay corpora exist (`test/detector-corpus-*.test.ts`, `test/eng-batching-native-replay.test.ts`). +- The PR lane reuses verified first-attempt passes when every consumed input is byte-identical (`scripts/e2e-shard-reuse.ts`); a change to a global touchfile or the paid runner invalidates every receipt, which is why #3033's pushes reran the whole gate. +- `scripts/ubicloud/ubi-runner.sh` has `up | sync | ssh | pull | down | gc` and `--pass NAME` for secrets; the CI image is built from `.github/docker/Dockerfile.ci` and published as `ghcr.io/garrytan/gstack/ci`. + +Gaps found while planning, each closed by this PR: + +1. `--case` writes its shard records into the normal project eval dir (`getProjectEvalDir()`, `~/.gstack/projects//evals`), which `eval:list`, `eval:compare` and `eval:summary` read as ordinary runs. Diagnostic trials are unlabelled today. +2. `--case --list` proves selection statically only. Both #3033 selector bugs (the CLI deadlocking when the library imported the running entry module, and a single-case file filtered by a test name that did not exist) passed static selection and failed only when a trial ran. +3. `--case` accepts only E2E ids (`test-paid-shards.ts:825`), so a red standalone judge has no single-case trial runner. +4. A case whose file registers only that case (`mode: 'file'`) runs its trials one after another (`paid-report.ts:93-100`), so 10 trials take 10 trial walls. +5. The repo copy of `ubi-runner.sh` destroys every `ubirun-*` VM older than 12 hours on each `up`, whoever owns it. The user-scope ubicloud skill's copy already tags VMs with `UBI_OWNER` and sweeps only the caller's VMs. +6. AGENTS.md (Validation discipline, item 7: "Rerun a failed case only after a concrete repair or a demonstrated launch correction") and TESTING_INTERNALS ("A red census is never rerun on unchanged inputs") forbid a measurement on unchanged inputs unless diagnostic measurement is carved out in writing. +7. `windows-free-tests.yml` can only be dispatched for the whole curated lane; there is no way to rerun one shard's exact file list. + +### The loop, as /ship executes it + +Step 6.3 becomes "If any eval fails: run the measure-then-fix loop below." The prose in the skill stays short: "run `gstack-eval-measure status`, do what its Next column says, and follow these rules". The mechanics live in one CLI, `bin/gstack-eval-measure` (resolved through the `runtime-root.ts` prelude like `gstack-detach`, so it works from any repository; `bun run eval:measure` is the alias inside gstack; source `scripts/eval-measure.ts`). Its `--help` works offline, lists every subcommand with one example, and says which values are deliberately not overridable (N, the bar, extend-once, mechanical `provider`) and why. Every step writes to one ledger, `.context/eval-repair/.json` (`gstack-slug`'s sanitized branch, so `/` never nests), which is the single current failure list that AGENTS.md's Validation discipline item 1 asks for. The CLI refuses to write the ledger into a tracked or unignored path and prints the `.gitignore` line to add. + +**Words.** A *trial* is one verdict-shaped run of the case (one judge panel for a judge, one session for a behavior case). A *batch* is N trials launched together. A *round* is one repair plus one fresh batch. A *loop* starts when `inventory` reads a new gate run's reds and ends when every red case reaches a terminal state; rounds count per loop, while gate runs and spend count per ship (per branch) and survive `reset --reason`, which archives a loop without deleting it. A *gate run* is one execution of the project's pre-merge eval command. + +**Command contract.** `status`, `decide`, `collect` and `start` take `--json` and return `{loop_id, cases: [{case, state, blockers, next}], spend, reserve, caps}`. Exit codes: 0 done or MEETS, 10 running, 20 BELOW or EXTEND (a diagnostic red, not a broken tool), 30 needs input (an approval or a classification), 40 refused (with a reason code), 50 infrastructure failure, lost or void batch, 2 usage. Every refusal prints a stable reason code, the problem, its cause, the exact next command and a `docs/troubleshooting.md` anchor, following the `lib/gate-outcomes.ts` pattern; tests pin the codes. + +| Subcommand | What it records or does | +|---|---| +| `doctor` | backend credentials, Ubicloud quota (`usage`), CI image tag resolution, model keys, `gh` auth; prints the backend it will use and why. Runs implicitly first in `start`. | +| `inventory [--run \| --dir \| --cases a,b]` | with no argument, reads the pointer Step 6.2 writes after every gate run (`.context/eval-repair/.last-gate.json`: CI run id or local eval dir); `--run` for a CI run, `--dir` for a local run, `--cases` for an adapter project. Opens a loop. Each red becomes a target of one kind: *case* (measurable), *file* (a keyless file or an id several files register, measured by file), or *run* (a missing slice artifact, an unattributed module-load error, an INFRA-only census), which is never measured: it routes to the existing INFRA re-dispatch rule or to a free fix of the load error. | +| `classify --trial / --route --tag --evidence ` | one immutable record per failed trial (a later record supersedes, never edits); the case's route is the summary of its trials; refuses `provider` unless `failure_cause` and its evidence say so | +| `proof --case --test [--negatives ] [--contract ]` | runs the free check twice itself: on the pre-fix snapshot (must fail on the capture) and on the current snapshot (must pass and still reject every negative), recording both tree hashes; `--contract` records the contract mapping | +| `probe --case ` | the selection probe below | +| `start --case [--backend ubicloud\|local] [--size S] [--image ref \| --build-image] [--dry-run]` | doctor, pre-registration, launch; idempotent per batch id (a repeat returns the running batch); `--dry-run` prints the plan, estimate and VM size without creating anything | +| `collect [--case ]` | before completion: "running, k/N done, next check ~HH:MM PT" with exit 10; after: pulls, decides, destroys | +| `decide --case ` | applies the bar to the case's batches | +| `note --case --scope … --design … --budget …` | the stop rule's re-evaluation note | +| `gate` | the only way the loop pushes: reserves the gate's estimate, pushes, adopts the one run for the head SHA (see Gate), records it and inventories its reds | +| `approve --cap batch\|case\|ship\|gate --usd N --question ` | records the approval, the question it answered and the time; like the ledger hashes, a tripwire against accidental edits, not proof of who answered (the PR table is the audit trail) | +| `status`, `table`, `cancel`, `reset --reason`, `demo` | state with a Next column; the PR-body table; cancel and destroy; archive a loop; the credential-free demo | + +1. **Inventory.** `gstack-eval-measure inventory` writes one row per red verdict: case id, kind, run id, source SHA, `failure_class`, `failure_cause`, evidence and Expected/Received (from `eval:pass-rates --run ` for CI runs, the eval store for local runs). It reconciles the runner's red count with the named reds and any module-load errors, sorts each red into a case, file or run target, and prints the gate's predicted all-green probability (see the PR-body report). Without `gh` auth it says exactly that and how to fix it. +2. **Classify each red** into one route (table below) with a fine-grained tag and a diagnosis that cites evidence. `provider` is assigned mechanically, only when `failure_cause` is `api_error`, or `pre_turn_infra` with provider or transport evidence (an HTTP 429 or 5xx, a connection reset); the agent cannot choose it. A `pre_turn_infra` without that evidence (a crash or module-load failure) is a harness defect. A `refusal` is not mechanical: it usually means the fixture or prompt invites it, so it is classified with a citation like any other red. A red with `failure_cause: unknown` or no readable capture is `unclassified` and is measured before anything is fixed. +3. **Free proof first.** If the route is fix-test-infra, write or extend a free check from the captured events and record it with `proof`: it must fail on the captured red, pass after the fix, and still reject every known-bad example (see "Repairs must not weaken the test"). No paid trial runs for that case until this step is green. +4. **Selection probe (no model spend).** `gstack-eval-measure probe --case ` runs one trial through the exact CLI path the paid trial will use, with every provider endpoint pointed at a closed local port, an invalid key, and `GSTACK_EVAL_CLIENT_RETRIES=0`, a new override the SDK runner (default 3 retries), the PTY and Codex launchers and `llm-judge.ts`'s retry loop honor. Session-ledger rows are written when a session ends, too late to stop on, so each of the four runners (claude-p, Agent SDK, PTY, Codex) and the judge caller gains one line, written only when `GSTACK_PROBE_FILE` is set, recording `{test_name, runner}` the moment it is about to send its first model request. The probe passes when the started tests match the case's selection mode (name mode: exactly one test, mapping to `` through `caseIdForTestName()`; file mode: at least one test, all from the case's file, none from another file), every started test reached a probe line, and no other test ran; the probe then stops the process group. It fails at a wall of 120 seconds or the case's recorded setup time plus 60 seconds, whichever is larger (injectable for tests). Each failure mode has its own reason code; a missing local tool (Bun, the Claude CLI) is a setup failure with a remedy, and only a selection failure is a harness defect to fix before spending. The same probe runs first on the VM before its trials, so a remote environment problem stops the batch before any model call (the VM's own minutes are billed and shown). An adapter without a probe records "probe unavailable", and its first batch's selection is the check: a batch with 0 or mismatched trial results is void and is not a round. +5. **Pre-register and estimate.** Before any result exists, the ledger records per case and batch: N, the bar that will decide it, the input identity, the estimate and the caps (see Cost guard), with a schema version and a hash of each pre-registered block. The CLI prints the estimate and continues without asking when it is under the caps. A pre-registered field edited outside the CLI is refused ("ledger edited outside gstack-eval-measure: field X"), and a lock (pid, host, start time) stops two sessions writing one ledger; a lock whose pid is dead or older than its TTL is reclaimed and the reclaim shows in `status`. +6. **Measure.** Baseline batch on the current head: N trials in parallel on detached Ubicloud VMs (see Infrastructure). `start` first freezes the source: a temporary index (`GIT_INDEX_FILE`) takes `git add -A` of the working tree, `git write-tree` gives one tree hash covering tracked, untracked-unignored and deleted files, and that tree, not the live directory, is what gets measured and identified. Each VM receives `git archive` of the tree into a fresh repository with that tree committed, so linked worktrees work and half-finished edits to other cases never ride along; local trials check out the same tree into a temporary worktree. The baseline is skipped only when step 3 reproduced the red deterministically from captured events; the ledger then records "before: reproduced by replay, 0 paid trials". `start` prints the deadline in PT and a suggested wake time; the skill prose names the host's wait (a Capy `wait` timer, or polling the `gstack-detach` sentinel on Claude Code and Codex). +7. **Decide** with the measurement bar and its precedence rules: MEETS, EXTEND (N more on identical inputs, pooled, once), fix round, or BELOW. A batch below strict whose failed trials nobody has classified returns *needs-classify* (exit 30) with Next "classify t2, t6, t9, then decide"; EXTEND is possible only after every failure has a route or is marked unclassified after review. The VM-side runner applies the early-stop rule itself (the pure `measure-bar.ts` is synced with the tree) and cancels remaining trials once the outcome is certain; an early-stopped batch has its own state, never confused with a void one. +8. **Fix round.** Fix at the cause, rerun its free proof, rerun the probe if the trial path changed, then measure a fresh batch. A fix changes the case's inputs, so the new batch starts a new pool; trials never pool across a fix. Never raise a budget, lower a threshold, change a kind or add a verdict retry to reach the bar. +9. **Stop rule.** After 3 fix rounds on one case without MEETS, stop patching and re-evaluate (see Stop rule). +10. **Gate.** When every case that went red MEETS, `gstack-eval-measure gate` runs exactly one gate for the new head SHA and is the only way the loop pushes. With an open PR on gstack, the push itself starts `evals.yml` (`pull_request`, with its verified-pass reuse), and `gate` adopts that run; it never also calls `eval:bg:pr`, whose duplicate check matches only dispatches, so that pairing would buy two gates. With no PR yet, `gate` dispatches through `eval:bg:pr` or runs the project's command locally under `gstack-detach`, as Step 6.2 does. It reserves the gate's estimate before pushing, and it reads every `evals.yml` run for the branch's head SHAs from `gh`, so a gate started any other way is counted too. The red gate that opened the loop is gate 1. That run's verdicts are the lane verdict for Step 6. Later `/ship` pushes (review fixes, VERSION, CHANGELOG) follow Step 16's existing evidence rules and also go through `gate` when the loop is open. A new red starts the loop for that case. A red on a case the loop already brought to MEETS is re-measured on the gate's inputs before any fix; it counts as a round only when a fix follows. At most 3 gate runs per ship, counting gate 1 and surviving `reset`; a fourth needs Garry (see Terminal states). +11. **Report.** `gstack-eval-measure table` writes the PR-body table, the three success numbers and Step 19's evidence record. + +**Interrupted work.** Every operation is keyed by loop and batch id and is safe to repeat. A coordinator that slept, lost its connection or restarted runs `status`, which shows each running batch with its VM, deadline, reserved spend and the exact recovery command. A batch whose sentinel never arrives by its deadline, whose VM vanished, or whose results could not be pulled is *lost*: it is not a round, its estimate stays counted as spend, and it may be re-dispatched once. On the VM, each batch packs its results into one archive with a manifest (expected trial count, every file's hash); `collect` pulls it, checks the manifest, retries once after 60 seconds, and only then destroys the VM, recording "captures lost" if both pulls failed. `ubi-runner.sh down` today runs `destroy && log` and then deletes its local state unconditionally, so a failed destroy looks like success and loses the SSH key; the port keeps state until Ubicloud confirms the VM is gone and treats an already-absent VM as done, and `pull` reports "nothing matched" as a failure to the CLI. Runner state lives under `$GSTACK_STATE_ROOT/ubicloud/`, and every `status` and `start` destroys this owner's VMs that are past their recorded deadline plus 30 minutes and deletes stale `measure/*` scratch refs. `status` shows lost, void and early-stopped batches distinctly. + +**Credential-free demo.** `gstack-eval-measure demo` runs the paid E2E's seeded fixture locally with no keys and no VM: inventory, probe, a 10-trial baseline at 7/10, the prepared fix, 10/10, and the table, in under two minutes with the expected output printed. It is the five-minute introduction for a new maintainer and a free test of the whole command sequence. + +Free-test reds (Step 5) use the same loop; see Free-suite flakes. + +``` +red verdict(s) ──> inventory ──> classify ──┬─ fix-test-infra ─> free proof ─┐ + ├─ unclassified ─────────────────┤ + ├─ fix-product / runtime / cut ──┤ + └─ qualifies (provider, ...) ────┤ + v + probe (no model spend; local, then VM) ─> estimate ─> measure batch (N, diagnostic) + │ + ┌──────────── EXTEND (once, pooled) <─┤ + v │ + decide ──BELOW──> fix round (≤3, then re-evaluate, ≤1 reset) ─┐ + │ │ + MEETS named red <── second exhaustion <──┘ + │ │ + all reds MEET? ── yes ─> gate (≤3 per ship) ─> GREEN ─> report + │ + RED ──> new case: loop again + named red ──> STOP: needs-decision (waive | quarantine proposal | revert) +``` + +### Routes and classification + +Six routes decide what happens; the fine tag is evidence for the ledger and the PR table. + +| Route | Fine tags | How it is recognised | What the loop does | Free proof before paid trials | Can count toward the 8/10 bar | +|---|---|---|---|---|---| +| fix-product | `product`, `contract` | the case passes on base and fails on head, or evidence points at changed product code (CLAUDE.md blame protocol); `failure_cause: contract` | fix the product | the free test that pins the behavior or contract | no: a regression | +| fix-test-infra | `detector`, `test-assertion`, `grader`, `fixture` | correct behavior, wrong verdict: a progress detector matched only exact wording (plan-ceo-split-overflow); a regex lacked "authenticity" (cso-diff-mode); a grader prompt mis-scored good output (office-hours preservation judge); a wrong fixture or scripted actor | fix the evaluator at the cause | replay over captured events plus the corpus, or the judge's control bundles (judge cost only), with known-bad controls | no while the fix is pending | +| cut-work | `budget`, `hang` | `session_timeout`, headroom above `HEADROOM_FAIL`, `provider_stall`, or an `observer_timeout` whose transcript shows no progress (office-hours marathon) | cut the work: smaller fixture, fewer steps, split the case; never a larger budget. A hang is first diagnosed as budget, detector or provider | `eval:pass-rates --headroom` before and after; the session ledger row | no: a timeout or hang | +| runtime | `runtime` | the failure follows the runtime, not the code (Bun 1.4.0 on Windows) | pin, work around or upgrade the runtime in the same PR; reproduce on the pinned runtime first | a free test on the affected runtime (Windows lane dispatch) | no | +| qualifies | `provider`, `judge-noise`, `model-miss` | `provider`: mechanically, `failure_cause` is `api_error`, or `pre_turn_infra` with provider or transport evidence. `judge-noise`: a judge sample at the threshold under the v3 median gate. `model-miss`: the model did the wrong thing within the case's tolerated deviation, citing the transcript line where it deviated and the free checks that ruled out detector, assertion and grader | nothing to fix; measure | none | yes | +| unclassified | `unclassified` | no evidence for any route above | measure first to get more captures, then reclassify | none | no | + +A red whose route has a pending fix is fixed, not accepted. `gstack-eval-measure decide` refuses MEETS (qualified) while any failed trial lacks a route, or a `model-miss` lacks its transcript citation, and the PR body lists every qualified MEETS under its own heading. + +### Repairs must not weaken the test + +A fix-test-infra or cut-work repair changes what the case checks, so it carries a short contract mapping in the ledger: the contract the case named before, the contract after, and why they are the same. Its free proof keeps every known-bad example the old check rejected (the case's corpus negatives plus at least one synthetic bad output written for this repair) and shows the new check still rejects them. A repair that cannot keep its negatives, or that narrows the named contract, is a scope change and goes through the stop rule's re-evaluation, not a fix round. + +### The measurement bar and pooling + +The bar is the one Garry accepted after #3033. It filters ship-time reds; it is not a claim that the case is healthy, and pass-rate history stays the drift detector. Each row also prints its next-gate red risk at the measured rate (for a behavior case, 1 − P(at least 2 of 3)). + +Unit of measurement: + +- `rule` case: one trial is one run of the case. N = 10. +- `judge` case (E2E judge-kind or standalone `LLM_JUDGE_TOUCHFILES` judge): one trial is one 3-sample `judgePanel()` gated on its median, exactly as the verdict runs. N = 10. +- `behavior` case: trials launch in whole panels of 3 (`#~t`), N = 12 (4 panels), and the bar applies to the per-trial count with the same proportions. + +Decision for a batch of N trials (counts round up): + +- **MEETS (strict):** passes ≥ 0.9·N, so ≥ 9/10 (≥ 11/12). +- **MEETS (qualified):** passes ≥ 0.8·N, so ≥ 8/10 (≥ 10/12), every failed trial is classified, and every route is qualifies (none is a timeout, a hang, a regression or a route with a pending fix). +- **EXTEND:** not MEETS but passes ≥ 0.7·N (7/10, or 8/10 that does not qualify). Run N more trials on identical inputs and decide once on the pooled 2N with the same proportions: ≥ 18/20 strict, or ≥ 16/20 qualified. There is no third batch. +- **BELOW:** anything else (≤ 6/10), or a pooled 2N that does not meet. Go to a fix round. + +Precedence, applied in this order before the counts above: + +1. A contract violation, or a product regression from this branch, in any trial: fix-product. Never MEETS, whatever the count. +2. Below strict (< 0.9·N) with any red on a route that has a pending fix (fix-test-infra, cut-work, runtime): a fix round now. No EXTEND, because extending would only re-measure a known defect. +3. Otherwise the counts decide: strict, qualified, EXTEND or BELOW. Strict needs no classification, as Garry's bar says. A strict MEETS that left a red with a known fix still gets that fix in this PR (discovered defects are repaired, not deferred), and because the fix changes the case's own inputs, the case is measured once more on the fixed tree; it is MEETS only on that tree. +4. Evidence that arrives during an EXTEND batch (a capture that reveals a fixable defect) cancels the extension and starts a fix round; the extension's trials are kept in the ledger as diagnostic evidence. + +**When MEETS goes stale.** MEETS is bound to the frozen tree it was measured on. At every `gate`, the CLI recomputes each MEETS case's series identity from the gate's tree. If the case's own bytes (its test file, fixtures, prompt files) changed, MEETS is stale and the case is measured again before the gate. If only shared harness inputs changed (another case's fix in a shared helper), MEETS stands with a "shared inputs changed" mark in `status` and the PR table, and the gate is the check; this avoids a re-measurement cascade every time a shared detector is fixed. + +Identical inputs means the same case series identity (case-owned bytes plus `HARNESS_VERSION`, from `scripts/eval-trial-series.ts`), the same full consumed-input fingerprint, model, Claude CLI version, policy version, CI image digest, vCPU per trial, the per-key concurrency limit and runner setup. The series identity is computed from the frozen tree's temporary index, because `scripts/eval-trial-series.ts` reads blob ids with `git ls-files -s` and would otherwise hash the index rather than the working-tree bytes being measured. `gstack-eval-measure` refuses to pool batches whose identities differ and names the field that differs. + +Why these numbers (binomial, per-trial rate p): + +| true p | ≤ 8/10 (strict misses) | MEETS strict, one batch | MEETS strict with one extension | MEETS if every red qualified, with extension | +|---|---|---|---|---| +| 0.97 | 3.5% | 96.5% | 98.9% | 99.97% | +| 0.95 | 8.6% | 91.4% | 95.9% | 99.8% | +| 0.90 | 26.4% | 73.6% | 80.4% | 97.2% | +| 0.80 | 62.4% | 37.6% | 40.8% | 75.3% | +| 0.70 | 85.1% | 14.9% | 15.6% | 42.3% | + +The last column is why only the qualifies route counts: a 70% case whose reds all "qualified" would pass the qualified bar 42% of the time, so a red with a known defect behind it must be fixed instead, and `provider` cannot be chosen by the agent. + +### Stop rule and terminal states + +A fix round is one concrete repair plus one fresh measurement. `gstack-eval-measure` counts rounds per case in the ledger and refuses a fourth measurement until `note` has recorded a re-evaluation answering three questions with evidence: + +- **Scope:** is the case testing the contract it names, or something incidental (wording, ordering, a detector's view of progress)? +- **Design:** would an outcome check, a structured signal, a smaller fixture or a split case make the result deterministic? +- **Budget fit:** does one trial fit Garry's ~10-minute eval target and its armed budget (`eval:pass-rates --headroom`)? A case whose single trial cannot fit gets less work, not more time. + +The agent then picks the new approach itself and continues; the counter resets once. Every case ends in one of two terminal states, and the loop has two approval gates on the way: + +| | When | What /ship does | +|---|---|---| +| terminal: MEETS | the bar is met on the case's frozen tree, and MEETS is not stale (see When MEETS goes stale) | eligible for the gate | +| terminal: named red | the second approach is exhausted, or a fix needs a policy, threshold, kind or budget change | stops before the gate with a needs-decision report for Garry: waive under the existing explicit exception path, file a quarantine proposal (diagnostic trials can be cited in its reason but never count toward its trial minimum), or revert the change that caused it. The gate never runs on a named red's account. | +| approval gate: spend cap | the next batch would pass a measurement cap | asks once with the estimate and the ledger; approval is recorded with `approve`; declining makes the case a named red | +| approval gate: gate cap | a fourth gate run on this ship would be needed (gate 1 is the red gate that opened the loop) | asks once with the reds so far and the predicted all-green probability | + +Only policy, threshold, budget and cap questions go to Garry. Routine spend under the caps never asks. Every question uses one fixed template: the problem in plain English, the options with what each costs and what happens next, a recommended option, and one question answerable with a letter. + +### Cost guard + +- **Estimate first.** Per trial: the mean `cost_usd` of the case's current series from pass-rate history; else the case's recorded wall from `scripts/paid-test-durations.json` times the tier's mean cost per second. Trials whose harness bills nothing (`cost_known: false`: PTY and Codex) use the wall-based estimate and are labelled estimated. Plus VM time at the Ubicloud list rate for the size used. The estimate prints with its source. +- **Caps (defaults, overridable per project).** Measurement and gates are capped separately, so the two never contradict each other. Measurement: ask only when one batch is estimated above $30, a case's cumulative measurement spend would pass $60, or the ship's total measurement spend (model plus VM time, free-suite batches included) would pass $150. Gates: capped by count (3 per ship, above), not dollars; their cost is reserved before each push and reported. The worst case without a question is therefore three gates (about $270 for a full gstack gate) plus $150 of measurement. Caps count per ship (per branch, across `reset`) against known spend plus the estimate reserved for every trial whose cost is unknown, so unknown billing cannot slip under a cap. A project overrides them with a `caps:` line in its Eval measurement block or `GSTACK_MEASURE_CAPS`; `status` prints the effective values and their source. +- **Actuals.** Each batch records its known cost and the count of trials whose cost is unknown, using the existing `formatCost()` wording. The table shows the sum and the reserve. +- **Rate limits.** The ledger caps concurrent model sessions per API key across all running batches (default 10, recorded in the identity), so several red cases measured at once do not throttle each other or other threads' gates. +- **Void batches.** If more than 30% of a batch's trials fail with affirmative provider or transport evidence (an `api_error`, a 429 or 5xx, a connection reset, at any turn), the batch is void (an outage or throttle, not a measurement) and may be re-dispatched once, mirroring the census INFRA rule. A `pre_turn_infra` failure without that evidence never voids a batch: crashes and module-load failures are harness defects, and their evidence is kept. Both batches are reported. + +### Eval-store labelling + +Diagnostic trials must never become verdicts or enter EVAL_POLICY v3 pooling or series history. Three independent guards, so any one failing still leaves two: + +1. **Separate place.** Every diagnostic trial writes under `$GSTACK_STATE_ROOT/projects//eval-diagnostics///`, never `getProjectEvalDir()`. `runCaseDiagnosis()` takes the eval dir from `gstack-eval-measure`, and a bare `--case` defaults there too. +2. **Separate schema and filename.** The per-trial ledger is `diagnostic-trials.jsonl` with schema `gstack-diagnostic-trial/v1`. It never matches the `trial-outcomes*.jsonl` glob that `eval:pass-rates` imports, and `parseTrialOutcomes()` rejects its schema, so older checkouts reject it too. +3. **Explicit label.** Shard children run with `GSTACK_EVAL_PURPOSE=diagnostic`; every `EvalResult` and trial record they write carries `purpose: "diagnostic"` and `diagnostic: { loop_id, case, batch, round }`, and diagnostic `EvalResult` files use their own filename prefix and schema version, so `isFinalizedEvalResultFile()` and the legacy `backfillEvalFiles()` import in `eval:pass-rates` (which read eval JSON independently of the JSONL parser) reject them structurally. `runPaidReport()` and the `trial-outcomes` writer refuse a record with `purpose: "diagnostic"`; `eval:pass-rates --dir` skips them with a printed count; `eval:list`, `eval:compare` and `eval:summary` mark them `[diagnostic]` and exclude them from trends. + +Two reuse paths could also carry diagnostic results into verdict evidence, and both are closed at their owners: `prepareWorkflowJudgeCache()` (`test/helpers/workflow-judge-cache.ts`, whose cache purpose is hard-coded to `gate`) neither looks up nor publishes receipts when `GSTACK_EVAL_PURPOSE=diagnostic`, and diagnostic runs unset `EVALS_CACHE_DIR` and `EVALS_CACHE_PR` so `e2e-shard-reuse.ts` never writes receipts for them. + +`docs/TESTING_INTERNALS.md` gains a short "Diagnostic measurement (not verdict retries)" subsection under the eval verdict policy, and AGENTS.md item 7 and the census rule gain one carve-out sentence each: a diagnostic measurement under this loop may run on unchanged inputs; it never yields, replaces or pools into a verdict. `EVAL_POLICY` constants and version are unchanged. + +### Infrastructure: detached trials on Ubicloud, run the way CI runs them + +The coordinator machine may sleep (a Capy turn ending, a laptop lid), so the trial process must not live there. + +Why Ubicloud and not a CI dispatch: a CI run needs the measured code pushed, and every push to the PR branch starts the PR's own paid lane (`evals.yml` runs on `pull_request`, `cancel-in-progress`), so each fix round would buy an unwanted gate run and cancel the one in flight. Ubicloud measures the frozen working tree, including uncommitted fixes, and runs nothing else. A paid measurement workflow dispatched on a scratch ref would also avoid the PR lane; it is the recorded alternative (TODOS.md) if the VM path proves costly to keep faithful. + +- **One shared CI trial script.** CI's paid step is more than the image: it runs as the `runner` user (the image creates it but does not select it, and Claude's permission-bypass mode refuses root), restores dependencies from `/opt/node_modules_cache`, builds the checkout, seeds interactive configuration, registers runtime resources and authenticates Codex into its own home (`evals.yml`, around lines 360-450). That sequence moves into one script, `scripts/ci/paid-trial-env.sh`, which both `evals.yml` and the measurement backend call, so they cannot drift. SDK, PTY and standalone-judge trials are each qualified on it once (the real-path smoke). +- **Same CPU per trial as CI.** CI runs paid slices on `ubicloud-standard-8` with `EVALS_JOBS=2`, about 4 vCPU per trial. Measurement keeps that: `standard-16` VMs running 4 trials each, so a 10-trial batch uses 3 VMs and a 12-trial behavior batch 3 VMs. vCPU per trial is in the pooling identity, because a crowded VM inflates walls into fake timeouts, which is exactly what the cut-work route reads. +- **Launch.** `start` runs `doctor` and the local probe, then per VM: `ubi-runner.sh up` (owner-tagged), the frozen tree as a fresh repository, and a new `ubi-runner.sh env NAME --pass K...` subcommand. `env` writes a separate Docker-format file (`~/.ubirun-docker.env`, literal `KEY=value` lines, mode 0600, values with newlines rejected), because the existing `~/.ubirun-env` holds shell `export K=%q` lines that `docker run --env-file` cannot read. One `ssh` then starts, under `gstack-detach --label measure--b --timeout `, the remote probe and then the trials through `scripts/ci/paid-trial-env.sh` inside the CI image with `docker run --user runner --env-file ~/.ubirun-docker.env` (never `-e K=V`, which would put values in the VM's process list) and `GSTACK_EVAL_PURPOSE=diagnostic`. It records the VM names, deadline and log paths in the ledger and returns at once. +- **The image.** The tag is the one `evals.yml` computes with `hashFiles(Dockerfile.ci, bun.lock, patches/**)`; `measure-remote.ts` reimplements that hash and a test checks it against a published tag. When the tag is not published (a fork, or a changed Dockerfile not yet built), the VM builds the image, which takes longer than the batch target; `start` says so in its estimate. The image digest is in the pooling identity. +- **Dead-man switch.** The VM schedules `sudo shutdown -h +`. A halted VM may still bill, so the switch only stops runaway trials; destruction is the coordinator's `status`/`start` sweep of VMs past deadline plus 30 minutes. +- **Collect.** `collect` (run when a timer wakes the coordinator) polls the remote logs for `### gstack-detach EXIT= ###`, pulls and checks each archive, writes `diagnostic-trials.jsonl`, prints the decision and destroys the VMs, on every path (success, failure, missing sentinel after the deadline, `cancel`). The ledger's `vm` field is cleared only after Ubicloud confirms the VM is gone. +- **The runner.** `scripts/ubicloud/ubi-runner.sh` is replaced wholesale by the user-scope skill copy (owner tagging with `UBI_OWNER`, owner-scoped `gc`, `usage`, the quota-refused message) with its tests, plus the `env` subcommand and the `down` and `pull` fixes above, so the copies stop drifting. +- **Parallel cases.** Red cases are measured in parallel, each with its own VMs, after checking `usage` against the shared quota and within the per-key session cap. +- **Local backend.** Projects without Ubicloud use `gstack-detach` on the coordinator with parallel jobs defaulting to half the machine's cores; the job count is part of the pooling identity, and the ledger records that a sleeping coordinator can lose the batch. +- **Isolation for parallel file-mode trials.** Each trial already gets its own `GSTACK_EVAL_DIR` and `TMPDIR`; it also gets its own `HOME` and `GSTACK_STATE_ROOT`, so a file-mode case running beside itself cannot share ports, fixture dirs or state. + +The ~10-minute target applies to a measurement batch. A typical repair path is sequential: about 2 minutes for inventory and probe, a 10- to 15-minute baseline batch, the fix, a 10- to 15-minute re-measure, then the gate (~12 minutes when everything reruns, less with reuse), so about 40 to 50 minutes from first red to verdict, against six full gates (over an hour of gate wall plus repair time between runs, ~$540) on #3033. + +### Free-suite flakes and Windows lanes + +A free-test red goes through the same steps with no model spend (its VM time still counts toward the ship's measurement cap): classify, free proof, then measure by rerunning the failing shard's exact file list, in the same order, from the run's plan artifact. Before calling a free red a product bug, the loop checks the known local traps (Bun version against the CI pin, the Capy git author wrapper) and records them as `runtime` when they apply. + +- **A diagnostic plan, not a replayed CI plan.** Today's plan validator requires the original revision and every free file exactly once, and the Windows lane enables retries whose verifier accepts a failed first attempt followed by a pass. Measurement therefore uses a new diagnostic plan type (`--diagnostic-plan`): source-plan provenance (run id, shard), the frozen tree's hash, the ordered file list and repetition ids 1..N, with retries disabled and only first-attempt outcomes counted. The executor and verifier learn this plan type together. +- Linux: on one Ubicloud VM (`test:ubicloud` setup), the diagnostic plan's N = 10 repetitions, four isolated copies at a time (each with its own temp state, `HOME` and state root). +- Windows: `windows-free-tests.yml` gains two `workflow_dispatch` inputs, `files` (the exact newline-separated list) and `repeat` (1 to 10). They reach the plan step only through `env:`, never `${{ inputs.files }}` inside `run:` (script injection), and the plan step builds the diagnostic plan from them with retries disabled. The measured code is pushed to a scratch ref, `measure/`, and the dispatch targets it: every workflow in `.github/workflows/` triggers only on pull requests, pushes to main, schedules or dispatch, so the scratch push starts nothing else, and the ref is deleted after collection. The dispatch's concurrency group is already per run (`windows-free-${{ github.run_id }}`), so measurement dispatches never cancel the PR's own Windows lane or each other. +- The free bar is 10/10 after the fix. A free test is meant to be deterministic, so any failure in the measurement is a defect to fix; the before-rate is reported for the table. `GSTACK_FREE_RETRY_FLAKY` stays a CI safety net, not evidence. + +### Other projects: an explicit adapter, never a guess + +`/ship` runs the automatic loop for another project only when its CLAUDE.md or AGENTS.md declares an "Eval measurement" block (version 1): + +``` +eval-measurement: v1 +trial: +results: exit-code | junit +kind: rule | behavior | judge (optional; default rule) +probe: (optional) +cost: (optional; without it the first batch asks) +caps: batch=$30 case=$60 ship=$150 (optional; measurement caps) +setup: