## Measure-then-fix loop (a red paid eval or a failed free-suite shard) Every rerun here is **diagnostic, not a verdict**: it never changes the recorded verdict of the run that failed, its artifacts are labeled `diagnostic` under `.context/ship-measure/` (never the eval history), and the lane verdict comes only from the one full gate run at the end. Never raise a budget, lower a threshold, add a verdict retry or skip a case to get a pass. `ship-measure` below means `bun --no-env-file --config=/dev/null run ~/.claude/skills/gstack/scripts/ship-measure.ts`. **Failed free-suite shard (Step 5).** Before calling the failure pre-existing or pushing, rerun that shard's exact file list (in run order) with `ship-measure free --files ` (gstack local runs: `--shard `): 10 reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with its own HOME and state root, the flaky retry off, under a 10-minute cap. `ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one Ubicloud VM. If the runner cannot take a file list, rerun the whole failing command 10 times. Report the pass rate, the completed count (for example 6 of 10 completed at the cap) and one failure capture, then continue the triage. **Red paid eval (Step 6).** For each red case: 1. **Classify.** Read the red line's cause, evidence and Expected/Received (`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run --case `). Name it: product change from this branch, detector or actor defect, fixture defect, provider event, or model miss. A detector fix gets a free replay test before any paid trial. A red that is infrastructure (provider outage, runner loss) may skip measurement: `ship-measure skip --case --reason ""` labels it `unmeasured` in the PR body. It never counts as a pass. 2. **Show the plan.** Print `ship-measure table` (trials, bar and budget per kind, from the `ship_measure_*` gstack config keys) and the estimate: per-trial cost from pass-rate history times trials, or "estimate unknown". 3. **Measure.** `ship-measure measure --case --round baseline [--cost-per-trial ]` runs the case alone on the current head at its kind's count, in parallel, through the project's documented single-case eval command, `--command ' {case}'` (found the way Step 4 finds test commands; gstack needs none and accepts judge ids). With no documented command, ask once and record the answer in the report. Exit 2 needs approval: ask once with its reason, then rerun with `--approved`. 4. **Decide by the exit code** (MEETS 9/10, MEETS-qualified 8/10 with every red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12): - 0: MEETS or MEETS-qualified; done. - 5: needs-classify. Record each listed red from its capture: `ship-measure classify --case --round --trials <2,5> --class --evidence ""`. Qualifying: `provider` (only on the trial's own provider error), `judge-noise`, `model-miss` (cite the line). Fix round: `timeout`, `hang`, `regression`, `fixable`. `unclassified`: reviewed, no evidence. - 6: EXTEND. `ship-measure extend --case --round ` once, on the unchanged tree; never a third batch. - 1: BELOW, a fix round. 5. **Fix at the cause.** Root-cause the failed trials from their captures in `.context/ship-measure///tNN/` and change the cause. 6. **Re-measure.** `ship-measure measure --case --round round- --fix ""`, then item 4. Call a case fixed only with a named causal change and every failure in the closing measurement explained; report "observed k/n after fix at ", never a proven rate. 7. **Stop.** Exit 3 is a named red (round limit, budget, or two void batches): stop, do not push, and report `ship-measure report` with every trial. **Gate once.** When every red case measures at or above target, run the full gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack: the pull request's `evals` workflow), the push is the gate run: push and read that run, and do not also start the gate command (gstack's `bun run eval:bg:pr` on a pushed head dispatches a second paid run). Put the `ship-measure report` table (estimated and actual spend per case) in the PR body.