gstack/ship/sections/measure.md

Measure-then-fix loop (a red paid eval or a failed free-suite shard)

Every rerun here is diagnostic, not a verdict: it never changes the recorded verdict of the run that failed, its artifacts are labeled diagnostic under .context/ship-measure/ (never the eval history), and the lane verdict comes only from the one full gate run at the end. Never raise a budget, lower a threshold, add a verdict retry or skip a case to get a pass. ship-measure below means bun run ~/.claude/skills/gstack/scripts/ship-measure.ts.

Failed free-suite shard (Step 5). Before calling the failure pre-existing or pushing, rerun that shard's exact file list (in run order) with ship-measure free --files <f1,f2,...> (gstack local runs: --shard <i>): 10 reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with its own HOME and state root, the flaky retry off, under a 10-minute cap. ship_rerun_backend=ubicloud with UBICLOUD_API_KEY set runs them on one Ubicloud VM. If the runner cannot take a file list, rerun the whole failing command 10 times. Report the pass rate, the completed count (for example 6 of 10 completed at the cap) and one failure capture, then continue the triage.

Red paid eval (Step 6). For each red case:

1. Classify. Read the red line's cause, evidence and Expected/Received (failure_cause, failure_detail; gstack: bun run eval:pass-rates --run <run> --case <id>). Name it: product change from this branch, detector or actor defect, fixture defect, provider event, or model miss. A detector fix gets a free replay test before any paid trial. A red that is infrastructure (provider outage, runner loss) may skip measurement: ship-measure skip --case <id> --reason "<why>" labels it unmeasured in the PR body. It never counts as a pass.
2. Show the plan. Print ship-measure table (trials, bar and budget per kind, from the ship_measure_* gstack config keys) and the estimate: per-trial cost from pass-rate history times trials, or "estimate unknown".
3. Measure. ship-measure measure --case <id> --round baseline [--cost-per-trial <usd>] runs the case alone on the current head at its kind's count, in parallel, through the project's documented single-case eval command, --command '<command> {case}' (found the way Step 4 finds test commands; gstack needs none and accepts judge ids). With no documented command, ask once and record the answer in the report. Exit 2 needs approval: ask once with its reason, then rerun with --approved.
4. Decide by the exit code (MEETS 9/10, MEETS-qualified 8/10 with every red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12):
   • 0: MEETS or MEETS-qualified; done.
   • 5: needs-classify. Record each listed red from its capture: ship-measure classify --case <id> --round <r> --trials <2,5> --class <class> --evidence "<path:line>". Qualifying: provider (only on the trial's own provider error), judge-noise, model-miss (cite the line). Fix round: timeout, hang, regression, fixable. unclassified: reviewed, no evidence.
   • 6: EXTEND. ship-measure extend --case <id> --round <r> once, on the unchanged tree; never a third batch.
   • 1: BELOW, a fix round.
5. Fix at the cause. Root-cause the failed trials from their captures in .context/ship-measure/<case>/<round>/tNN/ and change the cause.
6. Re-measure. ship-measure measure --case <id> --round round-<n> --fix "<cause and change>", then item 4. Call a case fixed only with a named causal change and every failure in the closing measurement explained; report "observed k/n after fix at ", never a proven rate.
7. Stop. Exit 3 is a named red (round limit, budget, or two void batches): stop, do not push, and report ship-measure report with every trial.

Gate once. When every red case measures at or above target, run the full gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack: the pull request's evals workflow), the push is the gate run: push and read that run, and do not also start the gate command (gstack's bun run eval:bg:pr on a pushed head dispatches a second paid run). Put the ship-measure report table (estimated and actual spend per case) in the PR body.