diff --git a/ship/SKILL.md b/ship/SKILL.md index 0cca3fe07..f47b6b638 100644 --- a/ship/SKILL.md +++ b/ship/SKILL.md @@ -486,6 +486,7 @@ sections. Read a section in full before doing its step; do not work from memory. |------|-------------------| | App Store/TestFlight distribution is requested for an Apple app (.xcodeproj, .xcworkspace, or an app-product Swift package) — read at Step 0.9 before the branch gate; an Apple repository-landing request follows the normal pipeline | `sections/apple-release.md` | | running the test suites and (if prompt files changed) the eval suites (Steps 4-6) | `sections/tests.md` | +| a paid eval case is red or a free-suite shard failed (Steps 5-6) — read from the tests section; diagnostic reruns, then the gate once | `sections/measure.md` | | auditing test coverage of the diff (Step 7) | `sections/test-coverage.md` | | auditing plan completion, verification, and scope drift (Step 8) | `sections/plan-completion.md` | | the pre-landing review and specialist dispatch (Step 9) | `sections/review-army.md` | diff --git a/ship/sections/manifest.json b/ship/sections/manifest.json index a3a564361..ac2bba19f 100644 --- a/ship/sections/manifest.json +++ b/ship/sections/manifest.json @@ -16,6 +16,12 @@ "title": "Test bootstrap, run, triage + eval suites", "trigger": "running the test suites and (if prompt files changed) the eval suites (Steps 4-6)" }, + { + "id": "measure", + "file": "measure.md", + "title": "Measure-then-fix loop for a red eval or free shard", + "trigger": "a paid eval case is red or a free-suite shard failed (Steps 5-6) \u2014 read from the tests section; diagnostic reruns, then the gate once" + }, { "id": "test-coverage", "file": "test-coverage.md", diff --git a/ship/sections/measure.md b/ship/sections/measure.md new file mode 100644 index 000000000..a340352f0 --- /dev/null +++ b/ship/sections/measure.md @@ -0,0 +1,54 @@ + + +## Measure-then-fix loop (a red paid eval or a failed free-suite shard) + +Every rerun here is **diagnostic, not a verdict**: it never changes the recorded +verdict of the run that failed, its artifacts are labeled `diagnostic` under +`.context/ship-measure/` (never the eval history), and the lane verdict comes +only from the one full gate run at the end. Never raise a budget, lower a +threshold, add a verdict retry or skip a case to get a pass. +`ship-measure` below means `bun run ~/.claude/skills/gstack/scripts/ship-measure.ts`. + +**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or +pushing, rerun that shard's exact file list (in run order) with +`ship-measure free --files ` (gstack local runs: `--shard `): 10 +reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with +its own HOME and state root, the flaky retry off, under a 10-minute cap. +`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one +Ubicloud VM. If the runner cannot take a file list, rerun the whole failing +command 10 times. Report the pass rate, the completed count (for example 6 of 10 +completed at the cap) and one failure capture, then continue the triage. + +**Red paid eval (Step 6).** For each red case: + +1. **Classify.** Read the red line's cause, evidence and Expected/Received + (`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run --case `). + Name it: product change from this branch, detector or actor defect, fixture + defect, provider event, or model miss. A detector fix gets a free replay test + before any paid trial. A red that is infrastructure (provider outage, runner + loss) may skip measurement: `ship-measure skip --case --reason ""` + labels it `unmeasured` in the PR body. It never counts as a pass. +2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and + budget per kind, from the `ship_measure_*` gstack config keys) and the + estimate: per-trial cost from pass-rate history times trials, or "estimate unknown". +3. **Measure.** `ship-measure measure --case --round baseline [--cost-per-trial ]` + runs the case alone on the current head at its kind's count, in parallel. It + uses the project's documented single-case eval command, + `--command ' {case}'`, found the way Step 4 finds test commands + (CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no + `--command`: `scripts/test-paid-shards.ts --tier --case `, and it + accepts standalone judge ids. With no documented single-case command, ask + once and record the answer in the report. Exit 2 means it needs approval: + ask once with the reason it printed, then rerun with `--approved`. +4. **Fix at the cause.** Root-cause the failed trials from their captures in + `.context/ship-measure///tNN/` and change the cause. +5. **Re-measure.** `ship-measure measure --case --round round- --fix ""`. + Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call + a case fixed only with a named causal change and every failure in the closing + measurement explained; report "observed k/n after fix at ", never a proven rate. +6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget + exhausted): stop, do not push, and report `ship-measure report` with every trial. + +**Gate once.** When every red case measures at or above target, run the full +gate command once (Step 6 item 2). Put the `ship-measure report` table (estimated +and actual spend per case) in the PR body. diff --git a/ship/sections/measure.md.tmpl b/ship/sections/measure.md.tmpl new file mode 100644 index 000000000..5568794a9 --- /dev/null +++ b/ship/sections/measure.md.tmpl @@ -0,0 +1,52 @@ +## Measure-then-fix loop (a red paid eval or a failed free-suite shard) + +Every rerun here is **diagnostic, not a verdict**: it never changes the recorded +verdict of the run that failed, its artifacts are labeled `diagnostic` under +`.context/ship-measure/` (never the eval history), and the lane verdict comes +only from the one full gate run at the end. Never raise a budget, lower a +threshold, add a verdict retry or skip a case to get a pass. +`ship-measure` below means `bun run ~/.claude/skills/gstack/scripts/ship-measure.ts`. + +**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or +pushing, rerun that shard's exact file list (in run order) with +`ship-measure free --files ` (gstack local runs: `--shard `): 10 +reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with +its own HOME and state root, the flaky retry off, under a 10-minute cap. +`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one +Ubicloud VM. If the runner cannot take a file list, rerun the whole failing +command 10 times. Report the pass rate, the completed count (for example 6 of 10 +completed at the cap) and one failure capture, then continue the triage. + +**Red paid eval (Step 6).** For each red case: + +1. **Classify.** Read the red line's cause, evidence and Expected/Received + (`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run --case `). + Name it: product change from this branch, detector or actor defect, fixture + defect, provider event, or model miss. A detector fix gets a free replay test + before any paid trial. A red that is infrastructure (provider outage, runner + loss) may skip measurement: `ship-measure skip --case --reason ""` + labels it `unmeasured` in the PR body. It never counts as a pass. +2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and + budget per kind, from the `ship_measure_*` gstack config keys) and the + estimate: per-trial cost from pass-rate history times trials, or "estimate unknown". +3. **Measure.** `ship-measure measure --case --round baseline [--cost-per-trial ]` + runs the case alone on the current head at its kind's count, in parallel. It + uses the project's documented single-case eval command, + `--command ' {case}'`, found the way Step 4 finds test commands + (CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no + `--command`: `scripts/test-paid-shards.ts --tier --case `, and it + accepts standalone judge ids. With no documented single-case command, ask + once and record the answer in the report. Exit 2 means it needs approval: + ask once with the reason it printed, then rerun with `--approved`. +4. **Fix at the cause.** Root-cause the failed trials from their captures in + `.context/ship-measure///tNN/` and change the cause. +5. **Re-measure.** `ship-measure measure --case --round round- --fix ""`. + Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call + a case fixed only with a named causal change and every failure in the closing + measurement explained; report "observed k/n after fix at ", never a proven rate. +6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget + exhausted): stop, do not push, and report `ship-measure report` with every trial. + +**Gate once.** When every red case measures at or above target, run the full +gate command once (Step 6 item 2). Put the `ship-measure report` table (estimated +and actual spend per case) in the PR body. diff --git a/ship/sections/tests.md b/ship/sections/tests.md index 44b7efb2f..9e0725028 100644 --- a/ship/sections/tests.md +++ b/ship/sections/tests.md @@ -228,7 +228,10 @@ log=...` summary lines — each carries the lane's exit code and a per-run log file (no shared /tmp collisions between concurrent ships). Read the log files for failure detail. -**If any test fails:** Do NOT immediately stop. Apply the Test Failure Ownership Triage: +**If any test fails:** Do NOT immediately stop. When a free-suite shard failed, +first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that +shard's file list as it says (diagnostic, not a verdict). Then apply the Test +Failure Ownership Triage: ## Test Failure Ownership Triage @@ -391,7 +394,10 @@ satisfy coverage. **3. Check results and save evidence for Step 19.** -- **If any eval fails:** Show failures and available costs, then **STOP**. +- **If any eval fails:** Do not rerun the full gate. Show failures and available + costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its + loop for each red case: classify, measure alone, fix at the cause, re-measure, + then gate once. **STOP** only at its named-red stop. - **If all selected evals pass:** Record actual counts, any reused evidence and its source, and available costs. Continue to Step 7. diff --git a/ship/sections/tests.md.tmpl b/ship/sections/tests.md.tmpl index 1ae7de964..fdb381ad0 100644 --- a/ship/sections/tests.md.tmpl +++ b/ship/sections/tests.md.tmpl @@ -38,7 +38,10 @@ log=...` summary lines — each carries the lane's exit code and a per-run log file (no shared /tmp collisions between concurrent ships). Read the log files for failure detail. -**If any test fails:** Do NOT immediately stop. Apply the Test Failure Ownership Triage: +**If any test fails:** Do NOT immediately stop. When a free-suite shard failed, +first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that +shard's file list as it says (diagnostic, not a verdict). Then apply the Test +Failure Ownership Triage: {{TEST_FAILURE_TRIAGE}} @@ -105,7 +108,10 @@ satisfy coverage. **3. Check results and save evidence for Step 19.** -- **If any eval fails:** Show failures and available costs, then **STOP**. +- **If any eval fails:** Do not rerun the full gate. Show failures and available + costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its + loop for each red case: classify, measure alone, fix at the cause, re-measure, + then gate once. **STOP** only at its named-red stop. - **If all selected evals pass:** Record actual counts, any reused evidence and its source, and available costs. Continue to Step 7. diff --git a/test/helpers/carve-guards.ts b/test/helpers/carve-guards.ts index 07901f4eb..474830706 100644 --- a/test/helpers/carve-guards.ts +++ b/test/helpers/carve-guards.ts @@ -166,7 +166,7 @@ export const CARVE_GUARDS: Record = { // wave's headline capability) grows the union to 1.195x. Deliberate: // the section is on-demand (loads only for Apple store targets), so // per-invocation cost for non-iOS ships is one manifest line. - maxSizeRatio: 1.432, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + maxSizeRatio: 1.455, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + A6 measure-then-fix loop: the lazily read measure.md section (classify, measure alone, fix, re-measure, gate once; free-shard reruns with the retry off) and its two pointers in tests.md; measured 272,944 / 187,706 = 1.4541 (2026-10-06). }, 'plan-ceo-review': { skill: 'plan-ceo-review', diff --git a/test/ship-measure-loop-skill.test.ts b/test/ship-measure-loop-skill.test.ts new file mode 100644 index 000000000..ca73750f4 --- /dev/null +++ b/test/ship-measure-loop-skill.test.ts @@ -0,0 +1,61 @@ +/** + * A6 acceptance: the generated ship skill runs the measure-then-fix loop on a + * red paid eval (classify, measure alone, fix at the cause, re-measure, gate + * once) and measures a failed free shard before calling it pre-existing, and + * it states the "diagnostic, not verdict" rule. Every ship-measure command and + * flag the skill names exists in scripts/ship-measure.ts. + */ +import { describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; + +const ROOT = path.resolve(import.meta.dir, '..'); + +describe('the generated ship skill runs the loop (A6 acceptance)', () => { + const tests = fs.readFileSync(path.join(ROOT, 'ship/sections/tests.md'), 'utf8'); + const loop = fs.readFileSync(path.join(ROOT, 'ship/sections/measure.md'), 'utf8'); + const skeleton = fs.readFileSync(path.join(ROOT, 'ship/SKILL.md'), 'utf8'); + + test('a red eval no longer stops /ship: Step 6 reads the loop instead of rerunning the gate', () => { + expect(tests).not.toContain('Show failures and available costs, then **STOP**.'); + const step6 = tests.slice(tests.indexOf('## Step 6: Eval Suites')); + expect(step6).toMatch(/If any eval fails:\*\* Do not rerun the full gate\./); + expect(step6).toContain('Read `~/.claude/skills/gstack/ship/sections/measure.md`'); + expect(step6).toContain('classify, measure alone, fix at the cause, re-measure,\n then gate once'); + expect(skeleton).toContain('| `sections/measure.md` |'); + }); + + test('the loop steps run in order: classify, plan, measure, fix, re-measure, stop, then gate once', () => { + const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Fix at the cause.**', '5. **Re-measure.**', '6. **Stop.**', '**Gate once.**']; + const positions = steps.map(step => loop.indexOf(step)); + expect(positions.every(p => p >= 0)).toBe(true); + expect([...positions].sort((a, b) => a - b)).toEqual(positions); + expect(loop).toContain('run the full\ngate command once'); + }); + + test('the diagnostic-not-verdict rule, the honest-fix rule and the unmeasured label are stated', () => { + expect(loop).toContain('Every rerun here is **diagnostic, not a verdict**: it never changes the recorded\nverdict of the run that failed'); + expect(loop).toContain('Never raise a budget, lower a\nthreshold, add a verdict retry or skip a case to get a pass.'); + expect(loop).toContain('report "observed k/n after fix at ", never a proven rate'); + expect(loop).toContain('labels it `unmeasured` in the PR body. It never counts as a pass.'); + expect(loop).toContain('ask\n once and record the answer in the report'); + }); + + test('a failed free shard is measured before it is called pre-existing, with the retry off', () => { + expect(tests).toContain('When a free-suite shard failed,\nfirst Read `~/.claude/skills/gstack/ship/sections/measure.md`'); + expect(tests.indexOf('sections/measure.md')).toBeLessThan(tests.indexOf('Then apply the Test\nFailure Ownership Triage')); + const free = loop.slice(loop.indexOf('**Failed free-suite shard (Step 5).**'), loop.indexOf('**Red paid eval (Step 6).**')); + for (const phrase of ['Before calling the failure pre-existing or\npushing', 'exact file list', 'rerun 1 alone as the baseline', 'the flaky retry off', + '10-minute cap', 'ship_rerun_backend=ubicloud', 'rerun the whole failing\ncommand', 'completed at the cap']) expect(free).toContain(phrase); + }); + + test('every ship-measure command and flag the skill names exists in the runner', () => { + const source = fs.readFileSync(path.join(ROOT, 'scripts/ship-measure.ts'), 'utf8'); + const named = [...loop.matchAll(/ship-measure (\w+)([^`]*)`/g)]; + expect(named.length).toBeGreaterThan(5); + for (const [, command, rest] of named) { + expect(source).toMatch(new RegExp(`command [!=]== '${command}'`)); + for (const [flag] of rest!.matchAll(/--[a-z-]+/g)) expect(source).toContain(`'${flag}'`); + } + }); +});