diff --git a/ship/sections/measure.md b/ship/sections/measure.md index 7246d3de5..1edcec777 100644 --- a/ship/sections/measure.md +++ b/ship/sections/measure.md @@ -28,26 +28,35 @@ completed at the cap) and one failure capture, then continue the triage. before any paid trial. A red that is infrastructure (provider outage, runner loss) may skip measurement: `ship-measure skip --case --reason ""` labels it `unmeasured` in the PR body. It never counts as a pass. -2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and - budget per kind, from the `ship_measure_*` gstack config keys) and the - estimate: per-trial cost from pass-rate history times trials, or "estimate unknown". +2. **Show the plan.** Print `ship-measure table` (trials, bar and budget per + kind, from the `ship_measure_*` gstack config keys) and the estimate: + per-trial cost from pass-rate history times trials, or "estimate unknown". 3. **Measure.** `ship-measure measure --case --round baseline [--cost-per-trial ]` - runs the case alone on the current head at its kind's count, in parallel. It - uses the project's documented single-case eval command, - `--command ' {case}'`, found the way Step 4 finds test commands - (CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no - `--command`: `scripts/test-paid-shards.ts --tier --case `, and it - accepts standalone judge ids. With no documented single-case command, ask - once and record the answer in the report. Exit 2 means it needs approval: - ask once with the reason it printed, then rerun with `--approved`. -4. **Fix at the cause.** Root-cause the failed trials from their captures in + runs the case alone on the current head at its kind's count, in parallel, + through the project's documented single-case eval command, + `--command ' {case}'` (found the way Step 4 finds test commands; + gstack needs none and accepts judge ids). With no documented command, ask + once and record the answer in the report. Exit 2 needs approval: ask once + with its reason, then rerun with `--approved`. +4. **Decide by the exit code** (MEETS 9/10, MEETS-qualified 8/10 with every + red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12): + - 0: MEETS or MEETS-qualified; done. + - 5: needs-classify. Record each listed red from its capture: + `ship-measure classify --case --round --trials <2,5> --class --evidence ""`. + Qualifying: `provider` (only on the trial's own provider error), + `judge-noise`, `model-miss` (cite the line). Fix round: `timeout`, + `hang`, `regression`, `fixable`. `unclassified`: reviewed, no evidence. + - 6: EXTEND. `ship-measure extend --case --round ` once, on the + unchanged tree; never a third batch. + - 1: BELOW, a fix round. +5. **Fix at the cause.** Root-cause the failed trials from their captures in `.context/ship-measure///tNN/` and change the cause. -5. **Re-measure.** `ship-measure measure --case --round round- --fix ""`. - Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call - a case fixed only with a named causal change and every failure in the closing - measurement explained; report "observed k/n after fix at ", never a proven rate. -6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget - exhausted): stop, do not push, and report `ship-measure report` with every trial. +6. **Re-measure.** `ship-measure measure --case --round round- --fix ""`, + then item 4. Call a case fixed only with a named causal change and every + failure in the closing measurement explained; + report "observed k/n after fix at ", never a proven rate. +7. **Stop.** Exit 3 is a named red (round limit, budget, or two void + batches): stop, do not push, and report `ship-measure report` with every trial. **Gate once.** When every red case measures at or above target, run the full gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack: diff --git a/ship/sections/measure.md.tmpl b/ship/sections/measure.md.tmpl index 8fcccf2a7..6dfd0e02a 100644 --- a/ship/sections/measure.md.tmpl +++ b/ship/sections/measure.md.tmpl @@ -26,26 +26,35 @@ completed at the cap) and one failure capture, then continue the triage. before any paid trial. A red that is infrastructure (provider outage, runner loss) may skip measurement: `ship-measure skip --case --reason ""` labels it `unmeasured` in the PR body. It never counts as a pass. -2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and - budget per kind, from the `ship_measure_*` gstack config keys) and the - estimate: per-trial cost from pass-rate history times trials, or "estimate unknown". +2. **Show the plan.** Print `ship-measure table` (trials, bar and budget per + kind, from the `ship_measure_*` gstack config keys) and the estimate: + per-trial cost from pass-rate history times trials, or "estimate unknown". 3. **Measure.** `ship-measure measure --case --round baseline [--cost-per-trial ]` - runs the case alone on the current head at its kind's count, in parallel. It - uses the project's documented single-case eval command, - `--command ' {case}'`, found the way Step 4 finds test commands - (CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no - `--command`: `scripts/test-paid-shards.ts --tier --case `, and it - accepts standalone judge ids. With no documented single-case command, ask - once and record the answer in the report. Exit 2 means it needs approval: - ask once with the reason it printed, then rerun with `--approved`. -4. **Fix at the cause.** Root-cause the failed trials from their captures in + runs the case alone on the current head at its kind's count, in parallel, + through the project's documented single-case eval command, + `--command ' {case}'` (found the way Step 4 finds test commands; + gstack needs none and accepts judge ids). With no documented command, ask + once and record the answer in the report. Exit 2 needs approval: ask once + with its reason, then rerun with `--approved`. +4. **Decide by the exit code** (MEETS 9/10, MEETS-qualified 8/10 with every + red qualifying, EXTEND 7/10; behavior 11/12, 10/12, 9/12): + - 0: MEETS or MEETS-qualified; done. + - 5: needs-classify. Record each listed red from its capture: + `ship-measure classify --case --round --trials <2,5> --class --evidence ""`. + Qualifying: `provider` (only on the trial's own provider error), + `judge-noise`, `model-miss` (cite the line). Fix round: `timeout`, + `hang`, `regression`, `fixable`. `unclassified`: reviewed, no evidence. + - 6: EXTEND. `ship-measure extend --case --round ` once, on the + unchanged tree; never a third batch. + - 1: BELOW, a fix round. +5. **Fix at the cause.** Root-cause the failed trials from their captures in `.context/ship-measure///tNN/` and change the cause. -5. **Re-measure.** `ship-measure measure --case --round round- --fix ""`. - Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call - a case fixed only with a named causal change and every failure in the closing - measurement explained; report "observed k/n after fix at ", never a proven rate. -6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget - exhausted): stop, do not push, and report `ship-measure report` with every trial. +6. **Re-measure.** `ship-measure measure --case --round round- --fix ""`, + then item 4. Call a case fixed only with a named causal change and every + failure in the closing measurement explained; + report "observed k/n after fix at ", never a proven rate. +7. **Stop.** Exit 3 is a named red (round limit, budget, or two void + batches): stop, do not push, and report `ship-measure report` with every trial. **Gate once.** When every red case measures at or above target, run the full gate once (Step 6 item 2). When the project's CI runs that gate on push (gstack: diff --git a/test/helpers/carve-guards.ts b/test/helpers/carve-guards.ts index 282c33a47..92d6b8305 100644 --- a/test/helpers/carve-guards.ts +++ b/test/helpers/carve-guards.ts @@ -166,7 +166,7 @@ export const CARVE_GUARDS: Record = { // wave's headline capability) grows the union to 1.195x. Deliberate: // the section is on-demand (loads only for Apple store targets), so // per-invocation cost for non-iOS ships is one manifest line. - maxSizeRatio: 1.495, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + Oct 6 fix wave A3/CEO-12: free text reaches commands only through agent-written mktemp files (mktemp block, write rule, path rebuild and refusal at each posting, prompt and brief site); measured 275,422 / 187,706 = 1.4673 (2026-10-06). Combined Oct 6 wave head (B free-text files + D measure loop + A rework): measured 280,363 / 187,706 = 1.4936 (2026-10-06). + gate-once push rule (no second paid gate after a CI push); measured 280,573 / 187,706 = 1.4947 (2026-10-06). + maxSizeRatio: 1.4975, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + Oct 6 fix wave A3/CEO-12: free text reaches commands only through agent-written mktemp files (mktemp block, write rule, path rebuild and refusal at each posting, prompt and brief site); measured 275,422 / 187,706 = 1.4673 (2026-10-06). Combined Oct 6 wave head (B free-text files + D measure loop + A rework): measured 280,363 / 187,706 = 1.4936 (2026-10-06). + gate-once push rule (no second paid gate after a CI push); measured 280,573 / 187,706 = 1.4947 (2026-10-06). + the measurement bar's decision step (exit-code branches for MEETS, MEETS-qualified, needs-classify with the qualifying and fix-round classes, EXTEND once; the measure prose was tightened to offset most of it); measured 281,058 / 187,706 = 1.4973 (2026-10-06). }, 'plan-ceo-review': { skill: 'plan-ceo-review', diff --git a/test/ship-measure-loop-skill.test.ts b/test/ship-measure-loop-skill.test.ts index 52789c442..7e16b3c34 100644 --- a/test/ship-measure-loop-skill.test.ts +++ b/test/ship-measure-loop-skill.test.ts @@ -1,7 +1,7 @@ /** * A6 acceptance: the generated ship skill runs the measure-then-fix loop on a - * red paid eval (classify, measure alone, fix at the cause, re-measure, gate - * once) and measures a failed free shard before calling it pre-existing, and + * red paid eval (classify, measure alone, decide by the measurement bar, fix + * at the cause, re-measure, gate once) and measures a failed free shard before calling it pre-existing, and * it states the "diagnostic, not verdict" rule. Every ship-measure command and * flag the skill names exists in scripts/ship-measure.ts. */ @@ -25,8 +25,8 @@ describe('the generated ship skill runs the loop (A6 acceptance)', () => { expect(skeleton).toContain('| `sections/measure.md` |'); }); - test('the loop steps run in order: classify, plan, measure, fix, re-measure, stop, then gate once', () => { - const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Fix at the cause.**', '5. **Re-measure.**', '6. **Stop.**', '**Gate once.**']; + test('the loop steps run in order: classify, plan, measure, decide, fix, re-measure, stop, then gate once', () => { + const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Decide by the exit code**', '5. **Fix at the cause.**', '6. **Re-measure.**', '7. **Stop.**', '**Gate once.**']; const positions = steps.map(step => loop.indexOf(step)); expect(positions.every(p => p >= 0)).toBe(true); expect([...positions].sort((a, b) => a - b)).toEqual(positions); @@ -36,6 +36,17 @@ describe('the generated ship skill runs the loop (A6 acceptance)', () => { expect(loop).toContain('do not also start the gate command'); }); + test('the decision branches on every exit code the runner returns, with the bar and its rules', () => { + const decideStep = loop.slice(loop.indexOf('4. **Decide by the exit code**'), loop.indexOf('5. **Fix at the cause.**')); + for (const code of ['- 0: MEETS or MEETS-qualified', '- 5: needs-classify', '- 6: EXTEND', '- 1: BELOW']) expect(decideStep).toContain(code); + for (const cls of ['provider', 'judge-noise', 'model-miss', 'timeout', 'hang', 'regression', 'fixable', 'unclassified']) expect(decideStep).toContain(`\`${cls}\``); + expect(decideStep).toMatch(/MEETS 9\/10, MEETS-qualified 8\/10[\s\S]*EXTEND 7\/10; behavior 11\/12, 10\/12, 9\/12/); + expect(decideStep).toMatch(/never a third batch/i); + expect(loop.slice(loop.indexOf('7. **Stop.**'))).toContain('Exit 3 is a named red'); + const source = fs.readFileSync(path.join(ROOT, 'scripts/lib/measure-bar.ts'), 'utf8'); + for (const cls of ['provider', 'judge-noise', 'model-miss', 'timeout', 'hang', 'regression', 'fixable', 'unclassified']) expect(source).toContain(`'${cls}'`); + }); + test('the diagnostic-not-verdict rule, the honest-fix rule and the unmeasured label are stated', () => { expect(loop).toContain('Every rerun here is **diagnostic, not a verdict**: it never changes the recorded\nverdict of the run that failed'); expect(loop).toContain('Never raise a budget, lower a\nthreshold, add a verdict retry or skip a case to get a pass.');