mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-09 10:18:32 +01:00
feat(ship): a red eval runs classify, measure, fix, re-measure, gate once (A6)
Step 6 no longer stops at the first red paid eval: it reads the new lazily loaded measure section and runs the loop for each red case, and Step 5 measures a failed free-suite shard (diagnostic reruns, retry off) before the triage can call it pre-existing. The section states the diagnostic- not-verdict rule, the per-kind table, the ask-once and named-red stops, the unmeasured label and the honest-fix wording (observed k/n after fix at <cause>). A free test pins the steps and their order in the generated skill and checks every command and flag it names exists in the runner. Ship's parity cap rises to 1.455 (measured 272,944 / 187,706 = 1.4541).
This commit is contained in:
garrytan committed
1 parent
7dc6cd2c00
commit
4454ad67e3
8 files changed
+191
-5
No files matched your search
@@ -486,6 +486,7 @@ sections. Read a section in full before doing its step; do not work from memory.
|
||||
|------|-------------------|
|
||||
| App Store/TestFlight distribution is requested for an Apple app (.xcodeproj, .xcworkspace, or an app-product Swift package) — read at Step 0.9 before the branch gate; an Apple repository-landing request follows the normal pipeline | `sections/apple-release.md` |
|
||||
| running the test suites and (if prompt files changed) the eval suites (Steps 4-6) | `sections/tests.md` |
|
||||
| a paid eval case is red or a free-suite shard failed (Steps 5-6) — read from the tests section; diagnostic reruns, then the gate once | `sections/measure.md` |
|
||||
| auditing test coverage of the diff (Step 7) | `sections/test-coverage.md` |
|
||||
| auditing plan completion, verification, and scope drift (Step 8) | `sections/plan-completion.md` |
|
||||
| the pre-landing review and specialist dispatch (Step 9) | `sections/review-army.md` |
|
||||
|
||||
@@ -16,6 +16,12 @@
|
||||
"title": "Test bootstrap, run, triage + eval suites",
|
||||
"trigger": "running the test suites and (if prompt files changed) the eval suites (Steps 4-6)"
|
||||
},
|
||||
{
|
||||
"id": "measure",
|
||||
"file": "measure.md",
|
||||
"title": "Measure-then-fix loop for a red eval or free shard",
|
||||
"trigger": "a paid eval case is red or a free-suite shard failed (Steps 5-6) \u2014 read from the tests section; diagnostic reruns, then the gate once"
|
||||
},
|
||||
{
|
||||
"id": "test-coverage",
|
||||
"file": "test-coverage.md",
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
<!-- AUTO-GENERATED from measure.md.tmpl — do not edit directly -->
|
||||
<!-- Regenerate: bun run gen:skill-docs -->
|
||||
## Measure-then-fix loop (a red paid eval or a failed free-suite shard)
|
||||
|
||||
Every rerun here is **diagnostic, not a verdict**: it never changes the recorded
|
||||
verdict of the run that failed, its artifacts are labeled `diagnostic` under
|
||||
`.context/ship-measure/` (never the eval history), and the lane verdict comes
|
||||
only from the one full gate run at the end. Never raise a budget, lower a
|
||||
threshold, add a verdict retry or skip a case to get a pass.
|
||||
`ship-measure` below means `bun run ~/.claude/skills/gstack/scripts/ship-measure.ts`.
|
||||
|
||||
**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or
|
||||
pushing, rerun that shard's exact file list (in run order) with
|
||||
`ship-measure free --files <f1,f2,...>` (gstack local runs: `--shard <i>`): 10
|
||||
reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with
|
||||
its own HOME and state root, the flaky retry off, under a 10-minute cap.
|
||||
`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one
|
||||
Ubicloud VM. If the runner cannot take a file list, rerun the whole failing
|
||||
command 10 times. Report the pass rate, the completed count (for example 6 of 10
|
||||
completed at the cap) and one failure capture, then continue the triage.
|
||||
|
||||
**Red paid eval (Step 6).** For each red case:
|
||||
|
||||
1. **Classify.** Read the red line's cause, evidence and Expected/Received
|
||||
(`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run <run> --case <id>`).
|
||||
Name it: product change from this branch, detector or actor defect, fixture
|
||||
defect, provider event, or model miss. A detector fix gets a free replay test
|
||||
before any paid trial. A red that is infrastructure (provider outage, runner
|
||||
loss) may skip measurement: `ship-measure skip --case <id> --reason "<why>"`
|
||||
labels it `unmeasured` in the PR body. It never counts as a pass.
|
||||
2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and
|
||||
budget per kind, from the `ship_measure_*` gstack config keys) and the
|
||||
estimate: per-trial cost from pass-rate history times trials, or "estimate unknown".
|
||||
3. **Measure.** `ship-measure measure --case <id> --round baseline [--cost-per-trial <usd>]`
|
||||
runs the case alone on the current head at its kind's count, in parallel. It
|
||||
uses the project's documented single-case eval command,
|
||||
`--command '<command> {case}'`, found the way Step 4 finds test commands
|
||||
(CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no
|
||||
`--command`: `scripts/test-paid-shards.ts --tier <tier> --case <id>`, and it
|
||||
accepts standalone judge ids. With no documented single-case command, ask
|
||||
once and record the answer in the report. Exit 2 means it needs approval:
|
||||
ask once with the reason it printed, then rerun with `--approved`.
|
||||
4. **Fix at the cause.** Root-cause the failed trials from their captures in
|
||||
`.context/ship-measure/<case>/<round>/tNN/` and change the cause.
|
||||
5. **Re-measure.** `ship-measure measure --case <id> --round round-<n> --fix "<cause and change>"`.
|
||||
Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call
|
||||
a case fixed only with a named causal change and every failure in the closing
|
||||
measurement explained; report "observed k/n after fix at <cause>", never a proven rate.
|
||||
6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget
|
||||
exhausted): stop, do not push, and report `ship-measure report` with every trial.
|
||||
|
||||
**Gate once.** When every red case measures at or above target, run the full
|
||||
gate command once (Step 6 item 2). Put the `ship-measure report` table (estimated
|
||||
and actual spend per case) in the PR body.
|
||||
@@ -0,0 +1,52 @@
|
||||
## Measure-then-fix loop (a red paid eval or a failed free-suite shard)
|
||||
|
||||
Every rerun here is **diagnostic, not a verdict**: it never changes the recorded
|
||||
verdict of the run that failed, its artifacts are labeled `diagnostic` under
|
||||
`.context/ship-measure/` (never the eval history), and the lane verdict comes
|
||||
only from the one full gate run at the end. Never raise a budget, lower a
|
||||
threshold, add a verdict retry or skip a case to get a pass.
|
||||
`ship-measure` below means `bun run ~/.claude/skills/gstack/scripts/ship-measure.ts`.
|
||||
|
||||
**Failed free-suite shard (Step 5).** Before calling the failure pre-existing or
|
||||
pushing, rerun that shard's exact file list (in run order) with
|
||||
`ship-measure free --files <f1,f2,...>` (gstack local runs: `--shard <i>`): 10
|
||||
reruns, rerun 1 alone as the baseline and the rest labeled parallel, each with
|
||||
its own HOME and state root, the flaky retry off, under a 10-minute cap.
|
||||
`ship_rerun_backend=ubicloud` with `UBICLOUD_API_KEY` set runs them on one
|
||||
Ubicloud VM. If the runner cannot take a file list, rerun the whole failing
|
||||
command 10 times. Report the pass rate, the completed count (for example 6 of 10
|
||||
completed at the cap) and one failure capture, then continue the triage.
|
||||
|
||||
**Red paid eval (Step 6).** For each red case:
|
||||
|
||||
1. **Classify.** Read the red line's cause, evidence and Expected/Received
|
||||
(`failure_cause`, `failure_detail`; gstack: `bun run eval:pass-rates --run <run> --case <id>`).
|
||||
Name it: product change from this branch, detector or actor defect, fixture
|
||||
defect, provider event, or model miss. A detector fix gets a free replay test
|
||||
before any paid trial. A red that is infrastructure (provider outage, runner
|
||||
loss) may skip measurement: `ship-measure skip --case <id> --reason "<why>"`
|
||||
labels it `unmeasured` in the PR body. It never counts as a pass.
|
||||
2. **Show the plan.** Print `ship-measure table` (trials, pass bar, panels and
|
||||
budget per kind, from the `ship_measure_*` gstack config keys) and the
|
||||
estimate: per-trial cost from pass-rate history times trials, or "estimate unknown".
|
||||
3. **Measure.** `ship-measure measure --case <id> --round baseline [--cost-per-trial <usd>]`
|
||||
runs the case alone on the current head at its kind's count, in parallel. It
|
||||
uses the project's documented single-case eval command,
|
||||
`--command '<command> {case}'`, found the way Step 4 finds test commands
|
||||
(CLAUDE.md/AGENTS.md, then package scripts). gstack's default needs no
|
||||
`--command`: `scripts/test-paid-shards.ts --tier <tier> --case <id>`, and it
|
||||
accepts standalone judge ids. With no documented single-case command, ask
|
||||
once and record the answer in the report. Exit 2 means it needs approval:
|
||||
ask once with the reason it printed, then rerun with `--approved`.
|
||||
4. **Fix at the cause.** Root-cause the failed trials from their captures in
|
||||
`.context/ship-measure/<case>/<round>/tNN/` and change the cause.
|
||||
5. **Re-measure.** `ship-measure measure --case <id> --round round-<n> --fix "<cause and change>"`.
|
||||
Exit 0 is at or above target. Exit 1 is below target: return to step 4. Call
|
||||
a case fixed only with a named causal change and every failure in the closing
|
||||
measurement explained; report "observed k/n after fix at <cause>", never a proven rate.
|
||||
6. **Stop.** Exit 3 is a named red (repair-round limit or the per-case budget
|
||||
exhausted): stop, do not push, and report `ship-measure report` with every trial.
|
||||
|
||||
**Gate once.** When every red case measures at or above target, run the full
|
||||
gate command once (Step 6 item 2). Put the `ship-measure report` table (estimated
|
||||
and actual spend per case) in the PR body.
|
||||
@@ -228,7 +228,10 @@ log=...` summary lines — each carries the lane's exit code and a per-run log
|
||||
file (no shared /tmp collisions between concurrent ships). Read the log files
|
||||
for failure detail.
|
||||
|
||||
**If any test fails:** Do NOT immediately stop. Apply the Test Failure Ownership Triage:
|
||||
**If any test fails:** Do NOT immediately stop. When a free-suite shard failed,
|
||||
first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that
|
||||
shard's file list as it says (diagnostic, not a verdict). Then apply the Test
|
||||
Failure Ownership Triage:
|
||||
|
||||
## Test Failure Ownership Triage
|
||||
|
||||
@@ -391,7 +394,10 @@ satisfy coverage.
|
||||
|
||||
**3. Check results and save evidence for Step 19.**
|
||||
|
||||
- **If any eval fails:** Show failures and available costs, then **STOP**.
|
||||
- **If any eval fails:** Do not rerun the full gate. Show failures and available
|
||||
costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its
|
||||
loop for each red case: classify, measure alone, fix at the cause, re-measure,
|
||||
then gate once. **STOP** only at its named-red stop.
|
||||
- **If all selected evals pass:** Record actual counts, any reused evidence and
|
||||
its source, and available costs. Continue to Step 7.
|
||||
|
||||
|
||||
@@ -38,7 +38,10 @@ log=...` summary lines — each carries the lane's exit code and a per-run log
|
||||
file (no shared /tmp collisions between concurrent ships). Read the log files
|
||||
for failure detail.
|
||||
|
||||
**If any test fails:** Do NOT immediately stop. Apply the Test Failure Ownership Triage:
|
||||
**If any test fails:** Do NOT immediately stop. When a free-suite shard failed,
|
||||
first Read `~/.claude/skills/gstack/ship/sections/measure.md` and rerun that
|
||||
shard's file list as it says (diagnostic, not a verdict). Then apply the Test
|
||||
Failure Ownership Triage:
|
||||
|
||||
{{TEST_FAILURE_TRIAGE}}
|
||||
|
||||
@@ -105,7 +108,10 @@ satisfy coverage.
|
||||
|
||||
**3. Check results and save evidence for Step 19.**
|
||||
|
||||
- **If any eval fails:** Show failures and available costs, then **STOP**.
|
||||
- **If any eval fails:** Do not rerun the full gate. Show failures and available
|
||||
costs, then Read `~/.claude/skills/gstack/ship/sections/measure.md` and run its
|
||||
loop for each red case: classify, measure alone, fix at the cause, re-measure,
|
||||
then gate once. **STOP** only at its named-red stop.
|
||||
- **If all selected evals pass:** Record actual counts, any reused evidence and
|
||||
its source, and available costs. Continue to Step 7.
|
||||
|
||||
|
||||
@@ -166,7 +166,7 @@ export const CARVE_GUARDS: Record<string, CarveGuard> = {
|
||||
// wave's headline capability) grows the union to 1.195x. Deliberate:
|
||||
// the section is on-demand (loads only for Apple store targets), so
|
||||
// per-invocation cost for non-iOS ships is one manifest line.
|
||||
maxSizeRatio: 1.432, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06).
|
||||
maxSizeRatio: 1.455, // Shared advisory identity/dedup + critical-severity validation: 248,065 union bytes / 187,706 baseline = 1.3216 (2026-09-17). + test value bar in the lazy Step 7 section (value cards, weak paths, gate table, base control, machine checks; ~13.6KB): measured 1.396 (2026-09-29). + W1 guarded state-root resolution (`eval gstack-paths; : "${GSTACK_STATE_ROOT:?…}"`) in the Context Recovery preamble, the eureka log and each state-writing bash block; measured 1.401 (2026-09-30). + the shared QA review step's plan-check timing rule (plan checks and their revalidation run on --timeout-ms after smoke expiry); measured 1.4022 (2026-09-30). + the shared TEST_BOOTSTRAP owned-change undo rules (s03 H1-H3, M1) and the value-bar coverage wording (s03 M2/M3); measured 1.4044 (2026-10-02). + severe fix wave ship-review lane: G1 NO_VERSION dispatch, title and CHANGELOG arms (versionless repos ship without inventing a version), F1 parent-side plan binding before the audit dispatch (binding, docs/designs candidates, exact not-run line) and D3's data-not-instructions line; +2,126 union bytes, measured 1.4135 (2026-10-03). + INV-1 consumer lines (outside review `unverified`/`unavailable` is missing coverage in the readiness note and PR body); measured 1.4148 (2026-10-03). + v1.91.19.0 severe fix wave: verdict-form outside-review callers with sandbox preflight and first-use notice (INV-1/B1/Q2), P0/P1 gate and unverified wording, bound-plan discovery (F1), visible LEARNINGS failures (B5), carved external pointers; measured 267,752 / 187,706 = 1.4264 (2026-10-03). + v1.91.18.0 (#3023) merged into the v1.91.19.0 wave; measured 267,874 / 187,706 = 1.4271 (2026-10-04). + Oct 6 fix wave: quota_exhausted preflight branch (#3051) and quoted-heredoc issue template (#3046); measured 268,199 / 187,706 = 1.4288 (2026-10-06). + Step 10's inline no-PR check before the Greptile dispatch (#3020); measured 268,655 / 187,706 = 1.4313 (2026-10-06). + A6 measure-then-fix loop: the lazily read measure.md section (classify, measure alone, fix, re-measure, gate once; free-shard reruns with the retry off) and its two pointers in tests.md; measured 272,944 / 187,706 = 1.4541 (2026-10-06).
|
||||
},
|
||||
'plan-ceo-review': {
|
||||
skill: 'plan-ceo-review',
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
/**
|
||||
* A6 acceptance: the generated ship skill runs the measure-then-fix loop on a
|
||||
* red paid eval (classify, measure alone, fix at the cause, re-measure, gate
|
||||
* once) and measures a failed free shard before calling it pre-existing, and
|
||||
* it states the "diagnostic, not verdict" rule. Every ship-measure command and
|
||||
* flag the skill names exists in scripts/ship-measure.ts.
|
||||
*/
|
||||
import { describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
|
||||
describe('the generated ship skill runs the loop (A6 acceptance)', () => {
|
||||
const tests = fs.readFileSync(path.join(ROOT, 'ship/sections/tests.md'), 'utf8');
|
||||
const loop = fs.readFileSync(path.join(ROOT, 'ship/sections/measure.md'), 'utf8');
|
||||
const skeleton = fs.readFileSync(path.join(ROOT, 'ship/SKILL.md'), 'utf8');
|
||||
|
||||
test('a red eval no longer stops /ship: Step 6 reads the loop instead of rerunning the gate', () => {
|
||||
expect(tests).not.toContain('Show failures and available costs, then **STOP**.');
|
||||
const step6 = tests.slice(tests.indexOf('## Step 6: Eval Suites'));
|
||||
expect(step6).toMatch(/If any eval fails:\*\* Do not rerun the full gate\./);
|
||||
expect(step6).toContain('Read `~/.claude/skills/gstack/ship/sections/measure.md`');
|
||||
expect(step6).toContain('classify, measure alone, fix at the cause, re-measure,\n then gate once');
|
||||
expect(skeleton).toContain('| `sections/measure.md` |');
|
||||
});
|
||||
|
||||
test('the loop steps run in order: classify, plan, measure, fix, re-measure, stop, then gate once', () => {
|
||||
const steps = ['1. **Classify.**', '2. **Show the plan.**', '3. **Measure.**', '4. **Fix at the cause.**', '5. **Re-measure.**', '6. **Stop.**', '**Gate once.**'];
|
||||
const positions = steps.map(step => loop.indexOf(step));
|
||||
expect(positions.every(p => p >= 0)).toBe(true);
|
||||
expect([...positions].sort((a, b) => a - b)).toEqual(positions);
|
||||
expect(loop).toContain('run the full\ngate command once');
|
||||
});
|
||||
|
||||
test('the diagnostic-not-verdict rule, the honest-fix rule and the unmeasured label are stated', () => {
|
||||
expect(loop).toContain('Every rerun here is **diagnostic, not a verdict**: it never changes the recorded\nverdict of the run that failed');
|
||||
expect(loop).toContain('Never raise a budget, lower a\nthreshold, add a verdict retry or skip a case to get a pass.');
|
||||
expect(loop).toContain('report "observed k/n after fix at <cause>", never a proven rate');
|
||||
expect(loop).toContain('labels it `unmeasured` in the PR body. It never counts as a pass.');
|
||||
expect(loop).toContain('ask\n once and record the answer in the report');
|
||||
});
|
||||
|
||||
test('a failed free shard is measured before it is called pre-existing, with the retry off', () => {
|
||||
expect(tests).toContain('When a free-suite shard failed,\nfirst Read `~/.claude/skills/gstack/ship/sections/measure.md`');
|
||||
expect(tests.indexOf('sections/measure.md')).toBeLessThan(tests.indexOf('Then apply the Test\nFailure Ownership Triage'));
|
||||
const free = loop.slice(loop.indexOf('**Failed free-suite shard (Step 5).**'), loop.indexOf('**Red paid eval (Step 6).**'));
|
||||
for (const phrase of ['Before calling the failure pre-existing or\npushing', 'exact file list', 'rerun 1 alone as the baseline', 'the flaky retry off',
|
||||
'10-minute cap', 'ship_rerun_backend=ubicloud', 'rerun the whole failing\ncommand', 'completed at the cap']) expect(free).toContain(phrase);
|
||||
});
|
||||
|
||||
test('every ship-measure command and flag the skill names exists in the runner', () => {
|
||||
const source = fs.readFileSync(path.join(ROOT, 'scripts/ship-measure.ts'), 'utf8');
|
||||
const named = [...loop.matchAll(/ship-measure (\w+)([^`]*)`/g)];
|
||||
expect(named.length).toBeGreaterThan(5);
|
||||
for (const [, command, rest] of named) {
|
||||
expect(source).toMatch(new RegExp(`command [!=]== '${command}'`));
|
||||
for (const [flag] of rest!.matchAll(/--[a-z-]+/g)) expect(source).toContain(`'${flag}'`);
|
||||
}
|
||||
});
|
||||
});
|
||||
Reference in new issue
Block a user