From 88da124dbc68ad886f4aef467d4b09375041ad86 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 18 Nov 2026 01:39:04 +0800 Subject: [PATCH] Plan a trial of /ship without its separate conductor subagent (GST-3003) (#4456) Adds the locked plan for a paid trial: does /ship cost less and catch as much when the pull request's own thread runs it, instead of a separate subagent? Why: that subagent's own steps cost about $22 of a $35 /ship run, and the keep-or-drop decision is written down here before anything is spent. Nothing to do: no code changes; paid runs start only after this merges and the budget is approved. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Review-Finding: p1/outside-voice#1 reader=outside-voice --- changelog.d/capy-plan-a-trial-of.md | 1 + docs/evals/ship-conductor-fold-prereg.md | 190 +++++++++++++++++++++++ 2 files changed, 191 insertions(+) create mode 100644 changelog.d/capy-plan-a-trial-of.md create mode 100644 docs/evals/ship-conductor-fold-prereg.md diff --git a/changelog.d/capy-plan-a-trial-of.md b/changelog.d/capy-plan-a-trial-of.md new file mode 100644 index 000000000..0f4dad300 --- /dev/null +++ b/changelog.d/capy-plan-a-trial-of.md @@ -0,0 +1 @@ +- **A locked plan for a trial of /ship without its separate conductor subagent (GST-3003).** `docs/evals/ship-conductor-fold-prereg.md` fixes, before any paid run, the bars, arms (today's /ship, the folded /ship and a no-gstack review), cases, scoring and stop rules for testing whether a pull request's own thread can run /ship's phases and start its reviewers itself, at no extra cost and with nothing missed that today's /ship catches. diff --git a/docs/evals/ship-conductor-fold-prereg.md b/docs/evals/ship-conductor-fold-prereg.md new file mode 100644 index 000000000..ad5479b52 --- /dev/null +++ b/docs/evals/ship-conductor-fold-prereg.md @@ -0,0 +1,190 @@ +# Prereg: does /ship need its conductor subagent? (GST-3003) + +frozen: 2026-11-17 + +Status: locked at merge, before any paid call. Owner: GStack Minder (GSTA-36); runner GSTA-589. Paid runs start only +after this file merges and the owner approves the budget below. + +## Question +When a Capy thread ships a pull request today, it starts one extra subagent, the conductor, whose only job is to run +/ship's phases (`ship/DRIVER.md`) and start /ship's reviewers. A profile of 12 recent runs (gstack 5dcd76b) counted +the conductor's own steps at $21.62 of a $34.66 run, carrying about 231k tokens of context a step. Since #4451 +(0ee59146), `gstack-ship-phase run|program` runs one DRIVER phase per call, the receipt counts `calls=`, and a run +prints `STOP: past 100 minutes`. So the pull request's own thread could run each phase itself and start the reviewers +itself, with no conductor in between. + +Does that folded /ship cost no more than today's /ship, and does it miss nothing today's /ship catches? + +## Bars +- lesson: whether a frontier model shipping a pull request needs a separate subagent to run its review procedure, or can run the procedure itself at no loss in what review catches +- cost per run, F minus C (thread plus all its subagents, cost.dollars): pass when the paired 95% interval's upper bound is at or below $0 or the interval contains $0; lower is better +- planted-bug catch rate, F minus C: pass when no case is a structural loss (below) and the paired 95% interval's upper bound is not below 0; higher is better +- real-problem catch rate, F minus C: the same bar as planted bugs, over the real problems found in the case (below); higher is better +- reader fidelity in F: every reviewer the run's printed review plan names is started and collected, in 30 of 30 F trials; higher is better +- wall time per run and `calls=` from the receipt: reported paired with 95% intervals, no bar (diagnostics) + +A **structural loss** is a case where C catches the problem in all 3 of its trials and F in none of its 3. If the two +arms are equally good, the chance of seeing at least one structural loss somewhere in 10 cases is at most 14.6% (worst +case, at a per-trial catch rate of 0.5). A literal "F misses one thing C caught once" rule with one trial each would +fire 83% of the time at a shared catch rate of 0.8, so it would decide on noise, not on the fold. + +## Decisions +- if F costs no more than C and misses nothing C catches (every bar above passes): the conductor goes. A follow-up pull request makes the folded procedure ship/DRIVER.md's default for Capy threads, and the crew rule that starts one /ship subagent per head (R104, CREW.md rule 4) is retired through GStack Minder and FOUN-32 +- if both cost and catch rates are a tie inside the run-to-run variation (both 95% intervals contain 0, no structural loss, fidelity 30 of 30): the conductor goes, as above, because the fold removes one agent layer at no measured loss; the results say plainly that differences smaller than the stated mdd could not be seen +- if F costs clearly more (the cost interval's lower bound is above $0): the conductor stays, R104 stands, and the results name the phase where the fold's extra spend went +- if F misses something C catches (a structural loss, a catch-rate interval entirely below 0, or fidelity under 30 of 30): the conductor stays, and the fold's reviewer dispatch gets one FIX-METHOD cycle; any retry is a new preregistration +- if the no-gstack arm's planted-bug catch rate is not below the better /ship arm's (paired interval contains 0 or is above it) at a quarter of that arm's cost or less: /ship's review panel gets one R48 "Why null?" cycle under G3 (which reviewers earn their place), and Garry hears the result as a cut-candidate question; the conductor decision above is unchanged +- if the no-gstack arm catches less, or costs more than a quarter of the better /ship arm: nothing changes beyond the conductor decision; its row is reported beside the others +- if the run stops incomplete (spend cap, a harness stop, or a scorer or judge below its bar): no decision is taken, R104 stands, and every finished trial is reported + +## Arms +- `claude-opus-5-5#ship-conductor` (C): today's /ship. A fresh Capy thread on Claude Opus 5.5 (reasoning high) starts one /ship subagent on a fresh machine, which runs ship/DRIVER.md through `gstack-ship-phase` and starts its reviewers. Planned at $38 a run ($34.66 counted mean plus about $3 for the thread itself) +- `claude-opus-5-5#ship-folded` (F): the same thread, brief and gstack revision, except that the thread runs ship/DRIVER.md itself, one `gstack-ship-phase` program per phase, and starts the reviewers as its own subagents. Planned at $38 a run (the same as C, so the budget does not assume the saving) +- `claude-opus-5-5#no-gstack` (N): the same model and case, no gstack, a plain review prompt (below). Scored on catches and cost only. Planned at $6 a run + +## Briefs +Reviewer models are /ship's own in both C and F (the second-model reviewer from `voice-models.json`); only who runs +the procedure differs. GPT-6.1 Sol is not an arm: the decision concerns Capy threads, which run on Opus 5.5 (R38). + +Briefs, frozen. Shared part: "You own branch `` on garrytan/gstack; its base is ``. Install gstack for +ship, then run /ship at the branch head against ``. Stop at `SHIP_RECEIPT:` or wherever /ship stops, skip +`gstack-ship-queue ready`, and never merge. Report the receipt line, the `CALLS:` line and the stop reason if any." +C adds: "Run /ship the usual way: one fresh /ship subagent (machine: fresh) runs it." F adds: "Run /ship yourself in +this thread: follow ship/DRIVER.md phase by phase with gstack-ship-phase, and start its reviewers as your own +subagents. Do not start a subagent to run /ship." + +## Comparison +stack: loaded + +C and F load gstack by design; the question is about gstack's /ship procedure. + +## No-gstack arm +- arms: `claude-opus-5-5#no-gstack` +- inputs: the same 10 cases, the same branch and base, the same planted bug +- prompt: "Review this PR (branch `` against ``, checked out here) for bugs that would fail a test, break production or lose data. List each as path:line and the cause." No SKILL.md, preamble, hooks, installed skills or routing lines +- panel: the same kind (`behavior`, 3 independent trials per case), the same catch rule and the same real-problem judge +- harness: the Claude Agent SDK with the `claude_code` preset, reasoning effort high, Bash, Read and Grep tools; cost from the SDK's reported total at list price. This differs from C and F (Capy threads), because a Capy thread cannot hide the owner's drives; its cost is comparable to theirs in size, not to the cent +- report: its row sits beside C and F in every results table and verdict, including when it wins +- masking: a container on a runner VM that sees no Capy drive, holding only the case checkout +- install: `test ! -e ~/.local/share/gstack && test ! -e ~/.claude/skills` inside the container before each run +- audit: each transcript is scanned for reads under `.capy/drive`, `.local/share/gstack` or `.claude/skills`; a trial that read a gstack skill file is contaminated and scores zero + +## Cases +Ten cases, the same ten in every arm (fully paired, the strongest form of matching by size class). A separate +sealing subagent picks them after this file merges, by these rules: + +1. **Host pull request.** A garrytan/gstack pull request merged on or after 2026-07-01 (after Opus 5.5's training + cutoff), whose revert applies cleanly to one frozen main commit, and whose reverted tree passes typecheck and the + host's own tests. Excluded: changes to docs, generated files, VERSION or CHANGELOG only, and changes to /ship or + review tooling itself (`ship/`, `lib/ship-*`, `bin/gstack-ship-*`, `bin/gstack-review-*`). +2. **Size class**, by changed lines without generated files and lockfiles: 3 small (under 150), 4 medium (150 to + 600), 3 large (over 600). At least 3 of the 10 are hosts that `gstack-review-plan` sends to the full panel + (security, user data, money or merge-check code), so reviewer dispatch runs at its widest. +3. **Planted bug.** One hunk of at most 15 changed lines inside the host's own diff that re-introduces a bug a later + gstack commit fixed (a revert, or a fix whose description or regression test names the failure), adapted only as + far as needed to apply. Before sealing, each plant must (a) make that fix's regression test, or a test the sealer + writes from the fix's description, fail on the planted tree and pass without the plant, which proves it is a real + defect at the blocking floor; and (b) leave the host's existing tests passing. A plant the existing suite already + catches would be caught the same way by C and F, spending about $76 a trial pair to learn nothing about the + conductor, so the trial uses plants that a reviewer, or a test the run writes, has to find. +4. **Branches.** Per case, one base branch: frozen main with the host reverted and this file and its results file + removed, as one commit with a plain message. Per trial, one branch: the base plus the host's commit with the + plant folded in, under the host's original title. Pull requests target the case's base branch, never main, so + nothing a trial does can land on main; every trial pull request is closed unmerged and its branches deleted once + it is scored. +5. **Labels.** The sealer writes one private label file: per case the host and its merge commit, base commit, size + and review class, the plant (path, labelled line, the hunk, the source fix, 2 to 4 cause words), and the random + seed that orders the cases. Its SHA-256 is posted on GST-3003 before the first trial starts, and the results + print it. The file never enters the repository. + +## Procedure +Trials run in waves. Each wave starts C and F on the same case and trial at the same time, each in a new thread with +no earlier context, so time of day and CI load hit both alike; N runs separately, since it shares nothing with them. +Wave 1 is the pilot: trial 1 of the first 3 cases in all three arms. It counts toward the run unless a stop rule +fires. The order of cases follows the sealed seed. + +Measured per trial: cost.dollars of the arm's thread and all its subagents at its final report; wall time from the +brief to the final report; `calls=` from `SHIP_RECEIPT:` and the `CALLS:` line; whether `STOP: past 100 minutes` +printed; for F, the reviewers the printed plan named against those started and collected. + +## Scoring +judge: gpt-6-astra +labels: 48 +agreement: 0.6 +paired: yes +interval: 95% +mdd: about $7 a run on cost (20% of C's $35; estimated from the profile's 121 to 308 conductor steps, so a per-run spread near $8, and 3 trials a case), and about 35 percentage points on catch rate; the pilot's measured spread is printed beside these and the bars do not move + +**Catches, by rule (no judge).** A trial catches its planted bug when either (a) a finding in the run's findings +record (`merged.jsonl`) or final report names the labelled path, a line within 3 of the labelled line, and at least +one cause word; or (b) the sealed regression test for the plant (from the label file, run by the runner, never shown to +the arm) passes at the run's final head, and one of the run's findings or commit messages names that file. A case's catch rate in an arm is its catches over 3 trials. The script that applies this +rule is checked against the decoys below before any trial runs. + +**Real problems, by judge.** Every finding that is not the plant and that the run marked blocking, or that led to a +fix, goes to GPT-6 Astra with the case diff. It answers two questions: does this meet the blocking floor (would fail +a test, break production, or lose data or secrets), and which other findings name the same problem. A case's real +problems are the union of judged-real problems across all arms and trials; each arm's catch rate on them uses the +rule above, with the judge's grouping in place of the planted label. The judge sees findings as A, B and C, never an +arm name, model or cost. It is checked first against 48 hand-checked labels: findings from merged gstack pull +requests outside the case set, 24 that led to a fix and 24 dismissed below the floor, each read against its diff by +the runner and frozen with a SHA-256 before the judge sees any. `gstack-eval-plan judge --check` must print +`JUDGE_AGREEMENT: ok` (Cohen's kappa at least 0.6) before its numbers count. + +**Comparisons.** Every comparison is paired by case: the per-case difference of means (cost, wall time, `calls=`) +or of catch rates, with a 95% t interval over the 10 cases. Every trial is reported; nothing is averaged away. + +## Decoys +Before any trial, the catch script scores 50 synthetic finding lists built from the sealed labels, 5 per case: the +right file with the line 4 or more away; the right line with no cause word; the cause words on another file; a final +head with the planted lines reworded but the bug kept (the sealed regression test still fails); and one exact match. It must score the 40 decoys 0 and the 10 +matches 1. A decoy pass proves the rule rejects near misses and does not credit a cosmetic edit as a fix. + +## Blinding +- Arms see a real project: the garrytan/gstack checkout, plain branch names taken from each host's own branch, and + the host's own commit title. No branch, file, commit message, prompt or label the trial controls carries an eval word. +- This file and its results file are removed from every case base, and the label file never enters the repository. + The installed gstack copy (`~/.local/share/gstack`) still holds this file, so every C and F transcript is scanned + for reads of it or of the label file's path, and a trial that read either scores zero. +- Each arm's checkout keeps git history, where the original fix commit lives. Lookups of the planted path's history + (`git log -S`, `git log -p`, `git blame` on that file) are counted per trial, and catches made after one are + reported separately as well as in the totals. +- The judge sees A, B and C, never an arm, model or price. + +## Stop rules +- Spend cap: $3,250 cost.dollars for everything in the budget table (the approved figure replaces it if lower). `budget_exhausted` stops the run with every finished trial reported, and the run does not resume. +- Per trial: $150 or 4 hours. A trial stopped there is recorded as incomplete, with its cost at the stop counted (a lower bound, flagged), and scored on what it caught by then. +- Pilot: if any pilot C or F trial ends without a receipt for a harness reason (machine, install, CI wiring), the run stops; the fix is a dated change to this file before the first counted trial, or a new preregistration if a bar, arm or decision changes. +- The catch script below 50 of 50 on decoys, or the judge below kappa 0.6: stop before any trial; fix the script or the judge prompt and relabel; never lower a bar. +- Projection: if the pilot's projection for all 90 trials plus the fixed costs exceeds the approved budget, stop and ask again with the new figure. +- No retries of a counted trial, with the one exception AGENTS.md allows: a red case may rerun once at the same head as a new panel, reported as flaky. Harness or infrastructure failures never rerun. + +## Budget (all-in estimate, cost.dollars) +| Item | Count | Each | Total | +|---|---|---|---| +| C trials | 30 | $38 | $1,140 | +| F trials | 30 | $38 | $1,140 | +| N trials | 30 | $6 | $180 | +| Sealing: picking hosts and plants, proving each plant | 1 | $60 | $60 | +| Judge: 48 labels plus about 300 findings on GPT-6 Astra | 1 | $80 | $80 | +| Runner: waves, audits, scoring, results write-up | 1 | $150 | $150 | +| /ship of the results pull request | 1 | $35 | $35 | +| **Estimate** | | | **$2,785** | +| Cap (estimate plus about 15%) | | | $3,250 | + +That sits under R71's $4,000 guideline for an eval with a GPT-6 Astra judge. If F saves what the profile predicts +(about $10 a run), the actual total is near $2,500. Machines, CI minutes and the runner VM are not in cost.dollars and +are reported separately. Each case gets 3 trials because AGENTS.md gives a `behavior` case a panel of 3, and because +one trial each would let noise decide the "misses nothing" bar (see Bars). + +## Held-out exposure +| set | prior exposure (which runs, which models) | reservation (what this run may read) | +|---|---|---| +| the 10 sealed cases (hosts, plants, labels) | none: picked after this file merges; hosts and fixes are public on GitHub, all merged after Opus 5.5's June 2026 training cutoff | arms read only their case checkout; the runner reads labels to score; the judge reads findings and the case diff, never the labels | +| the 48 judge labels | findings already read by /ship reviewers on their own pull requests, none in the case set | the judge reads them once for `judge --check` | + +## After the run +Results go to `docs/evals/ship-conductor-fold-results.md`, opening with two or three plain takeaways, with one table +holding a filled row for C, F and N, checked with `gstack-eval-plan results --check` before any number leaves the +run. Counts that may be zero (structural losses, contaminated trials, history lookups) are reported in prose, outside +the tables. A null gets one R48 "Why null?" cycle and a KEEP, FIX-METHOD or CUT-CANDIDATE verdict.