Plan a trial of /ship without its separate conductor subagent (GST-3003) (#4456)

Adds the locked plan for a paid trial: does /ship cost less and catch as
much when the pull request's own thread runs it, instead of a separate
subagent?

Why: that subagent's own steps cost about $22 of a $35 /ship run, and
the keep-or-drop decision is written down here before anything is spent.

Nothing to do: no code changes; paid runs start only after this merges
and the budget is approved.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Review-Finding: p1/outside-voice#1 reader=outside-voice
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-17 17:39:04 +00:00
1 parent a9da30363c
commit 88da124dbc
2 files changed
+191

No files matched your search

+1
View File
@@ -0,0 +1 @@
- **A locked plan for a trial of /ship without its separate conductor subagent (GST-3003).** `docs/evals/ship-conductor-fold-prereg.md` fixes, before any paid run, the bars, arms (today's /ship, the folded /ship and a no-gstack review), cases, scoring and stop rules for testing whether a pull request's own thread can run /ship's phases and start its reviewers itself, at no extra cost and with nothing missed that today's /ship catches.
+190
View File
@@ -0,0 +1,190 @@
# Prereg: does /ship need its conductor subagent? (GST-3003)
frozen: 2026-11-17
Status: locked at merge, before any paid call. Owner: GStack Minder (GSTA-36); runner GSTA-589. Paid runs start only
after this file merges and the owner approves the budget below.
## Question
When a Capy thread ships a pull request today, it starts one extra subagent, the conductor, whose only job is to run
/ship's phases (`ship/DRIVER.md`) and start /ship's reviewers. A profile of 12 recent runs (gstack 5dcd76b) counted
the conductor's own steps at $21.62 of a $34.66 run, carrying about 231k tokens of context a step. Since #4451
(0ee59146), `gstack-ship-phase run|program` runs one DRIVER phase per call, the receipt counts `calls=`, and a run
prints `STOP: past 100 minutes`. So the pull request's own thread could run each phase itself and start the reviewers
itself, with no conductor in between.
Does that folded /ship cost no more than today's /ship, and does it miss nothing today's /ship catches?
## Bars
- lesson: whether a frontier model shipping a pull request needs a separate subagent to run its review procedure, or can run the procedure itself at no loss in what review catches
- cost per run, F minus C (thread plus all its subagents, cost.dollars): pass when the paired 95% interval's upper bound is at or below $0 or the interval contains $0; lower is better
- planted-bug catch rate, F minus C: pass when no case is a structural loss (below) and the paired 95% interval's upper bound is not below 0; higher is better
- real-problem catch rate, F minus C: the same bar as planted bugs, over the real problems found in the case (below); higher is better
- reader fidelity in F: every reviewer the run's printed review plan names is started and collected, in 30 of 30 F trials; higher is better
- wall time per run and `calls=` from the receipt: reported paired with 95% intervals, no bar (diagnostics)
A **structural loss** is a case where C catches the problem in all 3 of its trials and F in none of its 3. If the two
arms are equally good, the chance of seeing at least one structural loss somewhere in 10 cases is at most 14.6% (worst
case, at a per-trial catch rate of 0.5). A literal "F misses one thing C caught once" rule with one trial each would
fire 83% of the time at a shared catch rate of 0.8, so it would decide on noise, not on the fold.
## Decisions
- if F costs no more than C and misses nothing C catches (every bar above passes): the conductor goes. A follow-up pull request makes the folded procedure ship/DRIVER.md's default for Capy threads, and the crew rule that starts one /ship subagent per head (R104, CREW.md rule 4) is retired through GStack Minder and FOUN-32
- if both cost and catch rates are a tie inside the run-to-run variation (both 95% intervals contain 0, no structural loss, fidelity 30 of 30): the conductor goes, as above, because the fold removes one agent layer at no measured loss; the results say plainly that differences smaller than the stated mdd could not be seen
- if F costs clearly more (the cost interval's lower bound is above $0): the conductor stays, R104 stands, and the results name the phase where the fold's extra spend went
- if F misses something C catches (a structural loss, a catch-rate interval entirely below 0, or fidelity under 30 of 30): the conductor stays, and the fold's reviewer dispatch gets one FIX-METHOD cycle; any retry is a new preregistration
- if the no-gstack arm's planted-bug catch rate is not below the better /ship arm's (paired interval contains 0 or is above it) at a quarter of that arm's cost or less: /ship's review panel gets one R48 "Why null?" cycle under G3 (which reviewers earn their place), and Garry hears the result as a cut-candidate question; the conductor decision above is unchanged
- if the no-gstack arm catches less, or costs more than a quarter of the better /ship arm: nothing changes beyond the conductor decision; its row is reported beside the others
- if the run stops incomplete (spend cap, a harness stop, or a scorer or judge below its bar): no decision is taken, R104 stands, and every finished trial is reported
## Arms
- `claude-opus-5-5#ship-conductor` (C): today's /ship. A fresh Capy thread on Claude Opus 5.5 (reasoning high) starts one /ship subagent on a fresh machine, which runs ship/DRIVER.md through `gstack-ship-phase` and starts its reviewers. Planned at $38 a run ($34.66 counted mean plus about $3 for the thread itself)
- `claude-opus-5-5#ship-folded` (F): the same thread, brief and gstack revision, except that the thread runs ship/DRIVER.md itself, one `gstack-ship-phase` program per phase, and starts the reviewers as its own subagents. Planned at $38 a run (the same as C, so the budget does not assume the saving)
- `claude-opus-5-5#no-gstack` (N): the same model and case, no gstack, a plain review prompt (below). Scored on catches and cost only. Planned at $6 a run
## Briefs
Reviewer models are /ship's own in both C and F (the second-model reviewer from `voice-models.json`); only who runs
the procedure differs. GPT-6.1 Sol is not an arm: the decision concerns Capy threads, which run on Opus 5.5 (R38).
Briefs, frozen. Shared part: "You own branch `<branch>` on garrytan/gstack; its base is `<base>`. Install gstack for
ship, then run /ship at the branch head against `<base>`. Stop at `SHIP_RECEIPT:` or wherever /ship stops, skip
`gstack-ship-queue ready`, and never merge. Report the receipt line, the `CALLS:` line and the stop reason if any."
C adds: "Run /ship the usual way: one fresh /ship subagent (machine: fresh) runs it." F adds: "Run /ship yourself in
this thread: follow ship/DRIVER.md phase by phase with gstack-ship-phase, and start its reviewers as your own
subagents. Do not start a subagent to run /ship."
## Comparison
stack: loaded
C and F load gstack by design; the question is about gstack's /ship procedure.
## No-gstack arm
- arms: `claude-opus-5-5#no-gstack`
- inputs: the same 10 cases, the same branch and base, the same planted bug
- prompt: "Review this PR (branch `<branch>` against `<base>`, checked out here) for bugs that would fail a test, break production or lose data. List each as path:line and the cause." No SKILL.md, preamble, hooks, installed skills or routing lines
- panel: the same kind (`behavior`, 3 independent trials per case), the same catch rule and the same real-problem judge
- harness: the Claude Agent SDK with the `claude_code` preset, reasoning effort high, Bash, Read and Grep tools; cost from the SDK's reported total at list price. This differs from C and F (Capy threads), because a Capy thread cannot hide the owner's drives; its cost is comparable to theirs in size, not to the cent
- report: its row sits beside C and F in every results table and verdict, including when it wins
- masking: a container on a runner VM that sees no Capy drive, holding only the case checkout
- install: `test ! -e ~/.local/share/gstack && test ! -e ~/.claude/skills` inside the container before each run
- audit: each transcript is scanned for reads under `.capy/drive`, `.local/share/gstack` or `.claude/skills`; a trial that read a gstack skill file is contaminated and scores zero
## Cases
Ten cases, the same ten in every arm (fully paired, the strongest form of matching by size class). A separate
sealing subagent picks them after this file merges, by these rules:
1. **Host pull request.** A garrytan/gstack pull request merged on or after 2026-07-01 (after Opus 5.5's training
cutoff), whose revert applies cleanly to one frozen main commit, and whose reverted tree passes typecheck and the
host's own tests. Excluded: changes to docs, generated files, VERSION or CHANGELOG only, and changes to /ship or
review tooling itself (`ship/`, `lib/ship-*`, `bin/gstack-ship-*`, `bin/gstack-review-*`).
2. **Size class**, by changed lines without generated files and lockfiles: 3 small (under 150), 4 medium (150 to
600), 3 large (over 600). At least 3 of the 10 are hosts that `gstack-review-plan` sends to the full panel
(security, user data, money or merge-check code), so reviewer dispatch runs at its widest.
3. **Planted bug.** One hunk of at most 15 changed lines inside the host's own diff that re-introduces a bug a later
gstack commit fixed (a revert, or a fix whose description or regression test names the failure), adapted only as
far as needed to apply. Before sealing, each plant must (a) make that fix's regression test, or a test the sealer
writes from the fix's description, fail on the planted tree and pass without the plant, which proves it is a real
defect at the blocking floor; and (b) leave the host's existing tests passing. A plant the existing suite already
catches would be caught the same way by C and F, spending about $76 a trial pair to learn nothing about the
conductor, so the trial uses plants that a reviewer, or a test the run writes, has to find.
4. **Branches.** Per case, one base branch: frozen main with the host reverted and this file and its results file
removed, as one commit with a plain message. Per trial, one branch: the base plus the host's commit with the
plant folded in, under the host's original title. Pull requests target the case's base branch, never main, so
nothing a trial does can land on main; every trial pull request is closed unmerged and its branches deleted once
it is scored.
5. **Labels.** The sealer writes one private label file: per case the host and its merge commit, base commit, size
and review class, the plant (path, labelled line, the hunk, the source fix, 2 to 4 cause words), and the random
seed that orders the cases. Its SHA-256 is posted on GST-3003 before the first trial starts, and the results
print it. The file never enters the repository.
## Procedure
Trials run in waves. Each wave starts C and F on the same case and trial at the same time, each in a new thread with
no earlier context, so time of day and CI load hit both alike; N runs separately, since it shares nothing with them.
Wave 1 is the pilot: trial 1 of the first 3 cases in all three arms. It counts toward the run unless a stop rule
fires. The order of cases follows the sealed seed.
Measured per trial: cost.dollars of the arm's thread and all its subagents at its final report; wall time from the
brief to the final report; `calls=` from `SHIP_RECEIPT:` and the `CALLS:` line; whether `STOP: past 100 minutes`
printed; for F, the reviewers the printed plan named against those started and collected.
## Scoring
judge: gpt-6-astra
labels: 48
agreement: 0.6
paired: yes
interval: 95%
mdd: about $7 a run on cost (20% of C's $35; estimated from the profile's 121 to 308 conductor steps, so a per-run spread near $8, and 3 trials a case), and about 35 percentage points on catch rate; the pilot's measured spread is printed beside these and the bars do not move
**Catches, by rule (no judge).** A trial catches its planted bug when either (a) a finding in the run's findings
record (`merged.jsonl`) or final report names the labelled path, a line within 3 of the labelled line, and at least
one cause word; or (b) the sealed regression test for the plant (from the label file, run by the runner, never shown to
the arm) passes at the run's final head, and one of the run's findings or commit messages names that file. A case's catch rate in an arm is its catches over 3 trials. The script that applies this
rule is checked against the decoys below before any trial runs.
**Real problems, by judge.** Every finding that is not the plant and that the run marked blocking, or that led to a
fix, goes to GPT-6 Astra with the case diff. It answers two questions: does this meet the blocking floor (would fail
a test, break production, or lose data or secrets), and which other findings name the same problem. A case's real
problems are the union of judged-real problems across all arms and trials; each arm's catch rate on them uses the
rule above, with the judge's grouping in place of the planted label. The judge sees findings as A, B and C, never an
arm name, model or cost. It is checked first against 48 hand-checked labels: findings from merged gstack pull
requests outside the case set, 24 that led to a fix and 24 dismissed below the floor, each read against its diff by
the runner and frozen with a SHA-256 before the judge sees any. `gstack-eval-plan judge --check` must print
`JUDGE_AGREEMENT: ok` (Cohen's kappa at least 0.6) before its numbers count.
**Comparisons.** Every comparison is paired by case: the per-case difference of means (cost, wall time, `calls=`)
or of catch rates, with a 95% t interval over the 10 cases. Every trial is reported; nothing is averaged away.
## Decoys
Before any trial, the catch script scores 50 synthetic finding lists built from the sealed labels, 5 per case: the
right file with the line 4 or more away; the right line with no cause word; the cause words on another file; a final
head with the planted lines reworded but the bug kept (the sealed regression test still fails); and one exact match. It must score the 40 decoys 0 and the 10
matches 1. A decoy pass proves the rule rejects near misses and does not credit a cosmetic edit as a fix.
## Blinding
- Arms see a real project: the garrytan/gstack checkout, plain branch names taken from each host's own branch, and
the host's own commit title. No branch, file, commit message, prompt or label the trial controls carries an eval word.
- This file and its results file are removed from every case base, and the label file never enters the repository.
The installed gstack copy (`~/.local/share/gstack`) still holds this file, so every C and F transcript is scanned
for reads of it or of the label file's path, and a trial that read either scores zero.
- Each arm's checkout keeps git history, where the original fix commit lives. Lookups of the planted path's history
(`git log -S`, `git log -p`, `git blame` on that file) are counted per trial, and catches made after one are
reported separately as well as in the totals.
- The judge sees A, B and C, never an arm, model or price.
## Stop rules
- Spend cap: $3,250 cost.dollars for everything in the budget table (the approved figure replaces it if lower). `budget_exhausted` stops the run with every finished trial reported, and the run does not resume.
- Per trial: $150 or 4 hours. A trial stopped there is recorded as incomplete, with its cost at the stop counted (a lower bound, flagged), and scored on what it caught by then.
- Pilot: if any pilot C or F trial ends without a receipt for a harness reason (machine, install, CI wiring), the run stops; the fix is a dated change to this file before the first counted trial, or a new preregistration if a bar, arm or decision changes.
- The catch script below 50 of 50 on decoys, or the judge below kappa 0.6: stop before any trial; fix the script or the judge prompt and relabel; never lower a bar.
- Projection: if the pilot's projection for all 90 trials plus the fixed costs exceeds the approved budget, stop and ask again with the new figure.
- No retries of a counted trial, with the one exception AGENTS.md allows: a red case may rerun once at the same head as a new panel, reported as flaky. Harness or infrastructure failures never rerun.
## Budget (all-in estimate, cost.dollars)
| Item | Count | Each | Total |
|---|---|---|---|
| C trials | 30 | $38 | $1,140 |
| F trials | 30 | $38 | $1,140 |
| N trials | 30 | $6 | $180 |
| Sealing: picking hosts and plants, proving each plant | 1 | $60 | $60 |
| Judge: 48 labels plus about 300 findings on GPT-6 Astra | 1 | $80 | $80 |
| Runner: waves, audits, scoring, results write-up | 1 | $150 | $150 |
| /ship of the results pull request | 1 | $35 | $35 |
| **Estimate** | | | **$2,785** |
| Cap (estimate plus about 15%) | | | $3,250 |
That sits under R71's $4,000 guideline for an eval with a GPT-6 Astra judge. If F saves what the profile predicts
(about $10 a run), the actual total is near $2,500. Machines, CI minutes and the runner VM are not in cost.dollars and
are reported separately. Each case gets 3 trials because AGENTS.md gives a `behavior` case a panel of 3, and because
one trial each would let noise decide the "misses nothing" bar (see Bars).
## Held-out exposure
| set | prior exposure (which runs, which models) | reservation (what this run may read) |
|---|---|---|
| the 10 sealed cases (hosts, plants, labels) | none: picked after this file merges; hosts and fixes are public on GitHub, all merged after Opus 5.5's June 2026 training cutoff | arms read only their case checkout; the runner reads labels to score; the judge reads findings and the case diff, never the labels |
| the 48 judge labels | findings already read by /ship reviewers on their own pull requests, none in the case set | the judge reads them once for `judge --check` |
## After the run
Results go to `docs/evals/ship-conductor-fold-results.md`, opening with two or three plain takeaways, with one table
holding a filled row for C, F and N, checked with `gstack-eval-plan results --check` before any number leaves the
run. Counts that may be zero (structural losses, contaminated trials, history lookups) are reported in prose, outside
the tables. A null gets one R48 "Why null?" cycle and a KEEP, FIX-METHOD or CUT-CANDIDATE verdict.