mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-18 06:04:33 +01:00
Plan a trial of /ship without its separate conductor subagent (GST-3003) (#4456)
Adds the locked plan for a paid trial: does /ship cost less and catch as much when the pull request's own thread runs it, instead of a separate subagent? Why: that subagent's own steps cost about $22 of a $35 /ship run, and the keep-or-drop decision is written down here before anything is spent. Nothing to do: no code changes; paid runs start only after this merges and the budget is approved. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> Review-Finding: p1/outside-voice#1 reader=outside-voice
This commit is contained in:
1 parent
a9da30363c
commit
88da124dbc
2 files changed
+191
No files matched your search
@@ -0,0 +1 @@
|
||||
- **A locked plan for a trial of /ship without its separate conductor subagent (GST-3003).** `docs/evals/ship-conductor-fold-prereg.md` fixes, before any paid run, the bars, arms (today's /ship, the folded /ship and a no-gstack review), cases, scoring and stop rules for testing whether a pull request's own thread can run /ship's phases and start its reviewers itself, at no extra cost and with nothing missed that today's /ship catches.
|
||||
@@ -0,0 +1,190 @@
|
||||
# Prereg: does /ship need its conductor subagent? (GST-3003)
|
||||
|
||||
frozen: 2026-11-17
|
||||
|
||||
Status: locked at merge, before any paid call. Owner: GStack Minder (GSTA-36); runner GSTA-589. Paid runs start only
|
||||
after this file merges and the owner approves the budget below.
|
||||
|
||||
## Question
|
||||
When a Capy thread ships a pull request today, it starts one extra subagent, the conductor, whose only job is to run
|
||||
/ship's phases (`ship/DRIVER.md`) and start /ship's reviewers. A profile of 12 recent runs (gstack 5dcd76b) counted
|
||||
the conductor's own steps at $21.62 of a $34.66 run, carrying about 231k tokens of context a step. Since #4451
|
||||
(0ee59146), `gstack-ship-phase run|program` runs one DRIVER phase per call, the receipt counts `calls=`, and a run
|
||||
prints `STOP: past 100 minutes`. So the pull request's own thread could run each phase itself and start the reviewers
|
||||
itself, with no conductor in between.
|
||||
|
||||
Does that folded /ship cost no more than today's /ship, and does it miss nothing today's /ship catches?
|
||||
|
||||
## Bars
|
||||
- lesson: whether a frontier model shipping a pull request needs a separate subagent to run its review procedure, or can run the procedure itself at no loss in what review catches
|
||||
- cost per run, F minus C (thread plus all its subagents, cost.dollars): pass when the paired 95% interval's upper bound is at or below $0 or the interval contains $0; lower is better
|
||||
- planted-bug catch rate, F minus C: pass when no case is a structural loss (below) and the paired 95% interval's upper bound is not below 0; higher is better
|
||||
- real-problem catch rate, F minus C: the same bar as planted bugs, over the real problems found in the case (below); higher is better
|
||||
- reader fidelity in F: every reviewer the run's printed review plan names is started and collected, in 30 of 30 F trials; higher is better
|
||||
- wall time per run and `calls=` from the receipt: reported paired with 95% intervals, no bar (diagnostics)
|
||||
|
||||
A **structural loss** is a case where C catches the problem in all 3 of its trials and F in none of its 3. If the two
|
||||
arms are equally good, the chance of seeing at least one structural loss somewhere in 10 cases is at most 14.6% (worst
|
||||
case, at a per-trial catch rate of 0.5). A literal "F misses one thing C caught once" rule with one trial each would
|
||||
fire 83% of the time at a shared catch rate of 0.8, so it would decide on noise, not on the fold.
|
||||
|
||||
## Decisions
|
||||
- if F costs no more than C and misses nothing C catches (every bar above passes): the conductor goes. A follow-up pull request makes the folded procedure ship/DRIVER.md's default for Capy threads, and the crew rule that starts one /ship subagent per head (R104, CREW.md rule 4) is retired through GStack Minder and FOUN-32
|
||||
- if both cost and catch rates are a tie inside the run-to-run variation (both 95% intervals contain 0, no structural loss, fidelity 30 of 30): the conductor goes, as above, because the fold removes one agent layer at no measured loss; the results say plainly that differences smaller than the stated mdd could not be seen
|
||||
- if F costs clearly more (the cost interval's lower bound is above $0): the conductor stays, R104 stands, and the results name the phase where the fold's extra spend went
|
||||
- if F misses something C catches (a structural loss, a catch-rate interval entirely below 0, or fidelity under 30 of 30): the conductor stays, and the fold's reviewer dispatch gets one FIX-METHOD cycle; any retry is a new preregistration
|
||||
- if the no-gstack arm's planted-bug catch rate is not below the better /ship arm's (paired interval contains 0 or is above it) at a quarter of that arm's cost or less: /ship's review panel gets one R48 "Why null?" cycle under G3 (which reviewers earn their place), and Garry hears the result as a cut-candidate question; the conductor decision above is unchanged
|
||||
- if the no-gstack arm catches less, or costs more than a quarter of the better /ship arm: nothing changes beyond the conductor decision; its row is reported beside the others
|
||||
- if the run stops incomplete (spend cap, a harness stop, or a scorer or judge below its bar): no decision is taken, R104 stands, and every finished trial is reported
|
||||
|
||||
## Arms
|
||||
- `claude-opus-5-5#ship-conductor` (C): today's /ship. A fresh Capy thread on Claude Opus 5.5 (reasoning high) starts one /ship subagent on a fresh machine, which runs ship/DRIVER.md through `gstack-ship-phase` and starts its reviewers. Planned at $38 a run ($34.66 counted mean plus about $3 for the thread itself)
|
||||
- `claude-opus-5-5#ship-folded` (F): the same thread, brief and gstack revision, except that the thread runs ship/DRIVER.md itself, one `gstack-ship-phase` program per phase, and starts the reviewers as its own subagents. Planned at $38 a run (the same as C, so the budget does not assume the saving)
|
||||
- `claude-opus-5-5#no-gstack` (N): the same model and case, no gstack, a plain review prompt (below). Scored on catches and cost only. Planned at $6 a run
|
||||
|
||||
## Briefs
|
||||
Reviewer models are /ship's own in both C and F (the second-model reviewer from `voice-models.json`); only who runs
|
||||
the procedure differs. GPT-6.1 Sol is not an arm: the decision concerns Capy threads, which run on Opus 5.5 (R38).
|
||||
|
||||
Briefs, frozen. Shared part: "You own branch `<branch>` on garrytan/gstack; its base is `<base>`. Install gstack for
|
||||
ship, then run /ship at the branch head against `<base>`. Stop at `SHIP_RECEIPT:` or wherever /ship stops, skip
|
||||
`gstack-ship-queue ready`, and never merge. Report the receipt line, the `CALLS:` line and the stop reason if any."
|
||||
C adds: "Run /ship the usual way: one fresh /ship subagent (machine: fresh) runs it." F adds: "Run /ship yourself in
|
||||
this thread: follow ship/DRIVER.md phase by phase with gstack-ship-phase, and start its reviewers as your own
|
||||
subagents. Do not start a subagent to run /ship."
|
||||
|
||||
## Comparison
|
||||
stack: loaded
|
||||
|
||||
C and F load gstack by design; the question is about gstack's /ship procedure.
|
||||
|
||||
## No-gstack arm
|
||||
- arms: `claude-opus-5-5#no-gstack`
|
||||
- inputs: the same 10 cases, the same branch and base, the same planted bug
|
||||
- prompt: "Review this PR (branch `<branch>` against `<base>`, checked out here) for bugs that would fail a test, break production or lose data. List each as path:line and the cause." No SKILL.md, preamble, hooks, installed skills or routing lines
|
||||
- panel: the same kind (`behavior`, 3 independent trials per case), the same catch rule and the same real-problem judge
|
||||
- harness: the Claude Agent SDK with the `claude_code` preset, reasoning effort high, Bash, Read and Grep tools; cost from the SDK's reported total at list price. This differs from C and F (Capy threads), because a Capy thread cannot hide the owner's drives; its cost is comparable to theirs in size, not to the cent
|
||||
- report: its row sits beside C and F in every results table and verdict, including when it wins
|
||||
- masking: a container on a runner VM that sees no Capy drive, holding only the case checkout
|
||||
- install: `test ! -e ~/.local/share/gstack && test ! -e ~/.claude/skills` inside the container before each run
|
||||
- audit: each transcript is scanned for reads under `.capy/drive`, `.local/share/gstack` or `.claude/skills`; a trial that read a gstack skill file is contaminated and scores zero
|
||||
|
||||
## Cases
|
||||
Ten cases, the same ten in every arm (fully paired, the strongest form of matching by size class). A separate
|
||||
sealing subagent picks them after this file merges, by these rules:
|
||||
|
||||
1. **Host pull request.** A garrytan/gstack pull request merged on or after 2026-07-01 (after Opus 5.5's training
|
||||
cutoff), whose revert applies cleanly to one frozen main commit, and whose reverted tree passes typecheck and the
|
||||
host's own tests. Excluded: changes to docs, generated files, VERSION or CHANGELOG only, and changes to /ship or
|
||||
review tooling itself (`ship/`, `lib/ship-*`, `bin/gstack-ship-*`, `bin/gstack-review-*`).
|
||||
2. **Size class**, by changed lines without generated files and lockfiles: 3 small (under 150), 4 medium (150 to
|
||||
600), 3 large (over 600). At least 3 of the 10 are hosts that `gstack-review-plan` sends to the full panel
|
||||
(security, user data, money or merge-check code), so reviewer dispatch runs at its widest.
|
||||
3. **Planted bug.** One hunk of at most 15 changed lines inside the host's own diff that re-introduces a bug a later
|
||||
gstack commit fixed (a revert, or a fix whose description or regression test names the failure), adapted only as
|
||||
far as needed to apply. Before sealing, each plant must (a) make that fix's regression test, or a test the sealer
|
||||
writes from the fix's description, fail on the planted tree and pass without the plant, which proves it is a real
|
||||
defect at the blocking floor; and (b) leave the host's existing tests passing. A plant the existing suite already
|
||||
catches would be caught the same way by C and F, spending about $76 a trial pair to learn nothing about the
|
||||
conductor, so the trial uses plants that a reviewer, or a test the run writes, has to find.
|
||||
4. **Branches.** Per case, one base branch: frozen main with the host reverted and this file and its results file
|
||||
removed, as one commit with a plain message. Per trial, one branch: the base plus the host's commit with the
|
||||
plant folded in, under the host's original title. Pull requests target the case's base branch, never main, so
|
||||
nothing a trial does can land on main; every trial pull request is closed unmerged and its branches deleted once
|
||||
it is scored.
|
||||
5. **Labels.** The sealer writes one private label file: per case the host and its merge commit, base commit, size
|
||||
and review class, the plant (path, labelled line, the hunk, the source fix, 2 to 4 cause words), and the random
|
||||
seed that orders the cases. Its SHA-256 is posted on GST-3003 before the first trial starts, and the results
|
||||
print it. The file never enters the repository.
|
||||
|
||||
## Procedure
|
||||
Trials run in waves. Each wave starts C and F on the same case and trial at the same time, each in a new thread with
|
||||
no earlier context, so time of day and CI load hit both alike; N runs separately, since it shares nothing with them.
|
||||
Wave 1 is the pilot: trial 1 of the first 3 cases in all three arms. It counts toward the run unless a stop rule
|
||||
fires. The order of cases follows the sealed seed.
|
||||
|
||||
Measured per trial: cost.dollars of the arm's thread and all its subagents at its final report; wall time from the
|
||||
brief to the final report; `calls=` from `SHIP_RECEIPT:` and the `CALLS:` line; whether `STOP: past 100 minutes`
|
||||
printed; for F, the reviewers the printed plan named against those started and collected.
|
||||
|
||||
## Scoring
|
||||
judge: gpt-6-astra
|
||||
labels: 48
|
||||
agreement: 0.6
|
||||
paired: yes
|
||||
interval: 95%
|
||||
mdd: about $7 a run on cost (20% of C's $35; estimated from the profile's 121 to 308 conductor steps, so a per-run spread near $8, and 3 trials a case), and about 35 percentage points on catch rate; the pilot's measured spread is printed beside these and the bars do not move
|
||||
|
||||
**Catches, by rule (no judge).** A trial catches its planted bug when either (a) a finding in the run's findings
|
||||
record (`merged.jsonl`) or final report names the labelled path, a line within 3 of the labelled line, and at least
|
||||
one cause word; or (b) the sealed regression test for the plant (from the label file, run by the runner, never shown to
|
||||
the arm) passes at the run's final head, and one of the run's findings or commit messages names that file. A case's catch rate in an arm is its catches over 3 trials. The script that applies this
|
||||
rule is checked against the decoys below before any trial runs.
|
||||
|
||||
**Real problems, by judge.** Every finding that is not the plant and that the run marked blocking, or that led to a
|
||||
fix, goes to GPT-6 Astra with the case diff. It answers two questions: does this meet the blocking floor (would fail
|
||||
a test, break production, or lose data or secrets), and which other findings name the same problem. A case's real
|
||||
problems are the union of judged-real problems across all arms and trials; each arm's catch rate on them uses the
|
||||
rule above, with the judge's grouping in place of the planted label. The judge sees findings as A, B and C, never an
|
||||
arm name, model or cost. It is checked first against 48 hand-checked labels: findings from merged gstack pull
|
||||
requests outside the case set, 24 that led to a fix and 24 dismissed below the floor, each read against its diff by
|
||||
the runner and frozen with a SHA-256 before the judge sees any. `gstack-eval-plan judge --check` must print
|
||||
`JUDGE_AGREEMENT: ok` (Cohen's kappa at least 0.6) before its numbers count.
|
||||
|
||||
**Comparisons.** Every comparison is paired by case: the per-case difference of means (cost, wall time, `calls=`)
|
||||
or of catch rates, with a 95% t interval over the 10 cases. Every trial is reported; nothing is averaged away.
|
||||
|
||||
## Decoys
|
||||
Before any trial, the catch script scores 50 synthetic finding lists built from the sealed labels, 5 per case: the
|
||||
right file with the line 4 or more away; the right line with no cause word; the cause words on another file; a final
|
||||
head with the planted lines reworded but the bug kept (the sealed regression test still fails); and one exact match. It must score the 40 decoys 0 and the 10
|
||||
matches 1. A decoy pass proves the rule rejects near misses and does not credit a cosmetic edit as a fix.
|
||||
|
||||
## Blinding
|
||||
- Arms see a real project: the garrytan/gstack checkout, plain branch names taken from each host's own branch, and
|
||||
the host's own commit title. No branch, file, commit message, prompt or label the trial controls carries an eval word.
|
||||
- This file and its results file are removed from every case base, and the label file never enters the repository.
|
||||
The installed gstack copy (`~/.local/share/gstack`) still holds this file, so every C and F transcript is scanned
|
||||
for reads of it or of the label file's path, and a trial that read either scores zero.
|
||||
- Each arm's checkout keeps git history, where the original fix commit lives. Lookups of the planted path's history
|
||||
(`git log -S`, `git log -p`, `git blame` on that file) are counted per trial, and catches made after one are
|
||||
reported separately as well as in the totals.
|
||||
- The judge sees A, B and C, never an arm, model or price.
|
||||
|
||||
## Stop rules
|
||||
- Spend cap: $3,250 cost.dollars for everything in the budget table (the approved figure replaces it if lower). `budget_exhausted` stops the run with every finished trial reported, and the run does not resume.
|
||||
- Per trial: $150 or 4 hours. A trial stopped there is recorded as incomplete, with its cost at the stop counted (a lower bound, flagged), and scored on what it caught by then.
|
||||
- Pilot: if any pilot C or F trial ends without a receipt for a harness reason (machine, install, CI wiring), the run stops; the fix is a dated change to this file before the first counted trial, or a new preregistration if a bar, arm or decision changes.
|
||||
- The catch script below 50 of 50 on decoys, or the judge below kappa 0.6: stop before any trial; fix the script or the judge prompt and relabel; never lower a bar.
|
||||
- Projection: if the pilot's projection for all 90 trials plus the fixed costs exceeds the approved budget, stop and ask again with the new figure.
|
||||
- No retries of a counted trial, with the one exception AGENTS.md allows: a red case may rerun once at the same head as a new panel, reported as flaky. Harness or infrastructure failures never rerun.
|
||||
|
||||
## Budget (all-in estimate, cost.dollars)
|
||||
| Item | Count | Each | Total |
|
||||
|---|---|---|---|
|
||||
| C trials | 30 | $38 | $1,140 |
|
||||
| F trials | 30 | $38 | $1,140 |
|
||||
| N trials | 30 | $6 | $180 |
|
||||
| Sealing: picking hosts and plants, proving each plant | 1 | $60 | $60 |
|
||||
| Judge: 48 labels plus about 300 findings on GPT-6 Astra | 1 | $80 | $80 |
|
||||
| Runner: waves, audits, scoring, results write-up | 1 | $150 | $150 |
|
||||
| /ship of the results pull request | 1 | $35 | $35 |
|
||||
| **Estimate** | | | **$2,785** |
|
||||
| Cap (estimate plus about 15%) | | | $3,250 |
|
||||
|
||||
That sits under R71's $4,000 guideline for an eval with a GPT-6 Astra judge. If F saves what the profile predicts
|
||||
(about $10 a run), the actual total is near $2,500. Machines, CI minutes and the runner VM are not in cost.dollars and
|
||||
are reported separately. Each case gets 3 trials because AGENTS.md gives a `behavior` case a panel of 3, and because
|
||||
one trial each would let noise decide the "misses nothing" bar (see Bars).
|
||||
|
||||
## Held-out exposure
|
||||
| set | prior exposure (which runs, which models) | reservation (what this run may read) |
|
||||
|---|---|---|
|
||||
| the 10 sealed cases (hosts, plants, labels) | none: picked after this file merges; hosts and fixes are public on GitHub, all merged after Opus 5.5's June 2026 training cutoff | arms read only their case checkout; the runner reads labels to score; the judge reads findings and the case diff, never the labels |
|
||||
| the 48 judge labels | findings already read by /ship reviewers on their own pull requests, none in the case set | the judge reads them once for `judge --check` |
|
||||
|
||||
## After the run
|
||||
Results go to `docs/evals/ship-conductor-fold-results.md`, opening with two or three plain takeaways, with one table
|
||||
holding a filled row for C, F and N, checked with `gstack-eval-plan results --check` before any number leaves the
|
||||
run. Counts that may be zero (structural losses, contaminated trials, history lookups) are reported in prose, outside
|
||||
the tables. A null gets one R48 "Why null?" cycle and a KEEP, FIX-METHOD or CUT-CANDIDATE verdict.
|
||||
Reference in new issue
Block a user