From 5b445ae937dabc2370549c396e2a38a596b23ea1 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sun, 1 Nov 2026 23:57:39 -0800 Subject: [PATCH] v1.92.64.0 ship: pin Step 11's outside voice to the certified model, refuse same-family voices, docs audit on Sonnet 5.5 (GSTA-167 PR B) (#3698) * ship: pin the outside voice to the certified model, refuse same-family voices, docs audit on Sonnet (GSTA-167 PR B) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog.d: GSTA-167 PR B fragment Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (outside voice with no provider prefix STOPs) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: Step 11 review fixes (resume on dispatched voice models, case-insensitive family, every outside voice checked) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog.d: GSTA-167 PR B review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: outside-voice default and voice-models.json pin; Step 14.5 runs once on Sonnet (GSTA-167 PR B) Adds the prb-docs-2 auditor's docs/unattended.md paragraph and fixes capy/skill/gstack/SKILL.md:49 at the same byte count (22,365 B), so the 5,312-token context-budget ceiling is unchanged. The ship-driver test pins the new 14.5 wording instead of the dropped haiku aside. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: trim capy SKILL.md:49 so it stays under the token ceiling beside #3798 '(14.5 runs once)' -> '(14.5 once)' and 'Otherwise' -> 'Else' (22,355 B, 5,310 tokens on main; 22,371 B, about 5,314 tokens merged with #3798 against its 5,315 ceiling). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 15 ++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- capy/skill/gstack/SKILL.md | 2 +- docs/unattended.md | 8 +++ lib/capy-review-pass.js | 49 +++++++++++-- package.json | 2 +- ship/DRIVER.md | 6 +- ship/DRIVER.md.tmpl | 6 +- test/review-pass.test.ts | 126 ++++++++++++++++++++++++++++++++- test/ship-driver.test.ts | 4 +- 11 files changed, 204 insertions(+), 18 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5e8b31011..fe9576817 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [1.92.64.0] - 2026-11-02 + +**/ship's Step 11 outside voice now defaults to the model it was tested with, and a same-family outside voice is refused.** On a lane running an Anthropic model, the pass program uses `openai/gpt-6.1-sol` at medium reasoning when no outside-voice model is given. That is the pairing no-loss rounds 3 to 6 and A3668 ran. Until now the lane picked the model (gpt-6-astra in replays 6c, 7e and 8g), and the only check was that some model was given, so a Claude outside voice beside a Claude native voice would have run as "outside" (GSTA-167 plan R2 and O2). + +### Changed +- **Outside-voice default, Anthropic lanes only** (`lib/capy-review-pass.js`): with no `models['outside-voice']` and an `anthropic/` `laneModel`, the voice runs on `openai/gpt-6.1-sol` at medium. Any other lane, or no lane identity, still stops with "none was given", because an OpenAI lane's pairing is not certified. +- **Same-family STOP**: an outside voice whose provider (the part before `/`, compared without regard to case) matches the lane model or the native voice's model stops the pass before either voice is created. The check runs again on every rerun of the same manifest and before a queued voice is started. An outside voice with no provider prefix, or with no lane model and no native model to compare against, also stops. +- **Voices resume on the models they were dispatched with**: the pass records each Step 11 voice's model and the lane model in `voice-models.json` in the pass directory before creating it. A rerun with different inputs starts a queued voice on its recorded model and checks it against the recorded native model, so it can't pair two voices of one family. +- **Overrides are labeled**: each outside voice prints an `OUTSIDE_VOICE:` line when it is created or started from the queue, saying whether it is the certified configuration. A different model or reasoning level still runs, marked "outside the certified configuration". +- **Step 14.5 docs audit runs once on Sonnet 5.5** (`ship/DRIVER.md`). In replay C5 the docs audit first ran on a smaller model whose reply lacked its JSON line, then ran again on Sonnet, which wasted 1.55M tokens. To stay under the driver's 12 KB cap, one sentence that repeated Phase 4 and 5 ("A fix repeats Phases 3-4.") was removed. + +### Tests +- `test/review-pass.test.ts`: six new fake-host cases: a same-family override stops; a rerun or a queued start still refuses it; a queued voice resumes on its recorded model; a missing lane identity, an unprefixed id or an unprefixed lane stops; an Anthropic lane gets the certified default with every brief byte-identical to naming that model; an override with changed reasoning runs and is labeled uncertified. The Step 11 case now passes `laneModel: undefined` to keep its "none was given" check, since an Anthropic lane gets the default. +- `test/ship-driver.test.ts`: the audit-reader assertion names Step 14.5's Sonnet 5.5 run. + ## [1.92.63.0] - 2026-11-02 **A slot restamp after the docs audit no longer stops `finish` with `DOCS: stale (CHANGELOG.md)` (GST-906).** When a PR lost its merge slot after Step 14.5, the new slot's restamp re-heads the CHANGELOG entry and moves VERSION, and on the lost-slot path it merges main first. `gstack-ship-release finish` read all of that as a docs change, so the only ways forward were a third docs audit, which the policy forbids, or an owner-accepted named risk. Reported by GBRA-258 on gbrain #8372. diff --git a/VERSION b/VERSION index 18a77cfd2..3fa9ff907 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.92.63.0 +1.92.64.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 817029559..8fa12fa27 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.92.63.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.92.64.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/capy/skill/gstack/SKILL.md b/capy/skill/gstack/SKILL.md index c6d85c010..bb617b0c0 100644 --- a/capy/skill/gstack/SKILL.md +++ b/capy/skill/gstack/SKILL.md @@ -46,7 +46,7 @@ Capy checklist for `/autoplan`: drive `gstack-autoplan`, never a hand adaptation Token caps on /ship: pass a brief's cap as `--token-cap `. /ship reads `cost.tokens` and `cost.credits` from `tools.thread.get({ thread_id: "" })` (its own jam; a subagent reading root `"this"` starts `--subagent --reading-from this --meter-scope tree`) inside the same execute program as the `gstack-ship-budget` command that takes them as `--reading --credits `, never alone. `ship/DRIVER.md` starts `--mode observe`: each program ends with `note`, and only a note past the cap stops (`docs/ship-budget.md`). -/ship on Capy is about a dozen calls, each one execute program (`ship/DRIVER.md`). Each review pass and Step 11 is one call to the checked-in pass program (fill only `input`): `const t0 = await now(); const p = (await tools.capy.bash({ command: ". '' && gstack-review-dispatch program", description: "Pass" })).output.split("\n"); return p[0] + "\n" + (await new (Object.getPrototypeOf(async () => {}).constructor)("tools", "sleep", "now", "input", JSON.parse(p[1]))(tools, sleep, now, { startedAt: t0, launch: "", cwd: "", base: "", out: "/p", pass: "-p", laneModel: "", planFlags: [], logStart: ["review"], jam: "", record: "", step: "9.2" }));` It plans, logs `logStart` (`reservation` mode from `RESERVATION_ID:`), sends each brief on `machine: "shared"`, then polls: one `tools.subagent.list` per ~20 s tick, returning 200 s after the program's first line, where a reader is settled only at status `done` or `failed` or with a `summary` (a just-created `idle` reader has not run); it starts readers under `rejected` with `tools.subagent.start` as slots free and stops on one stuck 20 minutes. It ends on `NEXT:` (`poll`: rerun) or `STOP:`. `models` maps roles to models (`outside-voice`: another family); Step 11 adds `stage: "voices"`; a full repeat pass, `diffBase: ""`. Steps 7, 8 and 14.5 (`step: "14.5"`) use `extras`, JSON `{name, prompt_file, first, model}` (prompts verbatim from their sections, Step 7 `first`); they cannot block, so `model: { model_id: "anthropic/claude-sonnet-5-5", reasoning_mode: "low" }` (`anthropic/claude-haiku-5-5` also fits). Waits: never let 5 minutes pass between calls. While readers run, never end the turn on `wait`; rerun the loader. Otherwise re-issue `wait` with `duration_seconds: 200` and operation ids. At Step 22 report the `CALLS:` line of `gstack-ship-budget calls --events ` on your jam's transcript since /ship began. A small waiting child saves nothing. +/ship on Capy is about a dozen calls, each an execute program (`ship/DRIVER.md`). Each review pass and Step 11 is one call to the checked-in pass program (fill only `input`): `const t0 = await now(); const p = (await tools.capy.bash({ command: ". '' && gstack-review-dispatch program", description: "Pass" })).output.split("\n"); return p[0] + "\n" + (await new (Object.getPrototypeOf(async () => {}).constructor)("tools", "sleep", "now", "input", JSON.parse(p[1]))(tools, sleep, now, { startedAt: t0, launch: "", cwd: "", base: "", out: "/p", pass: "-p", laneModel: "", planFlags: [], logStart: ["review"], jam: "", record: "", step: "9.2" }));` It plans, logs `logStart` (`reservation` mode from `RESERVATION_ID:`), sends each brief on `machine: "shared"`, then polls: one `tools.subagent.list` per ~20 s tick, returning 200 s after the program's first line, where a reader is settled only at status `done` or `failed` or with a `summary` (a just-created `idle` reader has not run); it starts readers under `rejected` with `tools.subagent.start` as slots free and stops on one stuck 20 minutes. It ends on `NEXT:` (`poll`: rerun) or `STOP:`. `models` maps roles to models (`outside-voice`: another family, pinned in `voice-models.json`); Step 11 adds `stage: "voices"`; a full repeat pass, `diffBase: ""`. Steps 7, 8 and 14.5 (`step: "14.5"`) use `extras`, JSON `{name, prompt_file, first, model}` (prompts verbatim from their sections, Step 7 `first`); they cannot block, so `model: { model_id: "anthropic/claude-sonnet-5-5", reasoning_mode: "low" }` (14.5 once). Waits: never let 5 minutes pass between calls. While readers run, never end the turn on `wait`; rerun the loader. Else re-issue `wait` with `duration_seconds: 200` and operation ids. At Step 22 report the `CALLS:` line of `gstack-ship-budget calls --events ` on your transcript since /ship began. A small waiting child saves nothing. ## Plan, approve, ship diff --git a/docs/unattended.md b/docs/unattended.md index 822f38e73..cb6ea320a 100644 --- a/docs/unattended.md +++ b/docs/unattended.md @@ -230,6 +230,14 @@ credits outside coverage. A parent reads the rows with `gstack-review-read` and treats `outside_status: unavailable` as missing coverage, not as a clean outside review. +The pass program (`lib/capy-review-pass.js`) picks the outside voice's model +when `input.models['outside-voice']` is empty and the lane model is +`anthropic/`: `openai/gpt-6.1-sol` at medium reasoning, the pairing the no-loss +rounds ran. Any other lane must name a model. An outside voice from the same +provider as the lane model or the native voice stops the pass (`STOP:`) before +either voice is created, including on a rerun, and each outside voice prints an +`OUTSIDE_VOICE:` line saying whether it is the certified configuration. + ### The carry across a base merge or a test-only fix When the parent merges the base in or fixes a test after the receipt was diff --git a/lib/capy-review-pass.js b/lib/capy-review-pass.js index e65aeacc0..9511695e9 100644 --- a/lib/capy-review-pass.js +++ b/lib/capy-review-pass.js @@ -91,8 +91,35 @@ const briefText = async (r) => { }; const roleArgs = (r) => ' --role ' + q(r.role) + (r.name ? ' --name ' + q(r.name) : ''); const REASONING = { medium: 'medium', light: 'low' }; -const modelOf = (r) => r.model || (input.models || {})[r.role === 'specialist-batched' ? 'specialist' : r.role] - || (REASONING[r.effort] && input.laneModel ? { model_id: input.laneModel, reasoning_mode: REASONING[r.effort] } : undefined); +const family = (id) => (typeof id === 'string' && id.includes('/') ? id.slice(0, id.indexOf('/')).toLowerCase() : ''); +const CERTIFIED_OUTSIDE = { model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'medium' }; +const voiceFile = input.out.replace(/\/+$/, '') + '/voice-models.json'; +let voices = { laneModel: null, readers: {} }; +let laneFamily = family(input.laneModel); +const modelOf = (r) => r.model || (r.reader in voices.readers ? voices.readers[r.reader] || undefined + : (input.models || {})[r.role === 'specialist-batched' ? 'specialist' : r.role] + || (r.role === 'outside-voice' && laneFamily === 'anthropic' ? CERTIFIED_OUTSIDE : undefined) + || (REASONING[r.effort] && input.laneModel ? { model_id: input.laneModel, reasoning_mode: REASONING[r.effort] } : undefined)); +const outsideNote = (r, source) => { + const ov = modelOf(r); + const certified = laneFamily === 'anthropic' && ov.model_id === CERTIFIED_OUTSIDE.model_id && ov.reasoning_mode === CERTIFIED_OUTSIDE.reasoning_mode; + return 'OUTSIDE_VOICE: ' + ov.model_id + ' reasoning=' + (ov.reasoning_mode || 'default') + ' ' + source + + (certified ? ' (certified configuration)' : '; outside the certified configuration (' + CERTIFIED_OUTSIDE.model_id + ' at ' + CERTIFIED_OUTSIDE.reasoning_mode + ' beside an anthropic lane)'); +}; +const familyStop = (readers) => { + const native = readers.find((x) => x.role === 'adversarial-native'); + const nativeFamily = family(native && modelOf(native) ? modelOf(native).model_id : undefined); + for (const ov of readers.filter((x) => x.role === 'outside-voice' && !['created', 'collected', 'failed'].includes(x.state))) { + const om = modelOf(ov); + if (!om) return "the outside voice needs input.models['outside-voice'] (a model of another family); none was given"; + const lane = voices.laneModel || input.laneModel; + if (!laneFamily && !nativeFamily) return 'the outside voice ' + om.model_id + ' cannot be checked against another family: ' + (lane ? 'the lane model ' + lane + ' has no provider prefix and no native voice model was given' : 'neither input.laneModel nor a native voice model was given'); + const f = family(om.model_id); + if (!f) return 'the outside voice ' + om.model_id + ' has no provider prefix, so its family cannot be checked'; + if (f === laneFamily || f === nativeFamily) return 'the outside voice ' + om.model_id + ' is the ' + f + ' family, the same as the ' + (f === nativeFamily ? 'native voice' : 'lane model') + "; Step 11 needs input.models['outside-voice'] from another family"; + } + return null; +}; const safe = (s) => s.replace(/[^a-zA-Z0-9._-]/g, '_'); const active = input.stage === 'voices' ? ['voices'] : ['first', 'specialists', 'red-team']; const beforeKey = input.stage === 'voices' ? 'before_voices' : 'before'; @@ -100,6 +127,11 @@ const beforeKey = input.stage === 'voices' ? 'before_voices' : 'before'; try { let m = await load(); if (input.stage !== 'voices' && m.next === 'fix-check') active.push('voices'); + if (active.includes('voices')) { + const v = await sh('cat ' + q(voiceFile) + ' 2>/dev/null || true'); + if (v.out.trim()) voices = JSON.parse(v.out); + if (voices.laneModel) laneFamily = family(voices.laneModel); + } if (input.stage === 'voices' && !m.readers.some((r) => r.stage === 'voices')) return finish(['VOICES: none (the pass plan lists no Step 11 voice)', 'NEXT: judge']); const seen = {}; for (const s of input.logStart || []) { @@ -125,12 +157,17 @@ try { await mark(r.reader, found[0].status === 'draft' ? 'queued' : 'created', { path: found[0].path }); } m = await load(); + const sameFamily = active.includes('voices') && familyStop(m.readers); + if (sameFamily) return stop(sameFamily); for (const r of m.readers.filter((x) => active.includes(x.stage) && x.state === 'queued')) { const t = byPath.get(r.path); if (t && t.status !== 'draft') { await mark(r.reader, 'created', {}); continue; } const model = modelOf(r); const s = await tools.subagent.start({ indices: [Number(r.path)], machine: 'shared', ...(model ? { model } : {}) }); - if ((s.started || []).length) await mark(r.reader, 'created', {}); + if ((s.started || []).length) { + if (r.role === 'outside-voice') lines.push(outsideNote(r, 'as dispatched, started from the queue')); + await mark(r.reader, 'created', {}); + } else if (r.state_at && (await now()) - Date.parse(r.state_at) > stuckMs) return stop(r.reader + ' (subagent ' + r.path + ') still queued after ' + Math.round(stuckMs / 60000) + ' minutes; free a subagent slot, then rerun'); } for (const r of m.readers.filter((x) => active.includes(x.stage) && x.state === 'created')) { @@ -167,7 +204,6 @@ try { lines.push(...s.out.trim().split('\n')); m = await load(); } - if (m.readers.some((x) => x.stage === stage && x.role === 'outside-voice' && x.state === 'rendered' && !modelOf(x))) return stop("the outside voice needs input.models['outside-voice'] (a model of another family); none was given"); const unmapped = m.readers.find((x) => x.stage === stage && x.state === 'rendered' && REASONING[x.effort] && !modelOf(x)); if (unmapped) return stop(unmapped.reader + ' runs at effort=' + unmapped.effort + ', which needs input.laneModel (your model id) or input.models; none was given'); for (const r of m.readers.filter((x) => x.stage === stage && x.state === 'rendered')) { @@ -184,6 +220,11 @@ try { const b = await reading(); if (b) { await sh(bin + ' mark --manifest ' + q(mf) + ' --before ' + b.tokens + ',' + b.credits + (beforeKey === 'before_voices' ? ' --voices' : '')); m[beforeKey] = b; } } + if (r.role === 'outside-voice') lines.push(outsideNote(r, r.model || (input.models || {})['outside-voice'] ? 'given' : 'default for an anthropic lane')); + if (r.stage === 'voices') { + voices = { laneModel: voices.laneModel || input.laneModel || null, readers: { ...voices.readers, [r.reader]: model || null } }; + await tools.capy.write({ path: voiceFile, content: JSON.stringify(voices) }); + } await mark(r.reader, 'intent', {}); const res = await tools.subagent.create({ tasks: [{ title: r.title, prompt: text }], start: { machine: 'shared', ...(model ? { model } : {}) } }); const idx = String(res.tasks[0].index); diff --git a/package.json b/package.json index fcdaacd15..f10ba2e02 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.92.63", + "version": "1.92.64", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/ship/DRIVER.md b/ship/DRIVER.md index ac19dd0bd..8e3b4f4ca 100644 --- a/ship/DRIVER.md +++ b/ship/DRIVER.md @@ -24,11 +24,11 @@ version, CHANGELOG, commits), **Documentation** (Step 14.5) and **Next steps** ( outside-voice and documentation brief, its severity and confidence rules and gates come from its section verbatim (placeholders filled); this file never shortens a reviewer's brief. A blocking reader's prompt is its `gstack-review-dispatch render` -file, unchanged; its findings reach `merged.jsonl` only via `collect`; `judge` refuses a drop or downgrade. Audit readers (Steps 7, 8, 14.5) cannot block, so each runs on a -small model at low effort; their findings go to the PR body. +file, unchanged; its findings reach `merged.jsonl` only via `collect`; `judge` refuses a drop or downgrade. Audit readers (Steps 7, 8, 14.5) cannot block, so each runs at low effort on a small model, +Step 14.5 once on Sonnet 5.5; findings go to the PR body. **Calls.** One pass with no fixes is 9 calls: Phases 1-8 and the report; at most 12 with a STOP's -section read or a fix-check, plus a keep-alive per 200 s waited. A fix repeats Phases 3-4. +section read or a fix-check, plus a keep-alive per 200 s waited. **Budget: observe.** `gstack-ship-budget start --mode observe` records, never admits: each program ends with `gstack-ship-budget note --step --reading R --credits C --record `, R and C read in that program diff --git a/ship/DRIVER.md.tmpl b/ship/DRIVER.md.tmpl index 0298fb93e..cefeda67f 100644 --- a/ship/DRIVER.md.tmpl +++ b/ship/DRIVER.md.tmpl @@ -22,11 +22,11 @@ version, CHANGELOG, commits), **Documentation** (Step 14.5) and **Next steps** ( outside-voice and documentation brief, its severity and confidence rules and gates come from its section verbatim (placeholders filled); this file never shortens a reviewer's brief. A blocking reader's prompt is its `gstack-review-dispatch render` -file, unchanged; its findings reach `merged.jsonl` only via `collect`; `judge` refuses a drop or downgrade. Audit readers (Steps 7, 8, 14.5) cannot block, so each runs on a -small model at low effort; their findings go to the PR body. +file, unchanged; its findings reach `merged.jsonl` only via `collect`; `judge` refuses a drop or downgrade. Audit readers (Steps 7, 8, 14.5) cannot block, so each runs at low effort on a small model, +Step 14.5 once on Sonnet 5.5; findings go to the PR body. **Calls.** One pass with no fixes is 9 calls: Phases 1-8 and the report; at most 12 with a STOP's -section read or a fix-check, plus a keep-alive per 200 s waited. A fix repeats Phases 3-4. +section read or a fix-check, plus a keep-alive per 200 s waited. **Budget: observe.** `gstack-ship-budget start --mode observe` records, never admits: each program ends with `gstack-ship-budget note --step --reading R --credits C --record `, R and C read in that program diff --git a/test/review-pass.test.ts b/test/review-pass.test.ts index 9d440c612..e3ed0fee3 100644 --- a/test/review-pass.test.ts +++ b/test/review-pass.test.ts @@ -182,7 +182,7 @@ function host(dir: string, env: NodeJS.ProcessEnv, o: { output?: (title: string) if (mode === 'after') throw new Error('create result lost'); return { tasks: [{ index, title: t.title }], ...(rejected ? { rejected: [{ index, error: 'cap' }] } : {}) }; }, - start: async ({ indices }: { indices: number[] }) => { starts.push(indices); for (const i of indices) subs.get(String(i))!.status = 'working'; return { started: indices.map(index => ({ index })), rejected: [] }; }, + start: async ({ indices, model }: { indices: number[]; model?: unknown }) => { starts.push(indices); for (const i of indices) { const s = subs.get(String(i))!; s.status = 'working'; if (model) s.model = model; } return { started: indices.map(index => ({ index })), rejected: [] }; }, list: async () => { advance(); return { tasks: [...subs].map(([p, s]) => ({ path: p, title: s.title, status: s.status })), total: subs.size }; }, get: async ({ path: p }: { path: string }) => ({ path: p, status: subs.get(p)!.status, summary: subs.get(p)!.summary }), }, @@ -338,7 +338,7 @@ describe('Capy pass program (fake host, real bins)', () => { const h = host(dir, env); const base = { base: 'main', out, pass: 'x7p1', planFlags: ['--depth-floor', 'full'] }; expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); - expect(await runPass(h, { ...base, stage: 'voices' })).toContain("STOP: the outside voice needs input.models['outside-voice']"); + expect(await runPass(h, { ...base, stage: 'voices', laneModel: undefined })).toContain("STOP: the outside voice needs input.models['outside-voice']"); expect(h.creates.some(c => c.title.startsWith('ship x7p1 ') && /adversarial|outside/.test(c.title))).toBe(false); const models = { 'outside-voice': { model_id: 'openai/gpt-6.1-sol' }, 'adversarial-native': { model_id: 'anthropic/claude-opus-5-5', reasoning_mode: 'high' } }; expect(await runPass(h, { ...base, stage: 'voices', models })).toContain('NEXT: judge the voices'); @@ -346,6 +346,128 @@ describe('Capy pass program (fake host, real bins)', () => { expect(h.creates.find(c => c.title === 'ship x7p1 adversarial-native')!.model).toEqual(models['adversarial-native']); }, 240_000); + // Value: protects=Step 11's outside voice is never the native voice's or the lane's model family; fails_when=the STOP checks only that some model was given (lib/capy-review-pass.js before GSTA-167 PR B), so a Claude outside voice beside a Claude native voice runs as "outside"; + // why_new=GSTA-167 plan R2 (autoplan-20261031135232); seam=none; no_claim=which family pairs find more + test('a same-family explicit outside voice STOPs before either voice is created', async () => { + const { dir, env } = repo(); + const h = host(dir, env); + const base = { base: 'main', out: path.join(dir, '.state/run/p1'), pass: 'x14p1', planFlags: ['--depth-floor', 'full'] }; + expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); + expect(await runPass(h, { ...base, stage: 'voices', models: { 'outside-voice': { model_id: 'anthropic/claude-sonnet-5-5', reasoning_mode: 'medium' } } })) + .toContain("STOP: the outside voice anthropic/claude-sonnet-5-5 is the anthropic family, the same as the lane model; Step 11 needs input.models['outside-voice'] from another family"); + expect(await runPass(h, { ...base, stage: 'voices', models: { 'outside-voice': { model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'medium' }, 'adversarial-native': { model_id: 'openai/gpt-6-astra', reasoning_mode: 'high' } } })) + .toContain('STOP: the outside voice openai/gpt-6.1-sol is the openai family, the same as the native voice'); + expect(h.creates.some(c => /^ship x14p1 (adversarial-native|outside-voice)$/.test(c.title))).toBe(false); + }, 240_000); + + // Value: protects=a Step 11 rerun on one manifest keeps refusing a same-family outside voice, including one queued at the subagent cap that tools.subagent.start would pick up; fails_when=the family check runs only on the first render, so a resumed or queued voice starts unchecked; + // why_new=GSTA-167 plan R2; seam=none; no_claim=the host's cap timing + test('a Step 11 rerun on one manifest still refuses a same-family outside voice, rendered or queued', async () => { + const same = { 'outside-voice': { model_id: 'anthropic/claude-sonnet-5-5', reasoning_mode: 'medium' } }; + const refused = /STOP: the outside voice anthropic\/claude-sonnet-5-5 is the anthropic family/; + const { dir, env } = repo(); + const h = host(dir, env); + const base = { base: 'main', out: path.join(dir, '.state/run/p1'), pass: 'x15p1', planFlags: ['--depth-floor', 'full'] }; + expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); + expect(await runPass(h, { ...base, stage: 'voices', models: same })).toMatch(refused); + expect(await runPass(h, { ...base, stage: 'voices', models: same })).toMatch(refused); + expect(h.creates.some(c => /^ship x15p1 (adversarial-native|outside-voice)$/.test(c.title))).toBe(false); + + const b = repo(); + const h2 = host(b.dir, b.env, { reject: t => t.endsWith('outside-voice') }); + const base2 = { ...base, out: path.join(b.dir, '.state/run/p1'), pass: 'x18p1' }; + expect(await runPass(h2, base2)).toContain('NEXT: judge (Phase 4)'); + expect(await runPass(h2, { ...base2, stage: 'voices', budgetMs: 1 })).toMatch(/DISPATCHED: outside-voice path=\d+ queued/); + const starts = h2.starts.length; + const recorded = path.join(base2.out, 'voice-models.json'); + const kept = fs.readFileSync(recorded, 'utf8'); + fs.rmSync(recorded); + expect(await runPass(h2, { ...base2, stage: 'voices', models: same })).toMatch(refused); + expect(h2.starts.length).toBe(starts); + fs.writeFileSync(recorded, kept); + const resumed = await runPass(h2, { ...base2, stage: 'voices', models: same }); + expect(resumed).toContain('OUTSIDE_VOICE: openai/gpt-6.1-sol reasoning=medium as dispatched, started from the queue (certified configuration)'); + expect(h2.creates.find(c => c.title === 'ship x18p1 outside-voice')!.model).toEqual({ model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'medium' }); + }, 300_000); + + // Value: protects=a queued outside voice resumes on the models recorded when the voices were dispatched, so changed rerun inputs cannot pair it with a native voice of its own family; fails_when=the rerun's input decides the queued voice's model and the native family (outside-voice finding on #3698: two OpenAI voices started); + // why_new=/ship Step 11 review of #3698; seam=none; no_claim=inputs that change between reruns on purpose + test('a queued outside voice resumes on its dispatched model, checked against the dispatched native family', async () => { + const { dir, env } = repo(); + const h = host(dir, env, { reject: t => t.endsWith('outside-voice') }); + const base = { base: 'main', out: path.join(dir, '.state/run/p1'), pass: 'x19p1', planFlags: ['--depth-floor', 'full'] }; + expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); + const first = { 'adversarial-native': { model_id: 'openai/gpt-6-astra', reasoning_mode: 'high' }, 'outside-voice': { model_id: 'google/gemini-4-argon', reasoning_mode: 'medium' } }; + expect(await runPass(h, { ...base, stage: 'voices', models: first, budgetMs: 1 })).toMatch(/DISPATCHED: outside-voice path=\d+ queued/); + const res = await runPass(h, { ...base, stage: 'voices' }); + expect(res).not.toContain('STOP:'); + expect(res).toContain('OUTSIDE_VOICE: google/gemini-4-argon reasoning=medium as dispatched, started from the queue; outside the certified configuration'); + expect(h.creates.find(c => c.title === 'ship x19p1 outside-voice')!.model).toEqual(first['outside-voice']); + expect(h.creates.find(c => c.title === 'ship x19p1 adversarial-native')!.model).toEqual(first['adversarial-native']); + }, 300_000); + + // Value: protects=a lane whose family is unknown or not Anthropic gets no default outside voice, and an explicit one cannot skip the family check; fails_when=the pin applies to a lane it was not certified for, or an outside voice runs with nothing to compare its family to; + // why_new=GSTA-167 plan R2; seam=none; no_claim=an OpenAI lane's pairing, which is uncertified + test('a missing lane identity STOPs the outside voice: none given, or given with nothing to compare its family to', async () => { + const { dir, env } = repo(); + const h = host(dir, env); + const base = { base: 'main', out: path.join(dir, '.state/run/p1'), pass: 'x16p1', planFlags: ['--depth-floor', 'full'] }; + expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); + const none = "STOP: the outside voice needs input.models['outside-voice'] (a model of another family); none was given"; + expect(await runPass(h, { ...base, stage: 'voices', laneModel: undefined })).toContain(none); + expect(await runPass(h, { ...base, stage: 'voices', laneModel: 'openai/gpt-6.1-sol' })).toContain(none); + expect(await runPass(h, { ...base, stage: 'voices', laneModel: undefined, models: { 'outside-voice': { model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'medium' } } })) + .toContain('STOP: the outside voice openai/gpt-6.1-sol cannot be checked against another family: neither input.laneModel nor a native voice model was given'); + expect(await runPass(h, { ...base, stage: 'voices', models: { 'outside-voice': { model_id: 'claude-sonnet-5-5' } } })) + .toContain('STOP: the outside voice claude-sonnet-5-5 has no provider prefix, so its family cannot be checked'); + expect(await runPass(h, { ...base, stage: 'voices', models: { 'outside-voice': { model_id: 'Anthropic/claude-sonnet-5-5' } } })) + .toContain('STOP: the outside voice Anthropic/claude-sonnet-5-5 is the anthropic family, the same as the lane model'); + expect(await runPass(h, { ...base, stage: 'voices', laneModel: 'claude-opus-5-5', models: { 'outside-voice': { model_id: 'openai/gpt-6.1-sol' } } })) + .toContain('STOP: the outside voice openai/gpt-6.1-sol cannot be checked against another family: the lane model claude-opus-5-5 has no provider prefix and no native voice model was given'); + expect(h.creates.some(c => /^ship x16p1 (adversarial-native|outside-voice)$/.test(c.title))).toBe(false); + }, 240_000); + + const certified = { model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'medium' }; + async function voicesRun(dir: string, env: NodeJS.ProcessEnv, sub: string, models?: Record) { + const out = path.join(dir, '.state/run', sub); + const h = host(dir, env); + const base = { base: 'main', out, pass: 'x17p1', planFlags: ['--depth-floor', 'full'] }; + expect(await runPass(h, base)).toContain('NEXT: judge (Phase 4)'); + const res = await runPass(h, { ...base, stage: 'voices', ...(models ? { models } : {}) }); + const m = JSON.parse(fs.readFileSync(path.join(out, 'pass-manifest.json'), 'utf8')) as Manifest; + for (const r of m.readers) expect(h.creates.find(c => c.title === r.title)!.prompt).toBe(fs.readFileSync(r.file!, 'utf8')); + const shas = Object.fromEntries(h.creates.map(c => [c.title, promptHash(c.prompt.split(out).join(''))])); + return { res, shas, models: Object.fromEntries(h.creates.map(c => [c.title, c.model])) }; + } + + // Value: protects=an Anthropic lane's Step 11 gets the certified outside voice (openai/gpt-6.1-sol at medium, as rounds 3-6 and A3668 ran) without naming it, and every brief goes out byte for byte as it does with that model named; fails_when=the lane must pick the outside voice (gpt-6-astra in 6c/7e/8g), or the pin changes any reader's brief or another reader's model; + // why_new=GSTA-167 plan R2; seam=none; no_claim=the certified pair's recall, which arm-L6 measured + test('an anthropic lane defaults the outside voice to the certified model, with every brief byte-identical to naming it', async () => { + const { dir, env } = repo(); + const pinned = await voicesRun(dir, env, 'pin'); + expect(pinned.res).toContain('OUTSIDE_VOICE: openai/gpt-6.1-sol reasoning=medium default for an anthropic lane (certified configuration)'); + expect(pinned.res).toContain('NEXT: judge the voices'); + expect(pinned.models['ship x17p1 outside-voice']).toEqual(certified); + const named = await voicesRun(dir, env, 'named', { 'outside-voice': certified }); + expect(named.res).toContain('OUTSIDE_VOICE: openai/gpt-6.1-sol reasoning=medium given (certified configuration)'); + expect(named.shas).toEqual(pinned.shas); + expect(named.models).toEqual(pinned.models); + }, 300_000); + + // Value: protects=an explicit cross-family override is accepted but its pass output says it is outside the certified configuration; fails_when=a changed outside-voice model or reasoning passes silently as if certified; + // why_new=GSTA-167 plan R2; seam=none; no_claim=the uncertified configuration's recall + test('an explicit outside-voice override with changed reasoning runs and is recorded as uncertified, other briefs unchanged', async () => { + const { dir, env } = repo(); + const high = { model_id: 'openai/gpt-6.1-sol', reasoning_mode: 'high' }; + const over = await voicesRun(dir, env, 'over', { 'outside-voice': high }); + expect(over.res).toContain('OUTSIDE_VOICE: openai/gpt-6.1-sol reasoning=high given; outside the certified configuration (openai/gpt-6.1-sol at medium beside an anthropic lane)'); + expect(over.res).toContain('NEXT: judge the voices'); + expect(over.models['ship x17p1 outside-voice']).toEqual(high); + const named = await voicesRun(dir, env, 'named', { 'outside-voice': certified }); + expect(over.shas).toEqual(named.shas); + for (const t of Object.keys(named.models)) if (t !== 'ship x17p1 outside-voice') expect(over.models[t]).toEqual(named.models[t]); + }, 300_000); + test('an audit reader marked first settles before any specialist starts; its output is returned, never merged', async () => { const { dir, env } = repo(); const out = path.join(dir, '.state/run/p1'); diff --git a/test/ship-driver.test.ts b/test/ship-driver.test.ts index 58f9dc471..2558d83d3 100644 --- a/test/ship-driver.test.ts +++ b/test/ship-driver.test.ts @@ -222,11 +222,11 @@ describe('variant G: the call plan, observe budget, waits and audit readers (GST // Value: protects=audits that cannot block (Steps 7, 8, 14.5) run on a small model at low effort with their sections' own briefs, and skip their reader when --audit decides them; fails_when=an audit reader inherits the lane's model and effort (#36: the coverage audit alone was 2.45M at Opus), or an audit brief is paraphrased; // why_new=audit model routing is new; seam=none; no_claim=the audits' recall test('audit readers run on a small model at low effort, named for Capy, and Phase 3 dispatches only what READER names', () => { - expect(flat(doc)).toContain('Audit readers (Steps 7, 8, 14.5) cannot block, so each runs on a small model at low effort; their findings go to the PR body'); + expect(flat(doc)).toContain('Audit readers (Steps 7, 8, 14.5) cannot block, so each runs at low effort on a small model, Step 14.5 once on Sonnet 5.5; findings go to the PR body'); expect(flat(doc)).toContain('`Coverage:` and Step 8\'s line come from `--audit` when no reader ran'); const capy = read('capy/skill/gstack/SKILL.md'); expect(capy).toContain('model: { model_id: "anthropic/claude-sonnet-5-5", reasoning_mode: "low" }'); - expect(capy).toContain('`anthropic/claude-haiku-5-5` also fits'); + expect(capy).toContain('(14.5 once)'); }); // Value: protects=Step 9.2.1's report-only probes run in the parent's own Phase 4 program, as the section says, with its Reads in Phase 3, and