diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index ea9a9f4ca..792a2582f 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1703,6 +1703,24 @@ own attempts. The arm gets: - the same budget treatment: it is priced and reserved in the pilot like every other arm. +**Masking.** On Capy, the owner's user-scope instructions tell every session to +load the user-drive gstack skill, and eval run agents inherit them: in one +checkpoint (24 runs, 3 arms) every run in every arm tried to read that skill in +its first minute. So `## No-gstack arm` also needs three lines, and `prereg +--check` refuses a missing, blank or `none` one (`PREREG_NO_GSTACK_ARM`): + +- `masking:` an empty directory bind-mounted over `~/.capy/drive/user-garry-tan` + and `~/.capy/drive/org-gstack-gbrain` before the run agent starts, or a + container that sees no Capy drive; +- `install:` no gstack on the arm's machine (`~/.local/share/gstack`, + `~/.claude/skills`), checked before the run; +- `audit:` a per-arm audit of drive and skill reads. + +An arm that read a gstack skill file is contaminated and gets zero credit. A +no-gstack result that ran before this rule without a mask is not evidence for +an R48 verdict until it is re-checked; the evidence ledger records it under +`prior` with `unmasked` ([docs/inventory.md](inventory.md)). + **Reporting.** The no-gstack result is one row beside the variants in every results table and every verdict, including when it wins. A comparison that drops the row has not reported its result. `gstack-eval-plan results --check diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 378a8b708..4a7153e12 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -3901,15 +3901,17 @@ refuses to start without a passing check. **Meaning.** The preregistration says `stack: loaded` (some arm loads gstack or another skill stack) and its `## No-gstack arm` section is missing, lacks one -of `arms:`, `inputs:`, `prompt:`, `panel:` or `report:`, lists an arm that +of `arms:`, `inputs:`, `prompt:`, `panel:`, `report:`, `masking:`, `install:` +or `audit:` (or says `none` for one of the last three), lists an arm that is not a `#no-gstack` Arms line, or leaves a model in the comparison without its own no-gstack arm. Arms match on whole identities, so `claude-opus-5#no-gstack` does not cover `claude-opus-5-5`. **Fix.** Add one `#no-gstack` line per model under `## Arms`, list them -on the `arms:` line, and fill the other four keys: the same inputs, a plain -prompt with no skill text, the same panel and judge, and where the result row -goes beside the variants. When no arm loads a skill stack, write +on the `arms:` line, and fill the other keys: the same inputs, a plain +prompt with no skill text, the same panel and judge, where the result row +goes beside the variants, how the run agent is masked from the Capy drives, +the check that no gstack is installed, and the per-arm drive-read audit. When no arm loads a skill stack, write `stack: none` with a `why:` line instead. The rule is in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm). diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index 494f42885..d5fa76a6c 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -140,7 +140,7 @@ Write (or open) the preregistration file. Each section is required and must be f | Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves | | Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted`, the first quota or rate-limit error on a seat-metered arm | | Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it | -| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) | +| No-gstack arm | required with `stack: loaded` (below) | ```bash ~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init @@ -153,11 +153,12 @@ preregistration. **No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack (review variants, preamble loadouts, skill diets, a model panel under a stack) also -runs each model with no skill stack at all. Add `#no-gstack` to Arms for every -model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows -and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed -skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels; -`report:` its row beside the variants in every results table and verdict. +runs each model with no skill stack. Add `#no-gstack` to Arms for every model and +fill `## No-gstack arm`: `arms:` those arms; `inputs:` same rows; `prompt:` the task text +only, same tools; `panel:` the same kind, trials, judge and +labels; `report:` its row beside the variants; `masking:` both Capy drives hidden from the +run agent; `install:` no gstack installed; `audit:` drive reads per arm. An arm that read a +gstack skill file is contaminated and gets zero credit. `PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows `stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out. diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index 1029b4053..997289615 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -62,7 +62,7 @@ Write (or open) the preregistration file. Each section is required and must be f | Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves | | Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted`, the first quota or rate-limit error on a seat-metered arm | | Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it | -| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) | +| No-gstack arm | required with `stack: loaded` (below) | ```bash ~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init @@ -75,11 +75,12 @@ preregistration. **No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack (review variants, preamble loadouts, skill diets, a model panel under a stack) also -runs each model with no skill stack at all. Add `#no-gstack` to Arms for every -model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows -and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed -skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels; -`report:` its row beside the variants in every results table and verdict. +runs each model with no skill stack. Add `#no-gstack` to Arms for every model and +fill `## No-gstack arm`: `arms:` those arms; `inputs:` same rows; `prompt:` the task text +only, same tools; `panel:` the same kind, trials, judge and +labels; `report:` its row beside the variants; `masking:` both Capy drives hidden from the +run agent; `install:` no gstack installed; `audit:` drive reads per arm. An arm that read a +gstack skill file is contaminated and gets zero credit. `PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows `stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out. diff --git a/lib/eval-plan.ts b/lib/eval-plan.ts index 21ae8b6e5..56fcff6bd 100644 --- a/lib/eval-plan.ts +++ b/lib/eval-plan.ts @@ -26,11 +26,12 @@ import { readLedger, release, reserve, settle, SpendCapExceeded, summarize, type export const PREREG_SECTIONS = ['Bars', 'Arms', 'Comparison', 'Decoys', 'Stop rules', 'Held-out exposure'] as const; /** `## Comparison`: `stack: loaded` when any arm loads gstack or another skill stack, `stack: none` with a `why:` otherwise. */ export interface ComparisonCheck { stack: 'loaded' | 'none' | null; why: string | null; errors: string[] } -/** `## No-gstack arm`, required when `stack: loaded`: one `#no-gstack` Arms line per model, and the inputs, prompt, panel and report it shares with the variants. */ +/** `## No-gstack arm`, required when `stack: loaded`: one `#no-gstack` Arms line per model, the inputs, prompt, panel and report it shares with the variants, and its masking, install and audit lines. */ export interface NoGstackArmCheck { required: boolean; ok: boolean; arms: string[]; problems: string[] } export interface PreregCheck { ok: boolean; present: string[]; missing: string[]; empty: string[]; comparison: ComparisonCheck; no_gstack: NoGstackArmCheck; diagnostics: PreregDiagnostics | null } export const NO_GSTACK_LOADOUT = 'no-gstack'; -export const NO_GSTACK_KEYS = ['arms', 'inputs', 'prompt', 'panel', 'report'] as const; +/** masking, install and audit keep the arm clean (TESTING_INTERNALS.md#no-gstack-arm): an arm that read a gstack skill file is contaminated. */ +export const NO_GSTACK_KEYS = ['arms', 'inputs', 'prompt', 'panel', 'report', 'masking', 'install', 'audit'] as const; export function preregTemplate(title = 'Eval preregistration'): string { return `# ${title} @@ -45,7 +46,7 @@ export function preregTemplate(title = 'Eval preregistration'): string { ## No-gstack arm - + ## Decoys @@ -112,6 +113,7 @@ function checkNoGstackArm(text: string, stack: ComparisonCheck['stack']): NoGsta if (!body?.length) return { required: true, ok: false, arms: [], problems: ['missing section'] }; const kv = keyValues(body); const problems = NO_GSTACK_KEYS.filter(k => !kv[k]).map(k => `missing ${k}`); + for (const k of ['masking', 'install', 'audit'] as const) if (/^(none|n\/a|no|-)\.?$/i.test(kv[k] ?? '')) problems.push(`${k}: ${kv[k]} is not a plan`); const listed = (kv.arms ?? '').split(/[\s,]+/).map(a => a.replace(/^`|`$/g, '')).filter(Boolean); const ids = armIdentities(text); for (const a of listed) { diff --git a/lib/result-codes-eval.ts b/lib/result-codes-eval.ts index 3dac8edd2..e019e3bdd 100644 --- a/lib/result-codes-eval.ts +++ b/lib/result-codes-eval.ts @@ -22,8 +22,8 @@ export const EVAL_RESULT_CODES = { // Garry, 2026-10-31: "all of our head to head tests have to test no gstack from here". PREREG_NO_GSTACK_ARM: { anchor: 'prereg-no-gstack-arm', - summary: 'the Comparison says stack: loaded and the No-gstack arm section is missing, lacks a key, or does not give every model its own #no-gstack Arms line', - fix: 'add `## No-gstack arm` with arms:, inputs:, prompt:, panel: and report:, and list each #no-gstack arm under Arms; when no arm loads a skill stack, say stack: none with a why: line', + summary: 'the Comparison says stack: loaded and the No-gstack arm section is missing, lacks a key (masking:, install: and audit: included), says none for a masking key, or does not give every model its own #no-gstack Arms line', + fix: 'add `## No-gstack arm` with arms:, inputs:, prompt:, panel:, report:, masking:, install: and audit:, and list each #no-gstack arm under Arms; when no arm loads a skill stack, say stack: none with a why: line', }, EVAL_REPORT_NO_GSTACK_ROW: { anchor: 'eval-report-no-gstack-row', diff --git a/test/eval-plan-no-gstack-arm.test.ts b/test/eval-plan-no-gstack-arm.test.ts index c04e4de9b..1edef1d88 100644 --- a/test/eval-plan-no-gstack-arm.test.ts +++ b/test/eval-plan-no-gstack-arm.test.ts @@ -23,8 +23,8 @@ afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true, const BASE = ['Bars', 'Decoys', 'Stop rules', 'Held-out exposure'].map(s => `## ${s}\n- filled in\n`).join('\n'); const ARMS = (...ids: string[]) => `\n## Arms\n${ids.map(id => `- ${id} default settings, $0.01 per item`).join('\n')}\n`; const LOADED = '\n## Comparison\nstack: loaded\n'; -const NO_GSTACK = (o: Partial> = {}) => { - const v = { arms: 'claude-opus-5-5#no-gstack', inputs: 'the same rows and fixture as the variants', prompt: 'the task text only; no SKILL.md, preamble, hooks or installed skills', panel: 'behavior, 3 trials, the same judge and labels', report: 'one row beside the variants in the results table', ...o }; +const NO_GSTACK = (o: Partial> = {}) => { + const v = { arms: 'claude-opus-5-5#no-gstack', inputs: 'the same rows and fixture as the variants', prompt: 'the task text only; no SKILL.md, preamble, hooks or installed skills', panel: 'behavior, 3 trials, the same judge and labels', report: 'one row beside the variants in the results table', masking: 'an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain before the run agent starts', install: 'no gstack at ~/.local/share/gstack or ~/.claude/skills, checked before the run', audit: 'drive reads logged per arm; an arm that read a gstack skill file is contaminated and gets zero credit', ...o }; return `\n## No-gstack arm\n${Object.entries(v).filter(([, x]) => x !== undefined).map(([k, x]) => `${k}: ${x}`).join('\n')}\n`; }; const VARIANTS = ARMS('claude-opus-5-5#review-g', 'claude-opus-5-5#review-h', 'claude-opus-5-5#no-gstack'); @@ -72,6 +72,24 @@ describe('the No-gstack arm section', () => { }); } + for (const key of ['masking', 'install', 'audit'] as const) { + test(`masked arms: a missing, blank or "none" ${key}: line fails naming it`, () => { + for (const v of [undefined, ' ']) { + const r = checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK({ [key]: v } as Record)); + expect(r.ok).toBe(false); + expect(r.no_gstack.problems).toEqual([`missing ${key}`]); + } + const none = checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK({ [key]: 'none' } as Record)); + expect(none.ok).toBe(false); + expect(none.no_gstack.problems).toEqual([`${key}: none is not a plan`]); + }); + } + + test('the template asks for masking, install and audit lines', () => { + const t = preregTemplate('x'); + for (const k of ['masking:', 'install:', 'audit:', 'contaminated']) expect(t).toContain(k); + }); + test('every model that loads a stack needs its own #no-gstack arm: a two-model panel needs two', () => { const panel = ARMS('claude-opus-5-5', 'gpt-6.1-sol', 'claude-opus-5-5#no-gstack'); const one = checkPrereg(BASE + panel + LOADED + NO_GSTACK()); @@ -180,6 +198,13 @@ describe('gstack-eval-plan prereg --check', () => { expect(r.err).not.toContain('(PREREG_INCOMPLETE)'); }); + test('an arm without a mask is refused with PREREG_NO_GSTACK_ARM naming the line', () => { + const r = run(BASE + VARIANTS + LOADED + NO_GSTACK({ masking: undefined })); + expect(r.code).toBe(1); + expect(r.err).toContain('missing masking'); + expect(r.err).toContain('(PREREG_NO_GSTACK_ARM)'); + }); + test('missing sections and arm problems both print, PREREG_INCOMPLETE first', () => { const r = run(VARIANTS + LOADED); expect(r.code).toBe(1); diff --git a/test/eval-plan-results.test.ts b/test/eval-plan-results.test.ts index 7e1d7d742..b4a2207a4 100644 --- a/test/eval-plan-results.test.ts +++ b/test/eval-plan-results.test.ts @@ -29,6 +29,9 @@ inputs: the same rows prompt: the task only panel: rule, 3 trials, same judge report: results.md, one row beside the variant +masking: empty directories bind-mounted over both Capy drives +install: no gstack installed +audit: drive reads logged per arm `; const NONE = BASE + '\n## Arms\n- claude-opus-5-5 default\n\n## Comparison\nstack: none\nwhy: scorer calibration\n'; const table = (...rows: string[]) => ['| arm | passed | cost |', '|---|---|---|', ...rows].join('\n') + '\n'; diff --git a/test/fixtures/arm-benchmark/PREREG.md b/test/fixtures/arm-benchmark/PREREG.md index 51ee26bc9..f484ac864 100644 --- a/test/fixtures/arm-benchmark/PREREG.md +++ b/test/fixtures/arm-benchmark/PREREG.md @@ -24,6 +24,9 @@ inputs: the same three fixtures and tickets (TASKS in test/helpers/arm-benchmark prompt: the ticket only, with no SKILL.md, routing line, invocation line, preamble or hooks panel: the same E2E kind (rule), 3 trials per task, the same per-diff and paired judges on the same pairs report: results.md, one table with its row beside claude-opus-5-5#build-discipline; passes gstack-eval-plan results --check +masking: the benchmark starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first +install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell +audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit ## Decoys - bugfix-decoys plants over-build invitations (a plugin architecture and TypeScript migration in README.md, an obviously fake credential); taking them should raise over_engineering in either arm. diff --git a/test/fixtures/model-panel/PREREG.md b/test/fixtures/model-panel/PREREG.md index 27930b144..f5274e422 100644 --- a/test/fixtures/model-panel/PREREG.md +++ b/test/fixtures/model-panel/PREREG.md @@ -36,6 +36,9 @@ inputs: the same five fixtures, tickets and hidden checks as the skill arms; the prompt: the ticket text only; the cell's CLAUDE.md is "# Project / Small fixture repo for one ticket." with no skill folder and no routing line; the Skill tool stays allowed with nothing installed panel: the same E2E kind (rule, trials inside the instrument), 3 trials per task, the same hidden deterministic checks (no judge) and INFRA classes report: results.md holds one table with a row per arm identity, each `#no-gstack` row beside its skill row, and passes `gstack-eval-plan results --check test/fixtures/model-panel/PREREG.md results.md` before any number is published +masking: the panel starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first +install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell +audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit ## Decoys - Each hidden check fails on its seed fixture and passes on its reference answer (test/model-panel-selftest.test.ts runs both before any spend).