* eval-plan: a no-gstack arm needs masking, install and audit lines prereg --check now refuses a loaded comparison whose ## No-gstack arm section lacks masking:, install: or audit:, or says none for one of them (PREREG_NO_GSTACK_ARM). On Capy, run agents inherit the instruction to load the user-drive gstack skill, so the arm hides both Capy drives, checks that no gstack is installed, and audits drive reads; an arm that read a gstack skill file is contaminated and gets zero credit (TESTING_INTERNALS.md #no-gstack-arm). The prereg template, /eval-plan, troubleshooting and the two passing prereg fixtures carry the new lines. eval-plan's paragraph was dieted to hold its context-budget ceiling. Evidence: safety: keeps the no-gstack control uncontaminated (Captain order, FOUN-32 via GGST-6) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE docs/inventory/evidence.json holds one row per inventory key: the skill folders discoverTemplates() finds (root skill as gstack, symlinked folders excluded) plus seven shared parts, 71 keys. lib/inventory.ts validates each row by its status's own fields, with named codes (LEDGER_*), and joins the ledger with lib/context-bill.ts buildBill. bin/gstack-inventory writes docs/inventory.md, --check validates (exit 1 with codes), --json prints. Paid dispatches per run for /ship and /autoplan are quoted from their own docs and found by line, so a moved citation fails --check (INVENTORY_SOURCE_MISSING). Every current row is grandfathered and seeded from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json): in-flight fair tests and proposed registry rows as plans, earlier unfair results under prior with why (stub, unmasked, ceiling, ...), five safety rows. gstack-skill-bill prints the skill's EVIDENCE line. Evidence: plan none (owner ruling R56, GST-914); the ledger itself is the measurement record Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ETHOS 9, Earn Its Place: factory, /write-skill and /autoplan carry it Garry, 2026-11-02: "Also work this principle of only include it if it is beneficial and good into gstack". ETHOS.md principle 9 states the rule, the fair test (claude_code preset, frontier models per R54, masked no-gstack arm), the R48 Why-null cycle, the safety exception, and measure biggest first, linking docs/inventory.md. factory.md item 8 cites it, trimmed elsewhere to 3,298 of 3,300 bytes with no rule dropped. /write-skill's edit ladder step 2 is the evidence rule and section 4 adds the Evidence: commit line. /autoplan's CEO phase lists each part a plan adds to gstack with its evidence or owned, dated plan; two redundant lines went to keep autoplan under its size pins (pins lowered by 17). CONTRIBUTING.md "Adding a new skill" and docs/skills.md point at the ledger. Skill-Size: 3273→3666 bytes (write-skill/SKILL.md.tmpl) Skill-Size: 12738→12721 bytes (autoplan/sections/ceo-phase.md) Evidence: plan none (owner ruling R56, GST-914); write-skill and autoplan rows stay grandfathered untested Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fixup! inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: list the evidence table's other names and its unmapped rows Ledger rows carry aliases (the name GEVA-2's table uses where it differs from the key, such as upgrade for gstack-upgrade), and docs/inventory.md lists them beside the installed-name rule. Unmapped rows are labeled as evidence-table rows, and the ranking notes that a skill's on-demand files can also be a shared row's. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the first inventory snapshot and the R56 changelog fragment docs/inventory.md is generated by bin/gstack-inventory at the previous commit: 71 parts ranked by load per run, 0 helps, 0 no-effect, 0 hurts, 16 untested with a plan (4 fair tests under way, 12 proposed), 50 untested with plan none, 5 safety. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's step procedure is a fair-test CUT-CANDIDATE (GST-814) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after investigate's no-effect row Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: only the seeded backlog may be grandfathered, so a new part's row is checked (review fix) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: devex-review and retro name their GEVA-2 registry rows (R56-11, R56-12) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: a concrete plan is a path, gstack-inventory never writes into a gstack install, and the ledger checks aliases, r48 and prior.why (review fixes) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's no-effect row notes that only Opus 5.5 was measured Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: R56 counts after investigate's row; grandfathering and the install guard Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register inventory-grandfathered.json as a free fixture free-suite (1) ate56d3312cfailed FREE_FIXTURES inventory: the pinned grandfather fixture fromd0b3028b5had no paid touchfile and no FREE_FIXTURES entry. Its only reader is test/inventory.test.ts. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
5.8 KiB
Model panel: four models, each with and without the skill (GST-468)
Supersedes the GSTA-173 preregistration (2026-10-30, commit d209821) for every
panel run after the commit that adds this file. The line GSTA-173 published
stays tied to its own run (2026-10-30-0105, four skill arms, no no-gstack arm).
Frozen before the first run under it; nothing below changes after a result is
seen. Instrument: test/skill-e2e-model-panel.test.ts with
test/helpers/model-panel-harness.ts; free selftest:
test/model-panel-selftest.test.ts.
Bars
- Pass rate per arm: scored (non-INFRA) cells whose hidden deterministic check passes, with a Wilson 95% interval. Descriptive only: the panel gates nothing and no arm "wins".
- Publication floor: an arm needs PANEL_MIN_COMPLETED = 12 scored cells before any number about it is published; below that its row says INCOMPLETE.
- Skill effect: a model's skill arm and its no-gstack arm differ only when their Wilson intervals do not overlap; otherwise the report says "no measured difference on this panel".
- Ceiling: GSTA-173 passed 58 of 59 scored cells, so this panel is near ceiling. A no-difference result between a skill arm and its no-gstack arm says nothing about whether the skill adds value, and the report must not word it as "gstack adds nothing".
- Suggestion line: the GSTA-173 wording rule for particle-1 (suggestionLine, usefulness floor 0.6) is unchanged and reads the four skill arms only.
- Harness bar: every no-gstack cell records consulted=false (no skill is installed to consult); a true value fails the task's harness assertion.
Arms
- claude-opus-5-5 Opus 5.5, Anthropic route, build-discipline skill and its CLAUDE.md routing line, reserve $2.00 per cell
- claude-opus-5-5#no-gstack Opus 5.5, Anthropic route, plain CLAUDE.md, no skill, reserve $2.00 per cell
- claude-sonnet-5-5 Sonnet 5.5, Anthropic route, build-discipline skill and routing line, reserve $1.00 per cell
- claude-sonnet-5-5#no-gstack Sonnet 5.5, Anthropic route, plain CLAUDE.md, no skill, reserve $1.00 per cell
- particle-ai/particle-1 particle-1, OpenRouter route with no Anthropic credential, build-discipline skill and routing line, reserve $0.15 per cell
- particle-ai/particle-1#no-gstack particle-1, OpenRouter route with no Anthropic credential, plain CLAUDE.md, no skill, reserve $0.15 per cell
- deepseek/deepseek-v4.1-flash-1017 DeepSeek V4.1 Flash, OpenRouter route with no Anthropic credential, build-discipline skill and routing line, reserve $0.15 per cell
- deepseek/deepseek-v4.1-flash-1017#no-gstack DeepSeek V4.1 Flash, OpenRouter route with no Anthropic credential, plain CLAUDE.md, no skill, reserve $0.15 per cell
Comparison
stack: loaded why: the skill arms load the build-discipline skill (extracted from ship/SKILL.md) and a routing line telling the model to invoke it
No-gstack arm
arms: claude-opus-5-5#no-gstack, claude-sonnet-5-5#no-gstack, particle-ai/particle-1#no-gstack, deepseek/deepseek-v4.1-flash-1017#no-gstack
inputs: the same five fixtures, tickets and hidden checks as the skill arms; the same model, route, credential handling (armEnv by route), tools, turn limit, timeout and spend reservation; every arm runs in Claude Code with max 40 turns, an 8-minute timeout and the tools Bash, Read, Write, Edit, Glob, Grep and Skill, 5 tasks x 3 trials = 15 cells per arm, 120 cells in all
prompt: the ticket text only; the cell's CLAUDE.md is "# Project / Small fixture repo for one ticket." with no skill folder and no routing line; the Skill tool stays allowed with nothing installed
panel: the same E2E kind (rule, trials inside the instrument), 3 trials per task, the same hidden deterministic checks (no judge) and INFRA classes
report: results.md holds one table with a row per arm identity, each <model>#no-gstack row beside its skill row, and passes gstack-eval-plan results --check test/fixtures/model-panel/PREREG.md results.md before any number is published
masking: the panel starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first
install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell
audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit
Decoys
- Each hidden check fails on its seed fixture and passes on its reference answer (test/model-panel-selftest.test.ts runs both before any spend).
- model-panel-bugfix carries decoy code paths next to the real bug; a fix in a decoy leaves the check red.
- model-panel-review seeds bugs that pass a quick read; a review that edits source files fails.
- model-panel-changelog lists build, test and tooling commits in COMMITS.md; an entry that includes them fails.
- model-panel-investigate wraps the symptom in a retry; a fix that only retries or edits the tests fails.
Stop rules
- Spend cap $15 (PANEL_SPEND_CAP_USD) in one ledger (/spend.json). Estimated spend $4-5: GSTA-173 spent $2.09 plus $0.15 at reservation for 60 cells, and this run has 120. A refused reservation stops the panel with STOP: reason=budget_exhausted and keeps every row already written.
- INFRA cells (startup timeout, no result event, MODEL_MIX, 402/429/5xx, error before the first turn) are excluded from pass rates, reported, and never rerun to change a verdict.
- One campaign per run id and PREREG.md sha256; a cell recorded twice in a campaign is refused, and rows from other campaigns are never pooled.
- No rerun after a result is seen.
Held-out exposure
| set | prior exposure | reservation |
|---|---|---|
| the five model-panel tasks (fixtures, tickets, checks) | GSTA-173 run 2026-10-30-0105: Opus 5.5, Sonnet 5.5, particle-1 and DeepSeek V4.1 Flash, skill loadout only | not held out; this run reads all five for all eight arms |