Files
gstack/test/fixtures/model-panel/PREREG.md
T
Garry Tanandcapy-ai[bot] ade35bc813 v1.94.29.0 R56: everything in gstack earns its place (principle, evidence gates, inventory) (#3837)
* eval-plan: a no-gstack arm needs masking, install and audit lines

prereg --check now refuses a loaded comparison whose ## No-gstack arm
section lacks masking:, install: or audit:, or says none for one of them
(PREREG_NO_GSTACK_ARM). On Capy, run agents inherit the instruction to load
the user-drive gstack skill, so the arm hides both Capy drives, checks that
no gstack is installed, and audits drive reads; an arm that read a gstack
skill file is contaminated and gets zero credit (TESTING_INTERNALS.md
#no-gstack-arm). The prereg template, /eval-plan, troubleshooting and the
two passing prereg fixtures carry the new lines. eval-plan's paragraph was
dieted to hold its context-budget ceiling.

Evidence: safety: keeps the no-gstack control uncontaminated (Captain order, FOUN-32 via GGST-6)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE

docs/inventory/evidence.json holds one row per inventory key: the skill
folders discoverTemplates() finds (root skill as gstack, symlinked folders
excluded) plus seven shared parts, 71 keys. lib/inventory.ts validates each
row by its status's own fields, with named codes (LEDGER_*), and joins the
ledger with lib/context-bill.ts buildBill. bin/gstack-inventory writes
docs/inventory.md, --check validates (exit 1 with codes), --json prints.
Paid dispatches per run for /ship and /autoplan are quoted from their own
docs and found by line, so a moved citation fails --check
(INVENTORY_SOURCE_MISSING). Every current row is grandfathered and seeded
from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json):
in-flight fair tests and proposed registry rows as plans, earlier unfair
results under prior with why (stub, unmasked, ceiling, ...), five safety
rows. gstack-skill-bill prints the skill's EVIDENCE line.

Evidence: plan none (owner ruling R56, GST-914); the ledger itself is the measurement record
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* ETHOS 9, Earn Its Place: factory, /write-skill and /autoplan carry it

Garry, 2026-11-02: "Also work this principle of only include it if it is
beneficial and good into gstack". ETHOS.md principle 9 states the rule, the
fair test (claude_code preset, frontier models per R54, masked no-gstack
arm), the R48 Why-null cycle, the safety exception, and measure biggest
first, linking docs/inventory.md. factory.md item 8 cites it, trimmed
elsewhere to 3,298 of 3,300 bytes with no rule dropped. /write-skill's edit
ladder step 2 is the evidence rule and section 4 adds the Evidence: commit
line. /autoplan's CEO phase lists each part a plan adds to gstack with its
evidence or owned, dated plan; two redundant lines went to keep autoplan
under its size pins (pins lowered by 17). CONTRIBUTING.md "Adding a new
skill" and docs/skills.md point at the ledger.

Skill-Size: 3273→3666 bytes (write-skill/SKILL.md.tmpl)
Skill-Size: 12738→12721 bytes (autoplan/sections/ceo-phase.md)
Evidence: plan none (owner ruling R56, GST-914); write-skill and autoplan rows stay grandfathered untested
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fixup! inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: list the evidence table's other names and its unmapped rows

Ledger rows carry aliases (the name GEVA-2's table uses where it differs from the key, such as upgrade for gstack-upgrade), and docs/inventory.md lists them beside the installed-name rule. Unmapped rows are labeled as evidence-table rows, and the ranking notes that a skill's on-demand files can also be a shared row's.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the first inventory snapshot and the R56 changelog fragment

docs/inventory.md is generated by bin/gstack-inventory at the previous commit: 71 parts ranked by load per run, 0 helps, 0 no-effect, 0 hurts, 16 untested with a plan (4 fair tests under way, 12 proposed), 50 untested with plan none, 5 safety.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: investigate's step procedure is a fair-test CUT-CANDIDATE (GST-814)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the inventory snapshot after investigate's no-effect row

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: only the seeded backlog may be grandfathered, so a new part's row is checked (review fix)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: devex-review and retro name their GEVA-2 registry rows (R56-11, R56-12)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the inventory snapshot after the review fixes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: a concrete plan is a path, gstack-inventory never writes into a gstack install, and the ledger checks aliases, r48 and prior.why (review fixes)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* inventory: investigate's no-effect row notes that only Opus 5.5 was measured

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: the inventory snapshot after the review fixes

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: R56 counts after investigate's row; grandfathering and the install guard

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: register inventory-grandfathered.json as a free fixture

free-suite (1) at e56d3312c failed FREE_FIXTURES inventory: the pinned grandfather fixture from d0b3028b5 had no paid touchfile and no FREE_FIXTURES entry. Its only reader is test/inventory.test.ts.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-03 17:08:30 +00:00

5.8 KiB

Model panel: four models, each with and without the skill (GST-468)

Supersedes the GSTA-173 preregistration (2026-10-30, commit d209821) for every panel run after the commit that adds this file. The line GSTA-173 published stays tied to its own run (2026-10-30-0105, four skill arms, no no-gstack arm). Frozen before the first run under it; nothing below changes after a result is seen. Instrument: test/skill-e2e-model-panel.test.ts with test/helpers/model-panel-harness.ts; free selftest: test/model-panel-selftest.test.ts.

Bars

  • Pass rate per arm: scored (non-INFRA) cells whose hidden deterministic check passes, with a Wilson 95% interval. Descriptive only: the panel gates nothing and no arm "wins".
  • Publication floor: an arm needs PANEL_MIN_COMPLETED = 12 scored cells before any number about it is published; below that its row says INCOMPLETE.
  • Skill effect: a model's skill arm and its no-gstack arm differ only when their Wilson intervals do not overlap; otherwise the report says "no measured difference on this panel".
  • Ceiling: GSTA-173 passed 58 of 59 scored cells, so this panel is near ceiling. A no-difference result between a skill arm and its no-gstack arm says nothing about whether the skill adds value, and the report must not word it as "gstack adds nothing".
  • Suggestion line: the GSTA-173 wording rule for particle-1 (suggestionLine, usefulness floor 0.6) is unchanged and reads the four skill arms only.
  • Harness bar: every no-gstack cell records consulted=false (no skill is installed to consult); a true value fails the task's harness assertion.

Arms

  • claude-opus-5-5 Opus 5.5, Anthropic route, build-discipline skill and its CLAUDE.md routing line, reserve $2.00 per cell
  • claude-opus-5-5#no-gstack Opus 5.5, Anthropic route, plain CLAUDE.md, no skill, reserve $2.00 per cell
  • claude-sonnet-5-5 Sonnet 5.5, Anthropic route, build-discipline skill and routing line, reserve $1.00 per cell
  • claude-sonnet-5-5#no-gstack Sonnet 5.5, Anthropic route, plain CLAUDE.md, no skill, reserve $1.00 per cell
  • particle-ai/particle-1 particle-1, OpenRouter route with no Anthropic credential, build-discipline skill and routing line, reserve $0.15 per cell
  • particle-ai/particle-1#no-gstack particle-1, OpenRouter route with no Anthropic credential, plain CLAUDE.md, no skill, reserve $0.15 per cell
  • deepseek/deepseek-v4.1-flash-1017 DeepSeek V4.1 Flash, OpenRouter route with no Anthropic credential, build-discipline skill and routing line, reserve $0.15 per cell
  • deepseek/deepseek-v4.1-flash-1017#no-gstack DeepSeek V4.1 Flash, OpenRouter route with no Anthropic credential, plain CLAUDE.md, no skill, reserve $0.15 per cell

Comparison

stack: loaded why: the skill arms load the build-discipline skill (extracted from ship/SKILL.md) and a routing line telling the model to invoke it

No-gstack arm

arms: claude-opus-5-5#no-gstack, claude-sonnet-5-5#no-gstack, particle-ai/particle-1#no-gstack, deepseek/deepseek-v4.1-flash-1017#no-gstack inputs: the same five fixtures, tickets and hidden checks as the skill arms; the same model, route, credential handling (armEnv by route), tools, turn limit, timeout and spend reservation; every arm runs in Claude Code with max 40 turns, an 8-minute timeout and the tools Bash, Read, Write, Edit, Glob, Grep and Skill, 5 tasks x 3 trials = 15 cells per arm, 120 cells in all prompt: the ticket text only; the cell's CLAUDE.md is "# Project / Small fixture repo for one ticket." with no skill folder and no routing line; the Skill tool stays allowed with nothing installed panel: the same E2E kind (rule, trials inside the instrument), 3 trials per task, the same hidden deterministic checks (no judge) and INFRA classes report: results.md holds one table with a row per arm identity, each <model>#no-gstack row beside its skill row, and passes gstack-eval-plan results --check test/fixtures/model-panel/PREREG.md results.md before any number is published masking: the panel starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit

Decoys

  • Each hidden check fails on its seed fixture and passes on its reference answer (test/model-panel-selftest.test.ts runs both before any spend).
  • model-panel-bugfix carries decoy code paths next to the real bug; a fix in a decoy leaves the check red.
  • model-panel-review seeds bugs that pass a quick read; a review that edits source files fails.
  • model-panel-changelog lists build, test and tooling commits in COMMITS.md; an entry that includes them fails.
  • model-panel-investigate wraps the symptom in a retry; a fix that only retries or edits the tests fails.

Stop rules

  • Spend cap $15 (PANEL_SPEND_CAP_USD) in one ledger (/spend.json). Estimated spend $4-5: GSTA-173 spent $2.09 plus $0.15 at reservation for 60 cells, and this run has 120. A refused reservation stops the panel with STOP: reason=budget_exhausted and keeps every row already written.
  • INFRA cells (startup timeout, no result event, MODEL_MIX, 402/429/5xx, error before the first turn) are excluded from pass rates, reported, and never rerun to change a verdict.
  • One campaign per run id and PREREG.md sha256; a cell recorded twice in a campaign is refused, and rows from other campaigns are never pooled.
  • No rerun after a result is seen.

Held-out exposure

set prior exposure reservation
the five model-panel tasks (fixtures, tickets, checks) GSTA-173 run 2026-10-30-0105: Opus 5.5, Sonnet 5.5, particle-1 and DeepSeek V4.1 Flash, skill loadout only not held out; this run reads all five for all eight arms