mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-08 17:58:13 +01:00
69f0fe181aa19ffda85591e8a765f2c0b3e6cdfd
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Garry Tanandcapy-ai[bot]
|
ade35bc813 |
v1.94.29.0 R56: everything in gstack earns its place (principle, evidence gates, inventory) (#3837)
* eval-plan: a no-gstack arm needs masking, install and audit lines prereg --check now refuses a loaded comparison whose ## No-gstack arm section lacks masking:, install: or audit:, or says none for one of them (PREREG_NO_GSTACK_ARM). On Capy, run agents inherit the instruction to load the user-drive gstack skill, so the arm hides both Capy drives, checks that no gstack is installed, and audits drive reads; an arm that read a gstack skill file is contaminated and gets zero credit (TESTING_INTERNALS.md #no-gstack-arm). The prereg template, /eval-plan, troubleshooting and the two passing prereg fixtures carry the new lines. eval-plan's paragraph was dieted to hold its context-budget ceiling. Evidence: safety: keeps the no-gstack control uncontaminated (Captain order, FOUN-32 via GGST-6) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE docs/inventory/evidence.json holds one row per inventory key: the skill folders discoverTemplates() finds (root skill as gstack, symlinked folders excluded) plus seven shared parts, 71 keys. lib/inventory.ts validates each row by its status's own fields, with named codes (LEDGER_*), and joins the ledger with lib/context-bill.ts buildBill. bin/gstack-inventory writes docs/inventory.md, --check validates (exit 1 with codes), --json prints. Paid dispatches per run for /ship and /autoplan are quoted from their own docs and found by line, so a moved citation fails --check (INVENTORY_SOURCE_MISSING). Every current row is grandfathered and seeded from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json): in-flight fair tests and proposed registry rows as plans, earlier unfair results under prior with why (stub, unmasked, ceiling, ...), five safety rows. gstack-skill-bill prints the skill's EVIDENCE line. Evidence: plan none (owner ruling R56, GST-914); the ledger itself is the measurement record Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ETHOS 9, Earn Its Place: factory, /write-skill and /autoplan carry it Garry, 2026-11-02: "Also work this principle of only include it if it is beneficial and good into gstack". ETHOS.md principle 9 states the rule, the fair test (claude_code preset, frontier models per R54, masked no-gstack arm), the R48 Why-null cycle, the safety exception, and measure biggest first, linking docs/inventory.md. factory.md item 8 cites it, trimmed elsewhere to 3,298 of 3,300 bytes with no rule dropped. /write-skill's edit ladder step 2 is the evidence rule and section 4 adds the Evidence: commit line. /autoplan's CEO phase lists each part a plan adds to gstack with its evidence or owned, dated plan; two redundant lines went to keep autoplan under its size pins (pins lowered by 17). CONTRIBUTING.md "Adding a new skill" and docs/skills.md point at the ledger. Skill-Size: 3273→3666 bytes (write-skill/SKILL.md.tmpl) Skill-Size: 12738→12721 bytes (autoplan/sections/ceo-phase.md) Evidence: plan none (owner ruling R56, GST-914); write-skill and autoplan rows stay grandfathered untested Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fixup! inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: list the evidence table's other names and its unmapped rows Ledger rows carry aliases (the name GEVA-2's table uses where it differs from the key, such as upgrade for gstack-upgrade), and docs/inventory.md lists them beside the installed-name rule. Unmapped rows are labeled as evidence-table rows, and the ranking notes that a skill's on-demand files can also be a shared row's. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the first inventory snapshot and the R56 changelog fragment docs/inventory.md is generated by bin/gstack-inventory at the previous commit: 71 parts ranked by load per run, 0 helps, 0 no-effect, 0 hurts, 16 untested with a plan (4 fair tests under way, 12 proposed), 50 untested with plan none, 5 safety. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's step procedure is a fair-test CUT-CANDIDATE (GST-814) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after investigate's no-effect row Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: only the seeded backlog may be grandfathered, so a new part's row is checked (review fix) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: devex-review and retro name their GEVA-2 registry rows (R56-11, R56-12) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: a concrete plan is a path, gstack-inventory never writes into a gstack install, and the ledger checks aliases, r48 and prior.why (review fixes) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's no-effect row notes that only Opus 5.5 was measured Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: R56 counts after investigate's row; grandfathering and the install guard Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register inventory-grandfathered.json as a free fixture free-suite (1) at |
||
|
Garry Tanandcapy-ai[bot]
|
4eff47c5cf |
v1.92.2.0 evals: no-gstack arms for the model panel, arm-benchmark, safety and routing evals (#3708)
* feat(eval-plan): results --check refuses a results table without its no-gstack rows Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(model-panel): a <model>#no-gstack arm per model, identity-keyed cells, one campaign, results.md (GST-468) Each panel model also runs as <model>#no-gstack: the same fixture, ticket, tools, limits, route and reservation, with no skill folder and no routing line. The arm identity keys the ledger attempt, test name, row and summary; candidate folders are named after the fixture project and the seed commit has a neutral author. summarize reads one campaign (run id and PREREG.md sha256) and refuses a duplicate cell; suggestionLine reads the skill arms only, so the preregistered particle-1 line is unchanged. renderResults writes results.md, which passes gstack-eval-plan results --check against the new test/fixtures/model-panel/PREREG.md (supersedes GSTA-173 for later runs). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(arm-benchmark): no-gstack arm on a pinned model, 3 paired trials, both-order paired judge, spend ledger (GST-469) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(evals): no-gstack arm for the safety-rule and router evals (GST-470) GSTACK_SAFETY_ARM and GSTACK_ROUTER_ARM take a third value, none: the same fixture, tools and user request with no skill excerpt, rule text, step file or installed skill. Under none every behavioral check is recorded as an observation (none-arm-observations.jsonl) and never fails the case or writes contract-violations.jsonl; harness preconditions still assert. The four safety prompt builders (ship-evidence, codex-boundary, codex-consult, design-risk) and the router fixtures move into helpers; the rule, removed, current and before prompts stay byte-identical, proven against reference copies of main's code. pair-agent and ios-demo refuse none with the reason (NO_NONE_ARM). GSTACK_ROUTER_ARM now throws on an unknown value instead of silently running current. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: changelog fragment for the no-gstack arms in existing evals Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(arm-benchmark): arm-pair paired-judge calibration run (bar met 6/6, $7.22) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(paid-shards): keep loop-named panel and arm-benchmark cases selectable by --case The lanes moved each cell's testName into helpers, so the static detector no longer saw a computed registration and B4 failed for all eight cases. Same runtime test names; the paid runs are unaffected. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(tests): keep FREE_FIXTURES and the placeholder allowlist current for the no-gstack arms scripts/judge-calibration.ts is now an arm-benchmark paid touchfile, so it can no longer be a free consumer of test/fixtures/judge-calibration/** (CI free-suite 8). The eval-plan Step 5 fence names <prereg-path> and <results-path>; allowlist both as PATH placeholders (CI free-suite 19). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(eval-plan): cover a complete table after a partial one, --json output and the results usage error Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(eval-plan): keep the Step 7 coverage test as reviewed Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: name the changelog fragment after the branch slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: arm-benchmark project folder names and the arm-pair calibration corpus Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: keep the judge-calibration fixture README as reviewed Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(skills): describe eval-plan results --check Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> |
||
|
Garry Tanandcapy-ai[bot]
|
0cd2b4a9f9 |
v1.91.373.0 feat(eval-plan): require a no-gstack arm in every head-to-head preregistration (#3690)
* feat(eval-plan): require a no-gstack arm in every head-to-head preregistration Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(eval-plan): lint loadout arms by their model; refuse arms sharing a ledger attempt Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> |