From 274fb34537edc2d90e021a27d91a0a7bc61125f8 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Fri, 6 Nov 2026 06:36:50 -0800 Subject: [PATCH] docs(inventory): record L1 verdicts on untold routing and told /investigate (GST-1629) (#4076) * docs(inventory): record L1 verdicts on untold routing and told /investigate (GST-1629) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(inventory): regenerate the snapshot with L1's verdicts (GST-1629) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- changelog.d/capy-gst-1629-l1-inventory.md | 1 + docs/inventory.md | 45 ++++++++++++----------- docs/inventory/evidence.json | 14 ++++++- 3 files changed, 36 insertions(+), 24 deletions(-) create mode 100644 changelog.d/capy-gst-1629-l1-inventory.md diff --git a/changelog.d/capy-gst-1629-l1-inventory.md b/changelog.d/capy-gst-1629-l1-inventory.md new file mode 100644 index 000000000..a44572dec --- /dev/null +++ b/changelog.d/capy-gst-1629-l1-inventory.md @@ -0,0 +1 @@ +- **The R56 inventory records L1's verdicts on gstack's untold routing and /investigate's told record (GST-1629).** GEVA-2's L1 run (gstack-evals#247; Opus 5.5 on the claude_code preset, 80 SWE-rebench tasks, a no-gstack arm, pins 82db89ab to 699a41cb) is now in `docs/inventory/evidence.json` and the regenerated `docs/inventory.md`: the untold routing package (route hook, contracts and the description rewrite) helps, B1 +21.3 points [+12.5, +30.0], KEEP, and needs routing or contracts on; the description rewrite alone had no effect (0 of 80); told record completeness is untested, B3 +10.0 [0, +20], FIX-METHOD because probe I failed its blind audit; pass rate O5 +5.0 at a 65% control, MDD 18.7, is descriptive only. /investigate keeps its no-effect status and CUT-CANDIDATE, which measured something else. diff --git a/docs/inventory.md b/docs/inventory.md index 1c5e0d19d..dc8bb8c0b 100644 --- a/docs/inventory.md +++ b/docs/inventory.md @@ -1,6 +1,6 @@ # gstack inventory: every part, its load per run and its evidence -Generated by `bin/gstack-inventory` from [evidence.json](inventory/evidence.json), the hand-kept ledger, and lib/context-bill.ts. Measured at commit 56c1fb045 (2026-11-04). This is a dated snapshot: rerun `bin/gstack-inventory` to refresh it; `bin/gstack-inventory --check` validates the ledger. The rule: [ETHOS.md: 9. Earn Its Place](../ETHOS.md#9-earn-its-place). +Generated by `bin/gstack-inventory` from [evidence.json](inventory/evidence.json), the hand-kept ledger, and lib/context-bill.ts. Measured at commit cf1333466 (2026-11-06). This is a dated snapshot: rerun `bin/gstack-inventory` to refresh it; `bin/gstack-inventory --check` validates the ledger. The rule: [ETHOS.md: 9. Earn Its Place](../ETHOS.md#9-earn-its-place). Token counts are offline estimates (estimate: calibrated bytes/token per content class, about ±40%), not billed tokens. @@ -19,9 +19,9 @@ Load per run: a skill's SKILL.md plus the files it must read on every use (`perI | Rank | Part | Kind | Load per run (tokens) | On demand (tokens) | Always-on (tokens) | Loads | Status | Source, plan or reason | |---|---|---|---|---|---|---|---|---| -| 1 | ship-review-panel | shared | 23,370 | n/a | n/a | every /ship run, by depth | untested | plan: GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | +| 1 | ship-review-panel | shared | 23,369 | n/a | n/a | every /ship run, by depth | untested | plan: GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 2 | design-review | skill | 20,159 | 7,778 | 101 | every use | untested | plan: GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | -| 3 | ship | skill | 14,587 | 70,907 | 91 | every use | untested | plan: GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | +| 3 | ship | skill | 14,571 | 71,199 | 91 | every use | untested | plan: GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 4 | plan-ceo-review | skill | 14,394 | 21,203 | 313 | every use | untested | plan: GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 5 | review | skill | 14,268 | 41,459 | 185 | every use | helps | sources: gstack-evals #207 (baa599a9) wiki/board/2026-11-04-geva-271-r2-after-pin-results, gstack-evals #211 (81daa79) wiki/board/2026-11-04-geva-281-r2-after-pin-results-home-field; R48 KEEP; caveats: one unit: /review and the contract Stop hook were measured together, and neither gets credit alone; 42 of 66 gstack sessions ran the review readers only after the hook blocked them (24 unprompted), and that split is not randomized; home-field bias in gstack's favor: 41 of the 42 blocking bugs in stacks-r1 came from gstack-family reviews, so +22.6 may run high; a suite seeded from other reviewers' bugs would remove the bias (GEVA-2 lists it as future work); development suite: stacks-r1 is a development suite, so the result makes no release claim and holds only for these 33 units at these settings | | 6 | devex-review | skill | 14,025 | 0 | 74 | every use | untested | plan: GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | @@ -47,21 +47,21 @@ Load per run: a skill's SKILL.md plus the files it must read on every use (`perI | 26 | canary | skill | 7,034 | 0 | 66 | every use | untested | plan: none (grandfathered) | | 27 | qa | skill | 6,957 | 13,909 | 74 | every use | untested | plan: none (grandfathered) | | 28 | scrape | skill | 6,647 | 0 | 80 | every use | untested | plan: none (grandfathered) | -| 29 | captain | skill | 6,550 | 9,194 | 127 | every use | untested | plan: none (grandfathered) | +| 29 | captain | skill | 6,550 | 9,517 | 127 | every use | untested | plan: none (grandfathered) | | 30 | preamble | shared | 5,743 | n/a | n/a | every tier 2+ skill run, inside that skill's load | untested | plan: I1 ceremony cut (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-108-ceremony-cut-prereg (#122, main 409ff47) with its G3 amendment (#128), preset, Opus 5.5 and particle-1 (grandfathered) | | 31 | gstack-upgrade | skill | 5,726 | 0 | 59 | every use | untested | plan: none (grandfathered) | | 32 | context-restore | skill | 5,715 | 1,734 | 84 | every use | untested | plan: none (grandfathered) | | 33 | document-generate | skill | 5,466 | 0 | 110 | every use | untested | plan: none (grandfathered) | | 34 | ios-qa | skill | 5,368 | 1,378 | 84 | every use | untested | plan: none (grandfathered) | | 35 | skillify | skill | 5,317 | 0 | 80 | every use | untested | plan: none (grandfathered) | -| 36 | capy-skill | shared | 5,315 | n/a | n/a | every Capy session | untested | plan: none (grandfathered) | +| 36 | capy-skill | shared | 5,295 | n/a | n/a | every Capy session | untested | plan: none (grandfathered) | | 37 | setup-deploy | skill | 5,216 | 0 | 73 | every use | untested | plan: none (grandfathered) | | 38 | qa-only | skill | 5,205 | 3,460 | 68 | every use | untested | plan: none (grandfathered) | | 39 | eval-plan | skill | 5,107 | 0 | 108 | every use | untested | plan: none (grandfathered) | -| 40 | investigate | skill | 5,059 | 3,729 | 598 | every use | no-effect | sources: GST-814 (HARN-1, 2026-11-02), org-gstack-gbrain handoffs/HARN-14/hard-follow-2026-11/decision-rule.md; R48 CUT-CANDIDATE | +| 40 | investigate | skill | 5,059 | 3,729 | 598 | every use | no-effect | sources: GST-814 (HARN-1, 2026-11-02), org-gstack-gbrain handoffs/HARN-14/hard-follow-2026-11/decision-rule.md; R48 CUT-CANDIDATE; caveats: L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), untold: the routing package (route hook, contracts and the description rewrite) helps, untold /investigate opens 17/80 vs 0/80, B1 +21.3 [+12.5, +30.0], R48 KEEP; it needs routing or contracts on, and hook vs contracts aren't split; the description rewrite alone had no effect (off-u 0/80); L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), told: /investigate record completeness with the contract Stop hook is untested, B3 +10.0 [0, +20]; probe I failed its blind audit, R48 FIX-METHOD; the weak item is 'Fix names an edited file', and 5 of 7 misses were fixes written by an inline Python heredoc; L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), pass rate: O5 +5.0 at a 65% control, MDD 18.7, descriptive only | | 41 | health | skill | 5,041 | 0 | 68 | every use | untested | plan: none (grandfathered) | | 42 | gstack | skill | 4,990 | n/a | 59 | every use | untested | plan: none (grandfathered) | -| 43 | pair-agent | skill | 4,811 | 0 | 63 | every use | untested | plan: none (grandfathered) | +| 43 | pair-agent | skill | 4,824 | 0 | 63 | every use | untested | plan: none (grandfathered) | | 44 | make-pdf | skill | 4,730 | 1,767 | 65 | every use | untested | plan: none (grandfathered) | | 45 | context-save | skill | 4,637 | 0 | 65 | every use | untested | plan: none (grandfathered) | | 46 | cso | skill | 4,456 | 4,759 | 95 | every use | safety | reason: security: supported security findings with explicit coverage; kept on its security reason (GEVA-2 marks it exempt; GStack Minder may reclassify it) | @@ -71,9 +71,9 @@ Load per run: a skill's SKILL.md plus the files it must read on every use (`perI | 50 | document-release | skill | 4,049 | 10,260 | 72 | every use | untested | plan: none (grandfathered) | | 51 | diagram | skill | 3,895 | 0 | 112 | every use | untested | plan: none (grandfathered) | | 52 | benchmark-models | skill | 3,822 | 0 | 77 | every use | untested | plan: none (grandfathered) | -| 53 | runaway-stopper | skill | 3,672 | 571 | 78 | every use | untested | plan: docs/evals/runaway-stopper-prereg.md (owner GStack Minder (GSTA-36), measure by 2026-11-24) | -| 54 | ubicloud | skill | 3,638 | 0 | 83 | every use | untested | plan: none (grandfathered) | -| 55 | modernize | skill | 3,418 | 0 | 94 | every use | untested | plan: none (grandfathered) | +| 53 | runaway-stopper | skill | 3,693 | 571 | 78 | every use | untested | plan: docs/evals/runaway-stopper-prereg.md (owner GStack Minder (GSTA-36), measure by 2026-11-24) | +| 54 | ubicloud | skill | 3,676 | 0 | 83 | every use | untested | plan: none (grandfathered) | +| 55 | modernize | skill | 3,526 | 0 | 94 | every use | untested | plan: none (grandfathered) | | 56 | setup-browser-cookies | skill | 3,289 | 0 | 79 | every use | untested | plan: none (grandfathered) | | 57 | deslop-shared-libs | skill | 3,100 | 0 | 64 | every use | untested | plan: none (grandfathered) | | 58 | landing-report | skill | 2,971 | 0 | 71 | every use | untested | plan: none (grandfathered) | @@ -98,9 +98,9 @@ Untested parts, largest load per run first. A fair test runs the claude_code pre | Order | Part | Load per run (tokens) | Plan | |---|---|---|---| -| 1 | ship-review-panel | 23,370 | GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | +| 1 | ship-review-panel | 23,369 | GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 2 | design-review | 20,159 | GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | -| 3 | ship | 14,587 | GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | +| 3 | ship | 14,571 | GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 4 | plan-ceo-review | 14,394 | GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 5 | devex-review | 14,025 | GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | | 6 | plan-design-review | 13,351 | GEVA-2 registry row R56-7 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) | @@ -132,13 +132,13 @@ Untested parts, largest load per run first. A fair test runs the claude_code pre | 32 | document-generate | 5,466 | none (grandfathered) | | 33 | ios-qa | 5,368 | none (grandfathered) | | 34 | skillify | 5,317 | none (grandfathered) | -| 35 | capy-skill | 5,315 | none (grandfathered) | +| 35 | capy-skill | 5,295 | none (grandfathered) | | 36 | setup-deploy | 5,216 | none (grandfathered) | | 37 | qa-only | 5,205 | none (grandfathered) | | 38 | eval-plan | 5,107 | none (grandfathered) | | 39 | health | 5,041 | none (grandfathered) | | 40 | gstack | 4,990 | none (grandfathered) | -| 41 | pair-agent | 4,811 | none (grandfathered) | +| 41 | pair-agent | 4,824 | none (grandfathered) | | 42 | make-pdf | 4,730 | none (grandfathered) | | 43 | context-save | 4,637 | none (grandfathered) | | 44 | open-gstack-browser | 4,429 | none (grandfathered) | @@ -147,9 +147,9 @@ Untested parts, largest load per run first. A fair test runs the claude_code pre | 47 | document-release | 4,049 | none (grandfathered) | | 48 | diagram | 3,895 | none (grandfathered) | | 49 | benchmark-models | 3,822 | none (grandfathered) | -| 50 | runaway-stopper | 3,672 | docs/evals/runaway-stopper-prereg.md (owner GStack Minder (GSTA-36), measure by 2026-11-24) | -| 51 | ubicloud | 3,638 | none (grandfathered) | -| 52 | modernize | 3,418 | none (grandfathered) | +| 50 | runaway-stopper | 3,693 | docs/evals/runaway-stopper-prereg.md (owner GStack Minder (GSTA-36), measure by 2026-11-24) | +| 51 | ubicloud | 3,676 | none (grandfathered) | +| 52 | modernize | 3,526 | none (grandfathered) | | 53 | setup-browser-cookies | 3,289 | none (grandfathered) | | 54 | deslop-shared-libs | 3,100 | none (grandfathered) | | 55 | landing-report | 2,971 | none (grandfathered) | @@ -188,11 +188,11 @@ Each count is quoted from the file and line named; `bin/gstack-inventory --check | Skill | What | Count | Source | |---|---|---|---| -| ship | Step 9 review, pass 1, light depth, docs or tests only | 1 reader (the core checklist) | docs/review-policy.md:295 | -| ship | Step 9 review, pass 1, light depth, code | 3 readers (core, one batched specialist, native adversarial), plus the outside voice on a diff that publishes numbers | docs/review-policy.md:296 | -| ship | Step 9 review, pass 1, standard depth | the batched specialist, the Red Team at 200 runtime lines, adversarial and outside voice | docs/review-policy.md:297 | -| ship | Step 9 review, pass 1, full depth | the batched specialist on the whole diff and the Red Team always | docs/review-policy.md:298 | -| ship | targeted reader on money, gate, auth or secret code | +1 at every depth | docs/review-policy.md:446 | +| ship | Step 9 review, pass 1, light depth, docs or tests only | 1 reader (the core checklist) | docs/review-policy.md:298 | +| ship | Step 9 review, pass 1, light depth, code | 3 readers (core, one batched specialist, native adversarial), plus the outside voice on a diff that publishes numbers | docs/review-policy.md:299 | +| ship | Step 9 review, pass 1, standard depth | the batched specialist, the Red Team at 200 runtime lines, adversarial and outside voice | docs/review-policy.md:300 | +| ship | Step 9 review, pass 1, full depth | the batched specialist on the whole diff and the Red Team always | docs/review-policy.md:301 | +| ship | targeted reader on money, gate, auth or secret code | +1 at every depth | docs/review-policy.md:449 | | ship | audits (Steps 7, 8 and 14.5) | 3 | lib/ship-budget-calibration.json:3 | | ship | measured: one /ship child (gstack-evals#31) | 10 review dispatches | lib/ship-budget-calibration.json:43 | | ship | prior cost of one dispatch on the Capy meter | 7,430,000 tokens | lib/ship-budget-calibration.json:56 | @@ -248,3 +248,4 @@ Each count is quoted from the file and line named; `bin/gstack-inventory --check - Evidence-table row "review: Review Army and adversarial pass": a section inside /review, not its own key; R2 measures it as part of the review row's plan. - Evidence-table row "ship: QA probes (Step 9.2.1)": a step inside /ship with no separate size; folded into the ship row (GEVA-2 registry row R56-4). - Evidence-table row "gstack as a whole (installed, any skill)": not a single text part; GEVA-2's real-world outcome lane measures it (gstack-evals plan #111), not a measurement plan for any one part. +- Evidence-table row "untold routing package (route hook, contracts and the description rewrite)": two Claude Code hooks and four description lines, not one text part with a key. L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb): helps, untold /investigate opens 17/80 vs 0/80, B1 +21.3 [+12.5, +30.0], R48 KEEP; it needs routing or contracts on, and hook vs contracts aren't split; the description rewrite alone had no effect (off-u 0/80). The investigate row carries it as a caveat. diff --git a/docs/inventory/evidence.json b/docs/inventory/evidence.json index e724304f0..c2ce800cd 100644 --- a/docs/inventory/evidence.json +++ b/docs/inventory/evidence.json @@ -231,7 +231,12 @@ } ], "seed": "GEVA-2, 2026-11-02: investigate: measurement under way (R3)", - "note": "Only Opus 5.5 (claude-opus-5-5) was measured; GPT-6.1 Sol was not, so this verdict covers one of the two frontier models ETHOS.md 9 names." + "note": "Only Opus 5.5 (claude-opus-5-5) was measured; GPT-6.1 Sol was not, so this verdict covers one of the two frontier models ETHOS.md 9 names. L1 (gstack-evals#247, wiki/board/2026-11-06-geva-350-l1-overhaul-results) measured the after pin's untold routing package and told record completeness, not this row's procedure follow rate, so the no-effect status and its CUT-CANDIDATE stand; its three verdicts are this row's caveats. GPT-6.1 Sol has not run L1.", + "caveats": [ + "L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), untold: the routing package (route hook, contracts and the description rewrite) helps, untold /investigate opens 17/80 vs 0/80, B1 +21.3 [+12.5, +30.0], R48 KEEP; it needs routing or contracts on, and hook vs contracts aren't split; the description rewrite alone had no effect (off-u 0/80)", + "L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), told: /investigate record completeness with the contract Stop hook is untested, B3 +10.0 [0, +20]; probe I failed its blind audit, R48 FIX-METHOD; the weak item is 'Fix names an edited file', and 5 of 7 misses were fixes written by an inline Python heredoc", + "L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb), pass rate: O5 +5.0 at a 65% control, MDD 18.7, descriptive only" + ] }, "ios-clean": { "status": "untested", @@ -597,7 +602,8 @@ "seed": "GEVA-2, 2026-11-02: always-on skill catalog (descriptions): measurement under way (Run D)", "aliases": [ "always-on skill catalog (descriptions)" - ] + ], + "note": "L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, pins 82db89ab → 699a41cb): the description rewrite alone had no effect (off-u, routing and contracts off with the rewritten descriptions kept: 0/80 untold /investigate opens). The rewrite helps only inside the untold routing package; see the investigate row's caveats and this ledger's unmapped rows." }, "capy-skill": { "status": "untested", @@ -678,6 +684,10 @@ { "row": "gstack as a whole (installed, any skill)", "why": "not a single text part; GEVA-2's real-world outcome lane measures it (gstack-evals plan #111), not a measurement plan for any one part" + }, + { + "row": "untold routing package (route hook, contracts and the description rewrite)", + "why": "two Claude Code hooks and four description lines, not one text part with a key. L1 (GEVA-2, gstack-evals#247, merged 28be250; Opus 5.5, claude_code preset, 80 SWE-rebench tasks, no-gstack arm, pins 82db89ab → 699a41cb): helps, untold /investigate opens 17/80 vs 0/80, B1 +21.3 [+12.5, +30.0], R48 KEEP; it needs routing or contracts on, and hook vs contracts aren't split; the description rewrite alone had no effect (off-u 0/80). The investigate row carries it as a caveat" } ] }