mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 23:37:53 +01:00
v1.94.29.0 R56: everything in gstack earns its place (principle, evidence gates, inventory) (#3837)
* eval-plan: a no-gstack arm needs masking, install and audit lines prereg --check now refuses a loaded comparison whose ## No-gstack arm section lacks masking:, install: or audit:, or says none for one of them (PREREG_NO_GSTACK_ARM). On Capy, run agents inherit the instruction to load the user-drive gstack skill, so the arm hides both Capy drives, checks that no gstack is installed, and audits drive reads; an arm that read a gstack skill file is contaminated and gets zero credit (TESTING_INTERNALS.md #no-gstack-arm). The prereg template, /eval-plan, troubleshooting and the two passing prereg fixtures carry the new lines. eval-plan's paragraph was dieted to hold its context-budget ceiling. Evidence: safety: keeps the no-gstack control uncontaminated (Captain order, FOUN-32 via GGST-6) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE docs/inventory/evidence.json holds one row per inventory key: the skill folders discoverTemplates() finds (root skill as gstack, symlinked folders excluded) plus seven shared parts, 71 keys. lib/inventory.ts validates each row by its status's own fields, with named codes (LEDGER_*), and joins the ledger with lib/context-bill.ts buildBill. bin/gstack-inventory writes docs/inventory.md, --check validates (exit 1 with codes), --json prints. Paid dispatches per run for /ship and /autoplan are quoted from their own docs and found by line, so a moved citation fails --check (INVENTORY_SOURCE_MISSING). Every current row is grandfathered and seeded from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json): in-flight fair tests and proposed registry rows as plans, earlier unfair results under prior with why (stub, unmasked, ceiling, ...), five safety rows. gstack-skill-bill prints the skill's EVIDENCE line. Evidence: plan none (owner ruling R56, GST-914); the ledger itself is the measurement record Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ETHOS 9, Earn Its Place: factory, /write-skill and /autoplan carry it Garry, 2026-11-02: "Also work this principle of only include it if it is beneficial and good into gstack". ETHOS.md principle 9 states the rule, the fair test (claude_code preset, frontier models per R54, masked no-gstack arm), the R48 Why-null cycle, the safety exception, and measure biggest first, linking docs/inventory.md. factory.md item 8 cites it, trimmed elsewhere to 3,298 of 3,300 bytes with no rule dropped. /write-skill's edit ladder step 2 is the evidence rule and section 4 adds the Evidence: commit line. /autoplan's CEO phase lists each part a plan adds to gstack with its evidence or owned, dated plan; two redundant lines went to keep autoplan under its size pins (pins lowered by 17). CONTRIBUTING.md "Adding a new skill" and docs/skills.md point at the ledger. Skill-Size: 3273→3666 bytes (write-skill/SKILL.md.tmpl) Skill-Size: 12738→12721 bytes (autoplan/sections/ceo-phase.md) Evidence: plan none (owner ruling R56, GST-914); write-skill and autoplan rows stay grandfathered untested Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fixup! inventory: the R56 evidence ledger, gstack-inventory and skill-bill EVIDENCE Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: list the evidence table's other names and its unmapped rows Ledger rows carry aliases (the name GEVA-2's table uses where it differs from the key, such as upgrade for gstack-upgrade), and docs/inventory.md lists them beside the installed-name rule. Unmapped rows are labeled as evidence-table rows, and the ranking notes that a skill's on-demand files can also be a shared row's. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the first inventory snapshot and the R56 changelog fragment docs/inventory.md is generated by bin/gstack-inventory at the previous commit: 71 parts ranked by load per run, 0 helps, 0 no-effect, 0 hurts, 16 untested with a plan (4 fair tests under way, 12 proposed), 50 untested with plan none, 5 safety. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's step procedure is a fair-test CUT-CANDIDATE (GST-814) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after investigate's no-effect row Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: only the seeded backlog may be grandfathered, so a new part's row is checked (review fix) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: devex-review and retro name their GEVA-2 registry rows (R56-11, R56-12) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: a concrete plan is a path, gstack-inventory never writes into a gstack install, and the ledger checks aliases, r48 and prior.why (review fixes) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * inventory: investigate's no-effect row notes that only Opus 5.5 was measured Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: the inventory snapshot after the review fixes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: R56 counts after investigate's row; grandfathering and the install guard Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register inventory-grandfathered.json as a free fixture free-suite (1) ate56d3312cfailed FREE_FIXTURES inventory: the pinned grandfather fixture fromd0b3028b5had no paid touchfile and no FREE_FIXTURES entry. Its only reader is test/inventory.test.ts. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed
1 parent
36fd1b18ab
commit
ade35bc813
35 files changed
+1834
-62
No files matched your search
@@ -1,5 +1,21 @@
|
||||
# Changelog
|
||||
|
||||
## [1.94.29.0] - 2026-11-03
|
||||
|
||||
**Every part of gstack now has to earn its place.** Garry, 2026-11-02: "Also work this principle of only include it if it is beneficial and good into gstack". A new skill, step, instruction, check or default needs evidence that it helps on frontier models, or a reason, an owner and a dated plan to measure it. The first inventory reads untested almost everywhere: one part, `/investigate`, has a fair no-effect result (a CUT-CANDIDATE, measured on Opus 5.5 only), 3 have a fair test under way, 14 more have one proposed, and 5 safety guards stay on their stated reason.
|
||||
|
||||
### Added
|
||||
- **ETHOS.md principle 9, Earn Its Place.** A fair test is Claude Code's own `claude_code` system prompt, frontier models (today Opus 5.5 and GPT-6.1 Sol) and a masked no-gstack arm. A part a fair test shows doesn't help gets one "Why null?" cycle, then is kept, fixed or cut. Measure the biggest parts first. The Capy factory page cites it in item 8, still under its 3,300-byte budget.
|
||||
- **`docs/inventory/evidence.json`, the evidence ledger, and `bin/gstack-inventory`.** One row per part: every skill folder plus seven shared parts (preamble, catalog, the two Capy files, the /ship review panel and docs audit, the /autoplan phases), 71 keys in all. Each status has its own required fields (`helps`, `no-effect` and `hurts` need sources, frontier models, the preset prompt, a scope and a masked no-gstack arm, and the last two an R48 verdict; a new `untested` part needs a reason, a concrete plan, an owner and a `measure_by` date; `safety` needs a reason). Rows present today are grandfathered, seeded from the gstack-evals evidence table with their sources, plans and earlier results that don't count yet; no part added later can claim that (`test/fixtures/inventory-grandfathered.json`). `gstack-inventory` writes `docs/inventory.md` (parts ranked by load per run, a "measure next" order, preamble tiers, paid dispatches per run for /ship and /autoplan quoted from their own docs); `--check` validates the ledger; `--json` prints it. It works on the git repo you run it in and never writes into a gstack install.
|
||||
- **`gstack-skill-bill` prints `EVIDENCE:`**, the skill's ledger row, and `/write-skill`'s edit ladder step 2 is now the evidence rule, with an `Evidence:` commit line beside `Skill-Size:`.
|
||||
- **`/autoplan`'s CEO phase lists each part a plan adds to gstack** with its evidence or owned, dated plan; a missing one is a finding.
|
||||
|
||||
### Changed
|
||||
- **No-gstack arms must be masked.** `gstack-eval-plan prereg --check` now also requires `masking:`, `install:` and `audit:` in `## No-gstack arm` and refuses a blank or `none` value (`PREREG_NO_GSTACK_ARM`). On Capy, run agents inherit the instruction to load the user-drive gstack skill, so the arm hides the Capy drives and checks that no gstack is installed. An arm that read a gstack skill file is contaminated and gets zero credit (docs/TESTING_INTERNALS.md#no-gstack-arm).
|
||||
|
||||
### For contributors
|
||||
- CONTRIBUTING.md "Adding a new skill" adds the ledger row; `test/inventory.test.ts` fails on a skill without a valid row.
|
||||
|
||||
## [1.94.28.0] - 2026-11-03
|
||||
|
||||
**`gstack-ship-preflight --timeout` reaches Step 5's pregate (GST-1196).** Preflight ran `gstack-pregate` with no way to pass `--timeout`, so on a repo whose `.gstack/pregate.json` sets no `timeout_s`, a test group longer than 600 s always stopped with `PREGATE_INCOMPLETE`. The only workaround was a separate `gstack-pregate --timeout 1800` before finish (gstack-evals #176 and #179).
|
||||
|
||||
@@ -814,6 +814,7 @@ Write or change skill prose with `/write-skill` (edit ladder, levels, `gstack-sk
|
||||
3. The dynamic template discovery picks it up for every host render and for `gstack/llms.txt`. Two things it does not fill in: a row in `SKILL_REQUIREMENTS` in `lib/doctor-components.ts` (the doctor's `--for <skill>` table; `test/doctor-components.test.ts` fails on a skill directory without one, and the same gen run rerenders `bin/gstack-doctor-components.sh`), and the catalog rows readers find the skill through (README.md, AGENTS.md, `docs/skills.md`, `docs/PROJECT_STRUCTURE.md`, the root `SKILL.md.tmpl` router)
|
||||
4. Budget it: run `bun test/helpers/capture-context-budget.ts` and commit the refreshed `test/fixtures/context-budget.json` — the context-budget ratchet fails any skill without a ceiling; the catalog budget (`test/catalog-budget.test.ts`) fails until its cap is ratcheted by the protocol in its failure message, and the number is quoted in CLAUDE.md, this file and `docs/CONTRIBUTOR_REFERENCE.md`
|
||||
5. Commit `{skill}/SKILL.md` (and the generated `{skill}/sections/preamble.md` when the template's `preamble-tier` is 2 or higher), external host output is generated at setup time and gitignored
|
||||
6. Give it a row in the evidence ledger, `docs/inventory/evidence.json` ([ETHOS.md: 9. Earn Its Place](ETHOS.md#9-earn-its-place)): evidence that it helps on frontier models (`sources`, `models`, `prompt: preset`, `scope`, `masked: true`), or `status: untested` with its `reason`, a concrete `plan` (a `test/fixtures/skill-evals/<skill>/tasks.json` or an /eval-plan preregistration), an `owner` and a `measure_by` date. `test/inventory.test.ts` fails on a skill without a valid row; `bin/gstack-inventory --check` names the problem, and `bin/gstack-inventory` refreshes the dated snapshot in `docs/inventory.md`
|
||||
|
||||
## Conductor workspaces
|
||||
|
||||
|
||||
@@ -273,6 +273,42 @@ the eval that jumps twenty points usually found a shortcut, not a capability.
|
||||
|
||||
---
|
||||
|
||||
## 9. Earn Its Place
|
||||
|
||||
Garry, 2026-11-02:
|
||||
"Also work this principle of only include it if it is beneficial and good into gstack".
|
||||
|
||||
Every skill, step, instruction, check and default costs tokens and attention
|
||||
on every run that loads it. So each one earns its place. A new part comes with
|
||||
evidence that it helps on frontier models, or with a stated reason, an owner
|
||||
and a date by which a fair test will measure it.
|
||||
|
||||
A fair test runs Claude Code's own system prompt (the `claude_code` preset,
|
||||
not the Agent SDK's stub), on frontier models (today Opus 5.5 and GPT-6.1 Sol,
|
||||
plus at most one named low-cost model), against a no-gstack arm that is masked
|
||||
so it cannot read gstack. A result from the stub, from Haiku or Sonnet alone,
|
||||
or from an unmasked arm is a lead, not a verdict.
|
||||
|
||||
When a fair test shows an existing part doesn't help, it gets one "Why null?"
|
||||
cycle: did the model use the part, did the task fit, was there enough power and
|
||||
room below the ceiling, and does today's model do the job without it. Then
|
||||
keep it, fix the method, or cut it, and say which plainly.
|
||||
|
||||
Safety and security guards (careful, freeze, guard, unfreeze, cso) stay on
|
||||
their stated reason: they stop the rare bad run that an average score can't see.
|
||||
|
||||
Measure the biggest parts first, the ones that load the most on the most runs.
|
||||
[docs/inventory.md](docs/inventory.md) ranks every part by load and shows its
|
||||
evidence; `docs/inventory/evidence.json` is the ledger, and
|
||||
`bin/gstack-inventory --check` validates it.
|
||||
|
||||
**Anti-patterns:**
|
||||
- A new step because it sounds wise, with no test planned.
|
||||
- Keeping a part because it looked good on the stub or a small model.
|
||||
- Cutting a part after one null result, without the Why-null cycle.
|
||||
|
||||
---
|
||||
|
||||
## How They Work Together
|
||||
|
||||
Boil the Ocean says: **cover the whole requirement with the least code.**
|
||||
@@ -282,6 +318,7 @@ Attack the Premise says: **when fixes keep failing, test the belief they share.*
|
||||
Separate Before Serializing says: **remove the race before you referee it.**
|
||||
Make Operations Idempotent says: **design for the run that happens twice.**
|
||||
Explain the Number says: **a result you cannot explain is not yet a result.**
|
||||
Earn Its Place says: **every part shows it helps, or has a dated plan to.**
|
||||
|
||||
Together: search first, then build the complete version of the right thing.
|
||||
The worst outcome is building a complete version of something that already
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# gstack digest v1.94.28.0 — regenerate/re-copy after upgrading gstack
|
||||
# gstack digest v1.94.29.0 — regenerate/re-copy after upgrading gstack
|
||||
|
||||
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
|
||||
for agent hosts without a full skill install. The full skills add workflows,
|
||||
|
||||
@@ -11,11 +11,12 @@ announcement. Dispatching a reviewer never completes its step.
|
||||
- Premises: accept reasonable ones (P6). Queue clearly-wrong/challenged premises
|
||||
as User Challenges for Phase 4: assumption, reason and cost of proceeding.
|
||||
The user decides there; never stop mid-pipeline.
|
||||
- List each part the plan adds to gstack (skill, step, instruction, check, default) with its
|
||||
evidence or owned, dated test plan; a missing one is a finding.
|
||||
- Alternatives: pick highest completeness (P1). If tied, pick simplest (P5).
|
||||
If top 2 are close → mark TASTE DECISION.
|
||||
- Scope expansion: in blast radius + <1d CC → approve (P2). Outside → defer to TODOS.md (P3).
|
||||
Duplicates → reject (P4). Borderline (3-5 files) → mark TASTE DECISION.
|
||||
- All 11 review sections: run fully, auto-decide each issue, log every decision.
|
||||
|
||||
**Required execution checklist (CEO):**
|
||||
|
||||
@@ -38,8 +39,7 @@ Before every spec dispatch, including after each accepted spec fix, run:
|
||||
```bash
|
||||
bun "<SNAPSHOT_TOOL>" amend-input ceo "<ACTIVE_PLAN>" "<CEO_STEP0_CHECKPOINT>" "<RESTORE_PATH>" "<methodologyPath>"
|
||||
```
|
||||
This applies the recorded requirements and exports the complete current
|
||||
`Implementation plan`. Keep returned `checkpointPath` as the amendment baseline;
|
||||
Keep returned `checkpointPath` as the amendment baseline;
|
||||
use returned `reviewInputPath` as `<CEO_SPEC_INPUT>`. Read that file at every
|
||||
returned `readRanges` offset/limit through EOF, then read the CEO scope summary in full.
|
||||
Reconcile dispositions, scope counts, proposal IDs and actual heading/test references
|
||||
|
||||
@@ -9,11 +9,12 @@ announcement. Dispatching a reviewer never completes its step.
|
||||
- Premises: accept reasonable ones (P6). Queue clearly-wrong/challenged premises
|
||||
as User Challenges for Phase 4: assumption, reason and cost of proceeding.
|
||||
The user decides there; never stop mid-pipeline.
|
||||
- List each part the plan adds to gstack (skill, step, instruction, check, default) with its
|
||||
evidence or owned, dated test plan; a missing one is a finding.
|
||||
- Alternatives: pick highest completeness (P1). If tied, pick simplest (P5).
|
||||
If top 2 are close → mark TASTE DECISION.
|
||||
- Scope expansion: in blast radius + <1d CC → approve (P2). Outside → defer to TODOS.md (P3).
|
||||
Duplicates → reject (P4). Borderline (3-5 files) → mark TASTE DECISION.
|
||||
- All 11 review sections: run fully, auto-decide each issue, log every decision.
|
||||
|
||||
**Required execution checklist (CEO):**
|
||||
|
||||
@@ -36,8 +37,7 @@ Before every spec dispatch, including after each accepted spec fix, run:
|
||||
```bash
|
||||
bun "<SNAPSHOT_TOOL>" amend-input ceo "<ACTIVE_PLAN>" "<CEO_STEP0_CHECKPOINT>" "<RESTORE_PATH>" "<methodologyPath>"
|
||||
```
|
||||
This applies the recorded requirements and exports the complete current
|
||||
`Implementation plan`. Keep returned `checkpointPath` as the amendment baseline;
|
||||
Keep returned `checkpointPath` as the amendment baseline;
|
||||
use returned `reviewInputPath` as `<CEO_SPEC_INPUT>`. Read that file at every
|
||||
returned `readRanges` offset/limit through EOF, then read the CEO scope summary in full.
|
||||
Reconcile dispositions, scope counts, proposal IDs and actual heading/test references
|
||||
|
||||
Executable
+54
@@ -0,0 +1,54 @@
|
||||
#!/usr/bin/env -S bun --no-env-file --config=/dev/null
|
||||
// gstack-inventory — every gstack part, its load per run and its evidence (ETHOS.md: 9. Earn Its Place). Logic: lib/inventory.ts.
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { existsSync, writeFileSync } from "node:fs";
|
||||
import { join, resolve } from "node:path";
|
||||
import { buildInventory, checkInventory, DOC_PATH, installReason, LEDGER_PATH, renderInventory } from "../lib/inventory";
|
||||
|
||||
const USAGE = `Usage: gstack-inventory [--root <gstack checkout>] [--check | --json]
|
||||
gstack-inventory --help
|
||||
|
||||
Joins the evidence ledger (${LEDGER_PATH}) with gstack-context-bill's sizes and writes ${DOC_PATH}:
|
||||
parts ranked by load per run, each part's evidence status, a "measure next" order (largest untested
|
||||
first), shared-part sizes, paid dispatches per run for /ship and /autoplan, and the key list.
|
||||
--check validates the ledger (every key has one row; each row has its status's fields), the quoted
|
||||
dispatch sources and the shared-part files, and writes nothing. --json prints the inventory.
|
||||
The root defaults to the git repo you run it in; it never writes into a gstack install.
|
||||
Exit 0 when clean; 1 with one "<key>: <problem> (<CODE>)" line per problem; 2 on a bad argument.`;
|
||||
|
||||
const args = process.argv.slice(2);
|
||||
if (args.includes("--help") || args.includes("-h")) { console.log(USAGE); process.exit(0); }
|
||||
const rootAt = args.indexOf("--root");
|
||||
const rootArg = rootAt >= 0 ? args[rootAt + 1] : undefined;
|
||||
const unknown = args.find((a, i) => !["--check", "--json", "--root"].includes(a) && !(rootAt >= 0 && i === rootAt + 1));
|
||||
if (unknown !== undefined || (rootAt >= 0 && (!rootArg || rootArg.startsWith("--"))) || (args.includes("--check") && args.includes("--json"))) {
|
||||
console.error(unknown !== undefined ? `gstack-inventory: unknown argument '${unknown}'; valid: --root <dir> --check --json --help` : USAGE);
|
||||
process.exit(2);
|
||||
}
|
||||
const top = spawnSync("git", ["rev-parse", "--show-toplevel"], { encoding: "utf8", timeout: 10_000 });
|
||||
const root = resolve(rootArg ?? (top.status === 0 ? top.stdout.trim() : process.cwd()));
|
||||
if (rootArg === undefined && !existsSync(join(root, LEDGER_PATH))) {
|
||||
console.error(`gstack-inventory: ${root} is not a gstack checkout (no ${LEDGER_PATH}); run it in your gstack checkout or pass --root <dir>`);
|
||||
process.exit(2);
|
||||
}
|
||||
const report = (problems: { key: string; code: string; message: string }[]) => {
|
||||
for (const p of problems) console.error(`gstack-inventory: ${p.key}: ${p.message} (${p.code})`);
|
||||
};
|
||||
if (args.includes("--check")) {
|
||||
const checked = checkInventory(root);
|
||||
report(checked.problems);
|
||||
if (checked.problems.length) process.exit(1);
|
||||
console.log(`INVENTORY: ok keys=${checked.keys.length} ledger=${LEDGER_PATH} dispatch_sources=${checked.dispatches.length}`);
|
||||
process.exit(0);
|
||||
}
|
||||
const inv = buildInventory(root);
|
||||
report(inv.problems);
|
||||
if (args.includes("--json")) { console.log(JSON.stringify(inv, null, 2)); process.exit(inv.problems.length ? 1 : 0); }
|
||||
if (inv.problems.length) { console.error(`gstack-inventory: ${DOC_PATH} not written; fix the problems above, then rerun`); process.exit(1); }
|
||||
const installed = installReason(root);
|
||||
if (installed) {
|
||||
console.error(`gstack-inventory: refusing to write ${DOC_PATH} into ${root}: it is ${installed}, which gstack-upgrade owns; run it in your gstack checkout or pass --root <dir>`);
|
||||
process.exit(2);
|
||||
}
|
||||
writeFileSync(join(root, DOC_PATH), renderInventory(inv));
|
||||
console.log(`INVENTORY: wrote ${DOC_PATH} keys=${inv.keys.length} commit=${inv.commit}`);
|
||||
@@ -11,7 +11,8 @@ Names the skill's kind: gstack-checkout (edit SKILL.md.tmpl, then bun run gen:sk
|
||||
gstack-install (never edit; gstack-upgrade overwrites it) or user. BASELINE is the newest commit
|
||||
with a "Skill-Diet: <name>" trailer; DUE means the authored file grew over 10% since then.
|
||||
In a gstack checkout, EVAL TASKS names test/fixtures/skill-evals/<name>/tasks.json (or none) and
|
||||
EVAL_TASKS_INVALID flags a file that breaks docs/designs/SKILL_EVAL_CONTRACT.md's task format.
|
||||
EVAL_TASKS_INVALID flags a file that breaks docs/designs/SKILL_EVAL_CONTRACT.md's task format, and
|
||||
EVIDENCE prints the skill's row in docs/inventory/evidence.json: its status and source, plan or reason.
|
||||
A name must match exactly; pass a path when it names more than one skill.
|
||||
Exit 0 on a report, 2 on SKILL_NOT_FOUND, SKILL_AMBIGUOUS or SKILL_MD_MISSING (each with a fix line).`;
|
||||
|
||||
|
||||
@@ -1,17 +1,17 @@
|
||||
# How we work: the gstack software factory on Capy
|
||||
# How we work: gstack's factory on Capy
|
||||
|
||||
What seven Captain crews converged on (2026-10-29). Your crew's CREW.md and recorded approvals decide who may merge, update a base, spend or touch production; urgency changes priority, never permissions or gates. With no Captain, ask the person and never merge without their word. Story: capy/README.md; examples: your org Drive's software-factory/README.md, if any.
|
||||
What seven Captain crews agreed on. Your crew's CREW.md and recorded approvals decide who may merge, update a base, spend or touch production; urgency changes priority, never permissions or gates. With no Captain, ask the person; never merge without their word. Story: capy/README.md; examples: org Drive software-factory/README.md.
|
||||
|
||||
1. **Plan, then a yes.** New functionality starts with /autoplan sized to the work; the build waits for "do it". [SKILL.md: Plan, approve, ship]
|
||||
2. **Someone else merges.** Run /ship to a receipt on your final pushed head, send one READY quoting gstack's verify output verbatim, and stop. The merge owner merges on green and assigns versions. [docs/captain.md: The READY contract]
|
||||
3. **Block only on the floor.** A finding blocks if it would fail a test, break production, or lose data or secrets; the rest goes in the PR body or next wave. [docs/review-policy.md]
|
||||
4. **Fix waves.** When main goes red, hold /ship while one owner finds the root cause; bring main in once, as your rules allow. Fold deferred findings and test follow-ups into one PR. [docs/triage.md]
|
||||
5. **Severe first, same gates.** A severe user-facing bug jumps the queue; the plan can be short; receipt, checks and merge owner stay.
|
||||
6. **Trade findings, not small talk.** Another crew's finding goes to its Captain, through yours, with the evidence.
|
||||
5. **Severe first, same gates.** A severe user-facing bug jumps the queue with a short plan; receipt, checks and merge owner stay.
|
||||
6. **Trade findings, not small talk.** Another crew's finding goes to its Captain, through yours, with evidence.
|
||||
7. **Measure before you act.** Run the real system on the merged tree you ship; a real run outranks a reviewed plan. Each number names its run, commit or uncapped query; say what you did not verify.
|
||||
8. **Evals decide, and losses get reported.** Preregister bars, cap and stop rule before paying [/eval-plan]. Cheap proof (a keyless fake, a smoke) before the paid run. Use controls (no-skill and placebo arms, clean decoys). Report use and pass apart; quote spend from the meter.
|
||||
9. **Flakes: one rerun, a ledger, a daily fix.** Only the rerun the policy allows, never a harness or infra red; a second red is real; no test is loosened. [docs/captain.md: The daily flaky pass]
|
||||
8. **Evals decide, and losses get reported.** Preregister bars, cap and stop rule before paying [/eval-plan]. Cheap proof (a keyless fake, a smoke) first. Controls: no-skill and placebo arms, clean decoys. Report use and pass apart; quote spend from the meter. A new part needs proof it helps or a dated test plan [ETHOS.md: 9. Earn Its Place].
|
||||
9. **Flakes: one rerun, a ledger, a daily fix.** Only the policy's rerun, never for a harness or infra red; a second red is real; never loosen a test. [docs/captain.md: The daily flaky pass]
|
||||
10. **Quiet.** Message on a decision, a blocker, a READY or a finished phase. A broadcast needing no reply goes in the Captain's notes; never reply just to acknowledge.
|
||||
11. **Ask once, plainly.** One ask per decision, explained like the owner is five: your Captain's name, then a lettered line each: "A: <code> hit problem p, which happens when h. We could do 1) x, 2) y or 3) z. Recommendation x because why. A1 to take recommendation." "do it" takes every recommendation; "A2" takes option 2. Owner tasks: the exact link (in Capy: path and field), numbered simple steps, the reply word. Never re-ask a standing rule; check what you can yourself. [docs/captain.md: The ask format]
|
||||
11. **Ask once, plainly.** One ask per decision, explained like the owner is five: your Captain's name, then a lettered line each: "A: <code> hit problem p, which happens when h. We could do 1) x, 2) y or 3) z. Recommendation x because why. A1 to take recommendation." "do it" takes every recommendation; "A2" takes option 2. Owner tasks: the exact link (in Capy: path and field), numbered steps, the reply word. Never re-ask a standing rule; check what you can yourself. [docs/captain.md: The ask format]
|
||||
12. **Fresh threads, closed out.** One thread per unit of work (crew threads ask their Captain); past the context limit, hand off. Its Captain archives only its crew's threads; threads outside every crew are the owner's, even its automation's. Size reviews to the diff; read the token counter. [docs/ship-budget.md]
|
||||
13. **The owner's model preference.** `gstack-models resolve --role agent` picks agent models, except cross-family voices and eval subjects. [docs/model-policy.md: Your model preference]
|
||||
@@ -1871,6 +1871,24 @@ own attempts. The arm gets:
|
||||
- the same budget treatment: it is priced and reserved in the pilot like every
|
||||
other arm.
|
||||
|
||||
**Masking.** On Capy, the owner's user-scope instructions tell every session to
|
||||
load the user-drive gstack skill, and eval run agents inherit them: in one
|
||||
checkpoint (24 runs, 3 arms) every run in every arm tried to read that skill in
|
||||
its first minute. So `## No-gstack arm` also needs three lines, and `prereg
|
||||
--check` refuses a missing, blank or `none` one (`PREREG_NO_GSTACK_ARM`):
|
||||
|
||||
- `masking:` an empty directory bind-mounted over `~/.capy/drive/user-garry-tan`
|
||||
and `~/.capy/drive/org-gstack-gbrain` before the run agent starts, or a
|
||||
container that sees no Capy drive;
|
||||
- `install:` no gstack on the arm's machine (`~/.local/share/gstack`,
|
||||
`~/.claude/skills`), checked before the run;
|
||||
- `audit:` a per-arm audit of drive and skill reads.
|
||||
|
||||
An arm that read a gstack skill file is contaminated and gets zero credit. A
|
||||
no-gstack result that ran before this rule without a mask is not evidence for
|
||||
an R48 verdict until it is re-checked; the evidence ledger records it under
|
||||
`prior` with `unmasked` ([docs/inventory.md](inventory.md)).
|
||||
|
||||
**Reporting.** The no-gstack result is one row beside the variants in every
|
||||
results table and every verdict, including when it wins. A comparison that
|
||||
drops the row has not reported its result. `gstack-eval-plan results --check
|
||||
|
||||
@@ -0,0 +1,249 @@
|
||||
# gstack inventory: every part, its load per run and its evidence
|
||||
|
||||
Generated by `bin/gstack-inventory` from [evidence.json](inventory/evidence.json), the hand-kept ledger, and lib/context-bill.ts. Measured at commit cd67c41d8 (2026-11-02). This is a dated snapshot: rerun `bin/gstack-inventory` to refresh it; `bin/gstack-inventory --check` validates the ledger. The rule: [ETHOS.md: 9. Earn Its Place](../ETHOS.md#9-earn-its-place).
|
||||
|
||||
Token counts are offline estimates (estimate: calibrated bytes/token per content class, about ±40%), not billed tokens.
|
||||
|
||||
| Status | Parts |
|
||||
|---|---|
|
||||
| helps | 0 |
|
||||
| no-effect | 1 |
|
||||
| hurts | 0 |
|
||||
| untested, with a plan | 17 |
|
||||
| untested, plan none | 48 |
|
||||
| safety | 5 |
|
||||
|
||||
## Parts by load per run
|
||||
|
||||
Load per run: a skill's SKILL.md plus the files it must read on every use (`perInvocationTokens` in lib/context-bill.ts); a shared part's files. Tier 2+ skills also read the shared preamble once per session (its own row). On demand: the skill's other sections/ and references/ files, read when a step asks for them; files a shared row also counts, such as /ship's review sections, appear in both. Always-on: frontmatter every session's skill scanner reads.
|
||||
|
||||
| Rank | Part | Kind | Load per run (tokens) | On demand (tokens) | Always-on (tokens) | Loads | Status | Source, plan or reason |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | ship-review-panel | shared | 22,830 | n/a | n/a | every /ship run, by depth | untested | plan: GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 2 | design-review | skill | 20,159 | 7,778 | 101 | every use | untested | plan: GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 3 | ship | skill | 14,587 | 69,678 | 91 | every use | untested | plan: GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 4 | plan-ceo-review | skill | 14,394 | 21,203 | 313 | every use | untested | plan: GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 5 | review | skill | 13,990 | 41,146 | 75 | every use | untested | plan: R2 (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-02-geva-122-r2-amendment (#129, main eae6efc), told /review vs a no-gstack arm, Opus 5.5, preset on both arms (grandfathered) |
|
||||
| 6 | devex-review | skill | 13,911 | 0 | 74 | every use | untested | plan: GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 7 | plan-design-review | skill | 13,351 | 12,986 | 79 | every use | untested | plan: GEVA-2 registry row R56-7 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 8 | retro | skill | 12,257 | 1,514 | 225 | every use | untested | plan: GEVA-2 registry row R56-12 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 9 | plan-devex-review | skill | 11,104 | 20,474 | 80 | every use | untested | plan: GEVA-2 registry row R56-6 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 10 | autoplan-phases | shared | 10,709 | n/a | n/a | every /autoplan run, by scope | untested | plan: folded into GEVA-2 registry row R56-3 (autoplan); proposed, not yet preregistered (grandfathered) |
|
||||
| 11 | office-hours | skill | 10,424 | 23,078 | 280 | every use | untested | plan: none (grandfathered) |
|
||||
| 12 | autoplan | skill | 10,288 | 18,701 | 394 | every use | untested | plan: GEVA-2 registry row R56-3 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 13 | plan-eng-review | skill | 9,837 | 25,963 | 74 | every use | untested | plan: GEVA-2 registry row R56-2 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 14 | design-consultation | skill | 9,557 | 10,864 | 282 | every use | untested | plan: GEVA-2 registry row R56-10 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 15 | setup-gbrain | skill | 9,099 | 10,921 | 106 | every use | untested | plan: none (grandfathered) |
|
||||
| 16 | codex | skill | 8,991 | 10,358 | 68 | every use | untested | plan: none (grandfathered) |
|
||||
| 17 | sync-gbrain | skill | 8,863 | 0 | 85 | every use | untested | plan: none (grandfathered) |
|
||||
| 18 | design-shotgun | skill | 8,470 | 1,208 | 248 | every use | untested | plan: none (grandfathered) |
|
||||
| 19 | plan-tune | skill | 8,377 | 0 | 107 | every use | untested | plan: none (grandfathered) |
|
||||
| 20 | spec | skill | 8,305 | 7,759 | 92 | every use | untested | plan: none (grandfathered) |
|
||||
| 21 | browse | skill | 8,125 | 5,597 | 81 | every use | untested | plan: none (grandfathered) |
|
||||
| 22 | design-html | skill | 8,096 | 5,844 | 82 | every use | untested | plan: none (grandfathered) |
|
||||
| 23 | land-and-deploy | skill | 7,652 | 16,188 | 63 | every use | untested | plan: GEVA-2 registry row R56-8 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 24 | benchmark | skill | 7,430 | 961 | 72 | every use | untested | plan: none (grandfathered) |
|
||||
| 25 | catalog | shared | 7,291 | n/a | 7,291 | every session | untested | plan: Run D (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-125-untold-discovery-prereg (#125, main 9820b14), untold gstack vs placebo vs no gstack, preset vs stub, Opus 5.5, GPT-6.1 Sol and particle-1; also measures /investigate's description (grandfathered) |
|
||||
| 26 | investigate | skill | 7,086 | 3,729 | 490 | every use | no-effect | sources: GST-814 (HARN-1, 2026-11-02), org-gstack-gbrain handoffs/HARN-14/hard-follow-2026-11/decision-rule.md; R48 CUT-CANDIDATE |
|
||||
| 27 | canary | skill | 7,034 | 0 | 66 | every use | untested | plan: none (grandfathered) |
|
||||
| 28 | qa | skill | 6,951 | 13,877 | 74 | every use | untested | plan: none (grandfathered) |
|
||||
| 29 | scrape | skill | 6,647 | 0 | 80 | every use | untested | plan: none (grandfathered) |
|
||||
| 30 | captain | skill | 6,522 | 7,856 | 127 | every use | untested | plan: none (grandfathered) |
|
||||
| 31 | preamble | shared | 5,743 | n/a | n/a | every tier 2+ skill run, inside that skill's load | untested | plan: I1 ceremony cut (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-108-ceremony-cut-prereg (#122, main 409ff47) with its G3 amendment (#128), preset, Opus 5.5 and particle-1 (grandfathered) |
|
||||
| 32 | gstack-upgrade | skill | 5,726 | 0 | 59 | every use | untested | plan: none (grandfathered) |
|
||||
| 33 | context-restore | skill | 5,715 | 1,734 | 84 | every use | untested | plan: none (grandfathered) |
|
||||
| 34 | document-generate | skill | 5,466 | 0 | 110 | every use | untested | plan: none (grandfathered) |
|
||||
| 35 | ios-qa | skill | 5,368 | 1,378 | 84 | every use | untested | plan: none (grandfathered) |
|
||||
| 36 | skillify | skill | 5,317 | 0 | 80 | every use | untested | plan: none (grandfathered) |
|
||||
| 37 | capy-skill | shared | 5,309 | n/a | n/a | every Capy session | untested | plan: none (grandfathered) |
|
||||
| 38 | health | skill | 5,287 | 0 | 68 | every use | untested | plan: none (grandfathered) |
|
||||
| 39 | setup-deploy | skill | 5,216 | 0 | 73 | every use | untested | plan: none (grandfathered) |
|
||||
| 40 | qa-only | skill | 5,205 | 3,460 | 68 | every use | untested | plan: none (grandfathered) |
|
||||
| 41 | eval-plan | skill | 5,021 | 0 | 108 | every use | untested | plan: none (grandfathered) |
|
||||
| 42 | gstack | skill | 4,951 | n/a | 59 | every use | untested | plan: none (grandfathered) |
|
||||
| 43 | pair-agent | skill | 4,811 | 0 | 63 | every use | untested | plan: none (grandfathered) |
|
||||
| 44 | make-pdf | skill | 4,730 | 1,767 | 65 | every use | untested | plan: none (grandfathered) |
|
||||
| 45 | context-save | skill | 4,637 | 0 | 65 | every use | untested | plan: none (grandfathered) |
|
||||
| 46 | cso | skill | 4,456 | 4,759 | 95 | every use | safety | reason: security: supported security findings with explicit coverage; kept on its security reason (GEVA-2 marks it exempt; GStack Minder may reclassify it) |
|
||||
| 47 | open-gstack-browser | skill | 4,429 | 0 | 74 | every use | untested | plan: none (grandfathered) |
|
||||
| 48 | design-motion | skill | 4,295 | 4,995 | 86 | every use | untested | plan: none (grandfathered) |
|
||||
| 49 | test-audit | skill | 4,201 | 0 | 82 | every use | untested | plan: none (grandfathered) |
|
||||
| 50 | document-release | skill | 4,049 | 10,260 | 72 | every use | untested | plan: none (grandfathered) |
|
||||
| 51 | diagram | skill | 3,895 | 0 | 112 | every use | untested | plan: none (grandfathered) |
|
||||
| 52 | benchmark-models | skill | 3,822 | 0 | 77 | every use | untested | plan: none (grandfathered) |
|
||||
| 53 | ubicloud | skill | 3,593 | 0 | 83 | every use | untested | plan: none (grandfathered) |
|
||||
| 54 | modernize | skill | 3,429 | 0 | 106 | every use | untested | plan: none (grandfathered) |
|
||||
| 55 | setup-browser-cookies | skill | 3,184 | 0 | 79 | every use | untested | plan: none (grandfathered) |
|
||||
| 56 | deslop-shared-libs | skill | 3,100 | 0 | 64 | every use | untested | plan: none (grandfathered) |
|
||||
| 57 | landing-report | skill | 2,971 | 0 | 71 | every use | untested | plan: none (grandfathered) |
|
||||
| 58 | ship-docs-audit | shared | 2,730 | n/a | n/a | every /ship run | untested | plan: folded into GEVA-2 registry row R56-4 (ship); proposed, not yet preregistered (grandfathered) |
|
||||
| 59 | verify | skill | 2,679 | 0 | 91 | every use | untested | plan: none (grandfathered) |
|
||||
| 60 | ios-design-review | skill | 2,367 | 0 | 76 | every use | untested | plan: none (grandfathered) |
|
||||
| 61 | ios-sync | skill | 2,240 | 0 | 90 | every use | untested | plan: none (grandfathered) |
|
||||
| 62 | ios-clean | skill | 2,214 | 0 | 86 | every use | untested | plan: none (grandfathered) |
|
||||
| 63 | write-skill | skill | 2,158 | 795 | 100 | every use | untested | plan: none (grandfathered) |
|
||||
| 64 | ios-fix | skill | 2,096 | 0 | 69 | every use | untested | plan: none (grandfathered) |
|
||||
| 65 | learn | skill | 1,989 | 0 | 66 | every use | untested | plan: none (grandfathered) |
|
||||
| 66 | careful | skill | 1,126 | 0 | 171 | every use | safety | reason: safety guard: warns before destructive commands (rm -rf, DROP TABLE, force-push); kept on that reason, an average score cannot see the rare bad run it stops |
|
||||
| 67 | freeze | skill | 1,043 | 0 | 234 | every use | safety | reason: safety guard: hard-blocks edits outside one directory; kept on that reason |
|
||||
| 68 | guard | skill | 908 | 0 | 348 | every use | safety | reason: safety guard: careful and freeze together; kept on that reason |
|
||||
| 69 | capy-factory | shared | 846 | n/a | n/a | every Capy session | untested | plan: none (grandfathered) |
|
||||
| 70 | unfreeze | skill | 372 | 0 | 65 | every use | safety | reason: safety guard: the off switch for freeze's edit restriction; kept with the guard it releases |
|
||||
| 71 | claude-code | skill | n/a | n/a | n/a | every use | untested | plan: none (grandfathered) |
|
||||
|
||||
## Measure next
|
||||
|
||||
Untested parts, largest load per run first. A fair test runs the claude_code preset, frontier models (Opus 5.5 and GPT-6.1 Sol, plus at most one named low-cost primary) and a masked no-gstack arm ([TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm)). Safety rows are not on this list.
|
||||
|
||||
| Order | Part | Load per run (tokens) | Plan |
|
||||
|---|---|---|---|
|
||||
| 1 | ship-review-panel | 22,830 | GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 2 | design-review | 20,159 | GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 3 | ship | 14,587 | GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 4 | plan-ceo-review | 14,394 | GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 5 | review | 13,990 | R2 (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-02-geva-122-r2-amendment (#129, main eae6efc), told /review vs a no-gstack arm, Opus 5.5, preset on both arms (grandfathered) |
|
||||
| 6 | devex-review | 13,911 | GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 7 | plan-design-review | 13,351 | GEVA-2 registry row R56-7 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 8 | retro | 12,257 | GEVA-2 registry row R56-12 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 9 | plan-devex-review | 11,104 | GEVA-2 registry row R56-6 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 10 | autoplan-phases | 10,709 | folded into GEVA-2 registry row R56-3 (autoplan); proposed, not yet preregistered (grandfathered) |
|
||||
| 11 | office-hours | 10,424 | none (grandfathered) |
|
||||
| 12 | autoplan | 10,288 | GEVA-2 registry row R56-3 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 13 | plan-eng-review | 9,837 | GEVA-2 registry row R56-2 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 14 | design-consultation | 9,557 | GEVA-2 registry row R56-10 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 15 | setup-gbrain | 9,099 | none (grandfathered) |
|
||||
| 16 | codex | 8,991 | none (grandfathered) |
|
||||
| 17 | sync-gbrain | 8,863 | none (grandfathered) |
|
||||
| 18 | design-shotgun | 8,470 | none (grandfathered) |
|
||||
| 19 | plan-tune | 8,377 | none (grandfathered) |
|
||||
| 20 | spec | 8,305 | none (grandfathered) |
|
||||
| 21 | browse | 8,125 | none (grandfathered) |
|
||||
| 22 | design-html | 8,096 | none (grandfathered) |
|
||||
| 23 | land-and-deploy | 7,652 | GEVA-2 registry row R56-8 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered) (grandfathered) |
|
||||
| 24 | benchmark | 7,430 | none (grandfathered) |
|
||||
| 25 | catalog | 7,291 | Run D (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-125-untold-discovery-prereg (#125, main 9820b14), untold gstack vs placebo vs no gstack, preset vs stub, Opus 5.5, GPT-6.1 Sol and particle-1; also measures /investigate's description (grandfathered) |
|
||||
| 26 | canary | 7,034 | none (grandfathered) |
|
||||
| 27 | qa | 6,951 | none (grandfathered) |
|
||||
| 28 | scrape | 6,647 | none (grandfathered) |
|
||||
| 29 | captain | 6,522 | none (grandfathered) |
|
||||
| 30 | preamble | 5,743 | I1 ceremony cut (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-108-ceremony-cut-prereg (#122, main 409ff47) with its G3 amendment (#128), preset, Opus 5.5 and particle-1 (grandfathered) |
|
||||
| 31 | gstack-upgrade | 5,726 | none (grandfathered) |
|
||||
| 32 | context-restore | 5,715 | none (grandfathered) |
|
||||
| 33 | document-generate | 5,466 | none (grandfathered) |
|
||||
| 34 | ios-qa | 5,368 | none (grandfathered) |
|
||||
| 35 | skillify | 5,317 | none (grandfathered) |
|
||||
| 36 | capy-skill | 5,309 | none (grandfathered) |
|
||||
| 37 | health | 5,287 | none (grandfathered) |
|
||||
| 38 | setup-deploy | 5,216 | none (grandfathered) |
|
||||
| 39 | qa-only | 5,205 | none (grandfathered) |
|
||||
| 40 | eval-plan | 5,021 | none (grandfathered) |
|
||||
| 41 | gstack | 4,951 | none (grandfathered) |
|
||||
| 42 | pair-agent | 4,811 | none (grandfathered) |
|
||||
| 43 | make-pdf | 4,730 | none (grandfathered) |
|
||||
| 44 | context-save | 4,637 | none (grandfathered) |
|
||||
| 45 | open-gstack-browser | 4,429 | none (grandfathered) |
|
||||
| 46 | design-motion | 4,295 | none (grandfathered) |
|
||||
| 47 | test-audit | 4,201 | none (grandfathered) |
|
||||
| 48 | document-release | 4,049 | none (grandfathered) |
|
||||
| 49 | diagram | 3,895 | none (grandfathered) |
|
||||
| 50 | benchmark-models | 3,822 | none (grandfathered) |
|
||||
| 51 | ubicloud | 3,593 | none (grandfathered) |
|
||||
| 52 | modernize | 3,429 | none (grandfathered) |
|
||||
| 53 | setup-browser-cookies | 3,184 | none (grandfathered) |
|
||||
| 54 | deslop-shared-libs | 3,100 | none (grandfathered) |
|
||||
| 55 | landing-report | 2,971 | none (grandfathered) |
|
||||
| 56 | ship-docs-audit | 2,730 | folded into GEVA-2 registry row R56-4 (ship); proposed, not yet preregistered (grandfathered) |
|
||||
| 57 | verify | 2,679 | none (grandfathered) |
|
||||
| 58 | ios-design-review | 2,367 | none (grandfathered) |
|
||||
| 59 | ios-sync | 2,240 | none (grandfathered) |
|
||||
| 60 | ios-clean | 2,214 | none (grandfathered) |
|
||||
| 61 | write-skill | 2,158 | none (grandfathered) |
|
||||
| 62 | ios-fix | 2,096 | none (grandfathered) |
|
||||
| 63 | learn | 1,989 | none (grandfathered) |
|
||||
| 64 | capy-factory | 846 | none (grandfathered) |
|
||||
| 65 | claude-code | n/a | none (grandfathered) |
|
||||
|
||||
## Shared parts and preamble tiers
|
||||
|
||||
| Tier | Skills | What loads |
|
||||
|---|---|---|
|
||||
| 1 | benchmark, benchmark-models, browse, diagram, gstack, make-pdf, open-gstack-browser, scrape, setup-browser-cookies, verify | a short preamble inline in SKILL.md |
|
||||
| 2 | canary, captain, context-restore, context-save, design-html, design-motion, design-shotgun, document-generate, document-release, eval-plan, health, investigate, ios-clean, ios-design-review, ios-fix, ios-sync, landing-report, learn, modernize, pair-agent, plan-tune, retro, setup-deploy, setup-gbrain, skillify, sync-gbrain, test-audit, ubicloud, write-skill | the shared sections/preamble.md (row preamble), once per session |
|
||||
| 3 | autoplan, claude-code, codex, design-consultation, devex-review, ios-qa, office-hours, plan-ceo-review, plan-design-review, plan-devex-review, plan-eng-review, spec | the shared sections/preamble.md (row preamble), once per session |
|
||||
| 4 | design-review, land-and-deploy, qa, qa-only, review, ship | the shared sections/preamble.md (row preamble), once per session |
|
||||
| none | careful, cso, deslop-shared-libs, freeze, gstack-upgrade, guard, unfreeze | no preamble |
|
||||
|
||||
- **preamble**: shared preamble (sections/preamble.md, identical in every tier 2+ skill) (ship/sections/preamble.md); loads every tier 2+ skill run, inside that skill's load.
|
||||
- **catalog**: skill catalog (every skill's frontmatter); loads every session.
|
||||
- **capy-skill**: capy/skill/gstack/SKILL.md (capy/skill/gstack/SKILL.md); loads every Capy session.
|
||||
- **capy-factory**: capy/skill/gstack/factory.md (capy/skill/gstack/factory.md); loads every Capy session.
|
||||
- **ship-review-panel**: /ship review panel (review-army and adversarial sections) (ship/sections/review-army.md, ship/sections/adversarial.md); loads every /ship run, by depth.
|
||||
- **ship-docs-audit**: /ship docs audit (Step 14.5) (ship/sections/documentation.md); loads every /ship run.
|
||||
- **autoplan-phases**: /autoplan CEO, design, DX and eng phases (autoplan/sections/ceo-phase.md, autoplan/sections/design-phase.md, autoplan/sections/dx-phase.md, autoplan/sections/eng-phase.md); loads every /autoplan run, by scope.
|
||||
|
||||
## Paid dispatches per run
|
||||
|
||||
Each count is quoted from the file and line named; `bin/gstack-inventory --check` fails when a quoted line moves away.
|
||||
|
||||
| Skill | What | Count | Source |
|
||||
|---|---|---|---|
|
||||
| ship | Step 9 review, pass 1, light depth, docs or tests only | 1 reader (the core checklist) | docs/review-policy.md:194 |
|
||||
| ship | Step 9 review, pass 1, light depth, code | 3 readers (core, one batched specialist, native adversarial), plus the outside voice on a diff that publishes numbers | docs/review-policy.md:195 |
|
||||
| ship | Step 9 review, pass 1, standard depth | the batched specialist, the Red Team at 200 runtime lines, adversarial and outside voice | docs/review-policy.md:196 |
|
||||
| ship | Step 9 review, pass 1, full depth | the batched specialist on the whole diff and the Red Team always | docs/review-policy.md:197 |
|
||||
| ship | targeted reader on money, gate, auth or secret code | +1 at every depth | docs/review-policy.md:319 |
|
||||
| ship | audits (Steps 7, 8 and 14.5) | 3 | lib/ship-budget-calibration.json:3 |
|
||||
| ship | measured: one /ship child (gstack-evals#31) | 10 review dispatches | lib/ship-budget-calibration.json:43 |
|
||||
| ship | prior cost of one dispatch on the Capy meter | 7,430,000 tokens | lib/ship-budget-calibration.json:56 |
|
||||
| autoplan | voices per phase (native and outside) | 2 | autoplan/DRIVER.md.tmpl:25 |
|
||||
| autoplan | phases per round: CEO and Eng always, Design with UI scope, DX with developer-facing scope | 2 to 4, so 4 to 8 voice dispatches a round | autoplan/SKILL.md.tmpl:99 |
|
||||
| autoplan | CEO spec review loop | up to 3 launches | autoplan/sections/ceo-phase.md.tmpl:50 |
|
||||
| autoplan | review rounds per phase (new unattended runs) | up to 2 | lib/autoplan-budget.ts:23 |
|
||||
| autoplan | reserved per voice on Capy or under a spend cap | $5 | lib/autoplan-budget.ts:25 |
|
||||
|
||||
## Earlier results that do not count yet
|
||||
|
||||
- `stub`: ran on the Agent SDK stub prompt, not the claude_code preset.
|
||||
- `small-tier`: Haiku, Sonnet or another non-frontier model only.
|
||||
- `unmasked`: no mask recorded: the no-gstack arm predates the masking rule and was not re-checked.
|
||||
- `ceiling`: control at the ceiling, so no difference could show.
|
||||
- `no-true-no-gstack-arm`: the none arm still had gstack installed.
|
||||
- `no-exposure`: the model never used the part.
|
||||
- `not-claude-code`: ran outside Claude Code (Messages API, Codex or Capy).
|
||||
- `variant-only`: variant against variant, no no-gstack arm.
|
||||
|
||||
| Part | Result | Why it does not count | Note |
|
||||
|---|---|---|---|
|
||||
| investigate | gstack-evals #109 stacks round 1 TB2 (GEVA-89, GEVA-93), wiki/board/2026-11-01-geva-93-stacks-round-1-tb2-results | stub, no-exposure, unmasked | Opus +3 points [-3, +9] at 1.9x cost; debug report in 0 of 111 trials; particle-1 -5 [-14, +1] |
|
||||
| investigate | GSTA-12 Lane B quick look 2, wiki/board/2026-10-29-gsta-12-laneb-quick-2-results | stub, unmasked | Opus gstack minus none -7 [-13, -2] at 1.7x cost; did not replicate |
|
||||
| investigate | GSTA-30 Lane B quick look 3 (#89), wiki/board/2026-10-30-gsta-30-laneb-quick-3-results | stub, small-tier | gstack arms on Sonnet, particle-1, DeepSeek and Haiku; Opus ran the none arm only |
|
||||
| investigate | HARN-14 replication (harness-evals #181, unmerged) | stub, small-tier | Haiku 5.5: gstack minus none +3.5 [-12.3, +20.0] |
|
||||
| learn | GSTA-13, GEVA-54 and GEVA-65 Lane A runs (wiki/board/2026-10-28-gsta-13-quick-look-3-results and the GEVA-54 and GEVA-65 posts) | stub, no-true-no-gstack-arm, ceiling | Opus contract tasks 27/30 vs 27/30 |
|
||||
| learn | GEVA-65 Lane A rerun row 5 (GPT-6.1 Sol in Codex) | not-claude-code, no-true-no-gstack-arm | Sol failed the no-code-change safety task in 5 of 9 skill trials vs 0 of 3 without; small n, a lead to check, not a hurts verdict |
|
||||
| office-hours | GSTA-13 quick look 3 Lane A, wiki/board/2026-10-28-gsta-13-quick-look-3-results | stub, no-true-no-gstack-arm | contract tasks on Opus: gstack 74% vs none 28%; the tasks grade the skill's own contract, a manipulation check, not lift |
|
||||
| office-hours | GEVA-65 Lane A rerun, wiki/board/2026-10-31-geva-65-lanea-shakedown-rerun-results | stub, no-true-no-gstack-arm | Opus before 0.67, after 0.76, none 0.36 |
|
||||
| plan-ceo-review | CEO panel v2 admission (GEVA-142), wiki/board/2026-11-02-geva-142-ceo-panel-v2-admission-results (open PR #131) | ceiling, unmasked | preset, Opus 5.5 only: must-fix recall 1.00 in all three arms; plain invocation ran the procedure in 0 of 16 |
|
||||
| plan-ceo-review | Planning R1a pilot (GEVA-102), wiki/board/2026-11-01-geva-102-planning-r1a-pilot-stop (#114) | stub, ceiling, unmasked | none-opus 8/8; gstack-opus read the sections file in 0 of 4 |
|
||||
| review | gstack-evals #108 stacks round 1 review (GEVA-89), wiki/board/2026-11-01-geva-89-stacks-round-1-review-results | stub, no-exposure, unmasked | Opus 5.5 blocking recall 9.5% vs 7.1%, +2.4 points [-6.4, +11.1]; specialists and adversarial pass ran in 0 of 66 sessions |
|
||||
| review | gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results | stub, unmasked | gstack-opus minus none-opus +3.5 points [-9.6, +16.7] |
|
||||
| review | GSTA-11 quick look 1 review replay, wiki/board/2026-10-28-gsta-11-review-replay-results | not-claude-code, unmasked | /review's prompts through the Messages API; Opus recall 44% vs 39% plain, 12 bugs |
|
||||
| preamble | GEVA-100 exposure counts, org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100) | stub, no-exposure | ceremony sections ran in 0 of 66 Opus review sessions and 6 of 111 Opus TB2 trials; no effect estimate |
|
||||
| catalog | GSTA-12 Lane B quick look 1, wiki/board/2026-10-29-gsta-12-laneb-quick-results | stub, no-exposure, unmasked | untold skill opened in 0 of 180 gstack-arm trials; Opus gstack minus none +5 [-3, +17] |
|
||||
| ship-review-panel | gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results | stub, not-claude-code, unmasked | arm C minus none +56.9 points [40.5, 72.2]; the none arm ran on the stub, arm C in Capy at another gstack version, on a cohort mined from gstack's own catches |
|
||||
| ship-review-panel | GSTA-48 review no-loss arm D (#77), wiki/board/2026-10-30-gsta-48-review-noloss-arm-how-to | variant-only, not-claude-code | arm D vs pooled arm C, no no-gstack arm |
|
||||
|
||||
## Key list and aliases
|
||||
|
||||
71 keys: 64 skills (the folders `discoverTemplates()` finds, the root skill as `gstack`) and 7 shared parts.
|
||||
|
||||
- Skills: autoplan, benchmark, benchmark-models, browse, canary, captain, careful, claude-code, codex, context-restore, context-save, cso, design-consultation, design-html, design-motion, design-review, design-shotgun, deslop-shared-libs, devex-review, diagram, document-generate, document-release, eval-plan, freeze, gstack, gstack-upgrade, guard, health, investigate, ios-clean, ios-design-review, ios-fix, ios-qa, ios-sync, land-and-deploy, landing-report, learn, make-pdf, modernize, office-hours, open-gstack-browser, pair-agent, plan-ceo-review, plan-design-review, plan-devex-review, plan-eng-review, plan-tune, qa, qa-only, retro, review, scrape, setup-browser-cookies, setup-deploy, setup-gbrain, ship, skillify, spec, sync-gbrain, test-audit, ubicloud, unfreeze, verify, write-skill.
|
||||
- Shared: preamble, catalog, capy-skill, capy-factory, ship-review-panel, ship-docs-audit, autoplan-phases.
|
||||
- Installed names: `gstack-<name>` is key `<name>` (`gstack-ship` is `ship`); `gstack-upgrade` keeps its name; the root skill installs as `gstack`.
|
||||
- Other names (the ledger's `aliases`): gstack (root skill) → gstack; upgrade → gstack-upgrade; preamble (ceremony) → preamble; always-on skill catalog (descriptions) → catalog; capy/skill/gstack/SKILL.md (Capy context) → capy-skill; capy/skill/gstack/factory.md (Capy context) → capy-factory; ship: review panel (specialist, adversarial, targeted, outside voice, red team) → ship-review-panel; ship: docs audit (Step 14.5) → ship-docs-audit; autoplan: CEO, design, DX and eng phases, outside voice → autoplan-phases.
|
||||
|
||||
## Not in this inventory
|
||||
|
||||
- Out of scope: openclaw/skills/* and browser-skills/* (not gstack skills an agent loads from this checkout), and connect-chrome (a symlink to open-gstack-browser, counted once).
|
||||
- Evidence-table row "review: Review Army and adversarial pass": a section inside /review, not its own key; R2 measures it as part of the review row's plan.
|
||||
- Evidence-table row "ship: QA probes (Step 9.2.1)": a step inside /ship with no separate size; folded into the ship row (GEVA-2 registry row R56-4).
|
||||
- Evidence-table row "gstack as a whole (installed, any skill)": not a single text part; GEVA-2's real-world outcome lane measures it (gstack-evals plan #111), not a measurement plan for any one part.
|
||||
@@ -0,0 +1,661 @@
|
||||
{
|
||||
"schema_version": 1,
|
||||
"note": "R56 evidence ledger (ETHOS.md: 9. Earn Its Place). One row per inventory key (docs/inventory.md lists them); bin/gstack-inventory --check validates it and test/inventory.test.ts runs that check. Statuses and their required fields: helps, no-effect, hurts need sources, models (frontier per R54), prompt: preset, scope and masked: true, and no-effect and hurts also r48 {verdict, ref}; untested needs plan (a reference, or none only when grandfathered), and a new part also reason, owner and measure_by; safety needs reason. prior lists earlier results that do not count yet, with why. Seeded 2026-11-02 from GEVA-2's table (org-gstack-gbrain/research/gstack-part-evidence.json, gstack dac71371a); every row present at that base is grandfathered.",
|
||||
"seed_source": "org-gstack-gbrain/research/gstack-part-evidence.json (GEVA-156 for GST-915, 2026-11-02, gstack-evals main 409ff47)",
|
||||
"parts": {
|
||||
"autoplan": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-3 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: autoplan: untested (never measured)"
|
||||
},
|
||||
"benchmark": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: benchmark: untested (never measured)"
|
||||
},
|
||||
"benchmark-models": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: benchmark-models: untested (never measured)"
|
||||
},
|
||||
"browse": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: browse: untested (never measured)"
|
||||
},
|
||||
"canary": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: canary: untested (never measured)"
|
||||
},
|
||||
"captain": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: captain: untested (never measured)"
|
||||
},
|
||||
"careful": {
|
||||
"status": "safety",
|
||||
"grandfathered": true,
|
||||
"reason": "safety guard: warns before destructive commands (rm -rf, DROP TABLE, force-push); kept on that reason, an average score cannot see the rare bad run it stops",
|
||||
"seed": "GEVA-2, 2026-11-02: careful: exempt (safety guard)"
|
||||
},
|
||||
"claude-code": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: claude-code: untested (never measured)"
|
||||
},
|
||||
"codex": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "not in GEVA-2's table (2026-11-02); never measured"
|
||||
},
|
||||
"context-restore": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: context-restore: untested (never measured)"
|
||||
},
|
||||
"context-save": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: context-save: untested (never measured)"
|
||||
},
|
||||
"cso": {
|
||||
"status": "safety",
|
||||
"grandfathered": true,
|
||||
"reason": "security: supported security findings with explicit coverage; kept on its security reason (GEVA-2 marks it exempt; GStack Minder may reclassify it)",
|
||||
"seed": "GEVA-2, 2026-11-02: cso: exempt (security audit skill, treated as exempt; GStack Minder may reclassify)"
|
||||
},
|
||||
"design-consultation": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-10 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: design-consultation: untested (never measured)"
|
||||
},
|
||||
"design-html": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: design-html: untested (never measured)"
|
||||
},
|
||||
"design-motion": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: design-motion: untested (never measured)"
|
||||
},
|
||||
"design-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-9 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: design-review: untested (never measured)"
|
||||
},
|
||||
"design-shotgun": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: design-shotgun: untested (never measured)"
|
||||
},
|
||||
"deslop-shared-libs": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: deslop-shared-libs: untested (never measured)"
|
||||
},
|
||||
"devex-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-11 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: devex-review: untested (never measured)"
|
||||
},
|
||||
"diagram": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: diagram: untested (never measured)"
|
||||
},
|
||||
"document-generate": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: document-generate: untested (never measured)"
|
||||
},
|
||||
"document-release": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: document-release: untested (never measured)"
|
||||
},
|
||||
"eval-plan": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: eval-plan: untested (never measured)"
|
||||
},
|
||||
"freeze": {
|
||||
"status": "safety",
|
||||
"grandfathered": true,
|
||||
"reason": "safety guard: hard-blocks edits outside one directory; kept on that reason",
|
||||
"seed": "GEVA-2, 2026-11-02: freeze: exempt (safety guard)"
|
||||
},
|
||||
"gstack": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: gstack (root skill): untested (never measured)",
|
||||
"aliases": [
|
||||
"gstack (root skill)"
|
||||
]
|
||||
},
|
||||
"gstack-upgrade": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: upgrade: untested (never measured)",
|
||||
"aliases": [
|
||||
"upgrade"
|
||||
]
|
||||
},
|
||||
"guard": {
|
||||
"status": "safety",
|
||||
"grandfathered": true,
|
||||
"reason": "safety guard: careful and freeze together; kept on that reason",
|
||||
"seed": "GEVA-2, 2026-11-02: guard: exempt (safety guard)"
|
||||
},
|
||||
"health": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: health: untested (never measured)"
|
||||
},
|
||||
"investigate": {
|
||||
"status": "no-effect",
|
||||
"grandfathered": true,
|
||||
"sources": [
|
||||
"GST-814 (HARN-1, 2026-11-02)",
|
||||
"org-gstack-gbrain handoffs/HARN-14/hard-follow-2026-11/decision-rule.md"
|
||||
],
|
||||
"models": [
|
||||
"claude-opus-5-5"
|
||||
],
|
||||
"prompt": "preset",
|
||||
"scope": "skill step procedure follow rate: 0 of 46 gstack sessions ran any step (0 of 30 on the claude_code preset); body loaded every time; the outcome check voided at ceiling (no-gstack fixed 8 of 8)",
|
||||
"masked": true,
|
||||
"r48": {
|
||||
"verdict": "CUT-CANDIDATE",
|
||||
"ref": "GST-938 (GSTA-251 plans hooks-or-drop)"
|
||||
},
|
||||
"plan": "R3 (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-101-r48-follow-ups-prereg (#112, main f0b81c0) and its amendment (open PR #133), told /investigate vs no gstack on SWE-bench Verified, preset, Opus 5.5 then GPT-6.1 Sol",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "gstack-evals #109 stacks round 1 TB2 (GEVA-89, GEVA-93), wiki/board/2026-11-01-geva-93-stacks-round-1-tb2-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-exposure",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "Opus +3 points [-3, +9] at 1.9x cost; debug report in 0 of 111 trials; particle-1 -5 [-14, +1]"
|
||||
},
|
||||
{
|
||||
"ref": "GSTA-12 Lane B quick look 2, wiki/board/2026-10-29-gsta-12-laneb-quick-2-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "Opus gstack minus none -7 [-13, -2] at 1.7x cost; did not replicate"
|
||||
},
|
||||
{
|
||||
"ref": "GSTA-30 Lane B quick look 3 (#89), wiki/board/2026-10-30-gsta-30-laneb-quick-3-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"small-tier"
|
||||
],
|
||||
"note": "gstack arms on Sonnet, particle-1, DeepSeek and Haiku; Opus ran the none arm only"
|
||||
},
|
||||
{
|
||||
"ref": "HARN-14 replication (harness-evals #181, unmerged)",
|
||||
"why": [
|
||||
"stub",
|
||||
"small-tier"
|
||||
],
|
||||
"note": "Haiku 5.5: gstack minus none +3.5 [-12.3, +20.0]"
|
||||
}
|
||||
],
|
||||
"seed": "GEVA-2, 2026-11-02: investigate: measurement under way (R3)",
|
||||
"note": "Only Opus 5.5 (claude-opus-5-5) was measured; GPT-6.1 Sol was not, so this verdict covers one of the two frontier models ETHOS.md 9 names."
|
||||
},
|
||||
"ios-clean": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ios-clean: untested (never measured)"
|
||||
},
|
||||
"ios-design-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ios-design-review: untested (never measured)"
|
||||
},
|
||||
"ios-fix": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ios-fix: untested (never measured)"
|
||||
},
|
||||
"ios-qa": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ios-qa: untested (never measured)"
|
||||
},
|
||||
"ios-sync": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ios-sync: untested (never measured)"
|
||||
},
|
||||
"land-and-deploy": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-8 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: land-and-deploy: untested (never measured)"
|
||||
},
|
||||
"landing-report": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: landing-report: untested (never measured)"
|
||||
},
|
||||
"learn": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "GSTA-13, GEVA-54 and GEVA-65 Lane A runs (wiki/board/2026-10-28-gsta-13-quick-look-3-results and the GEVA-54 and GEVA-65 posts)",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-true-no-gstack-arm",
|
||||
"ceiling"
|
||||
],
|
||||
"note": "Opus contract tasks 27/30 vs 27/30"
|
||||
},
|
||||
{
|
||||
"ref": "GEVA-65 Lane A rerun row 5 (GPT-6.1 Sol in Codex)",
|
||||
"why": [
|
||||
"not-claude-code",
|
||||
"no-true-no-gstack-arm"
|
||||
],
|
||||
"note": "Sol failed the no-code-change safety task in 5 of 9 skill trials vs 0 of 3 without; small n, a lead to check, not a hurts verdict"
|
||||
}
|
||||
],
|
||||
"seed": "GEVA-2, 2026-11-02: learn: untested (unfair test: SDK stub; no true no-gstack arm; Opus at ceiling; Sol ran in Codex, not Claude Code)"
|
||||
},
|
||||
"make-pdf": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: make-pdf: untested (never measured)"
|
||||
},
|
||||
"modernize": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: modernize: untested (never measured)"
|
||||
},
|
||||
"office-hours": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "GSTA-13 quick look 3 Lane A, wiki/board/2026-10-28-gsta-13-quick-look-3-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-true-no-gstack-arm"
|
||||
],
|
||||
"note": "contract tasks on Opus: gstack 74% vs none 28%; the tasks grade the skill's own contract, a manipulation check, not lift"
|
||||
},
|
||||
{
|
||||
"ref": "GEVA-65 Lane A rerun, wiki/board/2026-10-31-geva-65-lanea-shakedown-rerun-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-true-no-gstack-arm"
|
||||
],
|
||||
"note": "Opus before 0.67, after 0.76, none 0.36"
|
||||
}
|
||||
],
|
||||
"seed": "GEVA-2, 2026-11-02: office-hours: untested (unfair test: SDK stub; no true no-gstack arm; tasks grade the skill's own contract, a manipulation check, not lift)"
|
||||
},
|
||||
"open-gstack-browser": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: open-gstack-browser: untested (never measured)"
|
||||
},
|
||||
"pair-agent": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: pair-agent: untested (never measured)"
|
||||
},
|
||||
"plan-ceo-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-5 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "CEO panel v2 admission (GEVA-142), wiki/board/2026-11-02-geva-142-ceo-panel-v2-admission-results (open PR #131)",
|
||||
"why": [
|
||||
"ceiling",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "preset, Opus 5.5 only: must-fix recall 1.00 in all three arms; plain invocation ran the procedure in 0 of 16"
|
||||
},
|
||||
{
|
||||
"ref": "Planning R1a pilot (GEVA-102), wiki/board/2026-11-01-geva-102-planning-r1a-pilot-stop (#114)",
|
||||
"why": [
|
||||
"stub",
|
||||
"ceiling",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "none-opus 8/8; gstack-opus read the sections file in 0 of 4"
|
||||
}
|
||||
],
|
||||
"r48": {
|
||||
"verdict": "FIX-METHOD",
|
||||
"ref": "CEO panel v2 admission (GEVA-142, open PR #131): FIX-METHOD on the panel, a harder panel is planned"
|
||||
},
|
||||
"seed": "GEVA-2, 2026-11-02: plan-ceo-review: untested (unfair test: ceiling, control 12 of 12 so no MDD; plain invocation ran the procedure in 0 of 16)"
|
||||
},
|
||||
"plan-design-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-7 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: plan-design-review: untested (never measured)"
|
||||
},
|
||||
"plan-devex-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-6 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: plan-devex-review: untested (never measured)"
|
||||
},
|
||||
"plan-eng-review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-2 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: plan-eng-review: untested (never measured)"
|
||||
},
|
||||
"plan-tune": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: plan-tune: untested (never measured)"
|
||||
},
|
||||
"qa": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: qa: untested (never measured)"
|
||||
},
|
||||
"qa-only": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: qa-only: untested (never measured)"
|
||||
},
|
||||
"retro": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-12 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: retro: untested (never measured)"
|
||||
},
|
||||
"review": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "R2 (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-02-geva-122-r2-amendment (#129, main eae6efc), told /review vs a no-gstack arm, Opus 5.5, preset on both arms",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "gstack-evals #108 stacks round 1 review (GEVA-89), wiki/board/2026-11-01-geva-89-stacks-round-1-review-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-exposure",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "Opus 5.5 blocking recall 9.5% vs 7.1%, +2.4 points [-6.4, +11.1]; specialists and adversarial pass ran in 0 of 66 sessions"
|
||||
},
|
||||
{
|
||||
"ref": "gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "gstack-opus minus none-opus +3.5 points [-9.6, +16.7]"
|
||||
},
|
||||
{
|
||||
"ref": "GSTA-11 quick look 1 review replay, wiki/board/2026-10-28-gsta-11-review-replay-results",
|
||||
"why": [
|
||||
"not-claude-code",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "/review's prompts through the Messages API; Opus recall 44% vs 39% plain, 12 bugs"
|
||||
}
|
||||
],
|
||||
"r48": {
|
||||
"verdict": "FIX-METHOD",
|
||||
"ref": "org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100); R2 is the fix"
|
||||
},
|
||||
"seed": "GEVA-2, 2026-11-02: review: measurement under way (R2)"
|
||||
},
|
||||
"scrape": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: scrape: untested (never measured)"
|
||||
},
|
||||
"setup-browser-cookies": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: setup-browser-cookies: untested (never measured)"
|
||||
},
|
||||
"setup-deploy": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: setup-deploy: untested (never measured)"
|
||||
},
|
||||
"setup-gbrain": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: setup-gbrain: untested (never measured)"
|
||||
},
|
||||
"ship": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-4 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"seed": "GEVA-2, 2026-11-02: ship: untested (never measured)"
|
||||
},
|
||||
"skillify": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: skillify: untested (never measured)"
|
||||
},
|
||||
"spec": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: spec: untested (never measured)"
|
||||
},
|
||||
"sync-gbrain": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: sync-gbrain: untested (never measured)"
|
||||
},
|
||||
"test-audit": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: test-audit: untested (never measured)"
|
||||
},
|
||||
"ubicloud": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: ubicloud: untested (never measured)"
|
||||
},
|
||||
"unfreeze": {
|
||||
"status": "safety",
|
||||
"grandfathered": true,
|
||||
"reason": "safety guard: the off switch for freeze's edit restriction; kept with the guard it releases",
|
||||
"seed": "GEVA-2, 2026-11-02: unfreeze: exempt (safety guard (undoes freeze))"
|
||||
},
|
||||
"verify": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "not in GEVA-2's table (2026-11-02); never measured"
|
||||
},
|
||||
"write-skill": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: write-skill: untested (never measured)"
|
||||
},
|
||||
"preamble": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "I1 ceremony cut (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-108-ceremony-cut-prereg (#122, main 409ff47) with its G3 amendment (#128), preset, Opus 5.5 and particle-1",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "GEVA-100 exposure counts, org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100)",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-exposure"
|
||||
],
|
||||
"note": "ceremony sections ran in 0 of 66 Opus review sessions and 6 of 111 Opus TB2 trials; no effect estimate"
|
||||
}
|
||||
],
|
||||
"r48": {
|
||||
"verdict": "CUT-CANDIDATE",
|
||||
"ref": "org-gstack-gbrain/research/why-null-stacks-r1.md (GEVA-100); taken by Garry as the I1 ceremony-cut eval"
|
||||
},
|
||||
"seed": "GEVA-2, 2026-11-02: preamble (ceremony): measurement under way (I1, ceremony-cut eval)",
|
||||
"aliases": [
|
||||
"preamble (ceremony)"
|
||||
]
|
||||
},
|
||||
"catalog": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "Run D (GEVA-2, 2026-11-02): gstack-evals wiki/plans/2026-11-01-geva-125-untold-discovery-prereg (#125, main 9820b14), untold gstack vs placebo vs no gstack, preset vs stub, Opus 5.5, GPT-6.1 Sol and particle-1; also measures /investigate's description",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "GSTA-12 Lane B quick look 1, wiki/board/2026-10-29-gsta-12-laneb-quick-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"no-exposure",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "untold skill opened in 0 of 180 gstack-arm trials; Opus gstack minus none +5 [-3, +17]"
|
||||
}
|
||||
],
|
||||
"seed": "GEVA-2, 2026-11-02: always-on skill catalog (descriptions): measurement under way (Run D)",
|
||||
"aliases": [
|
||||
"always-on skill catalog (descriptions)"
|
||||
]
|
||||
},
|
||||
"capy-skill": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: capy/skill/gstack/SKILL.md (Capy context): untested (never measured)",
|
||||
"aliases": [
|
||||
"capy/skill/gstack/SKILL.md (Capy context)"
|
||||
],
|
||||
"note": "loads only in Capy sessions, which the claude_code preset fair test cannot reach yet; needs a Captain call on how to test it"
|
||||
},
|
||||
"capy-factory": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "none",
|
||||
"seed": "GEVA-2, 2026-11-02: capy/skill/gstack/factory.md (Capy context): untested (never measured)",
|
||||
"aliases": [
|
||||
"capy/skill/gstack/factory.md (Capy context)"
|
||||
],
|
||||
"note": "loads only in Capy sessions, which the claude_code preset fair test cannot reach yet; needs a Captain call on how to test it"
|
||||
},
|
||||
"ship-review-panel": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "GEVA-2 registry row R56-1 (org-gstack-gbrain/research/gstack-part-evidence.json registry_rows; proposed, not yet preregistered)",
|
||||
"prior": [
|
||||
{
|
||||
"ref": "gstack-evals #120 R1 blind re-judge (GEVA-113), wiki/board/2026-11-01-geva-113-r1-blind-rejudge-results",
|
||||
"why": [
|
||||
"stub",
|
||||
"not-claude-code",
|
||||
"unmasked"
|
||||
],
|
||||
"note": "arm C minus none +56.9 points [40.5, 72.2]; the none arm ran on the stub, arm C in Capy at another gstack version, on a cohort mined from gstack's own catches"
|
||||
},
|
||||
{
|
||||
"ref": "GSTA-48 review no-loss arm D (#77), wiki/board/2026-10-30-gsta-48-review-noloss-arm-how-to",
|
||||
"why": [
|
||||
"variant-only",
|
||||
"not-claude-code"
|
||||
],
|
||||
"note": "arm D vs pooled arm C, no no-gstack arm"
|
||||
}
|
||||
],
|
||||
"seed": "GEVA-2, 2026-11-02: ship: review panel (specialist, adversarial, targeted, outside voice, red team): untested (unfair test: none arm on the SDK stub; arms from different harnesses and gstack versions; cohort mined from gstack's own catches)",
|
||||
"aliases": [
|
||||
"ship: review panel (specialist, adversarial, targeted, outside voice, red team)"
|
||||
]
|
||||
},
|
||||
"ship-docs-audit": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "folded into GEVA-2 registry row R56-4 (ship); proposed, not yet preregistered",
|
||||
"seed": "GEVA-2, 2026-11-02: ship: docs audit (Step 14.5): untested (never measured)",
|
||||
"aliases": [
|
||||
"ship: docs audit (Step 14.5)"
|
||||
]
|
||||
},
|
||||
"autoplan-phases": {
|
||||
"status": "untested",
|
||||
"grandfathered": true,
|
||||
"plan": "folded into GEVA-2 registry row R56-3 (autoplan); proposed, not yet preregistered",
|
||||
"seed": "GEVA-2, 2026-11-02: autoplan: CEO, design, DX and eng phases, outside voice: untested (never measured)",
|
||||
"aliases": [
|
||||
"autoplan: CEO, design, DX and eng phases, outside voice"
|
||||
]
|
||||
}
|
||||
},
|
||||
"unmapped": [
|
||||
{
|
||||
"row": "review: Review Army and adversarial pass",
|
||||
"why": "a section inside /review, not its own key; R2 measures it as part of the review row's plan"
|
||||
},
|
||||
{
|
||||
"row": "ship: QA probes (Step 9.2.1)",
|
||||
"why": "a step inside /ship with no separate size; folded into the ship row (GEVA-2 registry row R56-4)"
|
||||
},
|
||||
{
|
||||
"row": "gstack as a whole (installed, any skill)",
|
||||
"why": "not a single text part; GEVA-2's real-world outcome lane measures it (gstack-evals plan #111), not a measurement plan for any one part"
|
||||
}
|
||||
]
|
||||
}
|
||||
+5
-1
@@ -910,8 +910,12 @@ gstack's own templates and for any skill folder you own.
|
||||
add nothing a fresh session already does; edit the line that covers the idea instead of appending; put the fact on the
|
||||
lowest level that reaches the agent in time (description, SKILL.md, `references/`, the tool's `--help`, the tool's
|
||||
errors); write the rule and put the story in the commit message.
|
||||
- **Evidence.** A new skill, step, instruction, check or default needs evidence that it helps on frontier models, or a
|
||||
dated plan to measure it ([ETHOS.md: 9. Earn Its Place](../ETHOS.md#9-earn-its-place)). Commits carry an `Evidence:`
|
||||
line, and every gstack skill has a row in the evidence ledger, `docs/inventory/evidence.json`, which
|
||||
[docs/inventory.md](inventory.md) ranks by load per run.
|
||||
- **Size.** `gstack-skill-bill` prints bytes and estimated tokens per level, and in a gstack checkout the skill's eval
|
||||
task file (`EVAL TASKS:`). Commits carry
|
||||
task file (`EVAL TASKS:`) and its ledger row (`EVIDENCE:`). Commits carry
|
||||
`Skill-Size: <before>→<after> bytes`; a skill more than 10% over its last `Skill-Diet:` commit is DUE, and a red
|
||||
context-budget ratchet means diet, not a higher ceiling.
|
||||
- **Drift.** `test/skill-bin-drift.test.ts` fails when a template or rendered skill calls a gstack bin or flag that
|
||||
|
||||
@@ -4165,15 +4165,17 @@ refuses to start without a passing check.
|
||||
|
||||
**Meaning.** The preregistration says `stack: loaded` (some arm loads gstack or
|
||||
another skill stack) and its `## No-gstack arm` section is missing, lacks one
|
||||
of `arms:`, `inputs:`, `prompt:`, `panel:` or `report:`, lists an arm that
|
||||
of `arms:`, `inputs:`, `prompt:`, `panel:`, `report:`, `masking:`, `install:`
|
||||
or `audit:` (or says `none` for one of the last three), lists an arm that
|
||||
is not a `<model>#no-gstack` Arms line, or leaves a model in the comparison
|
||||
without its own no-gstack arm. Arms match on whole identities, so
|
||||
`claude-opus-5#no-gstack` does not cover `claude-opus-5-5`.
|
||||
|
||||
**Fix.** Add one `<model>#no-gstack` line per model under `## Arms`, list them
|
||||
on the `arms:` line, and fill the other four keys: the same inputs, a plain
|
||||
prompt with no skill text, the same panel and judge, and where the result row
|
||||
goes beside the variants. When no arm loads a skill stack, write
|
||||
on the `arms:` line, and fill the other keys: the same inputs, a plain
|
||||
prompt with no skill text, the same panel and judge, where the result row
|
||||
goes beside the variants, how the run agent is masked from the Capy drives,
|
||||
the check that no gstack is installed, and the per-arm drive-read audit. When no arm loads a skill stack, write
|
||||
`stack: none` with a `why:` line instead. The rule is in
|
||||
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
|
||||
|
||||
|
||||
+7
-6
@@ -140,7 +140,7 @@ Write (or open) the preregistration file. Each section is required and must be f
|
||||
| Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves |
|
||||
| Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted`, the first quota or rate-limit error on a seat-metered arm |
|
||||
| Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it |
|
||||
| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) |
|
||||
| No-gstack arm | required with `stack: loaded` (below) |
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init <path>
|
||||
@@ -153,11 +153,12 @@ preregistration.
|
||||
|
||||
**No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack
|
||||
(review variants, preamble loadouts, skill diets, a model panel under a stack) also
|
||||
runs each model with no skill stack at all. Add `<model>#no-gstack` to Arms for every
|
||||
model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows
|
||||
and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed
|
||||
skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels;
|
||||
`report:` its row beside the variants in every results table and verdict.
|
||||
runs each model with no skill stack. Add `<model>#no-gstack` to Arms for every model and
|
||||
fill `## No-gstack arm`: `arms:` those arms; `inputs:` same rows; `prompt:` the task text
|
||||
only, same tools; `panel:` the same kind, trials, judge and
|
||||
labels; `report:` its row beside the variants; `masking:` both Capy drives hidden from the
|
||||
run agent; `install:` no gstack installed; `audit:` drive reads per arm. An arm that read a
|
||||
gstack skill file is contaminated and gets zero credit.
|
||||
`PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows
|
||||
`stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out.
|
||||
|
||||
|
||||
@@ -62,7 +62,7 @@ Write (or open) the preregistration file. Each section is required and must be f
|
||||
| Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves |
|
||||
| Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted`, the first quota or rate-limit error on a seat-metered arm |
|
||||
| Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it |
|
||||
| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) |
|
||||
| No-gstack arm | required with `stack: loaded` (below) |
|
||||
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init <path>
|
||||
@@ -75,11 +75,12 @@ preregistration.
|
||||
|
||||
**No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack
|
||||
(review variants, preamble loadouts, skill diets, a model panel under a stack) also
|
||||
runs each model with no skill stack at all. Add `<model>#no-gstack` to Arms for every
|
||||
model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows
|
||||
and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed
|
||||
skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels;
|
||||
`report:` its row beside the variants in every results table and verdict.
|
||||
runs each model with no skill stack. Add `<model>#no-gstack` to Arms for every model and
|
||||
fill `## No-gstack arm`: `arms:` those arms; `inputs:` same rows; `prompt:` the task text
|
||||
only, same tools; `panel:` the same kind, trials, judge and
|
||||
labels; `report:` its row beside the variants; `masking:` both Capy drives hidden from the
|
||||
run agent; `install:` no gstack installed; `audit:` drive reads per arm. An arm that read a
|
||||
gstack skill file is contaminated and gets zero credit.
|
||||
`PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows
|
||||
`stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out.
|
||||
|
||||
|
||||
+5
-3
@@ -26,11 +26,12 @@ import { readLedger, release, reserve, settle, SpendCapExceeded, summarize, type
|
||||
export const PREREG_SECTIONS = ['Bars', 'Arms', 'Comparison', 'Decoys', 'Stop rules', 'Held-out exposure'] as const;
|
||||
/** `## Comparison`: `stack: loaded` when any arm loads gstack or another skill stack, `stack: none` with a `why:` otherwise. */
|
||||
export interface ComparisonCheck { stack: 'loaded' | 'none' | null; why: string | null; errors: string[] }
|
||||
/** `## No-gstack arm`, required when `stack: loaded`: one `<model>#no-gstack` Arms line per model, and the inputs, prompt, panel and report it shares with the variants. */
|
||||
/** `## No-gstack arm`, required when `stack: loaded`: one `<model>#no-gstack` Arms line per model, the inputs, prompt, panel and report it shares with the variants, and its masking, install and audit lines. */
|
||||
export interface NoGstackArmCheck { required: boolean; ok: boolean; arms: string[]; problems: string[] }
|
||||
export interface PreregCheck { ok: boolean; present: string[]; missing: string[]; empty: string[]; comparison: ComparisonCheck; no_gstack: NoGstackArmCheck; diagnostics: PreregDiagnostics | null }
|
||||
export const NO_GSTACK_LOADOUT = 'no-gstack';
|
||||
export const NO_GSTACK_KEYS = ['arms', 'inputs', 'prompt', 'panel', 'report'] as const;
|
||||
/** masking, install and audit keep the arm clean (TESTING_INTERNALS.md#no-gstack-arm): an arm that read a gstack skill file is contaminated. */
|
||||
export const NO_GSTACK_KEYS = ['arms', 'inputs', 'prompt', 'panel', 'report', 'masking', 'install', 'audit'] as const;
|
||||
|
||||
export function preregTemplate(title = 'Eval preregistration'): string {
|
||||
return `# ${title}
|
||||
@@ -45,7 +46,7 @@ export function preregTemplate(title = 'Eval preregistration'): string {
|
||||
<!-- \`stack: loaded\` when any arm loads gstack or another skill stack (review variants, preamble loadouts, skill diets, a model panel under a stack); otherwise \`stack: none\` and a \`why:\` line. -->
|
||||
|
||||
## No-gstack arm
|
||||
<!-- Required with \`stack: loaded\`: \`arms:\` one \`<model>#no-gstack\` Arms line per model; \`inputs:\` the same rows and fixture; \`prompt:\` the task only (no SKILL.md, preamble, hooks or installed skills); \`panel:\` the same E2E kind, trials and judge; \`report:\` its row beside the variants. -->
|
||||
<!-- Required with \`stack: loaded\`: \`arms:\` one \`<model>#no-gstack\` Arms line per model; \`inputs:\` the same rows and fixture; \`prompt:\` the task only (no SKILL.md, preamble, hooks or installed skills); \`panel:\` the same E2E kind, trials and judge; \`report:\` its row beside the variants; \`masking:\` how the run agent is kept from gstack (an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain before it starts, or a container that sees no Capy drive); \`install:\` no gstack at ~/.local/share/gstack or ~/.claude/skills; \`audit:\` the per-arm drive-read audit (an arm that read a gstack skill file is contaminated and gets zero credit). -->
|
||||
|
||||
## Decoys
|
||||
<!-- Inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves. -->
|
||||
@@ -112,6 +113,7 @@ function checkNoGstackArm(text: string, stack: ComparisonCheck['stack']): NoGsta
|
||||
if (!body?.length) return { required: true, ok: false, arms: [], problems: ['missing section'] };
|
||||
const kv = keyValues(body);
|
||||
const problems = NO_GSTACK_KEYS.filter(k => !kv[k]).map(k => `missing ${k}`);
|
||||
for (const k of ['masking', 'install', 'audit'] as const) if (/^(none|n\/a|no|-)\.?$/i.test(kv[k] ?? '')) problems.push(`${k}: ${kv[k]} is not a plan`);
|
||||
const listed = (kv.arms ?? '').split(/[\s,]+/).map(a => a.replace(/^`|`$/g, '')).filter(Boolean);
|
||||
const ids = armIdentities(text);
|
||||
for (const a of listed) {
|
||||
|
||||
@@ -0,0 +1,335 @@
|
||||
/**
|
||||
* inventory — every part of gstack, its load per run and its evidence (ETHOS.md: 9. Earn Its Place).
|
||||
*
|
||||
* Keys are the skill directories discoverTemplates() finds (the root skill as `gstack`) plus
|
||||
* SHARED_KEYS. The ledger, docs/inventory/evidence.json, is hand-kept: each row is checked by its
|
||||
* status's own fields (validateEntry). buildInventory joins the ledger with lib/context-bill.ts
|
||||
* buildBill (offline token estimates) and the dispatch counts ship and autoplan already document,
|
||||
* each found by its quoted line so a citation that drifts is named. Read-only except the CLI's write.
|
||||
*/
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { discoverTemplates } from '../scripts/discover-skills';
|
||||
import { buildBill, contentClass, TOKEN_DIVISOR, TOKEN_DIVISORS } from './context-bill';
|
||||
import { CAPY_INSTALL_ROOT } from './doctor-components';
|
||||
|
||||
export const LEDGER_PATH = 'docs/inventory/evidence.json';
|
||||
export const DOC_PATH = 'docs/inventory.md';
|
||||
export const ROOT_KEY = 'gstack';
|
||||
|
||||
export interface SharedPart { key: string; label: string; load: string; files: string[] }
|
||||
/** Shared parts: sized from their files (catalog from the bill's frontmatter). */
|
||||
export const SHARED_PARTS: SharedPart[] = [
|
||||
{ key: 'preamble', label: 'shared preamble (sections/preamble.md, identical in every tier 2+ skill)', load: 'every tier 2+ skill run, inside that skill\'s load', files: ['ship/sections/preamble.md'] },
|
||||
{ key: 'catalog', label: 'skill catalog (every skill\'s frontmatter)', load: 'every session', files: [] },
|
||||
{ key: 'capy-skill', label: 'capy/skill/gstack/SKILL.md', load: 'every Capy session', files: ['capy/skill/gstack/SKILL.md'] },
|
||||
{ key: 'capy-factory', label: 'capy/skill/gstack/factory.md', load: 'every Capy session', files: ['capy/skill/gstack/factory.md'] },
|
||||
{ key: 'ship-review-panel', label: '/ship review panel (review-army and adversarial sections)', load: 'every /ship run, by depth', files: ['ship/sections/review-army.md', 'ship/sections/adversarial.md'] },
|
||||
{ key: 'ship-docs-audit', label: '/ship docs audit (Step 14.5)', load: 'every /ship run', files: ['ship/sections/documentation.md'] },
|
||||
{ key: 'autoplan-phases', label: '/autoplan CEO, design, DX and eng phases', load: 'every /autoplan run, by scope', files: ['autoplan/sections/ceo-phase.md', 'autoplan/sections/design-phase.md', 'autoplan/sections/dx-phase.md', 'autoplan/sections/eng-phase.md'] },
|
||||
];
|
||||
export const SHARED_KEYS = SHARED_PARTS.map(p => p.key);
|
||||
|
||||
export interface Dispatch { skill: 'ship' | 'autoplan'; what: string; count: string; file: string; quote: string }
|
||||
/** Paid dispatches per run, from the budget tables and docs that already state them (no new instrumentation). */
|
||||
export const DISPATCHES: Dispatch[] = [
|
||||
{ skill: 'ship', what: 'Step 9 review, pass 1, light depth, docs or tests only', count: '1 reader (the core checklist)', file: 'docs/review-policy.md', quote: '| **light**, runtime 0 |' },
|
||||
{ skill: 'ship', what: 'Step 9 review, pass 1, light depth, code', count: '3 readers (core, one batched specialist, native adversarial), plus the outside voice on a diff that publishes numbers', file: 'docs/review-policy.md', quote: '| **light**, runtime above 0 |' },
|
||||
{ skill: 'ship', what: 'Step 9 review, pass 1, standard depth', count: 'the batched specialist, the Red Team at 200 runtime lines, adversarial and outside voice', file: 'docs/review-policy.md', quote: '| **standard** |' },
|
||||
{ skill: 'ship', what: 'Step 9 review, pass 1, full depth', count: 'the batched specialist on the whole diff and the Red Team always', file: 'docs/review-policy.md', quote: '| **full** |' },
|
||||
{ skill: 'ship', what: 'targeted reader on money, gate, auth or secret code', count: '+1 at every depth', file: 'docs/review-policy.md', quote: '`gstack-review-plan` dispatches one' },
|
||||
{ skill: 'ship', what: 'audits (Steps 7, 8 and 14.5)', count: '3', file: 'lib/ship-budget-calibration.json', quote: 'each audit dispatch (Steps 7, 8, 14.5)' },
|
||||
{ skill: 'ship', what: 'measured: one /ship child (gstack-evals#31)', count: '10 review dispatches', file: 'lib/ship-budget-calibration.json', quote: 'its 10 review dispatches' },
|
||||
{ skill: 'ship', what: 'prior cost of one dispatch on the Capy meter', count: '7,430,000 tokens', file: 'lib/ship-budget-calibration.json', quote: '"per_dispatch": 7430000' },
|
||||
{ skill: 'autoplan', what: 'voices per phase (native and outside)', count: '2', file: 'autoplan/DRIVER.md.tmpl', quote: 'per voice. Dispatch the native prompt to a subagent on your model and the outside prompt to' },
|
||||
{ skill: 'autoplan', what: 'phases per round: CEO and Eng always, Design with UI scope, DX with developer-facing scope', count: '2 to 4, so 4 to 8 voice dispatches a round', file: 'autoplan/SKILL.md.tmpl', quote: 'Phases MUST run in strict order: CEO → Design (if UI scope) → DX (if developer-facing' },
|
||||
{ skill: 'autoplan', what: 'CEO spec review loop', count: 'up to 3 launches', file: 'autoplan/sections/ceo-phase.md.tmpl', quote: 'three-launch cap' },
|
||||
{ skill: 'autoplan', what: 'review rounds per phase (new unattended runs)', count: 'up to 2', file: 'lib/autoplan-budget.ts', quote: 'export const DEFAULT_MAX_ROUNDS = 2;' },
|
||||
{ skill: 'autoplan', what: 'reserved per voice on Capy or under a spend cap', count: '$5', file: 'lib/autoplan-budget.ts', quote: 'export const DEFAULT_HOST_ESTIMATE_USD = 5;' },
|
||||
];
|
||||
|
||||
export const STATUSES = ['helps', 'no-effect', 'hurts', 'untested', 'safety'] as const;
|
||||
export type Status = (typeof STATUSES)[number];
|
||||
export const R48_VERDICTS = ['KEEP', 'FIX-METHOD', 'CUT-CANDIDATE'] as const;
|
||||
/** R54: the frontier models under test; a row may add at most one named low-cost primary, never Haiku or Sonnet. */
|
||||
export const FRONTIER_MODELS = ['claude-opus-5-5', 'gpt-6.1-sol'];
|
||||
/** Why an earlier result does not count as fair evidence yet. */
|
||||
export const PRIOR_WHY: Record<string, string> = {
|
||||
stub: 'ran on the Agent SDK stub prompt, not the claude_code preset',
|
||||
'small-tier': 'Haiku, Sonnet or another non-frontier model only',
|
||||
unmasked: 'no mask recorded: the no-gstack arm predates the masking rule and was not re-checked',
|
||||
ceiling: 'control at the ceiling, so no difference could show',
|
||||
'no-true-no-gstack-arm': 'the none arm still had gstack installed',
|
||||
'no-exposure': 'the model never used the part',
|
||||
'not-claude-code': 'ran outside Claude Code (Messages API, Codex or Capy)',
|
||||
'variant-only': 'variant against variant, no no-gstack arm',
|
||||
};
|
||||
const FIELDS = ['status', 'grandfathered', 'plan', 'prior', 'reason', 'owner', 'measure_by', 'sources', 'models', 'prompt', 'scope', 'masked', 'r48', 'seed', 'aliases', 'note'];
|
||||
|
||||
export type LedgerCode =
|
||||
| 'LEDGER_PARSE' | 'LEDGER_KEY_MISSING' | 'LEDGER_KEY_EXTRA' | 'LEDGER_STATUS_INVALID' | 'LEDGER_FIELD_UNKNOWN'
|
||||
| 'LEDGER_SOURCES_MISSING' | 'LEDGER_MODELS_NOT_FRONTIER' | 'LEDGER_PROMPT_NOT_PRESET' | 'LEDGER_SCOPE_MISSING'
|
||||
| 'LEDGER_UNMASKED' | 'LEDGER_R48_MISSING' | 'LEDGER_PLAN_MISSING' | 'LEDGER_PLAN_NONE_NEW' | 'LEDGER_PLAN_NOT_CONCRETE'
|
||||
| 'LEDGER_REASON_MISSING' | 'LEDGER_OWNER_MISSING' | 'LEDGER_MEASURE_BY_MISSING' | 'LEDGER_PRIOR_INVALID' | 'LEDGER_FIELD_INVALID'
|
||||
| 'INVENTORY_SOURCE_MISSING' | 'INVENTORY_PART_MISSING';
|
||||
export interface LedgerProblem { key: string; code: LedgerCode; message: string }
|
||||
export interface LedgerEntry { status: Status; grandfathered?: boolean; plan?: string; prior?: Array<{ ref: string; why: string[]; note?: string }>; reason?: string; owner?: string; measure_by?: string; sources?: string[]; models?: string[]; prompt?: string; scope?: string; masked?: boolean; r48?: { verdict: string; ref: string }; seed?: string; aliases?: string[]; note?: string }
|
||||
export interface Ledger { parts: Record<string, LedgerEntry>; unmapped?: Array<{ row: string; why: string }>; [k: string]: unknown }
|
||||
|
||||
const isObj = (v: unknown): v is Record<string, unknown> => typeof v === 'object' && v !== null && !Array.isArray(v);
|
||||
const str = (v: unknown): v is string => typeof v === 'string' && v.trim() !== '';
|
||||
const validR48 = (v: unknown) => isObj(v) && (R48_VERDICTS as readonly string[]).includes(v.verdict as string) && str(v.ref);
|
||||
|
||||
function frontierModels(v: unknown): boolean {
|
||||
if (!Array.isArray(v) || !v.length || !v.every(str)) return false;
|
||||
const ids = v.map(m => m.slice(m.lastIndexOf('/') + 1));
|
||||
const other = ids.filter(m => !FRONTIER_MODELS.includes(m));
|
||||
return ids.length > other.length && other.length <= 1 && !other.some(m => /haiku|sonnet/i.test(m));
|
||||
}
|
||||
|
||||
/** Problems with one ledger row, by its status's own fields (helps, no-effect, hurts, untested, safety). */
|
||||
export function validateEntry(key: string, entry: unknown): LedgerProblem[] {
|
||||
const problems: LedgerProblem[] = [];
|
||||
const add = (code: LedgerCode, message: string) => problems.push({ key, code, message });
|
||||
if (!isObj(entry)) { add('LEDGER_STATUS_INVALID', 'the row is not an object'); return problems; }
|
||||
for (const f of Object.keys(entry)) if (!FIELDS.includes(f)) add('LEDGER_FIELD_UNKNOWN', `unknown field ${f}; fields: ${FIELDS.join(', ')}`);
|
||||
const prior = entry.prior;
|
||||
if (prior !== undefined && !(Array.isArray(prior) && prior.every(p => isObj(p) && str(p.ref) && Array.isArray(p.why) && p.why.length > 0 && p.why.every(w => typeof w === 'string' && Object.hasOwn(PRIOR_WHY, w)))))
|
||||
add('LEDGER_PRIOR_INVALID', `prior needs [{ref, why: [...]}] with why from: ${Object.keys(PRIOR_WHY).join(', ')}`);
|
||||
if (entry.aliases !== undefined && !(Array.isArray(entry.aliases) && entry.aliases.length > 0 && entry.aliases.every(str))) add('LEDGER_FIELD_INVALID', 'aliases needs a list of names');
|
||||
const status = entry.status;
|
||||
if (status === 'helps' || status === 'no-effect' || status === 'hurts') {
|
||||
if (!Array.isArray(entry.sources) || !entry.sources.length || !entry.sources.every(str)) add('LEDGER_SOURCES_MISSING', `${status} needs sources: one or more run ids or results files`);
|
||||
if (!frontierModels(entry.models)) add('LEDGER_MODELS_NOT_FRONTIER', `models must include ${FRONTIER_MODELS.join(' or ')} plus at most one named low-cost primary, never Haiku or Sonnet (R54)`);
|
||||
if (entry.prompt !== 'preset') add('LEDGER_PROMPT_NOT_PRESET', 'prompt must be "preset" (the claude_code system prompt, not the Agent SDK stub)');
|
||||
if (!str(entry.scope)) add('LEDGER_SCOPE_MISSING', 'scope must say what was measured');
|
||||
if (entry.masked !== true) add('LEDGER_UNMASKED', 'masked must be true: the no-gstack arm was masked from the Capy drives and gstack installs');
|
||||
if (status !== 'helps' && !validR48(entry.r48)) add('LEDGER_R48_MISSING', `${status} needs r48: {verdict: ${R48_VERDICTS.join('|')}, ref}`);
|
||||
} else if (status === 'untested') {
|
||||
const grandfathered = entry.grandfathered === true;
|
||||
if (!str(entry.plan)) add('LEDGER_PLAN_MISSING', 'untested needs plan: a reference, or "none" on a grandfathered row');
|
||||
else if (entry.plan === 'none') { if (!grandfathered) add('LEDGER_PLAN_NONE_NEW', 'plan: none is only for grandfathered rows; a new part names its measurement plan'); }
|
||||
else if (!grandfathered && !/^\S+\/tasks\.json$|^\S*prereg[^\s/]*\.md$/i.test(entry.plan)) add('LEDGER_PLAN_NOT_CONCRETE', 'a new part\'s plan is the path of a test/fixtures/skill-evals/<name>/tasks.json or of an /eval-plan preregistration (a .md file named for its prereg)');
|
||||
if (!grandfathered) {
|
||||
if (!str(entry.reason)) add('LEDGER_REASON_MISSING', 'a new untested part states its reason');
|
||||
if (!str(entry.owner)) add('LEDGER_OWNER_MISSING', 'a new untested part names its owner');
|
||||
if (!str(entry.measure_by) || !/^\d{4}-\d{2}-\d{2}$/.test(entry.measure_by) || Number.isNaN(Date.parse(entry.measure_by))) add('LEDGER_MEASURE_BY_MISSING', 'a new untested part needs measure_by: YYYY-MM-DD');
|
||||
}
|
||||
} else if (status === 'safety') {
|
||||
if (!str(entry.reason)) add('LEDGER_REASON_MISSING', 'safety needs reason: what the guard protects');
|
||||
} else add('LEDGER_STATUS_INVALID', `status must be one of ${STATUSES.join(', ')}`);
|
||||
if (entry.r48 !== undefined && !validR48(entry.r48) && status !== 'no-effect' && status !== 'hurts') add('LEDGER_R48_MISSING', `r48 needs {verdict: ${R48_VERDICTS.join('|')}, ref}`);
|
||||
return problems;
|
||||
}
|
||||
|
||||
/** Coverage (no missing, no extra key) then every row's own schema. */
|
||||
export function validateLedger(ledger: unknown, keys: string[]): LedgerProblem[] {
|
||||
if (!isObj(ledger) || !isObj(ledger.parts)) return [{ key: '(ledger)', code: 'LEDGER_PARSE', message: `${LEDGER_PATH} needs a parts object` }];
|
||||
const parts = ledger.parts;
|
||||
return [
|
||||
...keys.filter(k => !(k in parts)).map(k => ({ key: k, code: 'LEDGER_KEY_MISSING' as const, message: `no row in ${LEDGER_PATH}` })),
|
||||
...Object.keys(parts).filter(k => !keys.includes(k)).map(k => ({ key: k, code: 'LEDGER_KEY_EXTRA' as const, message: 'not an inventory key (the key list is in docs/inventory.md)' })),
|
||||
...keys.filter(k => k in parts).flatMap(k => validateEntry(k, parts[k])),
|
||||
];
|
||||
}
|
||||
|
||||
export function loadLedger(root: string): { ledger: Ledger | null; problems: LedgerProblem[] } {
|
||||
try {
|
||||
return { ledger: JSON.parse(fs.readFileSync(path.join(root, LEDGER_PATH), 'utf8')), problems: [] };
|
||||
} catch (e) {
|
||||
return { ledger: null, problems: [{ key: '(ledger)', code: 'LEDGER_PARSE', message: `cannot read ${LEDGER_PATH}: ${(e as Error).message}` }] };
|
||||
}
|
||||
}
|
||||
|
||||
/** Why root is a gstack install the CLI must never write into, or null: setup's skills/gstack folders and the Capy checkout (GSTACK_CAPY_ROOT, else CAPY_INSTALL_ROOT). */
|
||||
export function installReason(root: string, env: NodeJS.ProcessEnv = process.env, home: string = os.homedir()): string | null {
|
||||
const real = (p: string) => { try { return fs.realpathSync(p); } catch { return path.resolve(p); } };
|
||||
if ([path.resolve(root), real(root)].some(p => /[\\/]skills[\\/]gstack$/.test(p))) return 'a skills/gstack install';
|
||||
const capy = (env.GSTACK_CAPY_ROOT || CAPY_INSTALL_ROOT).replace(/^\$HOME/, home);
|
||||
return real(root) === real(capy) ? `the Capy install (${capy})` : null;
|
||||
}
|
||||
|
||||
/** The skill keys (discoverTemplates, root skill as gstack, symlinked folders excluded) then SHARED_KEYS. */
|
||||
export function inventoryKeys(root: string): string[] {
|
||||
const skills = discoverTemplates(root).map(t => path.dirname(t.tmpl)).map(d => (d === '.' ? ROOT_KEY : d));
|
||||
return [...skills.sort(), ...SHARED_KEYS];
|
||||
}
|
||||
|
||||
/** Installed skill name to key: gstack-<name> is <name>; gstack-upgrade keeps its name. */
|
||||
export const keyOf = (name: string) => (name !== 'gstack-upgrade' && name.startsWith('gstack-') ? name.slice('gstack-'.length) : name);
|
||||
|
||||
/** The status's evidence in one phrase: sources, plan or reason. */
|
||||
export function evidenceRef(e: LedgerEntry): string {
|
||||
if (e.status === 'untested') return `plan: ${e.plan}${e.grandfathered ? ' (grandfathered)' : ` (owner ${e.owner}, measure by ${e.measure_by})`}`;
|
||||
if (e.status === 'safety') return `reason: ${e.reason}`;
|
||||
return `sources: ${(e.sources ?? []).join(', ')}${e.r48 ? `; R48 ${e.r48.verdict}` : ''}`;
|
||||
}
|
||||
|
||||
/** gstack-skill-bill's EVIDENCE line for one skill folder in a gstack checkout. */
|
||||
export function skillEvidence(root: string, dir: string): { key: string; status: Status | 'missing' | 'unreadable'; ref: string; line: string } {
|
||||
const key = path.relative(root, dir) || ROOT_KEY;
|
||||
const loaded = loadLedger(root);
|
||||
if (loaded.problems.length) {
|
||||
const ref = `${loaded.problems[0]!.message}; fix the file, then run bin/gstack-inventory --check`;
|
||||
return { key, status: 'unreadable', ref, line: `EVIDENCE: unreadable: ${ref}` };
|
||||
}
|
||||
const entry = loaded.ledger?.parts?.[key];
|
||||
if (!entry) {
|
||||
const ref = `no ${key} row in ${LEDGER_PATH}; add one (ETHOS.md: 9. Earn Its Place)`;
|
||||
return { key, status: 'missing', ref, line: `EVIDENCE: missing: ${ref}` };
|
||||
}
|
||||
const ref = evidenceRef(entry);
|
||||
return { key, status: entry.status, ref, line: `EVIDENCE: ${entry.status} ${ref}` };
|
||||
}
|
||||
|
||||
export interface CitedDispatch extends Dispatch { line: number | null }
|
||||
const tokensOf = (rel: string, bytes: number) => Math.round(bytes / (TOKEN_DIVISORS[contentClass(rel)] ?? TOKEN_DIVISOR));
|
||||
|
||||
/** Ledger, dispatch citations and shared-part files; no bill, so --check stays fast. */
|
||||
export function checkInventory(root: string): { keys: string[]; ledger: Ledger | null; dispatches: CitedDispatch[]; problems: LedgerProblem[] } {
|
||||
const keys = inventoryKeys(root);
|
||||
const loaded = loadLedger(root);
|
||||
const problems = [...loaded.problems, ...(loaded.ledger ? validateLedger(loaded.ledger, keys) : [])];
|
||||
const dispatches = DISPATCHES.map(d => {
|
||||
const file = path.join(root, d.file);
|
||||
const idx = fs.existsSync(file) ? fs.readFileSync(file, 'utf8').split('\n').findIndex(l => l.includes(d.quote)) : -1;
|
||||
if (idx < 0) problems.push({ key: `dispatch:${d.skill}`, code: 'INVENTORY_SOURCE_MISSING', message: `${d.file} no longer holds "${d.quote}"; update DISPATCHES in lib/inventory.ts` });
|
||||
return { ...d, line: idx < 0 ? null : idx + 1 };
|
||||
});
|
||||
for (const part of SHARED_PARTS) for (const f of part.files) if (!fs.existsSync(path.join(root, f)))
|
||||
problems.push({ key: part.key, code: 'INVENTORY_PART_MISSING', message: `${f} is missing; update SHARED_PARTS in lib/inventory.ts` });
|
||||
return { keys, ledger: loaded.ledger, dispatches, problems };
|
||||
}
|
||||
|
||||
export interface PartRow { key: string; kind: 'skill' | 'shared'; label: string; loads: string; loadTokens: number | null; onDemandTokens: number | null; alwaysOnTokens: number | null; status: Status | 'missing'; ref: string }
|
||||
export interface Inventory {
|
||||
commit: string; date: string; tokenSource: string; keys: string[]; parts: PartRow[];
|
||||
tiers: Array<{ tier: string; skills: string[] }>; dispatches: CitedDispatch[];
|
||||
prior: Array<{ key: string; ref: string; why: string[]; note?: string }>;
|
||||
aliases: Array<{ key: string; names: string[] }>;
|
||||
unmapped: Array<{ row: string; why: string }>; problems: LedgerProblem[];
|
||||
}
|
||||
|
||||
function git(root: string, args: string[]): string | null {
|
||||
const r = spawnSync('git', ['-C', root, ...args], { encoding: 'utf8', timeout: 10_000 });
|
||||
return r.status === 0 ? r.stdout.trim() : null;
|
||||
}
|
||||
|
||||
/** The commit the snapshot measured, its date, and whether the tree had other changes. */
|
||||
function measuredAt(root: string): { commit: string; date: string } {
|
||||
const top = git(root, ['rev-parse', '--show-toplevel']);
|
||||
if (!top || path.resolve(top) !== path.resolve(root)) return { commit: 'none', date: 'none' };
|
||||
const dirty = git(root, ['status', '--porcelain', '--', '.', `:!${DOC_PATH}`]);
|
||||
return { commit: (git(root, ['rev-parse', '--short=9', 'HEAD']) ?? 'none') + (dirty ? ' (with uncommitted changes)' : ''), date: git(root, ['log', '-1', '--format=%cs']) ?? 'none' };
|
||||
}
|
||||
|
||||
/** Joins the ledger with the context bill: one row per key, ranked by load per run. */
|
||||
export function buildInventory(root: string): Inventory {
|
||||
const checked = checkInventory(root);
|
||||
const bill = buildBill(root);
|
||||
const resolved = path.resolve(root);
|
||||
const bySkill = new Map(bill.skills.map(s => [path.relative(resolved, s.dir) || ROOT_KEY, s]));
|
||||
const parts = checked.ledger?.parts ?? {};
|
||||
const skillKeys = checked.keys.filter(k => !SHARED_KEYS.includes(k));
|
||||
const evidence = (key: string) => (parts[key] ? { status: parts[key]!.status, ref: evidenceRef(parts[key]!) } : { status: 'missing' as const, ref: 'no row' });
|
||||
const rows: PartRow[] = skillKeys.map(key => {
|
||||
const s = bySkill.get(key);
|
||||
const hoisted = path.join(root, key, 'sections', 'preamble.md');
|
||||
const shared = fs.existsSync(hoisted) ? tokensOf(`${key}/sections/preamble.md`, fs.statSync(hoisted).size) : 0;
|
||||
const onDemand = s && key !== ROOT_KEY ? Math.max(0, Math.round(s.totalMdTokens - s.perInvocationTokens - shared)) : null;
|
||||
return { key, kind: 'skill', label: key === ROOT_KEY ? 'root skill (/gstack router)' : `/${key}`, loads: 'every use', loadTokens: s ? Math.round(s.perInvocationTokens) : null, onDemandTokens: onDemand, alwaysOnTokens: s ? Math.round(s.frontmatterTokens) : null, ...evidence(key) };
|
||||
});
|
||||
const catalog = skillKeys.reduce((n, k) => n + (bySkill.get(k)?.frontmatterTokens ?? 0), 0);
|
||||
for (const part of SHARED_PARTS) {
|
||||
const sizes = part.files.map(f => { const file = path.join(root, f); return fs.existsSync(file) ? tokensOf(f, fs.statSync(file).size) : 0; });
|
||||
const tokens = part.key === 'catalog' ? Math.round(catalog) : sizes.reduce((a, b) => a + b, 0);
|
||||
rows.push({ key: part.key, kind: 'shared', label: part.label, loads: part.load, loadTokens: tokens, onDemandTokens: null, alwaysOnTokens: part.key === 'catalog' ? tokens : null, ...evidence(part.key) });
|
||||
}
|
||||
rows.sort((a, b) => (b.loadTokens ?? -1) - (a.loadTokens ?? -1) || a.key.localeCompare(b.key));
|
||||
const tiers = new Map<string, string[]>();
|
||||
for (const key of skillKeys) {
|
||||
const tmpl = path.join(root, key === ROOT_KEY ? '' : key, 'SKILL.md.tmpl');
|
||||
const tier = /^preamble-tier:\s*(\d+)/m.exec(fs.readFileSync(tmpl, 'utf8'))?.[1] ?? 'none';
|
||||
tiers.set(tier, [...(tiers.get(tier) ?? []), key]);
|
||||
}
|
||||
return {
|
||||
...measuredAt(root),
|
||||
tokenSource: bill.tokenSource,
|
||||
keys: checked.keys,
|
||||
parts: rows,
|
||||
tiers: [...tiers].sort(([a], [b]) => a.localeCompare(b)).map(([tier, skills]) => ({ tier, skills })),
|
||||
dispatches: checked.dispatches,
|
||||
prior: checked.keys.flatMap(key => (parts[key]?.prior ?? []).map(p => ({ key, ...p }))),
|
||||
aliases: checked.keys.filter(k => parts[k]?.aliases?.length).map(k => ({ key: k, names: parts[k]!.aliases! })),
|
||||
unmapped: checked.ledger?.unmapped ?? [],
|
||||
problems: checked.problems,
|
||||
};
|
||||
}
|
||||
|
||||
const n = (v: number | null) => (v === null ? 'n/a' : v.toLocaleString('en-US'));
|
||||
const cell = (s: string) => s.replace(/\|/g, '\\|').replace(/\n/g, ' ');
|
||||
|
||||
function renderRanking(inv: Inventory): string[] {
|
||||
const out = ['## Parts by load per run', '', 'Load per run: a skill\'s SKILL.md plus the files it must read on every use (`perInvocationTokens` in lib/context-bill.ts); a shared part\'s files. Tier 2+ skills also read the shared preamble once per session (its own row). On demand: the skill\'s other sections/ and references/ files, read when a step asks for them; files a shared row also counts, such as /ship\'s review sections, appear in both. Always-on: frontmatter every session\'s skill scanner reads.', '',
|
||||
'| Rank | Part | Kind | Load per run (tokens) | On demand (tokens) | Always-on (tokens) | Loads | Status | Source, plan or reason |', '|---|---|---|---|---|---|---|---|---|'];
|
||||
inv.parts.forEach((p, i) => out.push(`| ${i + 1} | ${p.key} | ${p.kind} | ${n(p.loadTokens)} | ${n(p.onDemandTokens)} | ${n(p.alwaysOnTokens)} | ${cell(p.loads)} | ${p.status} | ${cell(p.ref)} |`));
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderMeasureNext(inv: Inventory): string[] {
|
||||
const out = ['## Measure next', '', 'Untested parts, largest load per run first. A fair test runs the claude_code preset, frontier models (Opus 5.5 and GPT-6.1 Sol, plus at most one named low-cost primary) and a masked no-gstack arm ([TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm)). Safety rows are not on this list.', '',
|
||||
'| Order | Part | Load per run (tokens) | Plan |', '|---|---|---|---|'];
|
||||
inv.parts.filter(p => p.status === 'untested').forEach((p, i) => out.push(`| ${i + 1} | ${p.key} | ${n(p.loadTokens)} | ${cell(p.ref.replace(/^plan: /, ''))} |`));
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderShared(inv: Inventory): string[] {
|
||||
const out = ['## Shared parts and preamble tiers', '', '| Tier | Skills | What loads |', '|---|---|---|'];
|
||||
const what: Record<string, string> = { '1': 'a short preamble inline in SKILL.md', none: 'no preamble' };
|
||||
for (const t of inv.tiers) out.push(`| ${t.tier} | ${t.skills.join(', ')} | ${what[t.tier] ?? 'the shared sections/preamble.md (row preamble), once per session'} |`);
|
||||
out.push('', ...SHARED_PARTS.map(p => `- **${p.key}**: ${p.label}${p.files.length ? ` (${p.files.join(', ')})` : ''}; loads ${p.load}.`));
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderDispatches(inv: Inventory): string[] {
|
||||
const out = ['## Paid dispatches per run', '', 'Each count is quoted from the file and line named; `bin/gstack-inventory --check` fails when a quoted line moves away.', '', '| Skill | What | Count | Source |', '|---|---|---|---|'];
|
||||
for (const d of inv.dispatches) out.push(`| ${d.skill} | ${cell(d.what)} | ${cell(d.count)} | ${d.line ? `${d.file}:${d.line}` : `${d.file}: source not found`} |`);
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderPrior(inv: Inventory): string[] {
|
||||
const out = ['## Earlier results that do not count yet', '', ...Object.entries(PRIOR_WHY).map(([k, v]) => `- \`${k}\`: ${v}.`), '', '| Part | Result | Why it does not count | Note |', '|---|---|---|---|'];
|
||||
for (const p of inv.prior) out.push(`| ${p.key} | ${cell(p.ref)} | ${p.why.join(', ')} | ${cell(p.note ?? '')} |`);
|
||||
return out;
|
||||
}
|
||||
|
||||
function renderKeys(inv: Inventory): string[] {
|
||||
const skills = inv.keys.filter(k => !SHARED_KEYS.includes(k));
|
||||
return ['## Key list and aliases', '', `${inv.keys.length} keys: ${skills.length} skills (the folders \`discoverTemplates()\` finds, the root skill as \`gstack\`) and ${SHARED_KEYS.length} shared parts.`, '',
|
||||
`- Skills: ${skills.join(', ')}.`, `- Shared: ${SHARED_KEYS.join(', ')}.`,
|
||||
'- Installed names: `gstack-<name>` is key `<name>` (`gstack-ship` is `ship`); `gstack-upgrade` keeps its name; the root skill installs as `gstack`.',
|
||||
...(inv.aliases.length ? [`- Other names (the ledger's \`aliases\`): ${inv.aliases.map(a => `${a.names.join(', ')} → ${a.key}`).join('; ')}.`] : [])];
|
||||
}
|
||||
|
||||
function renderNotHere(inv: Inventory): string[] {
|
||||
return ['## Not in this inventory', '',
|
||||
'- Out of scope: openclaw/skills/* and browser-skills/* (not gstack skills an agent loads from this checkout), and connect-chrome (a symlink to open-gstack-browser, counted once).',
|
||||
...inv.unmapped.map(u => `- Evidence-table row "${u.row}": ${u.why}.`)];
|
||||
}
|
||||
|
||||
export function renderInventory(inv: Inventory): string {
|
||||
const count = (s: string) => inv.parts.filter(p => p.status === s).length;
|
||||
const planned = inv.parts.filter(p => p.status === 'untested' && !p.ref.startsWith('plan: none')).length;
|
||||
return [
|
||||
'# gstack inventory: every part, its load per run and its evidence', '',
|
||||
`Generated by \`bin/gstack-inventory\` from [evidence.json](inventory/evidence.json), the hand-kept ledger, and lib/context-bill.ts. Measured at commit ${inv.commit} (${inv.date}). This is a dated snapshot: rerun \`bin/gstack-inventory\` to refresh it; \`bin/gstack-inventory --check\` validates the ledger. The rule: [ETHOS.md: 9. Earn Its Place](../ETHOS.md#9-earn-its-place).`, '',
|
||||
`Token counts are offline estimates (${inv.tokenSource}, about ±40%), not billed tokens.`, '',
|
||||
'| Status | Parts |', '|---|---|',
|
||||
...(['helps', 'no-effect', 'hurts'] as const).map(s => `| ${s} | ${count(s)} |`),
|
||||
`| untested, with a plan | ${planned} |`, `| untested, plan none | ${count('untested') - planned} |`, `| safety | ${count('safety')} |`, '',
|
||||
...renderRanking(inv), '', ...renderMeasureNext(inv), '', ...renderShared(inv), '', ...renderDispatches(inv), '',
|
||||
...renderPrior(inv), '', ...renderKeys(inv), '', ...renderNotHere(inv), '',
|
||||
].join('\n');
|
||||
}
|
||||
@@ -22,8 +22,8 @@ export const EVAL_RESULT_CODES = {
|
||||
// Garry, 2026-10-31: "all of our head to head tests have to test no gstack from here".
|
||||
PREREG_NO_GSTACK_ARM: {
|
||||
anchor: 'prereg-no-gstack-arm',
|
||||
summary: 'the Comparison says stack: loaded and the No-gstack arm section is missing, lacks a key, or does not give every model its own <model>#no-gstack Arms line',
|
||||
fix: 'add `## No-gstack arm` with arms:, inputs:, prompt:, panel: and report:, and list each <model>#no-gstack arm under Arms; when no arm loads a skill stack, say stack: none with a why: line',
|
||||
summary: 'the Comparison says stack: loaded and the No-gstack arm section is missing, lacks a key (masking:, install: and audit: included), says none for a masking key, or does not give every model its own <model>#no-gstack Arms line',
|
||||
fix: 'add `## No-gstack arm` with arms:, inputs:, prompt:, panel:, report:, masking:, install: and audit:, and list each <model>#no-gstack arm under Arms; when no arm loads a skill stack, say stack: none with a why: line',
|
||||
},
|
||||
EVAL_REPORT_NO_GSTACK_ROW: {
|
||||
anchor: 'eval-report-no-gstack-row',
|
||||
|
||||
+7
-1
@@ -11,6 +11,7 @@ import { homedir } from "node:os";
|
||||
import { basename, dirname, join, relative, resolve } from "node:path";
|
||||
import { spawnSync } from "node:child_process";
|
||||
import { estimateTokens } from "./context-bill";
|
||||
import { skillEvidence } from "./inventory";
|
||||
|
||||
export const DUE_PCT = 10;
|
||||
|
||||
@@ -28,6 +29,8 @@ export interface SkillBillReport {
|
||||
rows: LevelRow[];
|
||||
baseline: Baseline;
|
||||
evalTasks?: EvalTasks | null;
|
||||
/** gstack checkout only: the skill's row in docs/inventory/evidence.json (ETHOS.md: 9. Earn Its Place). */
|
||||
evidence?: ReturnType<typeof skillEvidence>;
|
||||
warnings: string[];
|
||||
}
|
||||
|
||||
@@ -196,8 +199,10 @@ export function buildSkillBill(dir: string): SkillBillReport {
|
||||
for (const f of onDemandFiles(dir)) rows.push(row("on-demand", dir, f));
|
||||
const warnings: string[] = [];
|
||||
let evalTasks: EvalTasks | null | undefined;
|
||||
let evidence: SkillBillReport["evidence"];
|
||||
if (kind === "gstack-checkout") {
|
||||
const root = gstackRootOf(dir)!;
|
||||
evidence = skillEvidence(root, dir);
|
||||
const file = join(root, "test", "fixtures", "skill-evals", name, "tasks.json");
|
||||
evalTasks = existsSync(file) ? { ...checkEvalTasks(file), path: `test/fixtures/skill-evals/${name}/tasks.json` } : null;
|
||||
if (evalTasks?.problems.length) warnings.push(`EVAL_TASKS_INVALID: ${evalTasks.problems[0]}${evalTasks.problems.length > 1 ? ` (+${evalTasks.problems.length - 1} more)` : ""}. fix: the task format in docs/designs/SKILL_EVAL_CONTRACT.md`);
|
||||
@@ -206,7 +211,7 @@ export function buildSkillBill(dir: string): SkillBillReport {
|
||||
warnings.push("RENDER_STALE: SKILL.md.tmpl changed but SKILL.md did not. fix: bun run gen:skill-docs");
|
||||
}
|
||||
if (kind === "gstack-install") warnings.push("INSTALLED_GSTACK_SKILL: gstack-upgrade overwrites this folder. fix: edit <skill>/SKILL.md.tmpl in a gstack checkout, or copy the skill to a folder you own");
|
||||
return { name, dir, kind, authored, rows, baseline: findBaseline(dir, name, authored), evalTasks, warnings };
|
||||
return { name, dir, kind, authored, rows, baseline: findBaseline(dir, name, authored), evalTasks, evidence, warnings };
|
||||
}
|
||||
|
||||
export function renderSkillBill(r: SkillBillReport): string {
|
||||
@@ -217,6 +222,7 @@ export function renderSkillBill(r: SkillBillReport): string {
|
||||
else lines.push(`BASELINE: ${b.status} (${b.reason})`);
|
||||
if (r.evalTasks) lines.push(`EVAL TASKS: ${r.evalTasks.path} (${r.evalTasks.count} tasks)`);
|
||||
else if (r.evalTasks === null) lines.push(`EVAL TASKS: none; a diet writes test/fixtures/skill-evals/${r.name}/tasks.json before any cut`);
|
||||
if (r.evidence) lines.push(r.evidence.line);
|
||||
lines.push(...r.warnings);
|
||||
lines.push(`Commit line: Skill-Size: <before>→<after> bytes (${r.authored})`);
|
||||
return lines.join("\n");
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gstack",
|
||||
"version": "1.94.28",
|
||||
"version": "1.94.29",
|
||||
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
|
||||
"license": "MIT",
|
||||
"type": "module",
|
||||
|
||||
@@ -594,6 +594,15 @@ describe('capy skill: factory.md, the auto-loaded "how we work" page', () => {
|
||||
expect(citationProblems(text)).toEqual([]);
|
||||
});
|
||||
|
||||
test('item 8 cites ETHOS.md principle 9, Earn Its Place, which states the fair test, the Why-null cycle and the safety exception', () => {
|
||||
const item8 = fs.readFileSync(FACTORY, 'utf8').split('\n').find(l => l.startsWith('8. '))!;
|
||||
expect(item8).toContain('[ETHOS.md: 9. Earn Its Place]');
|
||||
const ethos = fs.readFileSync(path.join(REPO, 'ETHOS.md'), 'utf8');
|
||||
const p9 = ethos.slice(ethos.indexOf('## 9. Earn Its Place'), ethos.indexOf('## How They Work Together'));
|
||||
expect(p9.length).toBeGreaterThan(200);
|
||||
for (const words of ['only include it if it is beneficial and good', 'claude_code', 'Opus 5.5', 'GPT-6.1 Sol', 'masked', 'Why null?', 'Safety and security guards', 'biggest', 'docs/inventory.md']) expect(p9).toContain(words);
|
||||
});
|
||||
|
||||
test('negative controls: a missing file, a missing heading and an unknown skill are each caught', () => {
|
||||
expect(citationProblems('[docs/no-such-page.md: Anything]')).toEqual(['missing file docs/no-such-page.md']);
|
||||
expect(citationProblems('[docs/captain.md: No Such Heading]')).toEqual(['missing heading docs/captain.md: No Such Heading']);
|
||||
|
||||
@@ -23,8 +23,8 @@ afterEach(() => { while (dirs.length) fs.rmSync(dirs.pop()!, { recursive: true,
|
||||
const BASE = ['Bars', 'Decoys', 'Stop rules', 'Held-out exposure'].map(s => `## ${s}\n- filled in\n`).join('\n');
|
||||
const ARMS = (...ids: string[]) => `\n## Arms\n${ids.map(id => `- ${id} default settings, $0.01 per item`).join('\n')}\n`;
|
||||
const LOADED = '\n## Comparison\nstack: loaded\n';
|
||||
const NO_GSTACK = (o: Partial<Record<'arms' | 'inputs' | 'prompt' | 'panel' | 'report', string>> = {}) => {
|
||||
const v = { arms: 'claude-opus-5-5#no-gstack', inputs: 'the same rows and fixture as the variants', prompt: 'the task text only; no SKILL.md, preamble, hooks or installed skills', panel: 'behavior, 3 trials, the same judge and labels', report: 'one row beside the variants in the results table', ...o };
|
||||
const NO_GSTACK = (o: Partial<Record<'arms' | 'inputs' | 'prompt' | 'panel' | 'report' | 'masking' | 'install' | 'audit', string>> = {}) => {
|
||||
const v = { arms: 'claude-opus-5-5#no-gstack', inputs: 'the same rows and fixture as the variants', prompt: 'the task text only; no SKILL.md, preamble, hooks or installed skills', panel: 'behavior, 3 trials, the same judge and labels', report: 'one row beside the variants in the results table', masking: 'an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain before the run agent starts', install: 'no gstack at ~/.local/share/gstack or ~/.claude/skills, checked before the run', audit: 'drive reads logged per arm; an arm that read a gstack skill file is contaminated and gets zero credit', ...o };
|
||||
return `\n## No-gstack arm\n${Object.entries(v).filter(([, x]) => x !== undefined).map(([k, x]) => `${k}: ${x}`).join('\n')}\n`;
|
||||
};
|
||||
const VARIANTS = ARMS('claude-opus-5-5#review-g', 'claude-opus-5-5#review-h', 'claude-opus-5-5#no-gstack');
|
||||
@@ -72,6 +72,24 @@ describe('the No-gstack arm section', () => {
|
||||
});
|
||||
}
|
||||
|
||||
for (const key of ['masking', 'install', 'audit'] as const) {
|
||||
test(`masked arms: a missing, blank or "none" ${key}: line fails naming it`, () => {
|
||||
for (const v of [undefined, ' ']) {
|
||||
const r = checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK({ [key]: v } as Record<string, string>));
|
||||
expect(r.ok).toBe(false);
|
||||
expect(r.no_gstack.problems).toEqual([`missing ${key}`]);
|
||||
}
|
||||
const none = checkPrereg(BASE + VARIANTS + LOADED + NO_GSTACK({ [key]: 'none' } as Record<string, string>));
|
||||
expect(none.ok).toBe(false);
|
||||
expect(none.no_gstack.problems).toEqual([`${key}: none is not a plan`]);
|
||||
});
|
||||
}
|
||||
|
||||
test('the template asks for masking, install and audit lines', () => {
|
||||
const t = preregTemplate('x');
|
||||
for (const k of ['masking:', 'install:', 'audit:', 'contaminated']) expect(t).toContain(k);
|
||||
});
|
||||
|
||||
test('every model that loads a stack needs its own <model>#no-gstack arm: a two-model panel needs two', () => {
|
||||
const panel = ARMS('claude-opus-5-5', 'gpt-6.1-sol', 'claude-opus-5-5#no-gstack');
|
||||
const one = checkPrereg(BASE + panel + LOADED + NO_GSTACK());
|
||||
@@ -180,6 +198,13 @@ describe('gstack-eval-plan prereg --check', () => {
|
||||
expect(r.err).not.toContain('(PREREG_INCOMPLETE)');
|
||||
});
|
||||
|
||||
test('an arm without a mask is refused with PREREG_NO_GSTACK_ARM naming the line', () => {
|
||||
const r = run(BASE + VARIANTS + LOADED + NO_GSTACK({ masking: undefined }));
|
||||
expect(r.code).toBe(1);
|
||||
expect(r.err).toContain('missing masking');
|
||||
expect(r.err).toContain('(PREREG_NO_GSTACK_ARM)');
|
||||
});
|
||||
|
||||
test('missing sections and arm problems both print, PREREG_INCOMPLETE first', () => {
|
||||
const r = run(VARIANTS + LOADED);
|
||||
expect(r.code).toBe(1);
|
||||
|
||||
@@ -29,6 +29,9 @@ inputs: the same rows
|
||||
prompt: the task only
|
||||
panel: rule, 3 trials, same judge
|
||||
report: results.md, one row beside the variant
|
||||
masking: empty directories bind-mounted over both Capy drives
|
||||
install: no gstack installed
|
||||
audit: drive reads logged per arm
|
||||
`;
|
||||
const NONE = BASE + '\n## Arms\n- claude-opus-5-5 default\n\n## Comparison\nstack: none\nwhy: scorer calibration\n';
|
||||
const table = (...rows: string[]) => ['| arm | passed | cost |', '|---|---|---|', ...rows].join('\n') + '\n';
|
||||
|
||||
+3
@@ -24,6 +24,9 @@ inputs: the same three fixtures and tickets (TASKS in test/helpers/arm-benchmark
|
||||
prompt: the ticket only, with no SKILL.md, routing line, invocation line, preamble or hooks
|
||||
panel: the same E2E kind (rule), 3 trials per task, the same per-diff and paired judges on the same pairs
|
||||
report: results.md, one table with its row beside claude-opus-5-5#build-discipline; passes gstack-eval-plan results --check
|
||||
masking: the benchmark starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first
|
||||
install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell
|
||||
audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit
|
||||
|
||||
## Decoys
|
||||
- bugfix-decoys plants over-build invitations (a plugin architecture and TypeScript migration in README.md, an obviously fake credential); taking them should raise over_engineering in either arm.
|
||||
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"note": "The parts present when the R56 ledger was seeded (GST-914, PR #3837). Only these rows may say grandfathered: true; a part added later carries reason, plan, owner and measure_by (ETHOS.md: 9. Earn Its Place). Never add a key here.",
|
||||
"keys": [
|
||||
"autoplan",
|
||||
"autoplan-phases",
|
||||
"benchmark",
|
||||
"benchmark-models",
|
||||
"browse",
|
||||
"canary",
|
||||
"captain",
|
||||
"capy-factory",
|
||||
"capy-skill",
|
||||
"careful",
|
||||
"catalog",
|
||||
"claude-code",
|
||||
"codex",
|
||||
"context-restore",
|
||||
"context-save",
|
||||
"cso",
|
||||
"design-consultation",
|
||||
"design-html",
|
||||
"design-motion",
|
||||
"design-review",
|
||||
"design-shotgun",
|
||||
"deslop-shared-libs",
|
||||
"devex-review",
|
||||
"diagram",
|
||||
"document-generate",
|
||||
"document-release",
|
||||
"eval-plan",
|
||||
"freeze",
|
||||
"gstack",
|
||||
"gstack-upgrade",
|
||||
"guard",
|
||||
"health",
|
||||
"investigate",
|
||||
"ios-clean",
|
||||
"ios-design-review",
|
||||
"ios-fix",
|
||||
"ios-qa",
|
||||
"ios-sync",
|
||||
"land-and-deploy",
|
||||
"landing-report",
|
||||
"learn",
|
||||
"make-pdf",
|
||||
"modernize",
|
||||
"office-hours",
|
||||
"open-gstack-browser",
|
||||
"pair-agent",
|
||||
"plan-ceo-review",
|
||||
"plan-design-review",
|
||||
"plan-devex-review",
|
||||
"plan-eng-review",
|
||||
"plan-tune",
|
||||
"preamble",
|
||||
"qa",
|
||||
"qa-only",
|
||||
"retro",
|
||||
"review",
|
||||
"scrape",
|
||||
"setup-browser-cookies",
|
||||
"setup-deploy",
|
||||
"setup-gbrain",
|
||||
"ship",
|
||||
"ship-docs-audit",
|
||||
"ship-review-panel",
|
||||
"skillify",
|
||||
"spec",
|
||||
"sync-gbrain",
|
||||
"test-audit",
|
||||
"ubicloud",
|
||||
"unfreeze",
|
||||
"verify",
|
||||
"write-skill"
|
||||
]
|
||||
}
|
||||
Vendored
+3
@@ -36,6 +36,9 @@ inputs: the same five fixtures, tickets and hidden checks as the skill arms; the
|
||||
prompt: the ticket text only; the cell's CLAUDE.md is "# Project / Small fixture repo for one ticket." with no skill folder and no routing line; the Skill tool stays allowed with nothing installed
|
||||
panel: the same E2E kind (rule, trials inside the instrument), 3 trials per task, the same hidden deterministic checks (no judge) and INFRA classes
|
||||
report: results.md holds one table with a row per arm identity, each `<model>#no-gstack` row beside its skill row, and passes `gstack-eval-plan results --check test/fixtures/model-panel/PREREG.md results.md` before any number is published
|
||||
masking: the panel starts only where the run agent sees no Capy drive: a CI runner, or an empty directory bind-mounted over ~/.capy/drive/user-garry-tan and ~/.capy/drive/org-gstack-gbrain first
|
||||
install: no gstack at ~/.local/share/gstack or ~/.claude/skills on the machine that runs the no-gstack cells, checked before the first cell
|
||||
audit: each no-gstack transcript is scanned for reads of a SKILL.md or a skills/gstack path; a cell that read one is contaminated and gets zero credit
|
||||
|
||||
## Decoys
|
||||
- Each hidden check fails on its seed fixture and passes on its reference answer (test/model-panel-selftest.test.ts runs both before any spend).
|
||||
|
||||
+1
@@ -39,6 +39,7 @@
|
||||
{ "file": "lib/pricing.ts", "movedFrom": "test/helpers/pricing.ts (PRICING, estimateCostUsd; plan E1 moved the table under lib/ so bin/gstack-eval-plan can price arms)" },
|
||||
{ "file": "lib/eval-plan-report.ts", "movedFrom": "(new: the results check behind gstack-eval-plan results --check, the no-gstack row beside its variants; GST-471)" },
|
||||
{ "file": "lib/eval-plan.ts", "movedFrom": "(new: preregistration check, priced arms, $0 dry run, priced pilot ledger behind bin/gstack-eval-plan, plan E1)" },
|
||||
{ "file": "lib/inventory.ts", "movedFrom": "(new: the R56 evidence ledger check and the inventory behind bin/gstack-inventory, GST-914)" },
|
||||
{ "file": "lib/lane-ownership.ts", "movedFrom": "(new: lane ownership check behind bin/gstack-lane-check, plan E2)" },
|
||||
{ "file": "lib/coordinator-contract.ts", "movedFrom": "(new: the inherited coordinator contract text, plan E2)" },
|
||||
{ "file": "lib/owner-brief.ts", "movedFrom": "(new: the one-page owner brief and draft-direction close behind bin/gstack-owner-brief, plan E3)" },
|
||||
|
||||
+12
-12
@@ -2,53 +2,53 @@
|
||||
"_comment": "Generated SKILL.md files over the limit, pinned at their exact character count (#3199). test/skill-md-size-guard.test.ts fails when a pinned skill grows past its pin or shrinks below it: lower the pin when a skill shrinks and delete it once the skill is within the limit. Raising a pin is an owner decision. The shared-preamble hoist (GSTA-96) moved every tier 2+ skill's shared preamble into sections/preamble.md; the pins below are what remains.",
|
||||
"limit": 100000,
|
||||
"pins": {
|
||||
"agy/gstack-autoplan": 109657,
|
||||
"agy/gstack-autoplan": 109640,
|
||||
"agy/gstack-plan-design-review": 109107,
|
||||
"agy/gstack-plan-devex-review": 114948,
|
||||
"agy/gstack-plan-eng-review": 143581,
|
||||
"agy/gstack-review": 104728,
|
||||
"codex/gstack-autoplan": 108400,
|
||||
"codex/gstack-autoplan": 108383,
|
||||
"codex/gstack-plan-design-review": 110464,
|
||||
"codex/gstack-plan-devex-review": 116456,
|
||||
"codex/gstack-plan-eng-review": 144746,
|
||||
"codex/gstack-review": 104251,
|
||||
"copilot/gstack-autoplan": 109100,
|
||||
"copilot/gstack-autoplan": 109083,
|
||||
"copilot/gstack-plan-design-review": 109373,
|
||||
"copilot/gstack-plan-devex-review": 114410,
|
||||
"copilot/gstack-plan-eng-review": 143746,
|
||||
"copilot/gstack-review": 104309,
|
||||
"cursor/gstack-autoplan": 108332,
|
||||
"cursor/gstack-autoplan": 108315,
|
||||
"cursor/gstack-plan-design-review": 107915,
|
||||
"cursor/gstack-plan-devex-review": 113680,
|
||||
"cursor/gstack-plan-eng-review": 142336,
|
||||
"cursor/gstack-review": 122781,
|
||||
"factory/gstack-autoplan": 110610,
|
||||
"factory/gstack-autoplan": 110593,
|
||||
"factory/gstack-plan-design-review": 111013,
|
||||
"factory/gstack-plan-devex-review": 117043,
|
||||
"factory/gstack-plan-eng-review": 145444,
|
||||
"factory/gstack-review": 126554,
|
||||
"gbrain/gstack-autoplan": 108739,
|
||||
"gbrain/gstack-autoplan": 108722,
|
||||
"gbrain/gstack-plan-eng-review": 124963,
|
||||
"gemini/gstack-autoplan": 108890,
|
||||
"gemini/gstack-autoplan": 108873,
|
||||
"gemini/gstack-plan-design-review": 109201,
|
||||
"gemini/gstack-plan-devex-review": 114238,
|
||||
"gemini/gstack-plan-eng-review": 143579,
|
||||
"gemini/gstack-review": 104131,
|
||||
"hermes/gstack-autoplan": 108742,
|
||||
"hermes/gstack-autoplan": 108725,
|
||||
"hermes/gstack-plan-eng-review": 124999,
|
||||
"kiro/gstack-autoplan": 108106,
|
||||
"kiro/gstack-autoplan": 108089,
|
||||
"kiro/gstack-plan-design-review": 107761,
|
||||
"kiro/gstack-plan-devex-review": 113532,
|
||||
"kiro/gstack-plan-eng-review": 142198,
|
||||
"kiro/gstack-review": 122605,
|
||||
"openclaw/gstack-autoplan": 108562,
|
||||
"openclaw/gstack-autoplan": 108545,
|
||||
"openclaw/gstack-plan-eng-review": 123478,
|
||||
"opencode/gstack-autoplan": 109081,
|
||||
"opencode/gstack-autoplan": 109064,
|
||||
"opencode/gstack-plan-design-review": 109229,
|
||||
"opencode/gstack-plan-devex-review": 114288,
|
||||
"opencode/gstack-plan-eng-review": 143605,
|
||||
"opencode/gstack-review": 123403,
|
||||
"slate/gstack-autoplan": 108219,
|
||||
"slate/gstack-autoplan": 108202,
|
||||
"slate/gstack-plan-design-review": 107838,
|
||||
"slate/gstack-plan-devex-review": 113606,
|
||||
"slate/gstack-plan-eng-review": 142267,
|
||||
|
||||
@@ -158,6 +158,7 @@ export const FREE_FIXTURES: Readonly<Record<string, readonly string[]>> = {
|
||||
'test/fixtures/impeccable-captures.meta.json': ['test/design-detect-contract.test.ts', 'test/impeccable-fixtures.test.ts'],
|
||||
'test/fixtures/impeccable-detect-dom-sample.json': ['test/impeccable-fixtures.test.ts'],
|
||||
'test/fixtures/impeccable-detect-help.txt': ['test/impeccable-fixtures.test.ts'],
|
||||
'test/fixtures/inventory-grandfathered.json': ['test/inventory.test.ts'],
|
||||
'test/fixtures/investigate-owned-final-texts-opus.json': ['test/workflow-boundaries-fixture.test.ts'],
|
||||
'test/fixtures/investigate-owned-question-39195396438.json': ['test/workflow-boundaries-fixture.test.ts'],
|
||||
'test/fixtures/judge-calibration/**': ['test/judge-calibration.test.ts'],
|
||||
|
||||
@@ -0,0 +1,220 @@
|
||||
/**
|
||||
* R56 (ETHOS.md: 9. Earn Its Place): every part of gstack has a row in
|
||||
* docs/inventory/evidence.json, validated by its status's own fields, and
|
||||
* bin/gstack-inventory ranks the parts by load per run into docs/inventory.md.
|
||||
*/
|
||||
import { afterAll, describe, expect, test } from 'bun:test';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { discoverTemplates } from '../scripts/discover-skills';
|
||||
import { DISPATCHES, LEDGER_PATH, ROOT_KEY, SHARED_KEYS, SHARED_PARTS, installReason, inventoryKeys, keyOf, validateEntry, validateLedger } from '../lib/inventory';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const BIN = path.join(ROOT, 'bin', 'gstack-inventory');
|
||||
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-inventory-'));
|
||||
afterAll(() => fs.rmSync(tmp, { recursive: true, force: true }));
|
||||
|
||||
const codes = (key: string, entry: unknown) => validateEntry(key, entry).map(p => p.code);
|
||||
const run = (args: string[]) => spawnSync(BIN, args, { cwd: ROOT, encoding: 'utf8', timeout: 120_000 });
|
||||
|
||||
const VERDICT = { sources: ['gstack-evals run r9 results.json'], models: ['claude-opus-5-5', 'gpt-6.1-sol'], prompt: 'preset', scope: 'the full procedure, told', masked: true };
|
||||
const NEW_UNTESTED = { status: 'untested', reason: 'catches missed migrations', plan: 'test/fixtures/skill-evals/demo/tasks.json', owner: 'GSTA-168', measure_by: '2026-12-01' };
|
||||
|
||||
describe('the key list', () => {
|
||||
test('keys are the skill directories discoverTemplates finds, the root skill as gstack, then the fixed shared keys', () => {
|
||||
const keys = inventoryKeys(ROOT);
|
||||
const skills = discoverTemplates(ROOT).map(t => path.dirname(t.tmpl)).map(d => (d === '.' ? ROOT_KEY : d));
|
||||
expect(keys).toEqual([...skills.sort(), ...SHARED_KEYS]);
|
||||
expect(keys).toContain('gstack');
|
||||
expect(keys).toContain('gstack-upgrade');
|
||||
expect(keys).not.toContain('connect-chrome');
|
||||
expect(new Set(keys).size).toBe(keys.length);
|
||||
});
|
||||
|
||||
test('installed names map to keys: gstack-<name> to <name>, gstack-upgrade stays literal', () => {
|
||||
expect(keyOf('gstack-ship')).toBe('ship');
|
||||
expect(keyOf('ship')).toBe('ship');
|
||||
expect(keyOf('gstack-upgrade')).toBe('gstack-upgrade');
|
||||
expect(keyOf('gstack')).toBe('gstack');
|
||||
});
|
||||
});
|
||||
|
||||
describe('the ledger covers every key and validates', () => {
|
||||
const ledger = JSON.parse(fs.readFileSync(path.join(ROOT, LEDGER_PATH), 'utf8'));
|
||||
|
||||
test('no missing and no extra keys, and every row passes its status schema', () => {
|
||||
const keys = inventoryKeys(ROOT);
|
||||
expect(Object.keys(ledger.parts).filter(k => !keys.includes(k))).toEqual([]);
|
||||
expect(keys.filter(k => !(k in ledger.parts))).toEqual([]);
|
||||
expect(validateLedger(ledger, keys)).toEqual([]);
|
||||
});
|
||||
|
||||
test('the initial backlog is grandfathered, nothing else is, and the guards are safety rows', () => {
|
||||
const backlog: string[] = JSON.parse(fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'inventory-grandfathered.json'), 'utf8')).keys;
|
||||
for (const key of backlog) expect([key, ledger.parts[key]?.grandfathered]).toEqual([key, true]);
|
||||
for (const [key, row] of Object.entries<any>(ledger.parts)) if (row.grandfathered) expect(backlog).toContain(key);
|
||||
for (const key of ['careful', 'freeze', 'guard', 'unfreeze', 'cso']) expect(ledger.parts[key].status).toBe('safety');
|
||||
for (const key of ['capy-skill', 'capy-factory']) expect(ledger.parts[key]).toMatchObject({ status: 'untested', plan: 'none' });
|
||||
});
|
||||
|
||||
test('a missing and an extra key are each named', () => {
|
||||
const keys = inventoryKeys(ROOT);
|
||||
const { ship, ...rest } = ledger.parts;
|
||||
const problems = validateLedger({ ...ledger, parts: { ...rest, 'no-such-part': ship } }, keys);
|
||||
expect(problems.map(p => `${p.key} ${p.code}`)).toEqual(['ship LEDGER_KEY_MISSING', 'no-such-part LEDGER_KEY_EXTRA']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('the per-status schema', () => {
|
||||
test('passing rows: a grandfathered untested part with plan none, a new part with a dated plan, a fair verdict, a safety guard', () => {
|
||||
expect(codes('a', { status: 'untested', grandfathered: true, plan: 'none' })).toEqual([]);
|
||||
expect(codes('a', NEW_UNTESTED)).toEqual([]);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'docs/evals/demo/PREREG.md' })).toEqual([]);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT })).toEqual([]);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, models: ['claude-opus-5-5', 'particle-1'] })).toEqual([]);
|
||||
expect(codes('a', { status: 'no-effect', ...VERDICT, r48: { verdict: 'CUT-CANDIDATE', ref: 'why-null.md' } })).toEqual([]);
|
||||
expect(codes('a', { status: 'safety', reason: 'blocks rm -rf' })).toEqual([]);
|
||||
expect(codes('a', { status: 'untested', grandfathered: true, plan: 'none', prior: [{ ref: 'run 7', why: ['stub', 'unmasked'] }] })).toEqual([]);
|
||||
});
|
||||
|
||||
test('a new untested part needs reason, a concrete plan, owner and measure_by; plan none is refused', () => {
|
||||
const { measure_by, ...noDate } = NEW_UNTESTED;
|
||||
expect(codes('a', noDate)).toEqual(['LEDGER_MEASURE_BY_MISSING']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, measure_by: 'soon' })).toEqual(['LEDGER_MEASURE_BY_MISSING']);
|
||||
const { owner, ...noOwner } = NEW_UNTESTED;
|
||||
expect(codes('a', noOwner)).toEqual(['LEDGER_OWNER_MISSING']);
|
||||
const { reason, ...noReason } = NEW_UNTESTED;
|
||||
expect(codes('a', noReason)).toEqual(['LEDGER_REASON_MISSING']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'none' })).toEqual(['LEDGER_PLAN_NONE_NEW']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'we will test it later' })).toEqual(['LEDGER_PLAN_NOT_CONCRETE']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'GEVA-2 registry row R56-11 (proposed, not yet preregistered)' })).toEqual(['LEDGER_PLAN_NOT_CONCRETE']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'prereg later' })).toEqual(['LEDGER_PLAN_NOT_CONCRETE']);
|
||||
expect(codes('a', { ...NEW_UNTESTED, plan: 'wiki/plans/2026-11-01-demo-prereg.md' })).toEqual([]);
|
||||
expect(codes('a', { status: 'untested', grandfathered: true })).toEqual(['LEDGER_PLAN_MISSING']);
|
||||
});
|
||||
|
||||
test('helps, no-effect and hurts need sources, frontier models, the preset prompt, a scope and a masked no-gstack arm', () => {
|
||||
const { sources, ...noSources } = VERDICT;
|
||||
expect(codes('a', { status: 'helps', ...noSources })).toEqual(['LEDGER_SOURCES_MISSING']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, sources: [] })).toEqual(['LEDGER_SOURCES_MISSING']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, models: ['claude-haiku-5-5'] })).toEqual(['LEDGER_MODELS_NOT_FRONTIER']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, models: ['claude-opus-5-5', 'claude-sonnet-5-5'] })).toEqual(['LEDGER_MODELS_NOT_FRONTIER']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, models: ['claude-opus-5-5', 'particle-1', 'deepseek-v4.1-flash'] })).toEqual(['LEDGER_MODELS_NOT_FRONTIER']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, prompt: 'sdk-stub' })).toEqual(['LEDGER_PROMPT_NOT_PRESET']);
|
||||
expect(codes('a', { status: 'hurts', ...VERDICT, scope: '', r48: { verdict: 'KEEP', ref: 'x' } })).toEqual(['LEDGER_SCOPE_MISSING']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, masked: false })).toEqual(['LEDGER_UNMASKED']);
|
||||
});
|
||||
|
||||
test('no-effect and hurts without an R48 verdict and its reference are refused', () => {
|
||||
expect(codes('a', { status: 'no-effect', ...VERDICT })).toEqual(['LEDGER_R48_MISSING']);
|
||||
expect(codes('a', { status: 'hurts', ...VERDICT, r48: { verdict: 'DROP', ref: 'x' } })).toEqual(['LEDGER_R48_MISSING']);
|
||||
expect(codes('a', { status: 'hurts', ...VERDICT, r48: { verdict: 'KEEP' } })).toEqual(['LEDGER_R48_MISSING']);
|
||||
});
|
||||
|
||||
test('safety needs a reason; unknown statuses, unknown fields and bad prior entries are refused', () => {
|
||||
expect(codes('a', { status: 'safety' })).toEqual(['LEDGER_REASON_MISSING']);
|
||||
expect(codes('a', { status: 'exempt', reason: 'x' })).toEqual(['LEDGER_STATUS_INVALID']);
|
||||
expect(codes('a', { status: 'safety', reason: 'x', source: 'typo' })).toEqual(['LEDGER_FIELD_UNKNOWN']);
|
||||
expect(codes('a', { status: 'untested', grandfathered: true, plan: 'none', prior: [{ ref: 'run 7', why: ['vibes'] }] })).toEqual(['LEDGER_PRIOR_INVALID']);
|
||||
expect(codes('a', { status: 'untested', grandfathered: true, plan: 'none', prior: [{ why: ['stub'] }] })).toEqual(['LEDGER_PRIOR_INVALID']);
|
||||
expect(codes('a', { status: 'untested', grandfathered: true, plan: 'none', prior: [{ ref: 'run 7', why: ['toString'] }] })).toEqual(['LEDGER_PRIOR_INVALID']);
|
||||
});
|
||||
|
||||
test('a present r48 is checked on every status, and aliases is a list of names', () => {
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, r48: { verdict: 'DROP', ref: 'x' } })).toEqual(['LEDGER_R48_MISSING']);
|
||||
expect(codes('a', { status: 'helps', ...VERDICT, r48: {} })).toEqual(['LEDGER_R48_MISSING']);
|
||||
expect(codes('a', { status: 'safety', reason: 'x', aliases: 'other-name' })).toEqual(['LEDGER_FIELD_INVALID']);
|
||||
expect(codes('a', { status: 'safety', reason: 'x', aliases: [] })).toEqual(['LEDGER_FIELD_INVALID']);
|
||||
expect(codes('a', { status: 'safety', reason: 'x', aliases: ['other-name'] })).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
/** A small gstack-shaped tree: a root skill, two skills, the shared-part files and the cited dispatch sources. */
|
||||
function fixtureTree(dir: string, parts: (keys: string[]) => Record<string, unknown>): string {
|
||||
const write = (rel: string, text: string) => { fs.mkdirSync(path.dirname(path.join(dir, rel)), { recursive: true }); fs.writeFileSync(path.join(dir, rel), text); };
|
||||
const skill = (rel: string, name: string, tier: number, body: string) => {
|
||||
const fm = `---\nname: ${name}\npreamble-tier: ${tier}\ndescription: |\n Does ${name} things.\n---\n`;
|
||||
write(path.join(rel, 'SKILL.md.tmpl'), fm + body);
|
||||
write(path.join(rel, 'SKILL.md'), fm + '<!-- AUTO-GENERATED from SKILL.md.tmpl -->\n' + body);
|
||||
};
|
||||
skill('', 'gstack', 1, 'router\n');
|
||||
skill('alpha', 'alpha', 2, 'a'.repeat(4000) + '\n');
|
||||
skill('beta', 'beta', 2, 'b'.repeat(1000) + '\n');
|
||||
write('alpha/sections/preamble.md', 'p'.repeat(800));
|
||||
write('beta/sections/preamble.md', 'p'.repeat(800));
|
||||
for (const part of SHARED_PARTS) for (const f of part.files) write(f, 'x'.repeat(500) + '\n');
|
||||
for (const d of DISPATCHES) if (!fs.existsSync(path.join(dir, d.file))) write(d.file, 'filler\n');
|
||||
for (const d of DISPATCHES) fs.appendFileSync(path.join(dir, d.file), `${d.quote}\n`);
|
||||
const keys = inventoryKeys(dir);
|
||||
write(LEDGER_PATH, JSON.stringify({ schema_version: 1, parts: parts(keys), unmapped: [{ row: 'gstack as a whole', why: 'not a single text part' }] }, null, 2));
|
||||
return dir;
|
||||
}
|
||||
|
||||
const grandfathered = (keys: string[]) => Object.fromEntries(keys.map(k => [k, k === 'beta' ? { status: 'safety', grandfathered: true, reason: 'a guard' } : { status: 'untested', grandfathered: true, plan: 'none' }]));
|
||||
|
||||
describe('bin/gstack-inventory', () => {
|
||||
test('determinism: two runs on a fixed tree write the same docs/inventory.md and the same --json', () => {
|
||||
const dir = fixtureTree(path.join(tmp, 'fixed'), grandfathered);
|
||||
const first = run(['--root', dir]);
|
||||
expect(first.status, first.stderr).toBe(0);
|
||||
const a = fs.readFileSync(path.join(dir, 'docs/inventory.md'), 'utf8');
|
||||
const second = run(['--root', dir]);
|
||||
expect(second.status, second.stderr).toBe(0);
|
||||
expect(fs.readFileSync(path.join(dir, 'docs/inventory.md'), 'utf8')).toBe(a);
|
||||
expect(run(['--root', dir, '--json']).stdout).toBe(run(['--root', dir, '--json']).stdout);
|
||||
expect(a).toContain('Measured at commit none');
|
||||
expect(a.indexOf('| alpha |')).toBeLessThan(a.indexOf('| beta |'));
|
||||
const start = a.indexOf('## Measure next');
|
||||
const next = a.slice(start, a.indexOf('\n## ', start + 1));
|
||||
expect(next).toContain('alpha');
|
||||
expect(next).not.toContain('| beta |');
|
||||
expect(a).toContain('gstack as a whole');
|
||||
expect(a).toContain('gstack-<name>');
|
||||
for (const d of DISPATCHES) expect(a).toContain(`${d.file}:`);
|
||||
});
|
||||
|
||||
test('--check: ok on a complete ledger; a missing key exits 1 with its code', () => {
|
||||
const dir = fixtureTree(path.join(tmp, 'check'), grandfathered);
|
||||
const ok = run(['--root', dir, '--check']);
|
||||
expect(ok.status, ok.stderr).toBe(0);
|
||||
expect(ok.stdout).toMatch(/^INVENTORY: ok keys=\d+/m);
|
||||
const bad = fixtureTree(path.join(tmp, 'check-bad'), keys => { const p = grandfathered(keys); delete p.alpha; return p; });
|
||||
const r = run(['--root', bad, '--check']);
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toContain('alpha: no row in docs/inventory/evidence.json (LEDGER_KEY_MISSING)');
|
||||
expect(run(['--root', bad]).status).toBe(1);
|
||||
});
|
||||
|
||||
test('--check: a cited dispatch source that lost its quote is named', () => {
|
||||
const dir = fixtureTree(path.join(tmp, 'drift'), grandfathered);
|
||||
fs.writeFileSync(path.join(dir, DISPATCHES[0]!.file), 'rewritten\n');
|
||||
const r = run(['--root', dir, '--check']);
|
||||
expect(r.status).toBe(1);
|
||||
expect(r.stderr).toContain('(INVENTORY_SOURCE_MISSING)');
|
||||
});
|
||||
|
||||
test('a bad --root, a root outside a gstack checkout and a gstack install are refused, and nothing is written', () => {
|
||||
const dir = fixtureTree(path.join(tmp, 'args'), grandfathered);
|
||||
for (const args of [['--root', '--json'], ['--root', '--check'], ['--root']]) expect(run(args).status).toBe(2);
|
||||
expect(fs.existsSync(path.join(dir, 'docs/inventory.md'))).toBe(false);
|
||||
const outside = spawnSync(BIN, [], { cwd: tmp, encoding: 'utf8', timeout: 120_000 });
|
||||
expect(outside.status).toBe(2);
|
||||
expect(outside.stderr).toContain('is not a gstack checkout');
|
||||
const installed = fixtureTree(path.join(tmp, 'home', '.claude', 'skills', 'gstack'), grandfathered);
|
||||
const w = run(['--root', installed]);
|
||||
expect(w.status).toBe(2);
|
||||
expect(w.stderr).toContain('refusing to write docs/inventory.md');
|
||||
expect(fs.existsSync(path.join(installed, 'docs/inventory.md'))).toBe(false);
|
||||
expect(run(['--root', installed, '--check']).status).toBe(0);
|
||||
expect(installReason(dir, { GSTACK_CAPY_ROOT: dir })).toContain('the Capy install');
|
||||
expect(installReason(dir, {}, path.join(tmp, 'home'))).toBeNull();
|
||||
expect(installReason(path.join(tmp, 'capy-root'), {}, path.join(tmp, 'h'))).toBeNull();
|
||||
});
|
||||
|
||||
test('the real ledger passes --check', () => {
|
||||
const r = run(['--check']);
|
||||
expect(r.status, r.stderr).toBe(0);
|
||||
});
|
||||
});
|
||||
+37
-2
@@ -47,7 +47,7 @@ describe('gstack-skill-bill', () => {
|
||||
const root = gstackRepo();
|
||||
|
||||
test('a gstack checkout skill reports every level and its last diet', () => {
|
||||
const r = bill(root, ['demo']);
|
||||
const r = bill(root, ['./demo']);
|
||||
expect(r.status).toBe(0);
|
||||
expect(r.stdout).toContain('SKILL: demo (gstack-checkout)');
|
||||
expect(r.stdout).toMatch(/template\s+SKILL\.md\.tmpl/);
|
||||
@@ -58,7 +58,7 @@ describe('gstack-skill-bill', () => {
|
||||
|
||||
test('growth over 10% since the Skill-Diet commit prints DUE, and an unrendered template is RENDER_STALE', () => {
|
||||
fs.appendFileSync(path.join(root, 'demo', 'SKILL.md.tmpl'), 'y'.repeat(200));
|
||||
const r = bill(root, ['demo', '--json']);
|
||||
const r = bill(root, ['./demo', '--json']);
|
||||
const report = JSON.parse(r.stdout);
|
||||
expect(report.baseline.status).toBe('found');
|
||||
expect(report.baseline.due).toBe(true);
|
||||
@@ -114,6 +114,41 @@ describe('gstack-skill-bill', () => {
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-skill-bill EVIDENCE line (docs/inventory/evidence.json)', () => {
|
||||
const root = path.join(tmp, 'evidence-repo');
|
||||
const ledger = (parts: Record<string, unknown>) => {
|
||||
fs.mkdirSync(path.join(root, 'docs', 'inventory'), { recursive: true });
|
||||
fs.writeFileSync(path.join(root, 'docs', 'inventory', 'evidence.json'), JSON.stringify({ schema_version: 1, parts }));
|
||||
};
|
||||
fs.mkdirSync(path.join(root, 'bin'), { recursive: true });
|
||||
fs.writeFileSync(path.join(root, 'SKILL.md.tmpl'), 'router');
|
||||
fs.writeFileSync(path.join(root, 'VERSION'), '1.0.0');
|
||||
skill(path.join(root, 'demo'), 'demo', 'body\n', 'SKILL.md.tmpl');
|
||||
|
||||
test('each status prints its source, plan or reason; a skill with no row says so', () => {
|
||||
ledger({ demo: { status: 'untested', grandfathered: true, plan: 'none' } });
|
||||
expect(bill(root, ['./demo']).stdout).toContain('EVIDENCE: untested plan: none (grandfathered)');
|
||||
ledger({ demo: { status: 'untested', reason: 'r', plan: 'test/fixtures/skill-evals/demo/tasks.json', owner: 'GSTA-1', measure_by: '2026-12-01' } });
|
||||
expect(bill(root, ['./demo']).stdout).toContain('EVIDENCE: untested plan: test/fixtures/skill-evals/demo/tasks.json (owner GSTA-1, measure by 2026-12-01)');
|
||||
ledger({ demo: { status: 'helps', sources: ['run-9/results.json'], models: ['claude-opus-5-5'], prompt: 'preset', scope: 's', masked: true } });
|
||||
expect(bill(root, ['./demo']).stdout).toContain('EVIDENCE: helps sources: run-9/results.json');
|
||||
ledger({ demo: { status: 'safety', reason: 'blocks rm -rf' } });
|
||||
expect(bill(root, ['./demo']).stdout).toContain('EVIDENCE: safety reason: blocks rm -rf');
|
||||
ledger({});
|
||||
expect(bill(root, ['./demo']).stdout).toContain('EVIDENCE: missing: no demo row in docs/inventory/evidence.json');
|
||||
expect(JSON.parse(bill(root, ['./demo', '--json']).stdout).evidence).toMatchObject({ key: 'demo', status: 'missing' });
|
||||
fs.writeFileSync(path.join(root, 'docs', 'inventory', 'evidence.json'), '{bad');
|
||||
const broken = bill(root, ['./demo']).stdout;
|
||||
expect(broken).toContain('EVIDENCE: unreadable: cannot read docs/inventory/evidence.json');
|
||||
expect(broken).not.toContain('add one');
|
||||
});
|
||||
|
||||
test('the real checkout: a guard prints its safety reason', () => {
|
||||
const r = spawnSync(BIN, ['careful'], { cwd: path.resolve(import.meta.dir, '..'), encoding: 'utf-8', timeout: 30_000 });
|
||||
expect(r.stdout).toMatch(/^EVIDENCE: safety reason: .+/m);
|
||||
});
|
||||
});
|
||||
|
||||
describe('gstack-skill-bill baselines and the root skill', () => {
|
||||
test('the gstack root skill is a checkout skill, and Skill-Diet: gstack-upgrade is not its diet', () => {
|
||||
const root = path.join(tmp, 'gstack-root');
|
||||
|
||||
@@ -102,8 +102,8 @@ telemetry — it never blocks the workflow.
|
||||
|
||||
## 1. Find the skill
|
||||
|
||||
Run `~/.claude/skills/gstack/bin/gstack-skill-bill <name|folder>`. It resolves exact names only and prints the
|
||||
skill's kind and its size per level. A new skill: make the folder first (in a gstack checkout, follow CONTRIBUTING.md
|
||||
Run `~/.claude/skills/gstack/bin/gstack-skill-bill <name|folder>`. It prints the skill's kind,
|
||||
its size per level and its `EVIDENCE:` row. A new skill: make the folder first (in a gstack checkout, follow CONTRIBUTING.md
|
||||
"Adding a new skill"), then write every line by sections 2 and 3.
|
||||
- `user`: edit it in place.
|
||||
- `gstack-install`: never edit; gstack-upgrade overwrites it. Edit `<skill>/SKILL.md.tmpl` in a gstack checkout, or
|
||||
@@ -126,7 +126,9 @@ Put each fact on the lowest level that still reaches the agent before it needs i
|
||||
|
||||
Stop at the first step that handles the change.
|
||||
1. **Fix it below the skill.** A clearer error, a flag, a check or a test in the tool. Prose is for judgment calls.
|
||||
2. **Check the model needs it.** If a fresh session does the right thing without the line, add nothing.
|
||||
2. **Earn its place.** Add a skill, step, instruction, check or default only with evidence it helps frontier models
|
||||
or a dated plan to measure it (skill-evals `tasks.json` or an /eval-plan prereg). If a fresh session does it right
|
||||
without the line, add nothing.
|
||||
3. **Edit, don't append.** Change the line that already covers the idea; each meaning lives in one place.
|
||||
4. **Pick its level** (section 2).
|
||||
5. **Rule, not story.** One positive rule under a short name; the incident goes in the commit message.
|
||||
@@ -138,6 +140,8 @@ When code changes a name or flag a skill teaches, update the skill in the same c
|
||||
|
||||
- Run `~/.claude/skills/gstack/bin/gstack-skill-bill` before and after, and put `Skill-Size: <before>→<after> bytes (<file>)` in the commit
|
||||
message. DUE means it grew over 10% since its last `Skill-Diet:` commit: diet it instead of adding.
|
||||
- Add `Evidence: <run id, results file or commit>`, `Evidence: plan <tasks.json or prereg> measure_by <date>` or
|
||||
`Evidence: safety: <reason>`. A new gstack skill adds its `docs/inventory/evidence.json` row (`gstack-inventory --check`).
|
||||
- In a gstack checkout: `bun test test/skill-bin-drift.test.ts test/parity-suite.test.ts` and the budgets in
|
||||
CONTRIBUTING.md. A red context-budget ratchet means diet, not a higher ceiling.
|
||||
- Rerun the case that prompted the change in a fresh session.
|
||||
|
||||
@@ -29,8 +29,8 @@ allowed-tools:
|
||||
|
||||
## 1. Find the skill
|
||||
|
||||
Run `~/.claude/skills/gstack/bin/gstack-skill-bill <name|folder>`. It resolves exact names only and prints the
|
||||
skill's kind and its size per level. A new skill: make the folder first (in a gstack checkout, follow CONTRIBUTING.md
|
||||
Run `~/.claude/skills/gstack/bin/gstack-skill-bill <name|folder>`. It prints the skill's kind,
|
||||
its size per level and its `EVIDENCE:` row. A new skill: make the folder first (in a gstack checkout, follow CONTRIBUTING.md
|
||||
"Adding a new skill"), then write every line by sections 2 and 3.
|
||||
- `user`: edit it in place.
|
||||
- `gstack-install`: never edit; gstack-upgrade overwrites it. Edit `<skill>/SKILL.md.tmpl` in a gstack checkout, or
|
||||
@@ -53,7 +53,9 @@ Put each fact on the lowest level that still reaches the agent before it needs i
|
||||
|
||||
Stop at the first step that handles the change.
|
||||
1. **Fix it below the skill.** A clearer error, a flag, a check or a test in the tool. Prose is for judgment calls.
|
||||
2. **Check the model needs it.** If a fresh session does the right thing without the line, add nothing.
|
||||
2. **Earn its place.** Add a skill, step, instruction, check or default only with evidence it helps frontier models
|
||||
or a dated plan to measure it (skill-evals `tasks.json` or an /eval-plan prereg). If a fresh session does it right
|
||||
without the line, add nothing.
|
||||
3. **Edit, don't append.** Change the line that already covers the idea; each meaning lives in one place.
|
||||
4. **Pick its level** (section 2).
|
||||
5. **Rule, not story.** One positive rule under a short name; the incident goes in the commit message.
|
||||
@@ -65,6 +67,8 @@ When code changes a name or flag a skill teaches, update the skill in the same c
|
||||
|
||||
- Run `~/.claude/skills/gstack/bin/gstack-skill-bill` before and after, and put `Skill-Size: <before>→<after> bytes (<file>)` in the commit
|
||||
message. DUE means it grew over 10% since its last `Skill-Diet:` commit: diet it instead of adding.
|
||||
- Add `Evidence: <run id, results file or commit>`, `Evidence: plan <tasks.json or prereg> measure_by <date>` or
|
||||
`Evidence: safety: <reason>`. A new gstack skill adds its `docs/inventory/evidence.json` row (`gstack-inventory --check`).
|
||||
- In a gstack checkout: `bun test test/skill-bin-drift.test.ts test/parity-suite.test.ts` and the budgets in
|
||||
CONTRIBUTING.md. A red context-budget ratchet means diet, not a higher ceiling.
|
||||
- Rerun the case that prompted the change in a fresh session.
|
||||
|
||||
Reference in new issue
Block a user