v1.91.287.0 feat(write-skill): write, change and diet skills; /learn at half its template size (#3524)

* test(skill-evals): /learn before/after tasks, written before any cut (GSTA-133) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learnings): --help, unknown-flag errors and --stats on the learnings bins; cross-project search no longer drops every project when the state root path contains the slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * refactor(learn): diet /learn onto the levels: tool flags replace the inline stats program, one mode table, prune rules kept Skill-Diet: learn Skill-Size: 5797→2586 bytes (SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(write-skill): /write-skill skill, gstack-skill-bill, static skill-to-bin drift ratchet, shared skill eval contract Skill-Size: 0→3245 bytes (write-skill/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): guard the state-root lookup in Prune; re-measure the learn ceiling Skill-Size: 2586→2707 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (drift scanner, learnings flags, skill-bill baselines, eval fixtures) - skill-bin-drift: a redirect no longer ends a command, so `bin/x 2>/dev/null --bogus` is checked - gstack-learnings-search: a missing value or a non-positive --limit exits 2 like an unknown flag - gstack-skill-bill: the gstack root skill is a checkout skill; Skill-Diet trailers match the exact name; the baseline counts the committed bytes untrimmed - skill-evals task files get a free contract test and are registered in FREE_FIXTURES, with the drift allowlist - learn eval: the no-code-change safety task no longer asks the model to fix code (its check penalised obeying the user) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn,write-skill): full bin paths in the mode table; prune asks per entry unless the request names the action Skill-Diet: learn Skill-Size: 2707→2955 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): /write-skill, gstack-skill-bill, drift ratchet, eval contract, /learn diet Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: drift ratchet in CONTRIBUTING's static tests; /learn prune example asks per entry Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): report a checkout skill's eval task file and check it against the contract The skill-evals task-file test now exercises lib/skill-bill.ts's checkEvalTasks, with a negative control, instead of a test-only validator (test-of-test ratchet). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): EVAL TASKS path uses forward slashes on every platform; a non-object task entry is a problem, not a crash Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): BASH_COMPAT guard on in-window heredocs; count skill-e2e-write-skill in the paid census Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-31 22:14:25 +01:00 · 2026-10-28 02:58:29 +00:00
1 parent 398618259d
commit 7204a39a27
@@ -20,7 +20,7 @@ cuts to the coverage audit, review army, or voice directive have real quality co

 A second, harder ceiling guards the DISCOVERY surface: `test/catalog-budget.test.ts`
 caps the aggregate frontmatter `name` + `description` across all skills at
-`CATALOG_BUDGET_TOKEN_EQUIVALENTS` (1,359 today; each new skill ratchets it)
+`CATALOG_BUDGET_TOKEN_EQUIVALENTS` (1,393 today; each new skill ratchets it)
 token-equivalents (260-byte per-skill sub-cap), counted through the shared census
 in `test/helpers/skill-census.ts`. This one is enforced, not a warning — every
 host loads the full catalog every session, so growth here taxes every
@@ -53,6 +53,7 @@ gstack/
 ├── ship/            # Ship workflow skill
 ├── review/          # PR review skill (checklist.md is hand-written; design-checklist.md is GENERATED from lib/design-catalog.ts)
 ├── deslop-shared-libs/ # Recommendations-only audit for worthwhile shared-code extractions
+├── write-skill/     # /write-skill skill (edit ladder, levels, gstack-skill-bill, full diet in references/diet.md)
 ├── test-audit/      # Report-first sweep for low-value tests (test value bar, audit mode)
 ├── eval-plan/       # Preregistered paid evals: prereg, arms, $0 dry run, priced pilot (bin/gstack-eval-plan)
 ├── captain/         # /captain (also /crew): one controller, many lanes; templates/ holds CREW.md, AGENTS.md, NOTES.md, DIGEST.md, the flaky ledger, the lane report, the READY block, slot call, the lane brief, asks and the daily automation prompt
@@ -0,0 +1,71 @@
+# Skill eval contract (what gstack needs from the shared skill eval)
+
+Status: interface request from gstack (GSTA-133) to the gstack-evals Captain (GSTA-2), 2026-10-27. gstack ships no
+eval harness of its own; `/write-skill`'s diet mode calls whatever implements this contract. Until it exists, the diet
+procedure runs upstream [skill-diet](https://github.com/nicobailon/skill-diet)'s `scripts/eval.mjs` on the same task
+files and applies the policy below to its per-task table.
+
+## Call
+
+```
+<cli> skill-eval --arms before=<skill-dir>,after=<skill-dir>,none --tasks <tasks.json>
+                 --host claude|codex --model <id> --trials 3 --spend-cap <usd> --out <dir>
+                 [--dry-run] [--calibrate] [--tasks-only <id,...>]
+```
+
+- An arm is a skill folder (SKILL.md plus anything it reads) or `none`. A gstack skill passes its rendered folder.
+- `--dry-run` prints the session count and a price estimate and starts nothing. `--spend-cap` is enforced while
+  running: the run stops before the next session would pass it and the verdict is INCOMPLETE.
+- `--calibrate` runs only the three checks: the agent quotes a line only its arm's copy holds, says it has no skill
+  in the `none` arm, and runs a shell command. A real run calibrates first and stops on a failed check.
+- Runs locally on a developer or Capy machine first. CI use, if any, is a manual or periodic job; never the free suite.
+- Paid sessions bill through the caller's own keys, and the verdict records their cost.
+
+## Tasks
+
+`test/fixtures/skill-evals/<skill>/tasks.json` in the repository that owns the skill:
+
+```json
+[{ "id": "stats", "prompt": "one-line request", "setup": "sh \"$SKILL_EVAL_FIXTURES/seed.sh\" full",
+   "must": ["regex"], "must_not": ["regex"], "check": "shell command that exits 0", "safety": true }]
+```
+
+`setup` and `check` run in the session's work dir; `SKILL_EVAL_FIXTURES` is the tasks file's directory. `must`
+patterns match the final answer; `must_not` patterns match its code blocks. `safety: true` marks a task whose
+failure fails the diet on its own. Tasks are committed before the skill is cut.
+
+## Output
+
+`<out>/verdict.json`:
+
+```json
+{ "verdict": "PASS|FAIL|INCOMPLETE", "reasons": ["REGRESSION stats", "SAFETY no-code-change", "WORSE", "UNCOVERED empty"],
+  "harness_version": "…", "model": "…", "trials": 3,
+  "calibration": { "skill": true, "none": true, "shell": true },
+  "arms": { "before": { "pass_rate": 0.9, "tokens_per_run": { "input": 0, "output": 0, "cache_read": 0, "cache_write": 0 },
+                        "turns_per_run": 0, "opened_skill_rate": 1.0, "files_loaded": { "SKILL.md": 30 }, "cost_usd": 0 } },
+  "tasks": { "stats": { "before": [true, true, true], "after": [true, true, false], "none": [false, false, false] } },
+  "adjudication": { "blinded": true, "judge_model": "…", "disputed": [] } }
+```
+
+Raw transcripts stay under `<out>/runs/`.
+
+## Exit codes
+
+- `0` PASS
+- `1` FAIL (a regression, a safety failure, or WORSE)
+- `2` INCOMPLETE (spend cap reached, infra failure, timeouts)
+- `3` calibration failed
+
+## Policy
+
+- **Per task, not aggregate.** A task whose `before` panel passed (at least 2 of 3) must still pass in `after`
+  (at least 2 of 3), or the verdict is FAIL with `REGRESSION <id>`.
+- **Safety.** Any failed trial of a `safety` task in `after` is FAIL.
+- **Uncovered.** A task every arm fails is reported as `UNCOVERED <id>`; it is never dropped silently.
+- **WORSE.** `after`'s overall pass rate lower than `before`'s by more than two standard errors is FAIL.
+- **Opened the skill.** Under half of the skill-arm runs opening the skill is a warning: the scores then measure
+  the tool's own text.
+- **Trials are fixed before the run.** No trials are added after a result. One same-tree rerun of a red task is
+  allowed and recorded as flaky; a second failure stands.
+- **Blinded adjudication.** A disputed answer goes to a fresh judge that sees the task and the answer, not the arm.
@@ -42,6 +42,7 @@ Detailed guides for every gstack skill — philosophy, workflow, and examples.
 | [`/context-restore`](#context-restore) | **Restore State** | Resume from a saved context, even across Conductor workspace handoffs. |
 | [`/health`](#health) | **Code Quality Dashboard** | Wraps type checker, linter, tests, dead code detection. Computes a weighted 0-10 score; tracks trends over time. |
 | [`/deslop-shared-libs`](#deslop-shared-libs) | **Shared Code Reviewer** | Find worthwhile shared-code extractions in recent work. Recommendations only. |
+| [`/write-skill`](#write-skill) | **Skill Editor** | Write or improve an agent skill without bloat: edit ladder, size per level, skill-to-bin drift check, full diets with a before/after test. |
 | [`/test-audit`](#test-audit) | **Test Auditor** | Sweep existing tests for low-value, implementation-coupled or duplicate tests. Report-only unless you approve a batch. |
 | [`/landing-report`](#landing-report) | **Ship Queue Dashboard** | Read-only snapshot of the workspace-aware ship queue. Which version slots are claimed, which sibling workspaces have WIP. |
 | [`/captain`](#captain) | **Crew Controller** | Run many coding agents from one controller. Lanes build and send a READY block; the controller validates it, merges through trains (stamp on call, verify, merge with the expected head, verify again), keeps the digest, relays approvals in three fixed shapes and runs a daily flaky-fix pass. Also answers to `/crew`. |
@@ -885,6 +886,33 @@ checks do not run the history audit. Optional extractions are advisory and requi
 approval; they do not block a clean review or reduce its score. Actual defects
 keep their normal fix handling.

+## `/write-skill`
+
+Every skill is read in full each time it runs, so every extra line costs tokens on every run, and a skill that names a
+command the tool no longer has produces wrong calls that pass CI. `/write-skill` keeps skills lean and correct, for
+gstack's own templates and for any skill folder you own.
+
+- **Find the skill.** `gstack-skill-bill <name|folder>` resolves exact names only and says what you are holding: a
+  folder you own, a gstack checkout template (edit `SKILL.md.tmpl`, then `bun run gen:skill-docs`), or an installed
+  gstack skill, which it never edits because `gstack-upgrade` overwrites it.
+- **Edit ladder.** Stop at the first step that handles the change: fix the tool (a clearer error, a flag, a test);
+  add nothing a fresh session already does; edit the line that covers the idea instead of appending; put the fact on the
+  lowest level that reaches the agent in time (description, SKILL.md, `references/`, the tool's `--help`, the tool's
+  errors); write the rule and put the story in the commit message.
+- **Size.** `gstack-skill-bill` prints bytes and estimated tokens per level, and in a gstack checkout the skill's eval
+  task file (`EVAL TASKS:`). Commits carry
+  `Skill-Size: <before>→<after> bytes`; a skill more than 10% over its last `Skill-Diet:` commit is DUE, and a red
+  context-budget ratchet means diet, not a higher ceiling.
+- **Drift.** `test/skill-bin-drift.test.ts` fails when a template or rendered skill calls a gstack bin or flag that
+  does not exist (`bun run scripts/skill-bin-drift.ts --skill <name>` for one skill). Its allowlist only shrinks.
+- **Full diet.** Report first, write the eval tasks before any cut
+  (`test/fixtures/skill-evals/<skill>/tasks.json`), cut, then prove before vs after vs no skill with the shared skill
+  eval (`docs/designs/SKILL_EVAL_CONTRACT.md`). A diet ships only when no task that passed before fails after.
+
+`/write-skill` is not `/skillify`: `/skillify` codifies a browser scrape into a browser-skill. Adapted from
+[nicobailon/skill-diet](https://github.com/nicobailon/skill-diet) (MIT). The first diet, /learn, cut its template from
+5,797 to 2,955 bytes.
+
 ## `/test-audit`

 Find existing tests that cost more than they protect. `/review`, `/ship`, `/qa` and
@@ -1242,7 +1270,7 @@ Claude: 23 learnings for this project (14 high confidence, 6 medium, 3 low)

        3 potentially stale (referenced files deleted):
        - "auth middleware uses JWT" — auth/middleware.ts was deleted
-        [Prune these? Y/N]
+        [Remove, keep, or update each one?]
 ```

 ---