Garry Tanandcapy-ai[bot]
7204a39a27
* test(skill-evals): /learn before/after tasks, written before any cut (GSTA-133) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learnings): --help, unknown-flag errors and --stats on the learnings bins; cross-project search no longer drops every project when the state root path contains the slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * refactor(learn): diet /learn onto the levels: tool flags replace the inline stats program, one mode table, prune rules kept Skill-Diet: learn Skill-Size: 5797→2586 bytes (SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(write-skill): /write-skill skill, gstack-skill-bill, static skill-to-bin drift ratchet, shared skill eval contract Skill-Size: 0→3245 bytes (write-skill/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): guard the state-root lookup in Prune; re-measure the learn ceiling Skill-Size: 2586→2707 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (drift scanner, learnings flags, skill-bill baselines, eval fixtures) - skill-bin-drift: a redirect no longer ends a command, so `bin/x 2>/dev/null --bogus` is checked - gstack-learnings-search: a missing value or a non-positive --limit exits 2 like an unknown flag - gstack-skill-bill: the gstack root skill is a checkout skill; Skill-Diet trailers match the exact name; the baseline counts the committed bytes untrimmed - skill-evals task files get a free contract test and are registered in FREE_FIXTURES, with the drift allowlist - learn eval: the no-code-change safety task no longer asks the model to fix code (its check penalised obeying the user) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn,write-skill): full bin paths in the mode table; prune asks per entry unless the request names the action Skill-Diet: learn Skill-Size: 2707→2955 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): /write-skill, gstack-skill-bill, drift ratchet, eval contract, /learn diet Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: drift ratchet in CONTRIBUTING's static tests; /learn prune example asks per entry Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): report a checkout skill's eval task file and check it against the contract The skill-evals task-file test now exercises lib/skill-bill.ts's checkEvalTasks, with a negative control, instead of a test-only validator (test-of-test ratchet). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): EVAL TASKS path uses forward slashes on every platform; a non-object task entry is a problem, not a crash Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): BASH_COMPAT guard on in-window heredocs; count skill-e2e-write-skill in the paid census Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2.8 KiB
Full diet
Rewrite a whole skill onto the levels, then prove agents do at least as well with it. Adapted from nicobailon/skill-diet (MIT) at 5f41df1.
1. Report first (free)
Run ~/.claude/skills/gstack/bin/gstack-skill-bill <skill> and, in a gstack checkout, bun run scripts/skill-bin-drift.ts --skill <skill>.
Show the user the size per level, every drift finding, and the moves you propose (what goes to references/, to the
tool's --help or errors, or out). Ask go or no-go before writing eval tasks.
2. Consumers
List what depends on the skill's text: skills that link it, code that embeds it, tests that assert it, prompts that name it. In gstack also the paid E2E cases that touchfiles select for its folder.
3. Tool steps (skills that teach a tool)
- Prove each mismatch by running the tool or citing its source line: names it rejects, flags it ignores, operations the skill never mentions.
- The tool's help can replace skill text only when every operation has an entry, an unknown name or flag is an error that names the valid ones, and common failures say how to fix them. Fix the tool first when you own it.
- Keep a cheat sheet of the calls agents make most, one full call each, plus one example per call shape they get wrong. Name every other operation once.
4. Tasks before cuts
Write 8–15 one-line tasks while the old skill is the only version, and commit them before any cut, at
test/fixtures/skill-evals/<skill>/tasks.json ({id, prompt, setup?, must?, must_not?, check?, safety?}; setup
may call $SKILL_EVAL_FIXTURES/<script>). Cover each mode, each "don't" rule as a safety task, and each risky area.
Give each task a check that asserts the end state where it can.
5. Cut and write
Cut what the model already knows, what only a human needs (motivation, history: commit message or README), and every
word a sentence can lose without changing what the agent does. Name each rule with a short phrase; replace a list of
cases with the idea behind them and a list of don'ts with one positive test. End with one case → where to look line
per reference.
6. Verify
Run the before, after and no-skill versions through the shared skill eval
(docs/designs/SKILL_EVAL_CONTRACT.md), 3 trials each on the same model. Ship only when no task whose before panel
passed fails after, no safety task breaks, and after is not worse by more than chance. Until the shared eval is
available, run upstream skill-diet's scripts/eval.mjs at commit 5f41df1 (the version this text adapts) with the
same tasks and apply those rules to its per-task table;
when neither can run, write "eval: not run" and ship only the tool and drift fixes, not the cuts.
Commit with Skill-Diet: <name> and Skill-Size: <before>→<after> bytes (<file>) lines, and report tokens per run
before and after, the per-task table, and what moved where.