mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-10 10:40:32 +01:00
Garry Tanandcapy-ai[bot]
7204a39a27
* test(skill-evals): /learn before/after tasks, written before any cut (GSTA-133) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learnings): --help, unknown-flag errors and --stats on the learnings bins; cross-project search no longer drops every project when the state root path contains the slug Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * refactor(learn): diet /learn onto the levels: tool flags replace the inline stats program, one mode table, prune rules kept Skill-Diet: learn Skill-Size: 5797→2586 bytes (SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * feat(write-skill): /write-skill skill, gstack-skill-bill, static skill-to-bin drift ratchet, shared skill eval contract Skill-Size: 0→3245 bytes (write-skill/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): guard the state-root lookup in Prune; re-measure the learn ceiling Skill-Size: 2586→2707 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes (drift scanner, learnings flags, skill-bill baselines, eval fixtures) - skill-bin-drift: a redirect no longer ends a command, so `bin/x 2>/dev/null --bogus` is checked - gstack-learnings-search: a missing value or a non-positive --limit exits 2 like an unknown flag - gstack-skill-bill: the gstack root skill is a checkout skill; Skill-Diet trailers match the exact name; the baseline counts the committed bytes untrimmed - skill-evals task files get a free contract test and are registered in FREE_FIXTURES, with the drift allowlist - learn eval: the no-code-change safety task no longer asks the model to fix code (its check penalised obeying the user) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn,write-skill): full bin paths in the mode table; prune asks per entry unless the request names the action Skill-Diet: learn Skill-Size: 2707→2955 bytes (learn/SKILL.md.tmpl) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(changelog): /write-skill, gstack-skill-bill, drift ratchet, eval contract, /learn diet Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: drift ratchet in CONTRIBUTING's static tests; /learn prune example asks per entry Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): report a checkout skill's eval task file and check it against the contract The skill-evals task-file test now exercises lib/skill-bill.ts's checkEvalTasks, with a negative control, instead of a test-only validator (test-of-test ratchet). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(skill-bill): EVAL TASKS path uses forward slashes on every platform; a non-object task entry is a problem, not a crash Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(learn): BASH_COMPAT guard on in-window heredocs; count skill-e2e-write-skill in the paid census Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
51 lines
2.8 KiB
Markdown
51 lines
2.8 KiB
Markdown
# Full diet
|
||
|
||
Rewrite a whole skill onto the levels, then prove agents do at least as well with it. Adapted from
|
||
nicobailon/skill-diet (MIT) at 5f41df1.
|
||
|
||
## 1. Report first (free)
|
||
|
||
Run `~/.claude/skills/gstack/bin/gstack-skill-bill <skill>` and, in a gstack checkout, `bun run scripts/skill-bin-drift.ts --skill <skill>`.
|
||
Show the user the size per level, every drift finding, and the moves you propose (what goes to `references/`, to the
|
||
tool's `--help` or errors, or out). Ask go or no-go before writing eval tasks.
|
||
|
||
## 2. Consumers
|
||
|
||
List what depends on the skill's text: skills that link it, code that embeds it, tests that assert it, prompts that
|
||
name it. In gstack also the paid E2E cases that touchfiles select for its folder.
|
||
|
||
## 3. Tool steps (skills that teach a tool)
|
||
|
||
- Prove each mismatch by running the tool or citing its source line: names it rejects, flags it ignores, operations the
|
||
skill never mentions.
|
||
- The tool's help can replace skill text only when every operation has an entry, an unknown name or flag is an error
|
||
that names the valid ones, and common failures say how to fix them. Fix the tool first when you own it.
|
||
- Keep a cheat sheet of the calls agents make most, one full call each, plus one example per call shape they get wrong.
|
||
Name every other operation once.
|
||
|
||
## 4. Tasks before cuts
|
||
|
||
Write 8–15 one-line tasks while the old skill is the only version, and commit them before any cut, at
|
||
`test/fixtures/skill-evals/<skill>/tasks.json` (`{id, prompt, setup?, must?, must_not?, check?, safety?}`; `setup`
|
||
may call `$SKILL_EVAL_FIXTURES/<script>`). Cover each mode, each "don't" rule as a safety task, and each risky area.
|
||
Give each task a `check` that asserts the end state where it can.
|
||
|
||
## 5. Cut and write
|
||
|
||
Cut what the model already knows, what only a human needs (motivation, history: commit message or README), and every
|
||
word a sentence can lose without changing what the agent does. Name each rule with a short phrase; replace a list of
|
||
cases with the idea behind them and a list of don'ts with one positive test. End with one `case → where to look` line
|
||
per reference.
|
||
|
||
## 6. Verify
|
||
|
||
Run the before, after and no-skill versions through the shared skill eval
|
||
(docs/designs/SKILL_EVAL_CONTRACT.md), 3 trials each on the same model. Ship only when no task whose before panel
|
||
passed fails after, no safety task breaks, and after is not worse by more than chance. Until the shared eval is
|
||
available, run upstream skill-diet's `scripts/eval.mjs` at commit 5f41df1 (the version this text adapts) with the
|
||
same tasks and apply those rules to its per-task table;
|
||
when neither can run, write "eval: not run" and ship only the tool and drift fixes, not the cuts.
|
||
|
||
Commit with `Skill-Diet: <name>` and `Skill-Size: <before>→<after> bytes (<file>)` lines, and report tokens per run
|
||
before and after, the per-task table, and what moved where.
|