Files
gstack/docs/designs/SKILL_EVAL_CONTRACT.md
Garry Tanandcapy-ai[bot] 8be7f21385 v1.91.351.0 fix(skill-evals): close four Lane A check-command holes (GST-377) (#3664)
* fix(skill-evals): close four Lane A check-command holes (GST-377)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: register skill-eval-checks in the test-of-test baseline; docs: describe the tightened checks (GST-377)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-31 05:17:59 +00:00

99 lines
6.1 KiB
Markdown

# Skill eval contract (what gstack needs from the shared skill eval)
Status: interface request from gstack (GSTA-133) to the gstack-evals Captain (GSTA-2), 2026-10-27. gstack ships no
eval harness of its own; `/write-skill`'s diet mode calls whatever implements this contract. Until it exists, the diet
procedure runs upstream [skill-diet](https://github.com/nicobailon/skill-diet)'s `scripts/eval.mjs` on the same task
files and applies the policy below to its per-task table.
## Call
```
<cli> skill-eval --arms before=<skill-dir>,after=<skill-dir>,none --tasks <tasks.json>
--host claude|codex --model <id> --trials 3 --spend-cap <usd> --out <dir>
[--dry-run] [--calibrate] [--tasks-only <id,...>]
```
- An arm is a skill folder (SKILL.md plus anything it reads) or `none`. A gstack skill passes its rendered folder.
- `--dry-run` prints the session count and a price estimate and starts nothing. `--spend-cap` is enforced while
running: the run stops before the next session would pass it and the verdict is INCOMPLETE.
- `--calibrate` runs only the three checks: the agent quotes a line only its arm's copy holds, says it has no skill
in the `none` arm, and runs a shell command. A real run calibrates first and stops on a failed check.
- Runs locally on a developer or Capy machine first. CI use, if any, is a manual or periodic job; never the free suite.
- Paid sessions bill through the caller's own keys, and the verdict records their cost.
## Tasks
`test/fixtures/skill-evals/<skill>/tasks.json` in the repository that owns the skill:
```json
[{ "id": "stats", "prompt": "one-line request", "setup": "sh \"$SKILL_EVAL_FIXTURES/seed.sh\" full",
"must": ["regex"], "must_not": ["regex"], "check": "shell command that exits 0", "safety": true }]
```
`setup` and `check` run in the session's work dir; `SKILL_EVAL_FIXTURES` is the tasks file's directory. `must`
patterns match the final answer; `must_not` patterns match its code blocks. `safety: true` marks a task whose
failure fails the diet on its own. Tasks are committed before the skill is cut.
## Output
`<out>/verdict.json`:
```json
{ "verdict": "PASS|FAIL|INCOMPLETE", "reasons": ["REGRESSION stats", "SAFETY no-code-change", "WORSE", "UNCOVERED empty"],
"harness_version": "…", "model": "…", "trials": 3,
"calibration": { "skill": true, "none": true, "shell": true },
"arms": { "before": { "pass_rate": 0.9, "tokens_per_run": { "input": 0, "output": 0, "cache_read": 0, "cache_write": 0 },
"turns_per_run": 0, "opened_skill_rate": 1.0, "files_loaded": { "SKILL.md": 30 }, "cost_usd": 0 } },
"tasks": { "stats": { "before": [true, true, true], "after": [true, true, false], "none": [false, false, false] } },
"adjudication": { "blinded": true, "judge_model": "…", "disputed": [] } }
```
Raw transcripts stay under `<out>/runs/`.
## Exit codes
- `0` PASS
- `1` FAIL (a regression, a safety failure, or WORSE)
- `2` INCOMPLETE (spend cap reached, infra failure, timeouts)
- `3` calibration failed
## Policy
- **Per task, not aggregate.** A task whose `before` panel passed (at least 2 of 3) must still pass in `after`
(at least 2 of 3), or the verdict is FAIL with `REGRESSION <id>`.
- **Safety.** Any failed trial of a `safety` task in `after` is FAIL.
- **Uncovered.** A task every arm fails is reported as `UNCOVERED <id>`; it is never dropped silently.
- **WORSE.** `after`'s overall pass rate lower than `before`'s by more than two standard errors is FAIL.
- **Opened the skill.** Under half of the skill-arm runs opening the skill is a warning: the scores then measure
the tool's own text.
- **Trials are fixed before the run.** No trials are added after a result. One same-tree rerun of a red task is
allowed and recorded as flaky; a second failure stands.
- **Blinded adjudication.** A disputed answer goes to a fresh judge that sees the task and the answer, not the arm.
## Notes for implementers (2026-10-28)
Offered to gstack-evals' Lane A (interface v0.4). Nothing above changes; these are conventions gstack's task files
already assume, and one optional field.
- **State root.** gstack skills keep state under one root (docs/state-root.md): `GSTACK_STATE_ROOT` wins, then
`GSTACK_HOME`. Set both to `<work dir>/.gstack-home` for the agent, `setup` and `check`; `GSTACK_HOME` alone loses
to a `GSTACK_STATE_ROOT` inherited from the caller's shell, and the agent then writes the caller's real state. A
task's `seed.sh` creates `.gstack-home` and lists it in `.git/info/exclude`, so `git diff` checks ignore it and
`check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks). A
`check` that reads `.gstack-home` first requires it to be a real directory, not a symlink, because `find` does not
descend into a symlinked start point. /learn's checks read the one project folder named for the work dir, and
/office-hours' no-code check lists untracked files against `.git/info/exclude` alone, so a `.gitignore` the agent
writes hides nothing.
- **`resumable`** (optional, in `verdict.json` and any error record): `true` when an INCOMPLETE run can be finished
with `--resume <out>`, which runs only cells that never ran, under the same lock and trial count, so it adds no
trials after a result; `false` for a terminal stop, which needs a fix and a fresh run. A missing field reads as
`false`.
- **Lock.** A paid run needs `tasks.lock.json` committed beside the tasks file; `/write-skill`'s diet step 6 runs
`--lock` and commits it.
- **Sidecar.** Extensions live in `tasks.ext.json` beside the tasks file, keyed by task id, so this format stays
unchanged. gstack's free test checks only that each entry names a task in `tasks.json` and uses known keys.
- **Interactive tasks.** /office-hours' `must` patterns grade a headless run that ends where the skill stops for
input, as upstream skill-diet's runs do. On a host that answers questions from the sidecar's `user` replies, the final answer is
the session's last message, so the sidecar's `judge` line carries the behavior there. Open for Lane A: grade
`must` over the run's assistant messages on such hosts, or keep it on the final answer and rely on `judge`.