mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-19 22:54:36 +01:00
* fix(skill-evals): close four Lane A check-command holes (GST-377) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register skill-eval-checks in the test-of-test baseline; docs: describe the tightened checks (GST-377) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
99 lines
6.1 KiB
Markdown
99 lines
6.1 KiB
Markdown
# Skill eval contract (what gstack needs from the shared skill eval)
|
|
|
|
Status: interface request from gstack (GSTA-133) to the gstack-evals Captain (GSTA-2), 2026-10-27. gstack ships no
|
|
eval harness of its own; `/write-skill`'s diet mode calls whatever implements this contract. Until it exists, the diet
|
|
procedure runs upstream [skill-diet](https://github.com/nicobailon/skill-diet)'s `scripts/eval.mjs` on the same task
|
|
files and applies the policy below to its per-task table.
|
|
|
|
## Call
|
|
|
|
```
|
|
<cli> skill-eval --arms before=<skill-dir>,after=<skill-dir>,none --tasks <tasks.json>
|
|
--host claude|codex --model <id> --trials 3 --spend-cap <usd> --out <dir>
|
|
[--dry-run] [--calibrate] [--tasks-only <id,...>]
|
|
```
|
|
|
|
- An arm is a skill folder (SKILL.md plus anything it reads) or `none`. A gstack skill passes its rendered folder.
|
|
- `--dry-run` prints the session count and a price estimate and starts nothing. `--spend-cap` is enforced while
|
|
running: the run stops before the next session would pass it and the verdict is INCOMPLETE.
|
|
- `--calibrate` runs only the three checks: the agent quotes a line only its arm's copy holds, says it has no skill
|
|
in the `none` arm, and runs a shell command. A real run calibrates first and stops on a failed check.
|
|
- Runs locally on a developer or Capy machine first. CI use, if any, is a manual or periodic job; never the free suite.
|
|
- Paid sessions bill through the caller's own keys, and the verdict records their cost.
|
|
|
|
## Tasks
|
|
|
|
`test/fixtures/skill-evals/<skill>/tasks.json` in the repository that owns the skill:
|
|
|
|
```json
|
|
[{ "id": "stats", "prompt": "one-line request", "setup": "sh \"$SKILL_EVAL_FIXTURES/seed.sh\" full",
|
|
"must": ["regex"], "must_not": ["regex"], "check": "shell command that exits 0", "safety": true }]
|
|
```
|
|
|
|
`setup` and `check` run in the session's work dir; `SKILL_EVAL_FIXTURES` is the tasks file's directory. `must`
|
|
patterns match the final answer; `must_not` patterns match its code blocks. `safety: true` marks a task whose
|
|
failure fails the diet on its own. Tasks are committed before the skill is cut.
|
|
|
|
## Output
|
|
|
|
`<out>/verdict.json`:
|
|
|
|
```json
|
|
{ "verdict": "PASS|FAIL|INCOMPLETE", "reasons": ["REGRESSION stats", "SAFETY no-code-change", "WORSE", "UNCOVERED empty"],
|
|
"harness_version": "…", "model": "…", "trials": 3,
|
|
"calibration": { "skill": true, "none": true, "shell": true },
|
|
"arms": { "before": { "pass_rate": 0.9, "tokens_per_run": { "input": 0, "output": 0, "cache_read": 0, "cache_write": 0 },
|
|
"turns_per_run": 0, "opened_skill_rate": 1.0, "files_loaded": { "SKILL.md": 30 }, "cost_usd": 0 } },
|
|
"tasks": { "stats": { "before": [true, true, true], "after": [true, true, false], "none": [false, false, false] } },
|
|
"adjudication": { "blinded": true, "judge_model": "…", "disputed": [] } }
|
|
```
|
|
|
|
Raw transcripts stay under `<out>/runs/`.
|
|
|
|
## Exit codes
|
|
|
|
- `0` PASS
|
|
- `1` FAIL (a regression, a safety failure, or WORSE)
|
|
- `2` INCOMPLETE (spend cap reached, infra failure, timeouts)
|
|
- `3` calibration failed
|
|
|
|
## Policy
|
|
|
|
- **Per task, not aggregate.** A task whose `before` panel passed (at least 2 of 3) must still pass in `after`
|
|
(at least 2 of 3), or the verdict is FAIL with `REGRESSION <id>`.
|
|
- **Safety.** Any failed trial of a `safety` task in `after` is FAIL.
|
|
- **Uncovered.** A task every arm fails is reported as `UNCOVERED <id>`; it is never dropped silently.
|
|
- **WORSE.** `after`'s overall pass rate lower than `before`'s by more than two standard errors is FAIL.
|
|
- **Opened the skill.** Under half of the skill-arm runs opening the skill is a warning: the scores then measure
|
|
the tool's own text.
|
|
- **Trials are fixed before the run.** No trials are added after a result. One same-tree rerun of a red task is
|
|
allowed and recorded as flaky; a second failure stands.
|
|
- **Blinded adjudication.** A disputed answer goes to a fresh judge that sees the task and the answer, not the arm.
|
|
|
|
## Notes for implementers (2026-10-28)
|
|
|
|
Offered to gstack-evals' Lane A (interface v0.4). Nothing above changes; these are conventions gstack's task files
|
|
already assume, and one optional field.
|
|
|
|
- **State root.** gstack skills keep state under one root (docs/state-root.md): `GSTACK_STATE_ROOT` wins, then
|
|
`GSTACK_HOME`. Set both to `<work dir>/.gstack-home` for the agent, `setup` and `check`; `GSTACK_HOME` alone loses
|
|
to a `GSTACK_STATE_ROOT` inherited from the caller's shell, and the agent then writes the caller's real state. A
|
|
task's `seed.sh` creates `.gstack-home` and lists it in `.git/info/exclude`, so `git diff` checks ignore it and
|
|
`check` reads state there (/learn's `add` and `prune-stale`, /office-hours' design-doc and mockup checks). A
|
|
`check` that reads `.gstack-home` first requires it to be a real directory, not a symlink, because `find` does not
|
|
descend into a symlinked start point. /learn's checks read the one project folder named for the work dir, and
|
|
/office-hours' no-code check lists untracked files against `.git/info/exclude` alone, so a `.gitignore` the agent
|
|
writes hides nothing.
|
|
- **`resumable`** (optional, in `verdict.json` and any error record): `true` when an INCOMPLETE run can be finished
|
|
with `--resume <out>`, which runs only cells that never ran, under the same lock and trial count, so it adds no
|
|
trials after a result; `false` for a terminal stop, which needs a fix and a fresh run. A missing field reads as
|
|
`false`.
|
|
- **Lock.** A paid run needs `tasks.lock.json` committed beside the tasks file; `/write-skill`'s diet step 6 runs
|
|
`--lock` and commits it.
|
|
- **Sidecar.** Extensions live in `tasks.ext.json` beside the tasks file, keyed by task id, so this format stays
|
|
unchanged. gstack's free test checks only that each entry names a task in `tasks.json` and uses known keys.
|
|
- **Interactive tasks.** /office-hours' `must` patterns grade a headless run that ends where the skill stops for
|
|
input, as upstream skill-diet's runs do. On a host that answers questions from the sidecar's `user` replies, the final answer is
|
|
the session's last message, so the sidecar's `judge` line carries the behavior there. Open for Lane A: grade
|
|
`must` over the run's assistant messages on such hosts, or keep it on the final answer and rely on `judge`.
|