gstack/eval-plan/SKILL.md.tmpl

---
name: eval-plan
preamble-tier: 2
version: 1.0.0
description: |
  Preregister a paid eval (bars, arms, decoys, stop rules), dry-run it at zero cost,
  then run a three-item priced pilot whose spend table gates approval. (gstack)
triggers:
  - plan an eval
  - preregister an eval
  - how much will this eval cost
  - pilot the eval
allowed-tools:
  - Bash
  - Read
  - Write
  - Edit
  - Glob
  - Grep
  - AskUserQuestion
---

{{PREAMBLE}}

# /eval-plan: Preregistered, paid evals

Three steps stand between an eval idea and the first priced call, in this order:
the preregistration (what counts, what stops the run), the zero-cost dry run (the
pipeline works on the real input shapes, with prices in the table), and the
priced pilot (three items per arm, settled against the invoice, extrapolated
with an error band). Only then is the owner asked to approve the full run. Every
step prints one grep-able line; a parent agent reads the lines, not this prose.

Usage: `/eval-plan [<prereg.md>] [--input <rows.jsonl>] [--arms <model,model>] [--cap <usd>] [--total <items>]`

## Boundaries

- Spend nothing before Step 3, and in Step 3 only what the ledger admits. A reservation the
  ledger refuses (`EVAL_BUDGET_EXHAUSTED`) ends the step; never retry to make room, never
  split a call to slip under the cap.
- Preregistered text is frozen once the pilot starts: a bar, arm, decoy or stop rule
  changed after a priced call is a new eval with a new preregistration.
- Held-out sets named in the preregistration are never read by this skill, pasted into a
  prompt, or summarized; the dry run reads the development rows only.
- Admission is an estimate. `lib/pricing.ts` prices standard text tokens and excludes
  cache writes, tool fees, service-tier and long-context multipliers, media tokens and
  seat-metered runs; actual cost reconciles at settle time and `unknown` charges stay
  visible in every spend table.
- Under `SESSION_KIND: unattended`, the approval in Step 4 is written as a gate item and
  the run ends `gate_pending`; this skill never approves a priced run by itself.

## Step 1: Preregistration

Write (or open) the preregistration file. Each section is required and must be filled;
`<!-- -->` guidance does not count as content:

| Section | What it holds |
|---|---|
| Bars | one line per metric: name, pass bar, direction, what a miss means for the decision |
| Arms | one line per arm: the arm identity first (`<model>` or `<model>#<loadout>`), settings, per-item cost from Step 1b |
| Comparison | `stack: loaded` when any arm loads gstack or another skill stack; otherwise `stack: none` and a `why:` line |
| Decoys | inputs the scorer must reject or score low; how many, how they were made, what a decoy pass proves |
| Stop rules | the spend cap, a bar missed on the pilot, a scorer below its bar, `budget_exhausted`, the first quota or rate-limit error on a seat-metered arm |
| Held-out exposure | every held-out set the plan touches: prior exposure (which runs, which models), the reservation on it |
| No-gstack arm | required with `stack: loaded`: `arms:`, `inputs:`, `prompt:`, `panel:`, `report:` (below) |

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --init <path>
~/.claude/skills/gstack/bin/gstack-eval-plan prereg --check <path>
```

`PREREG: ok` continues. `PREREG: incomplete` names the missing or empty sections
(`PREREG_INCOMPLETE`, exit 1); fill them and rerun. Do not continue on an incomplete
preregistration.

**No-gstack arm.** A head-to-head where any arm loads gstack or another skill stack
(review variants, preamble loadouts, skill diets, a model panel under a stack) also
runs each model with no skill stack at all. Add `<model>#no-gstack` to Arms for every
model, and fill `## No-gstack arm`: `arms:` lists those arms; `inputs:` the same rows
and fixture; `prompt:` the task text only, with no SKILL.md, preamble, hooks or installed
skills, and the same tools; `panel:` the same E2E kind, trials, judge and blinding labels;
`report:` its row beside the variants in every results table and verdict.
`PREREG_NO_GSTACK_ARM` (exit 1) names what is missing. The PREREG line shows
`stack=loaded no_gstack_arm=ok` when it is complete, or `stack=none` for an opt-out.

When the summarizer counts diagnostics, declare them before the freeze commit in a
`## Diagnostics` section: `summarizer: <command naming {fixture}>`, `fixture: <synthetic
records file>`, and one `| diagnostic | expected |` row per diagnostic, with the count the
fixture is built to produce. The summarizer's last stdout line is one JSON object of counts.
Then run `prereg --check <path> --allow-repo-commands`. Without the flag, the summarizer is
listed and the check exits 3. Every `PREREG_DIAGNOSTIC:` line must say `ok`.
`PREREG_DIAGNOSTIC_UNTESTED` (exit 1) means a count, a missing diagnostic or an
undeclared one does not match its definition. Fix that before you freeze.

### Step 1b: Arms and their price

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan arms
```

Arms come from the project's model rules (`gstack-models resolve --role eval-arms`),
falling back to the repo's own policy document (`--policy-doc <path>`), or from
`--arms <picked-models>` when the user named them. Each `ARM:` line carries its source
and the per-item cost at the stated token estimate; `price=MISSING` (`EVAL_PRICE_MISSING`)
means the model has no row in `lib/pricing.ts`: add the row from the provider's pricing
page or drop the arm. Copy the priced arms into the Arms section. An arm identity
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.

When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
newest first; `docs/eval-models.md` in gstack), `arms`, `dry-run`, `pilot start` and
`price-gate` print one `EVAL_MODEL:` line per arm. Pass `--scope smoke` for a smoke run;
the default is `full`. A refused arm (`EVAL_MODEL_DENIED`, `EVAL_MODEL_OLD_GENERATION`,
`EVAL_MODEL_GENERATION_UNRESOLVED`) is replaced, never argued past.

### Step 1c: Candidate-side blinding

A candidate that can tell it is being graded behaves differently from one that
thinks it is at work, and a judge that knows which arm it is reading grades the
label. Three rules hold for every arm, written into the preregistration (a
`## Blinding` section naming the fixture root and the judge's labels) and checked
by reading the fixture tree before the freeze commit; `prereg --check` does not
read fixture trees, so this check is yours:

1. **Nothing the candidate sees carries an eval word.** No `eval`, `test`,
   `judge`, `experiment`, `rubric`, `score`, `compare`, `benchmark`,
   `candidate` or `arena` in any directory name, file name, prompt, commit
   message or environment variable the preregistration controls and the candidate
   can read. Two things are outside the rule: a project's own test directory, when
   the task is about those tests, and the skill text an arm loads as its own
   instructions (a gstack SKILL.md uses these words freely); the rule is about the
   fixture and the task, which the eval builds.
2. **The directory looks like a project.** Name the fixture root the way the
   project it imitates would (`billing-service/`, `docs-site/`), with the files
   that project would plausibly hold, never `fixture-a/`, `case-7/` or the
   model's name. A candidate working in `arm-b/` knows there is an `arm-a/`.
3. **The judge sees sanitized labels and the candidate's real reads.** The judge
   gets `A` and `B`, never a model id, arm name, price or run number, and it
   grades whether the candidate followed the chain (read the file the task
   pointed at, then the file that one named) from the files the transcript shows
   it actually opened, never from the candidate's own account of what it read.

A rule broken after a priced call is a new eval with a new preregistration, the
same as a changed bar.

## Step 2: zero-cost stub-model dry run

Run the pipeline over the real development rows with a stub model that costs nothing:

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan dry-run --input <path> --arms <picked-models> --retry-prompt <path> --out <outdir>
```

Add `--fields <id>` when the rows' prompt field is not `prompt`, and `--output-budget` with
the budget the run will use. The checklist printed must be all `ok`:

- `CHECK: price-table` — every arm has a price row.
- `CHECK: output-budget` — at or above the reasoning floor (8,192 tokens); a reasoning model
  spends output tokens thinking, and a smaller budget truncates the answer the scorer reads.
- `CHECK: retry-prompt` — the exact text the run sends on a retry, printed verbatim; a retry
  with no text is a repeat of the same failure at the same price.
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
  output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.

`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
anything else (`EVAL_DRY_RUN_FAILED`).

## Step 3: Priced pilot (n≈3)

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan pilot start --out <outdir> --cap <cap-usd> --arms <picked-models> --dry-run <path>
```

`pilot start` reserves three attempts per arm through the spend ledger (`spend.json` in
`--out`): admission is `spent + reserved + unknown + estimate <= cap`, under a file lock
shared with any concurrent worker. Run each `PILOT_ATTEMPT:` through the project's own
eval command, then settle it with the actual cost from the provider's usage field, or
`unknown` when the call was seat-metered or CLI-metered.

An arm on a subscription with no extra charge (say `claude -p` on a seat) is declared at
start with `--seat-metered <arm,...>`, each also in `--arms`. Its attempts hold none of the
cap, settle as `unknown` and get a `PILOT_SEAT_METERED:` line instead of a projection; an
arm that settles a real charge is projected as priced. The first quota or rate-limit error
on such an arm ends the run: no retry, no new attempt; release the unstarted ones. Settle
each attempt, then report:

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan pilot settle --out <outdir> --attempt <attempt-id> --usd <usd|unknown>
~/.claude/skills/gstack/bin/gstack-eval-plan pilot report --out <outdir> --total <total-items> --workers <workers>
```

`pilot report` prints the spend table (`spent`, `reserved`, `unknown charges`, the worst case
with one in-flight call per worker, the worst-case overrun against the cap) and
one `PILOT_ARM:` line per money arm (its own mean, deviation, projection and band), then
`PILOT_PROJECTION:` — the arms summed: cost per item, deviation, the projection to
`--total` items, and a 95% band. Regenerate the table at every later gate
of the run. Stop and report, without asking for approval, when the pilot missed a bar from
the preregistration, when `status=budget_exhausted`, or when `verdict=exceeds`.

## Step 4: The approval ask

Price every paid gate the plan names (a benchmark or eval command that spends) from the
repo's call-plan file; nothing is executed (`docs/eval-calls.md` in gstack). The gate
command is free text, so it reaches the pricer only as a file:

{{FREE_TEXT_FILE:GATE_FILE=gate-command}}

Write the exact gate command, one line, into the printed file, then price it:

```bash
GATE_FILE="$(git rev-parse --show-toplevel 2>/dev/null || pwd)/.gstack/tmp/<gate-file-name>"
~/.claude/skills/gstack/bin/gstack-eval-plan price-gate --gate-file "$GATE_FILE" --cap-usd <cap-usd>
```

Each `GATE_PRICE:` line goes into the ask. `EVAL_GATE_UNPRICED`, `EVAL_PLAN_STALE`,
`EVAL_PRICE_MISSING` or `EVAL_GATE_OVER_CAP` means that gate is not approvable as priced;
`--allow-unpriced "<reason>"` only on the owner's word.

Present, in this order and nothing before it: the `PILOT_PROJECTION:` line, the spend
table, each gate's `GATE_PRICE:` line, the pilot's per-bar results, and the preregistration's stop rules. Then ask one
question with the projection as the recommended option only when `verdict=within`:

- A) Run `<total-items>` items at the projected cost (recommended when within the cap)
- B) Cut arms or items; recompute the projection first
- C) Stop here; keep the pilot results as the record

Under `SESSION_KIND: unattended` write the question as an `approval` gate item through
`gstack-gate` and end with `GSTACK_RESULT: skill=eval-plan status=gate_pending run=<outdir>`.
The full run, when approved, keeps settling every attempt into the same ledger and reprints
the spend table at each gate; `budget_exhausted` stops it.

## Step 5: Results beside the no-gstack arm

Write the run's results as one markdown table in a results file, one row per arm identity
from the Arms section, each `<model>#no-gstack` row beside its variants. Before any number
leaves the run (a report, a READY block, a PR body), check it:

```bash
~/.claude/skills/gstack/bin/gstack-eval-plan results --check <prereg-path> <results-path>
```

`RESULTS: ok` lets the numbers go out. `EVAL_REPORT_NO_GSTACK_ROW` (exit 1) names the arm
whose filled row is missing from the table that holds the others; add it and rerun. A
preregistration with `stack: none` needs no row.

## Tier 3, after P7 (not in this skill yet)

Each of these waits for the adoption number in the multi-agent wave plan (P7); one line each so the shape is known:

- Instrument audit: read every failing transcript of one dev round before freezing the scorer; check generated worlds for duplicate identifiers, word-like codes and generic phrases; mutation tests in both directions. (tier 3, after P7)
- Steady state: background queues empty at snapshot time, queue depth recorded as a gate. (tier 3, after P7)
- Sealed paths: declared in the plan; the skill refuses to quote from them, self-checks output for their n-grams and adds a provenance line. (tier 3, after P7)
- Off-machine persistence: long runs sync through `gstack-detach --sync-cmd <cmd> --sync-every <min>`. (tier 3, after P7)
- Memory preflight: concurrency times measured per-process memory, refuse above 70% of RAM. (tier 3, after P7)
- Power tables: power and risk sentences cite the committed power output with a sensitivity table over the obvious rule alternatives. (tier 3, after P7)
- QA soak: hours of mixed ingest against a long-lived server with a `CPU pinned, no progress` watchdog. (tier 3, after P7)