diff --git a/eval-plan/SKILL.md b/eval-plan/SKILL.md index af3649093..c86726696 100644 --- a/eval-plan/SKILL.md +++ b/eval-plan/SKILL.md @@ -185,13 +185,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit `#` (for example `claude-opus-5-5#no-gstack`) is priced by its model and gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`. -Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus -at most one named low-cost primary. Haiku and Sonnet are not models under test: they -are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say) -belongs only in a preregistration that states it is asking about that tier. A result -measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict. -Judges, cross-family review voices and a harness's shipped housekeeping model are not -models under test, and the rule leaves them alone +Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named +low-cost primary); a Haiku or Sonnet arm only with a stated tier question (docs/TESTING_INTERNALS.md#frontier-models-under-test). When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families @@ -248,10 +243,7 @@ the budget the run will use. The checklist printed must be all `ok`: with no text is a repeat of the same failure at the same price. - `CHECK: reservation-fits-cap` (with `--cap `) — one call, the largest row plus the output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold. -- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one, - `warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put - that number in the budget before you swap the arms. `--tier-question ""` passes only when the preregistration states that question. +- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question ""`), `warn` if some; re-costs on frontier prices. - `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt. `DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on diff --git a/eval-plan/SKILL.md.tmpl b/eval-plan/SKILL.md.tmpl index daa4bf73c..e33a79590 100644 --- a/eval-plan/SKILL.md.tmpl +++ b/eval-plan/SKILL.md.tmpl @@ -107,13 +107,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit `#` (for example `claude-opus-5-5#no-gstack`) is priced by its model and gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`. -Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus -at most one named low-cost primary. Haiku and Sonnet are not models under test: they -are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say) -belongs only in a preregistration that states it is asking about that tier. A result -measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict. -Judges, cross-family review voices and a harness's shipped housekeeping model are not -models under test, and the rule leaves them alone +Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named +low-cost primary); a Haiku or Sonnet arm only with a stated tier question (docs/TESTING_INTERNALS.md#frontier-models-under-test). When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families @@ -170,10 +165,7 @@ the budget the run will use. The checklist printed must be all `ok`: with no text is a repeat of the same failure at the same price. - `CHECK: reservation-fits-cap` (with `--cap `) — one call, the largest row plus the output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold. -- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one, - `warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put - that number in the budget before you swap the arms. `--tier-question ""` passes only when the preregistration states that question. +- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question ""`), `warn` if some; re-costs on frontier prices. - `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt. `DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on diff --git a/test/fixtures/context-budget.json b/test/fixtures/context-budget.json index ba51a7dba..6aa0d150f 100644 --- a/test/fixtures/context-budget.json +++ b/test/fixtures/context-budget.json @@ -25,7 +25,7 @@ "diagram": 4286, "document-generate": 6013, "document-release": 4454, - "eval-plan": 4887, + "eval-plan": 5458, "freeze": 1115, "gstack": 5350, "gstack-upgrade": 6297,