fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling

The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the
exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining
growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458,
the capture helper's value); other skills' ceilings are left as they were.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-11-01 23:07:45 +00:00
1 parent 8ca567fb65
commit 857c298c56
3 files changed
+7 -23

No files matched your search

+3 -11
View File
@@ -185,13 +185,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus
at most one named low-cost primary. Haiku and Sonnet are not models under test: they
are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say)
belongs only in a preregistration that states it is asking about that tier. A result
measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict.
Judges, cross-family review voices and a harness's shipped housekeeping model are not
models under test, and the rule leaves them alone
Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named
low-cost primary); a Haiku or Sonnet arm only with a stated tier question
(docs/TESTING_INTERNALS.md#frontier-models-under-test).
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
@@ -248,10 +243,7 @@ the budget the run will use. The checklist printed must be all `ok`:
with no text is a repeat of the same failure at the same price.
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one,
`warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put
that number in the budget before you swap the arms. `--tier-question "<what you ask about
that tier>"` passes only when the preregistration states that question.
- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question "<why>"`), `warn` if some; re-costs on frontier prices.
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
+3 -11
View File
@@ -107,13 +107,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus
at most one named low-cost primary. Haiku and Sonnet are not models under test: they
are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say)
belongs only in a preregistration that states it is asking about that tier. A result
measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict.
Judges, cross-family review voices and a harness's shipped housekeeping model are not
models under test, and the rule leaves them alone
Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named
low-cost primary); a Haiku or Sonnet arm only with a stated tier question
(docs/TESTING_INTERNALS.md#frontier-models-under-test).
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
@@ -170,10 +165,7 @@ the budget the run will use. The checklist printed must be all `ok`:
with no text is a repeat of the same failure at the same price.
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one,
`warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put
that number in the budget before you swap the arms. `--tier-question "<what you ask about
that tier>"` passes only when the preregistration states that question.
- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question "<why>"`), `warn` if some; re-costs on frontier prices.
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
+1 -1
View File
@@ -25,7 +25,7 @@
"diagram": 4286,
"document-generate": 6013,
"document-release": 4454,
"eval-plan": 4887,
"eval-plan": 5458,
"freeze": 1115,
"gstack": 5350,
"gstack-upgrade": 6297,