mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-03 15:27:56 +01:00
fix(eval-plan): trim R54 skill prose and raise the eval-plan context ceiling
The R54 paragraph and CHECK: frontier bullet were cut to the minimum; the exemptions and verdict rule stay in docs/TESTING_INTERNALS.md. The remaining growth is the feature, so only the eval-plan eager ceiling moves (4887 -> 5458, the capture helper's value); other skills' ceilings are left as they were. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
8ca567fb65
commit
857c298c56
3 files changed
+7
-23
No files matched your search
+3
-11
@@ -185,13 +185,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit
|
||||
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
|
||||
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
|
||||
|
||||
Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus
|
||||
at most one named low-cost primary. Haiku and Sonnet are not models under test: they
|
||||
are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say)
|
||||
belongs only in a preregistration that states it is asking about that tier. A result
|
||||
measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict.
|
||||
Judges, cross-family review voices and a harness's shipped housekeeping model are not
|
||||
models under test, and the rule leaves them alone
|
||||
Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named
|
||||
low-cost primary); a Haiku or Sonnet arm only with a stated tier question
|
||||
(docs/TESTING_INTERNALS.md#frontier-models-under-test).
|
||||
|
||||
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
|
||||
@@ -248,10 +243,7 @@ the budget the run will use. The checklist printed must be all `ok`:
|
||||
with no text is a repeat of the same failure at the same price.
|
||||
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
|
||||
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
|
||||
- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one,
|
||||
`warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put
|
||||
that number in the budget before you swap the arms. `--tier-question "<what you ask about
|
||||
that tier>"` passes only when the preregistration states that question.
|
||||
- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question "<why>"`), `warn` if some; re-costs on frontier prices.
|
||||
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
|
||||
|
||||
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
|
||||
|
||||
+3
-11
@@ -107,13 +107,8 @@ page or drop the arm. Copy the priced arms into the Arms section. An arm identit
|
||||
`<model>#<loadout>` (for example `claude-opus-5-5#no-gstack`) is priced by its model and
|
||||
gets its own pilot attempts; naming one identity twice is `EVAL_ARM_DUPLICATE`.
|
||||
|
||||
Models under test are frontier models: today `claude-opus-5-5` and `gpt-6.1-sol`, plus
|
||||
at most one named low-cost primary. Haiku and Sonnet are not models under test: they
|
||||
are not stand-ins for a frontier model, and a tier-transfer arm (Haiku to Sonnet, say)
|
||||
belongs only in a preregistration that states it is asking about that tier. A result
|
||||
measured only on Haiku or Sonnet carries no KEEP, FIX-METHOD or CUT-CANDIDATE verdict.
|
||||
Judges, cross-family review voices and a harness's shipped housekeeping model are not
|
||||
models under test, and the rule leaves them alone
|
||||
Models under test are frontier models (`claude-opus-5-5`, `gpt-6.1-sol`, at most one named
|
||||
low-cost primary); a Haiku or Sonnet arm only with a stated tier question
|
||||
(docs/TESTING_INTERNALS.md#frontier-models-under-test).
|
||||
|
||||
When the repo commits `.gstack/eval-models.json` (allow, deny, per-scope lists, families
|
||||
@@ -170,10 +165,7 @@ the budget the run will use. The checklist printed must be all `ok`:
|
||||
with no text is a repeat of the same failure at the same price.
|
||||
- `CHECK: reservation-fits-cap` (with `--cap <cap-usd>`) — one call, the largest row plus the
|
||||
output budget, priced, fits under the cap; a provider refuses a reservation the cap cannot hold.
|
||||
- `CHECK: frontier` (printed when an arm is Haiku or Sonnet) — `refused` when every arm is one,
|
||||
`warn` when some are. Either way it prices the whole plan on Opus 5.5 and GPT-6.1 Sol: put
|
||||
that number in the budget before you swap the arms. `--tier-question "<what you ask about
|
||||
that tier>"` passes only when the preregistration states that question.
|
||||
- `CHECK: frontier` — Haiku/Sonnet arms: `refused` if all (pass `--tier-question "<why>"`), `warn` if some; re-costs on frontier prices.
|
||||
- `DRY_RUN_ROW:` lines name rows missing a field; fix the rows, not the prompt.
|
||||
|
||||
`DRY_RUN_RESULT: pass` writes `dry-run.json` to `--out`; the pilot refuses to start on
|
||||
|
||||
Vendored
+1
-1
@@ -25,7 +25,7 @@
|
||||
"diagram": 4286,
|
||||
"document-generate": 6013,
|
||||
"document-release": 4454,
|
||||
"eval-plan": 4887,
|
||||
"eval-plan": 5458,
|
||||
"freeze": 1115,
|
||||
"gstack": 5350,
|
||||
"gstack-upgrade": 6297,
|
||||
|
||||
Reference in new issue
Block a user