Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754) (#4384)

* Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Check zeros in every results table, apply Decisions and Scoring with stack: none, count only valid judge labels (GST-2754)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* Describe the /eval-plan result checks in the skills guide (GST-2754)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-16 02:55:19 +00:00
1 parent 2c0e07886b
commit fdf3f1e1b6
21 files changed
+643 -75

No files matched your search

+31
View File
@@ -2059,6 +2059,37 @@ it as the diagnostic. Candidate folders and git identity name a project, never
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
## Result checks
`/eval-plan` runs four checks so that a number which cannot change a decision, or
which comes from an unchecked instrument, never leaves the run. Each one is a
refusal from `bin/gstack-eval-plan` (logic in `lib/eval-plan-checks.ts` and
`lib/eval-plan-report.ts`), not advice:
| Check | When it runs | Refusal |
|---|---|---|
| `## Decisions`: one `if <result>: <what we do>` line per possible result, at least two, with different actions | `prereg --check`, before anything is spent | `PREREG_INCOMPLETE` |
| `## Scoring`: `judge: none`, or a judge model from another family than every arm with `labels:` (40 or more) and `agreement:` (kappa 0.6 or more); with two or more arms, `paired: yes`, `interval: 95%` and `mdd:` stated before the run | `prereg --check` | `PREREG_INCOMPLETE` |
| The judge's Cohen's kappa against `{"id", "human", "judge"}` hand labels, at the preregistered count and bar | `judge --check` before the approval ask; `results --check --labels` before any number goes out | `EVAL_JUDGE_UNCHECKED` |
| A zero in any results table comes from a column named after a `## Diagnostics` counter whose synthetic fixture plants at least one case (expected 1 or more) | `results --check` | `EVAL_REPORT_UNPLANTED_ZERO` |
| A results table with a failures column also has an `answer-only` column: failures by the grader's own rule on the final answer | `results --check` | `EVAL_REPORT_NO_ANSWER_ONLY` |
Why: in our own agent teams, a counter that never fired reported zero for one
model while an outside check with planted cases found every one, and a reported
rise in failures vanished once failures were counted by the grader's own rule.
A judge from the same family as the arm it grades, or one never compared with
people, gives numbers nobody can weigh. An eval whose every result leads to the
same action spends money to change nothing.
Model families come from the model id (`claude` is Anthropic, `gpt` and `o`
models OpenAI, `gemini` Google, `grok` xAI), else its first word. A judge
whose labels are all one class has undefined agreement and is refused.
A preregistration with a `frozen: YYYY-MM-DD` line before 2026-11-15 keeps the
rules it was frozen under: the PREREG line shows `result_checks=exempt` and
none of the four checks apply. Text frozen before a priced call is not edited
to meet a newer rule.
## Frontier models under test
Garry, 2026-11-01 (R54): "I think you are spending too much time with Haiku and
+7 -2
View File
@@ -966,7 +966,10 @@ You: /eval-plan how much will the retrieval eval cost before I run it?
`bin/gstack-eval-plan prereg --init` writes the preregistration (question, bars,
arms, decoys, stop rules, cap) and `--check` refuses to continue while a field is
blank (`PREREG_INCOMPLETE`). `arms` resolves the `eval-arms` model role against
blank (`PREREG_INCOMPLETE`), including the `## Decisions` and `## Scoring`
sections the [result checks](TESTING_INTERNALS.md#result-checks) require.
`judge --check <prereg> --labels <labels.jsonl>` scores a model judge against hand
labels (`EVAL_JUDGE_UNCHECKED`). `arms` resolves the `eval-arms` model role against
`lib/pricing.ts` and stops on an unpriced model (`EVAL_PRICE_MISSING`); `dry-run`
runs every row through the stub model for $0 and flags malformed rows
(`EVAL_DRY_RUN_FAILED`), and refuses a plan whose every arm is Haiku or Sonnet
@@ -981,7 +984,9 @@ stays out of the projection until one of its attempts settles a real charge.
A run stops at the first quota or rate-limit error on such an arm.
`results --check <prereg> <results.md>` refuses a finished results file unless one
table holds a filled row for every arm, each `<model>#no-gstack` row beside its
variants (`EVAL_REPORT_NO_GSTACK_ROW`); no number leaves the run before it passes.
variants (`EVAL_REPORT_NO_GSTACK_ROW`); it also checks a model judge's `--labels`,
unplanted zeros (`EVAL_REPORT_UNPLANTED_ZERO`) and failures columns without an
`answer-only` column (`EVAL_REPORT_NO_ANSWER_ONLY`); no number leaves the run before it passes.
The skill's closing section lists the tier-3 items (instrument audit, steady
state, sealed paths, off-machine persistence, memory preflight, power tables, QA
soak) as `tier 3, after P7`; none of them run yet.
+58 -6
View File
@@ -4763,12 +4763,19 @@ parentheses; grep the code.
### `PREREG_INCOMPLETE`
**Meaning.** `gstack-eval-plan prereg --check <file>` found a required
preregistration section missing or empty: `## Bars`, `## Arms`,
`## Comparison`, `## Decoys`, `## Stop rules` or `## Held-out exposure`
(every held-out set the plan touches, its prior exposure and the reservation
on it). It also fires when `## Comparison` has no `stack:` line, a value other
than `loaded` or `none`, or `stack: none` without a `why:` line, and when a
`## Diagnostics` declaration is malformed.
preregistration section missing or empty: `## Bars`, `## Decisions`,
`## Arms`, `## Comparison`, `## Scoring`, `## Decoys`, `## Stop rules` or
`## Held-out exposure` (every held-out set the plan touches, its prior
exposure and the reservation on it). It also fires when `## Comparison` has no
`stack:` line, a value other than `loaded` or `none`, or `stack: none` without
a `why:` line, and when a `## Diagnostics` declaration is malformed. The
result checks add two more (`decisions:` and `scoring:` in the message):
`## Decisions` needs two or more `if <result>: <action>` lines with different
actions, and `## Scoring` needs a `judge:` line, a model judge from another
model family than every arm with `labels:` of 40 or more and `agreement:` of
0.6 or more, and with two or more arms `paired: yes`, `interval: 95%` and
`mdd:`. A `frozen:` date before 2026-11-15 exempts both
([docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks)).
**Fix.** Fill the named sections before any priced call. A plan that cannot
name its stop rule or its held-out exposure is not preregistered; the pilot
@@ -4808,6 +4815,51 @@ in one table, and rerun the check. Publish no number before it prints
`RESULTS: ok`. A preregistration with `stack: none` needs no row. The rule is
in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
<a id="eval-judge-unchecked"></a>
### `EVAL_JUDGE_UNCHECKED`
**Meaning.** The preregistration's `## Scoring` names a model judge, and
`gstack-eval-plan judge --check` or `results --check` found no `--labels`
file, fewer labelled outputs than the preregistered count (at least 40), labels
that are all one class (agreement is undefined), a repeated id, or a Cohen's
kappa between the hand labels and the judge below the preregistered bar (at
least 0.6). Until it passes, the judge's numbers do not count.
**Fix.** Label outputs by hand from development or calibration rows, never
held-out ones, as JSONL rows `{"id", "human", "judge"}`, covering every kind
of output the judge scores, and pass the file with `--labels`. A judge below
its bar is replaced or its prompt fixed and relabelled; the bar is not
lowered. The rule is in
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
<a id="eval-report-unplanted-zero"></a>
### `EVAL_REPORT_UNPLANTED_ZERO`
**Meaning.** `gstack-eval-plan results --check` found a zero (`0`, `0%`,
`0/N`) in a column of any results table, and the preregistration's
`## Diagnostics` declares no counter by that column's name with a planted
case (`expected` of 1 or more). A counter that never fires also reports zero,
so a zero from a counter nobody saw find anything proves nothing.
**Fix.** Declare the counter under `## Diagnostics` with a synthetic fixture
that plants at least one case, run `prereg --check --allow-repo-commands`
until its `PREREG_DIAGNOSTIC:` line says ok, and name the results column after
the diagnostic. Otherwise leave the zero out of the report. The rule is in
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
<a id="eval-report-no-answer-only"></a>
### `EVAL_REPORT_NO_ANSWER_ONLY`
**Meaning.** A table in the results file has a column counting failures and no
column headed `answer-only` beside it. A broad failure count (from
transcripts, tool errors or a second rule) can rise while failures counted by
the grader's own rule on the final answer do not move, and the two readings
must be read side by side.
**Fix.** Add an `answer-only` column to the same table, counting failures by
the grader's rule on the final answer alone, and report both readings. The
rule is in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
<a id="eval-arm-duplicate"></a>
### `EVAL_ARM_DUPLICATE`