mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-16 13:21:30 +01:00
Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754) (#4384)
* Make /eval-plan check judges, zeros, failure counts and decisions before results count (GST-2754) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Check zeros in every results table, apply Decisions and Scoring with stack: none, count only valid judge labels (GST-2754) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Describe the /eval-plan result checks in the skills guide (GST-2754) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
2c0e07886b
commit
fdf3f1e1b6
21 files changed
+643
-75
No files matched your search
@@ -2059,6 +2059,37 @@ it as the diagnostic. Candidate folders and git identity name a project, never
|
||||
the arm. The model panel (`test/skill-e2e-model-panel.test.ts`) runs a
|
||||
`<model>#no-gstack` arm beside each model (GST-468, GST-469).
|
||||
|
||||
## Result checks
|
||||
|
||||
`/eval-plan` runs four checks so that a number which cannot change a decision, or
|
||||
which comes from an unchecked instrument, never leaves the run. Each one is a
|
||||
refusal from `bin/gstack-eval-plan` (logic in `lib/eval-plan-checks.ts` and
|
||||
`lib/eval-plan-report.ts`), not advice:
|
||||
|
||||
| Check | When it runs | Refusal |
|
||||
|---|---|---|
|
||||
| `## Decisions`: one `if <result>: <what we do>` line per possible result, at least two, with different actions | `prereg --check`, before anything is spent | `PREREG_INCOMPLETE` |
|
||||
| `## Scoring`: `judge: none`, or a judge model from another family than every arm with `labels:` (40 or more) and `agreement:` (kappa 0.6 or more); with two or more arms, `paired: yes`, `interval: 95%` and `mdd:` stated before the run | `prereg --check` | `PREREG_INCOMPLETE` |
|
||||
| The judge's Cohen's kappa against `{"id", "human", "judge"}` hand labels, at the preregistered count and bar | `judge --check` before the approval ask; `results --check --labels` before any number goes out | `EVAL_JUDGE_UNCHECKED` |
|
||||
| A zero in any results table comes from a column named after a `## Diagnostics` counter whose synthetic fixture plants at least one case (expected 1 or more) | `results --check` | `EVAL_REPORT_UNPLANTED_ZERO` |
|
||||
| A results table with a failures column also has an `answer-only` column: failures by the grader's own rule on the final answer | `results --check` | `EVAL_REPORT_NO_ANSWER_ONLY` |
|
||||
|
||||
Why: in our own agent teams, a counter that never fired reported zero for one
|
||||
model while an outside check with planted cases found every one, and a reported
|
||||
rise in failures vanished once failures were counted by the grader's own rule.
|
||||
A judge from the same family as the arm it grades, or one never compared with
|
||||
people, gives numbers nobody can weigh. An eval whose every result leads to the
|
||||
same action spends money to change nothing.
|
||||
|
||||
Model families come from the model id (`claude` is Anthropic, `gpt` and `o`
|
||||
models OpenAI, `gemini` Google, `grok` xAI), else its first word. A judge
|
||||
whose labels are all one class has undefined agreement and is refused.
|
||||
|
||||
A preregistration with a `frozen: YYYY-MM-DD` line before 2026-11-15 keeps the
|
||||
rules it was frozen under: the PREREG line shows `result_checks=exempt` and
|
||||
none of the four checks apply. Text frozen before a priced call is not edited
|
||||
to meet a newer rule.
|
||||
|
||||
## Frontier models under test
|
||||
|
||||
Garry, 2026-11-01 (R54): "I think you are spending too much time with Haiku and
|
||||
|
||||
+7
-2
@@ -966,7 +966,10 @@ You: /eval-plan how much will the retrieval eval cost before I run it?
|
||||
|
||||
`bin/gstack-eval-plan prereg --init` writes the preregistration (question, bars,
|
||||
arms, decoys, stop rules, cap) and `--check` refuses to continue while a field is
|
||||
blank (`PREREG_INCOMPLETE`). `arms` resolves the `eval-arms` model role against
|
||||
blank (`PREREG_INCOMPLETE`), including the `## Decisions` and `## Scoring`
|
||||
sections the [result checks](TESTING_INTERNALS.md#result-checks) require.
|
||||
`judge --check <prereg> --labels <labels.jsonl>` scores a model judge against hand
|
||||
labels (`EVAL_JUDGE_UNCHECKED`). `arms` resolves the `eval-arms` model role against
|
||||
`lib/pricing.ts` and stops on an unpriced model (`EVAL_PRICE_MISSING`); `dry-run`
|
||||
runs every row through the stub model for $0 and flags malformed rows
|
||||
(`EVAL_DRY_RUN_FAILED`), and refuses a plan whose every arm is Haiku or Sonnet
|
||||
@@ -981,7 +984,9 @@ stays out of the projection until one of its attempts settles a real charge.
|
||||
A run stops at the first quota or rate-limit error on such an arm.
|
||||
`results --check <prereg> <results.md>` refuses a finished results file unless one
|
||||
table holds a filled row for every arm, each `<model>#no-gstack` row beside its
|
||||
variants (`EVAL_REPORT_NO_GSTACK_ROW`); no number leaves the run before it passes.
|
||||
variants (`EVAL_REPORT_NO_GSTACK_ROW`); it also checks a model judge's `--labels`,
|
||||
unplanted zeros (`EVAL_REPORT_UNPLANTED_ZERO`) and failures columns without an
|
||||
`answer-only` column (`EVAL_REPORT_NO_ANSWER_ONLY`); no number leaves the run before it passes.
|
||||
The skill's closing section lists the tier-3 items (instrument audit, steady
|
||||
state, sealed paths, off-machine persistence, memory preflight, power tables, QA
|
||||
soak) as `tier 3, after P7`; none of them run yet.
|
||||
|
||||
+58
-6
@@ -4763,12 +4763,19 @@ parentheses; grep the code.
|
||||
### `PREREG_INCOMPLETE`
|
||||
|
||||
**Meaning.** `gstack-eval-plan prereg --check <file>` found a required
|
||||
preregistration section missing or empty: `## Bars`, `## Arms`,
|
||||
`## Comparison`, `## Decoys`, `## Stop rules` or `## Held-out exposure`
|
||||
(every held-out set the plan touches, its prior exposure and the reservation
|
||||
on it). It also fires when `## Comparison` has no `stack:` line, a value other
|
||||
than `loaded` or `none`, or `stack: none` without a `why:` line, and when a
|
||||
`## Diagnostics` declaration is malformed.
|
||||
preregistration section missing or empty: `## Bars`, `## Decisions`,
|
||||
`## Arms`, `## Comparison`, `## Scoring`, `## Decoys`, `## Stop rules` or
|
||||
`## Held-out exposure` (every held-out set the plan touches, its prior
|
||||
exposure and the reservation on it). It also fires when `## Comparison` has no
|
||||
`stack:` line, a value other than `loaded` or `none`, or `stack: none` without
|
||||
a `why:` line, and when a `## Diagnostics` declaration is malformed. The
|
||||
result checks add two more (`decisions:` and `scoring:` in the message):
|
||||
`## Decisions` needs two or more `if <result>: <action>` lines with different
|
||||
actions, and `## Scoring` needs a `judge:` line, a model judge from another
|
||||
model family than every arm with `labels:` of 40 or more and `agreement:` of
|
||||
0.6 or more, and with two or more arms `paired: yes`, `interval: 95%` and
|
||||
`mdd:`. A `frozen:` date before 2026-11-15 exempts both
|
||||
([docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks)).
|
||||
|
||||
**Fix.** Fill the named sections before any priced call. A plan that cannot
|
||||
name its stop rule or its held-out exposure is not preregistered; the pilot
|
||||
@@ -4808,6 +4815,51 @@ in one table, and rerun the check. Publish no number before it prints
|
||||
`RESULTS: ok`. A preregistration with `stack: none` needs no row. The rule is
|
||||
in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#no-gstack-arm).
|
||||
|
||||
<a id="eval-judge-unchecked"></a>
|
||||
### `EVAL_JUDGE_UNCHECKED`
|
||||
|
||||
**Meaning.** The preregistration's `## Scoring` names a model judge, and
|
||||
`gstack-eval-plan judge --check` or `results --check` found no `--labels`
|
||||
file, fewer labelled outputs than the preregistered count (at least 40), labels
|
||||
that are all one class (agreement is undefined), a repeated id, or a Cohen's
|
||||
kappa between the hand labels and the judge below the preregistered bar (at
|
||||
least 0.6). Until it passes, the judge's numbers do not count.
|
||||
|
||||
**Fix.** Label outputs by hand from development or calibration rows, never
|
||||
held-out ones, as JSONL rows `{"id", "human", "judge"}`, covering every kind
|
||||
of output the judge scores, and pass the file with `--labels`. A judge below
|
||||
its bar is replaced or its prompt fixed and relabelled; the bar is not
|
||||
lowered. The rule is in
|
||||
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
|
||||
|
||||
<a id="eval-report-unplanted-zero"></a>
|
||||
### `EVAL_REPORT_UNPLANTED_ZERO`
|
||||
|
||||
**Meaning.** `gstack-eval-plan results --check` found a zero (`0`, `0%`,
|
||||
`0/N`) in a column of any results table, and the preregistration's
|
||||
`## Diagnostics` declares no counter by that column's name with a planted
|
||||
case (`expected` of 1 or more). A counter that never fires also reports zero,
|
||||
so a zero from a counter nobody saw find anything proves nothing.
|
||||
|
||||
**Fix.** Declare the counter under `## Diagnostics` with a synthetic fixture
|
||||
that plants at least one case, run `prereg --check --allow-repo-commands`
|
||||
until its `PREREG_DIAGNOSTIC:` line says ok, and name the results column after
|
||||
the diagnostic. Otherwise leave the zero out of the report. The rule is in
|
||||
[docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
|
||||
|
||||
<a id="eval-report-no-answer-only"></a>
|
||||
### `EVAL_REPORT_NO_ANSWER_ONLY`
|
||||
|
||||
**Meaning.** A table in the results file has a column counting failures and no
|
||||
column headed `answer-only` beside it. A broad failure count (from
|
||||
transcripts, tool errors or a second rule) can rise while failures counted by
|
||||
the grader's own rule on the final answer do not move, and the two readings
|
||||
must be read side by side.
|
||||
|
||||
**Fix.** Add an `answer-only` column to the same table, counting failures by
|
||||
the grader's rule on the final answer alone, and report both readings. The
|
||||
rule is in [docs/TESTING_INTERNALS.md](TESTING_INTERNALS.md#result-checks).
|
||||
|
||||
<a id="eval-arm-duplicate"></a>
|
||||
### `EVAL_ARM_DUPLICATE`
|
||||
|
||||
|
||||
Reference in new issue
Block a user