diff --git a/CHANGELOG.md b/CHANGELOG.md index 4b55be44c..60f071b00 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ ## [Unreleased] -**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON, and false passes are counted against `old-a` alone, so the result counts against the new request only. +**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (both valid `old-a` a07 samples also scored 4/4/4), and false passes are counted against `old-a` alone, so the result counts against the new request only. ### Tests - `test/fixtures/judge-calibration/cookie-workflow/`: 24 items. p01 is the production bundle (`buildCookieWorkflowJudgeInput`); a02 and a05 are real earlier revisions; the rest are hand-edited variants rendered by the same builder. Labels were preregistered and committed before any request with the schema was sent. A separate run (openai/gpt-6.1-sol) then labeled every item blind, and a third reviewer (google/gemini-4-argon) adjudicated the two disagreements: a01 kept pass, a08 changed to fail. `runs/` holds every sample and the result. The cost was 216 calls and $2.81, priced in `manifest.json` before dispatch as a separately funded entry. diff --git a/TODOS.md b/TODOS.md index cbc1d45d7..ed7010554 100644 --- a/TODOS.md +++ b/TODOS.md @@ -784,7 +784,8 @@ Each item was deferred with a reason during the `/claude-api prompt-audit` clean (landed, but one provider refusal on corpus item f03 under the new prompt, matched by one under the old prompt) and the cookie workflow judge (GSTA-91: held on one new false pass on a07, zero new-request format errors, so it stays - on today's request). Reuse `test/fixtures/judge-calibration` + on today's request; today's request itself false-passes a07 in `old-b` and on + both valid `old-a` samples, and a08 in `old-a`, so the prose step targets those). Reuse `test/fixtures/judge-calibration` and `scripts/judge-calibration.ts`. Priority P3. Effort M. - **Uncalibrated workflow-judge configurations** — build a corpus and calibrate schema transport for the frontier configuration (review/SKILL.md workflow); it diff --git a/test/fixtures/judge-calibration/README.md b/test/fixtures/judge-calibration/README.md index 9ab1a66ea..c8c223cb2 100644 --- a/test/fixtures/judge-calibration/README.md +++ b/test/fixtures/judge-calibration/README.md @@ -11,7 +11,8 @@ Each `/corpus.json` holds 24 items: 8 clearly passing, 8 clearly failing has been sampled, with the same frozen requests. `expected` is the verdict a careful grader would reach under the eval's pass rule (`pass_rule`). The author labeled every item; a separate agent run then labeled every item blind (shuffled ids, no author -labels) before any schema was written. `second_review` records that verdict, and each +labels) before any schema was written (for `cookie-workflow`, before any request with the +schema was sent). `second_review` records that verdict, and each disagreement carries `author_expected` and the third reviewer's `adjudication`. `scripts/judge-calibration.ts` runs the comparison and `scripts/judge-calibration-configs.ts` @@ -21,7 +22,7 @@ and priced usage, and `comparison-1.result.json` holds the flip rates, agreement false passes, error rates, p95 latency, landing decision and the UC3 prose-step trigger. `manifest.json` prices every planned call before dispatch. -## Comparison (1) results (2026-10-03, `claude-fable-5-1`) +## Comparison (1) results (2026-10-03, `claude-fable-5-1`, unless the row says otherwise) Flips are panel-verdict differences on items where both panels completed: old-vs-old is `old-a` vs `old-b`, new-vs-old is `new` vs `old-a`. Agreement is items whose panel @@ -53,7 +54,8 @@ output tokens before the manifest was priced. Total W2 spend: $90.47 of $115. `cookie-workflow` (GSTA-91, separately funded) is held and stays on today's request. Its one new false pass, a07, scored 4/4/4 on all three new samples and on all three `old-b` samples; `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (the defect the -schema transport removes), so the comparison counts it against the new request only. The old +schema transport removes), so the comparison counts it against the new request only. Both +valid `old-a` a07 samples also scored 4/4/4. The old request failed to parse 2 of 144 samples; the new one 0 of 72. The failed landing condition triggers the UC3 prose step, which is deferred to TODOS. Its `manifest.json` entry was priced separately (`priced_at`, `funding`) and is maintained by hand: the `manifest` command rewrites