fix: pre-landing review fixes, pass 2 (GSTA-91)

/ship Step 9 Fix-First, second pass:
- README: the blind-review sentence and results heading now fit the cookie corpus
  (labels predate any request with the schema; the row carries its own date and
  model), and the cookie paragraph records that both valid old-a a07 samples
  also scored 4/4/4.
- CHANGELOG: same a07 fact.
- TODOS: today's cookie request false-passes a07 and a08; the deferred prose step
  targets those.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-26 00:58:23 +00:00
1 parent 38955634b7
commit 171d7a92a2
3 files changed
+8 -5

No files matched your search

+1 -1
View File
@@ -2,7 +2,7 @@
## [Unreleased]
**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON, and false passes are counted against `old-a` alone, so the result counts against the new request only.
**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (both valid `old-a` a07 samples also scored 4/4/4), and false passes are counted against `old-a` alone, so the result counts against the new request only.
### Tests
- `test/fixtures/judge-calibration/cookie-workflow/`: 24 items. p01 is the production bundle (`buildCookieWorkflowJudgeInput`); a02 and a05 are real earlier revisions; the rest are hand-edited variants rendered by the same builder. Labels were preregistered and committed before any request with the schema was sent. A separate run (openai/gpt-6.1-sol) then labeled every item blind, and a third reviewer (google/gemini-4-argon) adjudicated the two disagreements: a01 kept pass, a08 changed to fail. `runs/` holds every sample and the result. The cost was 216 calls and $2.81, priced in `manifest.json` before dispatch as a separately funded entry.
+2 -1
View File
@@ -784,7 +784,8 @@ Each item was deferred with a reason during the `/claude-api prompt-audit` clean
(landed, but one provider refusal on corpus item f03 under the new prompt,
matched by one under the old prompt) and the cookie workflow judge (GSTA-91:
held on one new false pass on a07, zero new-request format errors, so it stays
on today's request). Reuse `test/fixtures/judge-calibration`
on today's request; today's request itself false-passes a07 in `old-b` and on
both valid `old-a` samples, and a08 in `old-a`, so the prose step targets those). Reuse `test/fixtures/judge-calibration`
and `scripts/judge-calibration.ts`. Priority P3. Effort M.
- **Uncalibrated workflow-judge configurations** — build a corpus and calibrate
schema transport for the frontier configuration (review/SKILL.md workflow); it
+5 -3
View File
@@ -11,7 +11,8 @@ Each `<config>/corpus.json` holds 24 items: 8 clearly passing, 8 clearly failing
has been sampled, with the same frozen requests. `expected` is the verdict a careful
grader would reach under the eval's pass rule (`pass_rule`). The author labeled every
item; a separate agent run then labeled every item blind (shuffled ids, no author
labels) before any schema was written. `second_review` records that verdict, and each
labels) before any schema was written (for `cookie-workflow`, before any request with the
schema was sent). `second_review` records that verdict, and each
disagreement carries `author_expected` and the third reviewer's `adjudication`.
`scripts/judge-calibration.ts` runs the comparison and `scripts/judge-calibration-configs.ts`
@@ -21,7 +22,7 @@ and priced usage, and `comparison-1.result.json` holds the flip rates, agreement
false passes, error rates, p95 latency, landing decision and the UC3 prose-step
trigger. `manifest.json` prices every planned call before dispatch.
## Comparison (1) results (2026-10-03, `claude-fable-5-1`)
## Comparison (1) results (2026-10-03, `claude-fable-5-1`, unless the row says otherwise)
Flips are panel-verdict differences on items where both panels completed: old-vs-old
is `old-a` vs `old-b`, new-vs-old is `new` vs `old-a`. Agreement is items whose panel
@@ -53,7 +54,8 @@ output tokens before the manifest was priced. Total W2 spend: $90.47 of $115.
`cookie-workflow` (GSTA-91, separately funded) is held and stays on today's request. Its one
new false pass, a07, scored 4/4/4 on all three new samples and on all three `old-b` samples;
`old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (the defect the
schema transport removes), so the comparison counts it against the new request only. The old
schema transport removes), so the comparison counts it against the new request only. Both
valid `old-a` a07 samples also scored 4/4/4. The old
request failed to parse 2 of 144 samples; the new one 0 of 72. The failed landing condition
triggers the UC3 prose step, which is deferred to TODOS. Its `manifest.json` entry was priced
separately (`priced_at`, `funding`) and is maintained by hand: the `manifest` command rewrites