mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 00:44:07 +01:00
fix: pre-landing review fixes, pass 2 (GSTA-91)
/ship Step 9 Fix-First, second pass: - README: the blind-review sentence and results heading now fit the cookie corpus (labels predate any request with the schema; the row carries its own date and model), and the cookie paragraph records that both valid old-a a07 samples also scored 4/4/4. - CHANGELOG: same a07 fact. - TODOS: today's cookie request false-passes a07 and a08; the deferred prose step targets those. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
38955634b7
commit
171d7a92a2
3 files changed
+8
-5
No files matched your search
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON, and false passes are counted against `old-a` alone, so the result counts against the new request only.
|
||||
**The setup-browser-cookies workflow judge now has a preregistered calibration corpus, and its structured-output calibration is recorded as held, so the judge stays on today's request (GSTA-91).** In train-4 gate 40173759210, the cookie judge (claude-sonnet-5-5, free-text transport) returned JSON with an unescaped quote. The repo's fix for that shape is the JSON-schema transport, and `test/fixtures/judge-calibration/README.md` lets a configuration switch only after a calibration lands. The cookie configuration had no corpus until now. Its comparison-1 run removed the parse errors (old 2 of 144 samples, new 0 of 72) and matched or improved every other measure. It is held on one landing condition: a new false pass on a07. a07 scored 4/4/4 on all three new samples and on all three `old-b` samples. `old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (both valid `old-a` a07 samples also scored 4/4/4), and false passes are counted against `old-a` alone, so the result counts against the new request only.
|
||||
|
||||
### Tests
|
||||
- `test/fixtures/judge-calibration/cookie-workflow/`: 24 items. p01 is the production bundle (`buildCookieWorkflowJudgeInput`); a02 and a05 are real earlier revisions; the rest are hand-edited variants rendered by the same builder. Labels were preregistered and committed before any request with the schema was sent. A separate run (openai/gpt-6.1-sol) then labeled every item blind, and a third reviewer (google/gemini-4-argon) adjudicated the two disagreements: a01 kept pass, a08 changed to fail. `runs/` holds every sample and the result. The cost was 216 calls and $2.81, priced in `manifest.json` before dispatch as a separately funded entry.
|
||||
|
||||
@@ -784,7 +784,8 @@ Each item was deferred with a reason during the `/claude-api prompt-audit` clean
|
||||
(landed, but one provider refusal on corpus item f03 under the new prompt,
|
||||
matched by one under the old prompt) and the cookie workflow judge (GSTA-91:
|
||||
held on one new false pass on a07, zero new-request format errors, so it stays
|
||||
on today's request). Reuse `test/fixtures/judge-calibration`
|
||||
on today's request; today's request itself false-passes a07 in `old-b` and on
|
||||
both valid `old-a` samples, and a08 in `old-a`, so the prose step targets those). Reuse `test/fixtures/judge-calibration`
|
||||
and `scripts/judge-calibration.ts`. Priority P3. Effort M.
|
||||
- **Uncalibrated workflow-judge configurations** — build a corpus and calibrate
|
||||
schema transport for the frontier configuration (review/SKILL.md workflow); it
|
||||
|
||||
+5
-3
@@ -11,7 +11,8 @@ Each `<config>/corpus.json` holds 24 items: 8 clearly passing, 8 clearly failing
|
||||
has been sampled, with the same frozen requests. `expected` is the verdict a careful
|
||||
grader would reach under the eval's pass rule (`pass_rule`). The author labeled every
|
||||
item; a separate agent run then labeled every item blind (shuffled ids, no author
|
||||
labels) before any schema was written. `second_review` records that verdict, and each
|
||||
labels) before any schema was written (for `cookie-workflow`, before any request with the
|
||||
schema was sent). `second_review` records that verdict, and each
|
||||
disagreement carries `author_expected` and the third reviewer's `adjudication`.
|
||||
|
||||
`scripts/judge-calibration.ts` runs the comparison and `scripts/judge-calibration-configs.ts`
|
||||
@@ -21,7 +22,7 @@ and priced usage, and `comparison-1.result.json` holds the flip rates, agreement
|
||||
false passes, error rates, p95 latency, landing decision and the UC3 prose-step
|
||||
trigger. `manifest.json` prices every planned call before dispatch.
|
||||
|
||||
## Comparison (1) results (2026-10-03, `claude-fable-5-1`)
|
||||
## Comparison (1) results (2026-10-03, `claude-fable-5-1`, unless the row says otherwise)
|
||||
|
||||
Flips are panel-verdict differences on items where both panels completed: old-vs-old
|
||||
is `old-a` vs `old-b`, new-vs-old is `new` vs `old-a`. Agreement is items whose panel
|
||||
@@ -53,7 +54,8 @@ output tokens before the manifest was priced. Total W2 spend: $90.47 of $115.
|
||||
`cookie-workflow` (GSTA-91, separately funded) is held and stays on today's request. Its one
|
||||
new false pass, a07, scored 4/4/4 on all three new samples and on all three `old-b` samples;
|
||||
`old-a`'s a07 panel is incomplete because one old sample returned invalid JSON (the defect the
|
||||
schema transport removes), so the comparison counts it against the new request only. The old
|
||||
schema transport removes), so the comparison counts it against the new request only. Both
|
||||
valid `old-a` a07 samples also scored 4/4/4. The old
|
||||
request failed to parse 2 of 144 samples; the new one 0 of 72. The failed landing condition
|
||||
triggers the UC3 prose step, which is deferred to TODOS. Its `manifest.json` entry was priced
|
||||
separately (`priced_at`, `funding`) and is maintained by hand: the `manifest` command rewrites
|
||||
|
||||
Reference in new issue
Block a user