mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-08 17:58:13 +01:00
docs(changelog): describe the new cookie calibration tests (GSTA-91)
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
ce9c41c6d4
commit
084bed27c5
1 file changed
+2
-1
+2
-1
@@ -7,7 +7,8 @@
|
||||
### Tests
|
||||
- `test/fixtures/judge-calibration/cookie-workflow/`: 24 items. p01 is the production bundle (`buildCookieWorkflowJudgeInput`); a02 and a05 are real earlier revisions; the rest are hand-edited variants rendered by the same builder. Labels were preregistered and committed before any request with the schema was sent. A separate run (openai/gpt-6.1-sol) then labeled every item blind, and a third reviewer (google/gemini-4-argon) adjudicated the two disagreements: a01 kept pass, a08 changed to fail. `runs/` holds every sample and the result. The cost was 216 calls and $2.81, priced in `manifest.json` before dispatch as a separately funded entry.
|
||||
- `scripts/judge-calibration-configs.ts`: the `cookie-workflow` configuration builds prompts with `buildWorkflowJudgePrompt(COOKIE_WORKFLOW_JUDGE, …)` and uses the eval's median floors of 4/3/4. Its new request differs from the old one only by `WORKFLOW_JUDGE_RESPONSE_SCHEMA`.
|
||||
- `test/judge-calibration.test.ts` covers the new configuration and its pass rule. The wiring test now requires the cookie registration to send the schema exactly when the recorded cookie calibration lands. It doesn't land, so the registration stays unchanged and 15 workflow registrations use the schema.
|
||||
- `test/judge-calibration.test.ts` covers the new configuration and its pass rule, including the actionability floor. The wiring test now requires the cookie registration to send the schema exactly when the recorded cookie calibration lands. It doesn't land, so the registration stays unchanged and 15 workflow registrations use the schema. The cookie registration may only spread `COOKIE_WORKFLOW_JUDGE`, so the calibrated request stays the one production sends.
|
||||
- Two new free tests: the calibration builds byte-for-byte the prompt the cookie eval sends, and `analyze()` reproduces the committed result from its labels, samples, stop record and request options.
|
||||
|
||||
### Gate integrity
|
||||
- No judge request, rubric, threshold, sample count, landing rule or case changed. The cookie judge keeps its free-text request. No gate exception is proposed.
|
||||
|
||||
Reference in new issue
Block a user