test(harness): record the llm-judge parse-message change as non-behavioral (GSTA-91)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed 2026-10-25 14:02:34 +00:00
1 parent 9faeebb537
commit e096ec6659
2 files changed
+2 -1

No files matched your search

+1
View File
@@ -19,6 +19,7 @@
- One test budget changed, approved by GStack Minder on 2026-10-25: the install-admission case's timeout goes from bun's 5 s default to 20 s, about 3.5 times the 5.7 s worst case measured above. No assertion changed.
- The QA-only late scenarios register each attempt with its real outer budget (1000 + 80 + 5000 = 6080 ms, the value `outerMs` already asserts) instead of a shortened 1580 ms Bun deadline, which left cleanup 500 ms of real time under load. Measured child totals with the fix under 24 busy loops, 20 runs each: late-timeout 2.4–2.7 s, late-retry 4.1–4.3 s, late-failure-retry 4.7–5.0 s, late-slow-retry 6.2–6.6 s, against the parent's unchanged 15 s spawn timeout. The mocked 1000 ms capture budget, the 80 ms drain and every existing assertion are unchanged. Verdict: justified, because the deadline was a fixture artifact shorter than the budget the case declares, and the clock freeze, not the deadline, is what removes the load sensitivity.
- `context-restore-fragment-match`: the failure surface moves from the whole transcript to the agent's report. Alpha or omega bytes in tool output no longer fail the case; the payments body in tool output no longer counts as loaded, so the report must show it. A run that resumes another checkpoint, or mixes two, still fails. `test/test-of-test-ratchet.test.ts` gains one BASELINE entry, the replay test, with its reason, and the new helper joins the touchfiles of the nine context-skills cases, as the derived-closure rule requires.
- `scripts/harness-version.json` records `test/helpers/llm-judge.ts` as non-behavioral at HARNESS_VERSION 6: only a parse failure's message changes, before any score is parsed, so no case's verdict or pass-rate series changes.
- No threshold, trial count, case kind, rubric, judge request or prompt changed.
## [1.91.211.0] - 2026-10-25
+1 -1
View File
@@ -29,7 +29,7 @@
"test/helpers/eval-budgets.ts": {"blob":"667d0e09b20fe5d7366f57b77b5fb9b04baa37ef","since":6},
"test/helpers/eval-store.ts": {"blob":"a50ea2c28f508ad9f35ea9353dd2f502ee0dcfc2","since":6,"nonBehavioral":"credential rejections are classed infra instead of assertion: no case's pass or fail changes; only failed trials whose provider rejected the run's key move class, so no rerun spends on a rejected key"},
"test/helpers/hermetic-env.ts": {"blob":"cc072895461dc52ea317d1beff72121342420d30","since":6},
"test/helpers/llm-judge.ts": {"blob":"d0bc6bbbe9f478603d11a1ece95e66c812ad34e6","since":6},
"test/helpers/llm-judge.ts": {"blob":"a3177665b662932ce485ccc976c2b36a1531a0bb","since":6,"nonBehavioral":"callJudge rewrites only the message of a JSON parse failure (names the judge output, stop reason, length and the retained llm-judge-response-parse-error event); it still throws a SyntaxError before any score is parsed, the request is unchanged, and no case's pass or fail changes"},
"test/helpers/paid-test-set.ts": {"blob":"ff9bfe43c43ec2bdd11114f7b8ec7c5736116f75","since":6},
"test/helpers/pty-screen.ts": {"blob":"4f77569f9ce11f9cee9f347216e468cd7c956820","since":6},
"test/helpers/pty/auq.ts": {"blob":"16e5ea2bb5b9f28f44808a1a09024dbd3e6f28c4","since":6},