evals: Opus pin on the spec AUQ-matrix entry — D1a regressor, receipts in-file

The periodic re-baseline for the capture default (Opus -> Sonnet)
found exactly one regressor across the seven-entry AUQ behavioral
matrix: spec failed twice under Sonnet ('never reached a question in
budget', 242s) while its six siblings passed; the controlled Opus
re-run passed cleanly (7/7 format, substance 5, 160s), and a second
run through the new per-entry model plumbing confirms. MatrixSkill
gains an optional model field wired into captureFirstAuq; only spec
sets it. TODOS gains the re-baseline receipts for the never-baselined
periodic tail (three setup-gbrain files + ship-idempotency, all
local-only).
This commit is contained in:
Garry Tan
2026-08-15 20:59:27 -07:00
parent 9c2d44253d
commit 5c67b11978
2 changed files with 18 additions and 0 deletions
+8
View File
@@ -197,6 +197,14 @@ claim true.
some orphans are deliberately manual (ios-device, opus-47 overlay harness), so a plain
glob is wrong — needs a curated exclude list.
**Fresh receipts (2026-08-16, v1.66.0.0 re-baseline):** the first full local
periodic run in this store gave the never-baselined tail its first results:
`skill-e2e-setup-gbrain-{bad-token,path4-local-pglite,remote}` all failed
(spawned-process exit 1 — likely live-gbrain interference on a dev box) and
`skill-e2e-ship-idempotency` timed out at the 1800s shard wall. None are in
the weekly matrix, so these failures are invisible to CI — exactly this
item's thesis. Start the burn-down with those four.
**Context / where to start:** `.github/workflows/evals-periodic.yml:71` (matrix),
`test/helpers/touchfiles.ts` E2E_TIERS (tier labels already exist per test), orphan
list generated via `comm -23` between `ls test/skill-e2e-*.test.ts` and the file lists