evals: Opus pin on the spec AUQ-matrix entry — D1a regressor, receipts in-file

The periodic re-baseline for the capture default (Opus -> Sonnet)
found exactly one regressor across the seven-entry AUQ behavioral
matrix: spec failed twice under Sonnet ('never reached a question in
budget', 242s) while its six siblings passed; the controlled Opus
re-run passed cleanly (7/7 format, substance 5, 160s), and a second
run through the new per-entry model plumbing confirms. MatrixSkill
gains an optional model field wired into captureFirstAuq; only spec
sets it. TODOS gains the re-baseline receipts for the never-baselined
periodic tail (three setup-gbrain files + ship-idempotency, all
local-only).
This commit is contained in:
Garry Tan
2026-08-15 20:59:27 -07:00
parent 9c2d44253d
commit 5c67b11978
2 changed files with 18 additions and 0 deletions
+10
View File
@@ -67,6 +67,8 @@ interface MatrixSkill {
skill: string;
fixtures: Record<string, string>;
scenario: string;
/** D1a regressor pin: explicit capture model when the Sonnet default measurably fails this entry. */
model?: string;
}
const MATRIX: MatrixSkill[] = [
@@ -99,6 +101,13 @@ const MATRIX: MatrixSkill[] = [
skill: 'spec',
fixtures: {},
scenario: 'Turn this vague intent into a precise spec: "add email notifications when a task is assigned to someone." Walk the spec workflow until the first AskUserQuestion.',
// D1a pin-on-regressors, with receipts (2026-08-16 re-baseline): under
// the Sonnet capture default this entry failed twice ("never reached a
// question in budget", 242s) while the six sibling entries passed; the
// controlled Opus re-run passed cleanly (7/7 format, substance 5, 160s).
// The spec workflow's long pre-question phase needs the stronger model
// to reach its first AskUserQuestion inside the turn budget.
model: 'claude-opus-4-7',
},
{
skill: 'design-consultation',
@@ -130,6 +139,7 @@ describeE2E('AUQ behavioral matrix (periodic)', () => {
scenario: m.scenario,
testName: `auq-matrix-${m.skill}`,
runId,
model: m.model,
});
} finally {
fs.rmSync(dir, { recursive: true, force: true });