test: run seven paid evals on the current default capture model (B8)

skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned
claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned
claude-sonnet-4-6; none tests a historical model, so they now capture with
resolveEvalModel('capture'), and the free harness tests that execute these
registrations receive the same resolver. The paid re-pin run passed all of
them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep
claude-opus-4-7: six of their cases failed on the default model (three
timeouts, a missing report file, a format miss and a posture score of 3), so
per the plan's fallback they keep their pins with a TODOS entry. The pre-spend
estimate and drop threshold are in docs/test-audit-2026-09.md.
This commit is contained in:
garrytan committed 2026-09-29 09:16:19 +00:00
1 parent 799f6ec36c
commit 67084a5caf
11 files changed
+55 -23

No files matched your search

+1 -1
View File
@@ -61,7 +61,7 @@ console.log(JSON.stringify({ type: 'result', subtype: 'success', is_error: false
// payload assertions. The negative case restores only the original typo.
expect(suiteSource).toContain('env: childEnv,');
let selectedSource = option === 'env' ? suiteSource : suiteSource.replace('env: childEnv,', 'extraEnv: childEnv,');
selectedSource = selectedSource.replace(/from '(\.\/helpers\/[^']+)'/g,
selectedSource = selectedSource.replace(/from '((?:\.\/helpers|\.\.\/lib)\/[^']+)'/g,
(_match, spec: string) => `from ${JSON.stringify(path.resolve(ROOT, 'test', spec))}`);
const suiteCopy = path.join(dir, 'writeback-suite.ts');
fs.writeFileSync(suiteCopy, selectedSource);