CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-13 00:19:03 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(evals): 300s budget floor for the two 90s design-consultation SDK tests

Browse Source 
Root cause of PR #2533's e2e-design CI failure: design-consultation-preview
failed 3 attempts at 0 turns/$0.00/93s — the session was up but the model's
first completion queued past the 90s inner budget under concurrent API load
(11 matrix jobs; the sibling research test booted its first tool at 4s, so
this is API-side queuing, not CPU boot contention). The test was selected
only because touchfiles.ts is a global touchfile; the tested behavior is
untouched by this branch.

90s budgets cannot absorb one slow first completion. Both 90s tests in the
file move to the repo's saturated-runner standard (300s inner / 360s outer,
matching review-dashboard-via and retro-base-branch). Deliberately NOT
re-arming the runner's inner timer on first stream event: an audit found
~100 outer bun-timeout literals sized inner+30-60s that a re-arm would
silently break — the structural options are written up in TODOS.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-12 10:44:47 -07:00
co-authored by Claude Fable 5
parent 6422c78f6f
commit f90d5a990e
2 changed files with 37 additions and 4 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-e2e-design.test.ts
+12 -4
View File
@@ -176,7 +176,13 @@ Include: color trends, typography patterns, and layout conventions you observed.
Do NOT generate a full DESIGN.md — just research notes.`,
workingDirectory: researchDir,
maxTurns: 8,
timeout: 90_000,
// 300s, not 90s: saturated-runner class (same as review-dashboard-via /
// retro-base-branch). PR #2533 CI observed the sibling preview test at
// 0 turns/$0.00 for 93s x3 attempts — session up, first completion
// queued past the budget under concurrent API load. 90s budgets cannot
// absorb one slow first completion; 300s is the repo's standard floor
// for CI SDK tests. Outer timeout below rises to 360s for headroom.
timeout: 300_000,
testName: 'design-consultation-research',
runId,
});
@@ -206,7 +212,7 @@ Do NOT generate a full DESIGN.md — just research notes.`,
}
try { fs.rmSync(researchDir, { recursive: true, force: true }); } catch {}
}, 120_000);
}, 360_000);
testConcurrentIfSelected('design-consultation-existing', async () => {
// Pre-create a minimal DESIGN.md (independent of core test)
@@ -274,7 +280,9 @@ Write a single HTML file to ${previewDir}/design-preview.html that shows:
Do NOT write DESIGN.md — only the preview HTML.`,
workingDirectory: previewDir,
maxTurns: 8,
timeout: 90_000,
// 300s, not 90s: this is the test that failed 3x at 0 turns/$0.00/93s
// on PR #2533 CI — see the research test's comment for the class.
timeout: 300_000,
testName: 'design-consultation-preview',
runId,
});
@@ -303,7 +311,7 @@ Do NOT write DESIGN.md — only the preview HTML.`,
}
try { fs.rmSync(previewDir, { recursive: true, force: true }); } catch {}
}, 120_000);
}, 360_000);
});
// --- Plan Design Review E2E (plan-mode) ---