mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-15 17:35:29 +02:00
fix(evals): 300s budget floor for the two 90s design-consultation SDK tests
Root cause of PR #2533's e2e-design CI failure: design-consultation-preview failed 3 attempts at 0 turns/$0.00/93s — the session was up but the model's first completion queued past the 90s inner budget under concurrent API load (11 matrix jobs; the sibling research test booted its first tool at 4s, so this is API-side queuing, not CPU boot contention). The test was selected only because touchfiles.ts is a global touchfile; the tested behavior is untouched by this branch. 90s budgets cannot absorb one slow first completion. Both 90s tests in the file move to the repo's saturated-runner standard (300s inner / 360s outer, matching review-dashboard-via and retro-base-branch). Deliberately NOT re-arming the runner's inner timer on first stream event: an audit found ~100 outer bun-timeout literals sized inner+30-60s that a re-arm would silently break — the structural options are written up in TODOS.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
6422c78f6f
commit
f90d5a990e
@@ -233,6 +233,31 @@ SKILL.md untouched). `bun test` is green again.
|
||||
|
||||
## Scope-gate follow-ups (filed via /plan-eng-review on the plan-mode auto-select-B change)
|
||||
|
||||
### P2: SDK eval budgets charge API-queue latency to the work budget — pick a structural fix
|
||||
|
||||
**What:** `runSkillTest`'s single `setTimeout(timeout)` arms at spawn, so session
|
||||
startup AND the model's first-completion queue time are charged against the
|
||||
test's work budget. Under concurrent load (11 CI matrix jobs, or local eval
|
||||
runs sharing the org API), a first completion can queue 60-90s+, producing the
|
||||
deterministic `0 turns / $0.00 / <budget>s x3 attempts` failure shape. Observed:
|
||||
`review-dashboard-via` (PR #2472, 180s→300s), `retro-base-branch` (240s→360s),
|
||||
`plan-ceo-plan-mode` (300s→420s, 2026-08-12), `design-consultation-preview`
|
||||
(90s→300s, PR #2533 CI). Every fix so far is a per-test budget bump.
|
||||
|
||||
**Why not just re-arm the timer on first stream event:** an audit (2026-08-12)
|
||||
found ~100 outer bun-timeout literals sized as inner+30-60s; re-arming the inner
|
||||
clock breaks every outer/inner relationship and needs a codemod of all of them.
|
||||
|
||||
**Options:** (a) two-phase timer in session-runner (startup grace, re-arm on
|
||||
first NDJSON line) + codemod outer literals to inner+grace+slack; (b) adopt a
|
||||
300s floor for all CI SDK budgets (statically enforceable — a free test can
|
||||
assert no `timeout: <300_000` in skill-e2e files) and stop re-litigating per
|
||||
test; (c) startup-spawn semaphore in the runner (bounds the boot stampede but
|
||||
not API-side queuing — evidence says queuing dominates, so likely insufficient
|
||||
alone). Recommend (b) short-term + (a) properly sequenced with the codemod.
|
||||
|
||||
**Depends on / blocked by:** none.
|
||||
|
||||
### P2: Wire the four demoted plan-mode/finding-floor PTY tests into periodic CI
|
||||
|
||||
**What:** `evals-periodic.yml` runs an explicit 9-file matrix; the four tests
|
||||
|
||||
Reference in New Issue
Block a user