CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-16 09:55:29 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(evals): preflight fails fast on spawn error, timeout, and exit 127

Browse Source 
The ping only grepped stdout for two connection strings — a missing
claude binary, a 30s timeout kill, or command-not-found all returned
'ok', and the fleet then burned ~30 shard timeouts discovering the
outage one child at a time. Cross-model finding (testing specialist +
codex adversarial). Other non-zero exits stay deliberately fail-open:
a flaky preflight must not block a runnable suite; pinned both ways.
This commit is contained in:
Garry Tan
2026-08-15 17:01:41 -07:00
parent 9e72b4ac7f
commit 1e35386c54
2 changed files with 43 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/helpers/anthropic-preflight.ts
+14
View File
@@ -27,6 +27,20 @@ export function preflightAnthropicApi(
['-c', 'echo "ping" | claude -p --max-turns 1 --output-format stream-json --verbose --dangerously-skip-permissions'],
{ stdio: 'pipe', timeout: 30_000 },
);
// Fail fast on the failure modes that guarantee EVERY shard fails too:
// spawn error (sh/claude unlaunchable), a timeout kill (signal set), or
// exit 127 (command not found). Anything else stays fail-open — a flaky
// preflight must not block a run the shards could complete (auth prompts
// and transient non-zero exits are the shards' problem to report).
if (check.error) {
throw new Error(`Anthropic preflight could not run (${check.error.message}) — aborting E2E suite before spawning shards.`);
}
if (check.signal) {
throw new Error(`Anthropic preflight timed out (killed with ${check.signal}) — aborting E2E suite. Check connectivity/auth and retry.`);
}
if (check.status === 127) {
throw new Error('Anthropic preflight: `claude` not found on PATH (exit 127) — aborting E2E suite before spawning shards.');
}
const output = check.stdout?.toString() || '';
if (output.includes('ConnectionRefused') || output.includes('Unable to connect')) {
throw new Error('Anthropic API unreachable — aborting E2E suite. Fix connectivity and retry.');