evals: preflight ping once in the sharded parent, not per shard

The Anthropic fail-fast ping ran at module load in every paid test file
importing e2e-helpers — ~30 paid claude -p calls (30s timeout each) per
full sharded run for one bit of information. The parent now pings once
before spawning shards and sets EVALS_PREFLIGHT_OK=1; the module-load
path honors the flag. Extracted to test/helpers/anthropic-preflight.ts
(injectable spawn seam) with regression pins in both directions: the
flag must skip, its absence must ping exactly once, dead API must throw.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-15 08:13:30 -07:00
co-authored by Claude Fable 5
parent f9e3fc8ab4
commit 6d109d3232
4 changed files with 102 additions and 9 deletions
+8 -1
View File
@@ -64,6 +64,7 @@ import {
} from './test-strict-output';
import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set';
import { getProjectEvalDir } from '../test/helpers/eval-store';
import { preflightAnthropicApi } from '../test/helpers/anthropic-preflight';
export { PAID_TEST_GLOBS, isPaidTestFile };
@@ -463,13 +464,19 @@ async function main(): Promise<number> {
return 0;
}
// One preflight ping in the parent; children skip theirs via the env flag.
// Before this, every shard's e2e-helpers module load re-pinged the API —
// ~30 paid claude -p calls (30s timeout each) per full run for one bit of
// information. A dead API now fails here, before any shard spawns.
preflightAnthropicApi(process.env);
const summary = await runPaidShards(shards, {
// Tier reaches the children only via EVALS_TIER below; the runtime
// E2E_TIERS filter inside each child is the real selection mechanism.
timeoutMs: options.timeoutMs,
jobs: options.jobs,
withinShardConcurrency: options.withinShardConcurrency,
env: { ...process.env, EVALS: '1', EVALS_TIER: options.tier },
env: { ...process.env, EVALS: '1', EVALS_TIER: options.tier, EVALS_PREFLIGHT_OK: '1' },
evalDirBase: process.env.GSTACK_EVAL_DIR || getProjectEvalDir(),
});
for (const line of formatSummary(summary)) console.log(line);