mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-22 05:57:18 +02:00
evals: preflight ping once in the sharded parent, not per shard
The Anthropic fail-fast ping ran at module load in every paid test file importing e2e-helpers — ~30 paid claude -p calls (30s timeout each) per full sharded run for one bit of information. The parent now pings once before spawning shards and sets EVALS_PREFLIGHT_OK=1; the module-load path honors the flag. Extracted to test/helpers/anthropic-preflight.ts (injectable spawn seam) with regression pins in both directions: the flag must skip, its absence must ping exactly once, dead API must throw. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
f9e3fc8ab4
commit
6d109d3232
@@ -64,6 +64,7 @@ import {
|
||||
} from './test-strict-output';
|
||||
import { PAID_TEST_GLOBS, isPaidTestFile } from '../test/helpers/paid-test-set';
|
||||
import { getProjectEvalDir } from '../test/helpers/eval-store';
|
||||
import { preflightAnthropicApi } from '../test/helpers/anthropic-preflight';
|
||||
|
||||
export { PAID_TEST_GLOBS, isPaidTestFile };
|
||||
|
||||
@@ -463,13 +464,19 @@ async function main(): Promise<number> {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// One preflight ping in the parent; children skip theirs via the env flag.
|
||||
// Before this, every shard's e2e-helpers module load re-pinged the API —
|
||||
// ~30 paid claude -p calls (30s timeout each) per full run for one bit of
|
||||
// information. A dead API now fails here, before any shard spawns.
|
||||
preflightAnthropicApi(process.env);
|
||||
|
||||
const summary = await runPaidShards(shards, {
|
||||
// Tier reaches the children only via EVALS_TIER below; the runtime
|
||||
// E2E_TIERS filter inside each child is the real selection mechanism.
|
||||
timeoutMs: options.timeoutMs,
|
||||
jobs: options.jobs,
|
||||
withinShardConcurrency: options.withinShardConcurrency,
|
||||
env: { ...process.env, EVALS: '1', EVALS_TIER: options.tier },
|
||||
env: { ...process.env, EVALS: '1', EVALS_TIER: options.tier, EVALS_PREFLIGHT_OK: '1' },
|
||||
evalDirBase: process.env.GSTACK_EVAL_DIR || getProjectEvalDir(),
|
||||
});
|
||||
for (const line of formatSummary(summary)) console.log(line);
|
||||
|
||||
Reference in New Issue
Block a user