mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-16 09:55:29 +02:00
fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier
- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an untrusted non-git dir (our temp skill dirs) — empirically verified. - gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error); dropped from the session runner and the benchmark adapter. A present- but-unusable CLI (deprecated individual code-assist auth path) now classifies as SKIP, not a false adapter failure; the benchmark live smoke skips on auth/rate_limit error codes (environmental) while still failing on timeout/unknown (the drift classes it exists to catch). - codex-e2e, gemini-e2e, and benchmark-providers gain the canonical whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3 (external service -> periodic) — the sharded gate runner now excludes all three (gate: 45 -> 42 shards). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
990d54a9e4
commit
bd11416d80
@@ -30,7 +30,15 @@ import * as os from 'os';
|
||||
// --- Prerequisites / gating ---
|
||||
|
||||
const evalsEnabled = !!process.env.EVALS;
|
||||
const describeIfEvals = evalsEnabled ? describe : describe.skip;
|
||||
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3) —
|
||||
// the header above says so, but without a whole-file guard the sharded gate
|
||||
// runner still selects this file into gate. The positive form below is the
|
||||
// canonical guard shape classifyPaidTestFile greps for.
|
||||
const tierOk = process.env.EVALS_TIER === 'periodic';
|
||||
const describeIfEvals = evalsEnabled && tierOk ? describe : describe.skip;
|
||||
if (evalsEnabled && !tierOk) {
|
||||
process.stderr.write('\nbenchmark-providers: SKIPPED — external-service test, periodic tier only\n');
|
||||
}
|
||||
|
||||
const PROMPT = 'Reply with exactly this text and nothing else: ok';
|
||||
|
||||
@@ -127,6 +135,15 @@ describeIfEvals('multi-provider benchmark adapters (live)', () => {
|
||||
}
|
||||
const result = await gemini.run({ prompt: PROMPT, workdir, timeoutMs: 120_000 });
|
||||
if (result.error) {
|
||||
// auth / rate_limit are ENVIRONMENT conditions the test can't act on
|
||||
// (e.g. Google deprecated the individual code-assist auth path — the
|
||||
// adapter classifies "no longer supported" as auth). A live smoke
|
||||
// reports them as a skip, not a false adapter failure. timeout/unknown
|
||||
// still fail: those are the drift classes this test exists to catch.
|
||||
if (result.error.code === 'auth' || result.error.code === 'rate_limit') {
|
||||
process.stderr.write(`\ngemini live smoke: SKIPPED — ${result.error.code}: ${result.error.reason.slice(0, 160)}\n`);
|
||||
return;
|
||||
}
|
||||
throw new Error(`gemini errored: ${result.error.code} — ${result.error.reason}`);
|
||||
}
|
||||
// Adapter must never report empty-success (#2159). After content/stats
|
||||
|
||||
Reference in New Issue
Block a user