fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier

- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an
  untrusted non-git dir (our temp skill dirs) — empirically verified.
- gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error);
  dropped from the session runner and the benchmark adapter. A present-
  but-unusable CLI (deprecated individual code-assist auth path) now
  classifies as SKIP, not a false adapter failure; the benchmark live
  smoke skips on auth/rate_limit error codes (environmental) while still
  failing on timeout/unknown (the drift classes it exists to catch).
- codex-e2e, gemini-e2e, and benchmark-providers gain the canonical
  whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3
  (external service -> periodic) — the sharded gate runner now excludes
  all three (gate: 45 -> 42 shards).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-13 10:34:25 -07:00
co-authored by Claude Fable 5
parent 990d54a9e4
commit bd11416d80
6 changed files with 100 additions and 21 deletions
+18 -1
View File
@@ -30,7 +30,15 @@ import * as os from 'os';
// --- Prerequisites / gating ---
const evalsEnabled = !!process.env.EVALS;
const describeIfEvals = evalsEnabled ? describe : describe.skip;
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3) —
// the header above says so, but without a whole-file guard the sharded gate
// runner still selects this file into gate. The positive form below is the
// canonical guard shape classifyPaidTestFile greps for.
const tierOk = process.env.EVALS_TIER === 'periodic';
const describeIfEvals = evalsEnabled && tierOk ? describe : describe.skip;
if (evalsEnabled && !tierOk) {
process.stderr.write('\nbenchmark-providers: SKIPPED — external-service test, periodic tier only\n');
}
const PROMPT = 'Reply with exactly this text and nothing else: ok';
@@ -127,6 +135,15 @@ describeIfEvals('multi-provider benchmark adapters (live)', () => {
}
const result = await gemini.run({ prompt: PROMPT, workdir, timeoutMs: 120_000 });
if (result.error) {
// auth / rate_limit are ENVIRONMENT conditions the test can't act on
// (e.g. Google deprecated the individual code-assist auth path — the
// adapter classifies "no longer supported" as auth). A live smoke
// reports them as a skip, not a false adapter failure. timeout/unknown
// still fail: those are the drift classes this test exists to catch.
if (result.error.code === 'auth' || result.error.code === 'rate_limit') {
process.stderr.write(`\ngemini live smoke: SKIPPED — ${result.error.code}: ${result.error.reason.slice(0, 160)}\n`);
return;
}
throw new Error(`gemini errored: ${result.error.code}${result.error.reason}`);
}
// Adapter must never report empty-success (#2159). After content/stats