mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-10 23:19:09 +02:00
fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier
- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an untrusted non-git dir (our temp skill dirs) — empirically verified. - gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error); dropped from the session runner and the benchmark adapter. A present- but-unusable CLI (deprecated individual code-assist auth path) now classifies as SKIP, not a false adapter failure; the benchmark live smoke skips on auth/rate_limit error codes (environmental) while still failing on timeout/unknown (the drift classes it exists to catch). - codex-e2e, gemini-e2e, and benchmark-providers gain the canonical whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3 (external service -> periodic) — the sharded gate runner now excludes all three (gate: 45 -> 42 shards). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
990d54a9e4
commit
bd11416d80
+11
-2
@@ -37,15 +37,24 @@ const CODEX_AVAILABLE = (() => {
|
||||
|
||||
const evalsEnabled = !!process.env.EVALS;
|
||||
|
||||
// Skip all tests if codex is not available or EVALS is not set.
|
||||
// External-service tests are periodic-tier (CLAUDE.md tiering rule 3):
|
||||
// "Requires external service (Codex, Gemini)? -> periodic". The positive
|
||||
// form below is the canonical whole-file guard shape — the sharded runner's
|
||||
// classifyPaidTestFile greps for it to exclude this file from gate.
|
||||
const tierOk = process.env.EVALS_TIER === 'periodic';
|
||||
|
||||
// Skip all tests if codex is not available, EVALS is not set, or we're in
|
||||
// the gate tier.
|
||||
// Note: Codex uses its own auth from ~/.codex/ config — no OPENAI_API_KEY env var needed.
|
||||
const SKIP = !CODEX_AVAILABLE || !evalsEnabled;
|
||||
const SKIP = !CODEX_AVAILABLE || !evalsEnabled || !tierOk;
|
||||
|
||||
const describeCodex = SKIP ? describe.skip : describe;
|
||||
|
||||
// Log why we're skipping (helpful for debugging CI)
|
||||
if (!evalsEnabled) {
|
||||
// Silent — same as Claude E2E tests, EVALS=1 required
|
||||
} else if (!tierOk) {
|
||||
process.stderr.write('\nCodex E2E: SKIPPED — external-service test, periodic tier only (EVALS_TIER === \'periodic\')\n');
|
||||
} else if (!CODEX_AVAILABLE) {
|
||||
process.stderr.write('\nCodex E2E: SKIPPED — codex binary not found (install: npm i -g @openai/codex)\n');
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user