fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier

- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an
  untrusted non-git dir (our temp skill dirs) — empirically verified.
- gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error);
  dropped from the session runner and the benchmark adapter. A present-
  but-unusable CLI (deprecated individual code-assist auth path) now
  classifies as SKIP, not a false adapter failure; the benchmark live
  smoke skips on auth/rate_limit error codes (environmental) while still
  failing on timeout/unknown (the drift classes it exists to catch).
- codex-e2e, gemini-e2e, and benchmark-providers gain the canonical
  whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3
  (external service -> periodic) — the sharded gate runner now excludes
  all three (gate: 45 -> 42 shards).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-13 10:34:25 -07:00
co-authored by Claude Fable 5
parent 990d54a9e4
commit bd11416d80
6 changed files with 100 additions and 21 deletions
+6 -2
View File
@@ -199,8 +199,12 @@ export async function runCodexSkill(opts: {
}
}
// Build codex exec command
const args = ['exec', prompt, '--json', '-s', sandbox];
// Build codex exec command.
// --skip-git-repo-check: newer codex CLIs refuse exec in an untrusted
// non-git directory ("Not inside a trusted directory and
// --skip-git-repo-check was not specified") — our temp skill dirs are
// exactly that. Empirically verified against codex on this machine.
const args = ['exec', prompt, '--json', '-s', sandbox, '--skip-git-repo-check'];
// Spawn codex with temp HOME so it discovers our installed skill.
// Hermetic scrub (test/helpers/hermetic-env.ts) with codex's auth surface