fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier

- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an
  untrusted non-git dir (our temp skill dirs) — empirically verified.
- gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error);
  dropped from the session runner and the benchmark adapter. A present-
  but-unusable CLI (deprecated individual code-assist auth path) now
  classifies as SKIP, not a false adapter failure; the benchmark live
  smoke skips on auth/rate_limit error codes (environmental) while still
  failing on timeout/unknown (the drift classes it exists to catch).
- codex-e2e, gemini-e2e, and benchmark-providers gain the canonical
  whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3
  (external service -> periodic) — the sharded gate runner now excludes
  all three (gate: 45 -> 42 shards).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-13 10:34:25 -07:00
co-authored by Claude Fable 5
parent 990d54a9e4
commit bd11416d80
6 changed files with 100 additions and 21 deletions
+7 -8
View File
@@ -103,10 +103,10 @@ export function resultFromGeminiStream(
* Headless flags always passed:
* --output-format stream-json — NDJSON events (message/tool_use/result)
* --yolo — auto-approve tools (non-interactive)
* --skip-trust — trust cwd for this session; required when
* workdir is a temp/untrusted folder (benchmarks
* use mkdtemp). Without it headless gemini exits
* before calling the model.
*
* --skip-trust is gone: gemini-cli 0.34 removed the flag ("Unknown arguments:
* skip-trust") — folder trust is settings-driven now and headless runs no
* longer need a flag for temp workdirs.
*/
export class GeminiAdapter implements ProviderAdapter {
readonly name = 'gemini';
@@ -136,10 +136,9 @@ export class GeminiAdapter implements ProviderAdapter {
async run(opts: RunOpts): Promise<RunResult> {
const start = Date.now();
// Default to --yolo (non-interactive) and stream-json output so we can parse
// tokens + tool calls. --skip-trust is required for headless/temp workdirs
// (gemini CLI otherwise exits: "not running in a trusted directory").
// Callers can override via extraArgs.
const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo', '--skip-trust'];
// tokens + tool calls. Callers can override via extraArgs. (--skip-trust was
// removed in gemini-cli 0.34; passing it errors at argv parse.)
const args = ['-p', opts.prompt, '--output-format', 'stream-json', '--yolo'];
if (opts.model) args.push('--model', opts.model);
if (opts.extraArgs) args.push(...opts.extraArgs);