mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-14 08:59:01 +02:00
fix(evals): absorb codex/gemini CLI drift; external-service tests go periodic-tier
- codex exec gains --skip-git-repo-check: newer CLIs refuse exec in an untrusted non-git dir (our temp skill dirs) — empirically verified. - gemini: --skip-trust was removed in gemini-cli 0.34 (argv parse error); dropped from the session runner and the benchmark adapter. A present- but-unusable CLI (deprecated individual code-assist auth path) now classifies as SKIP, not a false adapter failure; the benchmark live smoke skips on auth/rate_limit error codes (environmental) while still failing on timeout/unknown (the drift classes it exists to catch). - codex-e2e, gemini-e2e, and benchmark-providers gain the canonical whole-file EVALS_TIER === 'periodic' guard per CLAUDE.md tiering rule 3 (external service -> periodic) — the sharded gate runner now excludes all three (gate: 45 -> 42 shards). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
990d54a9e4
commit
bd11416d80
@@ -199,8 +199,12 @@ export async function runCodexSkill(opts: {
|
||||
}
|
||||
}
|
||||
|
||||
// Build codex exec command
|
||||
const args = ['exec', prompt, '--json', '-s', sandbox];
|
||||
// Build codex exec command.
|
||||
// --skip-git-repo-check: newer codex CLIs refuse exec in an untrusted
|
||||
// non-git directory ("Not inside a trusted directory and
|
||||
// --skip-git-repo-check was not specified") — our temp skill dirs are
|
||||
// exactly that. Empirically verified against codex on this machine.
|
||||
const args = ['exec', prompt, '--json', '-s', sandbox, '--skip-git-repo-check'];
|
||||
|
||||
// Spawn codex with temp HOME so it discovers our installed skill.
|
||||
// Hermetic scrub (test/helpers/hermetic-env.ts) with codex's auth surface
|
||||
|
||||
Reference in New Issue
Block a user