Applied from the adversarial subagent pass during /review on this branch:
- test/benchmark-cli.test.ts — new "NOT READY path fires when auth env
vars are stripped" test. The default dry-run test always showed OK on
dev machines with auth, hiding regressions in the remediation-hint
branch. Stripped env (no auth vars, HOME→empty tmpdir) now force-
exercises gpt + gemini NOT READY paths and asserts every NOT READY
line includes a concrete remediation hint (install/login/export).
(claude adapter's os.homedir() call is Bun-cached; the 2-of-3 adapter
coverage is sufficient to exercise the branch.)
- test/taste-engine.test.ts — session-cap test rewritten to seed the
profile with 50 entries + one real CLI call, instead of 55 sequential
subprocess spawns. Same coverage (FIFO eviction at the boundary), ~5s
faster CI time. Also pins first-casing-wins on the Geist/GEIST merge
assertion — bumpPref() keeps the first-arrival casing, so the test
documents that policy.
- test/skill-e2e-benchmark-providers.test.ts — workdir creation moved
from module-load into beforeAll, cleanup added in afterAll. Previous
shape leaked a /tmp/bench-e2e-* dir every CI run.
- test/publish-dry-run.test.ts — removed unused empty test/helpers
mkdirSync from the sandbox setup. The bin doesn't import from there,
so the empty dir was a footgun for future maintainers.
- test/helpers/providers/gpt.ts — expanded the inline comment on
`--skip-git-repo-check` to explicitly note that `-s read-only` is now
load-bearing safety (the trust prompt was the secondary boundary;
removing read-only while keeping skip-git-repo-check would be unsafe).
Net: 45 passing tests (was 44), session-cap test 5s faster, one real
regression surface covered that didn't exist before.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The gpt provider adapter spawns `codex exec -C <workdir>` with arbitrary
working directories (benchmark temp dirs, non-git paths). Without
`--skip-git-repo-check`, codex refuses to run and returns "Not inside a
trusted directory" — surfaced as a generic error.code='unknown' that
looks like an API failure.
Benchmarks don't care about codex's git-repo trust model; we just want
the prompt executed. Surfaced by the new provider live E2E test on a
temp workdir.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full spec Codex asked for: real provider adapters with auth
detection, normalized RunResult, pricing tables, tool compatibility
maps, parallel execution with error isolation, and table/JSON/markdown
output. Judge stays on Anthropic SDK as the single stable source of
quality scoring, gated behind --judge.
Codex flagged the original plan as massively under-scoped — the
existing runner is Claude-only and the judge is Anthropic-only. You
can't benchmark GPT or Gemini without real provider infrastructure.
This commit ships it.
New architecture:
test/helpers/providers/types.ts ProviderAdapter interface
test/helpers/providers/claude.ts wraps `claude -p --output-format json`
test/helpers/providers/gpt.ts wraps `codex exec --json`
test/helpers/providers/gemini.ts wraps `gemini -p --output-format stream-json --yolo`
test/helpers/pricing.ts per-model USD cost tables (quarterly)
test/helpers/tool-map.ts which tools each CLI exposes
test/helpers/benchmark-runner.ts orchestrator (Promise.allSettled)
test/helpers/benchmark-judge.ts Anthropic SDK quality scorer
bin/gstack-model-benchmark CLI entry
test/benchmark-runner.test.ts 9 unit tests (cost math, formatters, tool-map)
Per-provider error isolation:
- auth → record reason, don't abort batch
- timeout → record reason, don't abort batch
- rate_limit → record reason, don't abort batch
- binary_missing → record in available() check, skip if --skip-unavailable
Pricing correction: cached input tokens are disjoint from uncached
input tokens (Anthropic/OpenAI report them separately). Original
math subtracted them, producing negative costs. Now adds cached at
the 10% discount alongside the full uncached input cost.
CLI:
gstack-model-benchmark --prompt "..." --models claude,gpt,gemini
gstack-model-benchmark ./prompt.txt --output json --judge
gstack-model-benchmark ./prompt.txt --models claude --timeout-ms 60000
Output formats: table (default), json, markdown. Each shows model,
latency, in→out tokens, cost, quality (when --judge used), tool calls,
and any errors.
Known limitations for v1:
- Claude adapter approximates toolCalls as num_turns (stream-json
would give exact counts; v2 can upgrade).
- Live E2E tests (test/providers.e2e.test.ts) not included — they
require CI secrets for all three providers. Unit tests cover the
shape and math.
- Provider CLIs sometimes return non-JSON error text to stdout; the
parsers fall back to treating raw output as plain text in that case.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Garry Tan