refactor: capture plain-text CLI output, drop hardcoded provider schemas

The adapters parsed each vendor's proprietary JSON stream (Claude json,
Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a
per-model pricing table turned tokens into cost. That coupling was the
brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor
reshuffled its output, and it duplicated what any tool that instruments the
real model call already does. Braintrust owns scoring; it can't see a CLI
subprocess's tokens anyway, so computing cost ourselves meant maintaining
both a parser and a price table forever.

Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring
is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls;
the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all
three JSON parsers, and the Gemini stream-schema parser + its test. Gemini
auth detection (env/OAuth/.env) is kept — that's not schema parsing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:34:57 -07:00
co-authored by Claude Opus 4.8
parent 36f972f2e4
commit 7f4574e1d5
12 changed files with 84 additions and 441 deletions
@@ -42,8 +42,6 @@ test('production modules do not import from test directories', () => {
test('former test-helper paths re-export the production benchmark API', async () => {
const [
pricing,
helperPricing,
claude,
helperClaude,
gpt,
@@ -51,8 +49,6 @@ test('former test-helper paths re-export the production benchmark API', async ()
gemini,
helperGemini,
] = await Promise.all([
import('../lib/model-benchmark/pricing'),
import('./helpers/pricing'),
import('../lib/model-benchmark/providers/claude'),
import('./helpers/providers/claude'),
import('../lib/model-benchmark/providers/gpt'),
@@ -61,7 +57,6 @@ test('former test-helper paths re-export the production benchmark API', async ()
import('./helpers/providers/gemini'),
]);
expect(helperPricing.estimateCostUsd).toBe(pricing.estimateCostUsd);
expect(helperClaude.ClaudeAdapter).toBe(claude.ClaudeAdapter);
expect(helperGpt.GptAdapter).toBe(gpt.GptAdapter);
expect(helperGemini.GeminiAdapter).toBe(gemini.GeminiAdapter);