mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-21 20:30:47 +02:00
refactor: capture plain-text CLI output, drop hardcoded provider schemas
The adapters parsed each vendor's proprietary JSON stream (Claude json, Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a per-model pricing table turned tokens into cost. That coupling was the brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor reshuffled its output, and it duplicated what any tool that instruments the real model call already does. Braintrust owns scoring; it can't see a CLI subprocess's tokens anyway, so computing cost ourselves meant maintaining both a parser and a price table forever. Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls; the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all three JSON parsers, and the Gemini stream-schema parser + its test. Gemini auth detection (env/OAuth/.env) is kept — that's not schema parsing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
36f972f2e4
commit
7f4574e1d5
@@ -42,8 +42,6 @@ test('production modules do not import from test directories', () => {
|
||||
|
||||
test('former test-helper paths re-export the production benchmark API', async () => {
|
||||
const [
|
||||
pricing,
|
||||
helperPricing,
|
||||
claude,
|
||||
helperClaude,
|
||||
gpt,
|
||||
@@ -51,8 +49,6 @@ test('former test-helper paths re-export the production benchmark API', async ()
|
||||
gemini,
|
||||
helperGemini,
|
||||
] = await Promise.all([
|
||||
import('../lib/model-benchmark/pricing'),
|
||||
import('./helpers/pricing'),
|
||||
import('../lib/model-benchmark/providers/claude'),
|
||||
import('./helpers/providers/claude'),
|
||||
import('../lib/model-benchmark/providers/gpt'),
|
||||
@@ -61,7 +57,6 @@ test('former test-helper paths re-export the production benchmark API', async ()
|
||||
import('./helpers/providers/gemini'),
|
||||
]);
|
||||
|
||||
expect(helperPricing.estimateCostUsd).toBe(pricing.estimateCostUsd);
|
||||
expect(helperClaude.ClaudeAdapter).toBe(claude.ClaudeAdapter);
|
||||
expect(helperGpt.GptAdapter).toBe(gpt.GptAdapter);
|
||||
expect(helperGemini.GeminiAdapter).toBe(gemini.GeminiAdapter);
|
||||
|
||||
Reference in New Issue
Block a user