CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-21 12:20:48 +02:00
Code Issues Packages Projects Releases Wiki Activity

refactor: capture plain-text CLI output, drop hardcoded provider schemas

Browse Source 
The adapters parsed each vendor's proprietary JSON stream (Claude json,
Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a
per-model pricing table turned tokens into cost. That coupling was the
brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor
reshuffled its output, and it duplicated what any tool that instruments the
real model call already does. Braintrust owns scoring; it can't see a CLI
subprocess's tokens anyway, so computing cost ourselves meant maintaining
both a parser and a price table forever.

Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring
is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls;
the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all
three JSON parsers, and the Gemini stream-schema parser + its test. Gemini
auth detection (env/OAuth/.env) is kept — that's not schema parsing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:34:57 -07:00
co-authored by Claude Opus 4.8
parent 36f972f2e4
commit 7f4574e1d5
12 changed files with 84 additions and 441 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/benchmark-runner.test.ts
+4 -34
View File
@@ -1,44 +1,14 @@
/**
* Unit tests for benchmark pricing + tool-compatibility helpers.
* Unit tests for benchmark tool-compatibility helpers.
*
* Orchestration, scoring, and reporting moved to Braintrust
* (lib/model-benchmark/braintrust-eval.ts); those are covered by
* model-benchmark-braintrust.test.ts and the live e2e suite.
* Orchestration and scoring moved to Braintrust (lib/model-benchmark/braintrust-eval.ts);
* per-provider token/cost tracking was removed with the JSON-schema parsers.
* Provider capability coverage is what's left to pin here.
*/
import { test, expect } from 'bun:test';
import { estimateCostUsd, PRICING } from '../lib/model-benchmark/pricing';
import { missingTools, TOOL_COMPATIBILITY } from './helpers/tool-map';
test('estimateCostUsd returns 0 for unknown model (no crash)', () => {
const cost = estimateCostUsd({ input: 1000, output: 500 }, 'unknown-model-7b');
expect(cost).toBe(0);
});
test('estimateCostUsd computes correctly for known Claude model', () => {
// claude-opus-4-7: $15/MTok input, $75/MTok output
// 1M input + 0.5M output = $15 + $37.50 = $52.50
const cost = estimateCostUsd({ input: 1_000_000, output: 500_000 }, 'claude-opus-4-7');
expect(cost).toBeCloseTo(52.50, 2);
});
test('estimateCostUsd applies cached input discount alongside uncached input', () => {
// tokens.input is uncached-only; tokens.cached is disjoint cache-reads at 10%.
// 0 uncached input, 1M cached → 10% of 15 = $1.50
const cost1 = estimateCostUsd({ input: 0, output: 0, cached: 1_000_000 }, 'claude-opus-4-7');
expect(cost1).toBeCloseTo(1.50, 2);
// 500K uncached input + 500K cached → $7.50 + $0.75 = $8.25
const cost2 = estimateCostUsd({ input: 500_000, output: 0, cached: 500_000 }, 'claude-opus-4-7');
expect(cost2).toBeCloseTo(8.25, 2);
});
test('PRICING table covers the key model families', () => {
expect(PRICING['claude-opus-4-7']).toBeDefined();
expect(PRICING['claude-sonnet-4-6']).toBeDefined();
expect(PRICING['gpt-5.4']).toBeDefined();
expect(PRICING['gemini-2.5-pro']).toBeDefined();
});
test('missingTools reports unsupported tools per provider', () => {
// GPT/Codex doesn't expose Edit, Glob, Grep
expect(missingTools('gpt', ['Edit', 'Glob', 'Grep'])).toEqual(['Edit', 'Glob', 'Grep']);