CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-21 12:20:48 +02:00
Code Issues Packages Projects Releases Wiki Activity

refactor: capture plain-text CLI output, drop hardcoded provider schemas

Browse Source 
The adapters parsed each vendor's proprietary JSON stream (Claude json,
Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a
per-model pricing table turned tokens into cost. That coupling was the
brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor
reshuffled its output, and it duplicated what any tool that instruments the
real model call already does. Braintrust owns scoring; it can't see a CLI
subprocess's tokens anyway, so computing cost ourselves meant maintaining
both a parser and a price table forever.

Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring
is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls;
the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all
three JSON parsers, and the Gemini stream-schema parser + its test. Gemini
auth detection (env/OAuth/.env) is kept — that's not schema parsing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:34:57 -07:00
co-authored by Claude Opus 4.8
parent 36f972f2e4
commit 7f4574e1d5
12 changed files with 84 additions and 441 deletions
Download Patch File Download Diff File Expand all files Collapse all files
lib/model-benchmark/braintrust-eval.ts
+7 -17
View File
@@ -40,14 +40,10 @@ export interface BenchOpts {
judge?: boolean;
}
/** Per-case operational metrics the adapter reports — Braintrust doesn't time the CLI for us. */
/** Per-case operational metric Braintrust doesn't capture for a CLI subprocess: wall-clock. */
export interface CaseOps {
id: string;
durationMs: number;
tokensIn: number;
tokensOut: number;
costUsd: number;
toolCalls: number;
modelUsed: string;
error?: string;
}
@@ -126,23 +122,17 @@ export async function runProviderBenchmark(
const workdir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-bench-'));
try {
const r = await adapter.run({ prompt: input, workdir, timeoutMs });
const costUsd = adapter.estimateCost(r.tokens, r.modelUsed);
const op: CaseOps = {
ops.push({
id: c?.id ?? input.slice(0, 24),
durationMs: r.durationMs,
tokensIn: r.tokens.input,
tokensOut: r.tokens.output,
costUsd,
toolCalls: r.toolCalls,
modelUsed: r.modelUsed,
error: r.error ? `${r.error.code}: ${r.error.reason}` : undefined,
};
ops.push(op);
});
if (r.error) throw new Error(`${provider} failed: ${r.error.code} — ${r.error.reason}`);
// Empty output + zero tokens = provider never answered (silent auth/CLI failure).
// Fail loud so it can't masquerade as a 0.0 score.
if (!r.output.trim() && r.tokens.input === 0 && r.tokens.output === 0) {
throw new Error(`${provider} returned empty output with zero tokens (likely auth/CLI failure, not a real result)`);
// Empty output = provider never answered (silent auth/CLI failure). Fail
// loud so it can't masquerade as a 0.0 score.
if (!r.output.trim()) {
throw new Error(`${provider} returned empty output (likely auth/CLI failure, not a real result)`);
}
return r.output;
} finally {