mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-22 21:00:51 +02:00
feat: back model benchmark with Braintrust, drop in-house scoring
Braintrust now owns benchmark scoring, experiments, comparison, and reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim) plus operational metrics the CLIs report. runProviderBenchmark wraps each adapter as a Braintrust Eval task; a deterministic required-terms scorer replaces the in-house evaluation logic and the optional autoevals ClosedQA judge replaces judge.ts. Runs local by default under bun: with no BRAINTRUST_API_KEY it sets noSendLogs and ships nothing; setting the key opts into the cloud dashboard. An empty output with zero tokens is thrown so a silent auth/CLI failure can't masquerade as a 0.0 score. Deletes runner.ts and judge.ts (and their test-helper re-export shims); rewires bin/gstack-model-benchmark and adapts the benchmark tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
e71e7f7f41
commit
36f972f2e4
@@ -42,12 +42,8 @@ test('production modules do not import from test directories', () => {
|
||||
|
||||
test('former test-helper paths re-export the production benchmark API', async () => {
|
||||
const [
|
||||
runner,
|
||||
helperRunner,
|
||||
pricing,
|
||||
helperPricing,
|
||||
judge,
|
||||
helperJudge,
|
||||
claude,
|
||||
helperClaude,
|
||||
gpt,
|
||||
@@ -55,12 +51,8 @@ test('former test-helper paths re-export the production benchmark API', async ()
|
||||
gemini,
|
||||
helperGemini,
|
||||
] = await Promise.all([
|
||||
import('../lib/model-benchmark/runner'),
|
||||
import('./helpers/benchmark-runner'),
|
||||
import('../lib/model-benchmark/pricing'),
|
||||
import('./helpers/pricing'),
|
||||
import('../lib/model-benchmark/judge'),
|
||||
import('./helpers/benchmark-judge'),
|
||||
import('../lib/model-benchmark/providers/claude'),
|
||||
import('./helpers/providers/claude'),
|
||||
import('../lib/model-benchmark/providers/gpt'),
|
||||
@@ -69,9 +61,7 @@ test('former test-helper paths re-export the production benchmark API', async ()
|
||||
import('./helpers/providers/gemini'),
|
||||
]);
|
||||
|
||||
expect(helperRunner.runBenchmark).toBe(runner.runBenchmark);
|
||||
expect(helperPricing.estimateCostUsd).toBe(pricing.estimateCostUsd);
|
||||
expect(helperJudge.judgeEntries).toBe(judge.judgeEntries);
|
||||
expect(helperClaude.ClaudeAdapter).toBe(claude.ClaudeAdapter);
|
||||
expect(helperGpt.GptAdapter).toBe(gpt.GptAdapter);
|
||||
expect(helperGemini.GeminiAdapter).toBe(gemini.GeminiAdapter);
|
||||
|
||||
Reference in New Issue
Block a user