CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-21 12:20:48 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: back model benchmark with Braintrust, drop in-house scoring

Browse Source 
Braintrust now owns benchmark scoring, experiments, comparison, and
reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim)
plus operational metrics the CLIs report. runProviderBenchmark wraps each
adapter as a Braintrust Eval task; a deterministic required-terms scorer
replaces the in-house evaluation logic and the optional autoevals ClosedQA
judge replaces judge.ts.

Runs local by default under bun: with no BRAINTRUST_API_KEY it sets
noSendLogs and ships nothing; setting the key opts into the cloud dashboard.
An empty output with zero tokens is thrown so a silent auth/CLI failure
can't masquerade as a 0.0 score.

Deletes runner.ts and judge.ts (and their test-helper re-export shims);
rewires bin/gstack-model-benchmark and adapts the benchmark tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:12:10 -07:00
co-authored by Claude Opus 4.8
parent e71e7f7f41
commit 36f972f2e4
12 changed files with 321 additions and 470 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/benchmark-cli.test.ts
+18 -7
View File
@@ -1,7 +1,7 @@
/**
* gstack-model-benchmark CLI tests (offline).
*
* Covers CLI wiring that unit tests against benchmark-runner.ts can't see:
* Covers CLI wiring that unit tests can't see:
* - --dry-run auth/provider-list resolution
* - unknown provider WARN path
* - provider default (claude) when --models omitted
@@ -66,11 +66,22 @@ describe('gstack-model-benchmark --dry-run', () => {
expect(r.stdout).toContain('providers: claude');
});
test('--timeout-ms and --workdir flags flow through to dry-run report', () => {
const r = run(['--prompt', 'hi', '--timeout-ms', '9999', '--workdir', '/tmp', '--dry-run']);
test('--timeout-ms flows through to dry-run report', () => {
const r = run(['--prompt', 'hi', '--timeout-ms', '9999', '--dry-run']);
expect(r.status).toBe(0);
expect(r.stdout).toContain('timeout_ms: 9999');
expect(r.stdout).toContain('workdir: /tmp');
});
test('no prompt falls back to the corpus (not an error)', () => {
const r = run(['--models', 'claude', '--dry-run']);
expect(r.status).toBe(0);
expect(r.stdout).toMatch(/corpus:\s+\d+ case/);
});
test('upload is off by default (local only, nothing uploaded)', () => {
const r = run(['--prompt', 'hi', '--dry-run'], { env: { BRAINTRUST_API_KEY: '' } });
expect(r.status).toBe(0);
expect(r.stdout).toContain('local only');
});
test('--judge flag reported in dry-run output', () => {
@@ -196,9 +207,9 @@ describe('gstack-model-benchmark prompt resolution', () => {
expect(r.stdout).toContain('treat-me-as-inline');
});
test('missing prompt exits non-zero', () => {
test('no positional and no --prompt runs the corpus', () => {
const r = run(['--dry-run']);
expect(r.status).not.toBe(0);
expect(r.stderr).toContain('specify a prompt');
expect(r.status).toBe(0);
expect(r.stdout).toMatch(/corpus:\s+\d+ case/);
});
});