mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-21 12:20:48 +02:00
SinabinaandClaude Opus 4.8
36f972f2e4
Braintrust now owns benchmark scoring, experiments, comparison, and reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim) plus operational metrics the CLIs report. runProviderBenchmark wraps each adapter as a Braintrust Eval task; a deterministic required-terms scorer replaces the in-house evaluation logic and the optional autoevals ClosedQA judge replaces judge.ts. Runs local by default under bun: with no BRAINTRUST_API_KEY it sets noSendLogs and ships nothing; setting the key opts into the cloud dashboard. An empty output with zero tokens is thrown so a silent auth/CLI failure can't masquerade as a 0.0 score. Deletes runner.ts and judge.ts (and their test-helper re-export shims); rewires bin/gstack-model-benchmark and adapts the benchmark tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
22 lines
985 B
TypeScript
22 lines
985 B
TypeScript
/**
|
|
* GStack model benchmark — Braintrust entrypoint for `bun run` / CI.
|
|
*
|
|
* Braintrust owns scoring, comparison, and reporting. See
|
|
* lib/model-benchmark/braintrust-eval.ts for the core; the user-facing CLI is
|
|
* bin/gstack-model-benchmark.
|
|
*
|
|
* Local run, nothing leaves the machine:
|
|
* GSTACK_BENCH_PROVIDER=claude bun run evals/model-benchmark/gstack.eval.ts
|
|
*
|
|
* Opt into the Braintrust dashboard (consent-gated cloud):
|
|
* export BRAINTRUST_API_KEY=...
|
|
* for p in claude gpt gemini; do GSTACK_BENCH_PROVIDER=$p bun run evals/model-benchmark/gstack.eval.ts; done
|
|
*/
|
|
import { loadCorpus, runProviderBenchmark, type ProviderName } from '../../lib/model-benchmark/braintrust-eval';
|
|
|
|
const provider = (process.env.GSTACK_BENCH_PROVIDER ?? 'claude') as ProviderName;
|
|
const timeoutMs = Number(process.env.GSTACK_BENCH_TIMEOUT_MS ?? 300_000);
|
|
const judge = process.env.GSTACK_BENCH_JUDGE === '1';
|
|
|
|
await runProviderBenchmark(provider, loadCorpus(), { timeoutMs, judge });
|