CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-09-21 20:30:47 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: back model benchmark with Braintrust, drop in-house scoring

Browse Source 
Braintrust now owns benchmark scoring, experiments, comparison, and
reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim)
plus operational metrics the CLIs report. runProviderBenchmark wraps each
adapter as a Braintrust Eval task; a deterministic required-terms scorer
replaces the in-house evaluation logic and the optional autoevals ClosedQA
judge replaces judge.ts.

Runs local by default under bun: with no BRAINTRUST_API_KEY it sets
noSendLogs and ships nothing; setting the key opts into the cloud dashboard.
An empty output with zero tokens is thrown so a silent auth/CLI failure
can't masquerade as a 0.0 score.

Deletes runner.ts and judge.ts (and their test-helper re-export shims);
rewires bin/gstack-model-benchmark and adapts the benchmark tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:12:10 -07:00
co-authored by Claude Opus 4.8
parent e71e7f7f41
commit 36f972f2e4
12 changed files with 321 additions and 470 deletions
Download Patch File Download Diff File Expand all files Collapse all files
evals/model-benchmark/corpus.json
+17
View File
@@ -0,0 +1,17 @@
[
{
"id": "security-review",
"input": "Review a synthetic patch containing command injection, unsafe recursive delete, and path traversal. Do not mutate files. Report only.",
"required": ["command injection", "recursive delete", "path traversal", "no mutation"]
},
{
"id": "graph-indexer-consent",
"input": "Decide whether GStack should require a third-party graph indexer. Justify the install policy.",
"required": ["optional", "consent", "fallback"]
},
{
"id": "resolver-safety",
"input": "Diagnose a resolver that passes an untrusted logical skill name into path.resolve. Recommend the fix.",
"required": ["allowlist", "single segment", "traversal"]
}
]
evals/model-benchmark/gstack.eval.ts
+21
View File
@@ -0,0 +1,21 @@
/**
* GStack model benchmark — Braintrust entrypoint for `bun run` / CI.
*
* Braintrust owns scoring, comparison, and reporting. See
* lib/model-benchmark/braintrust-eval.ts for the core; the user-facing CLI is
* bin/gstack-model-benchmark.
*
* Local run, nothing leaves the machine:
* GSTACK_BENCH_PROVIDER=claude bun run evals/model-benchmark/gstack.eval.ts
*
* Opt into the Braintrust dashboard (consent-gated cloud):
* export BRAINTRUST_API_KEY=...
* for p in claude gpt gemini; do GSTACK_BENCH_PROVIDER=$p bun run evals/model-benchmark/gstack.eval.ts; done
*/
import { loadCorpus, runProviderBenchmark, type ProviderName } from '../../lib/model-benchmark/braintrust-eval';
const provider = (process.env.GSTACK_BENCH_PROVIDER ?? 'claude') as ProviderName;
const timeoutMs = Number(process.env.GSTACK_BENCH_TIMEOUT_MS ?? 300_000);
const judge = process.env.GSTACK_BENCH_JUDGE === '1';
await runProviderBenchmark(provider, loadCorpus(), { timeoutMs, judge });