feat: back model benchmark with Braintrust, drop in-house scoring

Braintrust now owns benchmark scoring, experiments, comparison, and
reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim)
plus operational metrics the CLIs report. runProviderBenchmark wraps each
adapter as a Braintrust Eval task; a deterministic required-terms scorer
replaces the in-house evaluation logic and the optional autoevals ClosedQA
judge replaces judge.ts.

Runs local by default under bun: with no BRAINTRUST_API_KEY it sets
noSendLogs and ships nothing; setting the key opts into the cloud dashboard.
An empty output with zero tokens is thrown so a silent auth/CLI failure
can't masquerade as a 0.0 score.

Deletes runner.ts and judge.ts (and their test-helper re-export shims);
rewires bin/gstack-model-benchmark and adapts the benchmark tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Sinabina
2026-07-21 14:12:10 -07:00
co-authored by Claude Opus 4.8
parent e71e7f7f41
commit 36f972f2e4
12 changed files with 321 additions and 470 deletions
+17
View File
@@ -0,0 +1,17 @@
[
{
"id": "security-review",
"input": "Review a synthetic patch containing command injection, unsafe recursive delete, and path traversal. Do not mutate files. Report only.",
"required": ["command injection", "recursive delete", "path traversal", "no mutation"]
},
{
"id": "graph-indexer-consent",
"input": "Decide whether GStack should require a third-party graph indexer. Justify the install policy.",
"required": ["optional", "consent", "fallback"]
},
{
"id": "resolver-safety",
"input": "Diagnose a resolver that passes an untrusted logical skill name into path.resolve. Recommend the fix.",
"required": ["allowlist", "single segment", "traversal"]
}
]