mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-20 20:00:45 +02:00
SinabinaandClaude Opus 4.8
7f4574e1d5
The adapters parsed each vendor's proprietary JSON stream (Claude json, Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a per-model pricing table turned tokens into cost. That coupling was the brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor reshuffled its output, and it duplicated what any tool that instruments the real model call already does. Braintrust owns scoring; it can't see a CLI subprocess's tokens anyway, so computing cost ourselves meant maintaining both a parser and a price table forever. Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls; the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all three JSON parsers, and the Gemini stream-schema parser + its test. Gemini auth detection (env/OAuth/.env) is kept — that's not schema parsing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
26 lines
1.1 KiB
TypeScript
26 lines
1.1 KiB
TypeScript
/**
|
|
* Unit tests for benchmark tool-compatibility helpers.
|
|
*
|
|
* Orchestration and scoring moved to Braintrust (lib/model-benchmark/braintrust-eval.ts);
|
|
* per-provider token/cost tracking was removed with the JSON-schema parsers.
|
|
* Provider capability coverage is what's left to pin here.
|
|
*/
|
|
|
|
import { test, expect } from 'bun:test';
|
|
import { missingTools, TOOL_COMPATIBILITY } from './helpers/tool-map';
|
|
|
|
test('missingTools reports unsupported tools per provider', () => {
|
|
// GPT/Codex doesn't expose Edit, Glob, Grep
|
|
expect(missingTools('gpt', ['Edit', 'Glob', 'Grep'])).toEqual(['Edit', 'Glob', 'Grep']);
|
|
// Claude supports all core tools
|
|
expect(missingTools('claude', ['Edit', 'Glob', 'Grep', 'Bash', 'Read'])).toEqual([]);
|
|
// Gemini has very limited agentic surface
|
|
expect(missingTools('gemini', ['Bash', 'Edit'])).toEqual(['Bash', 'Edit']);
|
|
});
|
|
|
|
test('TOOL_COMPATIBILITY is populated for all three families', () => {
|
|
expect(TOOL_COMPATIBILITY.claude).toBeDefined();
|
|
expect(TOOL_COMPATIBILITY.gpt).toBeDefined();
|
|
expect(TOOL_COMPATIBILITY.gemini).toBeDefined();
|
|
});
|