The adapters parsed each vendor's proprietary JSON stream (Claude json,
Codex JSONL, Gemini stream-json) to extract tokens/tool-calls, and a
per-model pricing table turned tokens into cost. That coupling was the
brittle hardcoding GStack 2 exists to avoid — it broke every time a vendor
reshuffled its output, and it duplicated what any tool that instruments the
real model call already does. Braintrust owns scoring; it can't see a CLI
subprocess's tokens anyway, so computing cost ourselves meant maintaining
both a parser and a price table forever.
Now each adapter runs the CLI in plain-text mode and returns stdout. Scoring
is unchanged (Braintrust reads the text). RunResult drops tokens/toolCalls;
the comparison table drops the Tokens/Cost columns. Deletes pricing.ts, all
three JSON parsers, and the Gemini stream-schema parser + its test. Gemini
auth detection (env/OAuth/.env) is kept — that's not schema parsing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
JSON-driven XCUITest runner plus SwiftUI/UIKit fixture apps that drive a
real app by bundle id. The runner resolves the actual switch inside a
SwiftUI Toggle wrapper, and when a center tap misses the full-width row it
retries on the control's trailing edge via an element-anchored normalized
offset (adaptive, not a raw screen coordinate), then verifies the switch
value actually changed. A tap that silently does nothing is now reported
as an interaction failure, not a false product defect. Validated on the
simulator and on a physical iPhone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Standalone planner that turns the IOSQAFlow JSON contract into an
xcodebuild XCUITest invocation for simulator or physical device, using
semantic selectors (identifier/role/label) and no coordinate taps.
Pure plan generation; covered by 7 unit tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A capability whose binary launches while the managed runtime hard-fails
(e.g. skill-API mismatch) was reported as `degraded`/ok:true/exit 0, diverging
from plain `gstack doctor` (ok:false/exit 1) for the identical report. Split
the branch so runtime `warn` stays `degraded` and runtime `fail` maps to
`failed`, and add a regression test for the runtime-fail + capability-pass case.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Braintrust now owns benchmark scoring, experiments, comparison, and
reporting. GStack keeps only the CLI-agent adapters (the unavoidable shim)
plus operational metrics the CLIs report. runProviderBenchmark wraps each
adapter as a Braintrust Eval task; a deterministic required-terms scorer
replaces the in-house evaluation logic and the optional autoevals ClosedQA
judge replaces judge.ts.
Runs local by default under bun: with no BRAINTRUST_API_KEY it sets
noSendLogs and ships nothing; setting the key opts into the cloud dashboard.
An empty output with zero tokens is thrown so a silent auth/CLI failure
can't masquerade as a 0.0 score.
Deletes runner.ts and judge.ts (and their test-helper re-export shims);
rewires bin/gstack-model-benchmark and adapts the benchmark tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The benchmark adapter only recognized GOOGLE_API_KEY and parsed the legacy
text/usage event shape. Recognize GEMINI_API_KEY and ~/.gemini/.env, parse
the current content/stats schema, ignore echoed user messages, and run
report-only plan mode with --skip-trust for disposable workspaces.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The contract previously named Context.dev the only authorized external
service and barred provider marketplaces. Loosen it so optional, off-by-
default, consent-gated eval backends (Braintrust and the like) are allowed;
keep the consent + typed-failure hygiene.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backing the model benchmark with Braintrust's local eval runner and its
autoevals scorer library instead of an in-house scoring engine.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New advisories landed after the integration work: fast-uri (High) and hono
(3x Medium) picked up fixed releases, and a Medium surfaced on
@hono/node-server. Bump the fast-uri and hono override pins to their fixed
patch releases (3.1.3, 4.12.27), clearing four findings.
The remaining @hono/node-server advisory is reachable only through the unused
@modelcontextprotocol/sdk transitive (no source imports it, no Hono server is
started); its only fix is a major bump the SDK pins against. Record that
assessment in .osv-scanner.toml so the scheduled scan stays honest instead of
alarming on an unreachable path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Create the browser smoke page under GITHUB_WORKSPACE so the runtime's local-file trust boundary permits it on every runner, and advance the immutable bootstrap channel to RC3.
Use an allowed local file URL for the browser smoke test, centralize keyless archive signing after native builds so Windows ARM64 does not require an unavailable Cosign binary, and advance the immutable bootstrap channel to RC2.
Publish signed prerelease artifacts from v2.0.0-rc.* tags, bind bootstrap trust to the immutable RC tag, make missing-release errors actionable, and install the six public skills from the canonical subpath.
Detect host-native browser tools before offering the isolated local Chromium fallback, add a common readiness fixture, harden managed browser startup, and verify standards installs expose one canonical QA skill.
Sinabina