gstack/docs/OVERLAY_BENCHMARK_CONTRACT.md

# Overlay benchmark behavior contract v4

The paid overlay fixtures verify supported behavior, with comparative efficacy
reported separately as research evidence. A passing case does **not** establish
marginal overlay benefit or resource non-regression. This replaces the previous
blocking efficacy contract; historical failures retain their original verdicts.

The contract is defined by `OVERLAY_CONTRACT` in
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
identifies version 4. Eval entry names include `contract-v4` so automated result
matching cannot silently compare the old and new contracts by the same name.
Selection IDs for the six retained paid cases stay unchanged.

## Blocking checks

Both arms use the real Claude Code preset; ON appends the resolved overlay and
OFF does not. Each retained fixture runs ten samples per arm on its pinned
model: `claude-opus-5-5`, plus `claude-sonnet-5-5` for the dedicated-tools `-sonnet` case.
Each model keeps its own results. The `opus-4-7-*` cases keep their overlay
(`model-overlays/opus-4-7.md`) but run on Opus 5.5 since GSTA-49.

Every planned sample in both arms must execute successfully, produce exactly
one successful native terminal result without an error flag, preserve workspace
scope, and yield a finite metric within the fixture's bounds. Missing reasoning
metadata, oracle process errors, partial sampling, deadlines, cleanup failures,
or recording failures keep the case failed. Missing observations are never
replaced with zero or discarded to obtain a passing arm.

A completed answer that fails the fixture's output check is a measured
correctness result, not an invalid measurement, in either arm: its record has
`measurementPassed: true`, `taskCorrect: false` and the check's message in
`answerError`. Only ON correctness gates the case, so an OFF wrong answer stays
in the comparison and an ON wrong answer fails the case. Version 3 counted an
OFF wrong answer as an invalid measurement (census 37198445662); v3 verdicts
are not reinterpreted.

| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
| --- | --- | --- | --- |
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
| `opus-4-7-effort-match-trivial` | Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens` |
| `opus-4-7-literal-interpretation` | Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |

The dedicated-tools family also has `-sonnet` and `-opus-5-5` cases. Output validation checks the native final
answer, so a correct word mentioned earlier cannot conceal a wrong final answer.
The read-only prompts explicitly request the JSON shape. These exact-answer
prompts are new in v2: compare ON/OFF within the same contract and fixture, not
across versions. In particular, never compare the current literal correctness
metric to historical counts of edited files.

OFF literal samples can validly complete 0..3 behaviors; that is the measured
control variable. OFF dedicated-tool samples can validly use Bash. ON must meet
the exact requirements above even when its comparison shows a large improvement;
for the dedicated-tools case the 20% Bash comparison itself is part of the gate.
Scope is checked again after the implementation oracle because importing a
repaired module can itself change files. Each oracle must emit its unique
completion marker after its assertions; an implementation exiting zero during
import cannot masquerade as a passing behavior.

## Comparative research results

The original numeric comparators remain unchanged: at least 20% fewer Bash calls
or reported reasoning tokens, and at least 20% more correct literal targets.
Their `criterionMet` values, means, and raw per-arm metrics remain in aggregate
artifacts. They do not decide the v2 behavior verdict.

| Comparison status | Meaning |
| --- | --- |
| `improved` | The original numeric criterion was met |
| `baseline_saturated` | Every OFF sample reached the metric's optimum; no demonstrated marginal benefit |
| `no_measured_improvement` | Complete measurements did not meet the original criterion |
| `regressed` | ON's mean was worse in the metric's declared direction |
| `unsupported_hypothesis` | The resolved overlay does not contain the claimed intervention |
| `incomplete` | Missing or invalid measurements; the behavior case also fails |

Equality at the optimum is behavior success with `criterionMet: false`, never a
percentage improvement. An effort case can pass correctness while its measured
reasoning-token comparison is `regressed`; that regression remains visible.
The product asks models to match effort to complexity and does not promise a
universal 20% token reduction. There is no calibrated resource ceiling,
non-inferiority margin, or powered sampling scheme in this contract. Any future
resource regression gate needs a separately agreed practical margin, baseline
distribution, sampling design, and decision rule before collecting new results.

## Retired fanout cases

The current resolved `model-overlays/opus-4-7.md` and its inherited `claude.md`
contain no fanout instruction. The paid cases `opus-4-7-fanout-toy`,
`opus-4-7-fanout-realistic`, and their `-sonnet` variants are retired, including
their paid wrappers and selection entries. They are not renamed successes.
Reintroducing a paid batching efficacy experiment requires an actual product
claim and a discriminating experiment.

Free regressions retain native first-message fragment grouping, tool-ID
deduplication, and the original fanout comparator. This metric measures message
batching and cannot establish wall-clock execution concurrency.

## Evidence and execution

Run directories and Bun/native attempt identities remain unique. Artifact files
use exclusive creation; retry attempts cannot overwrite earlier raw streams,
measurement records, workspace snapshots, or aggregate verdicts. Native events
are saved before yielding them, including streams that later throw. A validated
measurement records once; a separate aggregate records the case verdict with
zero additive cost and time. Failed writes retain partial evidence and fail the
case. Versioning never rewrites historical artifacts.

The paid runner executes one overlay wrapper per shard and one wrapper at a
time, disables Bun retries, and preserves native rate-limit retry evidence. At
finalization, workers must settle before workspace cleanup begins. Both share
the existing five-second grace; expiry fails the case and prevents late workers
from starting cleanup. Each fixture keeps its original turn cap, concurrency,
30-minute work budget, and recording grace. The six
wrappers remain periodic-tier cases; free tests exercise deterministic outcomes
without making model calls. A free pass validates the instrument, not live model
behavior or a positive efficacy claim.