Garry Tanandcapy-ai[bot]
635535d3c1
* feat(evals): Opus 5.5 is the paid gate's model under test; judge stays Fable (GSTA-49 TM1) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: re-baseline old-model paid pins onto the current generation (TM3) Repair the prosons capture prompt that tripped the reasoning_extraction classifier, and restructure /setup-browser-cookies so the Aside probe, decisions and target order are unambiguous. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG entry for the TM3 re-baseline under [Unreleased] Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: stamp v1.91.150.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
7.2 KiB
Overlay benchmark behavior contract v4
The paid overlay fixtures verify supported behavior, with comparative efficacy reported separately as research evidence. A passing case does not establish marginal overlay benefit or resource non-regression. This replaces the previous blocking efficacy contract; historical failures retain their original verdicts.
The contract is defined by OVERLAY_CONTRACT in
test/helpers/overlay-case-policy.ts. Every new measurement and aggregate record
identifies version 4. Eval entry names include contract-v4 so automated result
matching cannot silently compare the old and new contracts by the same name.
Selection IDs for the six retained paid cases stay unchanged.
Blocking checks
Both arms use the real Claude Code preset; ON appends the resolved overlay and
OFF does not. Each retained fixture runs ten samples per arm on its pinned
model: claude-opus-5-5, plus claude-sonnet-5-5 for the dedicated-tools -sonnet case.
Each model keeps its own results. The opus-4-7-* cases keep their overlay
(model-overlays/opus-4-7.md) but run on Opus 5.5 since GSTA-49.
Every planned sample in both arms must execute successfully, produce exactly one successful native terminal result without an error flag, preserve workspace scope, and yield a finite metric within the fixture's bounds. Missing reasoning metadata, oracle process errors, partial sampling, deadlines, cleanup failures, or recording failures keep the case failed. Missing observations are never replaced with zero or discarded to obtain a passing arm.
A completed answer that fails the fixture's output check is a measured
correctness result, not an invalid measurement, in either arm: its record has
measurementPassed: true, taskCorrect: false and the check's message in
answerError. Only ON correctness gates the case, so an OFF wrong answer stays
in the comparison and an ON wrong answer fails the case. Version 3 counted an
OFF wrong answer as an invalid measurement (census 37198445662); v3 verdicts
are not reinterpreted.
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
|---|---|---|---|
claude-dedicated-tools-vs-bash |
Read-only workspace; native final JSON maps all five src/ paths to their exact exported symbol names |
Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
opus-4-7-effort-match-trivial |
Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native usage.output_tokens_details.thinking_tokens |
opus-4-7-literal-interpretation |
Only src/auth.ts, src/billing.ts, and src/notifications.ts may change; public tests stay frozen; independent behavior checks run outside the writable fixture |
All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
The dedicated-tools family also has -sonnet and -opus-5-5 cases. Output validation checks the native final
answer, so a correct word mentioned earlier cannot conceal a wrong final answer.
The read-only prompts explicitly request the JSON shape. These exact-answer
prompts are new in v2: compare ON/OFF within the same contract and fixture, not
across versions. In particular, never compare the current literal correctness
metric to historical counts of edited files.
OFF literal samples can validly complete 0..3 behaviors; that is the measured control variable. OFF dedicated-tool samples can validly use Bash. ON must meet the exact requirements above even when its comparison shows a large improvement; for the dedicated-tools case the 20% Bash comparison itself is part of the gate. Scope is checked again after the implementation oracle because importing a repaired module can itself change files. Each oracle must emit its unique completion marker after its assertions; an implementation exiting zero during import cannot masquerade as a passing behavior.
Comparative research results
The original numeric comparators remain unchanged: at least 20% fewer Bash calls
or reported reasoning tokens, and at least 20% more correct literal targets.
Their criterionMet values, means, and raw per-arm metrics remain in aggregate
artifacts. They do not decide the v2 behavior verdict.
| Comparison status | Meaning |
|---|---|
improved |
The original numeric criterion was met |
baseline_saturated |
Every OFF sample reached the metric's optimum; no demonstrated marginal benefit |
no_measured_improvement |
Complete measurements did not meet the original criterion |
regressed |
ON's mean was worse in the metric's declared direction |
unsupported_hypothesis |
The resolved overlay does not contain the claimed intervention |
incomplete |
Missing or invalid measurements; the behavior case also fails |
Equality at the optimum is behavior success with criterionMet: false, never a
percentage improvement. An effort case can pass correctness while its measured
reasoning-token comparison is regressed; that regression remains visible.
The product asks models to match effort to complexity and does not promise a
universal 20% token reduction. There is no calibrated resource ceiling,
non-inferiority margin, or powered sampling scheme in this contract. Any future
resource regression gate needs a separately agreed practical margin, baseline
distribution, sampling design, and decision rule before collecting new results.
Retired fanout cases
The current resolved model-overlays/opus-4-7.md and its inherited claude.md
contain no fanout instruction. The paid cases opus-4-7-fanout-toy,
opus-4-7-fanout-realistic, and their -sonnet variants are retired, including
their paid wrappers and selection entries. They are not renamed successes.
Reintroducing a paid batching efficacy experiment requires an actual product
claim and a discriminating experiment.
Free regressions retain native first-message fragment grouping, tool-ID deduplication, and the original fanout comparator. This metric measures message batching and cannot establish wall-clock execution concurrency.
Evidence and execution
Run directories and Bun/native attempt identities remain unique. Artifact files use exclusive creation; retry attempts cannot overwrite earlier raw streams, measurement records, workspace snapshots, or aggregate verdicts. Native events are saved before yielding them, including streams that later throw. A validated measurement records once; a separate aggregate records the case verdict with zero additive cost and time. Failed writes retain partial evidence and fail the case. Versioning never rewrites historical artifacts.
The paid runner executes one overlay wrapper per shard and one wrapper at a time, disables Bun retries, and preserves native rate-limit retry evidence. At finalization, workers must settle before workspace cleanup begins. Both share the existing five-second grace; expiry fails the case and prevents late workers from starting cleanup. Each fixture keeps its original turn cap, concurrency, 30-minute work budget, and recording grace. The six wrappers remain periodic-tier cases; free tests exercise deterministic outcomes without making model calls. A free pass validates the instrument, not live model behavior or a positive efficacy claim.