mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-06 17:04:27 +01:00
Garry Tanandcapy-ai[bot]
635535d3c1
* feat(evals): Opus 5.5 is the paid gate's model under test; judge stays Fable (GSTA-49 TM1) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: re-baseline old-model paid pins onto the current generation (TM3) Repair the prosons capture prompt that tripped the reasoning_extraction classifier, and restructure /setup-browser-cookies so the Aside probe, decisions and target order are unambiguous. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: CHANGELOG entry for the TM3 re-baseline under [Unreleased] Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: stamp v1.91.150.0 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
116 lines
7.2 KiB
Markdown
116 lines
7.2 KiB
Markdown
# Overlay benchmark behavior contract v4
|
|
|
|
The paid overlay fixtures verify supported behavior, with comparative efficacy
|
|
reported separately as research evidence. A passing case does **not** establish
|
|
marginal overlay benefit or resource non-regression. This replaces the previous
|
|
blocking efficacy contract; historical failures retain their original verdicts.
|
|
|
|
The contract is defined by `OVERLAY_CONTRACT` in
|
|
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
|
|
identifies version 4. Eval entry names include `contract-v4` so automated result
|
|
matching cannot silently compare the old and new contracts by the same name.
|
|
Selection IDs for the six retained paid cases stay unchanged.
|
|
|
|
## Blocking checks
|
|
|
|
Both arms use the real Claude Code preset; ON appends the resolved overlay and
|
|
OFF does not. Each retained fixture runs ten samples per arm on its pinned
|
|
model: `claude-opus-5-5`, plus `claude-sonnet-5-5` for the dedicated-tools `-sonnet` case.
|
|
Each model keeps its own results. The `opus-4-7-*` cases keep their overlay
|
|
(`model-overlays/opus-4-7.md`) but run on Opus 5.5 since GSTA-49.
|
|
|
|
Every planned sample in both arms must execute successfully, produce exactly
|
|
one successful native terminal result without an error flag, preserve workspace
|
|
scope, and yield a finite metric within the fixture's bounds. Missing reasoning
|
|
metadata, oracle process errors, partial sampling, deadlines, cleanup failures,
|
|
or recording failures keep the case failed. Missing observations are never
|
|
replaced with zero or discarded to obtain a passing arm.
|
|
|
|
A completed answer that fails the fixture's output check is a measured
|
|
correctness result, not an invalid measurement, in either arm: its record has
|
|
`measurementPassed: true`, `taskCorrect: false` and the check's message in
|
|
`answerError`. Only ON correctness gates the case, so an OFF wrong answer stays
|
|
in the comparison and an ON wrong answer fails the case. Version 3 counted an
|
|
OFF wrong answer as an invalid measurement (census 37198445662); v3 verdicts
|
|
are not reinterpreted.
|
|
|
|
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
|
|
| --- | --- | --- | --- |
|
|
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
|
|
| `opus-4-7-effort-match-trivial` | Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens` |
|
|
| `opus-4-7-literal-interpretation` | Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
|
|
|
|
The dedicated-tools family also has `-sonnet` and `-opus-5-5` cases. Output validation checks the native final
|
|
answer, so a correct word mentioned earlier cannot conceal a wrong final answer.
|
|
The read-only prompts explicitly request the JSON shape. These exact-answer
|
|
prompts are new in v2: compare ON/OFF within the same contract and fixture, not
|
|
across versions. In particular, never compare the current literal correctness
|
|
metric to historical counts of edited files.
|
|
|
|
OFF literal samples can validly complete 0..3 behaviors; that is the measured
|
|
control variable. OFF dedicated-tool samples can validly use Bash. ON must meet
|
|
the exact requirements above even when its comparison shows a large improvement;
|
|
for the dedicated-tools case the 20% Bash comparison itself is part of the gate.
|
|
Scope is checked again after the implementation oracle because importing a
|
|
repaired module can itself change files. Each oracle must emit its unique
|
|
completion marker after its assertions; an implementation exiting zero during
|
|
import cannot masquerade as a passing behavior.
|
|
|
|
## Comparative research results
|
|
|
|
The original numeric comparators remain unchanged: at least 20% fewer Bash calls
|
|
or reported reasoning tokens, and at least 20% more correct literal targets.
|
|
Their `criterionMet` values, means, and raw per-arm metrics remain in aggregate
|
|
artifacts. They do not decide the v2 behavior verdict.
|
|
|
|
| Comparison status | Meaning |
|
|
| --- | --- |
|
|
| `improved` | The original numeric criterion was met |
|
|
| `baseline_saturated` | Every OFF sample reached the metric's optimum; no demonstrated marginal benefit |
|
|
| `no_measured_improvement` | Complete measurements did not meet the original criterion |
|
|
| `regressed` | ON's mean was worse in the metric's declared direction |
|
|
| `unsupported_hypothesis` | The resolved overlay does not contain the claimed intervention |
|
|
| `incomplete` | Missing or invalid measurements; the behavior case also fails |
|
|
|
|
Equality at the optimum is behavior success with `criterionMet: false`, never a
|
|
percentage improvement. An effort case can pass correctness while its measured
|
|
reasoning-token comparison is `regressed`; that regression remains visible.
|
|
The product asks models to match effort to complexity and does not promise a
|
|
universal 20% token reduction. There is no calibrated resource ceiling,
|
|
non-inferiority margin, or powered sampling scheme in this contract. Any future
|
|
resource regression gate needs a separately agreed practical margin, baseline
|
|
distribution, sampling design, and decision rule before collecting new results.
|
|
|
|
## Retired fanout cases
|
|
|
|
The current resolved `model-overlays/opus-4-7.md` and its inherited `claude.md`
|
|
contain no fanout instruction. The paid cases `opus-4-7-fanout-toy`,
|
|
`opus-4-7-fanout-realistic`, and their `-sonnet` variants are retired, including
|
|
their paid wrappers and selection entries. They are not renamed successes.
|
|
Reintroducing a paid batching efficacy experiment requires an actual product
|
|
claim and a discriminating experiment.
|
|
|
|
Free regressions retain native first-message fragment grouping, tool-ID
|
|
deduplication, and the original fanout comparator. This metric measures message
|
|
batching and cannot establish wall-clock execution concurrency.
|
|
|
|
## Evidence and execution
|
|
|
|
Run directories and Bun/native attempt identities remain unique. Artifact files
|
|
use exclusive creation; retry attempts cannot overwrite earlier raw streams,
|
|
measurement records, workspace snapshots, or aggregate verdicts. Native events
|
|
are saved before yielding them, including streams that later throw. A validated
|
|
measurement records once; a separate aggregate records the case verdict with
|
|
zero additive cost and time. Failed writes retain partial evidence and fail the
|
|
case. Versioning never rewrites historical artifacts.
|
|
|
|
The paid runner executes one overlay wrapper per shard and one wrapper at a
|
|
time, disables Bun retries, and preserves native rate-limit retry evidence. At
|
|
finalization, workers must settle before workspace cleanup begins. Both share
|
|
the existing five-second grace; expiry fails the case and prevents late workers
|
|
from starting cleanup. Each fixture keeps its original turn cap, concurrency,
|
|
30-minute work budget, and recording grace. The six
|
|
wrappers remain periodic-tier cases; free tests exercise deterministic outcomes
|
|
without making model calls. A free pass validates the instrument, not live model
|
|
behavior or a positive efficacy claim.
|