docs: overlay benchmark contract v3 (dedicated-tools gate = correct output + 20% fewer Bash)

This commit is contained in:
garrytan committed 2026-10-03 00:46:58 +00:00
1 parent 522e300d4e
commit b54b1e4d97
1 file changed
+4 -3
+4 -3
View File
@@ -7,7 +7,7 @@ blocking efficacy contract; historical failures retain their original verdicts.
The contract is defined by `OVERLAY_CONTRACT` in
`test/helpers/overlay-case-policy.ts`. Every new measurement and aggregate record
identifies version 2. Eval entry names include `contract-v2` so automated result
identifies version 3. Eval entry names include `contract-v3` so automated result
matching cannot silently compare the old and new contracts by the same name.
Selection IDs for the six retained paid cases stay unchanged.
@@ -26,7 +26,7 @@ observations are never replaced with zero or discarded to obtain a passing arm.
| Fixture family | Exact task and scope contract | Additional ON requirement | Measurement |
| --- | --- | --- | --- |
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Zero Bash calls in every ON sample | Total Bash tool calls |
| `claude-dedicated-tools-vs-bash` | Read-only workspace; native final JSON maps all five `src/` paths to their exact exported symbol names | Correct output in every ON sample, and at least 20% fewer Bash calls than OFF (when the OFF median is 0, zero Bash in every ON sample) | Total Bash tool calls |
| `opus-4-7-effort-match-trivial` | Read-only workspace; native final JSON contains the exact version string | Correct answer in every ON sample | Native `usage.output_tokens_details.thinking_tokens` |
| `opus-4-7-literal-interpretation` | Only `src/auth.ts`, `src/billing.ts`, and `src/notifications.ts` may change; public tests stay frozen; independent behavior checks run outside the writable fixture | All three target behaviors pass in every ON sample | Correct target behaviors, 0..3 |
@@ -39,7 +39,8 @@ metric to historical counts of edited files.
OFF literal samples can validly complete 0..3 behaviors; that is the measured
control variable. OFF dedicated-tool samples can validly use Bash. ON must meet
the exact requirements above even when its comparison shows a large improvement.
the exact requirements above even when its comparison shows a large improvement;
for the dedicated-tools case the 20% Bash comparison itself is part of the gate.
Scope is checked again after the implementation oracle because importing a
repaired module can itself change files. Each oracle must emit its unique
completion marker after its assertions; an implementation exiting zero during