The plan's metric order is diff-quality FIRST, but cells never ran the
fixtures' own run-tests.js, so a refusal, a broken implementation, and
working code were indistinguishable in aggregates (Codex adversarial catch).
Tasks with an oracle declare checkCmd; every cell records checks=pass|fail|none
in the report line and eval store. Selftest pins the oracle declarations and
that the planted bug fails its own check pre-fix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The selftest lived inside the paid skill-e2e-* file, so fixture-integrity
and plumbing pins executed weekly at best — a broken fixture would ship past
every gating check and be discovered when the periodic run burned money on a
dead instrument. Harness extracted to test/helpers/arm-benchmark-harness.ts,
selftest to test/arm-benchmark-selftest.test.ts (free suite). Touchfiles:
harness added to the three benchmark dep lists; the auq-repetition-cut-ab
tier comment now states the MANUAL re-run obligation honestly (periodic runs
force EVALS_ALL, so dep lists cannot auto-trigger it).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Harvest diffs against the recorded seed SHA (origin/main is movable by an
agent that commits AND pushes; a recorded SHA is not).
- Fixtures get a node_modules .gitignore and the git wrapper a 64MB
maxBuffer, so a vendored-dependency arm is scored instead of killing the
cell.
- The judge diff cap is a named constant with loud truncation (log +
judge_reasoning suffix).
- Judge prompt block markers carry a per-call random sentinel, so a diff
containing a faked closing marker cannot escape the data block.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Ponytail's honest-benchmark method pointed at gstack itself: 3 build-shaped
tasks (native-platform over-build trap, CRUD endpoint, bug fix with planted
decoys) x 2 arms, real claude -p sessions, scored on the git diff left
behind. A research instrument, not a release gate — no assertion compares
arm scores.
Arms use the PROVEN project-scope pattern: the with-arm installs a
build-discipline skill (extracted reuse-ladder + bounded-closer content, not
whole-file copies) into the fixture's .claude/skills/ with a CLAUDE.md
routing line and an explicit invocation; a live spike confirmed claude -p
discovers and invokes project-scope skills via the Skill tool (3 turns,
exact-output probe). Fixtures are git init + local bare origin; diff capture
is three lines of git, no worktree machinery.
Failure taxonomy: zero-diff arms are VALID scored cells (deterministic
0/none, no API call), harvest failures record harvest:null, judge_error
cells are excluded from aggregates but named in the report — nothing drops
silently. armJudge: fixed sonnet judge, 0-3 unrequested-structure rubric,
must name the construct or say none, bounded retry-on-malformed; callJudge
gains optional temperature/max_tokens (defaults unchanged). recordE2E now
populates tokens_used for every E2E. Eval schema v2: harvest gains
{insertions, deletions, net}, tolerant reads keep v1 runs comparable.
Registered periodic in E2E_TIERS + touchfiles (with the auq-repetition-cut
A/B); periodic detach timeout raised to the new shard-census floor. Free
selftest (8 tests, zero API) pins fixtures, extraction, arm asymmetry, diff
capture, judge plumbing, and the retry bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
