Garry Tan and Claude Fable 5
4c20eca33b
feat(evals): with-skill vs without-skill arm benchmark — measures whether gstack's behavioral layer earns its tokens
...
Ponytail's honest-benchmark method pointed at gstack itself: 3 build-shaped
tasks (native-platform over-build trap, CRUD endpoint, bug fix with planted
decoys) x 2 arms, real claude -p sessions, scored on the git diff left
behind. A research instrument, not a release gate — no assertion compares
arm scores.
Arms use the PROVEN project-scope pattern: the with-arm installs a
build-discipline skill (extracted reuse-ladder + bounded-closer content, not
whole-file copies) into the fixture's .claude/skills/ with a CLAUDE.md
routing line and an explicit invocation; a live spike confirmed claude -p
discovers and invokes project-scope skills via the Skill tool (3 turns,
exact-output probe). Fixtures are git init + local bare origin; diff capture
is three lines of git, no worktree machinery.
Failure taxonomy: zero-diff arms are VALID scored cells (deterministic
0/none, no API call), harvest failures record harvest:null, judge_error
cells are excluded from aggregates but named in the report — nothing drops
silently. armJudge: fixed sonnet judge, 0-3 unrequested-structure rubric,
must name the construct or say none, bounded retry-on-malformed; callJudge
gains optional temperature/max_tokens (defaults unchanged). recordE2E now
populates tokens_used for every E2E. Eval schema v2: harvest gains
{insertions, deletions, net}, tolerant reads keep v1 runs comparable.
Registered periodic in E2E_TIERS + touchfiles (with the auq-repetition-cut
A/B); periodic detach timeout raised to the new shard-census floor. Free
selftest (8 tests, zero API) pins fixtures, extraction, arm asymmetry, diff
capture, judge plumbing, and the retry bound.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com >
2026-08-28 02:07:56 +00:00
..
2026-08-28 02:07:56 +00:00
2026-08-28 02:07:56 +00:00
2026-08-15 22:20:30 -07:00
2026-08-28 01:54:36 +00:00
2026-03-18 23:57:59 -05:00
2026-08-15 22:20:30 -07:00
2026-08-16 19:34:41 -07:00
2026-05-17 08:26:36 -07:00
2026-08-27 09:50:31 -07:00
2026-08-19 16:53:19 -07:00
2026-08-27 09:50:31 -07:00
2026-08-28 01:41:29 +00:00
2026-08-14 22:02:07 -07:00
2026-04-19 17:50:31 +08:00
2026-08-19 11:42:55 -07:00
2026-06-21 07:15:19 -07:00
2026-08-27 09:50:31 -07:00
2026-08-19 11:42:55 -07:00
2026-08-19 11:42:55 -07:00
2026-05-29 08:35:00 -07:00
2026-08-19 11:42:55 -07:00
2026-08-19 11:42:55 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-05-20 07:35:01 -07:00
2026-04-08 22:21:28 -10:00
2026-05-29 08:35:00 -07:00
2026-06-07 19:13:24 -07:00
2026-06-07 19:13:24 -07:00
2026-08-27 09:50:31 -07:00
2026-06-07 19:13:24 -07:00
2026-08-14 09:28:56 -07:00
2026-05-30 14:57:07 -07:00
2026-08-16 19:34:41 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 11:42:19 -07:00
2026-08-16 19:34:41 -07:00
2026-04-23 07:25:20 -07:00
2026-08-15 09:37:04 -07:00
2026-08-18 17:00:04 -07:00
2026-08-15 22:20:30 -07:00
2026-08-18 17:00:04 -07:00
2026-08-27 09:50:31 -07:00
2026-08-16 19:34:41 -07:00
2026-05-09 08:06:47 -07:00
2026-08-16 19:34:41 -07:00
2026-08-27 09:50:31 -07:00
2026-05-16 12:32:33 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-08 09:28:45 -07:00
2026-06-07 19:13:24 -07:00
2026-08-15 09:37:04 -07:00
2026-05-28 18:21:09 -07:00
2026-08-15 22:20:30 -07:00
2026-06-09 22:29:23 -07:00
2026-06-12 15:38:53 -07:00
2026-08-16 19:34:41 -07:00
2026-05-30 12:09:10 -07:00
2026-05-28 18:21:09 -07:00
2026-05-28 18:21:09 -07:00
2026-05-21 21:21:07 -07:00
2026-06-07 19:13:24 -07:00
2026-04-26 13:55:13 -07:00
2026-08-15 22:20:30 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-08-15 22:20:30 -07:00
2026-08-07 15:08:12 -07:00
2026-08-15 22:20:30 -07:00
2026-08-27 08:46:41 -07:00
2026-08-24 08:55:58 -07:00
2026-08-14 22:02:07 -07:00
2026-08-14 22:02:07 -07:00
2026-05-20 07:35:01 -07:00
2026-08-15 11:42:19 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 09:37:04 -07:00
2026-08-24 08:55:58 -07:00
2026-08-15 11:42:19 -07:00
2026-06-08 06:20:58 -07:00
2026-08-15 22:20:30 -07:00
2026-06-21 07:15:19 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-05-17 08:26:36 -07:00
2026-06-08 06:20:58 -07:00
2026-05-14 17:20:48 -07:00
2026-08-27 09:50:31 -07:00
2026-05-21 21:21:07 -07:00
2026-04-24 01:38:21 -07:00
2026-08-19 11:42:55 -07:00
2026-08-16 19:34:41 -07:00
2026-08-19 11:42:55 -07:00
2026-08-15 11:42:19 -07:00
2026-05-17 08:26:36 -07:00
2026-08-19 11:42:55 -07:00
2026-05-30 14:57:07 -07:00
2026-08-08 09:28:45 -07:00
2026-08-16 19:34:41 -07:00
2026-08-15 22:20:30 -07:00
2026-07-10 16:51:16 -07:00
2026-05-21 18:55:55 -07:00
2026-08-14 09:28:56 -07:00
2026-08-15 09:37:04 -07:00
2026-08-16 19:34:41 -07:00
2026-08-28 01:45:53 +00:00
2026-05-20 06:56:41 -07:00
2026-08-14 22:02:07 -07:00
2026-05-21 21:21:07 -07:00
2026-08-15 11:42:19 -07:00
2026-05-28 18:21:09 -07:00
2026-08-19 11:42:55 -07:00
2026-08-19 16:53:19 -07:00
2026-05-30 08:54:46 -07:00
2026-08-15 11:42:19 -07:00
2026-06-08 06:20:58 -07:00
2026-06-08 06:20:58 -07:00
2026-06-14 11:40:57 -07:00
2026-08-08 09:28:45 -07:00
2026-08-16 19:34:41 -07:00
2026-08-08 09:28:45 -07:00
2026-08-14 09:28:56 -07:00
2026-08-24 08:55:58 -07:00
2026-08-16 19:34:41 -07:00
2026-06-08 06:39:38 -07:00
2026-08-16 19:34:41 -07:00
2026-08-24 08:55:58 -07:00
2026-08-19 11:42:55 -07:00
2026-08-14 22:02:07 -07:00
2026-06-21 07:15:19 -07:00
2026-08-19 11:42:55 -07:00
2026-08-19 11:42:55 -07:00
2026-07-10 03:09:13 -07:00
2026-08-27 09:50:31 -07:00
2026-05-29 08:35:00 -07:00
2026-06-07 21:38:21 -07:00
2026-08-19 16:53:19 -07:00
2026-08-27 09:50:31 -07:00
2026-08-16 19:34:41 -07:00
2026-08-19 11:42:55 -07:00
2026-08-16 19:34:41 -07:00
2026-05-28 18:21:09 -07:00
2026-08-14 22:02:07 -07:00
2026-04-28 01:17:54 -07:00
2026-05-14 17:20:48 -07:00
2026-05-25 10:57:15 -07:00
2026-08-19 11:42:55 -07:00
2026-04-26 13:55:13 -07:00
2026-08-24 08:55:58 -07:00
2026-08-14 09:28:56 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 09:53:31 -07:00
2026-08-14 22:02:07 -07:00
2026-08-18 17:00:04 -07:00
2026-08-14 22:02:07 -07:00
2026-08-17 07:50:37 -07:00
2026-07-14 16:35:30 -07:00
2026-08-15 11:42:19 -07:00
2026-07-14 16:35:30 -07:00
2026-06-14 11:40:57 -07:00
2026-04-18 15:05:42 +08:00
2026-05-30 14:57:07 -07:00
2026-08-15 09:37:04 -07:00
2026-08-27 09:50:31 -07:00
2026-05-21 21:21:07 -07:00
2026-06-21 07:15:19 -07:00
2026-05-01 19:51:51 -07:00
2026-05-07 20:14:59 -07:00
2026-08-08 09:28:45 -07:00
2026-08-15 11:42:19 -07:00
2026-05-20 07:35:01 -07:00
2026-05-30 14:57:07 -07:00
2026-04-19 08:38:19 +08:00
2026-08-15 11:42:19 -07:00
2026-08-15 11:42:19 -07:00
2026-08-15 11:42:19 -07:00
2026-08-14 22:02:07 -07:00
2026-08-27 09:50:31 -07:00
2026-04-25 11:52:48 -07:00
2026-08-14 22:02:07 -07:00
2026-08-14 22:02:07 -07:00
2026-08-14 22:02:07 -07:00
2026-08-27 09:50:31 -07:00
2026-08-14 22:02:07 -07:00
2026-08-27 09:50:31 -07:00
2026-08-08 09:28:45 -07:00
2026-04-17 00:45:13 -07:00
2026-08-18 17:00:04 -07:00
2026-08-14 22:02:07 -07:00
2026-05-26 16:50:03 -07:00
2026-05-30 12:09:10 -07:00
2026-08-15 09:37:04 -07:00
2026-05-28 18:21:09 -07:00
2026-08-27 09:50:31 -07:00
2026-05-06 19:37:53 -07:00
2026-08-14 22:02:07 -07:00
2026-06-07 22:04:18 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-14 09:28:56 -07:00
2026-08-16 19:34:41 -07:00
2026-08-14 22:02:07 -07:00
2026-08-08 09:28:45 -07:00
2026-08-16 19:34:41 -07:00
2026-04-18 15:05:42 +08:00
2026-05-30 08:54:46 -07:00
2026-08-15 09:37:04 -07:00
2026-05-30 08:54:46 -07:00
2026-08-28 01:50:31 +00:00
2026-08-16 19:34:41 -07:00
2026-05-30 08:54:46 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-05-30 08:54:46 -07:00
2026-05-30 12:09:10 -07:00
2026-06-07 06:51:10 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-05-20 06:56:41 -07:00
2026-05-20 06:56:41 -07:00
2026-08-16 19:34:41 -07:00
2026-05-30 12:09:10 -07:00
2026-06-14 11:40:57 -07:00
2026-05-29 08:35:00 -07:00
2026-05-29 08:35:00 -07:00
2026-08-16 09:53:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-05-29 08:35:00 -07:00
2026-05-29 08:35:00 -07:00
2026-04-24 01:38:21 -07:00
2026-06-04 11:14:43 -07:00
2026-06-21 07:15:19 -07:00
2026-07-09 23:49:09 -07:00
2026-08-19 11:42:55 -07:00
2026-08-16 19:34:41 -07:00
2026-08-16 19:34:41 -07:00
2026-08-19 11:42:55 -07:00
2026-08-24 08:55:58 -07:00
2026-08-14 22:02:07 -07:00
2026-08-18 17:00:04 -07:00
2026-05-14 21:19:58 -07:00
2026-05-29 18:06:19 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-24 08:55:58 -07:00
2026-08-19 16:53:19 -07:00
2026-08-19 16:53:19 -07:00
2026-08-14 22:02:07 -07:00
2026-08-16 19:34:41 -07:00
2026-08-19 16:53:19 -07:00
2026-08-16 19:34:41 -07:00
2026-08-15 11:42:19 -07:00
2026-08-27 08:46:41 -07:00
2026-05-30 12:09:10 -07:00
2026-08-16 19:34:41 -07:00
2026-05-30 12:09:10 -07:00
2026-08-15 11:42:19 -07:00
2026-04-18 23:58:59 +08:00
2026-08-14 09:28:56 -07:00
2026-08-16 19:34:41 -07:00
2026-06-04 11:14:43 -07:00
2026-04-19 08:38:19 +08:00
2026-08-15 22:20:30 -07:00
2026-08-15 22:20:30 -07:00
2026-06-12 15:38:53 -07:00
2026-08-27 09:50:31 -07:00
2026-08-28 02:07:56 +00:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 22:20:30 -07:00
2026-08-28 02:00:50 +00:00
2026-08-15 09:37:04 -07:00
2026-08-15 11:42:19 -07:00
2026-08-28 01:41:29 +00:00
2026-08-27 09:50:31 -07:00
2026-08-14 09:28:56 -07:00
2026-08-15 11:42:19 -07:00
2026-06-25 09:42:45 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 22:20:30 -07:00
2026-08-16 19:34:41 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-06-12 15:38:53 -07:00
2026-06-25 09:42:45 -07:00
2026-05-29 08:35:00 -07:00
2026-08-14 09:28:56 -07:00
2026-07-14 16:35:30 -07:00
2026-07-14 16:35:30 -07:00
2026-07-14 16:35:30 -07:00
2026-03-31 23:08:22 -06:00
2026-08-19 11:42:55 -07:00
2026-08-15 09:37:04 -07:00
2026-06-04 11:14:43 -07:00
2026-05-01 19:51:51 -07:00
2026-06-04 11:14:43 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 09:37:04 -07:00
2026-05-01 19:51:51 -07:00
2026-08-15 22:20:30 -07:00
2026-04-23 18:25:34 -07:00
2026-08-08 09:28:45 -07:00
2026-04-18 15:05:42 +08:00
2026-08-15 11:42:19 -07:00
2026-08-27 09:50:31 -07:00
2026-05-11 12:16:26 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-28 01:45:53 +00:00
2026-08-15 22:20:30 -07:00
2026-08-15 22:20:30 -07:00
2026-04-19 08:38:19 +08:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 08:46:41 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 09:37:04 -07:00
2026-08-15 22:20:30 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-05-26 21:36:53 -07:00
2026-08-27 09:50:31 -07:00
2026-03-13 21:08:12 -07:00
2026-05-29 08:35:00 -07:00
2026-08-15 22:20:30 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-08-27 09:50:31 -07:00
2026-05-25 10:57:15 -07:00
2026-08-15 22:20:30 -07:00
2026-05-29 08:35:00 -07:00
2026-08-15 11:42:19 -07:00
2026-04-19 17:50:31 +08:00
2026-04-22 01:06:22 -07:00
2026-08-27 09:50:31 -07:00
2026-06-21 07:15:19 -07:00
2026-05-30 12:09:10 -07:00
2026-06-08 05:41:07 -07:00
2026-08-18 17:00:04 -07:00
2026-08-19 16:53:19 -07:00
2026-05-25 10:57:15 -07:00
2026-08-15 22:20:30 -07:00
2026-08-15 22:20:30 -07:00
2026-08-28 02:07:56 +00:00
2026-08-27 09:50:31 -07:00
2026-08-16 09:53:31 -07:00
2026-05-30 12:09:10 -07:00
2026-08-16 19:34:41 -07:00
2026-08-19 16:53:19 -07:00
2026-08-14 22:02:07 -07:00
2026-05-24 01:43:51 -07:00
2026-08-19 11:42:55 -07:00
2026-08-16 19:34:41 -07:00
2026-08-14 22:02:07 -07:00
2026-08-15 09:37:04 -07:00
2026-08-19 16:53:19 -07:00
2026-08-14 22:02:07 -07:00
2026-04-06 14:41:06 -07:00
2026-08-27 09:50:31 -07:00