feat(evals): with-skill vs without-skill arm benchmark — measures whether gstack's behavioral layer earns its tokens

Ponytail's honest-benchmark method pointed at gstack itself: 3 build-shaped tasks (native-platform over-build trap, CRUD endpoint, bug fix with planted decoys) x 2 arms, real claude -p sessions, scored on the git diff left behind. A research instrument, not a release gate — no assertion compares arm scores. Arms use the PROVEN project-scope pattern: the with-arm installs a build-discipline skill (extracted reuse-ladder + bounded-closer content, not whole-file copies) into the fixture's .claude/skills/ with a CLAUDE.md routing line and an explicit invocation; a live spike confirmed claude -p discovers and invokes project-scope skills via the Skill tool (3 turns, exact-output probe). Fixtures are git init + local bare origin; diff capture is three lines of git, no worktree machinery. Failure taxonomy: zero-diff arms are VALID scored cells (deterministic 0/none, no API call), harvest failures record harvest:null, judge_error cells are excluded from aggregates but named in the report — nothing drops silently. armJudge: fixed sonnet judge, 0-3 unrequested-structure rubric, must name the construct or say none, bounded retry-on-malformed; callJudge gains optional temperature/max_tokens (defaults unchanged). recordE2E now populates tokens_used for every E2E. Eval schema v2: harvest gains {insertions, deletions, net}, tolerant reads keep v1 runs comparable. Registered periodic in E2E_TIERS + touchfiles (with the auq-repetition-cut A/B); periodic detach timeout raised to the new shard-census floor. Free selftest (8 tests, zero API) pins fixtures, extraction, arm asymmetry, diff capture, judge plumbing, and the retry bound. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-09-23 13:20:48 +02:00 · 2026-08-28 02:07:56 +00:00
parent 781f46d025
commit 4c20eca33b
21 changed files with 1076 additions and 10 deletions
@@ -0,0 +1,9 @@
+const form = document.getElementById('booking-form');
+const confirmation = document.getElementById('confirmation');
+
+form.addEventListener('submit', (event) => {
+  event.preventDefault();
+  const data = new FormData(form);
+  confirmation.textContent = `Booked for ${data.get('name')}. Confirmation sent to ${data.get('email')}.`;
+  confirmation.hidden = false;
+});
@@ -0,0 +1,20 @@
+<!doctype html>
+<html lang="en">
+<head>
+  <meta charset="utf-8">
+  <title>Trailhead Tours — Book a hike</title>
+  <link rel="stylesheet" href="styles.css">
+</head>
+<body>
+  <main>
+    <h1>Book a guided hike</h1>
+    <form id="booking-form">
+      <label>Name <input type="text" name="name" required></label>
+      <label>Email <input type="email" name="email" required></label>
+      <button type="submit">Book</button>
+    </form>
+    <p id="confirmation" hidden></p>
+  </main>
+  <script src="app.js"></script>
+</body>
+</html>
@@ -0,0 +1,14 @@
+body {
+  font-family: system-ui, sans-serif;
+  max-width: 32rem;
+  margin: 2rem auto;
+}
+
+label {
+  display: block;
+  margin-bottom: 0.75rem;
+}
+
+#confirmation {
+  color: #1a7f37;
+}