From bb56dfef960a53cdf3210f39cbc04454db435947 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Wed, 26 Aug 2026 16:27:23 +0000 Subject: [PATCH] docs: CHANGELOG accuracy fixes from the doc-release review Three factual corrections the Step 18 doc subagent caught in the fresh v1.70.1.0 entry: 5 tripwire tests (not 6), cost floor $0.63 per the cited eval store (not $0.59), and the visibility claim scoped to decision points (the re-run checklist mention survived the carve). Plus the E2E header's stale pending-burn-in note replaced with the observed numbers. Co-Authored-By: Claude Fable 5 --- CHANGELOG.md | 6 +++--- test/skill-e2e-ship-docsync.test.ts | 3 ++- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index b433239ca..29ad00d1d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,7 +5,7 @@ **Ship names its documentation subagent at every decision point.** **The handoff is now pinned by tests that fail loud if it ever goes quiet.** -`/ship` has dispatched `/document-release` as Step 18 since v0.18.2.0, but the v1.54.0.0 carve moved that step into an on-demand section and the always-loaded skeleton stopped saying "document-release" anywhere in the workflow body. The wiring was intact. The visibility was gone, and nothing tested the handoff. This release restores the visibility and locks it in: the section index, the STOP pointer, the Step 17 handoff line, and a new hoisted doc-sync invariant all name "the /document-release subagent" (subagent-framed on purpose, so an agent dispatches the isolated worker instead of running a weaker inline copy). A free tripwire pins the wording, carve-guard anchors pin each touchpoint independently, and a new gate-tier E2E proves a live agent actually fires the dispatch before creating the PR. +`/ship` has dispatched `/document-release` as Step 18 since v0.18.2.0, but the v1.54.0.0 carve moved that step into an on-demand section and the always-loaded skeleton stopped saying "document-release" at any decision point (one mention survived, buried in the re-run checklist). The wiring was intact. The visibility was gone, and nothing tested the handoff. This release restores the visibility and locks it in: the section index, the STOP pointer, the Step 17 handoff line, and a new hoisted doc-sync invariant all name "the /document-release subagent" (subagent-framed on purpose, so an agent dispatches the isolated worker instead of running a weaker inline copy). A free tripwire pins the wording, carve-guard anchors pin each touchpoint independently, and a new gate-tier E2E proves a live agent actually fires the dispatch before creating the PR. ### The numbers that matter @@ -14,8 +14,8 @@ Source: this branch's eval store (`~/.gstack/projects//evals/`, runs of `t | Property | Before | After | |--------|--------|-------| | Doc-sync subagent named in the Claude-host ship workflow body | 0 mentions at any decision point | 4 (section index, STOP pointer, Step 17 handoff, hoisted invariant) | -| Tests pinning the ship→document-release handoff | none | 6 free tripwire tests + 3 per-touchpoint carve anchors + 1 gate E2E | -| Live dispatch proof | never measured | 9/9 runs fire the dispatch before PR creation ($0.59-1.04, 234-319s each, sonnet-4-6) | +| Tests pinning the ship→document-release handoff | none | 5 free tripwire tests + 3 per-touchpoint carve anchors + 1 gate E2E | +| Live dispatch proof | never measured | 9/9 runs fire the dispatch before PR creation ($0.63-1.04, 234-319s each, sonnet-4-6) | | Always-loaded skeleton cost | 91,267 B | 91,764 B (+497 B, cap raised to 92,300) | Nine out of nine live runs is the line that matters. The E2E asserts on the actual tool-call stream, with a dispatch-specific matcher that a subagent merely quoting section text cannot satisfy, and a timeout-tolerant exit check that never softens the dispatch assert itself. diff --git a/test/skill-e2e-ship-docsync.test.ts b/test/skill-e2e-ship-docsync.test.ts index ba0d7f1df..58d586b2b 100644 --- a/test/skill-e2e-ship-docsync.test.ts +++ b/test/skill-e2e-ship-docsync.test.ts @@ -45,7 +45,8 @@ * test would make the assert tautological (and the prompt echoes into the * transcript, which is why asserts only ever read result.toolCalls). * - * Cost: ~$0.30-1/run. Gate tier (pending burn-in; see plan). + * Cost: observed $0.63-1.04/run, 234-319s (9/9 burn-in + review runs passed; + * gate tier confirmed). */ import { expect, beforeAll, afterAll } from 'bun:test'; import * as fs from 'fs';