mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-29 17:30:40 +02:00
v1.70.1.0 fix: ship names the /document-release subagent at every decision point (tripwire + gate E2E) (#2700)
* fix(ship): name the /document-release subagent at every Step 18 decision point The v1.54.0.0 carve moved Step 18 (documentation sync) into ship/sections/pr-body.md and the Claude-host skeleton stopped saying "document-release" anywhere in the workflow body — the dispatch became invisible at exactly the moments an agent decides whether to open the section. Restore visibility at three touchpoints, all subagent-framed (never bare-slash-framed, which would invite an inline Skill invocation that bypasses the fresh-context subagent + JSON contract): - manifest trigger (renders into the section-index row AND the STOP pointer): "dispatching the /document-release subagent to sync docs (Step 18) and then creating or updating the PR/MR (Step 19)" - Step 17 handoff line names Step 18's dispatch explicitly - new hoisted doc-sync invariant beside the PR-title invariant: the dispatch itself is never skipped; only a failed subagent is non-blocking Pin it in carve-guards: 'the /document-release subagent' (all three touchpoints) + 'dispatches the /document-release subagent' (invariant) must stay in the skeleton; the carved imperative 'Dispatch /document-release as a subagent' must stay carved. Skeleton cap 91,600 → 92,300 (measured 91,764; trigger renders twice). Goldens regenerated for all three hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin the ship→document-release Step 18 wiring with a free tripwire Five substring/structure asserts across the carved section, the Claude skeleton's three touchpoints, the manifest trigger, and the codex/factory goldens (inlined Step 18 ordered before Step 19). Claude-golden asserts deliberately omitted: host-config.test.ts already enforces golden == generated byte-for-byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: gate-tier E2E proving /ship dispatches the document-release subagent New skill-e2e-ship-docsync: a live agent gets the sliced Step 17→19 tail of the generated ship skeleton in a bare-remote git fixture (Steps 0-16 "done"), under a fake HOME so the STOP pointer and the Step 18 subagent prompt resolve to planted copies, with a stub document-release skill that returns the empty-result JSON contract. Hard assert: an Agent/Task tool-call matching /document-release/i exists in result.toolCalls and precedes any `gh pr create`. Neutral prompt (no STOP-Read priming, no document-release mention — the prompt echoes into the transcript, so asserts read toolCalls only). Hardening from review: throw-on-marker-drift fixture slice; per-test GSTACK_HOME + .redact-prepush-prompted marker (routes Step 17's credential guard to its silent branch — the hermetic GSTACK_HOME pin defeats a HOME-only override); 480s/540s timeouts (nested subagent adds wall clock the 300s sibling never carried); 'timeout' accepted in exitReason only because the dispatch assert is independently hard; whole-file describeE2ETier('gate') composed with diff selection (keeps the file out of the periodic shard census, which sits at its ceiling, and under the hard tier-alignment invariant). Registered as 'ship-docsync' in E2E_TOUCHFILES + E2E_TIERS (gate) in the same commit — touchfiles.test.ts rejects either half landing first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix stale document-release TODOS entry + three review-deferred items The SHIPPED entry still described the deleted Step 8.5 post-PR cat-delegation design from v0.8.4; replace with the current Step 18 subagent design and its test pins. Add the three P3 items deferred from the v1.69 plan review: dispatch receipt enforcement, land-and-deploy→canary dispatch-pin pattern, and the periodic shard-census boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pre-landing review fixes Testing-specialist findings, all mechanical: (1) pin the E2E fixture's git branch (-b main / init.defaultBranch=main) and assert every setup command's exit status so operator git config can't silently corrupt a paid run; (2) tighten the dispatch matcher to Step 18-prompt-specific markers (document-release/SKILL.md | executing the /document-release workflow) so a subagent merely quoting section text can't false-pass the regression assert (verified against recorded burn-in transcripts); (3) replace the subsumed carve-guards anchor with three non-overlapping per-touchpoint anchors (gerund/imperative/3rd-person) so each touchpoint is independently enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: red-team review fixes Five informational findings: TODOS shard-census arithmetic corrected (census is 67 with one free ungated slot; the SECOND ungated file trips the floor) and version pointer fixed (v0.18.2.0, not v0.18.1.0); the free tripwire now pins the two dispatch-matcher marker strings so a pr-body prompt reword fails the free suite instead of surfacing as a paid-tier mystery; the E2E matcher gains a section-paste exclusion (scaffold strings disqualify) — verified against all recorded runs; the E2E header documents the tierless test:evals invisibility tradeoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial review fixes Pin the E2E matcher's two EXCLUSION markers in the free tripwire (an unpinned 'Parent processing:' reword would silently deaden the section-paste guard while every test stayed green); add an ordering pin (the hoisted doc-sync invariant must sit above the pr-body STOP pointer — presence-only anchors can't catch drift below it); plant a third cwd-relative pr-body copy inside the fixture repo, gitignored so the agent never tries to commit test scaffolding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.70.1.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: CHANGELOG accuracy fixes from the doc-release review Three factual corrections the Step 18 doc subagent caught in the fresh v1.70.1.0 entry: 5 tripwire tests (not 6), cost floor $0.63 per the cited eval store (not $0.59), and the visibility claim scoped to decision points (the re-run checklist mention survived the carve). Plus the E2E header's stale pending-burn-in note replaced with the observed numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: raise bun-polyfill subprocess budget to 60s for degraded Windows runners The 50ms-sleep test blew the 20s budget on BOTH bun retry attempts on PR #2700's windows-latest runner (run 32989821401) — sustained AV/runner pressure, not just the documented cold-start. Same flake passed-on-rerun on the prompt-token-load-reduction branch yesterday. Budget only; every assertion still checks exact output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): run the ship-docsync gate E2E in the evals matrix + silent-skip tripwire The evals.yml matrix is hand-enumerated and the Run step never exported EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have self-skipped even with a row — a hollow green one layer deeper than the documented rehomed-monolith incident. Add the e2e-ship-docsync row with a row-level `tier: gate` property, exported as EVALS_TIER by the Run step (empty = unset for every existing row: all readers are `=== '<tier>'` or truthiness). New free tripwire test/evals-workflow-matrix.test.ts ratchets the class: matrix files must exist; gate-hosting files must have a row; whole-file-gated matrix files must carry a matching row tier; and the burn-down lists enforce their own cleanup. It enumerates the PRE-EXISTING holes found while wiring this (8 gate-hosting files with no row; codex/gemini rows running zero tests; the pty-plan-smoke row hollow since its files adopted describeE2ETier) — tracked in TODOS as the CI gate-lane hollow-coverage burn-down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
ad8400543c
commit
a3749bfa4b
@@ -2329,7 +2329,105 @@ Shipped as v0.5.0 on main. Includes `/plan-design-review` (report-only design au
|
||||
|
||||
### Auto-invoke /document-release from /ship — SHIPPED
|
||||
|
||||
Shipped in v0.8.3. Step 8.5 added to `/ship` — after creating the PR, `/ship` automatically reads `document-release/SKILL.md` and executes the doc update workflow. Zero-friction doc updates.
|
||||
Shipped in v0.8.4; redesigned twice since. Current design (v0.18.2.0+, carved in
|
||||
v1.54.0.0): `/ship` Step 18 (`ship/sections/pr-body.md`) dispatches
|
||||
`/document-release` as a general-purpose subagent AFTER Step 17 (push) and
|
||||
BEFORE Step 19 (PR creation); the subagent's JSON contract (`files_updated`,
|
||||
`commit_sha`, `pushed`, `documentation_section`) is baked into the initial PR
|
||||
body. Subagent failure is non-blocking. The skeleton names "the
|
||||
/document-release subagent" at three touchpoints (section-index trigger + STOP
|
||||
pointer, Step 17 handoff, hoisted doc-sync invariant). Pinned by
|
||||
`test/ship-document-release-dispatch.test.ts` + carve-guards anchors; behavior
|
||||
proven by the `ship-docsync` gate E2E (`test/skill-e2e-ship-docsync.test.ts`).
|
||||
|
||||
### Machine-checkable Step 18 dispatch receipt in /ship's Section self-check
|
||||
|
||||
**What:** Make ship's "Section self-check" verify a document-release dispatch
|
||||
actually occurred (a machine-checkable marker/receipt), instead of relying on
|
||||
prompt-level invariants alone.
|
||||
|
||||
**Why:** Prompt wording deters skipping but can't prove the dispatch happened.
|
||||
Two residual gaps from the v1.69 review are folded into this scope: (1) an
|
||||
agent invoking `/document-release` inline via the Skill tool bypasses the
|
||||
fresh-context subagent + JSON contract and no test can see it; (2) the ship
|
||||
RE-RUN path names document-release in the re-run list but no test asserts
|
||||
doc-sync on re-run.
|
||||
|
||||
**Context:** The `ship-docsync` E2E asserts the dispatch tool-call on the
|
||||
primary path; this TODO is the enforcement layer beyond wording. Start from
|
||||
ship's Section self-check (ship/SKILL.md.tmpl) and the Step 18 parent
|
||||
processing in ship/sections/pr-body.md.tmpl.
|
||||
|
||||
**Effort:** M (human) → S (CC+gstack)
|
||||
**Priority:** P3
|
||||
**Depends on:** ship-docsync E2E landed
|
||||
|
||||
### Apply the dispatch-pin + E2E pattern to /land-and-deploy → /canary
|
||||
|
||||
**What:** Same treatment ship→document-release got: name the handoff at the
|
||||
skeleton decision points, pin with carve-guards anchors + a free tripwire,
|
||||
prove with a toolCalls-assert E2E.
|
||||
|
||||
**Why:** Identical failure class — a carve or reword can silently strand the
|
||||
canary handoff out of the always-loaded skeleton, and nothing tests it today.
|
||||
|
||||
**Context:** Model files: `test/ship-document-release-dispatch.test.ts` (free
|
||||
pin) and `test/skill-e2e-ship-docsync.test.ts` (dispatch E2E, gate tier).
|
||||
|
||||
**Effort:** M (human) → S (CC+gstack)
|
||||
**Priority:** P3
|
||||
**Depends on:** None
|
||||
|
||||
### CI gate-lane hollow-coverage burn-down (evals.yml matrix)
|
||||
|
||||
**What:** `test/evals-workflow-matrix.test.ts` (added v1.70.1.0) ratchets two
|
||||
pre-existing CI coverage holes; burn them down. (1) Eight gate-hosting test
|
||||
files have no `evals.yml` matrix row, so CI never runs them
|
||||
(`KNOWN_MATRIX_GAPS` in the test enumerates them — notably the plan-mode and
|
||||
finding-floor smokes and the AUQ format-compliance gate). (2) Four matrix rows
|
||||
point at whole-file tier-gated files but set no row `tier:` property, so with
|
||||
`EVALS_TIER` unexported those suites self-skip: `codex-e2e`/`gemini-e2e` run
|
||||
ZERO tests and report green on every PR (vestigial rows; the periodic cron
|
||||
lane owns them — consider deleting the rows), and `e2e-pty-plan-smoke` spends
|
||||
~7 min on setup then skips every describe (hollow-green since the files
|
||||
adopted `describeE2ETier('gate')` — set `tier: gate` on the row to reactivate,
|
||||
after confirming the smokes still pass).
|
||||
|
||||
**Why:** "Gate tier blocks merge" is silently false for these files. Each fix
|
||||
is a deliberate cost/flake decision (activating paid suites on every PR), so
|
||||
they're enumerated instead of drive-by-fixed. The mechanism already exists:
|
||||
per-row `tier:` property, exported as `EVALS_TIER` by the Run step.
|
||||
|
||||
**Context:** Found 2026-08-26 on PR #2700 while adding the `ship-docsync` row.
|
||||
Fix = add/adjust the matrix row, then DELETE the corresponding burn-down entry
|
||||
(the tripwire fails on stale entries, so cleanup is enforced).
|
||||
|
||||
**Effort:** S per file (mechanical) + one burn-in run each to confirm green
|
||||
**Priority:** P2
|
||||
**Depends on:** None
|
||||
|
||||
### Periodic paid-test shard census is one ungated file from the detach-timeout floor
|
||||
|
||||
**What:** The periodic tier's shard census is 67 files — one ungated slot below
|
||||
the 68-file (17×4) ceiling. The next paid `skill-e2e-*` file WITHOUT a
|
||||
whole-file `describeE2ETier` self-gate lands at 68 (still 17 waves, floor
|
||||
32,130s ≤ 32,400s — passes); the SECOND ungated file trips 18 waves → 34,020s
|
||||
floor > the 32,400s configured detach timeout, and
|
||||
`test/eval-detach-timeout-floor.test.ts` fails with a confusing message.
|
||||
|
||||
**Why:** Whoever adds the second ungated periodic E2E gets a floor failure
|
||||
unrelated to their change. Fix options: raise the periodic detach timeout, or
|
||||
enforce whole-file tier self-gates on all paid files (upgrades them from the
|
||||
tier-alignment warn-only bucket to the hard invariant, and — bonus — restores
|
||||
tierless `bun run test:evals` coverage decisions to diff selection alone).
|
||||
|
||||
**Context:** `scripts/test-paid-shards.ts` `classifyPaidTestFile` counts
|
||||
ungated files in both tiers; `ship-docsync` composed `describeE2ETier('gate')`
|
||||
with diff selection specifically to avoid consuming the last free slot.
|
||||
|
||||
**Effort:** S
|
||||
**Priority:** P3
|
||||
**Depends on:** None
|
||||
|
||||
### `{{DOC_VOICE}}` shared resolver
|
||||
|
||||
|
||||
Reference in New Issue
Block a user