mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-29 01:10:50 +02:00
* fix(ship): name the /document-release subagent at every Step 18 decision point The v1.54.0.0 carve moved Step 18 (documentation sync) into ship/sections/pr-body.md and the Claude-host skeleton stopped saying "document-release" anywhere in the workflow body — the dispatch became invisible at exactly the moments an agent decides whether to open the section. Restore visibility at three touchpoints, all subagent-framed (never bare-slash-framed, which would invite an inline Skill invocation that bypasses the fresh-context subagent + JSON contract): - manifest trigger (renders into the section-index row AND the STOP pointer): "dispatching the /document-release subagent to sync docs (Step 18) and then creating or updating the PR/MR (Step 19)" - Step 17 handoff line names Step 18's dispatch explicitly - new hoisted doc-sync invariant beside the PR-title invariant: the dispatch itself is never skipped; only a failed subagent is non-blocking Pin it in carve-guards: 'the /document-release subagent' (all three touchpoints) + 'dispatches the /document-release subagent' (invariant) must stay in the skeleton; the carved imperative 'Dispatch /document-release as a subagent' must stay carved. Skeleton cap 91,600 → 92,300 (measured 91,764; trigger renders twice). Goldens regenerated for all three hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin the ship→document-release Step 18 wiring with a free tripwire Five substring/structure asserts across the carved section, the Claude skeleton's three touchpoints, the manifest trigger, and the codex/factory goldens (inlined Step 18 ordered before Step 19). Claude-golden asserts deliberately omitted: host-config.test.ts already enforces golden == generated byte-for-byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: gate-tier E2E proving /ship dispatches the document-release subagent New skill-e2e-ship-docsync: a live agent gets the sliced Step 17→19 tail of the generated ship skeleton in a bare-remote git fixture (Steps 0-16 "done"), under a fake HOME so the STOP pointer and the Step 18 subagent prompt resolve to planted copies, with a stub document-release skill that returns the empty-result JSON contract. Hard assert: an Agent/Task tool-call matching /document-release/i exists in result.toolCalls and precedes any `gh pr create`. Neutral prompt (no STOP-Read priming, no document-release mention — the prompt echoes into the transcript, so asserts read toolCalls only). Hardening from review: throw-on-marker-drift fixture slice; per-test GSTACK_HOME + .redact-prepush-prompted marker (routes Step 17's credential guard to its silent branch — the hermetic GSTACK_HOME pin defeats a HOME-only override); 480s/540s timeouts (nested subagent adds wall clock the 300s sibling never carried); 'timeout' accepted in exitReason only because the dispatch assert is independently hard; whole-file describeE2ETier('gate') composed with diff selection (keeps the file out of the periodic shard census, which sits at its ceiling, and under the hard tier-alignment invariant). Registered as 'ship-docsync' in E2E_TOUCHFILES + E2E_TIERS (gate) in the same commit — touchfiles.test.ts rejects either half landing first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix stale document-release TODOS entry + three review-deferred items The SHIPPED entry still described the deleted Step 8.5 post-PR cat-delegation design from v0.8.4; replace with the current Step 18 subagent design and its test pins. Add the three P3 items deferred from the v1.69 plan review: dispatch receipt enforcement, land-and-deploy→canary dispatch-pin pattern, and the periodic shard-census boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pre-landing review fixes Testing-specialist findings, all mechanical: (1) pin the E2E fixture's git branch (-b main / init.defaultBranch=main) and assert every setup command's exit status so operator git config can't silently corrupt a paid run; (2) tighten the dispatch matcher to Step 18-prompt-specific markers (document-release/SKILL.md | executing the /document-release workflow) so a subagent merely quoting section text can't false-pass the regression assert (verified against recorded burn-in transcripts); (3) replace the subsumed carve-guards anchor with three non-overlapping per-touchpoint anchors (gerund/imperative/3rd-person) so each touchpoint is independently enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: red-team review fixes Five informational findings: TODOS shard-census arithmetic corrected (census is 67 with one free ungated slot; the SECOND ungated file trips the floor) and version pointer fixed (v0.18.2.0, not v0.18.1.0); the free tripwire now pins the two dispatch-matcher marker strings so a pr-body prompt reword fails the free suite instead of surfacing as a paid-tier mystery; the E2E matcher gains a section-paste exclusion (scaffold strings disqualify) — verified against all recorded runs; the E2E header documents the tierless test:evals invisibility tradeoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial review fixes Pin the E2E matcher's two EXCLUSION markers in the free tripwire (an unpinned 'Parent processing:' reword would silently deaden the section-paste guard while every test stayed green); add an ordering pin (the hoisted doc-sync invariant must sit above the pr-body STOP pointer — presence-only anchors can't catch drift below it); plant a third cwd-relative pr-body copy inside the fixture repo, gitignored so the agent never tries to commit test scaffolding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.70.1.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: CHANGELOG accuracy fixes from the doc-release review Three factual corrections the Step 18 doc subagent caught in the fresh v1.70.1.0 entry: 5 tripwire tests (not 6), cost floor $0.63 per the cited eval store (not $0.59), and the visibility claim scoped to decision points (the re-run checklist mention survived the carve). Plus the E2E header's stale pending-burn-in note replaced with the observed numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: raise bun-polyfill subprocess budget to 60s for degraded Windows runners The 50ms-sleep test blew the 20s budget on BOTH bun retry attempts on PR #2700's windows-latest runner (run 32989821401) — sustained AV/runner pressure, not just the documented cold-start. Same flake passed-on-rerun on the prompt-token-load-reduction branch yesterday. Budget only; every assertion still checks exact output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): run the ship-docsync gate E2E in the evals matrix + silent-skip tripwire The evals.yml matrix is hand-enumerated and the Run step never exported EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have self-skipped even with a row — a hollow green one layer deeper than the documented rehomed-monolith incident. Add the e2e-ship-docsync row with a row-level `tier: gate` property, exported as EVALS_TIER by the Run step (empty = unset for every existing row: all readers are `=== '<tier>'` or truthiness). New free tripwire test/evals-workflow-matrix.test.ts ratchets the class: matrix files must exist; gate-hosting files must have a row; whole-file-gated matrix files must carry a matching row tier; and the burn-down lists enforce their own cleanup. It enumerates the PRE-EXISTING holes found while wiring this (8 gate-hosting files with no row; codex/gemini rows running zero tests; the pty-plan-smoke row hollow since its files adopted describeE2ETier) — tracked in TODOS as the CI gate-lane hollow-coverage burn-down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
136 lines
6.2 KiB
TypeScript
136 lines
6.2 KiB
TypeScript
/**
|
|
* /ship → /document-release Step 18 dispatch stays wired AND visible.
|
|
*
|
|
* The v1.54.0.0 carve (46c1fae7) moved Step 18 (documentation sync) out of
|
|
* the ship skeleton into sections/pr-body.md. The wiring survived, but on
|
|
* the Claude host the skeleton stopped saying "document-release" anywhere
|
|
* in the workflow body — the dispatch became invisible at the decision
|
|
* points and "the idea of running document-release as part of ship got
|
|
* lost". This tripwire pins both halves so neither can silently regress:
|
|
*
|
|
* - the carved section keeps the full dispatch contract (imperative,
|
|
* subagent_type, JSON return keys, non-blocking failure), and
|
|
* - the Claude skeleton names "the /document-release subagent" at its
|
|
* three touchpoints (manifest trigger via section-index + STOP pointer,
|
|
* Step 17 handoff, hoisted doc-sync invariant) while the imperative
|
|
* itself stays carved.
|
|
*
|
|
* Structural backstop lives in test/helpers/carve-guards.ts (ship entry);
|
|
* this named test documents the regression story. Behavioral proof is the
|
|
* ship-docsync E2E (test/skill-e2e-ship-docsync.test.ts).
|
|
*/
|
|
import { describe, test, expect } from 'bun:test';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
|
|
const ROOT = path.join(import.meta.dir, '..');
|
|
|
|
const SECTION_SITES = [
|
|
path.join(ROOT, 'ship', 'sections', 'pr-body.md'),
|
|
path.join(ROOT, 'ship', 'sections', 'pr-body.md.tmpl'),
|
|
];
|
|
|
|
// ship/SKILL.md only — NOT the claude golden: test/host-config.test.ts
|
|
// already enforces golden == generated byte-for-byte, so substring asserts
|
|
// on the golden would be pure duplication. The codex/factory goldens ARE
|
|
// asserted below because for them the check is content (the inlined
|
|
// section survived generation), which byte-equality alone doesn't prove.
|
|
const CLAUDE_SKELETON = path.join(ROOT, 'ship', 'SKILL.md');
|
|
const INLINED_GOLDENS = [
|
|
path.join(ROOT, 'test', 'fixtures', 'golden', 'codex-ship-SKILL.md'),
|
|
path.join(ROOT, 'test', 'fixtures', 'golden', 'factory-ship-SKILL.md'),
|
|
];
|
|
|
|
const CARVED_IMPERATIVE = 'Dispatch /document-release as a subagent';
|
|
|
|
describe('/ship Step 18 dispatches /document-release (carve visibility)', () => {
|
|
test('carved section carries the full dispatch contract', () => {
|
|
for (const file of SECTION_SITES) {
|
|
const content = fs.readFileSync(file, 'utf-8');
|
|
expect(content).toContain(
|
|
'## Step 18: Documentation sync (via subagent, before PR creation)'
|
|
);
|
|
expect(content).toContain(CARVED_IMPERATIVE);
|
|
expect(content).toContain('subagent_type: "general-purpose"');
|
|
// The JSON return contract Step 19 bakes into the PR body.
|
|
expect(content).toContain('"files_updated"');
|
|
expect(content).toContain('"commit_sha"');
|
|
expect(content).toContain('"pushed"');
|
|
expect(content).toContain('"documentation_section"');
|
|
// Deliberate design: docs sync never holds a ship hostage.
|
|
expect(content).toContain('Do not block /ship on subagent failure');
|
|
// These four strings are the ship-docsync E2E's dispatch-matcher markers
|
|
// (test/skill-e2e-ship-docsync.test.ts): the first two are INCLUSION
|
|
// markers (verbatim from the dictated Step 18 subagent prompt); the last
|
|
// two are EXCLUSION markers (section scaffolding that disqualifies a
|
|
// whole-section paste). Rewording any of them in pr-body.md.tmpl silently
|
|
// deadens the paid matcher; update all four in lockstep.
|
|
expect(content).toContain('You are executing the /document-release workflow');
|
|
expect(content).toContain('.claude/skills/gstack/document-release/SKILL.md');
|
|
expect(content).toContain('## Step 19: Create PR/MR');
|
|
expect(content).toContain('Parent processing:');
|
|
}
|
|
});
|
|
|
|
test('claude skeleton names the subagent at all three touchpoints', () => {
|
|
const content = fs.readFileSync(CLAUDE_SKELETON, 'utf-8');
|
|
// Manifest trigger — renders into the section-index row AND the STOP
|
|
// pointer, so it must appear at least twice.
|
|
const trigger =
|
|
'dispatching the /document-release subagent to sync docs (Step 18) and then creating or updating the PR/MR (Step 19)';
|
|
expect(content.split(trigger).length - 1).toBeGreaterThanOrEqual(2);
|
|
// Step 17 handoff.
|
|
expect(content).toContain(
|
|
'Step 18 (dispatch the /document-release subagent to sync docs)'
|
|
);
|
|
// Hoisted doc-sync invariant (beside the PR-title invariant).
|
|
expect(content).toContain('**Doc-sync invariant');
|
|
expect(content).toContain('dispatches the /document-release subagent');
|
|
// The STOP-Read pointer still routes to the carved section.
|
|
expect(content).toMatch(
|
|
/> \*\*STOP\.\*\*[^\n]*Read `[^`]*ship\/sections\/pr-body\.md`/
|
|
);
|
|
// Ordering pin: the hoisted invariant must sit ABOVE the pr-body STOP
|
|
// pointer (mustStayInSkeleton asserts presence only — a future edit could
|
|
// drift the paragraph below the STOP with every registry check green).
|
|
const invariantIdx = content.indexOf('**Doc-sync invariant');
|
|
const stopIdx = content.indexOf(
|
|
'> **STOP.** Before dispatching the /document-release subagent'
|
|
);
|
|
expect(invariantIdx).toBeGreaterThan(-1);
|
|
expect(stopIdx).toBeGreaterThan(invariantIdx);
|
|
});
|
|
|
|
test('the dispatch imperative stays carved out of the claude skeleton', () => {
|
|
const content = fs.readFileSync(CLAUDE_SKELETON, 'utf-8');
|
|
expect(content).not.toContain(CARVED_IMPERATIVE);
|
|
});
|
|
|
|
test('manifest trigger names the subagent', () => {
|
|
const manifest = JSON.parse(
|
|
fs.readFileSync(
|
|
path.join(ROOT, 'ship', 'sections', 'manifest.json'),
|
|
'utf-8'
|
|
)
|
|
);
|
|
const prBody = manifest.sections.find(
|
|
(s: { id: string }) => s.id === 'pr-body'
|
|
);
|
|
expect(prBody).toBeDefined();
|
|
expect(prBody.trigger).toContain('the /document-release subagent');
|
|
});
|
|
|
|
test('codex + factory goldens inline Step 18 before Step 19', () => {
|
|
for (const file of INLINED_GOLDENS) {
|
|
const content = fs.readFileSync(file, 'utf-8');
|
|
const step18 = content.indexOf(
|
|
'## Step 18: Documentation sync (via subagent, before PR creation)'
|
|
);
|
|
const step19 = content.indexOf('## Step 19: Create PR/MR');
|
|
expect(step18).toBeGreaterThan(-1);
|
|
expect(step19).toBeGreaterThan(step18);
|
|
expect(content).toContain(CARVED_IMPERATIVE);
|
|
}
|
|
});
|
|
});
|