mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-29 17:30:40 +02:00
v1.70.1.0 fix: ship names the /document-release subagent at every decision point (tripwire + gate E2E) (#2700)
* fix(ship): name the /document-release subagent at every Step 18 decision point The v1.54.0.0 carve moved Step 18 (documentation sync) into ship/sections/pr-body.md and the Claude-host skeleton stopped saying "document-release" anywhere in the workflow body — the dispatch became invisible at exactly the moments an agent decides whether to open the section. Restore visibility at three touchpoints, all subagent-framed (never bare-slash-framed, which would invite an inline Skill invocation that bypasses the fresh-context subagent + JSON contract): - manifest trigger (renders into the section-index row AND the STOP pointer): "dispatching the /document-release subagent to sync docs (Step 18) and then creating or updating the PR/MR (Step 19)" - Step 17 handoff line names Step 18's dispatch explicitly - new hoisted doc-sync invariant beside the PR-title invariant: the dispatch itself is never skipped; only a failed subagent is non-blocking Pin it in carve-guards: 'the /document-release subagent' (all three touchpoints) + 'dispatches the /document-release subagent' (invariant) must stay in the skeleton; the carved imperative 'Dispatch /document-release as a subagent' must stay carved. Skeleton cap 91,600 → 92,300 (measured 91,764; trigger renders twice). Goldens regenerated for all three hosts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: pin the ship→document-release Step 18 wiring with a free tripwire Five substring/structure asserts across the carved section, the Claude skeleton's three touchpoints, the manifest trigger, and the codex/factory goldens (inlined Step 18 ordered before Step 19). Claude-golden asserts deliberately omitted: host-config.test.ts already enforces golden == generated byte-for-byte. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test: gate-tier E2E proving /ship dispatches the document-release subagent New skill-e2e-ship-docsync: a live agent gets the sliced Step 17→19 tail of the generated ship skeleton in a bare-remote git fixture (Steps 0-16 "done"), under a fake HOME so the STOP pointer and the Step 18 subagent prompt resolve to planted copies, with a stub document-release skill that returns the empty-result JSON contract. Hard assert: an Agent/Task tool-call matching /document-release/i exists in result.toolCalls and precedes any `gh pr create`. Neutral prompt (no STOP-Read priming, no document-release mention — the prompt echoes into the transcript, so asserts read toolCalls only). Hardening from review: throw-on-marker-drift fixture slice; per-test GSTACK_HOME + .redact-prepush-prompted marker (routes Step 17's credential guard to its silent branch — the hermetic GSTACK_HOME pin defeats a HOME-only override); 480s/540s timeouts (nested subagent adds wall clock the 300s sibling never carried); 'timeout' accepted in exitReason only because the dispatch assert is independently hard; whole-file describeE2ETier('gate') composed with diff selection (keeps the file out of the periodic shard census, which sits at its ceiling, and under the hard tier-alignment invariant). Registered as 'ship-docsync' in E2E_TOUCHFILES + E2E_TIERS (gate) in the same commit — touchfiles.test.ts rejects either half landing first. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: fix stale document-release TODOS entry + three review-deferred items The SHIPPED entry still described the deleted Step 8.5 post-PR cat-delegation design from v0.8.4; replace with the current Step 18 subagent design and its test pins. Add the three P3 items deferred from the v1.69 plan review: dispatch receipt enforcement, land-and-deploy→canary dispatch-pin pattern, and the periodic shard-census boundary. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pre-landing review fixes Testing-specialist findings, all mechanical: (1) pin the E2E fixture's git branch (-b main / init.defaultBranch=main) and assert every setup command's exit status so operator git config can't silently corrupt a paid run; (2) tighten the dispatch matcher to Step 18-prompt-specific markers (document-release/SKILL.md | executing the /document-release workflow) so a subagent merely quoting section text can't false-pass the regression assert (verified against recorded burn-in transcripts); (3) replace the subsumed carve-guards anchor with three non-overlapping per-touchpoint anchors (gerund/imperative/3rd-person) so each touchpoint is independently enforced. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: red-team review fixes Five informational findings: TODOS shard-census arithmetic corrected (census is 67 with one free ungated slot; the SECOND ungated file trips the floor) and version pointer fixed (v0.18.2.0, not v0.18.1.0); the free tripwire now pins the two dispatch-matcher marker strings so a pr-body prompt reword fails the free suite instead of surfacing as a paid-tier mystery; the E2E matcher gains a section-paste exclusion (scaffold strings disqualify) — verified against all recorded runs; the E2E header documents the tierless test:evals invisibility tradeoff. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: adversarial review fixes Pin the E2E matcher's two EXCLUSION markers in the free tripwire (an unpinned 'Parent processing:' reword would silently deaden the section-paste guard while every test stayed green); add an ordering pin (the hoisted doc-sync invariant must sit above the pr-body STOP pointer — presence-only anchors can't catch drift below it); plant a third cwd-relative pr-body copy inside the fixture repo, gitignored so the agent never tries to commit test scaffolding. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: bump version and changelog (v1.70.1.0) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: CHANGELOG accuracy fixes from the doc-release review Three factual corrections the Step 18 doc subagent caught in the fresh v1.70.1.0 entry: 5 tripwire tests (not 6), cost floor $0.63 per the cited eval store (not $0.59), and the visibility claim scoped to decision points (the re-run checklist mention survived the carve). Plus the E2E header's stale pending-burn-in note replaced with the observed numbers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: raise bun-polyfill subprocess budget to 60s for degraded Windows runners The 50ms-sleep test blew the 20s budget on BOTH bun retry attempts on PR #2700's windows-latest runner (run 32989821401) — sustained AV/runner pressure, not just the documented cold-start. Same flake passed-on-rerun on the prompt-token-load-reduction branch yesterday. Budget only; every assertion still checks exact output. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(ci): run the ship-docsync gate E2E in the evals matrix + silent-skip tripwire The evals.yml matrix is hand-enumerated and the Run step never exported EVALS_TIER, so the new whole-file-gated ship-docsync E2E would have self-skipped even with a row — a hollow green one layer deeper than the documented rehomed-monolith incident. Add the e2e-ship-docsync row with a row-level `tier: gate` property, exported as EVALS_TIER by the Run step (empty = unset for every existing row: all readers are `=== '<tier>'` or truthiness). New free tripwire test/evals-workflow-matrix.test.ts ratchets the class: matrix files must exist; gate-hosting files must have a row; whole-file-gated matrix files must carry a matching row tier; and the burn-down lists enforce their own cleanup. It enumerates the PRE-EXISTING holes found while wiring this (8 gate-hosting files with no row; codex/gemini rows running zero tests; the pty-plan-smoke row hollow since its files adopted describeE2ETier) — tracked in TODOS as the CI gate-lane hollow-coverage burn-down. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Fable 5
Claude Fable 5
parent
ad8400543c
commit
a3749bfa4b
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* CI eval-matrix completeness tripwire — kills the silent-skip class where a
|
||||
* gate-tier test exists in the repo but the hand-enumerated matrix in
|
||||
* .github/workflows/evals.yml never runs it, so "gate tier blocks merge" is
|
||||
* quietly false in CI. This has happened before (see the "rehomed from the
|
||||
* deleted pre-split monolith" comment in evals.yml) and was found again on
|
||||
* PR #2700: nine gate-hosting files absent from the matrix, plus matrix rows
|
||||
* whose whole-file tier guards can never fire because the Run step exported
|
||||
* no EVALS_TIER.
|
||||
*
|
||||
* Ratchet, not amnesty: the KNOWN_* lists below enumerate the PRE-EXISTING
|
||||
* gaps with reasons, so no NEW gap can land while the backlog burns down
|
||||
* (same pattern as SCANNER_EXEMPT in egress-receipt-wiring). If you fix a
|
||||
* listed gap (add its matrix row / tier property), this test FAILS until you
|
||||
* remove the entry — stale exemptions are enforced, not decorative.
|
||||
*
|
||||
* Wiring pinned:
|
||||
* - every matrix `file:` path exists on disk (no stale rows),
|
||||
* - every gate-hosting paid file (whole-file gate self-gate, or named in the
|
||||
* dep list of a gate-tier E2E_TOUCHFILES key) appears in the matrix or in
|
||||
* KNOWN_MATRIX_GAPS,
|
||||
* - every matrix file with a whole-file tier guard has a matching row-level
|
||||
* `tier:` property (else the suite self-skips and the job is hollow-green)
|
||||
* or sits in KNOWN_TIER_UNSET.
|
||||
*/
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import { E2E_TOUCHFILES, E2E_TIERS } from './helpers/touchfiles-data';
|
||||
import { isPaidTestFile } from './helpers/paid-test-set';
|
||||
|
||||
const ROOT = path.join(import.meta.dir, '..');
|
||||
const WORKFLOW = path.join(ROOT, '.github', 'workflows', 'evals.yml');
|
||||
|
||||
/**
|
||||
* Pre-existing gate-hosting files with no matrix row (found 2026-08-26,
|
||||
* PR #2700). Adding a row activates real paid runs on every PR — a cost and
|
||||
* flake-surface decision per file, tracked in TODOS.md ("CI gate-lane
|
||||
* hollow-coverage burn-down"). Fix = add a matrix row (plus `tier: gate` when
|
||||
* the file is whole-file gated), then DELETE the entry here.
|
||||
*/
|
||||
const KNOWN_MATRIX_GAPS = new Set([
|
||||
'test/skill-e2e-ask-user-question-format-compliance.test.ts',
|
||||
'test/skill-e2e-hermetic-canary.test.ts',
|
||||
'test/skill-e2e-ios.test.ts',
|
||||
'test/skill-e2e-plan-ceo-finding-floor.test.ts',
|
||||
'test/skill-e2e-plan-ceo-plan-mode.test.ts',
|
||||
'test/skill-e2e-plan-design-with-ui.test.ts',
|
||||
'test/skill-e2e-plan-devex-finding-floor.test.ts',
|
||||
'test/skill-e2e-plan-devex-plan-mode.test.ts',
|
||||
]);
|
||||
|
||||
/**
|
||||
* Matrix files whose whole-file tier guard has no matching row `tier:`
|
||||
* property (pre-existing, found 2026-08-26). Consequences today:
|
||||
* - codex-e2e / gemini-e2e declare 'periodic' → both jobs run ZERO tests and
|
||||
* report green on every PR (vestigial rows; the periodic cron lane owns
|
||||
* these suites).
|
||||
* - the two PTY plan-mode smokes declare 'gate' → the e2e-pty-plan-smoke job
|
||||
* spends ~7 min on container setup and skill registration, then bun test
|
||||
* skips every describe — hollow-green since the files adopted
|
||||
* describeE2ETier.
|
||||
* Fixing either means deliberately (re)activating paid suites on every PR —
|
||||
* tracked in the same TODOS burn-down. Fix = add `tier:` to the row (or
|
||||
* delete the vestigial row), then DELETE the entry here.
|
||||
*/
|
||||
const KNOWN_TIER_UNSET = new Map([
|
||||
['test/codex-e2e.test.ts', 'periodic'],
|
||||
['test/gemini-e2e.test.ts', 'periodic'],
|
||||
['test/skill-e2e-office-hours-auto-mode.test.ts', 'gate'],
|
||||
['test/skill-e2e-plan-mode-no-op.test.ts', 'gate'],
|
||||
]);
|
||||
|
||||
interface MatrixRow {
|
||||
name: string;
|
||||
files: string[];
|
||||
tier?: string;
|
||||
}
|
||||
|
||||
/** Parse the `matrix: suite:` rows (name / file / optional tier) from evals.yml. */
|
||||
function parseMatrixRows(source: string): MatrixRow[] {
|
||||
const rows: MatrixRow[] = [];
|
||||
let current: MatrixRow | null = null;
|
||||
for (const line of source.split('\n')) {
|
||||
const name = line.match(/^\s+- name: (\S+)\s*$/);
|
||||
if (name) {
|
||||
if (current) rows.push(current);
|
||||
current = { name: name[1], files: [] };
|
||||
continue;
|
||||
}
|
||||
if (!current) continue;
|
||||
const file = line.match(/^\s+file: (.+?)\s*$/);
|
||||
if (file) current.files.push(...file[1].trim().split(/\s+/));
|
||||
const tier = line.match(/^\s+tier: (\S+)\s*$/);
|
||||
if (tier) current.tier = tier[1];
|
||||
// `steps:` ends the strategy block — stop before step-level keys leak in.
|
||||
if (/^\s{4}steps:\s*$/.test(line)) break;
|
||||
}
|
||||
if (current) rows.push(current);
|
||||
return rows.filter((r) => r.files.length > 0);
|
||||
}
|
||||
|
||||
const wholeFileTier = (source: string): string | null => {
|
||||
const m =
|
||||
/\b(?:describeE2ETier|e2eTierEnabled)\(\s*['"`](gate|periodic)['"`]/.exec(source) ||
|
||||
/EVALS_TIER\s*===\s*['"`](gate|periodic)['"`]/.exec(source);
|
||||
return m ? m[1] : null;
|
||||
};
|
||||
|
||||
const workflowSource = fs.readFileSync(WORKFLOW, 'utf-8');
|
||||
const rows = parseMatrixRows(workflowSource);
|
||||
const matrixFiles = new Map<string, MatrixRow>();
|
||||
for (const row of rows) for (const f of row.files) matrixFiles.set(f, row);
|
||||
|
||||
const paidFiles = fs
|
||||
.readdirSync(path.join(ROOT, 'test'))
|
||||
.filter((f) => f.endsWith('.test.ts'))
|
||||
.map((f) => `test/${f}`)
|
||||
.filter(isPaidTestFile);
|
||||
|
||||
describe('evals.yml matrix completeness (gate-lane silent-skip tripwire)', () => {
|
||||
test('matrix parse sanity: rows and known suites present', () => {
|
||||
expect(rows.length).toBeGreaterThanOrEqual(15);
|
||||
expect(matrixFiles.has('test/skill-e2e-workflow.test.ts')).toBe(true);
|
||||
expect(matrixFiles.has('test/skill-e2e-ship-docsync.test.ts')).toBe(true);
|
||||
});
|
||||
|
||||
test('every matrix file exists on disk', () => {
|
||||
const missing = [...matrixFiles.keys()].filter(
|
||||
(f) => !fs.existsSync(path.join(ROOT, f))
|
||||
);
|
||||
expect(missing).toEqual([]);
|
||||
});
|
||||
|
||||
test('every gate-hosting paid file is in the matrix (or the documented backlog)', () => {
|
||||
const gaps: string[] = [];
|
||||
for (const file of paidFiles) {
|
||||
const source = fs.readFileSync(path.join(ROOT, file), 'utf-8');
|
||||
const declaresGate = wholeFileTier(source) === 'gate';
|
||||
const inGateDeps = Object.entries(E2E_TOUCHFILES).some(
|
||||
([key, deps]) =>
|
||||
(E2E_TIERS as Record<string, string>)[key] === 'gate' &&
|
||||
(deps as string[]).includes(file)
|
||||
);
|
||||
if (!declaresGate && !inGateDeps) continue;
|
||||
if (matrixFiles.has(file) || KNOWN_MATRIX_GAPS.has(file)) continue;
|
||||
gaps.push(file);
|
||||
}
|
||||
expect(
|
||||
gaps,
|
||||
`Gate-hosting test file(s) missing from the evals.yml matrix — CI will ` +
|
||||
`never run them and "gate tier blocks merge" becomes silently false. ` +
|
||||
`Add a matrix row (with tier: gate when the file is whole-file gated). ` +
|
||||
`Do NOT extend KNOWN_MATRIX_GAPS for new files.`
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('matrix rows for whole-file-gated files carry a matching tier property', () => {
|
||||
const mismatches: string[] = [];
|
||||
for (const [file, row] of matrixFiles) {
|
||||
if (!fs.existsSync(path.join(ROOT, file))) continue;
|
||||
const declared = wholeFileTier(fs.readFileSync(path.join(ROOT, file), 'utf-8'));
|
||||
if (!declared) continue;
|
||||
if (row.tier === declared) continue;
|
||||
if (KNOWN_TIER_UNSET.get(file) === declared && row.tier === undefined) continue;
|
||||
mismatches.push(`${file} declares '${declared}' but row '${row.name}' has tier: ${row.tier ?? 'unset'}`);
|
||||
}
|
||||
expect(
|
||||
mismatches,
|
||||
`A whole-file tier guard with no matching row tier means the suite ` +
|
||||
`self-skips and the CI job reports a hollow green. Set tier: <declared> ` +
|
||||
`on the row (the Run step exports it as EVALS_TIER).`
|
||||
).toEqual([]);
|
||||
});
|
||||
|
||||
test('burn-down lists hold only live gaps (ratchet cleanup enforcement)', () => {
|
||||
const staleGaps = [...KNOWN_MATRIX_GAPS].filter(
|
||||
(f) => matrixFiles.has(f) || !fs.existsSync(path.join(ROOT, f))
|
||||
);
|
||||
expect(
|
||||
staleGaps,
|
||||
'Entry fixed or file removed — delete it from KNOWN_MATRIX_GAPS.'
|
||||
).toEqual([]);
|
||||
const staleTiers = [...KNOWN_TIER_UNSET.entries()].filter(([f, declared]) => {
|
||||
const row = matrixFiles.get(f);
|
||||
if (!row) return true; // row deleted — entry no longer applies
|
||||
if (row.tier === declared) return true; // fixed — entry must go
|
||||
if (!fs.existsSync(path.join(ROOT, f))) return true;
|
||||
return wholeFileTier(fs.readFileSync(path.join(ROOT, f), 'utf-8')) !== declared;
|
||||
});
|
||||
expect(
|
||||
staleTiers.map(([f]) => f),
|
||||
'Entry fixed, row removed, or guard changed — delete it from KNOWN_TIER_UNSET.'
|
||||
).toEqual([]);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user