Files
gstack/test/workflow-excerpt.test.ts
T
Garry Tan 943105f109 v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals (#2994)
* test: delete test-infrastructure dead code (G)

- exit-propagation drives the runner's real strict verdict
  (BunTestOutputClassifier + strictTestExitCode); delete the unused
  shardRunLooksTruncated predicate.
- delete skill-coverage-matrix registry + its gate (nothing reads it; the
  floor already iterates skillCensus()).
- delete touchfiles-facade export-parity tests (Bun fails missing imports
  at link time) and the duplicated E2E_TIERS tier-value test.
- delete brain-cache-spec TRANSPORT_DEFAULT_POLICY, SKILL_RUN_RETENTION_DAYS
  and the now-unused BrainTrustPolicy type with their literal tests.
  AUTOPLAN_PREFLIGHT_BUDGET_BYTES stays: skill-preflight-budget enforces it
  against real resolver output.
- delete audit-compliance's JSDoc-comment grep.

* test: replace product tests that fake the product with real-boundary tests (F)

- design: serve.test.ts drove an inline mirror server; now two tests run the
  real serve() on an ephemeral port (reload confinement, submit exit 0).
- setup-gbrain: rollback + voyage tests execute the template-extracted init
  blocks (3 sites) instead of drifted local bash copies.
- terminal-agent: internalHandler source greps replaced by a behavioral
  /internal/grant + /internal/revoke auth matrix (no/wrong/valid token).
- /health: server-security-surface and the server-auth / security-audit-r2 /
  sidebar-tabs source greps fold into one liveness-only check on the real
  body; the L4 sidecar wiring gets a behavioral /pty-inject-scan test.
- delete tautologies (browser-manager onDisconnect, memory-command #12),
  ios swiftui tap fixture self-check, memory-ingest put_page grep, detach
  source greps, sidebar-agent absence pins, dead-CSS pins + the dead CSS,
  security-audit-r2 Task 1 + the test-only meta-commands re-export,
  duplicate generated-SKILL.md checks.
- make-pdf coverage-gaps cases move into their owner test files.

* test: delete tests of dead eval code (A)

- A1: the retired Eng lexical oracle (evaluateEngSeedCoverage,
  isEngSeedDecisionAUQ), the completion-handoff detector and the retained
  corpus had no paid caller since v1.87.6; delete their 26 replay files,
  ~2.6k helper LOC and fixtures, and the dead blocks in 8 mixed files
  (live hasNativePlanTerminal / batching assertions stay).
- A2: dead viewport approvers in autoplan-artifact-permission and their 11
  replay files + fixtures; recorder/launcher cases stay.
- A3: never-wired oracles and seeders (autoplan-phase-order,
  eng-finding-fixture, ceo-paired-fixture, design-ui-scope,
  plan-skill-completion, pty-current-screen, required-reads,
  transcript-section-logger); plan-seed-submission now decodes through the
  production createPtyScreen; section manifests name their actual guard.
- A4: zero-reference helper exports, plus execGit and invokeAndObserve
  found by the reachability pass.
- 52 fixtures orphaned by the deletions; touchfile and selection-table
  entries for every deleted path.

* test: clean up the paid eval lane (B1-B4, B6, B7)

- B1: delete paid files that assert nothing or cannot pass meaningfully:
  skill-llm-eval-spec and skill-e2e-spec-execute (test.todo), gemini-e2e
  (+ gemini-session-runner; no gemini CLI in CI), ship-idempotency (red
  since v1.63), the two opus-4-7 *-sonnet overlay wrappers, conductor-prose
  (+ its source-evaluation replay), codex-e2e-plan-format; drop their keys,
  scripts and census rows.
- B2: skill-llm-eval grades browse/sections/command-list.md with one union
  judge that also carries the baseline score pin; regression-vs-baseline
  deleted (paid run: pass, c4/c4/a4).
- B3: memory-pipeline, ios-qa, ios-qa-swift-build and plan-tune-cathedral
  make no model calls; renamed out of the paid glob so they run on every
  PR. Swift builds need GSTACK_TEST_SWIFT=1; device stub deleted.
- B4: codex-e2e*, outside-voice, aside and ios-device cannot run in the CI
  image; excluded from the weekly lane with a tracked re-entry condition.
- B6: fold opus-47's negative routing controls into skill-routing-e2e
  journey-negatives (paid run: 3/3 unrouted) and delete the file.
- B7: delete the never-green brain-privacy-gate eval; a free
  gstack-skill-start test now proves consent precedes artifacts egress.

* test: retire the finding-count cluster and trim its helpers (C)

- C0/C1: the five never-green evals (skill-e2e-autoplan-chain and
  skill-e2e-plan-{ceo,eng,design,devex}-finding-count) failed on harness and
  budget, never on skill behavior; delete them, their touchfile/tier ids,
  AUTOPLAN_CHAIN_BUDGET and the dedicated eighth periodic slice (--slices 7).
- C2: delete the helper groups whose only paid consumers were those files
  (11 modules), trim claude-pty-runner and eng-seeded-coverage to the paid
  closure, and delete the free replay tests whose assertions exercised only
  that dead code (89 files, 135 orphaned fixtures). Blocks that used dead code
  only as input for a live subject keep their assertions: the multiSelect
  default moved to plan-review-decisions, runner PTY tests use inline caller
  policies, and the timer-safe budget checks moved to eng-finding-retry-budget.
- The eight production-touching files stay except ceo-current-decision-record
  (its template read only feeds the retired counter).
- CARVE_GUARDS.autoplan is behavioral 'none'; TODOS records the lost chain
  and per-finding cadence coverage with their re-entry tests.

* test: fold per-incident replay series into their detector owners (D)

Twelve detector families move into one owner test each: 73 incident files
become describe blocks in ceo-section-loading-fixture (stale-fill race),
model-overlays, coverage-audit-evidence, autoplan-phase-observer,
native-auto-decide, outside-voice-evidence, eng-first-review,
plan-count-completion, plan-count-file-permission, ceo-mode-option,
plan-scope-selection and plan-count-prerequisite. Each block keeps its original
code and fixture, so every case still runs; only tests asserting the incident
file's own touchfile registration are dropped (41). Touchfile lists that named
an incident now name its owner.

* test: start the plan-count history PTY on its readiness marker (H)

The fake CLI prints a startup marker and the runner waits for it instead of the
fixed 8 s startup sleep (8.6 s -> 0.9 s locally). eng-semantic-terminal's
sleeping registration cases went with C; plan-count-timeout keeps the fixed wait
because it asserts deadline behavior.

* test: derive paid touchfiles from each eval's static closure (E)

touchfiles.test.ts now checks, per key, that the paid file's static
test/helpers and test/fixtures closure (plus fixture paths it names in string
literals) is covered, and names the file, path, chain and key to fix when it is
not. Free *.test.ts files are no longer touchfiles, so editing a free replay
test stops selecting paid evals: 950 entries removed, 653 real closure paths
added. The hand-copied inventories go: periodic-fixture-selection,
fake-impeccable-touchfiles and 45 per-file selection examples. Selection for
the sample edits (plan-eng-review template, claude-pty-runner,
plan-count-fixture, gstack-config) loses no case under either profile.
CONTRIBUTING documents the rule and its lower bound.

* test: skip hollow tier shards and census judges in the paid planner (B5)

A paid file is now skipped for a tier lane only when every E2E id it registers
is known statically and none has that tier; ids come from the touchfile
registrations and literal testName/*IfSelected arguments, so a comment or
skill path that quotes another id cannot unschedule it, and computed names
keep today's scheduling. --list and the manifest show each skip as
"skipped: no E2E_TIERS id has tier <tier>". The weekly gate census drops the
LLM judges (--skip-judges); they still run in the periodic census and PR gate
lanes. Gate lane 52 -> 42 files, census 41; periodic 77 -> 69.

* test: run seven paid evals on the current default capture model (B8)

skill-e2e-{auq-matrix,plan-format,qa-bugs,retro,workflow} pinned
claude-opus-4-7 and skill-e2e-office-hours plus -brain-writeback pinned
claude-sonnet-4-6; none tests a historical model, so they now capture with
resolveEvalModel('capture'), and the free harness tests that execute these
registrations receive the same resolver. The paid re-pin run passed all of
them. skill-e2e-{design,office-hours-phase4,plan-prosons,plan} keep
claude-opus-4-7: six of their cases failed on the default model (three
timeouts, a missing report file, a format miss and a posture score of 3), so
per the plan's fallback they keep their pins with a TODOS entry. The pre-spend
estimate and drop threshold are in docs/test-audit-2026-09.md.

* test: guard the reduced suite against new test-of-test files

- test/test-of-test-ratchet.test.ts records the 228 free tests that import only
  test/ code and fails on a new one, naming the owner test to extend instead;
  a stale baseline entry fails with the remove instruction.
- test/helpers/resolve-repo-path.ts is the one specifier/literal resolver for
  the ratchet and the touchfile closure invariant, with its own unit tests.
- CONTRIBUTING "Test tiers" describes the paid-failure workflow (fix, then one
  row in the detector's owner test) and the ratchet; TEST_PORTFOLIO gains the
  detector -> owner-test table and no longer claims an Autoplan chain eval.
- TODOS: automatic exclusion policy for chronically red periodic files (P3),
  the deferred native-completion table collapse, the unused CEO payment
  seeder; the PTY readiness item is narrowed to the paid runner.
- docs/test-audit-2026-09.md collects the triage, security mapping, inventories,
  selection proof, behavior-commit decisions and retained false positives.

* v1.91.8.0 test: smaller suite, derived paid selection, retired never-green evals

Release metadata for the test-reduction branch: VERSION 1.91.8.0 (1.91.7.0 is
claimed by #2983), CHANGELOG with the measured before/after table and a
contributor section, durations re-recorded on Ubicloud standard-16 (857 files,
0 failures), the agents digest, CONTRIBUTING's after-measurement row, the B8
fallback TODOS entry, and the after metrics, kept-vs-plan notes, B8 run and
census estimate in docs/test-audit-2026-09.md.

* fix(ubicloud): skip retrieval globs that match nothing instead of reporting a failed pull

* test: count issue-numbered Design findings in the UI-scope gate eval
2026-09-29 08:58:49 -07:00

353 lines
24 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { describe, expect, test } from 'bun:test';
import { ENG_REVIEW_EXCERPT, readWorkflowExcerpt } from './helpers/workflow-excerpt';
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { spawnSync } from 'child_process';
function expectOutsideReviewControlFlow(text: string, promptHeading: string): void {
const ceo = text.includes('**Record the disabled outcome:**');
const markers = [ceo ? '**Record the disabled outcome:**' : '**Disabled is a terminal branch', promptHeading, '**If `CODEX_MODE: ready`', '\n**Native fallback —'];
const indices = markers.map(marker => text.indexOf(marker));
expect(indices.every(index => index >= 0)).toBe(true);
expect(indices).toEqual([...indices].sort((a, b) => a - b));
const disabled = text.slice(indices[0], indices[1]);
if (ceo) {
expect(disabled).toContain('"outside_status":"disabled"');
expect(disabled.replace(/\s+/g, ' ')).toContain('without a challenge, CLI invocation, Agent/Task fallback or questions about outside findings');
expect(disabled).toContain('_DISABLED_REVIEW_MODE=');
expect(disabled).toContain('if [ "$_DISABLED_REVIEW_MODE" = disabled ]');
} else {
expect(disabled).toContain('persist `outside_status: disabled`');
expect(disabled.replace(/\s+/g, ' ')).toMatch(/Do not construct a (?:review prompt|challenge), invoke an outside CLI, dispatch an Agent\/Task fallback/);
}
expect(text.slice(indices[1], indices[2])).toContain('(skip only on `disabled`)');
const fallback = text.slice(indices[3]);
if (text.includes('**Outcome routing:**')) {
const routing = text.slice(text.indexOf('**Outcome routing:**'), indices[0]);
expect(routing).toContain('Other preflight mode, including harness mismatch');
expect(routing).toContain('Outside execution or output validation fails');
expect(routing).toContain('Retain its output and diagnosis, finish termination, then use Native fallback');
expect(routing).toContain('No prompt, outside process or native replacement');
expect(fallback.replace(/\s+/g, ' ')).toContain('Immediately before dispatch, check the preflight result again: disabled means no replacement');
} else if (ceo) {
expect(fallback.replace(/\s+/g, ' ')).toContain('Other preflight failures retain their printed diagnosis, including harness mismatch');
expect(fallback.replace(/\s+/g, ' ')).toContain('These failures do not block the review; they use the bounded fallback below');
} else {
expect(fallback).toContain('The disabled branch never reaches this fallback.');
expect(fallback.replace(/\s+/g, ' ')).toMatch(/Otherwise, use this fallback for missing\/broken CLI, failed authentication\/model selection, a failed preflight(?: \(including harness mismatch\))?, or a failed outside invocation\./);
}
const dispatch = fallback.indexOf('Dispatch via the Agent tool');
expect(dispatch).toBeGreaterThan(0);
const recheck = fallback.slice(0, dispatch);
if (ceo) {
expect(recheck.replace(/\s+/g, ' ')).toContain('Immediately before dispatch, recheck the preflight result');
expect(recheck.replace(/\s+/g, ' ')).toContain('`CODEX_MODE: disabled`, return to **Record the disabled outcome** without dispatching');
} else if (!text.includes('**Outcome routing:**')) {
expect(recheck).toContain('Immediately before dispatching, check the preflight result again.');
expect(recheck).toContain('`CODEX_MODE: disabled`, finish this section with `outside_status: disabled`;');
expect(recheck).toContain('do not dispatch.');
}
expect(fallback).toContain('Availability/native fallback is not outside completion.');
}
describe('workflow judge excerpts', () => {
test('expands ship sections in execution order, not alphabetical order', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
const headings = ['## Step 3:', '## Step 4:', '## Step 7:', '## Step 8:', '## Step 9:', '## Step 10:', '## Step 11:', '## Step 11.5:', '## Step 12:', '## Step 13:', '## Step 14:'];
const indices = headings.map(heading => text.indexOf(heading));
expect(indices.every(index => index >= 0)).toBe(true);
expect(indices).toEqual([...indices].sort((a, b) => a - b));
});
test('ship uses project-native commands and never jumps over mandatory gates', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
expect(text).toContain("Use the project's test commands discovered in Step 4");
expect(text).toContain('**Project-native path:**');
expect(text).toContain('Use the documented selector and pre-merge command.');
expect(text).not.toMatch(/skipping evals[^\n]*Step 9/);
const reviewAndTriage = text.slice(text.indexOf('## Step 9:'), text.indexOf('## Step 11:'));
expect(reviewAndTriage.match(/continue to Step 12/i)).toBeNull();
expect(text).not.toContain('Steps 4-6:');
expect(text).toContain('During pre-flight, read the existing review log');
expect(text).toContain('Save the JSON `baseVersion` as `BASE_VERSION`');
expect(text).not.toContain('GIT_SEQUENCE_EDITOR');
expect(text).not.toContain("--exec 'true'");
expect(text).not.toContain('-X ours');
expect(text).toContain('````text\nYou are running a ship-workflow');
});
test('ship review shortcuts retain dedup and fixes repeat the whole review cycle', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
expect(text).toContain("Continue to Step 9.2 with the core/design-lite findings and an empty specialist list, then the parent's Exploratory QA step and Step 9.3 (cross-review dedup)");
expect(text).toContain('## Step 9.4: Fix-First and persistence');
expect(text.replace(/\s+/g, ' ')).toContain('Run checklist/design, specialists (9.1), merge/Red Team (9.2), exploratory QA (9.2.1), dedup (9.3), then fixes and logging (9.4)');
expect(text.replace(/\s+/g, ' ')).toContain('**Fixes applied below the cap:** Insert Step 5, affected Steps 6–8 and all of Step 9 before the pending Step 10 in the work list. Tests must pass or retain approval for the same verified pre-existing failures and scope');
const audit = text.slice(text.indexOf('## Step 7:'), text.indexOf('## Step 8:'));
expect(audit).not.toContain('Scope Challenge');
expect(text.replace(/\s+/g, ' ')).toContain('Keep actual outcomes and incomplete flags; VERIFY_RESULT stays fail for plan-check exceptions');
});
test('ship excerpt preserves readable detours, audit fallback and final input decisions', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules').replace(/\s+/g, ' ');
expect(text).toContain('For another repair, repeat rule 2 without discarding pending work');
expect(text).toContain('A further Step 9 fix affecting 6–8 makes the list `5 → 6 → 7 → 8 → 9 → 10 → 11 → 11.5`');
expect(text).toContain('The unchanged release steps follow');
expect(text).toContain('All three snapshots must match');
expect(text).toContain('does not mean the failed or unrun probes passed');
expect(text).toContain('Fallback recovers the audit; it does not pass or bypass the coverage gate');
expect(text).toContain('Skip only the plan completion audit');
expect(text).toContain('Continue with Step 8.1, Scope Drift and Prior Learnings');
expect(text).toContain('Step 9 QA still runs');
expect(text).toContain('Use this example only after confirming that every allowed edit is release metadata');
expect(text).not.toContain('Every listed change below is metadata:');
expect(text).not.toContain('No plan file found:** Skip entirely');
});
test('a sliced section is not appended again with its generated header', () => {
const text = readWorkflowExcerpt('plan-design-review/SKILL.md', '## Review Sections', '## CRITICAL RULE');
expect(text.match(/## Review Sections/g)).toHaveLength(1);
expect(text).not.toContain('## CRITICAL RULE');
expect(text).not.toContain('AUTO-GENERATED');
});
test('ship publishes existing PRs only after shared body composition and scan', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
const publish = text.slice(text.indexOf('## Step 19:'), text.indexOf('## Step 20:'));
const compose = publish.indexOf('PR_BODY_FILE=$(mktemp)');
const scan = publish.indexOf('gstack-redact --from-file "$PR_BODY_FILE"');
const edit = publish.indexOf('gh pr edit --body-file');
expect(compose).toBeGreaterThan(0);
expect(scan).toBeGreaterThan(compose);
expect(edit).toBeGreaterThan(scan);
expect(publish.indexOf('Print the existing URL')).toBeGreaterThan(edit);
expect(text).not.toContain('Phase 8e.5');
expect(text).toContain('never create an empty commit');
const review = text.slice(text.indexOf('## Step 9:'), text.indexOf('## Step 10:'));
expect(review.indexOf('## Confidence Calibration')).toBeLessThan(review.indexOf('1. Read'));
const flat = review.replace(/\s+/g, ' ');
expect(flat).toContain('**No edits in this pass:** Resolve the required-probe gate below. Only after it clears may you continue to Step 10');
expect(flat).toContain('**Dispatched reviewer output missing:** STOP');
expect(flat).toContain('Retain queued fixes and restore coverage');
expect(flat).toContain('**Third fixing cycle reached (`CYCLES >= 3`):** STOP and report recurring findings with `converged:false`; do not run a fourth fixing cycle');
expect(flat).toContain('With completed checklist and dispatched reviewers, failed/unavailable required probes block continuation');
});
test('ship approval gates stay outside the subagent prompts', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
for (const [step, next, gate] of [[7, 8, '**7. Coverage gate:**'], [8, 9, '### Gate Logic']] as const) {
const section = text.slice(text.indexOf(`## Step ${step}:`), text.indexOf(`## Step ${next}:`));
const prompt = section.match(/````text\n([\s\S]*?)\n````/)![1];
expect(prompt).not.toContain(gate);
expect(prompt).not.toContain('Use AskUserQuestion:');
expect(prompt).not.toContain('commit as');
expect(section.indexOf(gate)).toBeGreaterThan(section.indexOf('\n````\n'));
}
expect(text).toContain('"partial":N,"not_done":N');
expect(text).toContain('each Y\'d item with the user\'s free-text evidence and each D\'d item with "intentionally dropped"');
});
test('expands a body before the end marker in the skeleton', () => {
const text = readWorkflowExcerpt('document-release/SKILL.md', '# Document Release:', '## Important Rules');
expect(text).toContain('## Step 2:');
expect(text).toContain('## Step 9:');
});
test('documentation review precedes publication and keeps changelog protection', () => {
const text = readWorkflowExcerpt('document-release/SKILL.md', '# Document Release:', '## Important Rules');
expect(text).toContain('DOC_DIFF_BASE=$(git merge-base origin/<base> HEAD 2>/dev/null || git merge-base <base> HEAD) || exit 1');
const reviewStart = text.indexOf('## Codex Documentation Review');
const commit = text.indexOf('## Step 9:');
expect(reviewStart).toBeGreaterThanOrEqual(0);
expect(commit).toBeGreaterThan(reviewStart);
const review = text.slice(reviewStart, commit);
expectOutsideReviewControlFlow(review, '**Construct the doc-review prompt**');
expect(review).toContain('Skip the apply gate, persist `status: unavailable`, `outside_status: unavailable`, and `source: none`');
expect(review).toContain('present the findings, then use AskUserQuestion ONCE:');
expect(review).toContain('On A or per-finding approvals, make the approved edits yourself');
expect(text).toContain('Step 9 then commits and pushes those edits');
expect(text).toContain('Entries scoring <2 need attention, not replacement');
expect(text).not.toContain('Flag and rewrite');
expect(text).toContain('if VERSION is absent, use the completion date only');
});
test('Eng preparation and decision procedure precede the four review sections', async () => {
const { marked } = await import('marked');
const { skillPath, startMarker, endMarker } = ENG_REVIEW_EXCERPT;
const eng = readWorkflowExcerpt(skillPath, startMarker, endMarker);
const stages = ['## Review preparation', '## Retrospective learning', '## Confidence Calibration', '## Decision procedure',
'### Prepare an unanswered choice', '## Scope Challenge', '### A. Assess the target',
'### B. Resolve complexity selectors', '### C. Resolve findings', '## Review Sections',
'### 1. Architecture review', '### 2. Code quality review', '### 3. Test review', '### 4. Performance review']
.map(heading => eng.indexOf(heading));
expect(stages.every(index => index >= 0)).toBe(true);
expect(stages).toEqual([...stages].sort((a, b) => a - b));
expect(eng.match(/^## Decision procedure$/gm)).toHaveLength(1);
const procedure = eng.slice(eng.indexOf('## Decision procedure'), eng.indexOf('## Scope Challenge'));
const headings = marked.lexer(procedure).filter(token => token.type === 'heading' && token.depth === 3);
expect(headings.map(token => token.text)).toEqual(['Prepare an unanswered choice', 'Send once and wait', 'Record the answer']);
expect(procedure).toContain("**Pending-record checkpoint.**");
expect(procedure).toContain('### Send once and wait');
expect(procedure).toContain("### Record the answer");
const outputs = ['### TODOS.md updates', '## Approval readiness', '## Required outputs', '## Implementation Tasks',
'### Unresolved decisions', '### Completion summary', '## Plan File Review Report',
'### Write to the report file', '## Review Log'].map(heading => eng.indexOf(heading));
expect(outputs.every(index => index > stages[stages.length - 1]!)).toBe(true);
expect(outputs).toEqual([...outputs].sort((a, b) => a - b));
});
test('CEO mode handoff precedes its route and spec review stays within persistence', () => {
const ceo = readWorkflowExcerpt('plan-ceo-review/SKILL.md', '## Step 0: Nuclear Scope Challenge', '## Review Sections');
const positions = ['### 0D.', '### 0E. Mode Selection', '**Mode handoff:**',
'Follow the selected mode\'s route:', '### 0F.', '### 0G. Mode-Specific Analysis',
'### 0H.', '#### Spec Review Loop', '### 0I. Temporal Interrogation']
.map(heading => ceo.indexOf(heading));
expect(positions.every(index => index >= 0)).toBe(true);
expect(positions).toEqual([...positions].sort((a, b) => a - b));
const persistence = ceo.slice(positions[6], positions[8]);
expect(persistence.match(/^#### Spec Review Loop$/gm)).toHaveLength(1);
expect(persistence).not.toMatch(/^## Spec Review Loop$/m);
expect(ceo.slice(positions[2], positions[3])).toContain('Auto-decided review mode → <selected mode> (your preference)');
expect(ceo.slice(positions[2], positions[3])).toContain('Mode: <selected mode>; approved decisions: <rows or none>');
});
test('CEO Step 0 headings follow their sequential execution labels', () => {
const ceo = readWorkflowExcerpt('plan-ceo-review/SKILL.md', '## Step 0: Nuclear Scope Challenge', '## Review Sections');
const labels = [...ceo.matchAll(/^### (0[A-Z](?:-[A-Za-z]+)?)\. /gm)].map(match => match[1]);
expect(labels).toEqual(['0A', '0B', '0C', '0D', '0E', '0F', '0G', '0H', '0I']);
});
test('CEO capture locates Mode Selection by name for current and frozen skill copies', () => {
const dir = mkdtempSync(join(tmpdir(), 'ceo-semantic-capture-'));
const helper = join(import.meta.dir, 'helpers', 'auq-sdk-capture.ts');
const runner = join(import.meta.dir, 'helpers', 'agent-sdk-runner.ts');
const script = join(dir, 'capture.ts');
writeFileSync(script, `import { mock } from 'bun:test';
import * as fs from 'node:fs';
import * as path from 'node:path';
const calls = [];
mock.module(${JSON.stringify(runner)}, () => ({resolveClaudeBinary:()=>'/fixture/claude',runAgentSdkTest: async options => {
calls.push(options);
void options.canUseTool('AskUserQuestion', {questions:[{header:'Mode',question:'captured mode choice',
options:['SCOPE EXPANSION','SELECTIVE EXPANSION','HOLD SCOPE','SCOPE REDUCTION'].map(label=>({label,description:''}))}]},
{toolUseID:'native-mode',signal:options.signal});
options.signal.throwIfAborted();
throw Error('capture failed to stop before answering');
}}));
const {captureModeSelectionAuq, verboseSkill} = await import(${JSON.stringify(helper)});
const current = fs.readFileSync(${JSON.stringify(join(import.meta.dir, '..', 'plan-ceo-review', 'SKILL.md'))}, 'utf8');
const results = [];
for (const [variant, skill] of [['current', current], ['frozen', verboseSkill()]]) {
const planDir = path.join(${JSON.stringify(dir)}, variant);
fs.mkdirSync(path.join(planDir, 'plan-ceo-review'), {recursive:true});
fs.writeFileSync(path.join(planDir, 'plan-ceo-review', 'SKILL.md'), skill);
fs.writeFileSync(path.join(planDir, 'plan.md'), 'Review this plan.');
results.push({variant, heading:skill.match(/^### (0[A-Z])\\. Mode Selection/m)?.[1],
captured:await captureModeSelectionAuq({planDir, testName:'free-semantic-capture', model:'fake-model'})});
}
console.log(JSON.stringify({calls, results}));
`);
try {
const child = spawnSync(process.execPath, [script], { encoding: 'utf8', timeout: 10_000 });
expect(child.status, `${child.error ?? ''}\n${child.stderr}`).toBe(0);
const { calls, results } = JSON.parse(child.stdout.trim().split('\n').at(-1)!);
expect(results).toEqual([
{ variant: 'current', heading: expect.stringMatching(/^0[A-Z]$/), captured: expect.stringContaining('captured mode choice') },
{ variant: 'frozen', heading: '0F', captured: expect.stringContaining('captured mode choice') },
]);
expect(calls).toHaveLength(2);
for (const call of calls) {
expect(call.userPrompt).toContain('Proceed to Mode Selection,');
expect(call.userPrompt).not.toMatch(/Step 0[A-Z]/);
expect(call.userPrompt).toContain(join(call.workingDirectory, 'plan-ceo-review', 'SKILL.md'));
expect(call.userPrompt).toContain('Do NOT search for, Glob, find, or read any OTHER SKILL.md');
expect(call.userPrompt).toContain('Ask the user through the AskUserQuestion tool and wait for their answer.');
expect(call).toMatchObject({ allowedTools: ['Read', 'Write', 'AskUserQuestion'], maxTurns: 12, maxRetries: 0, model: 'fake-model' });
}
} finally { rmSync(dir, { recursive: true, force: true }); }
});
test('Eng LLM scope and pending decisions precede the test artifact', () => {
const eng = readWorkflowExcerpt('plan-eng-review/SKILL.md', '## Scope gate', '## Section self-check');
const tests = eng.slice(eng.indexOf('### 3. Test review'), eng.indexOf('### 4. Performance review'));
const scope = tests.indexOf('### LLM/eval scope');
const decisions = tests.indexOf('**Step 5. Add missing tests to the plan:**');
const stop = tests.indexOf("**STOP for each pending decision.**", decisions);
const artifact = tests.indexOf('### Test Plan Artifact');
expect(0 <= scope && scope < decisions && decisions < stop && stop < artifact).toBe(true);
expect(tests.match(/For LLM\/prompt changes:/g)).toHaveLength(1);
expect(tests.slice(artifact)).not.toContain("**STOP for each pending decision.**");
const fastPath = tests.slice(tests.indexOf('**Fast path:**'), scope);
expect(fastPath).toContain('Still check LLM/eval scope and produce the Test Plan Artifact');
});
test('plan review evidence and design approval rules precede their use', () => {
const eng = readWorkflowExcerpt('plan-eng-review/SKILL.md', '## Scope gate', '## Section self-check');
expect(eng.indexOf('## Confidence Calibration')).toBeLessThan(eng.indexOf('### 1. Architecture review'));
expect(eng).toContain('quote the motivating plan requirement');
expectOutsideReviewControlFlow(eng, '**Construct the plan review prompt**');
expect(eng).toContain('Agreement between reviewers is evidence, not approval');
expect(eng).toContain('new or reopened choices still need their own answers');
const pendingDecision = eng.slice(eng.indexOf('### Send once and wait'), eng.indexOf("### Record the answer"));
expect(pendingDecision).toContain("**STOP until the actual answer arrives.**");
expect(pendingDecision.replace(/\s+/g, ' ')).toContain("Do not apply a remedy, make another call, start the next section or call ExitPlanMode while the choice awaits an answer");
expect(eng.replace(/\s+/g, ' ')).toContain("Use a scoped Edit to save this record and only the authorized working-plan amendments. Leave other choices unchanged");
const design = readWorkflowExcerpt('plan-design-review/SKILL.md', '## Review Sections', '## CRITICAL RULE');
expect(design).toContain('wait for approval, then edit the plan and re-rate');
const pass4 = design.slice(design.indexOf('### Pass 4:'), design.indexOf('### Pass 5:'));
expect(pass4.match(/^### /gm)).toHaveLength(1);
expect(pass4).toMatch(/^#### Design Hard Rules$/m);
expect(pass4.indexOf('**Pass 4 evaluation:**')).toBeLessThan(pass4.indexOf('\n#### Design Hard Rules'));
expect(pass4.indexOf('#### Design Hard Rules')).toBeLessThan(pass4.indexOf('FIX TO 10:'));
expect(pass4).toContain('caps this pass below 8');
});
test('fails closed for missing excerpt markers', () => {
expect(() => readWorkflowExcerpt('ship/SKILL.md', '# missing', null)).toThrow('Start marker not found');
expect(() => readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '# missing')).toThrow('End marker not found');
});
test('retro judge includes compare semantics and unambiguous report inputs', () => {
const text = readWorkflowExcerpt('retro/SKILL.md', '## Instructions', '## Tone');
expect(text).toContain('## Compare Mode');
expect(text).toContain('does not require saved history');
expect(text).toContain('one second before the current start');
expect(text).toContain('PRs referenced');
expect(text).toContain('prs_merged: null');
expect(text).toContain('not newly added test cases');
expect(text).toContain('`streak_days` is the live **team** streak');
expect(text).toContain('draft the tweetable summary using the format in Step 14, then save');
expect(text).toContain('### Shipping Streaks');
expect(text).toContain('### Shortcut Debt');
expect(text.indexOf('## Capture Learnings')).toBeGreaterThan(text.indexOf('### Step 14:'));
expect(text).not.toContain('$(date');
expect(text.match(/today="<today>"/g)).toHaveLength(2);
const judge = readFileSync(join(import.meta.dir, 'skill-llm-eval.test.ts'), 'utf8');
expect(judge).toMatch(/skillPath: 'retro\/SKILL.md',[\s\S]*?endMarker: '## Tone'/);
});
test('deploy gates and navigation timing formulas are executable as documented', () => {
const land = readFileSync(join(import.meta.dir, '../land-and-deploy/SKILL.md.tmpl'), 'utf8');
expect(land).not.toContain('Skip Step 3, go to Step 4');
expect(land).toContain('continue to Step 3.4, then Step 3.5 before merging');
const benchmark = readFileSync(join(import.meta.dir, '../benchmark/SKILL.md.tmpl'), 'utf8');
const timings = { startTime: 0, domInteractive: 600, domComplete: 1200, loadEventEnd: 1400 };
for (const [label, expected] of [['DOM Interactive', 600], ['DOM Complete', 1200], ['Full Load', 1400]] as const) {
const formula = benchmark.match(new RegExp(`\\*\\*${label}\\*\\*: \x60([^\x60]+)\x60`))![1];
const actual = new Function(...Object.keys(timings), `return ${formula}`)(...Object.values(timings));
expect(actual).toBe(expected);
}
});
test('ship commits logical chunks without rewriting existing checkpoint commits', () => {
const text = readWorkflowExcerpt('ship/SKILL.md', '# Ship:', '## Important Rules');
const commit = text.slice(text.indexOf('## Step 15:'), text.indexOf('## Step 16:'));
expect(commit).toContain('Make bisectable commits');
expect(commit).toContain('if already committed, continue to Step 16');
expect(commit).toContain('Each commit must work independently');
expect(commit).not.toMatch(/rebase|reset|squash|fixup|WIP_TODO|gstack-context/);
expect(text).not.toContain('Step 15.0');
});
});