Files
gstack/test/plan-reality.test.ts
T
Garry Tanandcapy-ai[bot] 8695938e74 v1.92.15.0 feat: plan reviews check facts and keep one task list; /office-hours hands its wireframe to design review (#340, #513, #973, #1048, #1791) (#3775)
* G1: gstack-plan-reality check opens every file:line receipt (#973)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G1: office-hours spec reviewer checks paths and flags through a bin-enforced probe (#973)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2: plan-review reruns reconcile the last task list (gstack-tasks prior/supersede) (#1791)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2: plan reviews write task lists through gstack-tasks start/add/close

The emit block shrinks to three calls: prior (now defaulting --dir and
--branch, and printing how to reconcile), start (creates the empty
branch-named list) and add (one validated row per task). close records a
done or dropped prior task. The plan-review result codes move to
lib/result-codes-plan.ts so lib/result-codes.ts stays under the module-size
ratchet; generated plan-review sections shrink, so their size pins go down.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* F2: office-hours saves the sketch beside the design doc; /plan-design-review reads it

Step 5 copies sketch.png/html to <PROJECT_DIR>/{user}-{branch}-sketch-{datetime}.*
and writes a Wireframe: line (state-root relative) or Wireframe: none (<reason>).
The design review section finds the design doc before Pass 1 and opens the
wireframe, text-only when the file is missing. G4's signals line rides along
in the same generated office-hours files; tests in planning-wave-prose.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G3: plan reviews tie storage to queries, not file count (#1048)

One bullet in each of plan-eng-review and plan-ceo-review: fields queried,
constrained or updated on their own get explicit columns or tables; JSON suits
a document read and written whole. Written fresh; the reporter's patch is not
ported.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* evals: GSTA-228 planning-wave cases with branch, main and no-gstack arms

planted facts (G1, behavior), data model (G3, behavior), task-list rerun (G2,
rule) and closing signals (G4, behavior). GSTACK_WAVE_ARM picks the arm; only
the branch arm asserts the preregistered bar, every arm writes a score row.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: GSTA-228 planning wave fragment

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G1: the probe takes subcommand words for fixed-subcommand CLIs

The planted-facts eval's one branch miss: the reviewer asked to probe
"git log", the probe refused it, and git's top-level help cannot show a log
flag, so the fabricated --since-commit stayed "unverified".
probe --help <command> [<subcommand> [<subcommand>]] now runs
<cli> <sub> --help for git, gh, docker, cargo, go, npm and uv (never node,
bun, python3, make, yarn, pnpm or a gstack bin, where the word could name a
script), in an empty temporary directory with pagers set to cat, falling back
to -h when --help has no usage line (a minimized system's missing man page).

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2: task text goes through a rows file, never a command line

The free suite's generated-bash lint (INV-3) refused the add command's
<title>/<finding> placeholders: free text reaches a command only as a file the
agent writes with its file-write tool. start now also creates an empty rows
file in the repository's git-excluded .gstack/tmp and records the run; the
agent writes task and disposition rows there; add validates every row,
stamps phase, run, branch and commit, appends tasks to the list and
dispositions to task-dispositions-<phase>-<branch>.jsonl, then supersedes the
branch's earlier lists. close is gone (dispositions are rows).

The new planning-wave cases get measured periodic walls in
scripts/paid-test-durations.json so the periodic census still packs into 40
slices, the schema case declares the preamble sources its eng-template owners
need, and the paid file count is 114.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2: fold start into add so each plan review emits one bash fence

The rows file lives at .gstack/tmp/tasks-<phase>-rows.jsonl; add writes this
run's list (empty when there are no rows), appends dispositions, deletes the
rows file and supersedes the branch's earlier lists. Lowers 27 size pins.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2 tests: give gstack-tasks a private HOME and state root

CI shard 13 home containment caught gstack-slug writing ~/.gstack/slug-cache
entries from the tasks-prior fixtures.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: gstack-tasks has no start command

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G1/G2: tighten the probe and the rows file after ship review

The probe now accepts only each CLI's listed built-in subcommands (no uv run,
docker run, go test, npm exec or gh extensions), refuses gstack bins that do not
handle --help and never retries them with -h, and runs with a throwaway HOME and
an environment without keys. gstack-tasks resolves the slug through
runScriptSync (Windows) and prior deletes a rows file an unfinished run left.
Adds tests for the dropped disposition and the stale rows file.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G2: add refuses while an earlier task is neither carried nor closed

Ship review (state matrix): a rerun that skipped RECONCILE silently retired
open tasks. add now names each undecided <run_id>/<id> and writes nothing.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* G1/G2: fixes from the ship fix-check and Step 11 voices

Probe: find the repository root without running git, drop relative and
in-repository PATH entries for lookup and for the child, keep HOME and
toolchain locations (rustup, mise, asdf) but no keys, drop cargo fmt/clippy
(external cargo-* programs), restore planted env in the test.
gstack-tasks: resolve the exclude path against the cwd (subdirectories),
resolve --dir, stamp an empty branch on a detached HEAD like the aggregator,
name the file from --branch, and refuse a same-second second list instead of
appending. Receipts: line 0, reversed ranges, dangling links and paths
through files are unresolved instead of crashing.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* changelog: probe limits and the reconcile refusal

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: probe subcommand counts per CLI (ship docs audit)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-01 06:11:17 +00:00

398 lines
27 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Plan D1: reality rows in plan reviews. Pins the row ids, the `REALITY:` line
* grammar, the checker's missing/unreceipted/n/a branches, the bin's exit codes
* and `PLAN_REALITY:` line, the unattended `incomplete` verdict, and that the
* rendered plan-review skills and autoplan phase files carry the contract.
* Each tier 1 row has a positive fixture from this wave's own review record
* (test/fixtures/multi-agent-wave/reality-rows.json, receipts verified against
* the record) and a decoy that must not count. Prose is not pinned.
*/
import { describe, expect, test } from 'bun:test';
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { EXIT } from '../lib/headless-artifacts';
import { PLAN_REALITY_CHECKER, REALITY_ROWS, REALITY_SCOPES, TIER1_ROW_IDS, checkPlanReality, checkRealityRows, parseRealityLines, parseScopes, renderIncidentTable, renderRows, requiredRows, vaguePlanTerms } from '../lib/plan-reality';
import { extractHeading } from '../lib/autoplan-prompts';
import { RESULT_CODES } from '../lib/result-codes';
import { generatePlanRealityRows } from '../scripts/resolvers/plan-reality';
import type { TemplateContext } from '../scripts/resolvers/types';
const ROOT = path.resolve(import.meta.dir, '..');
const BIN = path.join(ROOT, 'bin', 'gstack-plan-reality');
const FIXTURE = JSON.parse(fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'multi-agent-wave', 'reality-rows.json'), 'utf8')) as {
rows: Record<string, { positive: string; source: { file: string; lines: [number, number]; contains: string }; decoy: string; decoy_kind: string; metric?: Record<string, string> }>;
};
const positives = Object.values(FIXTURE.rows).map(r => r.positive).join('\n');
function run(args: string[], input?: string, env: Record<string, string> = {}) {
const r = spawnSync('bun', [BIN, ...args], { cwd: ROOT, encoding: 'utf8', timeout: 30_000, input, env: { ...process.env, GSTACK_SESSION_KIND: '', ...env } });
return { code: r.status, out: r.stdout, err: r.stderr };
}
describe('reality row table', () => {
test('seven tier 1 rows, every row has a check and an incident, ids are unique and scopes known', () => {
expect(TIER1_ROW_IDS).toEqual(['premise-table', 'already-done', 'surface-check', 'binding-decisions', 'numbers', 'deferred-asks', 'pr-count']);
expect(REALITY_ROWS.filter(r => r.tier === 2).length).toBe(11);
expect(new Set(REALITY_ROWS.map(r => r.id)).size).toBe(REALITY_ROWS.length);
for (const r of REALITY_ROWS) {
expect(r.check.length).toBeGreaterThan(20);
expect(r.incident.length).toBeGreaterThan(20);
expect(REALITY_SCOPES).toContain(r.scope);
expect(r.tier === 1 ? 'always' : 'scoped').toBe(r.scope === 'always' ? 'always' : 'scoped');
}
});
test('required rows: tier 1 always, tier 2 only for a detected scope', () => {
expect(requiredRows([]).map(r => r.id)).toEqual([...TIER1_ROW_IDS]);
expect(requiredRows(['db']).map(r => r.id)).toContain('transaction-semantics');
expect(requiredRows(['db']).map(r => r.id)).not.toContain('recall');
expect(parseScopes('db, search,bogus')).toEqual({ scopes: ['db', 'search'], unknown: ['bogus'] });
});
});
describe('REALITY: line grammar', () => {
test('parses id, status, receipt (path:line, path:line-line, sha) and key=value metrics; tolerates list and table prefixes', () => {
const lines = parseRealityLines([
'REALITY: premise-table finding 27 checked unverified=3 docs/x/ceo-native.md:11-37',
'- REALITY: already-done pass 0 hits 91bbd9e',
'| REALITY: surface-check finding bun upgrade bin/gstack-capy-install:52 |',
'REALITY: numbers pass 7 PRs 48 min',
'REALITY: transaction-semantics n/a scope db absent',
'reality: not-a-row pass lower-case prefix is not the grammar',
].join('\n'));
expect(lines.map(l => [l.id, l.status, l.receipt])).toEqual([
['premise-table', 'finding', 'docs/x/ceo-native.md:11-37'],
['already-done', 'pass', '91bbd9e'],
['surface-check', 'finding', 'bin/gstack-capy-install:52'],
['numbers', 'pass', null],
['transaction-semantics', 'n/a', null],
]);
expect(lines[0]!.metrics.unverified).toBe('3');
});
});
describe('checkRealityRows', () => {
test('every tier 1 positive fixture from the review record passes and the premise metric is scored', () => {
const r = checkRealityRows(positives, { phase: 'ceo', sessionKind: 'unattended' });
expect(r.missing).toEqual([]);
expect(r.unreceipted).toEqual([]);
expect(r.verdict).toBe('complete');
expect(r.metrics).toEqual({ unverified: '3' });
expect(r.line).toBe('PLAN_REALITY: phase=ceo rows=7/7 missing=none unverified=3 verdict=complete');
});
test('each positive receipt points into the review record at the stated lines', () => {
for (const [id, row] of Object.entries(FIXTURE.rows)) {
const text = fs.readFileSync(path.join(ROOT, row.source.file), 'utf8').split('\n');
const [from, to] = row.source.lines;
const excerpt = text.slice(from - 1, to).join('\n');
expect(excerpt, `${id}: ${row.source.file}:${from}-${to}`).toContain(row.source.contains);
const parsed = parseRealityLines(row.positive)[0]!;
expect(parsed.id).toBe(id);
expect(parsed.receipt, `${id} carries a receipt`).not.toBeNull();
expect(row.positive).toContain(`${row.source.file}:${from}`);
}
});
test('each decoy fails for its own row while the other six positives stand (negative control per row)', () => {
for (const [id, row] of Object.entries(FIXTURE.rows)) {
const others = Object.entries(FIXTURE.rows).filter(([k]) => k !== id).map(([, r]) => r.positive);
const r = checkRealityRows([...others, row.decoy].join('\n'), { phase: 'eng' });
expect([...r.missing, ...r.unreceipted], `${id} decoy (${row.decoy_kind}) must not count`).toEqual([id]);
expect(r.verdict).toBe('repair');
}
});
test('a review that only talks about the rows in prose has every row missing', () => {
const prose = 'The premises were checked and already-done work was considered. Surface check done. Numbers estimated.';
const r = checkRealityRows(prose, { phase: 'ceo', sessionKind: 'unattended' });
expect(r.missing).toEqual([...TIER1_ROW_IDS]);
expect(r.verdict).toBe('incomplete');
expect(r.line).toContain('rows=0/7');
});
test('verdict word: unattended → incomplete, anything else → repair; complete needs no session kind', () => {
const partial = positives.split('\n').slice(0, 6).join('\n');
expect(checkRealityRows(partial, { phase: 'eng', sessionKind: 'unattended' }).verdict).toBe('incomplete');
expect(checkRealityRows(partial, { phase: 'eng', sessionKind: 'interactive' }).verdict).toBe('repair');
expect(checkRealityRows(partial, { phase: 'eng' }).verdict).toBe('repair');
expect(checkRealityRows(partial, { phase: 'eng' }).missing).toEqual(['pr-count']);
});
test('tier 2 rows: required only under their scope; n/a is accepted off-scope and a tier 1 n/a is missing', () => {
const withNa = `${positives}\nREALITY: transaction-semantics n/a no db change`;
expect(checkRealityRows(withNa, { phase: 'eng' }).missing).toEqual([]);
expect(checkRealityRows(withNa, { phase: 'eng', scopes: ['db'] }).missing).toEqual(['transaction-semantics']);
const dbRow = `${positives}\nREALITY: transaction-semantics pass idle, pooled and reserved traced lib/pool.ts:40-88`;
expect(checkRealityRows(dbRow, { phase: 'eng', scopes: ['db'] }).line).toContain('rows=8/8');
expect(checkRealityRows(positives.replace(/^REALITY: numbers finding/m, 'REALITY: numbers n/a'), { phase: 'eng' }).missing).toEqual(['numbers']);
});
test('a later real line outranks an earlier n/a for the same row; unknown row ids are reported, not counted', () => {
const r = checkRealityRows(`REALITY: already-done n/a\n${positives}\nREALITY: premises pass lookalike plan.md:1`, { phase: 'ceo' });
expect(r.missing).toEqual([]);
expect(r.unknownRows).toEqual(['premises']);
});
});
describe('gstack-plan-reality bin', () => {
test('--help prints the exit table and the line shape; no command is usage', () => {
const help = run(['--help']);
expect(help.code).toBe(EXIT.ok);
expect(help.out).toContain('Exit codes: 0 ok · 1 fail · 2 usage · 3 refused or needs a flag');
expect(help.out).toContain('REALITY: <row> pass|finding|n/a <summary> <file:line>');
expect(run([]).code).toBe(EXIT.usage);
expect(run(['rows']).code).toBe(EXIT.usage);
expect(run(['rows', '--phase', 'eng', '--scope', 'bogus']).code).toBe(EXIT.usage);
expect(run(['frob', '--phase', 'eng']).code).toBe(EXIT.usage);
});
test('rows prints the applicable rows; --scope adds tier 2; --json is JSON only', () => {
const plain = run(['rows', '--phase', 'eng']);
expect(plain.code).toBe(EXIT.ok);
for (const id of TIER1_ROW_IDS) expect(plain.out).toContain(`- ${id} (tier 1)`);
expect(plain.out).not.toContain('recall');
expect(run(['rows', '--phase', 'eng', '--scope', 'search']).out).toContain('- recall (tier 2, scope search)');
expect(run(['rows', '--phase', 'eng', '--deep']).out).toContain('Incident:');
const json = JSON.parse(run(['rows', '--phase', 'ceo', '--json']).out);
expect(json.rows.map((r: { id: string }) => r.id)).toEqual([...TIER1_ROW_IDS]);
});
test('check: complete exits 0; a miss exits 1 with REALITY_ROW_MISSING; unattended prints verdict=incomplete; stdin and --json', () => {
const tmp = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-plan-reality-'));
const file = path.join(tmp, 'review.md');
fs.writeFileSync(file, `prose\n${positives}\n`);
const ok = run(['check', '--phase', 'ceo', file]);
expect(ok.code).toBe(EXIT.ok);
expect(ok.out.trim()).toBe('PLAN_REALITY: phase=ceo rows=7/7 missing=none unverified=3 verdict=complete');
const partial = positives.split('\n').slice(1).join('\n');
const miss = run(['check', '--phase', 'eng', '-'], partial, { GSTACK_SESSION_KIND: 'unattended' });
expect(miss.code).toBe(EXIT.fail);
expect(miss.out).toContain('missing=premise-table verdict=incomplete');
expect(miss.err).toContain('(REALITY_ROW_MISSING)');
expect(miss.err).toContain('fix:');
const flag = run(['check', '--phase', 'eng', '-', '--session-kind', 'unattended'], partial);
expect(flag.out).toContain('verdict=incomplete');
const interactive = run(['check', '--phase', 'eng', '-'], partial);
expect(interactive.out).toContain('verdict=repair');
const unreceipted = run(['check', '--phase', 'ceo', '-', '--json'], `${partial}\n${FIXTURE.rows['premise-table']!.decoy}`);
expect(unreceipted.code).toBe(EXIT.fail);
const parsed = JSON.parse(unreceipted.out);
expect(parsed.unreceipted).toEqual(['premise-table']);
expect(unreceipted.err).toContain('(REALITY_RECEIPT_MISSING)');
const unreceiptedText = run(['check', '--phase', 'ceo', '-'], `${partial}\n${FIXTURE.rows['premise-table']!.decoy}`);
expect(unreceiptedText.out).toMatch(/^PLAN_REALITY: phase=ceo .*\bunreceipted=premise-table\b.* verdict=repair$/m);
expect(run(['check', '--phase', 'ceo', path.join(tmp, 'absent.md')]).code).toBe(EXIT.fail);
fs.rmSync(tmp, { recursive: true, force: true });
});
});
describe('captured tier 1 n/a (plan-mode-no-op, paid run 39004466422)', () => {
// The named-target plan-eng-review session marked deferred-asks n/a, check returned repair, and the repair loop ran
// past the 300 s capture before the first question.
const captured = 'REALITY: deferred-asks n/a no linked issue; the plan cites no request to compare against package.json:2';
test('check rejects the captured line as a missing row, and the skill text forbids it', () => {
const others = Object.entries(FIXTURE.rows).filter(([id]) => id !== 'deferred-asks').map(([, row]) => row.positive).join('\n');
const rejected = run(['check', '--phase', 'eng', '-'], `${others}\n${captured}\n`);
expect(rejected.code).toBe(EXIT.fail);
expect(rejected.out).toMatch(/^PLAN_REALITY: phase=eng rows=6\/7 missing=deferred-asks .*verdict=repair$/m);
expect(rejected.err).toContain('(REALITY_ROW_MISSING)');
const repaired = run(['check', '--phase', 'eng', '-'], `${others}\n${captured.replace(' n/a ', ' pass ')}\n`);
expect(repaired.out).toMatch(/^PLAN_REALITY: phase=eng rows=7\/7 missing=none .*verdict=complete$/m);
const rule = 'tier 1 always, never `n/a`: premise-table, already-done, surface-check, binding-decisions, numbers, deferred-asks, pr-count; tier 2 by detected scope, `--scope <csv>`). Every row cites a file:line receipt (with no linked issue, deferred-asks is `pass`)';
for (const phase of ['plan-eng-review', 'plan-ceo-review', 'plan-devex-review']) {
expect(generatePlanRealityRows({ skillName: phase, host: 'claude', paths: {} } as unknown as TemplateContext).replace(/\s+/g, ' ')).toContain(rule);
}
for (const rel of ['plan-eng-review/sections/review-sections.md', 'plan-ceo-review/SKILL.md']) {
expect(fs.readFileSync(path.join(ROOT, rel), 'utf8').replace(/\s+/g, ' ')).toContain(rule);
}
});
});
describe('rendered templates carry the contract', () => {
const read = (rel: string) => fs.readFileSync(path.join(ROOT, rel), 'utf8');
const ctx = (skillName: string) => ({ skillName, host: 'claude', paths: {} } as unknown as TemplateContext);
test('resolver variants name the bin, the tier 1 rows, the line shape and the unattended rule', () => {
const eng = generatePlanRealityRows(ctx('plan-eng-review'));
expect(eng).toContain('gstack-plan-reality rows --phase eng');
expect(eng).toContain('gstack-plan-reality check --phase eng');
for (const id of TIER1_ROW_IDS) expect(eng).toContain(id);
expect(eng).toContain('REALITY: <row> pass|finding|n/a <summary> <file:line>');
expect(generatePlanRealityRows(ctx('plan-ceo-review'))).toContain('--phase ceo');
expect(generatePlanRealityRows(ctx('plan-devex-review'))).toContain('--phase dx');
// Generated by /ship Step 7 coverage audit (GSTA-91, PR #3421).
// Value: protects=the operative block says its check prints the PLAN_REALITY line the summary row asks for;
// fails_when=that pointer is dropped and PLAN_REALITY is undefined in the block; why_new=only the summary variant
// was asserted to name PLAN_REALITY; seam=none; no_claim=the bin's output (pinned by the check tests above)
for (const skill of ['plan-eng-review', 'plan-ceo-review']) {
expect(generatePlanRealityRows(ctx(skill)), skill).toMatch(/gstack-plan-reality check --phase [a-z]+ [^\n]*`PLAN_REALITY`/);
}
const deep = generatePlanRealityRows(ctx('plan-eng-review'), ['deep']);
for (const r of REALITY_ROWS) expect(deep).toContain(`| \`${r.id}\` | ${r.tier} | ${r.scope} |`);
expect(deep).toBe(deep.replace(/\n{3,}/g, '\n\n'));
expect(generatePlanRealityRows(ctx('autoplan'), ['autoplan', 'eng'])).toContain('check --phase eng');
expect(generatePlanRealityRows(ctx('autoplan'), ['pregate'])).toContain('`incomplete`');
expect(generatePlanRealityRows(ctx('plan-eng-review'), ['summary'])).toContain('PLAN_REALITY');
expect(renderIncidentTable(REALITY_ROWS).split('\n').length).toBe(REALITY_ROWS.length + 2);
expect(renderRows(requiredRows([]), { deep: false }).split('\n').length).toBe(TIER1_ROW_IDS.length);
});
test('plan-eng-review: Scope Challenge block, deep incident table and Completion summary line', () => {
const sections = read('plan-eng-review/sections/review-sections.md');
expect(sections).toContain('gstack-plan-reality rows --phase eng');
expect(sections).toContain('### Reality rows (why each exists)');
expect(sections).toContain('| `pr-count` | 1 | always |');
expect(sections).toContain('- Reality rows: ___/___ applicable rows emitted (`PLAN_REALITY` line), ___ unverified premises');
expect(read('plan-eng-review/sections/checklist.md')).toContain('| Reality rows (`REALITY:` lines, `PLAN_REALITY` check) | Reality rows (why each exists) | always |');
});
test('plan-ceo-review: 0A/0B block in the skeleton, Completion Summary row and checklist row', () => {
const skeleton = read('plan-ceo-review/SKILL.md');
const at = skeleton.indexOf('gstack-plan-reality rows --phase ceo');
expect(at).toBeGreaterThan(skeleton.indexOf('### 0B. Existing Code Leverage'));
expect(at).toBeLessThan(skeleton.indexOf('### 0C. Dream State Mapping'));
expect(read('plan-ceo-review/sections/review-sections.md')).toContain('| Reality rows | ___/___ rows (PLAN_REALITY), ___ unverified |');
expect(read('plan-ceo-review/sections/review-sections.md')).toContain('### Reality rows\n');
expect(read('plan-ceo-review/sections/checklist.md')).toContain('| Reality rows (`REALITY:` lines, `PLAN_REALITY` check) | Reality rows | always |');
});
test('autoplan: CEO and Eng phase files run the check; the Pre-Gate names incomplete for unattended misses', () => {
expect(read('autoplan/sections/ceo-phase.md')).toContain('gstack-plan-reality check --phase ceo');
expect(read('autoplan/sections/eng-phase.md')).toContain('gstack-plan-reality check --phase eng');
const skeleton = read('autoplan/SKILL.md');
const pregate = skeleton.slice(skeleton.indexOf('## Pre-Gate Verification'), skeleton.indexOf('## Phase 4: Final Approval Gate'));
expect(pregate).toContain('`REALITY:` lines');
expect(pregate).toContain('verdict=complete');
expect(pregate).toContain('`incomplete`');
});
test('the two result codes have troubleshooting anchors', () => {
const doc = read('docs/troubleshooting.md');
for (const code of ['REALITY_ROW_MISSING', 'REALITY_RECEIPT_MISSING'] as const) {
expect(doc).toContain(`<a id="${RESULT_CODES[code].anchor}"></a>`);
}
});
});
describe('field wave item 12: a receipt per measured number, the rule text for a rule claim', () => {
const others = (id: string) => Object.entries(FIXTURE.rows).filter(([k]) => k !== id).map(([, r]) => r.positive).join('\n');
const FIVE = 'REALITY: numbers pass p50 120ms, p95 340ms, 13% faster, 28k tokens, cost $41 [measured: test/bench.ts:10]';
test('five measured numbers sharing one receipt leave four unreceipted, each printed with its clause', () => {
const r = checkRealityRows(`${others('numbers')}\n${FIVE}`, { phase: 'eng', sessionKind: 'unattended' });
expect(r.missing).toEqual([]);
expect(r.unreceipted).toEqual([]);
expect(r.unreceipted_numbers).toEqual(['p50 120ms', 'p95 340ms', '13% faster', '28k tokens']);
expect(r.numberLines).toEqual(['NUMBER_UNRECEIPTED: p50 120ms', 'NUMBER_UNRECEIPTED: p95 340ms', 'NUMBER_UNRECEIPTED: 13% faster', 'NUMBER_UNRECEIPTED: 28k tokens']);
expect(r.verdict).toBe('incomplete');
expect(r.line).toContain(' unreceipted_numbers=4 ');
expect(checkRealityRows(`${others('numbers')}\n${FIVE}`, { phase: 'eng' }).verdict).toBe('repair');
});
test('each clause with its own receipt (file:line, script path or a command in backticks) passes; unmeasured counts, versions and ids need none', () => {
const own = 'REALITY: numbers pass p95 340ms lib/x.ts:4, 13% faster `bun run bench --json`, 28k tokens scripts/count-tokens.ts, recall 0.82 vs 0.79 on gstack 1.91.95.0 per F2 and #3160 [estimated]';
const r = checkRealityRows(`${others('numbers')}\n${own}`, { phase: 'eng' });
expect(r.unreceipted_numbers).toEqual([]);
expect(r.verdict).toBe('complete');
});
test('a rule claim without the rule text is unquoted; the quoted rule or a file:line link to it passes', () => {
const bare = 'REALITY: binding-decisions pass the decision rule was written first 91bbd9e';
const r = checkRealityRows(`${others('binding-decisions')}\n${bare}`, { phase: 'eng', sessionKind: 'unattended' });
expect(r.unreceipted).toEqual([]);
expect(r.unquoted).toEqual([bare]);
expect(r.line).toContain(' unquoted=1 ');
expect(r.verdict).toBe('incomplete');
const quoted = 'REALITY: binding-decisions pass the decision rule was written first: "stop if recall drops below the base on any bucket" 91bbd9e';
expect(checkRealityRows(`${others('binding-decisions')}\n${quoted}`, { phase: 'eng' }).verdict).toBe('complete');
const linked = 'REALITY: binding-decisions pass the decision rule was written first, see docs/plan.md:4';
expect(checkRealityRows(`${others('binding-decisions')}\n${linked}`, { phase: 'eng' }).unquoted).toEqual([]);
});
test('receipts on dot paths count; a sha receipt needs a hex letter or a commit/backtick context', () => {
expect(parseRealityLines('REALITY: pr-count pass one PR .gstack/ship-policy.json:1')[0]!.receipt).toBe('.gstack/ship-policy.json:1');
expect(parseRealityLines('REALITY: pr-count pass ci lane .github/workflows/ci.yml:12-20')[0]!.receipt).toBe('.github/workflows/ci.yml:12-20');
expect(parseRealityLines('REALITY: numbers pass run took 20261012 cycles')[0]!.receipt).toBeNull();
expect(parseRealityLines('REALITY: already-done pass base clean at commit 1234567')[0]!.receipt).toBe('commit 1234567');
expect(parseRealityLines('REALITY: already-done pass base clean at `7654321` today')[0]!.receipt).toBe('`7654321`');
});
test('bin: NUMBER_UNRECEIPTED lines follow the PLAN_REALITY line and the verdict exits 1 with a fix', () => {
const r = run(['check', '--phase', 'eng', '-', '--session-kind', 'unattended'], `${others('numbers')}\n${FIVE}`);
expect(r.code).toBe(EXIT.fail);
expect(r.out.split('\n').filter(Boolean)).toEqual([
expect.stringMatching(/^PLAN_REALITY: phase=eng rows=7\/7 missing=none unreceipted_numbers=4 .*verdict=incomplete$/),
'NUMBER_UNRECEIPTED: p50 120ms', 'NUMBER_UNRECEIPTED: p95 340ms', 'NUMBER_UNRECEIPTED: 13% faster', 'NUMBER_UNRECEIPTED: 28k tokens',
'RECEIPT_UNRESOLVED: numbers test/bench.ts:10 no such file',
]);
expect(r.err).toContain('(REALITY_RECEIPT_MISSING)');
expect(r.err).toContain('fix:');
});
test('checkPlanReality(planText): plan prose gets the same number and rule checks, pure and session-kind mapped', () => {
const plan = 'Cold install measured 32.5 s; the warm rerun takes 13 s (bin/gstack-capy-install:91).\n```\nlatency 900 ms in a fenced example\n```\nThe decision rule was written first.\nThe preregistered rule is below.\n> stop when recall drops below the base\n';
const r = checkPlanReality(plan, { sessionKind: 'unattended' });
expect(r.unreceipted_numbers).toEqual(['Cold install measured 32.5 s']);
expect(r.unquoted).toEqual(['The decision rule was written first.']);
expect(r.verdict).toBe('incomplete');
expect(r.line).toBe('PLAN_REALITY: scope=plan unreceipted_numbers=1 unquoted=1 verdict=incomplete');
expect(r.numberLines).toEqual(['NUMBER_UNRECEIPTED: Cold install measured 32.5 s']);
expect(checkPlanReality('Add a retry with a 30 s cap (lib/worker.ts:12).\n').line).toBe('PLAN_REALITY: scope=plan unreceipted_numbers=0 unquoted=0 verdict=complete');
expect(checkPlanReality(plan).verdict).toBe('repair');
});
test('false-positive bound: the field-wave plan itself raises only a small, pinned count', () => {
const text = fs.readFileSync(path.join(ROOT, 'docs', 'designs', 'GSTACK_FIELD_WAVE_2026_10_12.md'), 'utf8');
const r = checkPlanReality(extractHeading(text, 'Implementation plan')!);
expect(r.unquoted).toEqual([]);
expect(r.unreceipted_numbers.length).toBeGreaterThan(0);
expect(r.unreceipted_numbers.length).toBeLessThanOrEqual(6);
});
});
describe('next wave B6/B10: PLAN_REALITY flags without false positives; plan quotes need no receipt', () => {
const others = (id: string) => Object.entries(FIXTURE.rows).filter(([k]) => k !== id).map(([, r]) => r.positive).join('\n');
test('B6: item and section references are not quantities; a phrase in backticks is named, not claimed (plan scope)', () => {
const plan = "B1. `rec`: one word takes every recommendation (item 5).\nB6. When the plan names `minimum effect`, `margin`, `decision rule` or a dollar cost, the gate flags it (items 10, 14–19; section 3).\n";
const r = checkPlanReality(plan);
expect(r.unreceipted_numbers).toEqual([]);
expect(r.unquoted).toEqual([]);
expect(checkPlanReality('Startup takes roughly 9 s on a cold machine.\n').unreceipted_numbers).toEqual(['Startup takes roughly 9 s on a cold machine.']);
expect(checkPlanReality('The decision rule was written first.\n').unquoted).toEqual(['The decision rule was written first.']);
expect(PLAN_REALITY_CHECKER).toBe('plan-reality/3');
});
test('B-f negative control: a measured number inside backticks in review scope stays flagged; a command in backticks still receipts', () => {
const quiet = 'REALITY: numbers pass the cold install took `86 s` on Capy';
expect(checkRealityRows(`${others('numbers')}\n${quiet}`, { phase: 'eng' }).unreceipted_numbers).toEqual(['the cold install took `86 s` on Capy']);
const cmd = 'REALITY: numbers pass the cold install took 86 s `bun run bench --json` on Capy';
expect(checkRealityRows(`${others('numbers')}\n${cmd}`, { phase: 'eng' }).unreceipted_numbers).toEqual([]);
});
test('B10: in reviewer scope a number quoted verbatim from the reviewed plan needs no receipt; a reworded or unknown quote does', () => {
const plan = 'Add exponential backoff: base 500 ms, factor 2, cap 30 s, jitter 10%.\n';
const quoted = 'REALITY: numbers pass the plan says "base 500 ms, factor 2, cap 30 s" and gives no reason for the cap (plan.md:1)';
expect(checkRealityRows(`${others('numbers')}\n${quoted}`, { phase: 'eng', planText: plan }).unreceipted_numbers).toEqual([]);
expect(checkRealityRows(`${others('numbers')}\n${quoted}`, { phase: 'eng' }).unreceipted_numbers.length).toBeGreaterThan(0);
const reworded = 'REALITY: numbers pass the plan says "a 45 s cap" and gives no reason';
expect(checkRealityRows(`${others('numbers')}\n${reworded}`, { phase: 'eng', planText: plan }).unreceipted_numbers.length).toBe(1);
const own = 'REALITY: numbers pass the plan says "cap 30 s"; retries took 900 ms in my run';
expect(checkRealityRows(`${others('numbers')}\n${own}`, { phase: 'eng', planText: plan }).unreceipted_numbers).toEqual(['retries took 900 ms in my run']);
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'plan-reality-quote-'));
fs.writeFileSync(path.join(dir, 'plan.md'), plan);
const bin = run(['check', '--phase', 'eng', '--plan', path.join(dir, 'plan.md'), '-'], `${others('numbers')}\n${quoted}`);
expect(bin.code, bin.err).toBe(EXIT.ok);
expect(run(['check', '--phase', 'eng', '-'], `${others('numbers')}\n${quoted}`).code).toBe(EXIT.fail);
fs.rmSync(dir, { recursive: true, force: true });
});
test('B6: vaguePlanTerms flags minimum effect, margin, decision rule and dollar cost only without a number, arm list or rule body', () => {
const plan = [
'Ship when the minimum effect is reached.',
'Keep a safety margin on the spend.',
'A margin of 15% over the pilot price.',
'The decision rule:',
'- arm A: keep the default',
'- arm B: switch',
'We price the dollar cost later (item 9).',
'The decision rule is "stop when recall drops below the base".',
'Name `minimum effect` in the template.',
'```', 'margin inside a fence', '```',
].join('\n');
expect(vaguePlanTerms(plan).map(v => [v.phrase, v.line])).toEqual([['minimum effect', 1], ['margin', 2], ['dollar cost', 7]]);
});
});