eval(plan-eng-review-artifact): periodic checkpoint resumes at Test review; the fresh full run moves to marathon

A fresh interactive /plan-eng-review cannot reach Test review inside the
600 s periodic budget: startup plus one-at-a-time Scope Challenge questions
took 595 s with six questions and no Test review in census 37174266054 and
in a local run (one earlier fresh run passed at 498 s). Per the eval-speed
policy, the long run moves to the non-blocking marathon lane and the
blocking case becomes a focused checkpoint.

- plan-eng-review-artifact (periodic, 600 s unchanged): resumes the real
  interactive skill from a saved review record: the plan, the Scope
  Challenge result, Sections 1-2 complete, and a Decision ledger rebuilt
  from that census run's six actual native questions and answers
  (test/fixtures/plan-eng-artifact-resume-37174266054.json). The live skill
  runs Section 3, Test review. Assertion unchanged: exactly one new
  eng-review test plan in the run's own state root, about the dashboard
  change. Paid trial (local, Claude Code 2.1.284): PASS, 364 s.
- plan-eng-review-artifact-full (marathon, PTY_LONG_MS): the fresh
  start-to-finish run with the same assertion; budget measured beside it.
- test/plan-eng-resume.test.ts proves the rebuilt record is a valid resume
  point and rejects missing, unanswered, edited or duplicated records.
- Registry: touchfiles, tiers, kinds, test-name labels, durations, paid
  file count 115, PTY consumer list; CHANGELOG and TODOS.
This commit is contained in:
garrytan committed 2026-10-04 04:22:44 +00:00
1 parent 307ba98623
commit 7249f01464
14 files changed
+405 -43

No files matched your search

+1 -1
View File
@@ -98,7 +98,7 @@ The first two rows are the ones to care about: a review that read nothing, or fo
- Eval children no longer receive credential env vars that carry a suffix (`GITHUB_TOKEN_1`, `GITHUB_APP_PRIVATE_KEY_BASE64`); admit one explicitly with `extraAllow` (#2949). Chunked large-push coverage for the pre-push guard (#2915).
- The test typecheck ratchet ignores the order tsc prints an object type's members in.
- The weekly eval report counts only cases that ran: tests deselected for another tier or case shard are listed by reason with no credit, every result is attributed to its case (0 unattributed, was 72), and no PR-lane eval shard runs over 10 minutes (the deploy eval is split per case).
- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. The `/plan-eng-review` QA test-plan eval now drives the real interactive review instead of a shortcut prompt that skipped the workflow.
- Codex evals run on their own CI job where Codex's sandbox can start, and pass only when Codex completed a command; a review whose sandbox failed now fails instead of passing. The `/codex review` eval passes only when Codex actually reviewed the diff. The AUQ matrix checks the `(recommended)` marker on option labels, where the auto-decide hook reads it. The `/plan-eng-review` QA test-plan eval now runs the real interactive review instead of a shortcut prompt that skipped the workflow: a periodic checkpoint resumes it at Test review from a recorded Scope Challenge, and the full fresh run is a non-blocking marathon case.
Contributed by @CermakM (#2843), @Haijie-Lee (#2192), @SomSamantray (#2965), @Vaughan-g-aus (#2329), @BharadwajMittapelli (#2729), @maxpetrusenkoagent (#1931, #1932, #1981), @Infiniteyieldai (#2884), @afshaker (#2778), @time-attack (#2232, #2229), @jzeisweiss (#2670), @liutiming (#2783), @agile-operators (#1437), @benjaminberes-bp (#2369), @lewispeel (#2885), @Yugz29 (#2899), @fcmerle (#2962), @cdecook23 (#2784), @mb1810 (#2913), @ntdatt812 (#2949), @pneumorea-ai (#2915), @gabrielrondon (#2811), @kevingrasso32 (#2876), @xwang4-svg (#2354), @DizzyWesterwald (#2936), @oli548 (#2771), @shofel (#1968), @ryanlaiwy (#1633), @yinanli1917-cloud (#2281), @Spooks444 (#2260), @punksterlabs (#2439), @himiaocc (#2492), @jbetala7 (#1419, #1777), @mamedov (#2891), @Lockyer228 (#2651), @topcoder1 (#2263), @jonahberg (#2707), @AlinValentin7 (#3015), @philhie (#2208), @Alchemist-DevAI (#2334), @merit-blake (#2343), @day-of-davon (#2952), @loulanyue (#2797, #2792, #2794, #2796), @Bit0ps (#2840), @szsunyuan (#2799), @frosimanuel (#2818), @JiayuuWang (#1096), @RyanAlberts (#1728), @exGeni (#2704), @Jino00 (#2812), @snig-17 (#2809) and @SholtoMc (#2037).
+5 -1
View File
@@ -82,7 +82,11 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
checker and office-hours-brain-writeback's prompt are fixed (3/3).
plan-eng-review-artifact (0/4 since #3017) measured a shortcut prompt that
skipped the workflow; it now drives the real interactive /plan-eng-review in
a PTY and wrote its QA test plan at 498 s in one CI-image paid run.
a PTY. A fresh run needs more than its 600 s budget (498 s pass; 595 s
timeouts in census 37174266054 and a local run), so the periodic case
resumes at Test review from that census run's real Scope Challenge ledger
(pass, 364 s) and the fresh run is the marathon case
plan-eng-review-artifact-full.
- **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per
reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed
first. **Effort:** S. **Priority:** P3.
+1
View File
@@ -87,6 +87,7 @@ export const CASE_TEST_NAMES: Record<string, string> = {
'cso-infra-scope': '/cso --infra finds an attacker-to-credential execution path',
'plan-ceo-review-plan-mode': 'first terminal outcome is asked (Step 0 fires before any plan write)',
'plan-eng-review-artifact': 'an interactive review writes one QA test plan about the reviewed change',
'plan-eng-review-artifact-full': 'a fresh interactive review reaches Test review and writes one QA test plan',
};
export const CASE_KEY_SEPARATOR = '#';
+1
View File
@@ -172,6 +172,7 @@
"test/skill-e2e-plan-design-finding-floor.test.ts": 155538,
"test/skill-e2e-plan-design-plan-mode.test.ts": 106608,
"test/skill-e2e-plan-devex-peer-comparison-classification.test.ts": 79786,
"test/skill-e2e-plan-eng-artifact-full.test.ts": 595009,
"test/skill-e2e-plan-eng-artifact.test.ts": 498474,
"test/skill-e2e-plan-eng-finding-floor.test.ts": 388666,
"test/skill-e2e-plan-eng-multi-finding-batching.test.ts": 597500,
+1 -1
View File
@@ -139,7 +139,7 @@ test('live periodic census fits the declared CI wall including setup', () => {
test('registered allocation is deterministic and preserves every discovered file', () => {
const files = collectPaidTestFiles();
expect(files).toHaveLength(114);
expect(files).toHaveLength(115);
expect(files).toContain('test/skill-e2e-ship-skip.test.ts');
const m = livePlan(files);
expect(livePlan([...files].reverse())).toEqual(m);
+126
View File
@@ -0,0 +1,126 @@
{
"source": "periodic census 37174266054 (eval-slices 2), plan-eng-review-artifact PTY run on a8ace39: the six native AskUserQuestion calls the real /plan-eng-review sent during Scope Challenge, with the runner's actual answers. Public question fields only.",
"calls": [
{
"toolUseId": "toolu_015bMxAKzS2jn8i47R3fc2Pj",
"answeredAt": "2026-10-04T03:35:45.294Z",
"question": "D1 — Fix the synchronous read of an async fetch in Dashboard\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\" reading /api/stats.\nELI10: fetchStats() hands back a promise, an IOU for data that arrives later, but Dashboard() treats the IOU as if it were the data. So users and revenue are always undefined. The one thing the plan exists to do, show user count and revenue, never happens.\nStakes if we pick wrong: the dashboard ships with blank metrics and no error; nobody notices until a human stares at the page.\nRecommendation: A because async/await is the explicit, standard fix and keeps error handling (R2) readable when it lands.\nCompleteness: A=10/10, B=9/10\nPros / cons:\nA) Make Dashboard async and await fetchStats (recommended) (human: ~30 min / CC: ~2 min)\n ✅ Dashboard resolves to real {users, revenue} values, so the plan's contract actually holds\n ✅ Explicit control flow; main() in app.ts awaits it and returns Promise<Stats>; low maintenance\n ❌ main() becomes async, so any future caller must await it (no callers exist in the repo today)\nB) Return the promise chain from Dashboard without async/await (human: ~20 min / CC: ~2 min)\n ✅ Smallest diff: one .then() mapping to {users, revenue}; same resolved value as A\n ✅ No new keywords; main() can keep its one-line body and return the promise\n ❌ Then-chains get harder to read once the r.ok check, catch and timeout land in the same function\nNet: a couple of keywords versus a Dashboard that actually returns data; either fixes the P0, A ages better.",
"header": "Async fix",
"options": [
{
"label": "Make Dashboard async and await fetchStats (recommended)",
"description": "Dashboard becomes `async` and awaits fetchStats(); main() in app.ts becomes async and awaits Dashboard(). Resolves to real {users, revenue}. Required proof: unit test with mocked fetch asserting the resolved values. R2/R3/R4 stay pending. greet() unchanged. (human: ~30 min / CC: ~2 min)"
},
{
"label": "Return the promise chain from Dashboard without async/await",
"description": "Dashboard returns fetchStats().then(d => ({ users: d.users, revenue: d.revenue })); main() returns that promise. Same resolved value as A. Required proof: same unit test. R2/R3/R4 stay pending. greet() unchanged. (human: ~20 min / CC: ~2 min)"
}
],
"answer": "Make Dashboard async and await fetchStats (recommended)",
"ledgerId": "R1"
},
{
"toolUseId": "toolu_01L6up5bAgKvNxnb8VniPhB2",
"answeredAt": "2026-10-04T03:36:49.867Z",
"question": "D2 — What should Dashboard do when /api/stats fails?\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\"; R1 (async fix) approved.\nELI10: Today the code assumes the server always answers with valid JSON. If the network is down, the server returns a 500, or it sends an HTML error page, the code blows up with an unhandled error and the user sees nothing. We need to decide how a failure is reported to whoever renders the dashboard.\nStakes if we pick wrong: a server hiccup becomes a blank screen or a console-only exception, and the next person who calls Dashboard() forgets to catch it and we're back to a silent failure.\nRecommendation: A because a result object makes the failure path a value the caller cannot ignore, and the plan's own job is to report metrics or say why it can't.\nCompleteness: A=10/10, B=9/10, C=0/10\nPros / cons:\nA) Resolve to an ok/error result object (recommended) (human: ~1.5 h / CC: ~5 min)\n ✅ Callers get { ok, stats | error } and the type system forces them to handle the failure state\n ✅ Dashboard() never rejects, so main() cannot reintroduce an unhandled promise rejection\n ❌ Slightly more verbose call sites than a bare value; every caller unwraps the result\nB) Throw a typed StatsError from Dashboard (human: ~1.5 h / CC: ~5 min)\n ✅ Idiomatic async/await: happy path stays a plain Stats value; errors carry kind and HTTP status\n ✅ Works with any upstream error boundary that already catches rejections\n ❌ A caller that forgets try/catch (main() has none today) recreates the same silent failure\nC) Leave failures unhandled (human: 0 / CC: 0)\n ✅ No code change\n ❌ 500s and offline users produce unhandled rejections or SyntaxError; the plan's \"returns metrics\" promise is unreliable\nNet: forcing callers to look at the error (A) versus idiomatic throws they can forget (B); either beats C.",
"header": "Error handling",
"options": [
{
"label": "Resolve to an ok/error result object (recommended)",
"description": "Dashboard() resolves to `{ ok: true, stats }` or `{ ok: false, error: { kind: 'network' | 'http' | 'parse', status?, message } }`; checks `r.ok`, catches network and JSON-parse failures; never rejects. Required proof: unit tests for happy path and each of the three failure classes with mocked fetch. R1 stays approved; R3/R4/R7 stay pending. (human: ~1.5 h / CC: ~5 min)"
},
{
"label": "Throw a typed StatsError from Dashboard",
"description": "Dashboard() returns Stats on success and rejects with `StatsError { kind, status?, cause }` for network, non-2xx and parse failures; main() wraps the call in try/catch. Required proof: same four tests. R1 stays approved; R3/R4/R7 stay pending. (human: ~1.5 h / CC: ~5 min)"
},
{
"label": "Leave failures unhandled",
"description": "No change: fetch rejections and SyntaxError propagate out of Dashboard() and main(). No tests added for failure paths. R1 stays approved; R3/R4/R7 stay pending."
}
],
"answer": "Resolve to an ok/error result object (recommended)",
"ledgerId": "R2"
},
{
"toolUseId": "toolu_01WRokhgJVFLKtJcfYwA48zB",
"answeredAt": "2026-10-04T03:37:56.510Z",
"question": "D3 — Pin down the /api/stats contract before building the client\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\"; R1 and R2 approved.\nELI10: The plan says \"fetch from /api/stats\" but nothing in this repo serves that URL, and nobody has written down what it returns, what happens when you're not logged in, or whether revenue is in dollars or cents. Building a client against an unwritten contract means the first real request is the first test.\nStakes if we pick wrong: the client parses a shape the server never sends (blank metrics again), or revenue numbers are off by 100x, or anonymous users can read revenue.\nRecommendation: A because writing the contract into the plan now gives R4 a type to validate against and the backend owner a concrete thing to confirm; it costs minutes.\nCompleteness: A=9/10, B=5/10, C=3/10\nPros / cons:\nA) Write the /api/stats contract into the plan (recommended) (human: ~1 h / CC: ~5 min)\n ✅ Shape, status codes, auth and revenue unit become explicit text the backend owner can confirm or correct\n ✅ Gives R4 (runtime validation) and the tests a single source of truth for the Stats type\n ❌ Still an assumption until the backend owner confirms; a mismatch means one more plan edit\nB) Investigate the backend first (human: ~2 h / CC: ~10 min)\n ✅ Contract copied from the real handler, so no guessing about shape or auth\n ✅ Surfaces early if the endpoint does not exist yet and needs its own ticket\n ❌ Blocks the client work until someone finds the producer; approves no implementation\nC) Build against the assumed shape (human: 0 / CC: 0)\n ✅ No extra writing; matches the plan as drafted\n ❌ Shape, auth and units stay undocumented; the first production request is the first integration test\nNet: a written, confirmable contract (A) versus waiting on discovery (B) versus hoping (C).",
"header": "API contract",
"options": [
{
"label": "Write the /api/stats contract into the plan (recommended)",
"description": "Plan records: `GET /api/stats` requires an authenticated session, client sends `credentials: 'same-origin'`; 200 → `{ users: number, revenue: number }` (revenue unit and currency marked \"confirm with backend owner\"); 401/403/5xx → R2 `http` error with status. This Stats shape is the type R4 validates against. R1/R2 stay approved; R4/R7 stay pending. (human: ~1 h / CC: ~5 min)"
},
{
"label": "Investigate the backend first",
"description": "Bounded investigation: locate the service/repo that serves `/api/stats` (or confirm it does not exist) and copy its real response shape, status codes and auth into the plan. Approves no implementation; R3 rows stay pending until the result is in. R1/R2 stay approved; R4/R7 stay pending. (human: ~2 h / CC: ~10 min)"
},
{
"label": "Build against the assumed shape",
"description": "Keep the plan as written: client assumes JSON with `users` and `revenue`, no documented status codes, auth or units. R1/R2 stay approved; R4/R7 stay pending."
}
],
"answer": "Write the /api/stats contract into the plan (recommended)",
"ledgerId": "R3"
},
{
"toolUseId": "toolu_01QuAeEMbCnnJHHbe4vuvwHE",
"answeredAt": "2026-10-04T03:39:03.639Z",
"question": "D4 — Should the stats request give up after a fixed time?\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\"; R1, R2, R3 approved.\nELI10: If the stats server hangs instead of answering, the current fetch waits as long as the browser lets it, often minutes. The user stares at a loading dashboard with no idea whether it will ever finish. A timeout turns \"forever\" into a clear \"couldn't load stats\" after a known number of seconds.\nStakes if we pick wrong: a slow backend incident becomes an indefinitely spinning dashboard, and support hears \"the dashboard is frozen\" instead of \"the dashboard said stats are unavailable\".\nRecommendation: A because AbortSignal.timeout is a one-line built-in and turns the worst failure mode into a handled error with a test.\nCompleteness: A=10/10, B=5/10\nPros / cons:\nA) 10 second timeout via AbortSignal.timeout (recommended) (human: ~45 min / CC: ~3 min)\n ✅ Hung backend becomes a { ok: false, error: { kind: 'timeout' } } result after 10 s; the UI can say so\n ✅ Platform built-in, no dependency; one extra error kind in the union approved in D2\n ❌ 10 s is a guess; a legitimately slow aggregation query would be reported as a failure\nB) No client-side timeout (human: 0 / CC: 0)\n ✅ No code change; never cuts off a slow but successful response\n ✅ Nothing extra to test or tune\n ❌ Hung backend means the dashboard spins until the browser gives up, often minutes, with no message\nNet: a bounded, explainable failure after 10 s versus trusting the backend to always answer.",
"header": "Request timeout",
"options": [
{
"label": "10 second timeout via AbortSignal.timeout (recommended)",
"description": "fetchStats passes `signal: AbortSignal.timeout(10_000)`; an abort (`err.name === 'TimeoutError'`) maps to `{ ok: false, error: { kind: 'timeout', message } }`, extending the D2 error union. Dashboard() still never rejects. Required proof: unit test with mocked fetch rejecting with a TimeoutError. R1/R2/R3 stay approved; R4 stays pending. (human: ~45 min / CC: ~3 min)"
},
{
"label": "No client-side timeout",
"description": "fetch keeps browser defaults; no `timeout` error kind; a hung backend leaves Dashboard() pending. R1/R2/R3 stay approved; R4 stays pending."
}
],
"answer": "10 second timeout via AbortSignal.timeout (recommended)",
"ledgerId": "R7"
},
{
"toolUseId": "toolu_01E4hjs1j86wmxGerQoXGkvf",
"answeredAt": "2026-10-04T03:40:33.915Z",
"question": "D5 — Should the client verify the stats response shape at runtime?\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\"; R1, R2, R3, R7 approved.\nELI10: TypeScript types only exist while you compile. At runtime the server can send anything, and today the code forwards whatever arrives without looking. If the backend renames a field or sends a string, the dashboard shows blank or wrong numbers with no error. A three-line check turns that into the same clear \"parse\" failure the D2 result object already carries.\nStakes if we pick wrong: a backend rename or a staging/prod contract drift shows up as blank or garbage revenue, not as an error anyone can act on.\nRecommendation: A because the guard is three lines, has no dependency, reuses the approved parse error kind and makes the D3 contract enforceable instead of aspirational.\nCompleteness: A=10/10, B=6/10\nPros / cons:\nA) Stats type plus a hand-written runtime guard (recommended) (human: ~1 h / CC: ~4 min)\n ✅ Contract drift (renamed field, string instead of number) becomes an explicit parse error, not undefined metrics\n ✅ No new dependency; the guard is the executable form of the D3 contract and is unit-tested\n ❌ Two more branches to keep in sync if the contract grows beyond two fields\nB) Stats type only, trust the server (human: ~15 min / CC: ~1 min)\n ✅ Compile-time safety for callers reading stats.users and stats.revenue\n ✅ Smallest change; nothing to update when the backend adds optional fields\n ❌ Runtime drift still flows through silently; the type is a promise the code never checks\nNet: enforce the contract at the boundary (A) or document it and hope (B); the price of A is three lines and two tests.",
"header": "Shape validation",
"options": [
{
"label": "Stats type plus a hand-written runtime guard (recommended)",
"description": "Add `export type Stats = { users: number; revenue: number }` and `isStats(x: unknown): x is Stats` (both fields finite numbers) in dashboard.ts; a body failing the guard resolves to `{ ok: false, error: { kind: 'parse', message } }`. No dependency. Required proof: unit tests for a valid body, a missing field and a wrong type. R1/R2/R3/R7 stay approved. (human: ~1 h / CC: ~4 min)"
},
{
"label": "Stats type only, trust the server",
"description": "Add the `Stats` type and cast `r.json()` to it; no runtime check, so shape drift passes through. R1/R2/R3/R7 stay approved. (human: ~15 min / CC: ~1 min)"
}
],
"answer": "Stats type plus a hand-written runtime guard (recommended)",
"ledgerId": "R4"
},
{
"toolUseId": "toolu_013ZVLcvRXw1iHDDywWdoQZ9",
"answeredAt": "2026-10-04T03:42:00.587Z",
"question": "D6 — Which test runner carries the approved tests?\nProject/branch/task: gstack-plan-count-Q1Nd6h on main, plan \"Add Dashboard\"; R1/R2/R3/R4/R7 approved, each with required unit tests.\nELI10: Five approved behaviors each come with tests, but the repo has no way to run a test: no package.json, no TypeScript config, no runner. We need to pick the tool that executes them. Node 22 can run TypeScript tests natively, or we add Vitest, the common choice for TypeScript projects that ship to a browser.\nStakes if we pick wrong: either the tests never get written because there is nowhere to put them, or we pin the project to a toolchain that fights the browser build later.\nRecommendation: A because the code targets a browser (relative fetch URL), Vitest handles the existing extensionless import untouched, and it is the boring default for TypeScript projects; node:test is a close second.\nCompleteness: A=10/10, B=10/10, C=0/10\nPros / cons:\nA) Vitest + TypeScript (recommended) (human: ~2 h / CC: ~8 min)\n ✅ Runs the existing app.ts import as-is; vi.stubGlobal makes fetch mocking one line per test\n ✅ Same toolchain family a browser build (Vite) will use, so one config lineage instead of two\n ❌ Adds vitest and its transitive deps to a two-file repo; install size is the price\nB) node:test with native type stripping (human: ~2 h / CC: ~8 min)\n ✅ Zero test dependencies; node --test and mock.method are in the standard library\n ✅ Verified viable here: Node v22.20.0 strips types unflagged\n ❌ app.ts:1 must import \"./dashboard.ts\" and the project pins Node >= 22.18; tsconfig needs the extension-rewrite flags\nC) No tests (human: 0 / CC: 0)\n ✅ No toolchain to maintain\n ❌ Every approved behavior ships unproven; the P0 async bug class has no regression guard\nNet: boring browser-aligned toolchain (A) versus zero dependencies with a source tweak and a Node pin (B); C drops all required proof.",
"header": "Test runner",
"options": [
{
"label": "Vitest + TypeScript (recommended)",
"description": "Add `package.json` (`\"test\": \"vitest run\"`, `\"typecheck\": \"tsc --noEmit\"`), `tsconfig.json` (`strict`, `lib: [\"dom\",\"es2022\"]`), devDependencies `vitest` and `typescript`. Tests live in `dashboard.test.ts` and `app.test.ts`; fetch mocked with `vi.stubGlobal`. No source import changes. R1/R2/R3/R4/R7 stay approved; R6 stays pending. (human: ~2 h / CC: ~8 min)"
},
{
"label": "node:test with native type stripping",
"description": "Add `package.json` (`\"test\": \"node --test '**/*.test.ts'\"`, `engines.node >= 22.18`), `tsconfig.json` with `allowImportingTsExtensions` + `rewriteRelativeImportExtensions`, devDependency `typescript` for typecheck only. Change app.ts:1 to import `\"./dashboard.ts\"`. fetch mocked with `mock.method`. R1/R2/R3/R4/R7 stay approved; R6 stays pending. (human: ~2 h / CC: ~8 min)"
},
{
"label": "No tests",
"description": "No toolchain or tests added; approved required proof cannot run. R1/R2/R3/R4/R7 stay approved as behaviors; R6 stays pending."
}
],
"answer": "Vitest + TypeScript (recommended)",
"ledgerId": "R6"
}
],
"ledgerIds": "From the questions' own cross-references: D2 cites R1 approved, D3 R1-R2, D4 R1-R3, D5 R1-R3 and R7 (so D4 was R7), D6 R1-R4 and R7 (so D5 was R4) and names R6 as its own choice."
}
+31
View File
@@ -0,0 +1,31 @@
/** The dashboard plan and sources that /plan-eng-review's QA test-plan cases review. */
/** Working-plan path in the prompt; rewritten per run to a private temporary path. */
export const FIXTURE_PLAN_PATH = '/tmp/gstack-test-plan-eng-artifact.md';
export const DASHBOARD_PLAN = [
'Proceed directly to the requested engineering review; skip the optional /office-hours prerequisite.',
`Please review this plan thoroughly. As you go, write your plan-mode plan to ${FIXTURE_PLAN_PATH} (use Edit/Write to that exact path).`,
'',
'# Plan: Add Dashboard',
'',
'## Changes',
'1. New `dashboard.ts` with Dashboard component and fetchStats API call',
'2. Updated `app.ts` to import and use Dashboard',
'',
'## Architecture',
'- Dashboard fetches from `/api/stats` endpoint',
'- Returns user count and revenue metrics',
].join('\n');
export const SOURCES = {
'app.ts': 'import { Dashboard } from "./dashboard";\nexport function greet() { return "hello"; }\nexport function main() { return Dashboard(); }\n',
'dashboard.ts': `export function Dashboard() {
const data = fetchStats();
return { users: data.users, revenue: data.revenue };
}
function fetchStats() {
return fetch('/api/stats').then(r => r.json());
}
`,
};
+98
View File
@@ -0,0 +1,98 @@
/**
* A /plan-eng-review resume point: the working plan plus the review record the
* real skill saves (Scope Challenge result, section dispositions and the
* Decision ledger in review-sections.md's record format), built from actual
* native AskUserQuestion calls and their actual answers. A checkpoint case
* hands it to the live skill, which continues at Section 3 (Test review).
*/
export type EngResumeCall = {
toolUseId: string;
answeredAt?: string;
ledgerId: string;
question: string;
header: string;
options: Array<{ label: string; description: string }>;
answer: string;
};
const SELECTORS = ['A', 'B', 'C', 'D'];
const title = (question: string) => question.split('\n', 1)[0]!.replace(/^D\d+\s*—\s*/, '');
export function engResumeReport(plan: string, calls: EngResumeCall[]): string {
const ids = calls.map(call => call.ledgerId).join(', ');
const records = calls.map((call, index) => {
const selected = call.options.findIndex(option => option.label === call.answer);
return [
`### ${call.ledgerId}: ${call.header} — ${title(call.question)}`,
`Finding: S${index + 1}, Scope Challenge, reviewer: Claude (plan-eng-review)`,
'Plan baseline: original proposal (Plan: Add Dashboard)',
`Runtime evidence: as stated in D${index + 1}`,
'State: approved',
'',
'Comparison grid:',
'',
`| Choice | Current | ${call.options.map((_, i) => SELECTORS[i]).join(' | ')} |`,
`|---|---|${call.options.map(() => '---|').join('')}`,
`| ${call.ledgerId} ${call.header.toLowerCase()} | pending | ${call.options.map(option => option.label).join(' | ')} |`,
'',
`Question D${index + 1}:`,
call.question,
`Header: ${call.header}`,
'Options:',
...call.options.flatMap((option, i) => [`${SELECTORS[i]}) ${option.label}`, option.description]),
'',
`Actual answer: ${SELECTORS[selected]}) ${call.answer} (D${index + 1}${call.answeredAt ? `, answered ${call.answeredAt}` : ''})`,
`Accepted scope: ${call.options[selected]!.description}`,
'History: —',
].join('\n');
});
return [
plan.trimEnd(),
'',
'## Scope Challenge record',
'',
'Scope Challenge result: scope accepted as-is (complexity gate skipped: 2 changed files, no new classes/services).',
`Findings S1–S${calls.length} resolved through D1–D${calls.length}; dispositions: ${ids} approved.`,
'',
'## Review progress',
'',
`- Scope Challenge: complete (D1–D${calls.length} answered).`,
`- Section 1, Architecture review: complete. No new or reopened choices; dispositions: ${ids} carried forward.`,
'- Section 2, Code quality review: complete. No new or reopened choices.',
'- Next: Section 3, Test review.',
'',
'## Decision ledger',
'',
records.join('\n\n'),
'',
].join('\n');
}
/** Why a report is not a valid resume point for these calls; empty when it is. */
export function engResumeProblems(report: string, calls: EngResumeCall[]): string[] {
const problems: string[] = [];
if (report.split('\n## Decision ledger\n').length !== 2) problems.push('expected exactly one Decision ledger');
if (!/^Scope Challenge result: scope accepted as-is\b/m.test(report)) problems.push('missing Scope Challenge result');
for (const section of ['Section 1, Architecture review: complete', 'Section 2, Code quality review: complete', 'Next: Section 3, Test review'])
if (!report.includes(section)) problems.push(`missing progress: ${section}`);
const ledger = report.split('\n## Decision ledger\n')[1] ?? '';
const records = ledger.split(/^(?=### R\d+: )/m).filter(record => record.startsWith('### R'));
if (records.length !== calls.length) problems.push(`expected ${calls.length} ledger records, found ${records.length}`);
if (new Set(calls.map(call => call.ledgerId)).size !== calls.length) problems.push('ledger ids repeat');
calls.forEach((call, index) => {
const record = records.find(text => text.startsWith(`### ${call.ledgerId}: `));
const d = `D${index + 1}`;
if (!record) { problems.push(`${call.ledgerId}: missing record`); return; }
if (!call.question.startsWith(`${d} —`)) problems.push(`${call.ledgerId}: question is not ${d}`);
if (!record.includes(`Question ${d}:\n${call.question}\nHeader: ${call.header}\nOptions:\n`)) problems.push(`${call.ledgerId}: question or header differs from the native call`);
call.options.forEach((option, i) => {
if (!record.includes(`\n${SELECTORS[i]}) ${option.label}\n${option.description}\n`)) problems.push(`${call.ledgerId}: option ${SELECTORS[i]} differs from the native call`);
});
const selected = call.options.findIndex(option => option.label === call.answer);
if (selected < 0) problems.push(`${call.ledgerId}: actual answer is not an offered label`);
else if (!record.includes(`\nActual answer: ${SELECTORS[selected]}) ${call.answer} (${d}`)) problems.push(`${call.ledgerId}: actual answer not recorded`);
if (!/^State: approved$/m.test(record)) problems.push(`${call.ledgerId}: not approved`);
if (!/^Accepted scope: \S/m.test(record)) problems.push(`${call.ledgerId}: missing accepted scope`);
});
return problems;
}
+11 -2
View File
@@ -160,7 +160,14 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact.test.ts', 'test/helpers/workflow-excerpt.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/plan-eng-resume.ts', 'test/fixtures/plan-eng-artifact-resume-37174266054.json', 'test/helpers/workflow-excerpt.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'],
'plan-eng-review-artifact-full': [ 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/pty/**', 'test/helpers/pty/auq.ts', 'test/helpers/pty/binary.ts', 'test/helpers/pty/boundaries.ts', 'test/helpers/pty/classify.ts', 'test/helpers/pty/fake-session.ts', 'test/helpers/pty/judge.ts', 'test/helpers/pty/launch.ts', 'test/helpers/pty/plan-native.ts', 'test/helpers/pty/runners/counting.ts', 'test/helpers/pty/runners/floor.ts', 'test/helpers/pty/runners/observation.ts', 'test/helpers/pty/screen.ts', 'test/helpers/pty/session.ts', 'test/helpers/skill-census.ts',
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact-full.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/workflow-excerpt.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'],
'plan-review-report': [
@@ -1193,7 +1200,8 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
'plan-ceo-review-selective': 'periodic',
'plan-ceo-review-expansion-energy': 'periodic', // Demoted from gate (2026-08 audit): Opus generator + subjective 2-axis >=4/5 LLM-judge threshold in the merge lane — the exact class siblings were demoted for (a +21-line preamble change once flipped the score). CLAUDE.md's own rule: Opus model test -> periodic.
'plan-eng-review': 'periodic',
'plan-eng-review-artifact': 'periodic',
'plan-eng-review-artifact': 'periodic', // Checkpoint: resumes at Test review from a real Scope Challenge ledger
'plan-eng-review-artifact-full': 'marathon', // Full fresh /plan-eng-review through Test review (498 s pass; 595 s timeouts at 600 s before Test review)
'plan-eng-coverage-audit': 'gate',
'ship-coverage-value': 'gate',
'review-test-value': 'gate',
@@ -1631,6 +1639,7 @@ export const E2E_KINDS: Record<string, 'rule' | 'behavior' | 'judge'> = {
'plan-ceo-review-expansion-energy': 'behavior',
'plan-eng-review': 'rule',
'plan-eng-review-artifact': 'rule',
'plan-eng-review-artifact-full': 'rule',
'plan-eng-coverage-audit': 'rule',
'plan-review-report': 'rule',
'plan-ceo-review-plan-mode': 'rule',
+31
View File
@@ -0,0 +1,31 @@
import { describe, expect, test } from 'bun:test';
import { DASHBOARD_PLAN } from './helpers/plan-eng-artifact-fixture';
import { engResumeProblems, engResumeReport, type EngResumeCall } from './helpers/plan-eng-resume';
import resume from './fixtures/plan-eng-artifact-resume-37174266054.json';
const calls = resume.calls as EngResumeCall[];
const plan = DASHBOARD_PLAN.slice(DASHBOARD_PLAN.indexOf('# Plan: Add Dashboard'));
const report = engResumeReport(plan, calls);
describe('plan-eng-review-artifact resume point (census 37174266054)', () => {
test('the real Scope Challenge calls rebuild a valid resume point at Section 3', () => {
expect(calls.map(call => call.question.split(' ')[0])).toEqual(['D1', 'D2', 'D3', 'D4', 'D5', 'D6']);
expect(calls.every(call => call.options.some(option => option.label === call.answer))).toBe(true);
expect(engResumeProblems(report, calls)).toEqual([]);
expect(report.startsWith(plan)).toBe(true);
expect(report).toContain('Next: Section 3, Test review.');
for (const call of calls) expect(report).toContain(`### ${call.ledgerId}: ${call.header}`);
});
test.each([
['a missing record', (r: string) => r.replace(/### R7: [\s\S]*?History: —\n/, '')],
['a changed actual answer', (r: string) => r.replace('Actual answer: A) Vitest + TypeScript (recommended)', 'Actual answer: B) node:test with native type stripping')],
['an edited option', (r: string) => r.replace('A) Make Dashboard async and await fetchStats (recommended)\n', 'A) Make Dashboard synchronous\n')],
['a pending record', (r: string) => r.replace('State: approved', 'State: pending')],
['a duplicate ledger', (r: string) => r + '\n## Decision ledger\n'],
['no Scope Challenge result', (r: string) => r.replace(/^Scope Challenge result: .*$/m, '')],
['Section 2 still open', (r: string) => r.replace('Section 2, Code quality review: complete', 'Section 2, Code quality review: in progress')],
])('rejects %s', (_name, mutate) => {
expect(engResumeProblems(mutate(report), calls).length).toBeGreaterThan(0);
});
});
+1 -1
View File
@@ -7,7 +7,7 @@ const ptyIds = [
'auto-decide-preserved', 'plan-ceo-mode-routing', 'plan-design-with-ui-scope', 'plan-eng-finding-floor',
'auq-format-gate', 'carve-section-loading', 'office-hours-section-loading', 'office-hours-design-draft', 'plan-ceo-section-loading', 'ship-section-loading',
'plan-ceo-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor',
'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact',
'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact', 'plan-eng-review-artifact-full',
].sort();
test('PTY supervision controls select every current runner consumer', () => {
@@ -0,0 +1,71 @@
/**
* Marathon: a fresh /plan-eng-review, from startup through Test review, writes
* its QA test-plan artifact (paid, real-PTY, non-blocking).
*
* The periodic checkpoint (skill-e2e-plan-eng-artifact.test.ts) resumes after
* Scope Challenge and Sections 1-2; this case keeps the whole interactive path.
* Budget: PTY_LONG_MS. Measured on Claude Code 2.1.284: one fresh run passed at
* 498 s; census 37174266054 and a local run both spent 595 s on startup plus six
* one-at-a-time Scope Challenge questions without reaching Test review.
*
* Replaces the composing `plan-eng-review-artifact` case in skill-e2e-plan.test.ts.
* That case told a `claude -p` session to skip the preamble and questions and
* "go straight to the review"; in runs 37151477069, 37158847998 and 37162480720
* (and one local CI-image run) the model wrote the whole review into one file
* without executing the workflow, so it measured composing, not the skill. A
* full interactive run does write the artifact (37158847998's batching PTY run
* wrote eng-review-test-plan-20261003-224617.md during Test review).
*
* This case drives the real interactive /plan-eng-review over the same
* dashboard plan and sources, answers each question with its first option, and
* stops once the artifact exists or the report completes. The requirement is
* unchanged: exactly one new QA test plan under the run's own state root, about
* the reviewed dashboard change.
*/
import { expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { PTY_LONG_MS } from './helpers/eval-budgets';
import { DASHBOARD_PLAN, FIXTURE_PLAN_PATH, SOURCES } from './helpers/plan-eng-artifact-fixture';
import { describeE2ETier } from './helpers/e2e-gate';
import { engFirstReviewAUQ, engSetupAUQ, engStep0Boundary, hasCompletePlanReport, runPlanSkillCounting } from './helpers/claude-pty-runner';
const describeE2E = describeE2ETier('marathon');
describeE2E('/plan-eng-review QA test-plan artifact, full workflow (marathon)', () => {
test('a fresh interactive review reaches Test review and writes one QA test plan', async () => {
const startedAt = Date.now();
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-e2e-plan-eng-artifact-full-'));
const planPath = path.join(tmpDir, 'gstack-test-plan-eng-artifact.md');
try {
const obs = await runPlanSkillCounting({
skillName: 'plan-eng-review',
slashCommand: '/plan-eng-review',
followUpPrompt: DASHBOARD_PLAN.replaceAll(FIXTURE_PLAN_PATH, planPath),
fixtureFiles: SOURCES,
permissionPlanPath: planPath,
isLastStep0AUQ: engStep0Boundary,
isSetupAUQ: engSetupAUQ,
isFirstReviewAUQ: engFirstReviewAUQ,
// The artifact is the only verdict; stop once it exists or the report is complete.
isCollectionComplete: (_transcript, _fingerprints, testPlans) =>
testPlans.length > 0 || hasCompletePlanReport(planPath, startedAt, Date.now()),
reviewCountCeiling: Infinity,
preconfiguredReviewActor: true,
timeoutMs: PTY_LONG_MS,
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
});
const plans = obs.engTestPlans;
console.log(`plan-eng-review-artifact-full: outcome=${obs.outcome} elapsed=${obs.elapsedMs}ms review=${obs.reviewCount} test plans=${plans.map(plan => plan.file).join(', ') || 'none'}`);
if (!['plan_ready', 'completion_summary', 'collection_complete'].includes(obs.outcome)) {
throw new Error(`plan-eng-review-artifact-full: outcome=${obs.outcome} (${obs.summary})\n--- evidence ---\n${obs.evidence}`);
}
expect(plans, `expected one new eng-review test plan; outcome=${obs.outcome}`).toHaveLength(1);
expect(/dashboard|fetchStats|\/api\/stats/i.test(plans[0]!.content), 'the test plan covers the reviewed dashboard change').toBe(true);
} finally {
fs.rmSync(tmpDir, { recursive: true, force: true });
}
}, PTY_LONG_MS + 30_000);
});
+26 -37
View File
@@ -1,6 +1,6 @@
/**
* /plan-eng-review writes its QA test-plan artifact when run the way users run
* it (periodic, paid, real-PTY).
* /plan-eng-review writes its QA test-plan artifact when its live Test review
* runs (periodic checkpoint, paid, real-PTY).
*
* Replaces the composing `plan-eng-review-artifact` case in skill-e2e-plan.test.ts.
* That case told a `claude -p` session to skip the preamble and questions and
@@ -10,11 +10,17 @@
* full interactive run does write the artifact (37158847998's batching PTY run
* wrote eng-review-test-plan-20261003-224617.md during Test review).
*
* This case drives the real interactive /plan-eng-review over the same
* dashboard plan and sources, answers each question with its first option, and
* stops once the artifact exists or the report completes. The requirement is
* A fresh run spends its budget on startup and one-at-a-time Scope Challenge
* questions (census 37174266054: six questions, timeout at 595 s, no Test
* review). This checkpoint resumes the real interactive /plan-eng-review from
* the record a real run saves: the plan, Scope Challenge result, Sections 1-2
* dispositions and a Decision ledger rebuilt from that census run's six actual
* native questions and answers. The live skill then runs Section 3, Test
* review, answering each question with its first option, and the case stops
* once the artifact exists or the report completes. The requirement is
* unchanged: exactly one new QA test plan under the run's own state root, about
* the reviewed dashboard change.
* the reviewed dashboard change. The fresh start-to-finish path is the marathon
* case in skill-e2e-plan-eng-artifact-full.test.ts.
*/
import { expect, test } from 'bun:test';
@@ -22,51 +28,34 @@ import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { CAPTURE_LONG_MS } from './helpers/eval-budgets';
import { DASHBOARD_PLAN, SOURCES } from './helpers/plan-eng-artifact-fixture';
import { engResumeProblems, engResumeReport, type EngResumeCall } from './helpers/plan-eng-resume';
import resume from './fixtures/plan-eng-artifact-resume-37174266054.json';
import { describeE2ETier } from './helpers/e2e-gate';
import { engFirstReviewAUQ, engSetupAUQ, engStep0Boundary, hasCompletePlanReport, runPlanSkillCounting } from './helpers/claude-pty-runner';
const describeE2E = describeE2ETier('periodic');
/** Working-plan path in the prompt; rewritten per run to a private temporary path. */
const FIXTURE_PLAN_PATH = '/tmp/gstack-test-plan-eng-artifact.md';
const DASHBOARD_PLAN = [
'Proceed directly to the requested engineering review; skip the optional /office-hours prerequisite.',
`Please review this plan thoroughly. As you go, write your plan-mode plan to ${FIXTURE_PLAN_PATH} (use Edit/Write to that exact path).`,
'',
'# Plan: Add Dashboard',
'',
'## Changes',
'1. New `dashboard.ts` with Dashboard component and fetchStats API call',
'2. Updated `app.ts` to import and use Dashboard',
'',
'## Architecture',
'- Dashboard fetches from `/api/stats` endpoint',
'- Returns user count and revenue metrics',
].join('\n');
const SOURCES = {
'app.ts': 'import { Dashboard } from "./dashboard";\nexport function greet() { return "hello"; }\nexport function main() { return Dashboard(); }\n',
'dashboard.ts': `export function Dashboard() {
const data = fetchStats();
return { users: data.users, revenue: data.revenue };
}
function fetchStats() {
return fetch('/api/stats').then(r => r.json());
}
`,
};
describeE2E('/plan-eng-review QA test-plan artifact (periodic)', () => {
test('an interactive review writes one QA test plan about the reviewed change', async () => {
const startedAt = Date.now();
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-e2e-plan-eng-artifact-'));
const planPath = path.join(tmpDir, 'gstack-test-plan-eng-artifact.md');
const calls = resume.calls as EngResumeCall[];
const plan = DASHBOARD_PLAN.slice(DASHBOARD_PLAN.indexOf('# Plan: Add Dashboard'));
const report = engResumeReport(plan, calls);
expect(engResumeProblems(report, calls)).toEqual([]);
fs.writeFileSync(planPath, report);
try {
const obs = await runPlanSkillCounting({
skillName: 'plan-eng-review',
slashCommand: '/plan-eng-review',
followUpPrompt: DASHBOARD_PLAN.replaceAll(FIXTURE_PLAN_PATH, planPath),
followUpPrompt: [
'Proceed directly to the requested engineering review; skip the optional /office-hours prerequisite.',
`Resume my /plan-eng-review of this plan. Its report file is ${planPath}: it holds the plan, the completed Scope Challenge (D1-D${calls.length} answered) and Sections 1-2 (Architecture, Code quality) with no open choices, plus the Decision ledger. Read it first, keep using it as the report file, and do not redo or re-ask those stages. Continue at Section 3, Test review, and carry the review on from there.`,
'',
plan,
].join('\n'),
fixtureFiles: SOURCES,
permissionPlanPath: planPath,
isLastStep0AUQ: engStep0Boundary,
+1
View File
@@ -150,6 +150,7 @@ const BASELINE = [
'test/plan-create-permission.test.ts',
'test/plan-create-prepublication.test.ts',
'test/plan-edit-cropped-permission.test.ts',
'test/plan-eng-resume.test.ts', // harness owner for the plan-eng-review-artifact checkpoint: proves its resume point before paid spend
'test/plan-floor-dx-actor.test.ts',
'test/plan-floor-review.test.ts',
'test/plan-mode-evidence.test.ts',