diff --git a/TODOS.md b/TODOS.md index 4ce5715c8..d8f46ebdd 100644 --- a/TODOS.md +++ b/TODOS.md @@ -4,7 +4,7 @@ ### P2: `/plan-eng-review` retirement follow-ups (filed 2026-11-06, GST-1698 /ship review of #4108) -- **Stop the dots and generic harness targets from registering the retired skill.** `sourceSkills()` in `lib/harness-register.ts` lists every committed Claude-render skill and filters out only `gstack`, so `gstack-harness register --target dots|generic` still lists `plan-eng-review` in its router and gives it a folder in `all` mode. Filter `RETIRED_SKILLS` there (the Claude-render /autoplan reads Phase 3 from `~/.claude/skills/gstack/plan-eng-review/SKILL.md`, not from the harness payload) and add a case to `test/retired-skills.test.ts`. Raised by the outside voice (labelled blocking) and the native adversarial pass (non-blocking); below the blocking floor because nothing fails and the skill still works. **Effort:** S. **Priority:** P2. +- **Delete the retired `plan-eng-multi-finding-batching` case and its counter subsystem.** The case checks one interactive question per eng finding; /autoplan Phase 3 auto-decides findings, so no shipped entry reaches it. The follow-up to #4108 took it out of the weekly lane through `PERIODIC_CI_EXCLUDE` instead of deleting it, because about 20 free tests and fixtures depend on its counter (`createEngBatchingIssueCounter` in `test/helpers/eng-seeded-coverage.ts`, `test/eng-batching-saved-ledger.test.ts`, `test/detector-corpus-plan-eng-multi-finding-batching.test.ts`, the `eng-batching-*` fixtures). Retirement card: detects=an eng review that batches findings into one plan write; non_test_callers=0; stronger_proof=the ceo/design/devex finding floors keep the shared one-question-per-finding contract; unlocks=the counter helpers, their free tests and fixtures. Delete each helper only after checking it has no other caller (plan-eng-review-artifact shares `engFirstReviewAUQ`). **Effort:** M. **Priority:** P3. - **Point the external-host methodology file's preamble Read at `gstack-autoplan`.** Every external host's `gstack-autoplan/plan-eng-review.md` says to Read `sections/preamble.md` relative to the installed `gstack-plan-eng-review` directory, which no longer exists. Normally the matching preamble id skips the reread, but after compaction late in a run the agent follows a missing path. Fixing it changes Phase 3 bytes, so it needs its own byte-pin update. **Effort:** S. **Priority:** P3. - **Redirect `gstack-doctor --for plan-eng-review`.** It now exits 2 with "unknown skill"; map the retired name to `/autoplan`'s requirements and print a one-line redirect, so saved lane scripts keep working. **Effort:** S. **Priority:** P3. - **Decide whether the QM edition keeps `gstack-plan`.** `hosts/qm.ts` and `qm/edition.json` still build `gstack-plan` from the `plan-eng-review` sources under another name, so the retired-name guards don't see it. Product call for Garry. **Effort:** S. **Priority:** P3. @@ -545,8 +545,9 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10 a PTY. A fresh run needs more than its 600 s budget (498 s pass; 595 s timeouts in census 37174266054 and a local run), so the periodic case resumes at Test review from that census run's real Scope Challenge ledger - (pass, 364 s) and the fresh run is the marathon case - plan-eng-review-artifact-full. + (pass, 364 s) and the fresh run was the marathon case + plan-eng-review-artifact-full (retired with standalone /plan-eng-review, + GST-1698). - **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed first. **Effort:** S. **Priority:** P3. @@ -2104,7 +2105,8 @@ alone). Recommend (b) short-term + (a) properly sequenced with the codemod. **What:** `evals-periodic.yml` runs an explicit 9-file matrix; the four tests demoted to `periodic` in v1.62.0.0 (`skill-e2e-plan-eng-plan-mode`, `skill-e2e-plan-design-plan-mode`, `skill-e2e-plan-eng-finding-floor`, -`skill-e2e-plan-design-finding-floor`) are not in it, so they currently run +`skill-e2e-plan-design-finding-floor`; the two plan-eng files were deleted with +standalone /plan-eng-review in GST-1698) are not in it, so they currently run only locally/manually (`bun run test:periodic` or `eval:bg:periodic`). Wiring them needs a PTY-capable periodic job: the container skill-registration setup from evals.yml's `e2e-pty-plan-smoke` job (real-file SKILL.md copies for the diff --git a/changelog.d/capy-gst-1698-retired-followups.md b/changelog.d/capy-gst-1698-retired-followups.md new file mode 100644 index 000000000..5c7bcaa78 --- /dev/null +++ b/changelog.d/capy-gst-1698-retired-followups.md @@ -0,0 +1 @@ +- **The dots and generic harness targets no longer register the retired /plan-eng-review.** `gstack-harness register --target dots|generic` listed every committed skill except `gstack`, so after #4108 it still put `plan-eng-review` in its router and, in `all` mode, gave it a folder. It now skips every skill in `RETIRED_SKILLS`; `plan-eng-review/` stays in the source tree as /autoplan's Phase 3 methodology. The paid cases written for the standalone skill each have a recorded fate in `docs/TEST_PORTFOLIO.md`: four that check Phase 3 methodology or its required outputs stay (`plan-eng-review`, `plan-eng-coverage-audit`, `plan-eng-review-artifact`, `journey-plan-eng`), six that checked only the standalone interactive entry are deleted along with their rows in the AUQ matrix and the plan-mode no-op file, and `plan-eng-multi-finding-batching` leaves the weekly lane through `PERIODIC_CI_EXCLUDE` until its counter helpers are removed. (GST-1698) diff --git a/docs/TEST_PORTFOLIO.md b/docs/TEST_PORTFOLIO.md index 7cfbf631e..b63a2dcc6 100644 --- a/docs/TEST_PORTFOLIO.md +++ b/docs/TEST_PORTFOLIO.md @@ -286,6 +286,29 @@ or out-of-tier placeholders. A retained passing judge is reusable only when its complete request, rubric, parameters and relevant dependencies match; native behavioral acceptance follows its separate freshness contract. +## Paid cases of the retired standalone /plan-eng-review (GST-1698) + +The skill's methodology still ships: `plan-eng-review/` is /autoplan's Phase 3 +methodology source, bound into each run by the methodology snapshot. Cases that +check that methodology or Phase 3's required outputs stay. Cases that checked +the standalone interactive entry, which no longer ships, are retired. None was +quarantined: `CASE_QUARANTINE` holds flaky cases that keep running, and these +cases are not flaky, their entry point is gone. + +| Case | Fate | Why, and what still proves the contract | +| --- | --- | --- | +| `plan-eng-review` (SDK) | kept | Runs the methodology source over a plan, the bytes Phase 3 loads. | +| `plan-eng-coverage-audit` (gate) | kept | Phase 3 must produce the codepath-to-test diagram; the case reads the same methodology and sections. | +| `plan-eng-review-artifact` (PTY checkpoint) | kept | Phase 3 must leave a test plan on disk; the case's Test review writes it from the same section. | +| `journey-plan-eng` | kept, retargeted | Architecture-review requests now route to /autoplan (#4108). | +| `plan-eng-multi-finding-batching` | retired, excluded | Checks one interactive question per finding; Phase 3 auto-decides findings. `PERIODIC_CI_EXCLUDE` stops its weekly run; deleting it with its counter helpers is a TODOS.md item. | +| `plan-eng-review-plan-mode` | retired, deleted | Standalone plan-mode handshake; the ceo, design and devex plan-mode cases and `plan-mode-no-op` keep the shared contract. | +| `plan-eng-finding-floor` | retired, deleted | Interactive first-finding question; the ceo, design and devex floors keep the shared floor. | +| `plan-eng-review-artifact-full` | retired, deleted | Fresh standalone marathon run; the checkpoint case keeps the artifact requirement. | +| `plan-eng-review-format-coverage`, `plan-eng-review-format-kind` | retired, deleted | Format of standalone questions; the ceo format cases keep the shared question format. | +| `carve-section-loading-plan-eng-review` | retired, deleted | Model-driven section loading on standalone entry; Phase 3 gets the methodology through the snapshot instead. | +| `auq-matrix-plan-eng-review`, the eng rows of `plan-mode-no-op` | retired rows | Standalone entry; the other skills' rows keep both files' contracts. | + ## Remaining work, not claimed savings Report final acceptance with the delivery revision and its actual case inventory. diff --git a/docs/evals/detector-inventory.md b/docs/evals/detector-inventory.md index b6c3d4824..0e38df97e 100644 --- a/docs/evals/detector-inventory.md +++ b/docs/evals/detector-inventory.md @@ -31,7 +31,7 @@ Dispositions: | F3 | test/helpers/pty/runners/floor.ts:497 | screen Write render / plan-ready text → fail | Write/ExitPlanMode tool_use in run.transcript | no | (b) | | A1 | test/helpers/auq-sdk-capture.ts:30-43 `scoreAuqFormat` | question has ELI10:, Recommendation:, Pros / cons, ✅, ❌, Net:, (recommended) | AUQ tool input (its prose is the contract) | yes | (c) | -Cases that use S1-S8: plan-ceo-review-plan-mode, plan-devex-review-plan-mode, plan-mode-no-op, office-hours-auto-mode (gate); auto-decide-preserved, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke (periodic). S9-S10: plan-mode-no-op, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke. F1-F3: plan-ceo-finding-floor, plan-devex-finding-floor (gate); plan-design-finding-floor, plan-eng-finding-floor (periodic). A1: auq-format-gate (gate); the three standalone AUQ files (periodic). +Cases that use S1-S8: plan-ceo-review-plan-mode, plan-devex-review-plan-mode, plan-mode-no-op, office-hours-auto-mode (gate); auto-decide-preserved, plan-design-review-plan-mode-smoke (periodic). S9-S10: plan-mode-no-op, plan-design-review-plan-mode-smoke. F1-F3: plan-ceo-finding-floor, plan-devex-finding-floor (gate); plan-design-finding-floor (periodic). A1: auq-format-gate (gate); the three standalone AUQ files (periodic). plan-eng-review-plan-mode and plan-eng-finding-floor were deleted with standalone /plan-eng-review (GST-1698). ## Gate tier @@ -205,18 +205,18 @@ Gate cases with no phrase-only checks: the shared-libs review-lifecycle, revalid | plan-eng-multi-finding-batching | test/helpers/eng-seeded-coverage.ts:33-104 | distinct-issue identity per answered AUQ ("Issue N:", D/R-ID, ELI10 gates) | ledger R records, used only as fallback behind the prose gates | no | (b) | | plan-eng-multi-finding-batching | test/helpers/pty/classify.ts:555 via counting.ts:554 | screen completion text ends counting | review-log row / ExitPlanMode tool_use | no | (b) | | plan-eng-review-artifact | test/skill-e2e-plan-eng-artifact.test.ts:78 | test-plan artifact ~ dashboard/fetchStats/`/api/stats` | artifact schema has no field naming the change (scripts/resolvers/testing.ts:501-528) | no | (d) | -| carve-section-loading [plan-eng-review] | test/helpers/carve-section-case.ts:120 → test/helpers/auq-sdk-capture.ts:381 | report or output ~ report/review/summary/design doc/handoff | review-log row in the capture state root, deleted before the verdict (auq-sdk-capture.ts:368-370) | no | (b) | +| carve-section-loading [plan-eng-review] (deleted, GST-1698) | test/helpers/carve-section-case.ts:120 → test/helpers/auq-sdk-capture.ts:381 | report or output ~ report/review/summary/design doc/handoff | review-log row in the capture state root, deleted before the verdict (auq-sdk-capture.ts:368-370) | no | (b) | | carve-section-loading [12 other skills] | same | same loose regex | none (Bash not allowed, auq-sdk-capture.ts:358-359) | no | (d) | -The plan-eng-review-plan-mode and plan-design-review-plan-mode-smoke verdicts are S1-S10. plan-eng-finding-floor and plan-design-finding-floor verdicts are F1-F3. +The plan-design-review-plan-mode-smoke verdicts are S1-S10, and the plan-design-finding-floor verdicts are F1-F3. Their plan-eng twins were deleted with standalone /plan-eng-review (GST-1698). ### Question format: test/skill-e2e-plan-format.test.ts, test/skill-e2e-plan-prosons.test.ts, standalone AUQ files | Case | Check (file:line) | Matches | Structured source | User-visible | Disposition | |---|---|---|---|---|---| -| plan-ceo-review-format-mode, plan-eng-review-format-kind | test/skill-e2e-plan-format.test.ts:174-175, :377-378 | no `Completeness: N/10`; kind note present | captured question | yes | (c) | -| plan-ceo-review-format-approach, plan-eng-review-format-coverage | test/skill-e2e-plan-format.test.ts:241, :310 | `Completeness: N/10` present | captured question | yes | (c) | -| all four format cases | test/skill-e2e-plan-format.test.ts:177, :243, :312, :380 | recommendation quality | — | — | judge | +| plan-ceo-review-format-mode | test/skill-e2e-plan-format.test.ts:173-174 | no `Completeness: N/10`; kind note present | captured question | yes | (c) | +| plan-ceo-review-format-approach | test/skill-e2e-plan-format.test.ts:240 | `Completeness: N/10` present | captured question | yes | (c) | +| both format cases | test/skill-e2e-plan-format.test.ts:176, :242 | recommendation quality | — | — | judge | | plan-review-prosons-format | test/skill-e2e-plan-prosons.test.ts:184-196 | `D\d+ —`, ELI10:, Stakes…:, Recommendation:, Pros / cons:, ^Net:, ✅≥4, ❌≥2, (recommended) | captured question | yes | (c) | | plan-review-prosons-hardstop-neg | test/skill-e2e-plan-prosons.test.ts:246-249 | no hard-stop escape line; ✅/❌ counts | captured question | yes | (c) | | plan-review-prosons-neutral-neg | test/skill-e2e-plan-prosons.test.ts:299-303 | no `taste call`; (recommended); `Recommendation:.*because` | captured question | yes | (c) | @@ -315,6 +315,6 @@ Red cases that are not convert candidates: shared-libs-plan-callers (3 reds, act - Census commits `a8ace39` through `fd6854b` are not in this checkout, so the wave-reds line numbers could not be mapped to assertions; reds are attributed from the trial error text. - The skill-llm-eval.test.ts judge cases are outside `E2E_TIERS` and are not covered. - codex-ci-access.test.ts, codex-hardening.test.ts, codex-under-codex-detection.test.ts, e2e-shard-reuse.test.ts and skill-routing-e2e.test.ts name the umbrella keys but are free tests or route to the files audited above. -- plan-eng-review-artifact-full is `marathon` and out of scope. +- plan-eng-review-artifact-full was `marathon` and was deleted with standalone /plan-eng-review (GST-1698). - The aside-* cases need a live Aside on macOS and never run in CI. - "Not verified" in the structured-source column means the template writes the record, but the audit did not confirm that the fixture installs the helper. diff --git a/lib/harness-register.ts b/lib/harness-register.ts index 10a46a093..a5e619155 100644 --- a/lib/harness-register.ts +++ b/lib/harness-register.ts @@ -19,6 +19,7 @@ import * as fs from 'node:fs'; import * as os from 'node:os'; import * as path from 'node:path'; import { spawnSync } from 'node:child_process'; +import { RETIRED_SKILLS } from '../scripts/discover-skills'; import { bunIsolationArgs } from './bun-isolation'; import { runBashSync } from './script-invocation'; import { frontmatterList, routerLine } from './skill-triggers'; @@ -233,7 +234,7 @@ function skillDirs(root: string, triggersByName = new Map()): /** The skills a target's payload carries: the host render (made for this checkout as the install root) or the checkout's committed Claude render. */ export function sourceSkills(target: Target, checkout: string, scratch: string): SkillEntry[] { - if (target.render === 'claude') return skillDirs(checkout).filter(s => s.name !== 'gstack'); + if (target.render === 'claude') return skillDirs(checkout).filter(s => s.name !== 'gstack' && !RETIRED_SKILLS.includes(s.name)); const out = path.join(scratch, 'render'); fs.rmSync(out, { recursive: true, force: true }); const r = spawnSync(process.execPath, [...bunIsolationArgs(), 'scripts/gen-skill-docs.ts', '--host', target.name, '--install-root', checkout, '--out-dir', out], diff --git a/scripts/harness-pins/scripts/lib/paid-cases.ts.json b/scripts/harness-pins/scripts/lib/paid-cases.ts.json index be046eaa0..5162bf28b 100644 --- a/scripts/harness-pins/scripts/lib/paid-cases.ts.json +++ b/scripts/harness-pins/scripts/lib/paid-cases.ts.json @@ -1 +1 @@ -{"blob":"f862cf5241580c7593b9ee62eacf5ef046248d26","since":6,"nonBehavioral":"Claude-host verdicts are unchanged: the Codex host (EVALS_HOST=codex, codex-host-runner) runs only when selected and records its own per-model series; the Codex runner refactor (execCodex extraction, onSpawn hook) keeps its events and verdicts and adds token cost accounting; host fields on the plan/report are absent for claude"} +{"blob":"c984ba618be367f2201362a9592da7a8631accb5","since":6,"nonBehavioral":"Retired standalone /plan-eng-review cases (GST-1698): removed their budgets and paid-case rows; remaining files' walls shrink with their case counts, and no remaining case's budget or verdict changes"} diff --git a/scripts/harness-pins/test/helpers/eval-budgets.ts.json b/scripts/harness-pins/test/helpers/eval-budgets.ts.json index f903f69d5..485698131 100644 --- a/scripts/harness-pins/test/helpers/eval-budgets.ts.json +++ b/scripts/harness-pins/test/helpers/eval-budgets.ts.json @@ -1 +1 @@ -{"blob":"8cb433b0618cc218dacf21cc644de7947442d045","since":6,"nonBehavioral":"GST-1403 removes the last three standalone LLM judges (qa anti-refusal, voice directive tone, cross-skill greptile consistency): llm-judge.ts drops their three schemas and three prompt builders, which no remaining case calls, and eval-budgets.ts drops the deleted test/skill-llm-eval.test.ts file wall; no kept case's prompt, threshold, samples, budget or verdict can change"} +{"blob":"5a16324a47b7c3c770e64322ae357f90c405a249","since":6,"nonBehavioral":"Retired standalone /plan-eng-review cases (GST-1698): removed their budgets and paid-case rows; remaining files' walls shrink with their case counts, and no remaining case's budget or verdict changes"} diff --git a/scripts/lib/paid-cases.ts b/scripts/lib/paid-cases.ts index f862cf524..c984ba618 100644 --- a/scripts/lib/paid-cases.ts +++ b/scripts/lib/paid-cases.ts @@ -76,7 +76,6 @@ export const CASE_TEST_NAMES: Record = { 'cso-infra-scope': '/cso --infra finds an attacker-to-credential execution path', 'plan-ceo-review-plan-mode': 'first terminal outcome is asked (Step 0 fires before any plan write)', 'plan-eng-review-artifact': 'an interactive review writes one QA test plan about the reviewed change', - 'plan-eng-review-artifact-full': 'a fresh interactive review reaches Test review and writes one QA test plan', }; export const CASE_KEY_SEPARATOR = '#'; diff --git a/scripts/paid-test-durations.json b/scripts/paid-test-durations.json index 22b1f8c06..31a4ad2ec 100644 --- a/scripts/paid-test-durations.json +++ b/scripts/paid-test-durations.json @@ -116,7 +116,6 @@ "test/carve-section-loading.test.ts#carve-section-loading-land-and-deploy": 156293, "test/carve-section-loading.test.ts#carve-section-loading-plan-design-review": 223606, "test/carve-section-loading.test.ts#carve-section-loading-plan-devex-review": 384295, - "test/carve-section-loading.test.ts#carve-section-loading-plan-eng-review": 297627, "test/carve-section-loading.test.ts#carve-section-loading-qa": 149637, "test/carve-section-loading.test.ts#carve-section-loading-retro": 199931, "test/carve-section-loading.test.ts#carve-section-loading-review": 247902, @@ -173,11 +172,8 @@ "test/skill-e2e-plan-design-finding-floor.test.ts": 131042, "test/skill-e2e-plan-design-plan-mode.test.ts": 111838, "test/skill-e2e-plan-devex-peer-comparison-classification.test.ts": 77201, - "test/skill-e2e-plan-eng-artifact-full.test.ts": 595009, "test/skill-e2e-plan-eng-artifact.test.ts": 406795, - "test/skill-e2e-plan-eng-finding-floor.test.ts": 234397, "test/skill-e2e-plan-eng-multi-finding-batching.test.ts": 409161, - "test/skill-e2e-plan-eng-plan-mode.test.ts": 170914, "test/skill-e2e-plan-format.test.ts": 219670, "test/skill-e2e-plan-prosons.test.ts": 163659, "test/skill-e2e-plan-tune.test.ts": 1000, diff --git a/test/carve-section-loading.test.ts b/test/carve-section-loading.test.ts index b24b86c99..46c57238e 100644 --- a/test/carve-section-loading.test.ts +++ b/test/carve-section-loading.test.ts @@ -24,7 +24,6 @@ describeE2ETier('periodic')('carve section-loading', () => { test('carve-section-loading-land-and-deploy', () => runCarveSectionCase('land-and-deploy'), CAPTURE_LONG_MS); test('carve-section-loading-plan-design-review', () => runCarveSectionCase('plan-design-review'), CAPTURE_LONG_MS); test('carve-section-loading-plan-devex-review', () => runCarveSectionCase('plan-devex-review'), CAPTURE_LONG_MS); - test('carve-section-loading-plan-eng-review', () => runCarveSectionCase('plan-eng-review'), CAPTURE_LONG_MS); test('carve-section-loading-qa', () => runCarveSectionCase('qa'), CAPTURE_LONG_MS); test('carve-section-loading-retro', () => runCarveSectionCase('retro'), CAPTURE_LONG_MS); test('carve-section-loading-review', () => runCarveSectionCase('review'), CAPTURE_LONG_MS); diff --git a/test/carve-section-sharding.test.ts b/test/carve-section-sharding.test.ts index b12dc0944..f9be6a12b 100644 --- a/test/carve-section-sharding.test.ts +++ b/test/carve-section-sharding.test.ts @@ -9,6 +9,7 @@ import { CASE_SHARDED_FILES, DEFAULT_SHARD_TIMEOUT_MS, caseFile, expandCaseShard import { carveSectionCaseId } from './helpers/carve-section-case'; import { E2E_TIERS, E2E_TOUCHFILES, selectTests } from './helpers/touchfiles'; import { E2E_KINDS } from './helpers/touchfiles-data'; +import { RETIRED_SKILLS } from '../scripts/discover-skills'; describe('carved-skill cases each get a complete paid process budget', () => { const FILE = 'test/carve-section-loading.test.ts'; @@ -16,7 +17,8 @@ describe('carved-skill cases each get a complete paid process budget', () => { const source = fs.readFileSync(path.join(ROOT, FILE), 'utf8'); const cases = [...source.matchAll(/test\('carve-section-loading-([a-z-]+)', \(\) => runCarveSectionCase\('([a-z-]+)'\), CAPTURE_LONG_MS\)/g)] .map(match => ({ id: `carve-section-loading-${match[1]}`, skill: match[2]! })); - const generic = Object.values(CARVE_GUARDS).filter(guard => guard.behavioral === 'plan' || guard.behavioral === 'prompt').map(guard => guard.skill).sort(); + // A retired skill keeps its carve guard (its methodology still renders) but no section-loading case. + const generic = Object.values(CARVE_GUARDS).filter(guard => (guard.behavioral === 'plan' || guard.behavioral === 'prompt') && !RETIRED_SKILLS.includes(guard.skill)).map(guard => guard.skill).sort(); test('every generic registry entry has exactly one periodic case, under its own case id, and no external entry does', () => { expect(source).toContain("describeE2ETier('periodic')"); diff --git a/test/e2e-harness-audit.test.ts b/test/e2e-harness-audit.test.ts index 537845ab6..41bcd15e3 100644 --- a/test/e2e-harness-audit.test.ts +++ b/test/e2e-harness-audit.test.ts @@ -13,6 +13,7 @@ import { describe, test, expect } from 'bun:test'; import * as fs from 'fs'; import * as path from 'path'; +import { RETIRED_SKILLS } from '../scripts/discover-skills'; const ROOT = path.resolve(import.meta.dir, '..'); @@ -47,7 +48,8 @@ function skillTemplateDirs(): string[] { */ function findInteractiveSkills(): string[] { const interactive: string[] = []; - for (const skill of skillTemplateDirs()) { + // A retired skill keeps its template as methodology for another skill but has no interactive entry. + for (const skill of skillTemplateDirs().filter(name => !RETIRED_SKILLS.includes(name))) { const tmplPath = path.join(ROOT, skill, 'SKILL.md.tmpl'); if (!fs.existsSync(tmplPath)) continue; const content = fs.readFileSync(tmplPath, 'utf-8'); diff --git a/test/eng-finding-retry-budget.test.ts b/test/eng-finding-retry-budget.test.ts index d2ad31fd7..7dbbaa3af 100644 --- a/test/eng-finding-retry-budget.test.ts +++ b/test/eng-finding-retry-budget.test.ts @@ -1,6 +1,7 @@ import { expect, test } from 'bun:test'; import { resolvePaidShardBudget, retriesForFiles, planPaidShards, parseRunManifest, verifySliceResults, runPaidShard, buildRunManifest, paidShardWallUpperBoundMs, collectPaidTestFiles, selectPaidTestFiles, isOverlayTestFile, DEFAULT_SHARD_TIMEOUT_MS, DEFAULT_JOBS, parseCliOptions, expandCaseShards, expandTrialShards, shardFile, sliceExecutionOrder, sliceSupervisedWallMs, resolvePaidShardTimeoutMs } from '../scripts/test-paid-shards'; import { FINDING_RETRY_BUDGETS, ALL_TIERS, SHARD_RESERVE_MS } from './helpers/eval-budgets'; +import { PERIODIC_CI_EXCLUDE } from './helpers/periodic-exclude-data'; import fs from 'node:fs'; import os from 'node:os'; import path from 'node:path'; @@ -141,7 +142,7 @@ test('live periodic census fits the declared CI wall including setup', () => { test('registered allocation is deterministic and preserves every discovered file', () => { const files = collectPaidTestFiles(); - expect(files).toHaveLength(114); // - skill-llm-eval (last standalone judges removed, GST-1403); + skill-e2e-autoplan-ceo-only (GST-537); + skill-e2e-planning-wave (GSTA-228); + skill-e2e-model-panel (GSTA-173); + skill-e2e-write-skill (GSTA-133); + skill-e2e-ship-frozen-scope (frozen review scope, GSTA-113 PR 2); + skill-e2e-failure-routing (#3157/#3158); + skill-e2e-dx-error-contract (#3226); + skill-e2e-gate-integrity (HONEST_WORK_GATE_INTEGRITY W8); + skill-e2e-autoplan-schema-canary, -guard-pty and -long-session (Oct 7 wave, CEO-13, ENG-18, CEO-15); + skill-e2e-ship-measure-seeded-flake (v1.91.32.0); W5a: 15 carve wrappers became one case-sharded file; + codex-e2e-multiblock-live and skill-e2e-design-model-smoke (v1.91.30.0); + expect(files).toHaveLength(111); // - skill-e2e-plan-eng-plan-mode, -plan-eng-finding-floor and -plan-eng-artifact-full (standalone /plan-eng-review retired, GST-1698); - skill-llm-eval (last standalone judges removed, GST-1403); + skill-e2e-autoplan-ceo-only (GST-537); + skill-e2e-planning-wave (GSTA-228); + skill-e2e-model-panel (GSTA-173); + skill-e2e-write-skill (GSTA-133); + skill-e2e-ship-frozen-scope (frozen review scope, GSTA-113 PR 2); + skill-e2e-failure-routing (#3157/#3158); + skill-e2e-dx-error-contract (#3226); + skill-e2e-gate-integrity (HONEST_WORK_GATE_INTEGRITY W8); + skill-e2e-autoplan-schema-canary, -guard-pty and -long-session (Oct 7 wave, CEO-13, ENG-18, CEO-15); + skill-e2e-ship-measure-seeded-flake (v1.91.32.0); W5a: 15 carve wrappers became one case-sharded file; + codex-e2e-multiblock-live and skill-e2e-design-model-smoke (v1.91.30.0); expect(files).toContain('test/skill-e2e-ship-skip.test.ts'); const m = livePlan(files); expect(livePlan([...files].reverse())).toEqual(m); @@ -175,6 +176,7 @@ test('single-slice manifest retains all registered files with one allocation', ( for (const m of manifests) expect(m.entries.filter(e => e.status === 'planned').every(e => e.slice === 1)).toBe(true); for (const budget of FINDING_RETRY_BUDGETS) { const entries = manifests.flatMap(m => m.entries.filter(e => e.file === budget.file && e.status === 'planned')); + if (PERIODIC_CI_EXCLUDE[budget.file]) { expect(entries, budget.file).toHaveLength(0); continue; } expect(entries, budget.file).toHaveLength(1); expect(entries[0]!.budget).toEqual(resolvePaidShardBudget([budget.file])); } diff --git a/test/eval-kinds.test.ts b/test/eval-kinds.test.ts index d385ac044..0d2eb797c 100644 --- a/test/eval-kinds.test.ts +++ b/test/eval-kinds.test.ts @@ -62,7 +62,7 @@ describe('E2E_KINDS registry', () => { // Contract-shaped cases stay rule: ask-before-decide, plan-mode no-writes, // mandated steps, secrets, and the batching floor never ride a majority. for (const id of ['plan-ceo-mode-routing', 'plan-eng-multi-finding-batching', 'plan-design-review-plan-mode', - 'plan-eng-review-plan-mode', 'plan-ceo-section-loading', 'setup-gbrain-bad-token', 'qa-only-no-fix', 'review-sql-injection']) { + 'plan-ceo-section-loading', 'setup-gbrain-bad-token', 'qa-only-no-fix', 'review-sql-injection']) { expect(E2E_KINDS[id], id).toBe('rule'); } }); diff --git a/test/fixtures/paid-report-census/gate/manifest.json b/test/fixtures/paid-report-census/gate/manifest.json index eb7c0b628..365a0fffb 100644 --- a/test/fixtures/paid-report-census/gate/manifest.json +++ b/test/fixtures/paid-report-census/gate/manifest.json @@ -16,7 +16,7 @@ "status": "planned", "estimatedMs": 217184, "budget": { - "timeoutMs": 3720000, + "timeoutMs": 2520000, "source": "registered", "policyId": "skill-e2e-plan-mode-no-op-existing-retry-v1" } diff --git a/test/fixtures/paid-report-census/gate/paid-slice-1-a1/slice-1.json b/test/fixtures/paid-report-census/gate/paid-slice-1-a1/slice-1.json index 9717aaa4e..521b6cece 100644 --- a/test/fixtures/paid-report-census/gate/paid-slice-1-a1/slice-1.json +++ b/test/fixtures/paid-report-census/gate/paid-slice-1-a1/slice-1.json @@ -20,7 +20,7 @@ "executedTests": 5, "skippedTests": 0, "budget": { - "timeoutMs": 3720000, + "timeoutMs": 2520000, "source": "registered", "policyId": "skill-e2e-plan-mode-no-op-existing-retry-v1" } diff --git a/test/fixtures/touchfile-moved-code/w4-pty.json b/test/fixtures/touchfile-moved-code/w4-pty.json index 9efc32e89..394e12e47 100644 --- a/test/fixtures/touchfile-moved-code/w4-pty.json +++ b/test/fixtures/touchfile-moved-code/w4-pty.json @@ -15,7 +15,6 @@ "carve-section-loading-land-and-deploy", "carve-section-loading-plan-design-review", "carve-section-loading-plan-devex-review", - "carve-section-loading-plan-eng-review", "carve-section-loading-qa", "carve-section-loading-retro", "carve-section-loading-review", @@ -33,9 +32,7 @@ "plan-design-with-ui-scope", "plan-devex-finding-floor", "plan-devex-review-plan-mode", - "plan-eng-finding-floor", "plan-eng-multi-finding-batching", - "plan-eng-review-plan-mode", "plan-mode-no-op", "ship-section-loading" ], @@ -54,7 +51,6 @@ "carve-section-loading-land-and-deploy", "carve-section-loading-plan-design-review", "carve-section-loading-plan-devex-review", - "carve-section-loading-plan-eng-review", "carve-section-loading-qa", "carve-section-loading-retro", "carve-section-loading-review", @@ -72,9 +68,7 @@ "plan-design-with-ui-scope", "plan-devex-finding-floor", "plan-devex-review-plan-mode", - "plan-eng-finding-floor", "plan-eng-multi-finding-batching", - "plan-eng-review-plan-mode", "plan-mode-no-op", "ship-section-loading" ], diff --git a/test/fixtures/touchfile-moved-code/w5-review-resolvers.json b/test/fixtures/touchfile-moved-code/w5-review-resolvers.json index 4b8b23bd0..6751f2998 100644 --- a/test/fixtures/touchfile-moved-code/w5-review-resolvers.json +++ b/test/fixtures/touchfile-moved-code/w5-review-resolvers.json @@ -14,7 +14,6 @@ "carve-section-loading-land-and-deploy", "carve-section-loading-plan-design-review", "carve-section-loading-plan-devex-review", - "carve-section-loading-plan-eng-review", "carve-section-loading-qa", "carve-section-loading-retro", "carve-section-loading-review", @@ -35,13 +34,9 @@ "plan-devex-finding-floor", "plan-devex-review-plan-mode", "plan-eng-coverage-audit", - "plan-eng-finding-floor", "plan-eng-multi-finding-batching", "plan-eng-review", "plan-eng-review-artifact", - "plan-eng-review-format-coverage", - "plan-eng-review-format-kind", - "plan-eng-review-plan-mode", "plan-mode-no-op", "plan-review-prosons-format", "plan-review-report", diff --git a/test/harness-register.test.ts b/test/harness-register.test.ts index b483d491c..213910a09 100644 --- a/test/harness-register.test.ts +++ b/test/harness-register.test.ts @@ -13,6 +13,7 @@ import { BLOCK_BEGIN, BLOCK_END, CONTEXT_BLOCK_MAX, DOT_SKILLS_DIR, HARNESS_CODES, HYBRID_SKILLS, HarnessError, OPENCLAW_ALL_TEXTS, OPENCLAW_INSTRUCTION_MAX, ROUTER_DESCRIPTION, ROUTER_DESCRIPTION_MAX, TARGETS, contextBlock, contextScope, detect, listLedgers, readLedger, refresh, register, sourceSkills, unregister, type RegisterOptions, } from '../lib/harness-register'; +import { RETIRED_SKILLS } from '../scripts/discover-skills'; import { QM_PACK_LIMITS, qmRequest, registerQm, removeQmPack } from '../lib/harness-qm'; import { cleanupFixtures, cleanupSeed, makeFixture, makeSource, registryRows, runSetup, tree } from './helpers/install-fixture'; @@ -139,6 +140,17 @@ describe('gstack-harness register and unregister (skill folders)', () => { } }, 60_000); + test('Claude-render targets (dots, generic) never register a retired skill; /autoplan still carries its methodology', () => { + const d = tmp(); + const names = sourceSkills(TARGETS.dots, ROOT, d).map(s => s.name); + expect(names).toContain('autoplan'); + for (const retired of RETIRED_SKILLS) expect(names).not.toContain(retired); + expect(sourceSkills(TARGETS.generic, ROOT, d).map(s => s.name)).toEqual(names); + const r = register(opts(d, 'dots', { mode: 'all', skillsDir: path.join(d, 'all') })); + for (const retired of RETIRED_SKILLS) expect(fs.existsSync(path.join(r.dest, retired))).toBe(false); + expect(fs.existsSync(path.join(ROOT, 'plan-eng-review', 'SKILL.md'))).toBe(true); + }, 60_000); + test('generic: the context block is replaced in place, then cut out; the user\'s lines and edits survive', () => { const d = tmp(); const ctx = path.join(d, 'agent', 'AGENTS.md'); diff --git a/test/helpers/eval-budgets.ts b/test/helpers/eval-budgets.ts index 8cb433b06..5a16324a4 100644 --- a/test/helpers/eval-budgets.ts +++ b/test/helpers/eval-budgets.ts @@ -99,16 +99,14 @@ export const FILE_RETRY_BUDGETS = [ { file: 'test/skill-e2e-qa-callers.test.ts', attemptMs: 5 * (CAPTURE_MS + 15_000), caseMs: CAPTURE_MS + 15_000 }, { file: 'test/skill-e2e-shared-libs-paths.test.ts', attemptMs: 3 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, { file: 'test/skill-e2e-ship-docsync.test.ts', attemptMs: 4 * CAPTURE_LONG_MS + 8 * CAPTURE_MS, caseMs: CAPTURE_LONG_MS }, - { file: 'test/skill-e2e-auq-matrix.test.ts', attemptMs: 6 * CAPTURE_MS, caseMs: CAPTURE_MS }, - { file: 'test/skill-e2e-plan-format.test.ts', attemptMs: 4 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 }, + { file: 'test/skill-e2e-auq-matrix.test.ts', attemptMs: 5 * CAPTURE_MS, caseMs: CAPTURE_MS }, + { file: 'test/skill-e2e-plan-format.test.ts', attemptMs: 2 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 }, { file: 'test/skill-e2e-auto-decide-preserved.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, { file: 'test/skill-e2e-plan-ceo-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, - { file: 'test/skill-e2e-plan-eng-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, { file: 'test/skill-e2e-plan-design-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, { file: 'test/skill-e2e-plan-devex-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS }, - { file: 'test/skill-e2e-plan-mode-no-op.test.ts', attemptMs: 6 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, + { file: 'test/skill-e2e-plan-mode-no-op.test.ts', attemptMs: 4 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, { file: 'test/skill-e2e-plan-ceo-mode-routing.test.ts', attemptMs: 2 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, - { file: 'test/skill-e2e-plan-eng-plan-mode.test.ts', attemptMs: 2 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS }, { file: 'test/skill-e2e-plan-prosons.test.ts', attemptMs: 4 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 }, // Gate: six 300s cases + one 610s case; periodic: two 900s + two 600s // (plan-eng-review-artifact runs from its own PTY file). diff --git a/test/helpers/periodic-exclude-data.ts b/test/helpers/periodic-exclude-data.ts index 98b637b84..70f6b6bee 100644 --- a/test/helpers/periodic-exclude-data.ts +++ b/test/helpers/periodic-exclude-data.ts @@ -23,6 +23,10 @@ export const PERIODIC_CI_EXCLUDE: Record = { "test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/plan-eng-resume.ts', 'test/fixtures/plan-eng-artifact-resume-37174266054.json', 'test/helpers/workflow-excerpt.ts', "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", 'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'], - 'plan-eng-review-artifact-full': [ 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/pty/**', 'test/helpers/pty/auq.ts', 'test/helpers/pty/binary.ts', 'test/helpers/pty/boundaries.ts', 'test/helpers/pty/classify.ts', 'test/helpers/pty/fake-session.ts', 'test/helpers/pty/judge.ts', 'test/helpers/pty/launch.ts', 'test/helpers/pty/plan-native.ts', 'test/helpers/pty/runners/counting.ts', 'test/helpers/pty/runners/floor.ts', 'test/helpers/pty/runners/observation.ts', 'test/helpers/pty/screen.ts', 'test/helpers/pty/session.ts', 'test/helpers/skill-census.ts', - - 'scripts/resolvers/learnings.ts', - - "test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact-full.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/workflow-excerpt.ts', - "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", - 'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'], 'plan-review-report': [ 'test/helpers/office-hours-attempt.ts', @@ -231,28 +224,6 @@ export const E2E_TOUCHFILES: Record = { "test/fixtures/eng-option-b-scope-al.json", 'test/fixtures/pty-companion-cli.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/tasks-section.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'bin/gstack-context-recovery'], - 'plan-eng-review-plan-mode': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', - - 'test/fixtures/auto-decide-recommendation-361c.json', - - 'test/fixtures/auto-decide-target-361c.json', - - 'test/fixtures/autoplan-public-narration-ad.json', - 'test/helpers/plan-count-transcript.ts', 'test/fixtures/plan-count-cross-cwd-ancestry-0bcd.json', - 'scripts/resolvers/learnings.ts', - "test/fixtures/plan-scope-target-aw.json", - - - "test/fixtures/plan-scope-recovery-av.json", - 'test/fixtures/auto-decide-saved-ai.json', 'test/fixtures/auto-decide-retry-ai.json','bin/gstack-skill-start', 'bin/gstack-skill-end', 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/question-tuning.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/skill-e2e-plan-eng-plan-mode.test.ts', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt', 'test/helpers/plan-scope-selection.ts', 'test/fixtures/design-plan-scope-ag.json', 'test/fixtures/design-scope-selection-aj.json', 'test/helpers/native-auto-decide.ts', 'test/fixtures/auto-decide-current-declaration-6aef.json', 'test/fixtures/auto-decide-explanatory-mode-043a.json', 'test/fixtures/auto-decide-explanatory-mode-749df.json', 'test/fixtures/auto-decide-structured-77.json', 'test/helpers/auto-decision-state.ts', 'test/fixtures/auto-decide-state-cab3.json', 'bin/gstack-question-log', 'bin/gstack-question-preference', 'test/helpers/fake-plan-seed.ts', 'test/fixtures/native-auto-decide-ag.json', 'test/fixtures/eng-seeded-completion-ai.json', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/pty-screen.ts', 'test/fixtures/pty-screen/**', - - 'test/fixtures/design-scope-announcement-ao.json', - 'test/fixtures/design-scope-declaration-ak.json', - - "test/fixtures/eng-option-b-scope-al.json", - - "scripts/resolvers/preamble/generate-preamble-bash.ts", - 'test/fixtures/pty-companion-cli.ts', 'test/helpers/plan-seed-submission.ts', 'test/fixtures/plan-seed-cli.ts', 'test/helpers/owned-claude-transcript.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/testing.ts', 'test/helpers/plan-mode-evidence.ts', 'lib/redact-engine.ts', 'lib/redact-patterns.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'bin/gstack-context-recovery'], // PTY plan-mode smoke (whole file); the SDK plan-edit case below owns plan-design-review-plan-mode. 'plan-design-review-plan-mode-smoke': [ 'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', @@ -440,7 +411,6 @@ export const E2E_TOUCHFILES: Record = { 'carve-section-loading-land-and-deploy': ['land-and-deploy/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], 'carve-section-loading-plan-design-review': ['plan-design-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], 'carve-section-loading-plan-devex-review': ['plan-devex-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], - 'carve-section-loading-plan-eng-review': ['plan-eng-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'scripts/resolvers/learnings.ts', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'scripts/resolvers/testing.ts', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], 'carve-section-loading-qa': ['qa/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], 'carve-section-loading-retro': ['retro/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], 'carve-section-loading-review': ['review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'], @@ -457,22 +427,6 @@ export const E2E_TOUCHFILES: Record = { // review-phase AskUserQuestion). Uses runPlanSkillFloorCheck — minimal // "did agent fire ANY AUQ?" observer that exits early on first non-permission // numbered-option render. ~1-3 min typical wall time per test, ~$2-6 total. - 'plan-eng-finding-floor': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/pty-screen.ts', - 'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', 'test/fixtures/plan-create-prepublication-491.json', 'test/fixtures/plan-create-combined-permission-70b.json', 'test/fixtures/plan-floor-quote-70b.json', - - 'test/fixtures/plan-create-permission-361c.json', - 'test/helpers/plan-floor-review.ts', - - 'test/fixtures/plan-floor-routing-361c.json', - - 'test/fixtures/ceo-report-permission-fb10.json', 'test/fixtures/plan-floor-permission-fb10.json', 'test/helpers/plan-count-file-permission.ts', 'test/fixtures/plan-edit-cropped-permission-1579.json', 'test/fixtures/plan-count-permission-target-ad-v2.json', 'test/fixtures/design-crop-gutter-ap.json', 'test/fixtures/plan-count-crop-ak.json', 'test/fixtures/plan-count-long-edit-0bcd.json', 'test/fixtures/plan-count-cropped-wrap-6714.json', - 'scripts/resolvers/learnings.ts', - - "test/fixtures/plan-scope-recovery-av.json", 'bin/gstack-skill-start', 'bin/gstack-skill-end', 'plan-eng-review/**', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/fixtures/forcing-finding-seeds.ts', 'test/skill-e2e-plan-eng-finding-floor.test.ts', 'test/fixtures/eng-batching-t-calls.json', 'test/fixtures/eng-scope-y-calls.json', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt', 'test/fixtures/eng-binding-z-calls.json', 'test/fixtures/eng-binding-retry-z-calls.json', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-artifacts.ts', - "test/fixtures/eng-injected-export-aq.json", "test/fixtures/eng-library-hooks-aq.json", - - "scripts/resolvers/preamble/generate-preamble-bash.ts", - 'test/fixtures/pty-companion-cli.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/testing.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'bin/gstack-context-recovery'], 'plan-ceo-finding-floor': [ 'ETHOS.md','bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/pty-screen.ts', 'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', 'test/fixtures/plan-create-prepublication-491.json', 'test/fixtures/plan-create-combined-permission-70b.json', 'test/fixtures/plan-floor-quote-70b.json', @@ -601,20 +555,6 @@ export const E2E_TOUCHFILES: Record = { 'plan-ceo-review-format-approach': [ 'ETHOS.md', 'plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts', 'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/tasks-section.ts' ], - 'plan-eng-review-format-coverage': [ - 'scripts/resolvers/learnings.ts', - - "test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts', - "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", - 'scripts/resolvers/testing.ts', 'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts' - ], - 'plan-eng-review-format-kind': [ - 'scripts/resolvers/learnings.ts', - - "test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts', - "scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts", - 'scripts/resolvers/testing.ts', 'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts' - ], // v1.7.0.0 Pros/Cons format cadence + format + negative-escape evals. // Dependencies: same as format-mode + the 4 plan-review templates + overlay. @@ -1357,7 +1297,6 @@ export const E2E_TIERS: Record = { 'plan-ceo-review-expansion-energy': 'periodic', // Demoted from gate (2026-08 audit): Opus generator + subjective 2-axis >=4/5 LLM-judge threshold in the merge lane — the exact class siblings were demoted for (a +21-line preamble change once flipped the score). CLAUDE.md's own rule: Opus model test -> periodic. 'plan-eng-review': 'periodic', 'plan-eng-review-artifact': 'periodic', // Checkpoint: resumes at Test review from a real Scope Challenge ledger - 'plan-eng-review-artifact-full': 'marathon', // Full fresh /plan-eng-review through Test review (498 s pass; 595 s timeouts at 600 s before Test review) 'plan-eng-coverage-audit': 'gate', 'ship-coverage-value': 'gate', 'review-test-value': 'gate', @@ -1372,7 +1311,6 @@ export const E2E_TIERS: Record = { // the hardened ask-first gate + the collapsed-form detector lifted them from // always-failing to mostly-passing, but they are not deterministic gates. 'plan-ceo-review-plan-mode': 'gate', - 'plan-eng-review-plan-mode': 'periodic', 'plan-design-review-plan-mode': 'periodic', 'plan-design-review-plan-mode-smoke': 'periodic', 'plan-devex-review-plan-mode': 'gate', @@ -1409,7 +1347,6 @@ export const E2E_TIERS: Record = { 'carve-section-loading-land-and-deploy': 'periodic', 'carve-section-loading-plan-design-review': 'periodic', 'carve-section-loading-plan-devex-review': 'periodic', - 'carve-section-loading-plan-eng-review': 'periodic', 'carve-section-loading-qa': 'periodic', 'carve-section-loading-retro': 'periodic', 'carve-section-loading-review': 'periodic', @@ -1421,7 +1358,6 @@ export const E2E_TIERS: Record = { // run drives a full skill end-to-end (~25 min, ~$5/run). Sequential // execution during calibration; concurrent opt-in only after measured // comparison agrees (plan §D15). - 'plan-eng-finding-floor': 'periodic', // stochastic ask-first (see plan-mode-handshake note); periodic 'plan-ceo-finding-floor': 'gate', 'plan-design-finding-floor': 'periodic', // stochastic ask-first (see plan-mode-handshake note); periodic 'plan-devex-finding-floor': 'gate', @@ -1445,8 +1381,6 @@ export const E2E_TIERS: Record = { // AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark) 'plan-ceo-review-format-mode': 'periodic', 'plan-ceo-review-format-approach': 'periodic', - 'plan-eng-review-format-coverage': 'periodic', - 'plan-eng-review-format-kind': 'periodic', // Office-hours Phase 4 silent-auto-decide regression — periodic (Phase 4 // requires the agent to invent 2-3 architectures, more open-ended than the @@ -1798,11 +1732,9 @@ export const E2E_KINDS: Record = { 'plan-ceo-review-expansion-energy': 'behavior', 'plan-eng-review': 'rule', 'plan-eng-review-artifact': 'rule', - 'plan-eng-review-artifact-full': 'rule', 'plan-eng-coverage-audit': 'rule', 'plan-review-report': 'rule', 'plan-ceo-review-plan-mode': 'rule', - 'plan-eng-review-plan-mode': 'rule', 'plan-design-review-plan-mode': 'rule', 'plan-design-review-plan-mode-smoke': 'rule', 'ship-coverage-value': 'rule', @@ -1835,13 +1767,11 @@ export const E2E_KINDS: Record = { 'carve-section-loading-land-and-deploy': 'rule', 'carve-section-loading-plan-design-review': 'rule', 'carve-section-loading-plan-devex-review': 'rule', - 'carve-section-loading-plan-eng-review': 'rule', 'carve-section-loading-qa': 'rule', 'carve-section-loading-retro': 'rule', 'carve-section-loading-review': 'rule', 'carve-section-loading-setup-gbrain': 'rule', 'carve-section-loading-spec': 'rule', - 'plan-eng-finding-floor': 'rule', 'plan-ceo-finding-floor': 'rule', 'plan-design-finding-floor': 'rule', 'plan-devex-finding-floor': 'rule', @@ -1852,8 +1782,6 @@ export const E2E_KINDS: Record = { 'setup-gbrain-path4-local-pglite': 'rule', 'plan-ceo-review-format-mode': 'behavior', 'plan-ceo-review-format-approach': 'behavior', - 'plan-eng-review-format-coverage': 'behavior', - 'plan-eng-review-format-kind': 'behavior', 'office-hours-phase4-fork': 'behavior', 'llm-judge-recommendation': 'judge', 'plan-ceo-review-prosons-cadence': 'behavior', @@ -2037,10 +1965,6 @@ export const BEHAVIOR_WHY: Record = { "Mode-question wording (Completeness line vs kind note) is live formatting of one AskUserQuestion.", 'plan-ceo-review-format-approach': "Approach-menu Completeness wording is live formatting of one AskUserQuestion.", - 'plan-eng-review-format-coverage': - "Coverage-issue Completeness wording is live formatting of one AskUserQuestion.", - 'plan-eng-review-format-kind': - "Kind-note wording is live formatting of one AskUserQuestion.", 'plan-ceo-review-prosons-cadence': "Pros/Cons cadence on a hard-stop question is live formatting; either the escape or the full block is accepted.", 'plan-review-prosons-format': diff --git a/test/office-hours-attempt.test.ts b/test/office-hours-attempt.test.ts index c96bc8889..f52199b39 100644 --- a/test/office-hours-attempt.test.ts +++ b/test/office-hours-attempt.test.ts @@ -557,8 +557,6 @@ describe('session runner native CLI max-turns exit semantics', () => { const FORMAT_CASES = [ ['skill-e2e-plan-format.test.ts', 'plan-ceo-review-format-mode'], ['skill-e2e-plan-format.test.ts', 'plan-ceo-review-format-approach'], - ['skill-e2e-plan-format.test.ts', 'plan-eng-review-format-coverage'], - ['skill-e2e-plan-format.test.ts', 'plan-eng-review-format-kind'], ['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-format'], ['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-hardstop-neg'], ['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-neutral-neg'], diff --git a/test/paid-report-census.test.ts b/test/paid-report-census.test.ts index 9e5b9ff33..3d3f96c1e 100644 --- a/test/paid-report-census.test.ts +++ b/test/paid-report-census.test.ts @@ -5,7 +5,7 @@ * 2026-10-03 periodic and gate census artifacts (run 37151477069, commit * ac20ef1), whose report counted deselections as SKIPPED 82/282 and left * 72/24 testcases unattributed. Budget records follow the current declared - * policy (plan-mode-no-op: 3,720,000 ms since v1.91.18.0 added its devex member). + * policy (plan-mode-no-op: 2,520,000 ms since its plan-eng-review member retired, GST-1698). * The periodic fixture is re-shaped to the current case shards: qa-workflow runs * as its qa-fix-loop case shard (its three gate-tier siblings become sibling * deselections) and the review carve runs as a case of the consolidated diff --git a/test/paid-retry-supervision.test.ts b/test/paid-retry-supervision.test.ts index 09ddade6f..45812152f 100644 --- a/test/paid-retry-supervision.test.ts +++ b/test/paid-retry-supervision.test.ts @@ -22,16 +22,14 @@ const expectedWalls = { 'test/skill-e2e-shared-libs-paths.test.ts': 1_920_000, 'test/skill-e2e-ship-docsync.test.ts': 4_920_000, 'test/skill-e2e-auq-consistency.test.ts': 1_080_000, - 'test/skill-e2e-auq-matrix.test.ts': 1_920_000, - 'test/skill-e2e-plan-format.test.ts': 1_360_000, + 'test/skill-e2e-auq-matrix.test.ts': 1_620_000, + 'test/skill-e2e-plan-format.test.ts': 740_000, 'test/skill-e2e-auto-decide-preserved.test.ts': 1_020_000, 'test/skill-e2e-plan-ceo-finding-floor.test.ts': 1_020_000, - 'test/skill-e2e-plan-eng-finding-floor.test.ts': 1_020_000, 'test/skill-e2e-plan-design-finding-floor.test.ts': 1_020_000, 'test/skill-e2e-plan-devex-finding-floor.test.ts': 1_020_000, - 'test/skill-e2e-plan-mode-no-op.test.ts': 3_720_000, + 'test/skill-e2e-plan-mode-no-op.test.ts': 2_520_000, 'test/skill-e2e-plan-ceo-mode-routing.test.ts': 1_320_000, - 'test/skill-e2e-plan-eng-plan-mode.test.ts': 1_320_000, 'test/skill-e2e-plan-prosons.test.ts': 1_360_000, 'test/skill-e2e-plan.test.ts': 3_120_000, }; @@ -50,9 +48,9 @@ test('paid evals never retry: every paid file and registered row runs once', () } }); -test('registration covers exactly the sixteen demonstrated full-file retry gaps', () => { +test('registration covers exactly the fourteen demonstrated full-file retry gaps', () => { expect(Object.fromEntries(newBudgets.map(row => [row.file, row.shardMs]))).toEqual(expectedWalls); - expect(new Set(FILE_RETRY_BUDGETS.map(row => row.file)).size).toBe(18); + expect(new Set(FILE_RETRY_BUDGETS.map(row => row.file)).size).toBe(16); expect(STRICT_RETRY_CASE_BUDGETS.map(row => row.file)).toEqual([ ...FINDING_RETRY_BUDGETS.map(row => row.file), AUQ_CONSISTENCY_RETRY_BUDGET.file, ]); @@ -82,20 +80,19 @@ test('source allowances retain all captures, cases, and finalization grace', () expect(read('test/helpers/codex-eval.ts')).toContain('CODEX_EVAL_FINALIZE_MS = 2 * CODEX_DRAIN_GRACE_MS'); expect(read('test/helpers/codex-session-runner.ts')).toMatch(/CODEX_DRAIN_GRACE_MS\s*=\s*5_000/); expect(read('test/helpers/office-hours-attempt.ts')).toContain('OFFICE_HOURS_BUN_GRACE_MS = 10_000'); - for (const file of ['auto-decide-preserved', 'plan-ceo-finding-floor', 'plan-eng-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor']) { + for (const file of ['auto-decide-preserved', 'plan-ceo-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor']) { expect(timeoutExpressions(`test/skill-e2e-${file}.test.ts`)).toEqual(['PTY_MS']); if (file.endsWith('-finding-floor')) expect(read(`test/skill-e2e-${file}.test.ts`)).toContain('timeoutMs: CAPTURE_LONG_MS'); } const noop = read('test/skill-e2e-plan-mode-no-op.test.ts'); - expect(noop).toContain("['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'plan-devex-review'] as const"); - expect(noop).toContain("['plan-eng-review', 'plan-design-review'] as const"); + expect(noop).toContain("['plan-ceo-review', 'plan-design-review', 'plan-devex-review'] as const"); + expect(noop).toContain("['plan-design-review'] as const"); expect(timeoutExpressions('test/skill-e2e-plan-mode-no-op.test.ts')).toEqual(Array(2).fill('CAPTURE_LONG_MS')); expect(read('test/skill-e2e-plan-ceo-mode-routing.test.ts').match(/^\s*{ mode: '/gm)).toHaveLength(2); expect(timeoutExpressions('test/skill-e2e-plan-ceo-mode-routing.test.ts')).toEqual(['CAPTURE_LONG_MS']); - expect(timeoutExpressions('test/skill-e2e-plan-eng-plan-mode.test.ts')).toEqual(Array(2).fill('CAPTURE_LONG_MS')); expect(timeoutExpressions('test/skill-e2e-plan-prosons.test.ts')).toEqual(Array(4).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS')); - expect(timeoutExpressions('test/skill-e2e-plan-format.test.ts')).toEqual(Array(4).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS')); - expect(read('test/skill-e2e-auq-matrix.test.ts').match(/^ skill: '/gm)).toHaveLength(6); + expect(timeoutExpressions('test/skill-e2e-plan-format.test.ts')).toEqual(Array(2).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS')); + expect(read('test/skill-e2e-auq-matrix.test.ts').match(/^ skill: '/gm)).toHaveLength(5); expect(timeoutExpressions('test/skill-e2e-auq-matrix.test.ts')).toEqual(['CAPTURE_MS']); expect(timeoutExpressions('test/skill-e2e-plan.test.ts')).toEqual([ 'PTY_MS', 'PTY_MS', 'CAPTURE_LONG_MS', 'CAPTURE_LONG_MS', diff --git a/test/plan-floor-permission.test.ts b/test/plan-floor-permission.test.ts index 5249223f0..17d23c7e4 100644 --- a/test/plan-floor-permission.test.ts +++ b/test/plan-floor-permission.test.ts @@ -521,10 +521,10 @@ test('duplicate declarations and existing fixture files cannot be rewritten into expect(() => createPlanCountFixture(FORCING_FLOOR_CEO,{requestedPlanPath,files:{'gstack-test-plan-ceo-floor.md':'different source'}})).toThrow(); }); -test.each(['auq_observed','timeout','throw'])('all four actual floor registrations keep the declared actor and %s outcome', outcome => { +test.each(['auq_observed','timeout','throw'])('all three actual floor registrations keep the declared actor and %s outcome', outcome => { const dir = fs.mkdtempSync(path.join(os.tmpdir(),'floor-registration-free-')); const worker = path.join(dir,'registrations.test.ts'), facts = path.join(dir,'calls.jsonl'); - const files = ['ceo','eng','design','devex'].map(kind => path.join(ROOT,`test/skill-e2e-plan-${kind}-finding-floor.test.ts`)); + const files = ['ceo','design','devex'].map(kind => path.join(ROOT,`test/skill-e2e-plan-${kind}-finding-floor.test.ts`)); try { for (const file of files) expect(fs.statSync(file).isFile()).toBe(true); fs.writeFileSync(worker, ` @@ -557,7 +557,7 @@ ${files.map(file => `await import(${JSON.stringify(file)});`).join('\n')} const output = result.stdout.toString()+result.stderr.toString(); expect(result.exitCode,output).toBe(outcome === 'auq_observed' ? 0 : 1); expect(fs.readFileSync(facts,'utf8').trim().split('\n').map(line=>JSON.parse(line).kind).sort()) - .toEqual(['ceo','design','devex','eng']); + .toEqual(['ceo','design','devex']); if(outcome !== 'auq_observed') expect(output).toContain(outcome === 'throw' ? 'controlled runner failure' : 'floor test FAILED: outcome=timeout'); } finally {fs.rmSync(dir,{recursive:true,force:true});} },25_000); diff --git a/test/plan-mode-evidence.test.ts b/test/plan-mode-evidence.test.ts index 3976d1cd7..fdf3c634d 100644 --- a/test/plan-mode-evidence.test.ts +++ b/test/plan-mode-evidence.test.ts @@ -11,10 +11,10 @@ const SECRET = 'fixture-private-diagnostic-value'; // Import the actual paid registration in an isolated Bun process. Only model // execution is replaced; Bun assertions and the report-file validator are real. -for (const skill of ['design', 'eng'] as const) { +for (const skill of ['design'] as const) { const PAID_FILE = `test/skill-e2e-plan-${skill}-plan-mode.test.ts`; const skillName = `plan-${skill}-review`; - for (const scenario of ['success', 'missing-auto-select', 'scope-question', 'bad-report', 'bad-outcome', 'retry', 'unwritable', ...(skill === 'eng' ? ['missing-decisions'] : [])]) { + for (const scenario of ['success', 'missing-auto-select', 'scope-question', 'bad-report', 'bad-outcome', 'retry', 'unwritable']) { test(`${skill} plan-mode caller preserves its outcome and failure evidence: ${scenario}`, () => { const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'design-mode-evidence-')); const evalDir = path.join(dir, 'artifacts'); @@ -29,7 +29,7 @@ const runnerPath = ${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-run const { assertReportAtBottomIfPlanWritten, planFileHasDecisionsSection } = await import(runnerPath); const scenario = ${JSON.stringify(scenario)}; const report = ${JSON.stringify(path.join(dir, 'plan.md'))}; -fs.writeFileSync(report, '# Plan\\n' + (scenario === 'missing-decisions' ? '' : '## Decisions to confirm\\nPending.\\n') +fs.writeFileSync(report, '# Plan\\n## Decisions to confirm\\nPending.\\n' + '## GSTACK REVIEW REPORT\\nDone.\\n' + (scenario === 'bad-report' ? '## Unexpected trailing section\\n' : '')); let calls = 0; mock.module(runnerPath, () => ({ @@ -37,10 +37,10 @@ mock.module(runnerPath, () => ({ runPlanSkillObservation: async opts => { const call = ++calls; fs.appendFileSync(${JSON.stringify(events)}, JSON.stringify({ call, opts }) + '\\n'); - return { outcome: ['bad-report', 'missing-decisions'].includes(scenario) ? 'plan_ready' : scenario === 'bad-outcome' ? 'auto_decided' : 'asked', + return { outcome: scenario === 'bad-report' ? 'plan_ready' : scenario === 'bad-outcome' ? 'auto_decided' : 'asked', summary: 'Returned observation ' + call, elapsedMs: 42, evidence: 'Visible terminal tail ' + call + ': ' + process.env.DIAGNOSTIC_API_TOKEN, - ...(['bad-report', 'missing-decisions'].includes(scenario) ? { planFile: report } : {}), + ...(scenario === 'bad-report' ? { planFile: report } : {}), proseAUQEverObserved: true, waitingEverObserved: true, tokensObserved: { 'draft-plan': true, 'scope-announcement': false }, scopeGateQuestionObserved: scenario === 'scope-question', scopeGateAutoSelectObserved: !['missing-auto-select', 'retry', 'unwritable'].includes(scenario) }; @@ -68,14 +68,6 @@ await import(${JSON.stringify(path.join(ROOT, PAID_FILE))}); keys.push('initialPlanContent', 'trackTokens'); expect(opts.trackTokens.length).toBeGreaterThan(0); for (const token of opts.trackTokens) expect(opts.initialPlanContent).toContain(token); - if (skill === 'eng') { - keys.push('extraArgs', 'cwd', 'env'); - expect(opts.extraArgs).toEqual(['--disallowedTools', 'AskUserQuestion']); - // The seeded runner plan reviews its own fixture project, never this checkout. - expect(path.isAbsolute(opts.cwd)).toBe(true); - expect(path.relative(ROOT, opts.cwd).startsWith('..')).toBe(true); - expect(opts.env).toEqual({ DISABLE_AUTOUPDATER: '1', CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' }); - } } expect(Object.keys(opts).sort()).toEqual(keys.sort()); } @@ -113,7 +105,6 @@ await import(${JSON.stringify(path.join(ROOT, PAID_FILE))}); if (scenario === 'bad-report') expect(output).toContain('GSTACK REVIEW REPORT contract violation'); if (scenario === 'missing-auto-select') expect(output).toContain('scopeGateAutoSelectObserved'); if (scenario === 'scope-question') expect(output).toContain('scopeGateQuestionObserved'); - if (scenario === 'missing-decisions') expect(output).toContain('without a question'); } finally { fs.rmSync(dir, { recursive: true, force: true }); } }); } diff --git a/test/qa-supervision-selection.test.ts b/test/qa-supervision-selection.test.ts index 04bff0697..5da21c962 100644 --- a/test/qa-supervision-selection.test.ts +++ b/test/qa-supervision-selection.test.ts @@ -2,15 +2,15 @@ import { expect, test } from 'bun:test'; import { E2E_TOUCHFILES, selectTests } from './helpers/touchfiles'; const ptyIds = [ - 'plan-ceo-review-plan-mode', 'plan-eng-review-plan-mode', 'plan-design-review-plan-mode-smoke', + 'plan-ceo-review-plan-mode', 'plan-design-review-plan-mode-smoke', 'plan-devex-review-plan-mode', 'plan-mode-no-op', 'office-hours-auto-mode', - 'auto-decide-preserved', 'plan-ceo-mode-routing', 'plan-design-with-ui-scope', 'plan-eng-finding-floor', + 'auto-decide-preserved', 'plan-ceo-mode-routing', 'plan-design-with-ui-scope', // The two formerly keyless AUQ probes (W1b) import the same SDK capture helper as auq-format-gate, which reaches the PTY runner. 'auq-format-gate', 'auq-consistency', 'auq-verbose-vs-carved-ab', ...['benchmark', 'browse', 'codex', 'context-restore', 'design-consultation', 'design-html', 'design-shotgun', 'document-release', 'investigate', 'land-and-deploy', - 'plan-design-review', 'plan-devex-review', 'plan-eng-review', 'qa', 'retro', 'review', 'setup-gbrain', 'spec'].map(skill => `carve-section-loading-${skill}`), + 'plan-design-review', 'plan-devex-review', 'qa', 'retro', 'review', 'setup-gbrain', 'spec'].map(skill => `carve-section-loading-${skill}`), 'office-hours-section-loading', 'office-hours-design-draft', 'plan-ceo-section-loading', 'ship-section-loading', 'plan-ceo-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor', - 'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact', 'plan-eng-review-artifact-full', + 'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact', // The /autoplan guard's live cases drive the same PTY runner (Oct 7 wave). 'autoplan-guard-pty', 'autoplan-long-session', ].sort(); diff --git a/test/skill-e2e-auq-matrix.test.ts b/test/skill-e2e-auq-matrix.test.ts index ac9bd2ff2..ec0622bb1 100644 --- a/test/skill-e2e-auq-matrix.test.ts +++ b/test/skill-e2e-auq-matrix.test.ts @@ -70,11 +70,6 @@ interface MatrixSkill { } const MATRIX: MatrixSkill[] = [ - { - skill: 'plan-eng-review', - fixtures: { 'plan.md': FLAWED_PLAN }, - scenario: 'Read plan.md — that is the plan to review. It is a standalone plan document, not a codebase. Walk the review until the first AskUserQuestion (a per-issue finding or a scope decision).', - }, { skill: 'plan-design-review', fixtures: { 'plan.md': FLAWED_PLAN + '\n## UI\nA new pricing page with a comparison table, plan cards, and an upgrade modal.\n' }, diff --git a/test/skill-e2e-plan-ceo-finding-floor.test.ts b/test/skill-e2e-plan-ceo-finding-floor.test.ts index f0ff770c0..910332227 100644 --- a/test/skill-e2e-plan-ceo-finding-floor.test.ts +++ b/test/skill-e2e-plan-ceo-finding-floor.test.ts @@ -1,7 +1,9 @@ /** * /plan-ceo-review AskUserQuestion floor regression (gate, paid, real-PTY). * - * See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract. + * Contract: a seeded forcing finding must fire at least one AskUserQuestion + * before the review writes its plan or exits plan mode (the /plan-eng-review + * floor that first pinned it retired with that skill, GST-1698). */ import { test } from 'bun:test'; diff --git a/test/skill-e2e-plan-design-finding-floor.test.ts b/test/skill-e2e-plan-design-finding-floor.test.ts index a6aec58fa..f5c6326c6 100644 --- a/test/skill-e2e-plan-design-finding-floor.test.ts +++ b/test/skill-e2e-plan-design-finding-floor.test.ts @@ -1,7 +1,9 @@ /** * /plan-design-review AskUserQuestion floor regression (periodic, paid, real-PTY). * - * See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract. + * Contract: a seeded forcing finding must fire at least one AskUserQuestion + * before the review writes its plan or exits plan mode (the /plan-eng-review + * floor that first pinned it retired with that skill, GST-1698). */ import { test } from 'bun:test'; diff --git a/test/skill-e2e-plan-devex-finding-floor.test.ts b/test/skill-e2e-plan-devex-finding-floor.test.ts index f6ce96d45..53aae9b5d 100644 --- a/test/skill-e2e-plan-devex-finding-floor.test.ts +++ b/test/skill-e2e-plan-devex-finding-floor.test.ts @@ -1,7 +1,9 @@ /** * /plan-devex-review AskUserQuestion floor regression (gate, paid, real-PTY). * - * See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract. + * Contract: a seeded forcing finding must fire at least one AskUserQuestion + * before the review writes its plan or exits plan mode (the /plan-eng-review + * floor that first pinned it retired with that skill, GST-1698). */ import { test } from 'bun:test'; diff --git a/test/skill-e2e-plan-eng-artifact-full.test.ts b/test/skill-e2e-plan-eng-artifact-full.test.ts deleted file mode 100644 index dee392169..000000000 --- a/test/skill-e2e-plan-eng-artifact-full.test.ts +++ /dev/null @@ -1,71 +0,0 @@ -/** - * Marathon: a fresh /plan-eng-review, from startup through Test review, writes - * its QA test-plan artifact (paid, real-PTY, non-blocking). - * - * The periodic checkpoint (skill-e2e-plan-eng-artifact.test.ts) resumes after - * Scope Challenge and Sections 1-2; this case keeps the whole interactive path. - * Budget: PTY_LONG_MS. Measured on Claude Code 2.1.284: one fresh run passed at - * 498 s; census 37174266054 and a local run both spent 595 s on startup plus six - * one-at-a-time Scope Challenge questions without reaching Test review. - * - * Replaces the composing `plan-eng-review-artifact` case in skill-e2e-plan.test.ts. - * That case told a `claude -p` session to skip the preamble and questions and - * "go straight to the review"; in runs 37151477069, 37158847998 and 37162480720 - * (and one local CI-image run) the model wrote the whole review into one file - * without executing the workflow, so it measured composing, not the skill. A - * full interactive run does write the artifact (37158847998's batching PTY run - * wrote eng-review-test-plan-20261003-224617.md during Test review). - * - * This case drives the real interactive /plan-eng-review over the same - * dashboard plan and sources, answers each question with its first option, and - * stops once the artifact exists or the report completes. The requirement is - * unchanged: exactly one new QA test plan under the run's own state root, about - * the reviewed dashboard change. - */ - -import { expect, test } from 'bun:test'; -import * as fs from 'node:fs'; -import * as os from 'node:os'; -import * as path from 'node:path'; -import { PTY_LONG_MS } from './helpers/eval-budgets'; -import { DASHBOARD_PLAN, FIXTURE_PLAN_PATH, SOURCES } from './helpers/plan-eng-artifact-fixture'; -import { describeE2ETier } from './helpers/e2e-gate'; -import { engFirstReviewAUQ, engSetupAUQ, engStep0Boundary, hasCompletePlanReport, runPlanSkillCounting } from './helpers/claude-pty-runner'; - -const describeE2E = describeE2ETier('marathon'); - -describeE2E('/plan-eng-review QA test-plan artifact, full workflow (marathon)', () => { - test('a fresh interactive review reaches Test review and writes one QA test plan', async () => { - const startedAt = Date.now(); - const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-e2e-plan-eng-artifact-full-')); - const planPath = path.join(tmpDir, 'gstack-test-plan-eng-artifact.md'); - try { - const obs = await runPlanSkillCounting({ - skillName: 'plan-eng-review', - slashCommand: '/plan-eng-review', - followUpPrompt: DASHBOARD_PLAN.replaceAll(FIXTURE_PLAN_PATH, planPath), - fixtureFiles: SOURCES, - permissionPlanPath: planPath, - isLastStep0AUQ: engStep0Boundary, - isSetupAUQ: engSetupAUQ, - isFirstReviewAUQ: engFirstReviewAUQ, - // The artifact is the only verdict; stop once it exists or the report is complete. - isCollectionComplete: (_transcript, _fingerprints, testPlans) => - testPlans.length > 0 || hasCompletePlanReport(planPath, startedAt, Date.now()), - reviewCountCeiling: Infinity, - preconfiguredReviewActor: true, - timeoutMs: PTY_LONG_MS, - env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' }, - }); - const plans = obs.engTestPlans; - console.log(`plan-eng-review-artifact-full: outcome=${obs.outcome} elapsed=${obs.elapsedMs}ms review=${obs.reviewCount} test plans=${plans.map(plan => plan.file).join(', ') || 'none'}`); - if (!['plan_ready', 'completion_summary', 'collection_complete'].includes(obs.outcome)) { - throw new Error(`plan-eng-review-artifact-full: outcome=${obs.outcome} (${obs.summary})\n--- evidence ---\n${obs.evidence}`); - } - expect(plans, `expected one new eng-review test plan; outcome=${obs.outcome}`).toHaveLength(1); - expect(/dashboard|fetchStats|\/api\/stats/i.test(plans[0]!.content), 'the test plan covers the reviewed dashboard change').toBe(true); - } finally { - fs.rmSync(tmpDir, { recursive: true, force: true }); - } - }, PTY_LONG_MS + 30_000); -}); diff --git a/test/skill-e2e-plan-eng-artifact.test.ts b/test/skill-e2e-plan-eng-artifact.test.ts index 00e095d4d..921235334 100644 --- a/test/skill-e2e-plan-eng-artifact.test.ts +++ b/test/skill-e2e-plan-eng-artifact.test.ts @@ -19,8 +19,8 @@ * review, answering each question with its first option, and the case stops * once the artifact exists or the report completes. The requirement is * unchanged: exactly one new QA test plan under the run's own state root, about - * the reviewed dashboard change. The fresh start-to-finish path is the marathon - * case in skill-e2e-plan-eng-artifact-full.test.ts. + * the reviewed dashboard change. (The fresh start-to-finish marathon case + * retired with the standalone skill, GST-1698.) */ import { expect, test } from 'bun:test'; diff --git a/test/skill-e2e-plan-eng-finding-floor.test.ts b/test/skill-e2e-plan-eng-finding-floor.test.ts deleted file mode 100644 index 703e7bce0..000000000 --- a/test/skill-e2e-plan-eng-finding-floor.test.ts +++ /dev/null @@ -1,56 +0,0 @@ -/** - * /plan-eng-review AskUserQuestion floor regression (periodic, paid, real-PTY). - * - * Catches the May 2026 transcript bug where /plan-eng-review wrote a - * multi-section review plan to ~/.claude/plans/ and called ExitPlanMode - * without firing any AskUserQuestion. See - * `.context/attachments/pasted_text_2026-05-06_10-25-23.txt`. - * - * Uses runPlanSkillFloorCheck — a minimal "did the agent fire ANY AUQ?" - * observer that exits early on the first non-permission numbered-option - * render. See claude-pty-runner.ts for why this is separate from the - * runPlanSkillCounting harness used by periodic finding-count tests. - * - * Tier: periodic. Budget: 10 min (early exit on success ~30-90s typical). - * Cost: ~$0.50-$1.50 per run depending on early-exit timing. - */ - -import { test } from 'bun:test'; -import { CAPTURE_LONG_MS, PTY_MS } from './helpers/eval-budgets'; -import { describeE2ETier } from './helpers/e2e-gate'; -import { runPlanSkillFloorCheck } from './helpers/claude-pty-runner'; -import { FORCING_FLOOR_ENG } from './fixtures/forcing-finding-seeds'; - -const describeE2E = describeE2ETier('periodic'); - -describeE2E('/plan-eng-review AskUserQuestion floor (periodic)', () => { - test( - 'seeded forcing finding causes the agent to fire at least one AskUserQuestion', - async () => { - const obs = await runPlanSkillFloorCheck({ - skillName: 'plan-eng-review', - slashCommand: '/plan-eng-review', - followUpPrompt: FORCING_FLOOR_ENG, - requestedPlanPath: '/tmp/gstack-test-plan-eng-floor.md', - // LIVE-REPO CWD: PTY session needs the repo cwd — gstack skill - // registry + hermetic pre-trusted dir (hermetic-env trustedDirs). - cwd: process.cwd(), - timeoutMs: CAPTURE_LONG_MS, - env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' }, - }); - - if (obs.outcome !== 'auq_observed') { - throw new Error( - `floor test FAILED: outcome=${obs.outcome} elapsed=${obs.elapsedMs}ms\n` + - `summary: ${obs.summary}\n` + - `If outcome is plan_ready or completion_summary, this is the transcript-bug ` + - `regression — agent reached terminal without firing AskUserQuestion. See ` + - `.context/attachments/pasted_text_2026-05-06_10-25-23.txt.\n` + - `If outcome is timeout, agent may just be slow — re-run or increase budget.\n` + - `--- evidence (last 3KB) ---\n${obs.evidence}`, - ); - } - }, - PTY_MS, - ); -}); diff --git a/test/skill-e2e-plan-eng-plan-mode.test.ts b/test/skill-e2e-plan-eng-plan-mode.test.ts deleted file mode 100644 index 815c8929a..000000000 --- a/test/skill-e2e-plan-eng-plan-mode.test.ts +++ /dev/null @@ -1,183 +0,0 @@ -/** - * plan-eng-review plan-mode smoke (periodic, paid, real-PTY). - * - * See test/skill-e2e-plan-ceo-plan-mode.test.ts for the shared assertion - * contract. This file exercises the same contract against /plan-eng-review. - */ - -import { test, expect } from 'bun:test'; -import { spawnSync } from 'node:child_process'; -import * as fs from 'node:fs'; -import * as os from 'node:os'; -import * as path from 'node:path'; -import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets'; -import { describeE2ETier } from './helpers/e2e-gate'; -import { assertNoPlanFileDecisions, assertPlanModeWithEvidence, seededPlanTargeted } from './helpers/plan-mode-evidence'; -import { - runPlanSkillObservation, - planFileHasDecisionsSection, - assertReportAtBottomIfPlanWritten, -} from './helpers/claude-pty-runner'; - -const describeE2E = describeE2ETier('periodic'); - -// SEED_PLAN_FORCING_FINDINGS: 8+ files + custom-vs-builtin smell forces the -// Step 0 complexity check to trigger. Passed via runPlanSkillObservation's -// initialPlanContent (D3-B) so the spawned `claude` actually sees it. -const SEED_PLAN_FORCING_FINDINGS = ` -# Parallelize unit tests - -## Plan -Build a custom test runner: scripts/test-parallel.ts, scripts/test-shard-impl.ts, -scripts/test-merge-results.ts, scripts/test-progress.ts, scripts/test-watch.ts, -scripts/test-coverage.ts, scripts/test-cli.ts, scripts/test-config.ts. - -Add new TestRunner class, new ShardManager class, new ResultMerger class. - -Ignore Bun's native --shard flag because we want full control. - -## Files -- scripts/test-parallel.ts (new) -- scripts/test-shard-impl.ts (new) -- scripts/test-merge-results.ts (new) -- scripts/test-progress.ts (new) -- scripts/test-watch.ts (new) -- scripts/test-coverage.ts (new) -- scripts/test-cli.ts (new) -- scripts/test-config.ts (new) -- package.json (add scripts) - -## Tests -None planned — will add later. -`; - -/** - * The seeded plan proposes a custom test runner, so its Step 0 "what already - * exists" pass reads the working tree's test setup. Run it in a small project - * of its own, never the gstack checkout: there the agent explored gstack's own - * sharded runner, and the review's length tracked unrelated runner edits - * (a 300 s timeout in census 37228573062 after a runner refactor). - */ -function createRunnerPlanFixture(): { cwd: string; cleanup(): void } { - const cwd = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-plan-eng-smoke-'))); - const cleanup = () => fs.rmSync(cwd, { recursive: true, force: true }); - const files: Record = { - 'package.json': JSON.stringify({ name: 'tiny-math', private: true, type: 'module', scripts: { test: 'bun test' } }, null, 2) + '\n', - 'README.md': '# tiny-math\n\nA small Bun library. `bun test` runs the unit tests in test/.\n', - 'src/sum.ts': 'export function sum(values: number[]): number {\n return values.reduce((total, value) => total + value, 0);\n}\n', - 'test/sum.test.ts': "import { expect, test } from 'bun:test';\nimport { sum } from '../src/sum';\n\ntest('sums values', () => {\n expect(sum([1, 2, 3])).toBe(6);\n});\n", - }; - try { - for (const [name, content] of Object.entries(files)) { - fs.mkdirSync(path.dirname(path.join(cwd, name)), { recursive: true }); - fs.writeFileSync(path.join(cwd, name), content); - } - for (const args of [ - ['init', '-b', 'main'], - ['add', '--', ...Object.keys(files)], - ['-c', 'user.name=Plan Eng Fixture', '-c', 'user.email=plan-eng@example.test', '-c', 'commit.gpgsign=false', - 'commit', '--no-verify', '-m', 'Seed tiny-math'], - ['update-ref', 'refs/remotes/origin/main', 'HEAD'], - ]) { - const result = spawnSync('git', args, { cwd, encoding: 'utf8', timeout: 10_000 }); - if (result.error || result.status !== 0) throw new Error(`Could not initialize the plan-eng fixture: ${result.error?.message ?? result.stderr}`); - } - return { cwd, cleanup }; - } catch (error) { - cleanup(); - throw error; - } -} - -// Seed-only names; seeing one after the slash command shows the review read the seeded plan. -const SEED_PLAN_TOKENS = ['ShardManager', 'ResultMerger', 'test-shard-impl']; - -describeE2E('plan-eng-review plan-mode smoke (periodic)', () => { - test('reaches a terminal outcome (asked or plan_ready) without silent writes', async () => { - const obs = await runPlanSkillObservation({ - skillName: 'plan-eng-review', - inPlanMode: true, - timeoutMs: CAPTURE_MS, - }); - - assertPlanModeWithEvidence('plan-eng-review', 'reaches a terminal outcome (asked or plan_ready) without silent writes', obs, () => { - if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') { - throw new Error( - `plan-eng-review plan-mode smoke FAILED: outcome=${obs.outcome}\n` + - `summary: ${obs.summary}\n` + - `elapsed: ${obs.elapsedMs}ms\n` + - `--- evidence (last 2KB visible) ---\n${obs.evidence}`, - ); - } - expect(['asked', 'plan_ready']).toContain(obs.outcome); - assertReportAtBottomIfPlanWritten(obs); - assertNoPlanFileDecisions(obs, planFileHasDecisionsSection); - }); - }, CAPTURE_LONG_MS); - - // D3-B / D4-B: when a plan with guaranteed-finding-triggering complexity - // is seeded, the skill must ask (mcp AskUserQuestion or the prose fallback) - // before writing findings to the plan. The wrote_findings_before_asking - // outcome catches the transcript bug where the model writes findings to the - // plan before any question renders. A plan-file ## Decisions section is not - // a substitute for asking. - test('STOP gate fires when seeded plan forces Step 0 findings', async () => { - const fixture = createRunnerPlanFixture(); - let obs: Awaited>; - try { - obs = await runPlanSkillObservation({ - skillName: 'plan-eng-review', - inPlanMode: true, - cwd: fixture.cwd, - // As in auto-decide-preserved's standalone fixture: a first launch in a - // fresh folder can paint a product notice under the composer, which - // the plan-seed composer check (one footer row) never accepts. - env: { DISABLE_AUTOUPDATER: '1', CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' }, - initialPlanContent: SEED_PLAN_FORCING_FINDINGS, - // Force the Conductor-style path: native AUQ disallowed → the model - // must use mcp__*__AskUserQuestion or render the prose fallback - // (both observed as outcome='asked'). - extraArgs: ['--disallowedTools', 'AskUserQuestion'], - trackTokens: SEED_PLAN_TOKENS, - timeoutMs: CAPTURE_MS, - }); - } finally { - fixture.cleanup(); - } - - assertPlanModeWithEvidence('plan-eng-review', 'STOP gate fires when seeded plan forces Step 0 findings', obs, () => { - if ( - obs.outcome === 'wrote_findings_before_asking' || - obs.outcome === 'auto_decided' || - obs.outcome === 'silent_write' || - obs.outcome === 'exited' || - obs.outcome === 'timeout' - ) { - throw new Error( - `STOP-gate regression: outcome=${obs.outcome}\nsummary: ${obs.summary}\n` + - `elapsed: ${obs.elapsedMs}ms\n` + - `--- evidence (last 2KB) ---\n${obs.evidence}`, - ); - } - - assertReportAtBottomIfPlanWritten(obs); - assertNoPlanFileDecisions(obs, planFileHasDecisionsSection); - if (obs.outcome !== 'asked') { - throw new Error( - `STOP-gate regression: outcome=${obs.outcome} — the seeded findings reached ` + - `${obs.planFile ?? 'plan_ready'} without a question.\n` + - `--- evidence (last 2KB) ---\n${obs.evidence}`, - ); - } - - // Plan-mode scope-gate bypass: with a seeded plan in plan mode, the gate - // must NOT render its "What should I review?" menu, and the review must - // target the seeded plan. The announcement's exact wording is not graded. - // Unseeded test 1 keeps its lenient contract: with no plan drafted, the - // "ask as normal" fallback legitimately renders the question. - expect(obs.scopeGateQuestionObserved ?? false).toBe(false); - console.log(`[plan-eng plan-mode] scope announcement observed: ${obs.scopeGateAutoSelectObserved ?? false}; seed tokens: ${JSON.stringify(obs.tokensObserved ?? {})}`); - expect(seededPlanTargeted(obs, SEED_PLAN_TOKENS), 'seeded plan was not targeted (scopeGateAutoSelectObserved or seed-only plan tokens)').toBe(true); - }); - }, CAPTURE_LONG_MS); -}); diff --git a/test/skill-e2e-plan-format.test.ts b/test/skill-e2e-plan-format.test.ts index 143c1200e..48d4e0f35 100644 --- a/test/skill-e2e-plan-format.test.ts +++ b/test/skill-e2e-plan-format.test.ts @@ -1,5 +1,6 @@ /** - * AskUserQuestion format regression test for /plan-ceo-review and /plan-eng-review. + * AskUserQuestion format regression test for /plan-ceo-review (the /plan-eng-review + * cases retired with that skill, GST-1698). * * Context: a user on Opus 4.7 reported the RECOMMENDATION line and the * `Completeness: N/10` per-option score stopped appearing on AskUserQuestion @@ -82,7 +83,7 @@ We're building a new user dashboard that shows recent activity, notifications, a - Cache: Redis for dashboard aggregates `; -function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review' | 'plan-eng-review'): string { +function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review'): string { const planDir = fs.mkdtempSync(path.join(os.tmpdir(), tmpPrefix)); const run = (cmd: string, args: string[]) => spawnSync(cmd, args, { cwd: planDir, stdio: 'pipe', timeout: 5000 }); @@ -107,12 +108,6 @@ function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review' | 'plan-en return planDir; } -// The capture instruction passed to every case except the mode and approach menus (D2). -// Tells the agent to dump AskUserQuestion content to a file instead of calling a tool. -function captureInstruction(outFile: string): string { - return `Write the verbatim text of every AskUserQuestion you would have made to ${outFile} (one question per session, full text including options and recommendation line). Do NOT call any tool to ask the user. Do NOT paraphrase — include the exact prose you would have shown. This is a format-capture test, not an interactive session.`; -} - /** * D2: both approach-menu refusals in census 37195203538 (Fable 5.1, * `reasoning_extraction`, "duplicating model outputs") were anchored on the @@ -259,143 +254,6 @@ After writing the file, stop. Do not continue the review.`, }, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS); }); -// --- Case 3: plan-eng-review coverage-differentiated per-issue AskUserQuestion --- - -describeIfSelected('Plan Format — Eng Coverage Issue', ['plan-eng-review-format-coverage'], () => { - let planDir: string; - let outFile: string; - - beforeAll(() => { - planDir = setupPlanDir('skill-e2e-plan-format-eng-cov-', 'plan-eng-review'); - outFile = path.join(planDir, 'ask-capture.md'); - }); - - afterAll(() => { - try { fs.rmSync(planDir, { recursive: true, force: true }); } catch {} - }); - - testConcurrentIfSelected('plan-eng-review-format-coverage', async () => { - const judgeMetadata: Pick = {}; - await runRecordedOfficeHoursAttempt({ - collector: evalCollector, name: '/plan-eng-review-format-coverage', suite: 'Plan Format — Eng Coverage Issue', - model: resolveEvalModel('capture'), budgetMs: CAPTURE_MS, - judgeMetadata, - run: signal => runSkillTest({ - signal, - prompt: `Read plan-eng-review/SKILL.md for the review workflow. - -Read plan.md — that's the plan to review. This is a standalone plan document, not a codebase — skip any codebase exploration steps. - -During your review (Section 3 Test Review is the natural place), generate ONE AskUserQuestion about test coverage depth where the options are clearly coverage-differentiated. For example: - A) Full coverage: happy path + edge cases + error paths (Completeness 10/10) - B) Happy path only (Completeness 7/10) - C) Smoke test (Completeness 3/10) - -${captureInstruction(outFile)} - -After writing the file with that ONE question, stop. Do not continue the review.`, - workingDirectory: planDir, - maxTurns: 10, - timeout: CAPTURE_MS, - testName: 'plan-eng-review-format-coverage', - runId, - model: resolveEvalModel('capture'), - }), - validate: async (result, signal) => { - logCost('/plan-eng-review format (coverage)', result); - expect(['success', 'error_max_turns']).toContain(result.exitReason); - - expect(fs.existsSync(outFile)).toBe(true); - const captured = fs.readFileSync(outFile, 'utf-8'); - expect(captured.length).toBeGreaterThan(100); - - // Coverage-differentiated: Completeness: N/10 required. Recommendation - // presence checked by the judge. - expect(captured).toMatch(COMPLETENESS_RE); - - await assertRecommendationQuality({ - captured, - signal, - record: metadata => Object.assign(judgeMetadata, metadata), - evalCollector, - evalId: '/plan-eng-review-format-coverage', - evalTitle: 'Plan Format — Eng Coverage Issue', - result, - passed: ['success', 'error_max_turns'].includes(result.exitReason), - }); - }, - }); - }, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS); -}); - -// --- Case 4: plan-eng-review kind-differentiated per-issue AskUserQuestion --- - -describeIfSelected('Plan Format — Eng Kind Issue', ['plan-eng-review-format-kind'], () => { - let planDir: string; - let outFile: string; - - beforeAll(() => { - planDir = setupPlanDir('skill-e2e-plan-format-eng-kind-', 'plan-eng-review'); - outFile = path.join(planDir, 'ask-capture.md'); - }); - - afterAll(() => { - try { fs.rmSync(planDir, { recursive: true, force: true }); } catch {} - }); - - testConcurrentIfSelected('plan-eng-review-format-kind', async () => { - const judgeMetadata: Pick = {}; - await runRecordedOfficeHoursAttempt({ - collector: evalCollector, name: '/plan-eng-review-format-kind', suite: 'Plan Format — Eng Kind Issue', - model: resolveEvalModel('capture'), budgetMs: CAPTURE_MS, - judgeMetadata, - run: signal => runSkillTest({ - signal, - prompt: `Read plan-eng-review/SKILL.md for the review workflow. - -Read plan.md — that's the plan to review. This is a standalone plan document, not a codebase — skip any codebase exploration steps. - -During your review (Section 1 Architecture), generate ONE AskUserQuestion about an architectural choice where the options differ in kind, not in coverage. For example, "should we use Redis or Postgres for the cache layer?" — the options are different kinds of systems with different tradeoffs, not more-or-less-complete versions of the same thing. - -${captureInstruction(outFile)} - -After writing the file with that ONE question, stop. Do not continue the review.`, - workingDirectory: planDir, - maxTurns: 10, - timeout: CAPTURE_MS, - testName: 'plan-eng-review-format-kind', - runId, - model: resolveEvalModel('capture'), - }), - validate: async (result, signal) => { - logCost('/plan-eng-review format (kind)', result); - expect(['success', 'error_max_turns']).toContain(result.exitReason); - - expect(fs.existsSync(outFile)).toBe(true); - const captured = fs.readFileSync(outFile, 'utf-8'); - expect(captured.length).toBeGreaterThan(100); - - // Kind-differentiated: Completeness: N/10 must NOT appear, and the brief - // says the options are not comparable on coverage (any wording). - // Recommendation presence checked by the judge. - expect(captured).not.toMatch(COMPLETENESS_RE); - expect(captured).toMatch(KIND_NOTE_RE); - - await assertRecommendationQuality({ - captured, - signal, - record: metadata => Object.assign(judgeMetadata, metadata), - evalCollector, - evalId: '/plan-eng-review-format-kind', - evalTitle: 'Plan Format — Eng Kind Issue', - result, - passed: ['success', 'error_max_turns'].includes(result.exitReason), - }); - }, - }); - }, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS); -}); - afterAll(async () => { await finalizeEvalCollector(evalCollector); }); diff --git a/test/skill-e2e-plan-mode-no-op.test.ts b/test/skill-e2e-plan-mode-no-op.test.ts index e3b7e427d..11fa3ea79 100644 --- a/test/skill-e2e-plan-mode-no-op.test.ts +++ b/test/skill-e2e-plan-mode-no-op.test.ts @@ -16,11 +16,12 @@ * * Coverage grew with the scope-gate bypass (plan-mode auto-select B): * - plan-ceo-review: original preamble-misfire regression. - * - plan-eng-review / plan-design-review: the bypass must NOT fire outside + * - plan-design-review: the bypass must NOT fire outside * plan mode (scopeGateAutoSelectObserved stays false), and when the run * ends in 'asked', the question that fired must be the scope gate itself * (outside plan mode with no named target, the gate is the FIRST - * question by contract). + * question by contract). (plan-eng-review had the same two checks; they + * retired with that skill, GST-1698.) * - plan-devex-review: no no-target scope gate, so only the terminal * outcome and the absent plan-mode reminder are asserted. It reviews a * pasted developer-facing plan: without one it reviews the checkout's @@ -90,14 +91,14 @@ script against the task tracker. Prints to stdout; --out writes a file. `; describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => { - for (const skillName of ['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'plan-devex-review'] as const) { - const hasScopeGate = skillName === 'plan-eng-review' || skillName === 'plan-design-review'; + for (const skillName of ['plan-ceo-review', 'plan-design-review', 'plan-devex-review'] as const) { + const hasScopeGate = skillName === 'plan-design-review'; test(`${skillName} reaches a terminal outcome outside plan mode`, async () => { const obs = await runPlanSkillObservation({ skillName, inPlanMode: false, timeoutMs: CAPTURE_MS, - // eng/design: force the prose-fallback path. The unconditional + // design: force the prose-fallback path. The unconditional // gate-must-ask assert below pins the render shape the detector // anchors on, and only the --disallowedTools prose fallback makes // that shape CONTRACTUAL ("use exactly this shape" in the template); @@ -164,7 +165,7 @@ describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => { // review output), proving the target was used rather than the question // merely skipped. Also the over-trigger guard for the tightened // "explicit-only" exception wording. - for (const skillName of ['plan-eng-review', 'plan-design-review'] as const) { + for (const skillName of ['plan-design-review'] as const) { test(`${skillName} skips the scope gate for an explicitly-pasted target`, async () => { const obs = await runPlanSkillObservation({ skillName, diff --git a/test/touchfiles.test.ts b/test/touchfiles.test.ts index 8c1102bac..bf30cfd10 100644 --- a/test/touchfiles.test.ts +++ b/test/touchfiles.test.ts @@ -147,8 +147,7 @@ describe('selectTests', () => { const actual = selectTests(['scripts/resolvers/testing.ts'], E2E_TOUCHFILES); expect(actual.reason).toBe('diff'); expect(actual.selected.sort()).toEqual(expected); - for (const id of ['plan-eng-multi-finding-batching', - 'plan-eng-review-format-coverage', 'ship-section-loading']) { + for (const id of ['plan-eng-multi-finding-batching', 'ship-section-loading']) { expect(actual.selected).toContain(id); expect(E2E_TIERS[id]).toBe('periodic'); } @@ -633,7 +632,10 @@ describe('derived touchfile closure', () => { }; const maps = [['E2E_TOUCHFILES', E2E_TOUCHFILES], ['LLM_JUDGE_TOUCHFILES', LLM_JUDGE_TOUCHFILES]] as const; /** Paid files no key selects; they run only by tier or census. */ - const KEYLESS_PAID: Record = {}; + const KEYLESS_PAID: Record = { + // Template-literal ids (auq-matrix-${skill}) and no E2E map key; it runs on every periodic sweep. It only looked keyed while its plan-eng-review row quoted that SDK key (row retired with the skill, GST-1698). + 'test/skill-e2e-auq-matrix.test.ts': 'periodic self-gated matrix with template-literal ids', + }; test('no free test file is a touchfile', () => { const listed = maps.flatMap(([name, map]) => Object.entries(map).flatMap(([key, deps]) =>