CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-11-08 01:38:10 +01:00
Code Issues Packages Projects Releases Wiki Activity

test: harness targets skip retired skills; record the standalone /plan-eng-review paid cases' fates (GST-1698) (#4111)

Browse Source 
* test: harness targets skip retired skills; record each standalone /plan-eng-review paid case's fate (GST-1698)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: mark retired /plan-eng-review cases in the detector inventory and TODOS (GST-1698)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-11-07 06:02:49 +00:00
1 parent 3cf4bc9181
commit 94392a61ec
39 files changed
+121 -629
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

TODOS.md
+6 -4
View File
@@ -4,7 +4,7 @@
### P2: `/plan-eng-review` retirement follow-ups (filed 2026-11-06, GST-1698 /ship review of #4108)
- **Stop the dots and generic harness targets from registering the retired skill.** `sourceSkills()` in `lib/harness-register.ts` lists every committed Claude-render skill and filters out only `gstack`, so `gstack-harness register --target dots|generic` still lists `plan-eng-review` in its router and gives it a folder in `all` mode. Filter `RETIRED_SKILLS` there (the Claude-render /autoplan reads Phase 3 from `~/.claude/skills/gstack/plan-eng-review/SKILL.md`, not from the harness payload) and add a case to `test/retired-skills.test.ts`. Raised by the outside voice (labelled blocking) and the native adversarial pass (non-blocking); below the blocking floor because nothing fails and the skill still works. **Effort:** S. **Priority:** P2.
- **Delete the retired `plan-eng-multi-finding-batching` case and its counter subsystem.** The case checks one interactive question per eng finding; /autoplan Phase 3 auto-decides findings, so no shipped entry reaches it. The follow-up to #4108 took it out of the weekly lane through `PERIODIC_CI_EXCLUDE` instead of deleting it, because about 20 free tests and fixtures depend on its counter (`createEngBatchingIssueCounter` in `test/helpers/eng-seeded-coverage.ts`, `test/eng-batching-saved-ledger.test.ts`, `test/detector-corpus-plan-eng-multi-finding-batching.test.ts`, the `eng-batching-*` fixtures). Retirement card: detects=an eng review that batches findings into one plan write; non_test_callers=0; stronger_proof=the ceo/design/devex finding floors keep the shared one-question-per-finding contract; unlocks=the counter helpers, their free tests and fixtures. Delete each helper only after checking it has no other caller (plan-eng-review-artifact shares `engFirstReviewAUQ`). **Effort:** M. **Priority:** P3.
- **Point the external-host methodology file's preamble Read at `gstack-autoplan`.** Every external host's `gstack-autoplan/plan-eng-review.md` says to Read `sections/preamble.md` relative to the installed `gstack-plan-eng-review` directory, which no longer exists. Normally the matching preamble id skips the reread, but after compaction late in a run the agent follows a missing path. Fixing it changes Phase 3 bytes, so it needs its own byte-pin update. **Effort:** S. **Priority:** P3.
- **Redirect `gstack-doctor --for plan-eng-review`.** It now exits 2 with "unknown skill"; map the retired name to `/autoplan`'s requirements and print a one-line redirect, so saved lane scripts keep working. **Effort:** S. **Priority:** P3.
- **Decide whether the QM edition keeps `gstack-plan`.** `hosts/qm.ts` and `qm/edition.json` still build `gstack-plan` from the `plan-eng-review` sources under another name, so the retired-name guards don't see it. Product call for Garry. **Effort:** S. **Priority:** P3.
@@ -545,8 +545,9 @@ Deferred from the approved severe fix wave (docs/designs/SEVERE_FIX_WAVE_2026_10
a PTY. A fresh run needs more than its 600 s budget (498 s pass; 595 s
timeouts in census 37174266054 and a local run), so the periodic case
resumes at Test review from that census run's real Scope Challenge ledger
(pass, 364 s) and the fresh run is the marathon case
plan-eng-review-artifact-full.
(pass, 364 s) and the fresh run was the marathon case
plan-eng-review-artifact-full (retired with standalone /plan-eng-review,
GST-1698).
- **Gate-outcome telemetry** — count `not_run`/`unavailable` outcomes per
reason code from `lib/gate-outcomes.ts` so the most common gaps get fixed
first. **Effort:** S. **Priority:** P3.
@@ -2104,7 +2105,8 @@ alone). Recommend (b) short-term + (a) properly sequenced with the codemod.
**What:** `evals-periodic.yml` runs an explicit 9-file matrix; the four tests
demoted to `periodic` in v1.62.0.0 (`skill-e2e-plan-eng-plan-mode`,
`skill-e2e-plan-design-plan-mode`, `skill-e2e-plan-eng-finding-floor`,
`skill-e2e-plan-design-finding-floor`) are not in it, so they currently run
`skill-e2e-plan-design-finding-floor`; the two plan-eng files were deleted with
standalone /plan-eng-review in GST-1698) are not in it, so they currently run
only locally/manually (`bun run test:periodic` or `eval:bg:periodic`). Wiring
them needs a PTY-capable periodic job: the container skill-registration setup
from evals.yml's `e2e-pty-plan-smoke` job (real-file SKILL.md copies for the
changelog.d/capy-gst-1698-retired-followups.md
+1
View File
@@ -0,0 +1 @@
- **The dots and generic harness targets no longer register the retired /plan-eng-review.** `gstack-harness register --target dots|generic` listed every committed skill except `gstack`, so after #4108 it still put `plan-eng-review` in its router and, in `all` mode, gave it a folder. It now skips every skill in `RETIRED_SKILLS`; `plan-eng-review/` stays in the source tree as /autoplan's Phase 3 methodology. The paid cases written for the standalone skill each have a recorded fate in `docs/TEST_PORTFOLIO.md`: four that check Phase 3 methodology or its required outputs stay (`plan-eng-review`, `plan-eng-coverage-audit`, `plan-eng-review-artifact`, `journey-plan-eng`), six that checked only the standalone interactive entry are deleted along with their rows in the AUQ matrix and the plan-mode no-op file, and `plan-eng-multi-finding-batching` leaves the weekly lane through `PERIODIC_CI_EXCLUDE` until its counter helpers are removed. (GST-1698)
docs/TEST_PORTFOLIO.md
+23
View File
@@ -286,6 +286,29 @@ or out-of-tier placeholders. A retained passing judge is reusable only when its
complete request, rubric, parameters and relevant dependencies match; native
behavioral acceptance follows its separate freshness contract.
## Paid cases of the retired standalone /plan-eng-review (GST-1698)
The skill's methodology still ships: `plan-eng-review/` is /autoplan's Phase 3
methodology source, bound into each run by the methodology snapshot. Cases that
check that methodology or Phase 3's required outputs stay. Cases that checked
the standalone interactive entry, which no longer ships, are retired. None was
quarantined: `CASE_QUARANTINE` holds flaky cases that keep running, and these
cases are not flaky, their entry point is gone.
| Case | Fate | Why, and what still proves the contract |
| --- | --- | --- |
| `plan-eng-review` (SDK) | kept | Runs the methodology source over a plan, the bytes Phase 3 loads. |
| `plan-eng-coverage-audit` (gate) | kept | Phase 3 must produce the codepath-to-test diagram; the case reads the same methodology and sections. |
| `plan-eng-review-artifact` (PTY checkpoint) | kept | Phase 3 must leave a test plan on disk; the case's Test review writes it from the same section. |
| `journey-plan-eng` | kept, retargeted | Architecture-review requests now route to /autoplan (#4108). |
| `plan-eng-multi-finding-batching` | retired, excluded | Checks one interactive question per finding; Phase 3 auto-decides findings. `PERIODIC_CI_EXCLUDE` stops its weekly run; deleting it with its counter helpers is a TODOS.md item. |
| `plan-eng-review-plan-mode` | retired, deleted | Standalone plan-mode handshake; the ceo, design and devex plan-mode cases and `plan-mode-no-op` keep the shared contract. |
| `plan-eng-finding-floor` | retired, deleted | Interactive first-finding question; the ceo, design and devex floors keep the shared floor. |
| `plan-eng-review-artifact-full` | retired, deleted | Fresh standalone marathon run; the checkpoint case keeps the artifact requirement. |
| `plan-eng-review-format-coverage`, `plan-eng-review-format-kind` | retired, deleted | Format of standalone questions; the ceo format cases keep the shared question format. |
| `carve-section-loading-plan-eng-review` | retired, deleted | Model-driven section loading on standalone entry; Phase 3 gets the methodology through the snapshot instead. |
| `auq-matrix-plan-eng-review`, the eng rows of `plan-mode-no-op` | retired rows | Standalone entry; the other skills' rows keep both files' contracts. |
## Remaining work, not claimed savings
Report final acceptance with the delivery revision and its actual case inventory.
docs/evals/detector-inventory.md
+7 -7
View File
@@ -31,7 +31,7 @@ Dispositions:
| F3 | test/helpers/pty/runners/floor.ts:497 | screen Write render / plan-ready text → fail | Write/ExitPlanMode tool_use in run.transcript | no | (b) |
| A1 | test/helpers/auq-sdk-capture.ts:30-43 `scoreAuqFormat` | question has ELI10:, Recommendation:, Pros / cons, ✅, ❌, Net:, (recommended) | AUQ tool input (its prose is the contract) | yes | (c) |
Cases that use S1-S8: plan-ceo-review-plan-mode, plan-devex-review-plan-mode, plan-mode-no-op, office-hours-auto-mode (gate); auto-decide-preserved, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke (periodic). S9-S10: plan-mode-no-op, plan-eng-review-plan-mode, plan-design-review-plan-mode-smoke. F1-F3: plan-ceo-finding-floor, plan-devex-finding-floor (gate); plan-design-finding-floor, plan-eng-finding-floor (periodic). A1: auq-format-gate (gate); the three standalone AUQ files (periodic).
Cases that use S1-S8: plan-ceo-review-plan-mode, plan-devex-review-plan-mode, plan-mode-no-op, office-hours-auto-mode (gate); auto-decide-preserved, plan-design-review-plan-mode-smoke (periodic). S9-S10: plan-mode-no-op, plan-design-review-plan-mode-smoke. F1-F3: plan-ceo-finding-floor, plan-devex-finding-floor (gate); plan-design-finding-floor (periodic). A1: auq-format-gate (gate); the three standalone AUQ files (periodic). plan-eng-review-plan-mode and plan-eng-finding-floor were deleted with standalone /plan-eng-review (GST-1698).
## Gate tier
@@ -205,18 +205,18 @@ Gate cases with no phrase-only checks: the shared-libs review-lifecycle, revalid
| plan-eng-multi-finding-batching | test/helpers/eng-seeded-coverage.ts:33-104 | distinct-issue identity per answered AUQ ("Issue N:", D/R-ID, ELI10 gates) | ledger R records, used only as fallback behind the prose gates | no | (b) |
| plan-eng-multi-finding-batching | test/helpers/pty/classify.ts:555 via counting.ts:554 | screen completion text ends counting | review-log row / ExitPlanMode tool_use | no | (b) |
| plan-eng-review-artifact | test/skill-e2e-plan-eng-artifact.test.ts:78 | test-plan artifact ~ dashboard/fetchStats/`/api/stats` | artifact schema has no field naming the change (scripts/resolvers/testing.ts:501-528) | no | (d) |
| carve-section-loading [plan-eng-review] | test/helpers/carve-section-case.ts:120 → test/helpers/auq-sdk-capture.ts:381 | report or output ~ report/review/summary/design doc/handoff | review-log row in the capture state root, deleted before the verdict (auq-sdk-capture.ts:368-370) | no | (b) |
| carve-section-loading [plan-eng-review] (deleted, GST-1698) | test/helpers/carve-section-case.ts:120 → test/helpers/auq-sdk-capture.ts:381 | report or output ~ report/review/summary/design doc/handoff | review-log row in the capture state root, deleted before the verdict (auq-sdk-capture.ts:368-370) | no | (b) |
| carve-section-loading [12 other skills] | same | same loose regex | none (Bash not allowed, auq-sdk-capture.ts:358-359) | no | (d) |
The plan-eng-review-plan-mode and plan-design-review-plan-mode-smoke verdicts are S1-S10. plan-eng-finding-floor and plan-design-finding-floor verdicts are F1-F3.
The plan-design-review-plan-mode-smoke verdicts are S1-S10, and the plan-design-finding-floor verdicts are F1-F3. Their plan-eng twins were deleted with standalone /plan-eng-review (GST-1698).
### Question format: test/skill-e2e-plan-format.test.ts, test/skill-e2e-plan-prosons.test.ts, standalone AUQ files
| Case | Check (file:line) | Matches | Structured source | User-visible | Disposition |
|---|---|---|---|---|---|
| plan-ceo-review-format-mode, plan-eng-review-format-kind | test/skill-e2e-plan-format.test.ts:174-175, :377-378 | no `Completeness: N/10`; kind note present | captured question | yes | (c) |
| plan-ceo-review-format-approach, plan-eng-review-format-coverage | test/skill-e2e-plan-format.test.ts:241, :310 | `Completeness: N/10` present | captured question | yes | (c) |
| all four format cases | test/skill-e2e-plan-format.test.ts:177, :243, :312, :380 | recommendation quality | — | — | judge |
| plan-ceo-review-format-mode | test/skill-e2e-plan-format.test.ts:173-174 | no `Completeness: N/10`; kind note present | captured question | yes | (c) |
| plan-ceo-review-format-approach | test/skill-e2e-plan-format.test.ts:240 | `Completeness: N/10` present | captured question | yes | (c) |
| both format cases | test/skill-e2e-plan-format.test.ts:176, :242 | recommendation quality | — | — | judge |
| plan-review-prosons-format | test/skill-e2e-plan-prosons.test.ts:184-196 | `D\d+ —`, ELI10:, Stakes…:, Recommendation:, Pros / cons:, ^Net:, ✅≥4, ❌≥2, (recommended) | captured question | yes | (c) |
| plan-review-prosons-hardstop-neg | test/skill-e2e-plan-prosons.test.ts:246-249 | no hard-stop escape line; ✅/❌ counts | captured question | yes | (c) |
| plan-review-prosons-neutral-neg | test/skill-e2e-plan-prosons.test.ts:299-303 | no `taste call`; (recommended); `Recommendation:.*because` | captured question | yes | (c) |
@@ -315,6 +315,6 @@ Red cases that are not convert candidates: shared-libs-plan-callers (3 reds, act
- Census commits `a8ace39` through `fd6854b` are not in this checkout, so the wave-reds line numbers could not be mapped to assertions; reds are attributed from the trial error text.
- The skill-llm-eval.test.ts judge cases are outside `E2E_TIERS` and are not covered.
- codex-ci-access.test.ts, codex-hardening.test.ts, codex-under-codex-detection.test.ts, e2e-shard-reuse.test.ts and skill-routing-e2e.test.ts name the umbrella keys but are free tests or route to the files audited above.
- plan-eng-review-artifact-full is `marathon` and out of scope.
- plan-eng-review-artifact-full was `marathon` and was deleted with standalone /plan-eng-review (GST-1698).
- The aside-* cases need a live Aside on macOS and never run in CI.
- "Not verified" in the structured-source column means the template writes the record, but the audit did not confirm that the fixture installs the helper.
lib/harness-register.ts
+2 -1
View File
@@ -19,6 +19,7 @@ import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { spawnSync } from 'node:child_process';
import { RETIRED_SKILLS } from '../scripts/discover-skills';
import { bunIsolationArgs } from './bun-isolation';
import { runBashSync } from './script-invocation';
import { frontmatterList, routerLine } from './skill-triggers';
@@ -233,7 +234,7 @@ function skillDirs(root: string, triggersByName = new Map<string, string[]>()):
/** The skills a target's payload carries: the host render (made for this checkout as the install root) or the checkout's committed Claude render. */
export function sourceSkills(target: Target, checkout: string, scratch: string): SkillEntry[] {
if (target.render === 'claude') return skillDirs(checkout).filter(s => s.name !== 'gstack');
if (target.render === 'claude') return skillDirs(checkout).filter(s => s.name !== 'gstack' && !RETIRED_SKILLS.includes(s.name));
const out = path.join(scratch, 'render');
fs.rmSync(out, { recursive: true, force: true });
const r = spawnSync(process.execPath, [...bunIsolationArgs(), 'scripts/gen-skill-docs.ts', '--host', target.name, '--install-root', checkout, '--out-dir', out],
scripts/harness-pins/scripts/lib/paid-cases.ts.json
+1 -1
View File
@@ -1 +1 @@
{"blob":"f862cf5241580c7593b9ee62eacf5ef046248d26","since":6,"nonBehavioral":"Claude-host verdicts are unchanged: the Codex host (EVALS_HOST=codex, codex-host-runner) runs only when selected and records its own per-model series; the Codex runner refactor (execCodex extraction, onSpawn hook) keeps its events and verdicts and adds token cost accounting; host fields on the plan/report are absent for claude"}
{"blob":"c984ba618be367f2201362a9592da7a8631accb5","since":6,"nonBehavioral":"Retired standalone /plan-eng-review cases (GST-1698): removed their budgets and paid-case rows; remaining files' walls shrink with their case counts, and no remaining case's budget or verdict changes"}
scripts/harness-pins/test/helpers/eval-budgets.ts.json
+1 -1
View File
@@ -1 +1 @@
{"blob":"8cb433b0618cc218dacf21cc644de7947442d045","since":6,"nonBehavioral":"GST-1403 removes the last three standalone LLM judges (qa anti-refusal, voice directive tone, cross-skill greptile consistency): llm-judge.ts drops their three schemas and three prompt builders, which no remaining case calls, and eval-budgets.ts drops the deleted test/skill-llm-eval.test.ts file wall; no kept case's prompt, threshold, samples, budget or verdict can change"}
{"blob":"5a16324a47b7c3c770e64322ae357f90c405a249","since":6,"nonBehavioral":"Retired standalone /plan-eng-review cases (GST-1698): removed their budgets and paid-case rows; remaining files' walls shrink with their case counts, and no remaining case's budget or verdict changes"}
scripts/lib/paid-cases.ts
-1
View File
@@ -76,7 +76,6 @@ export const CASE_TEST_NAMES: Record<string, string> = {
'cso-infra-scope': '/cso --infra finds an attacker-to-credential execution path',
'plan-ceo-review-plan-mode': 'first terminal outcome is asked (Step 0 fires before any plan write)',
'plan-eng-review-artifact': 'an interactive review writes one QA test plan about the reviewed change',
'plan-eng-review-artifact-full': 'a fresh interactive review reaches Test review and writes one QA test plan',
};
export const CASE_KEY_SEPARATOR = '#';
scripts/paid-test-durations.json
-4
View File
@@ -116,7 +116,6 @@
"test/carve-section-loading.test.ts#carve-section-loading-land-and-deploy": 156293,
"test/carve-section-loading.test.ts#carve-section-loading-plan-design-review": 223606,
"test/carve-section-loading.test.ts#carve-section-loading-plan-devex-review": 384295,
"test/carve-section-loading.test.ts#carve-section-loading-plan-eng-review": 297627,
"test/carve-section-loading.test.ts#carve-section-loading-qa": 149637,
"test/carve-section-loading.test.ts#carve-section-loading-retro": 199931,
"test/carve-section-loading.test.ts#carve-section-loading-review": 247902,
@@ -173,11 +172,8 @@
"test/skill-e2e-plan-design-finding-floor.test.ts": 131042,
"test/skill-e2e-plan-design-plan-mode.test.ts": 111838,
"test/skill-e2e-plan-devex-peer-comparison-classification.test.ts": 77201,
"test/skill-e2e-plan-eng-artifact-full.test.ts": 595009,
"test/skill-e2e-plan-eng-artifact.test.ts": 406795,
"test/skill-e2e-plan-eng-finding-floor.test.ts": 234397,
"test/skill-e2e-plan-eng-multi-finding-batching.test.ts": 409161,
"test/skill-e2e-plan-eng-plan-mode.test.ts": 170914,
"test/skill-e2e-plan-format.test.ts": 219670,
"test/skill-e2e-plan-prosons.test.ts": 163659,
"test/skill-e2e-plan-tune.test.ts": 1000,
test/carve-section-loading.test.ts
-1
View File
@@ -24,7 +24,6 @@ describeE2ETier('periodic')('carve section-loading', () => {
test('carve-section-loading-land-and-deploy', () => runCarveSectionCase('land-and-deploy'), CAPTURE_LONG_MS);
test('carve-section-loading-plan-design-review', () => runCarveSectionCase('plan-design-review'), CAPTURE_LONG_MS);
test('carve-section-loading-plan-devex-review', () => runCarveSectionCase('plan-devex-review'), CAPTURE_LONG_MS);
test('carve-section-loading-plan-eng-review', () => runCarveSectionCase('plan-eng-review'), CAPTURE_LONG_MS);
test('carve-section-loading-qa', () => runCarveSectionCase('qa'), CAPTURE_LONG_MS);
test('carve-section-loading-retro', () => runCarveSectionCase('retro'), CAPTURE_LONG_MS);
test('carve-section-loading-review', () => runCarveSectionCase('review'), CAPTURE_LONG_MS);
test/carve-section-sharding.test.ts
+3 -1
View File
@@ -9,6 +9,7 @@ import { CASE_SHARDED_FILES, DEFAULT_SHARD_TIMEOUT_MS, caseFile, expandCaseShard
import { carveSectionCaseId } from './helpers/carve-section-case';
import { E2E_TIERS, E2E_TOUCHFILES, selectTests } from './helpers/touchfiles';
import { E2E_KINDS } from './helpers/touchfiles-data';
import { RETIRED_SKILLS } from '../scripts/discover-skills';
describe('carved-skill cases each get a complete paid process budget', () => {
const FILE = 'test/carve-section-loading.test.ts';
@@ -16,7 +17,8 @@ describe('carved-skill cases each get a complete paid process budget', () => {
const source = fs.readFileSync(path.join(ROOT, FILE), 'utf8');
const cases = [...source.matchAll(/test\('carve-section-loading-([a-z-]+)', \(\) => runCarveSectionCase\('([a-z-]+)'\), CAPTURE_LONG_MS\)/g)]
.map(match => ({ id: `carve-section-loading-${match[1]}`, skill: match[2]! }));
const generic = Object.values(CARVE_GUARDS).filter(guard => guard.behavioral === 'plan' || guard.behavioral === 'prompt').map(guard => guard.skill).sort();
// A retired skill keeps its carve guard (its methodology still renders) but no section-loading case.
const generic = Object.values(CARVE_GUARDS).filter(guard => (guard.behavioral === 'plan' || guard.behavioral === 'prompt') && !RETIRED_SKILLS.includes(guard.skill)).map(guard => guard.skill).sort();
test('every generic registry entry has exactly one periodic case, under its own case id, and no external entry does', () => {
expect(source).toContain("describeE2ETier('periodic')");
test/e2e-harness-audit.test.ts
+3 -1
View File
@@ -13,6 +13,7 @@
import { describe, test, expect } from 'bun:test';
import * as fs from 'fs';
import * as path from 'path';
import { RETIRED_SKILLS } from '../scripts/discover-skills';
const ROOT = path.resolve(import.meta.dir, '..');
@@ -47,7 +48,8 @@ function skillTemplateDirs(): string[] {
*/
function findInteractiveSkills(): string[] {
const interactive: string[] = [];
for (const skill of skillTemplateDirs()) {
// A retired skill keeps its template as methodology for another skill but has no interactive entry.
for (const skill of skillTemplateDirs().filter(name => !RETIRED_SKILLS.includes(name))) {
const tmplPath = path.join(ROOT, skill, 'SKILL.md.tmpl');
if (!fs.existsSync(tmplPath)) continue;
const content = fs.readFileSync(tmplPath, 'utf-8');
test/eng-finding-retry-budget.test.ts
+3 -1
View File
@@ -1,6 +1,7 @@
import { expect, test } from 'bun:test';
import { resolvePaidShardBudget, retriesForFiles, planPaidShards, parseRunManifest, verifySliceResults, runPaidShard, buildRunManifest, paidShardWallUpperBoundMs, collectPaidTestFiles, selectPaidTestFiles, isOverlayTestFile, DEFAULT_SHARD_TIMEOUT_MS, DEFAULT_JOBS, parseCliOptions, expandCaseShards, expandTrialShards, shardFile, sliceExecutionOrder, sliceSupervisedWallMs, resolvePaidShardTimeoutMs } from '../scripts/test-paid-shards';
import { FINDING_RETRY_BUDGETS, ALL_TIERS, SHARD_RESERVE_MS } from './helpers/eval-budgets';
import { PERIODIC_CI_EXCLUDE } from './helpers/periodic-exclude-data';
import fs from 'node:fs';
import os from 'node:os';
import path from 'node:path';
@@ -141,7 +142,7 @@ test('live periodic census fits the declared CI wall including setup', () => {
test('registered allocation is deterministic and preserves every discovered file', () => {
const files = collectPaidTestFiles();
expect(files).toHaveLength(114); // - skill-llm-eval (last standalone judges removed, GST-1403); + skill-e2e-autoplan-ceo-only (GST-537); + skill-e2e-planning-wave (GSTA-228); + skill-e2e-model-panel (GSTA-173); + skill-e2e-write-skill (GSTA-133); + skill-e2e-ship-frozen-scope (frozen review scope, GSTA-113 PR 2); + skill-e2e-failure-routing (#3157/#3158); + skill-e2e-dx-error-contract (#3226); + skill-e2e-gate-integrity (HONEST_WORK_GATE_INTEGRITY W8); + skill-e2e-autoplan-schema-canary, -guard-pty and -long-session (Oct 7 wave, CEO-13, ENG-18, CEO-15); + skill-e2e-ship-measure-seeded-flake (v1.91.32.0); W5a: 15 carve wrappers became one case-sharded file; + codex-e2e-multiblock-live and skill-e2e-design-model-smoke (v1.91.30.0);
expect(files).toHaveLength(111); // - skill-e2e-plan-eng-plan-mode, -plan-eng-finding-floor and -plan-eng-artifact-full (standalone /plan-eng-review retired, GST-1698); - skill-llm-eval (last standalone judges removed, GST-1403); + skill-e2e-autoplan-ceo-only (GST-537); + skill-e2e-planning-wave (GSTA-228); + skill-e2e-model-panel (GSTA-173); + skill-e2e-write-skill (GSTA-133); + skill-e2e-ship-frozen-scope (frozen review scope, GSTA-113 PR 2); + skill-e2e-failure-routing (#3157/#3158); + skill-e2e-dx-error-contract (#3226); + skill-e2e-gate-integrity (HONEST_WORK_GATE_INTEGRITY W8); + skill-e2e-autoplan-schema-canary, -guard-pty and -long-session (Oct 7 wave, CEO-13, ENG-18, CEO-15); + skill-e2e-ship-measure-seeded-flake (v1.91.32.0); W5a: 15 carve wrappers became one case-sharded file; + codex-e2e-multiblock-live and skill-e2e-design-model-smoke (v1.91.30.0);
expect(files).toContain('test/skill-e2e-ship-skip.test.ts');
const m = livePlan(files);
expect(livePlan([...files].reverse())).toEqual(m);
@@ -175,6 +176,7 @@ test('single-slice manifest retains all registered files with one allocation', (
for (const m of manifests) expect(m.entries.filter(e => e.status === 'planned').every(e => e.slice === 1)).toBe(true);
for (const budget of FINDING_RETRY_BUDGETS) {
const entries = manifests.flatMap(m => m.entries.filter(e => e.file === budget.file && e.status === 'planned'));
if (PERIODIC_CI_EXCLUDE[budget.file]) { expect(entries, budget.file).toHaveLength(0); continue; }
expect(entries, budget.file).toHaveLength(1);
expect(entries[0]!.budget).toEqual(resolvePaidShardBudget([budget.file]));
}
test/eval-kinds.test.ts
+1 -1
View File
@@ -62,7 +62,7 @@ describe('E2E_KINDS registry', () => {
// Contract-shaped cases stay rule: ask-before-decide, plan-mode no-writes,
// mandated steps, secrets, and the batching floor never ride a majority.
for (const id of ['plan-ceo-mode-routing', 'plan-eng-multi-finding-batching', 'plan-design-review-plan-mode',
'plan-eng-review-plan-mode', 'plan-ceo-section-loading', 'setup-gbrain-bad-token', 'qa-only-no-fix', 'review-sql-injection']) {
'plan-ceo-section-loading', 'setup-gbrain-bad-token', 'qa-only-no-fix', 'review-sql-injection']) {
expect(E2E_KINDS[id], id).toBe('rule');
}
});
test/fixtures/paid-report-census/gate/manifest.json
Vendored
+1 -1
View File
@@ -16,7 +16,7 @@
"status": "planned",
"estimatedMs": 217184,
"budget": {
"timeoutMs": 3720000,
"timeoutMs": 2520000,
"source": "registered",
"policyId": "skill-e2e-plan-mode-no-op-existing-retry-v1"
}
test/fixtures/paid-report-census/gate/paid-slice-1-a1/slice-1.json
Vendored
+1 -1
View File
@@ -20,7 +20,7 @@
"executedTests": 5,
"skippedTests": 0,
"budget": {
"timeoutMs": 3720000,
"timeoutMs": 2520000,
"source": "registered",
"policyId": "skill-e2e-plan-mode-no-op-existing-retry-v1"
}
test/fixtures/touchfile-moved-code/w4-pty.json
Vendored
-6
View File
@@ -15,7 +15,6 @@
"carve-section-loading-land-and-deploy",
"carve-section-loading-plan-design-review",
"carve-section-loading-plan-devex-review",
"carve-section-loading-plan-eng-review",
"carve-section-loading-qa",
"carve-section-loading-retro",
"carve-section-loading-review",
@@ -33,9 +32,7 @@
"plan-design-with-ui-scope",
"plan-devex-finding-floor",
"plan-devex-review-plan-mode",
"plan-eng-finding-floor",
"plan-eng-multi-finding-batching",
"plan-eng-review-plan-mode",
"plan-mode-no-op",
"ship-section-loading"
],
@@ -54,7 +51,6 @@
"carve-section-loading-land-and-deploy",
"carve-section-loading-plan-design-review",
"carve-section-loading-plan-devex-review",
"carve-section-loading-plan-eng-review",
"carve-section-loading-qa",
"carve-section-loading-retro",
"carve-section-loading-review",
@@ -72,9 +68,7 @@
"plan-design-with-ui-scope",
"plan-devex-finding-floor",
"plan-devex-review-plan-mode",
"plan-eng-finding-floor",
"plan-eng-multi-finding-batching",
"plan-eng-review-plan-mode",
"plan-mode-no-op",
"ship-section-loading"
],
test/fixtures/touchfile-moved-code/w5-review-resolvers.json
Vendored
-5
View File
@@ -14,7 +14,6 @@
"carve-section-loading-land-and-deploy",
"carve-section-loading-plan-design-review",
"carve-section-loading-plan-devex-review",
"carve-section-loading-plan-eng-review",
"carve-section-loading-qa",
"carve-section-loading-retro",
"carve-section-loading-review",
@@ -35,13 +34,9 @@
"plan-devex-finding-floor",
"plan-devex-review-plan-mode",
"plan-eng-coverage-audit",
"plan-eng-finding-floor",
"plan-eng-multi-finding-batching",
"plan-eng-review",
"plan-eng-review-artifact",
"plan-eng-review-format-coverage",
"plan-eng-review-format-kind",
"plan-eng-review-plan-mode",
"plan-mode-no-op",
"plan-review-prosons-format",
"plan-review-report",
test/harness-register.test.ts
+12
View File
@@ -13,6 +13,7 @@ import {
BLOCK_BEGIN, BLOCK_END, CONTEXT_BLOCK_MAX, DOT_SKILLS_DIR, HARNESS_CODES, HYBRID_SKILLS, HarnessError, OPENCLAW_ALL_TEXTS, OPENCLAW_INSTRUCTION_MAX, ROUTER_DESCRIPTION, ROUTER_DESCRIPTION_MAX, TARGETS,
contextBlock, contextScope, detect, listLedgers, readLedger, refresh, register, sourceSkills, unregister, type RegisterOptions,
} from '../lib/harness-register';
import { RETIRED_SKILLS } from '../scripts/discover-skills';
import { QM_PACK_LIMITS, qmRequest, registerQm, removeQmPack } from '../lib/harness-qm';
import { cleanupFixtures, cleanupSeed, makeFixture, makeSource, registryRows, runSetup, tree } from './helpers/install-fixture';
@@ -139,6 +140,17 @@ describe('gstack-harness register and unregister (skill folders)', () => {
}
}, 60_000);
test('Claude-render targets (dots, generic) never register a retired skill; /autoplan still carries its methodology', () => {
const d = tmp();
const names = sourceSkills(TARGETS.dots, ROOT, d).map(s => s.name);
expect(names).toContain('autoplan');
for (const retired of RETIRED_SKILLS) expect(names).not.toContain(retired);
expect(sourceSkills(TARGETS.generic, ROOT, d).map(s => s.name)).toEqual(names);
const r = register(opts(d, 'dots', { mode: 'all', skillsDir: path.join(d, 'all') }));
for (const retired of RETIRED_SKILLS) expect(fs.existsSync(path.join(r.dest, retired))).toBe(false);
expect(fs.existsSync(path.join(ROOT, 'plan-eng-review', 'SKILL.md'))).toBe(true);
}, 60_000);
test('generic: the context block is replaced in place, then cut out; the user\'s lines and edits survive', () => {
const d = tmp();
const ctx = path.join(d, 'agent', 'AGENTS.md');
test/helpers/eval-budgets.ts
+3 -5
View File
@@ -99,16 +99,14 @@ export const FILE_RETRY_BUDGETS = [
{ file: 'test/skill-e2e-qa-callers.test.ts', attemptMs: 5 * (CAPTURE_MS + 15_000), caseMs: CAPTURE_MS + 15_000 },
{ file: 'test/skill-e2e-shared-libs-paths.test.ts', attemptMs: 3 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-ship-docsync.test.ts', attemptMs: 4 * CAPTURE_LONG_MS + 8 * CAPTURE_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-auq-matrix.test.ts', attemptMs: 6 * CAPTURE_MS, caseMs: CAPTURE_MS },
{ file: 'test/skill-e2e-plan-format.test.ts', attemptMs: 4 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 },
{ file: 'test/skill-e2e-auq-matrix.test.ts', attemptMs: 5 * CAPTURE_MS, caseMs: CAPTURE_MS },
{ file: 'test/skill-e2e-plan-format.test.ts', attemptMs: 2 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 },
{ file: 'test/skill-e2e-auto-decide-preserved.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS },
{ file: 'test/skill-e2e-plan-ceo-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS },
{ file: 'test/skill-e2e-plan-eng-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS },
{ file: 'test/skill-e2e-plan-design-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS },
{ file: 'test/skill-e2e-plan-devex-finding-floor.test.ts', attemptMs: PTY_MS, caseMs: PTY_MS },
{ file: 'test/skill-e2e-plan-mode-no-op.test.ts', attemptMs: 6 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-plan-mode-no-op.test.ts', attemptMs: 4 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-plan-ceo-mode-routing.test.ts', attemptMs: 2 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-plan-eng-plan-mode.test.ts', attemptMs: 2 * CAPTURE_LONG_MS, caseMs: CAPTURE_LONG_MS },
{ file: 'test/skill-e2e-plan-prosons.test.ts', attemptMs: 4 * (CAPTURE_MS + 10_000), caseMs: CAPTURE_MS + 10_000 },
// Gate: six 300s cases + one 610s case; periodic: two 900s + two 600s
// (plan-eng-review-artifact runs from its own PTY file).
test/helpers/periodic-exclude-data.ts
+4
View File
@@ -23,6 +23,10 @@ export const PERIODIC_CI_EXCLUDE: Record<string, { reason: string; tracking: str
reason: 'needs OPENROUTER_API_KEY, which no CI workflow carries; a preregistered research instrument run on demand, not weekly coverage',
tracking: 'TODOS.md "CI-unrunnable paid evals" (re-entry: OPENROUTER_API_KEY in the periodic slice env and a new preregistration; review by 2026-12-28)',
},
'test/skill-e2e-plan-eng-multi-finding-batching.test.ts': {
reason: 'retired with standalone /plan-eng-review (GST-1698): it checks one interactive question per eng finding, and /autoplan Phase 3 auto-decides findings, so no shipped entry reaches the behavior; running it weekly is known waste',
tracking: 'TODOS.md "/plan-eng-review retirement follow-ups" (re-entry: none; delete the file with its batching counter helpers and fixtures)',
},
'test/skill-e2e-ios-device.test.ts': {
reason: 'needs a physical iPhone over USB/devicectl — manual hardware, not a CI runner capability',
tracking: 'TODOS.md "CI-unrunnable paid evals" (re-entry: the CLI/device is available in the CI image; review by 2026-12-28)',
test/helpers/touchfiles-data.ts
-76
View File
@@ -191,13 +191,6 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/plan-eng-resume.ts', 'test/fixtures/plan-eng-artifact-resume-37174266054.json', 'test/helpers/workflow-excerpt.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'],
'plan-eng-review-artifact-full': [ 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/pty/**', 'test/helpers/pty/auq.ts', 'test/helpers/pty/binary.ts', 'test/helpers/pty/boundaries.ts', 'test/helpers/pty/classify.ts', 'test/helpers/pty/fake-session.ts', 'test/helpers/pty/judge.ts', 'test/helpers/pty/launch.ts', 'test/helpers/pty/plan-native.ts', 'test/helpers/pty/runners/counting.ts', 'test/helpers/pty/runners/floor.ts', 'test/helpers/pty/runners/observation.ts', 'test/helpers/pty/screen.ts', 'test/helpers/pty/session.ts', 'test/helpers/skill-census.ts',
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'test/skill-e2e-plan-eng-artifact-full.test.ts', 'test/helpers/plan-eng-artifact-fixture.ts', 'test/helpers/workflow-excerpt.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/llm-judge.ts', 'test/helpers/office-hours-attempt.ts', 'test/helpers/office-hours-completion.ts', 'test/helpers/workflow-judge-input.ts'],
'plan-review-report': [
'test/helpers/office-hours-attempt.ts',
@@ -231,28 +224,6 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
"test/fixtures/eng-option-b-scope-al.json",
'test/fixtures/pty-companion-cli.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/tasks-section.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'bin/gstack-context-recovery'],
'plan-eng-review-plan-mode': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts',
'test/fixtures/auto-decide-recommendation-361c.json',
'test/fixtures/auto-decide-target-361c.json',
'test/fixtures/autoplan-public-narration-ad.json',
'test/helpers/plan-count-transcript.ts', 'test/fixtures/plan-count-cross-cwd-ancestry-0bcd.json',
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-target-aw.json",
"test/fixtures/plan-scope-recovery-av.json",
'test/fixtures/auto-decide-saved-ai.json', 'test/fixtures/auto-decide-retry-ai.json','bin/gstack-skill-start', 'bin/gstack-skill-end', 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/question-tuning.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/skill-e2e-plan-eng-plan-mode.test.ts', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt', 'test/helpers/plan-scope-selection.ts', 'test/fixtures/design-plan-scope-ag.json', 'test/fixtures/design-scope-selection-aj.json', 'test/helpers/native-auto-decide.ts', 'test/fixtures/auto-decide-current-declaration-6aef.json', 'test/fixtures/auto-decide-explanatory-mode-043a.json', 'test/fixtures/auto-decide-explanatory-mode-749df.json', 'test/fixtures/auto-decide-structured-77.json', 'test/helpers/auto-decision-state.ts', 'test/fixtures/auto-decide-state-cab3.json', 'bin/gstack-question-log', 'bin/gstack-question-preference', 'test/helpers/fake-plan-seed.ts', 'test/fixtures/native-auto-decide-ag.json', 'test/fixtures/eng-seeded-completion-ai.json', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/pty-screen.ts', 'test/fixtures/pty-screen/**',
'test/fixtures/design-scope-announcement-ao.json',
'test/fixtures/design-scope-declaration-ak.json',
"test/fixtures/eng-option-b-scope-al.json",
"scripts/resolvers/preamble/generate-preamble-bash.ts",
'test/fixtures/pty-companion-cli.ts', 'test/helpers/plan-seed-submission.ts', 'test/fixtures/plan-seed-cli.ts', 'test/helpers/owned-claude-transcript.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/testing.ts', 'test/helpers/plan-mode-evidence.ts', 'lib/redact-engine.ts', 'lib/redact-patterns.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'bin/gstack-context-recovery'],
// PTY plan-mode smoke (whole file); the SDK plan-edit case below owns plan-design-review-plan-mode.
'plan-design-review-plan-mode-smoke': [
'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts',
@@ -440,7 +411,6 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
'carve-section-loading-land-and-deploy': ['land-and-deploy/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-plan-design-review': ['plan-design-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-plan-devex-review': ['plan-devex-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-plan-eng-review': ['plan-eng-review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'scripts/resolvers/learnings.ts', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'scripts/resolvers/testing.ts', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-qa': ['qa/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-retro': ['retro/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
'carve-section-loading-review': ['review/**', 'setup', 'bin/gstack-bun-version.sh', 'bin/gstack-state-root.sh', 'lib/state-root.ts', 'bin/gstack-review-log', 'bin/gstack-review-read', 'lib/review-evidence.ts', 'bin/gstack-slug', 'bin/gstack-remote-identity.sh', 'bin/gstack-wtree', 'bin/gstack-config', 'bin/gstack-brain-enqueue', 'test/fixtures/autoplan-amend-input-77.json', 'test/fixtures/autoplan-phase-handoff-6714.json', 'test/fixtures/plan-scope-recovery-av.json', 'test/fixtures/design-scope-checkpoint-at.json', 'scripts/resolvers/composition.ts', 'autoplan/**', 'test/helpers/carve-guards.ts', 'scripts/resolvers/sections.ts', 'scripts/resolvers/redact-doc.ts', 'scripts/gen-skill-docs.ts', 'test/helpers/auq-sdk-capture.ts', 'test/helpers/session-runner.ts', 'bin/gstack-autoplan-snapshot.ts', 'test/fixtures/autoplan/t-ceo-omitted-obligations.json', 'test/fixtures/autoplan/u-ceo-original-loss.json', 'test/fixtures/autoplan/v-ceo-dangling-references.json', 'scripts/resolvers/preamble/generate-preamble-bash.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'test/carve-section-loading.test.ts', 'test/helpers/carve-section-case.ts', 'test/fixtures/design-html-section-complete.md', 'test/helpers/carve-plan-fixture.ts', 'test/fixtures/carve-existing-repository/**', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'test/helpers/agent-sdk-runner.ts', 'test/helpers/sdk-system-prompt.ts', 'test/helpers/auq-native-capture.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/capture-parity-baseline.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/llm-judge.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/parity-harness.ts', 'test/helpers/size-ledger.ts', 'test/size-ledger/**', 'test/helpers/plan-count-artifacts.ts', 'test/helpers/plan-count-file-permission.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-floor-review.ts', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/plan-skill-questions.ts', 'test/helpers/pty-screen.ts', 'test/helpers/pty-trust-dialog.ts', 'test/helpers/skill-census.ts'],
@@ -457,22 +427,6 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
// review-phase AskUserQuestion). Uses runPlanSkillFloorCheck — minimal
// "did agent fire ANY AUQ?" observer that exits early on first non-permission
// numbered-option render. ~1-3 min typical wall time per test, ~$2-6 total.
'plan-eng-finding-floor': ['bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/pty-screen.ts',
'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', 'test/fixtures/plan-create-prepublication-491.json', 'test/fixtures/plan-create-combined-permission-70b.json', 'test/fixtures/plan-floor-quote-70b.json',
'test/fixtures/plan-create-permission-361c.json',
'test/helpers/plan-floor-review.ts',
'test/fixtures/plan-floor-routing-361c.json',
'test/fixtures/ceo-report-permission-fb10.json', 'test/fixtures/plan-floor-permission-fb10.json', 'test/helpers/plan-count-file-permission.ts', 'test/fixtures/plan-edit-cropped-permission-1579.json', 'test/fixtures/plan-count-permission-target-ad-v2.json', 'test/fixtures/design-crop-gutter-ap.json', 'test/fixtures/plan-count-crop-ak.json', 'test/fixtures/plan-count-long-edit-0bcd.json', 'test/fixtures/plan-count-cropped-wrap-6714.json',
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'bin/gstack-skill-start', 'bin/gstack-skill-end', 'plan-eng-review/**', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completion-status.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts', 'test/helpers/claude-pty-runner.ts', 'test/helpers/pty/**', 'test/fixtures/design-ui-boxed-question.json', 'test/helpers/hermetic-skill-runtime.ts', 'test/helpers/pty-trust-dialog.ts', 'test/fixtures/forcing-finding-seeds.ts', 'test/skill-e2e-plan-eng-finding-floor.test.ts', 'test/fixtures/eng-batching-t-calls.json', 'test/fixtures/eng-scope-y-calls.json', 'test/fixtures/eng-d2-truncated-border-0bcd.json', 'test/fixtures/eng-d1-clipped-elision-1579.json', 'test/fixtures/eng-d2-planning-prelude-4d.json', 'test/fixtures/ceo-approach-z-call.json', 'test/fixtures/ceo-approach-z-screen.txt', 'test/fixtures/eng-binding-z-calls.json', 'test/fixtures/eng-binding-retry-z-calls.json', 'test/helpers/plan-floor-target.ts', 'test/helpers/plan-count-fixture.ts', 'test/helpers/plan-count-artifacts.ts',
"test/fixtures/eng-injected-export-aq.json", "test/fixtures/eng-library-hooks-aq.json",
"scripts/resolvers/preamble/generate-preamble-bash.ts",
'test/fixtures/pty-companion-cli.ts', 'lib/fs-atomic.ts', 'test/helpers/plan-skill-questions.ts', 'test/fixtures/eng-auq-validation-error.json', 'test/fixtures/bash-directory-permission.json', 'test/fixtures/design-tasks-bash-permission.json', 'test/fixtures/read-permission.json', 'test/fixtures/ceo-split-e5-numbered-description-491.json', 'test/helpers/plan-skill-question-events.ts', 'test/helpers/plan-skill-question-hook-scope.ts', 'test/helpers/skill-census.ts', 'scripts/resolvers/testing.ts', 'test/helpers/auto-decision-state.ts', 'test/helpers/autoplan-artifact-digest.ts', 'test/helpers/autoplan-artifact-permission.ts', 'test/helpers/autoplan-artifact-recorder.ts', 'test/helpers/dx-selected-navigation.ts', 'test/helpers/e2e-gate.ts', 'test/helpers/eng-cache-writer-decision.ts', 'test/helpers/native-auto-decide.ts', 'test/helpers/owned-claude-transcript.ts', 'test/helpers/plan-count-pending-exit.ts', 'test/helpers/plan-count-pending-question.ts', 'test/helpers/plan-count-transcript.ts', 'test/helpers/plan-scope-selection.ts', 'test/helpers/plan-seed-submission.ts', 'bin/gstack-context-recovery'],
'plan-ceo-finding-floor': [ 'ETHOS.md','bin/gstack-state-root.sh', 'lib/state-root.ts', 'test/helpers/pty-screen.ts',
'lib/claude-public-transcript.ts', 'lib/claude-journal-records.ts', 'lib/claude-owned-journal.ts', 'test/fixtures/plan-create-prepublication-491.json', 'test/fixtures/plan-create-combined-permission-70b.json', 'test/fixtures/plan-floor-quote-70b.json',
@@ -601,20 +555,6 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
'plan-ceo-review-format-approach': [ 'ETHOS.md', 'plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts',
'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/tasks-section.ts'
],
'plan-eng-review-format-coverage': [
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts'
],
'plan-eng-review-format-kind': [
'scripts/resolvers/learnings.ts',
"test/fixtures/plan-scope-recovery-av.json", 'plan-eng-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'scripts/resolvers/preamble-hoist.ts', 'model-overlays/opus-4-7.md', 'test/helpers/llm-judge.ts', 'test/skill-e2e-plan-format.test.ts', 'test/helpers/plan-format-kind-note.ts',
"scripts/resolvers/preamble/generate-preamble-bash.ts", "scripts/resolvers/preamble/generate-completion-status.ts",
'scripts/resolvers/testing.ts', 'test/helpers/office-hours-attempt.ts', 'scripts/resolvers/review-dashboard.ts', 'scripts/resolvers/plan-gates.ts', 'scripts/resolvers/spec-review.ts', 'scripts/resolvers/outside-voice-steps.ts', 'scripts/resolvers/review-scope.ts', 'scripts/resolvers/outside-voice.ts'
],
// v1.7.0.0 Pros/Cons format cadence + format + negative-escape evals.
// Dependencies: same as format-mode + the 4 plan-review templates + overlay.
@@ -1357,7 +1297,6 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
'plan-ceo-review-expansion-energy': 'periodic', // Demoted from gate (2026-08 audit): Opus generator + subjective 2-axis >=4/5 LLM-judge threshold in the merge lane — the exact class siblings were demoted for (a +21-line preamble change once flipped the score). CLAUDE.md's own rule: Opus model test -> periodic.
'plan-eng-review': 'periodic',
'plan-eng-review-artifact': 'periodic', // Checkpoint: resumes at Test review from a real Scope Challenge ledger
'plan-eng-review-artifact-full': 'marathon', // Full fresh /plan-eng-review through Test review (498 s pass; 595 s timeouts at 600 s before Test review)
'plan-eng-coverage-audit': 'gate',
'ship-coverage-value': 'gate',
'review-test-value': 'gate',
@@ -1372,7 +1311,6 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
// the hardened ask-first gate + the collapsed-form detector lifted them from
// always-failing to mostly-passing, but they are not deterministic gates.
'plan-ceo-review-plan-mode': 'gate',
'plan-eng-review-plan-mode': 'periodic',
'plan-design-review-plan-mode': 'periodic',
'plan-design-review-plan-mode-smoke': 'periodic',
'plan-devex-review-plan-mode': 'gate',
@@ -1409,7 +1347,6 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
'carve-section-loading-land-and-deploy': 'periodic',
'carve-section-loading-plan-design-review': 'periodic',
'carve-section-loading-plan-devex-review': 'periodic',
'carve-section-loading-plan-eng-review': 'periodic',
'carve-section-loading-qa': 'periodic',
'carve-section-loading-retro': 'periodic',
'carve-section-loading-review': 'periodic',
@@ -1421,7 +1358,6 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
// run drives a full skill end-to-end (~25 min, ~$5/run). Sequential
// execution during calibration; concurrent opt-in only after measured
// comparison agrees (plan §D15).
'plan-eng-finding-floor': 'periodic', // stochastic ask-first (see plan-mode-handshake note); periodic
'plan-ceo-finding-floor': 'gate',
'plan-design-finding-floor': 'periodic', // stochastic ask-first (see plan-mode-handshake note); periodic
'plan-devex-finding-floor': 'gate',
@@ -1445,8 +1381,6 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic' | 'marathon'> = {
// AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark)
'plan-ceo-review-format-mode': 'periodic',
'plan-ceo-review-format-approach': 'periodic',
'plan-eng-review-format-coverage': 'periodic',
'plan-eng-review-format-kind': 'periodic',
// Office-hours Phase 4 silent-auto-decide regression — periodic (Phase 4
// requires the agent to invent 2-3 architectures, more open-ended than the
@@ -1798,11 +1732,9 @@ export const E2E_KINDS: Record<string, 'rule' | 'behavior' | 'judge'> = {
'plan-ceo-review-expansion-energy': 'behavior',
'plan-eng-review': 'rule',
'plan-eng-review-artifact': 'rule',
'plan-eng-review-artifact-full': 'rule',
'plan-eng-coverage-audit': 'rule',
'plan-review-report': 'rule',
'plan-ceo-review-plan-mode': 'rule',
'plan-eng-review-plan-mode': 'rule',
'plan-design-review-plan-mode': 'rule',
'plan-design-review-plan-mode-smoke': 'rule',
'ship-coverage-value': 'rule',
@@ -1835,13 +1767,11 @@ export const E2E_KINDS: Record<string, 'rule' | 'behavior' | 'judge'> = {
'carve-section-loading-land-and-deploy': 'rule',
'carve-section-loading-plan-design-review': 'rule',
'carve-section-loading-plan-devex-review': 'rule',
'carve-section-loading-plan-eng-review': 'rule',
'carve-section-loading-qa': 'rule',
'carve-section-loading-retro': 'rule',
'carve-section-loading-review': 'rule',
'carve-section-loading-setup-gbrain': 'rule',
'carve-section-loading-spec': 'rule',
'plan-eng-finding-floor': 'rule',
'plan-ceo-finding-floor': 'rule',
'plan-design-finding-floor': 'rule',
'plan-devex-finding-floor': 'rule',
@@ -1852,8 +1782,6 @@ export const E2E_KINDS: Record<string, 'rule' | 'behavior' | 'judge'> = {
'setup-gbrain-path4-local-pglite': 'rule',
'plan-ceo-review-format-mode': 'behavior',
'plan-ceo-review-format-approach': 'behavior',
'plan-eng-review-format-coverage': 'behavior',
'plan-eng-review-format-kind': 'behavior',
'office-hours-phase4-fork': 'behavior',
'llm-judge-recommendation': 'judge',
'plan-ceo-review-prosons-cadence': 'behavior',
@@ -2037,10 +1965,6 @@ export const BEHAVIOR_WHY: Record<string, string> = {
"Mode-question wording (Completeness line vs kind note) is live formatting of one AskUserQuestion.",
'plan-ceo-review-format-approach':
"Approach-menu Completeness wording is live formatting of one AskUserQuestion.",
'plan-eng-review-format-coverage':
"Coverage-issue Completeness wording is live formatting of one AskUserQuestion.",
'plan-eng-review-format-kind':
"Kind-note wording is live formatting of one AskUserQuestion.",
'plan-ceo-review-prosons-cadence':
"Pros/Cons cadence on a hard-stop question is live formatting; either the escape or the full block is accepted.",
'plan-review-prosons-format':
test/office-hours-attempt.test.ts
-2
View File
@@ -557,8 +557,6 @@ describe('session runner native CLI max-turns exit semantics', () => {
const FORMAT_CASES = [
['skill-e2e-plan-format.test.ts', 'plan-ceo-review-format-mode'],
['skill-e2e-plan-format.test.ts', 'plan-ceo-review-format-approach'],
['skill-e2e-plan-format.test.ts', 'plan-eng-review-format-coverage'],
['skill-e2e-plan-format.test.ts', 'plan-eng-review-format-kind'],
['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-format'],
['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-hardstop-neg'],
['skill-e2e-plan-prosons.test.ts', 'plan-review-prosons-neutral-neg'],
test/paid-report-census.test.ts
+1 -1
View File
@@ -5,7 +5,7 @@
* 2026-10-03 periodic and gate census artifacts (run 37151477069, commit
* ac20ef1), whose report counted deselections as SKIPPED 82/282 and left
* 72/24 testcases unattributed. Budget records follow the current declared
* policy (plan-mode-no-op: 3,720,000 ms since v1.91.18.0 added its devex member).
* policy (plan-mode-no-op: 2,520,000 ms since its plan-eng-review member retired, GST-1698).
* The periodic fixture is re-shaped to the current case shards: qa-workflow runs
* as its qa-fix-loop case shard (its three gate-tier siblings become sibling
* deselections) and the review carve runs as a case of the consolidated
test/paid-retry-supervision.test.ts
+10 -13
View File
@@ -22,16 +22,14 @@ const expectedWalls = {
'test/skill-e2e-shared-libs-paths.test.ts': 1_920_000,
'test/skill-e2e-ship-docsync.test.ts': 4_920_000,
'test/skill-e2e-auq-consistency.test.ts': 1_080_000,
'test/skill-e2e-auq-matrix.test.ts': 1_920_000,
'test/skill-e2e-plan-format.test.ts': 1_360_000,
'test/skill-e2e-auq-matrix.test.ts': 1_620_000,
'test/skill-e2e-plan-format.test.ts': 740_000,
'test/skill-e2e-auto-decide-preserved.test.ts': 1_020_000,
'test/skill-e2e-plan-ceo-finding-floor.test.ts': 1_020_000,
'test/skill-e2e-plan-eng-finding-floor.test.ts': 1_020_000,
'test/skill-e2e-plan-design-finding-floor.test.ts': 1_020_000,
'test/skill-e2e-plan-devex-finding-floor.test.ts': 1_020_000,
'test/skill-e2e-plan-mode-no-op.test.ts': 3_720_000,
'test/skill-e2e-plan-mode-no-op.test.ts': 2_520_000,
'test/skill-e2e-plan-ceo-mode-routing.test.ts': 1_320_000,
'test/skill-e2e-plan-eng-plan-mode.test.ts': 1_320_000,
'test/skill-e2e-plan-prosons.test.ts': 1_360_000,
'test/skill-e2e-plan.test.ts': 3_120_000,
};
@@ -50,9 +48,9 @@ test('paid evals never retry: every paid file and registered row runs once', ()
}
});
test('registration covers exactly the sixteen demonstrated full-file retry gaps', () => {
test('registration covers exactly the fourteen demonstrated full-file retry gaps', () => {
expect(Object.fromEntries(newBudgets.map(row => [row.file, row.shardMs]))).toEqual(expectedWalls);
expect(new Set(FILE_RETRY_BUDGETS.map(row => row.file)).size).toBe(18);
expect(new Set(FILE_RETRY_BUDGETS.map(row => row.file)).size).toBe(16);
expect(STRICT_RETRY_CASE_BUDGETS.map(row => row.file)).toEqual([
...FINDING_RETRY_BUDGETS.map(row => row.file), AUQ_CONSISTENCY_RETRY_BUDGET.file,
]);
@@ -82,20 +80,19 @@ test('source allowances retain all captures, cases, and finalization grace', ()
expect(read('test/helpers/codex-eval.ts')).toContain('CODEX_EVAL_FINALIZE_MS = 2 * CODEX_DRAIN_GRACE_MS');
expect(read('test/helpers/codex-session-runner.ts')).toMatch(/CODEX_DRAIN_GRACE_MS\s*=\s*5_000/);
expect(read('test/helpers/office-hours-attempt.ts')).toContain('OFFICE_HOURS_BUN_GRACE_MS = 10_000');
for (const file of ['auto-decide-preserved', 'plan-ceo-finding-floor', 'plan-eng-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor']) {
for (const file of ['auto-decide-preserved', 'plan-ceo-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor']) {
expect(timeoutExpressions(`test/skill-e2e-${file}.test.ts`)).toEqual(['PTY_MS']);
if (file.endsWith('-finding-floor')) expect(read(`test/skill-e2e-${file}.test.ts`)).toContain('timeoutMs: CAPTURE_LONG_MS');
}
const noop = read('test/skill-e2e-plan-mode-no-op.test.ts');
expect(noop).toContain("['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'plan-devex-review'] as const");
expect(noop).toContain("['plan-eng-review', 'plan-design-review'] as const");
expect(noop).toContain("['plan-ceo-review', 'plan-design-review', 'plan-devex-review'] as const");
expect(noop).toContain("['plan-design-review'] as const");
expect(timeoutExpressions('test/skill-e2e-plan-mode-no-op.test.ts')).toEqual(Array(2).fill('CAPTURE_LONG_MS'));
expect(read('test/skill-e2e-plan-ceo-mode-routing.test.ts').match(/^\s*{ mode: '/gm)).toHaveLength(2);
expect(timeoutExpressions('test/skill-e2e-plan-ceo-mode-routing.test.ts')).toEqual(['CAPTURE_LONG_MS']);
expect(timeoutExpressions('test/skill-e2e-plan-eng-plan-mode.test.ts')).toEqual(Array(2).fill('CAPTURE_LONG_MS'));
expect(timeoutExpressions('test/skill-e2e-plan-prosons.test.ts')).toEqual(Array(4).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS'));
expect(timeoutExpressions('test/skill-e2e-plan-format.test.ts')).toEqual(Array(4).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS'));
expect(read('test/skill-e2e-auq-matrix.test.ts').match(/^ skill: '/gm)).toHaveLength(6);
expect(timeoutExpressions('test/skill-e2e-plan-format.test.ts')).toEqual(Array(2).fill('CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS'));
expect(read('test/skill-e2e-auq-matrix.test.ts').match(/^ skill: '/gm)).toHaveLength(5);
expect(timeoutExpressions('test/skill-e2e-auq-matrix.test.ts')).toEqual(['CAPTURE_MS']);
expect(timeoutExpressions('test/skill-e2e-plan.test.ts')).toEqual([
'PTY_MS', 'PTY_MS', 'CAPTURE_LONG_MS', 'CAPTURE_LONG_MS',
test/plan-floor-permission.test.ts
+3 -3
View File
@@ -521,10 +521,10 @@ test('duplicate declarations and existing fixture files cannot be rewritten into
expect(() => createPlanCountFixture(FORCING_FLOOR_CEO,{requestedPlanPath,files:{'gstack-test-plan-ceo-floor.md':'different source'}})).toThrow();
});
test.each(['auq_observed','timeout','throw'])('all four actual floor registrations keep the declared actor and %s outcome', outcome => {
test.each(['auq_observed','timeout','throw'])('all three actual floor registrations keep the declared actor and %s outcome', outcome => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(),'floor-registration-free-'));
const worker = path.join(dir,'registrations.test.ts'), facts = path.join(dir,'calls.jsonl');
const files = ['ceo','eng','design','devex'].map(kind => path.join(ROOT,`test/skill-e2e-plan-${kind}-finding-floor.test.ts`));
const files = ['ceo','design','devex'].map(kind => path.join(ROOT,`test/skill-e2e-plan-${kind}-finding-floor.test.ts`));
try {
for (const file of files) expect(fs.statSync(file).isFile()).toBe(true);
fs.writeFileSync(worker, `
@@ -557,7 +557,7 @@ ${files.map(file => `await import(${JSON.stringify(file)});`).join('\n')}
const output = result.stdout.toString()+result.stderr.toString();
expect(result.exitCode,output).toBe(outcome === 'auq_observed' ? 0 : 1);
expect(fs.readFileSync(facts,'utf8').trim().split('\n').map(line=>JSON.parse(line).kind).sort())
.toEqual(['ceo','design','devex','eng']);
.toEqual(['ceo','design','devex']);
if(outcome !== 'auq_observed') expect(output).toContain(outcome === 'throw' ? 'controlled runner failure' : 'floor test FAILED: outcome=timeout');
} finally {fs.rmSync(dir,{recursive:true,force:true});}
},25_000);
test/plan-mode-evidence.test.ts
+5 -14
View File
@@ -11,10 +11,10 @@ const SECRET = 'fixture-private-diagnostic-value';
// Import the actual paid registration in an isolated Bun process. Only model
// execution is replaced; Bun assertions and the report-file validator are real.
for (const skill of ['design', 'eng'] as const) {
for (const skill of ['design'] as const) {
const PAID_FILE = `test/skill-e2e-plan-${skill}-plan-mode.test.ts`;
const skillName = `plan-${skill}-review`;
for (const scenario of ['success', 'missing-auto-select', 'scope-question', 'bad-report', 'bad-outcome', 'retry', 'unwritable', ...(skill === 'eng' ? ['missing-decisions'] : [])]) {
for (const scenario of ['success', 'missing-auto-select', 'scope-question', 'bad-report', 'bad-outcome', 'retry', 'unwritable']) {
test(`${skill} plan-mode caller preserves its outcome and failure evidence: ${scenario}`, () => {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'design-mode-evidence-'));
const evalDir = path.join(dir, 'artifacts');
@@ -29,7 +29,7 @@ const runnerPath = ${JSON.stringify(path.join(ROOT, 'test/helpers/claude-pty-run
const { assertReportAtBottomIfPlanWritten, planFileHasDecisionsSection } = await import(runnerPath);
const scenario = ${JSON.stringify(scenario)};
const report = ${JSON.stringify(path.join(dir, 'plan.md'))};
fs.writeFileSync(report, '# Plan\\n' + (scenario === 'missing-decisions' ? '' : '## Decisions to confirm\\nPending.\\n')
fs.writeFileSync(report, '# Plan\\n## Decisions to confirm\\nPending.\\n'
+ '## GSTACK REVIEW REPORT\\nDone.\\n' + (scenario === 'bad-report' ? '## Unexpected trailing section\\n' : ''));
let calls = 0;
mock.module(runnerPath, () => ({
@@ -37,10 +37,10 @@ mock.module(runnerPath, () => ({
runPlanSkillObservation: async opts => {
const call = ++calls;
fs.appendFileSync(${JSON.stringify(events)}, JSON.stringify({ call, opts }) + '\\n');
return { outcome: ['bad-report', 'missing-decisions'].includes(scenario) ? 'plan_ready' : scenario === 'bad-outcome' ? 'auto_decided' : 'asked',
return { outcome: scenario === 'bad-report' ? 'plan_ready' : scenario === 'bad-outcome' ? 'auto_decided' : 'asked',
summary: 'Returned observation ' + call, elapsedMs: 42,
evidence: 'Visible terminal tail ' + call + ': ' + process.env.DIAGNOSTIC_API_TOKEN,
...(['bad-report', 'missing-decisions'].includes(scenario) ? { planFile: report } : {}),
...(scenario === 'bad-report' ? { planFile: report } : {}),
proseAUQEverObserved: true, waitingEverObserved: true, tokensObserved: { 'draft-plan': true, 'scope-announcement': false },
scopeGateQuestionObserved: scenario === 'scope-question',
scopeGateAutoSelectObserved: !['missing-auto-select', 'retry', 'unwritable'].includes(scenario) };
@@ -68,14 +68,6 @@ await import(${JSON.stringify(path.join(ROOT, PAID_FILE))});
keys.push('initialPlanContent', 'trackTokens');
expect(opts.trackTokens.length).toBeGreaterThan(0);
for (const token of opts.trackTokens) expect(opts.initialPlanContent).toContain(token);
if (skill === 'eng') {
keys.push('extraArgs', 'cwd', 'env');
expect(opts.extraArgs).toEqual(['--disallowedTools', 'AskUserQuestion']);
// The seeded runner plan reviews its own fixture project, never this checkout.
expect(path.isAbsolute(opts.cwd)).toBe(true);
expect(path.relative(ROOT, opts.cwd).startsWith('..')).toBe(true);
expect(opts.env).toEqual({ DISABLE_AUTOUPDATER: '1', CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' });
}
}
expect(Object.keys(opts).sort()).toEqual(keys.sort());
}
@@ -113,7 +105,6 @@ await import(${JSON.stringify(path.join(ROOT, PAID_FILE))});
if (scenario === 'bad-report') expect(output).toContain('GSTACK REVIEW REPORT contract violation');
if (scenario === 'missing-auto-select') expect(output).toContain('scopeGateAutoSelectObserved');
if (scenario === 'scope-question') expect(output).toContain('scopeGateQuestionObserved');
if (scenario === 'missing-decisions') expect(output).toContain('without a question');
} finally { fs.rmSync(dir, { recursive: true, force: true }); }
});
}
test/qa-supervision-selection.test.ts
+4 -4
View File
@@ -2,15 +2,15 @@ import { expect, test } from 'bun:test';
import { E2E_TOUCHFILES, selectTests } from './helpers/touchfiles';
const ptyIds = [
'plan-ceo-review-plan-mode', 'plan-eng-review-plan-mode', 'plan-design-review-plan-mode-smoke',
'plan-ceo-review-plan-mode', 'plan-design-review-plan-mode-smoke',
'plan-devex-review-plan-mode', 'plan-mode-no-op', 'office-hours-auto-mode',
'auto-decide-preserved', 'plan-ceo-mode-routing', 'plan-design-with-ui-scope', 'plan-eng-finding-floor',
'auto-decide-preserved', 'plan-ceo-mode-routing', 'plan-design-with-ui-scope',
// The two formerly keyless AUQ probes (W1b) import the same SDK capture helper as auq-format-gate, which reaches the PTY runner.
'auq-format-gate', 'auq-consistency', 'auq-verbose-vs-carved-ab', ...['benchmark', 'browse', 'codex', 'context-restore', 'design-consultation', 'design-html', 'design-shotgun', 'document-release', 'investigate', 'land-and-deploy',
'plan-design-review', 'plan-devex-review', 'plan-eng-review', 'qa', 'retro', 'review', 'setup-gbrain', 'spec'].map(skill => `carve-section-loading-${skill}`),
'plan-design-review', 'plan-devex-review', 'qa', 'retro', 'review', 'setup-gbrain', 'spec'].map(skill => `carve-section-loading-${skill}`),
'office-hours-section-loading', 'office-hours-design-draft', 'plan-ceo-section-loading', 'ship-section-loading',
'plan-ceo-finding-floor', 'plan-design-finding-floor', 'plan-devex-finding-floor',
'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact', 'plan-eng-review-artifact-full',
'plan-eng-multi-finding-batching', 'plan-ceo-split-overflow', 'plan-eng-review-artifact',
// The /autoplan guard's live cases drive the same PTY runner (Oct 7 wave).
'autoplan-guard-pty', 'autoplan-long-session',
].sort();
test/skill-e2e-auq-matrix.test.ts
-5
View File
@@ -70,11 +70,6 @@ interface MatrixSkill {
}
const MATRIX: MatrixSkill[] = [
{
skill: 'plan-eng-review',
fixtures: { 'plan.md': FLAWED_PLAN },
scenario: 'Read plan.md — that is the plan to review. It is a standalone plan document, not a codebase. Walk the review until the first AskUserQuestion (a per-issue finding or a scope decision).',
},
{
skill: 'plan-design-review',
fixtures: { 'plan.md': FLAWED_PLAN + '\n## UI\nA new pricing page with a comparison table, plan cards, and an upgrade modal.\n' },
test/skill-e2e-plan-ceo-finding-floor.test.ts
+3 -1
View File
@@ -1,7 +1,9 @@
/**
* /plan-ceo-review AskUserQuestion floor regression (gate, paid, real-PTY).
*
* See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract.
* Contract: a seeded forcing finding must fire at least one AskUserQuestion
* before the review writes its plan or exits plan mode (the /plan-eng-review
* floor that first pinned it retired with that skill, GST-1698).
*/
import { test } from 'bun:test';
test/skill-e2e-plan-design-finding-floor.test.ts
+3 -1
View File
@@ -1,7 +1,9 @@
/**
* /plan-design-review AskUserQuestion floor regression (periodic, paid, real-PTY).
*
* See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract.
* Contract: a seeded forcing finding must fire at least one AskUserQuestion
* before the review writes its plan or exits plan mode (the /plan-eng-review
* floor that first pinned it retired with that skill, GST-1698).
*/
import { test } from 'bun:test';
test/skill-e2e-plan-devex-finding-floor.test.ts
+3 -1
View File
@@ -1,7 +1,9 @@
/**
* /plan-devex-review AskUserQuestion floor regression (gate, paid, real-PTY).
*
* See test/skill-e2e-plan-eng-finding-floor.test.ts for the contract.
* Contract: a seeded forcing finding must fire at least one AskUserQuestion
* before the review writes its plan or exits plan mode (the /plan-eng-review
* floor that first pinned it retired with that skill, GST-1698).
*/
import { test } from 'bun:test';
test/skill-e2e-plan-eng-artifact-full.test.ts
-71
View File
@@ -1,71 +0,0 @@
/**
* Marathon: a fresh /plan-eng-review, from startup through Test review, writes
* its QA test-plan artifact (paid, real-PTY, non-blocking).
*
* The periodic checkpoint (skill-e2e-plan-eng-artifact.test.ts) resumes after
* Scope Challenge and Sections 1-2; this case keeps the whole interactive path.
* Budget: PTY_LONG_MS. Measured on Claude Code 2.1.284: one fresh run passed at
* 498 s; census 37174266054 and a local run both spent 595 s on startup plus six
* one-at-a-time Scope Challenge questions without reaching Test review.
*
* Replaces the composing `plan-eng-review-artifact` case in skill-e2e-plan.test.ts.
* That case told a `claude -p` session to skip the preamble and questions and
* "go straight to the review"; in runs 37151477069, 37158847998 and 37162480720
* (and one local CI-image run) the model wrote the whole review into one file
* without executing the workflow, so it measured composing, not the skill. A
* full interactive run does write the artifact (37158847998's batching PTY run
* wrote eng-review-test-plan-20261003-224617.md during Test review).
*
* This case drives the real interactive /plan-eng-review over the same
* dashboard plan and sources, answers each question with its first option, and
* stops once the artifact exists or the report completes. The requirement is
* unchanged: exactly one new QA test plan under the run's own state root, about
* the reviewed dashboard change.
*/
import { expect, test } from 'bun:test';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { PTY_LONG_MS } from './helpers/eval-budgets';
import { DASHBOARD_PLAN, FIXTURE_PLAN_PATH, SOURCES } from './helpers/plan-eng-artifact-fixture';
import { describeE2ETier } from './helpers/e2e-gate';
import { engFirstReviewAUQ, engSetupAUQ, engStep0Boundary, hasCompletePlanReport, runPlanSkillCounting } from './helpers/claude-pty-runner';
const describeE2E = describeE2ETier('marathon');
describeE2E('/plan-eng-review QA test-plan artifact, full workflow (marathon)', () => {
test('a fresh interactive review reaches Test review and writes one QA test plan', async () => {
const startedAt = Date.now();
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-e2e-plan-eng-artifact-full-'));
const planPath = path.join(tmpDir, 'gstack-test-plan-eng-artifact.md');
try {
const obs = await runPlanSkillCounting({
skillName: 'plan-eng-review',
slashCommand: '/plan-eng-review',
followUpPrompt: DASHBOARD_PLAN.replaceAll(FIXTURE_PLAN_PATH, planPath),
fixtureFiles: SOURCES,
permissionPlanPath: planPath,
isLastStep0AUQ: engStep0Boundary,
isSetupAUQ: engSetupAUQ,
isFirstReviewAUQ: engFirstReviewAUQ,
// The artifact is the only verdict; stop once it exists or the report is complete.
isCollectionComplete: (_transcript, _fingerprints, testPlans) =>
testPlans.length > 0 || hasCompletePlanReport(planPath, startedAt, Date.now()),
reviewCountCeiling: Infinity,
preconfiguredReviewActor: true,
timeoutMs: PTY_LONG_MS,
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
});
const plans = obs.engTestPlans;
console.log(`plan-eng-review-artifact-full: outcome=${obs.outcome} elapsed=${obs.elapsedMs}ms review=${obs.reviewCount} test plans=${plans.map(plan => plan.file).join(', ') || 'none'}`);
if (!['plan_ready', 'completion_summary', 'collection_complete'].includes(obs.outcome)) {
throw new Error(`plan-eng-review-artifact-full: outcome=${obs.outcome} (${obs.summary})\n--- evidence ---\n${obs.evidence}`);
}
expect(plans, `expected one new eng-review test plan; outcome=${obs.outcome}`).toHaveLength(1);
expect(/dashboard|fetchStats|\/api\/stats/i.test(plans[0]!.content), 'the test plan covers the reviewed dashboard change').toBe(true);
} finally {
fs.rmSync(tmpDir, { recursive: true, force: true });
}
}, PTY_LONG_MS + 30_000);
});
test/skill-e2e-plan-eng-artifact.test.ts
+2 -2
View File
@@ -19,8 +19,8 @@
* review, answering each question with its first option, and the case stops
* once the artifact exists or the report completes. The requirement is
* unchanged: exactly one new QA test plan under the run's own state root, about
* the reviewed dashboard change. The fresh start-to-finish path is the marathon
* case in skill-e2e-plan-eng-artifact-full.test.ts.
* the reviewed dashboard change. (The fresh start-to-finish marathon case
* retired with the standalone skill, GST-1698.)
*/
import { expect, test } from 'bun:test';
test/skill-e2e-plan-eng-finding-floor.test.ts
-56
View File
@@ -1,56 +0,0 @@
/**
* /plan-eng-review AskUserQuestion floor regression (periodic, paid, real-PTY).
*
* Catches the May 2026 transcript bug where /plan-eng-review wrote a
* multi-section review plan to ~/.claude/plans/ and called ExitPlanMode
* without firing any AskUserQuestion. See
* `.context/attachments/pasted_text_2026-05-06_10-25-23.txt`.
*
* Uses runPlanSkillFloorCheck — a minimal "did the agent fire ANY AUQ?"
* observer that exits early on the first non-permission numbered-option
* render. See claude-pty-runner.ts for why this is separate from the
* runPlanSkillCounting harness used by periodic finding-count tests.
*
* Tier: periodic. Budget: 10 min (early exit on success ~30-90s typical).
* Cost: ~$0.50-$1.50 per run depending on early-exit timing.
*/
import { test } from 'bun:test';
import { CAPTURE_LONG_MS, PTY_MS } from './helpers/eval-budgets';
import { describeE2ETier } from './helpers/e2e-gate';
import { runPlanSkillFloorCheck } from './helpers/claude-pty-runner';
import { FORCING_FLOOR_ENG } from './fixtures/forcing-finding-seeds';
const describeE2E = describeE2ETier('periodic');
describeE2E('/plan-eng-review AskUserQuestion floor (periodic)', () => {
test(
'seeded forcing finding causes the agent to fire at least one AskUserQuestion',
async () => {
const obs = await runPlanSkillFloorCheck({
skillName: 'plan-eng-review',
slashCommand: '/plan-eng-review',
followUpPrompt: FORCING_FLOOR_ENG,
requestedPlanPath: '/tmp/gstack-test-plan-eng-floor.md',
// LIVE-REPO CWD: PTY session needs the repo cwd — gstack skill
// registry + hermetic pre-trusted dir (hermetic-env trustedDirs).
cwd: process.cwd(),
timeoutMs: CAPTURE_LONG_MS,
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
});
if (obs.outcome !== 'auq_observed') {
throw new Error(
`floor test FAILED: outcome=${obs.outcome} elapsed=${obs.elapsedMs}ms\n` +
`summary: ${obs.summary}\n` +
`If outcome is plan_ready or completion_summary, this is the transcript-bug ` +
`regression — agent reached terminal without firing AskUserQuestion. See ` +
`.context/attachments/pasted_text_2026-05-06_10-25-23.txt.\n` +
`If outcome is timeout, agent may just be slow — re-run or increase budget.\n` +
`--- evidence (last 3KB) ---\n${obs.evidence}`,
);
}
},
PTY_MS,
);
});
test/skill-e2e-plan-eng-plan-mode.test.ts
-183
View File
@@ -1,183 +0,0 @@
/**
* plan-eng-review plan-mode smoke (periodic, paid, real-PTY).
*
* See test/skill-e2e-plan-ceo-plan-mode.test.ts for the shared assertion
* contract. This file exercises the same contract against /plan-eng-review.
*/
import { test, expect } from 'bun:test';
import { spawnSync } from 'node:child_process';
import * as fs from 'node:fs';
import * as os from 'node:os';
import * as path from 'node:path';
import { CAPTURE_MS, CAPTURE_LONG_MS } from './helpers/eval-budgets';
import { describeE2ETier } from './helpers/e2e-gate';
import { assertNoPlanFileDecisions, assertPlanModeWithEvidence, seededPlanTargeted } from './helpers/plan-mode-evidence';
import {
runPlanSkillObservation,
planFileHasDecisionsSection,
assertReportAtBottomIfPlanWritten,
} from './helpers/claude-pty-runner';
const describeE2E = describeE2ETier('periodic');
// SEED_PLAN_FORCING_FINDINGS: 8+ files + custom-vs-builtin smell forces the
// Step 0 complexity check to trigger. Passed via runPlanSkillObservation's
// initialPlanContent (D3-B) so the spawned `claude` actually sees it.
const SEED_PLAN_FORCING_FINDINGS = `
# Parallelize unit tests
## Plan
Build a custom test runner: scripts/test-parallel.ts, scripts/test-shard-impl.ts,
scripts/test-merge-results.ts, scripts/test-progress.ts, scripts/test-watch.ts,
scripts/test-coverage.ts, scripts/test-cli.ts, scripts/test-config.ts.
Add new TestRunner class, new ShardManager class, new ResultMerger class.
Ignore Bun's native --shard flag because we want full control.
## Files
- scripts/test-parallel.ts (new)
- scripts/test-shard-impl.ts (new)
- scripts/test-merge-results.ts (new)
- scripts/test-progress.ts (new)
- scripts/test-watch.ts (new)
- scripts/test-coverage.ts (new)
- scripts/test-cli.ts (new)
- scripts/test-config.ts (new)
- package.json (add scripts)
## Tests
None planned — will add later.
`;
/**
* The seeded plan proposes a custom test runner, so its Step 0 "what already
* exists" pass reads the working tree's test setup. Run it in a small project
* of its own, never the gstack checkout: there the agent explored gstack's own
* sharded runner, and the review's length tracked unrelated runner edits
* (a 300 s timeout in census 37228573062 after a runner refactor).
*/
function createRunnerPlanFixture(): { cwd: string; cleanup(): void } {
const cwd = fs.realpathSync(fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-plan-eng-smoke-')));
const cleanup = () => fs.rmSync(cwd, { recursive: true, force: true });
const files: Record<string, string> = {
'package.json': JSON.stringify({ name: 'tiny-math', private: true, type: 'module', scripts: { test: 'bun test' } }, null, 2) + '\n',
'README.md': '# tiny-math\n\nA small Bun library. `bun test` runs the unit tests in test/.\n',
'src/sum.ts': 'export function sum(values: number[]): number {\n return values.reduce((total, value) => total + value, 0);\n}\n',
'test/sum.test.ts': "import { expect, test } from 'bun:test';\nimport { sum } from '../src/sum';\n\ntest('sums values', () => {\n expect(sum([1, 2, 3])).toBe(6);\n});\n",
};
try {
for (const [name, content] of Object.entries(files)) {
fs.mkdirSync(path.dirname(path.join(cwd, name)), { recursive: true });
fs.writeFileSync(path.join(cwd, name), content);
}
for (const args of [
['init', '-b', 'main'],
['add', '--', ...Object.keys(files)],
['-c', 'user.name=Plan Eng Fixture', '-c', 'user.email=plan-eng@example.test', '-c', 'commit.gpgsign=false',
'commit', '--no-verify', '-m', 'Seed tiny-math'],
['update-ref', 'refs/remotes/origin/main', 'HEAD'],
]) {
const result = spawnSync('git', args, { cwd, encoding: 'utf8', timeout: 10_000 });
if (result.error || result.status !== 0) throw new Error(`Could not initialize the plan-eng fixture: ${result.error?.message ?? result.stderr}`);
}
return { cwd, cleanup };
} catch (error) {
cleanup();
throw error;
}
}
// Seed-only names; seeing one after the slash command shows the review read the seeded plan.
const SEED_PLAN_TOKENS = ['ShardManager', 'ResultMerger', 'test-shard-impl'];
describeE2E('plan-eng-review plan-mode smoke (periodic)', () => {
test('reaches a terminal outcome (asked or plan_ready) without silent writes', async () => {
const obs = await runPlanSkillObservation({
skillName: 'plan-eng-review',
inPlanMode: true,
timeoutMs: CAPTURE_MS,
});
assertPlanModeWithEvidence('plan-eng-review', 'reaches a terminal outcome (asked or plan_ready) without silent writes', obs, () => {
if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') {
throw new Error(
`plan-eng-review plan-mode smoke FAILED: outcome=${obs.outcome}\n` +
`summary: ${obs.summary}\n` +
`elapsed: ${obs.elapsedMs}ms\n` +
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
expect(['asked', 'plan_ready']).toContain(obs.outcome);
assertReportAtBottomIfPlanWritten(obs);
assertNoPlanFileDecisions(obs, planFileHasDecisionsSection);
});
}, CAPTURE_LONG_MS);
// D3-B / D4-B: when a plan with guaranteed-finding-triggering complexity
// is seeded, the skill must ask (mcp AskUserQuestion or the prose fallback)
// before writing findings to the plan. The wrote_findings_before_asking
// outcome catches the transcript bug where the model writes findings to the
// plan before any question renders. A plan-file ## Decisions section is not
// a substitute for asking.
test('STOP gate fires when seeded plan forces Step 0 findings', async () => {
const fixture = createRunnerPlanFixture();
let obs: Awaited<ReturnType<typeof runPlanSkillObservation>>;
try {
obs = await runPlanSkillObservation({
skillName: 'plan-eng-review',
inPlanMode: true,
cwd: fixture.cwd,
// As in auto-decide-preserved's standalone fixture: a first launch in a
// fresh folder can paint a product notice under the composer, which
// the plan-seed composer check (one footer row) never accepts.
env: { DISABLE_AUTOUPDATER: '1', CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC: '1' },
initialPlanContent: SEED_PLAN_FORCING_FINDINGS,
// Force the Conductor-style path: native AUQ disallowed → the model
// must use mcp__*__AskUserQuestion or render the prose fallback
// (both observed as outcome='asked').
extraArgs: ['--disallowedTools', 'AskUserQuestion'],
trackTokens: SEED_PLAN_TOKENS,
timeoutMs: CAPTURE_MS,
});
} finally {
fixture.cleanup();
}
assertPlanModeWithEvidence('plan-eng-review', 'STOP gate fires when seeded plan forces Step 0 findings', obs, () => {
if (
obs.outcome === 'wrote_findings_before_asking' ||
obs.outcome === 'auto_decided' ||
obs.outcome === 'silent_write' ||
obs.outcome === 'exited' ||
obs.outcome === 'timeout'
) {
throw new Error(
`STOP-gate regression: outcome=${obs.outcome}\nsummary: ${obs.summary}\n` +
`elapsed: ${obs.elapsedMs}ms\n` +
`--- evidence (last 2KB) ---\n${obs.evidence}`,
);
}
assertReportAtBottomIfPlanWritten(obs);
assertNoPlanFileDecisions(obs, planFileHasDecisionsSection);
if (obs.outcome !== 'asked') {
throw new Error(
`STOP-gate regression: outcome=${obs.outcome} — the seeded findings reached ` +
`${obs.planFile ?? 'plan_ready'} without a question.\n` +
`--- evidence (last 2KB) ---\n${obs.evidence}`,
);
}
// Plan-mode scope-gate bypass: with a seeded plan in plan mode, the gate
// must NOT render its "What should I review?" menu, and the review must
// target the seeded plan. The announcement's exact wording is not graded.
// Unseeded test 1 keeps its lenient contract: with no plan drafted, the
// "ask as normal" fallback legitimately renders the question.
expect(obs.scopeGateQuestionObserved ?? false).toBe(false);
console.log(`[plan-eng plan-mode] scope announcement observed: ${obs.scopeGateAutoSelectObserved ?? false}; seed tokens: ${JSON.stringify(obs.tokensObserved ?? {})}`);
expect(seededPlanTargeted(obs, SEED_PLAN_TOKENS), 'seeded plan was not targeted (scopeGateAutoSelectObserved or seed-only plan tokens)').toBe(true);
});
}, CAPTURE_LONG_MS);
});
test/skill-e2e-plan-format.test.ts
+3 -145
View File
@@ -1,5 +1,6 @@
/**
* AskUserQuestion format regression test for /plan-ceo-review and /plan-eng-review.
* AskUserQuestion format regression test for /plan-ceo-review (the /plan-eng-review
* cases retired with that skill, GST-1698).
*
* Context: a user on Opus 4.7 reported the RECOMMENDATION line and the
* `Completeness: N/10` per-option score stopped appearing on AskUserQuestion
@@ -82,7 +83,7 @@ We're building a new user dashboard that shows recent activity, notifications, a
- Cache: Redis for dashboard aggregates
`;
function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review' | 'plan-eng-review'): string {
function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review'): string {
const planDir = fs.mkdtempSync(path.join(os.tmpdir(), tmpPrefix));
const run = (cmd: string, args: string[]) =>
spawnSync(cmd, args, { cwd: planDir, stdio: 'pipe', timeout: 5000 });
@@ -107,12 +108,6 @@ function setupPlanDir(tmpPrefix: string, skillName: 'plan-ceo-review' | 'plan-en
return planDir;
}
// The capture instruction passed to every case except the mode and approach menus (D2).
// Tells the agent to dump AskUserQuestion content to a file instead of calling a tool.
function captureInstruction(outFile: string): string {
return `Write the verbatim text of every AskUserQuestion you would have made to ${outFile} (one question per session, full text including options and recommendation line). Do NOT call any tool to ask the user. Do NOT paraphrase — include the exact prose you would have shown. This is a format-capture test, not an interactive session.`;
}
/**
* D2: both approach-menu refusals in census 37195203538 (Fable 5.1,
* `reasoning_extraction`, "duplicating model outputs") were anchored on the
@@ -259,143 +254,6 @@ After writing the file, stop. Do not continue the review.`,
}, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS);
});
// --- Case 3: plan-eng-review coverage-differentiated per-issue AskUserQuestion ---
describeIfSelected('Plan Format — Eng Coverage Issue', ['plan-eng-review-format-coverage'], () => {
let planDir: string;
let outFile: string;
beforeAll(() => {
planDir = setupPlanDir('skill-e2e-plan-format-eng-cov-', 'plan-eng-review');
outFile = path.join(planDir, 'ask-capture.md');
});
afterAll(() => {
try { fs.rmSync(planDir, { recursive: true, force: true }); } catch {}
});
testConcurrentIfSelected('plan-eng-review-format-coverage', async () => {
const judgeMetadata: Pick<EvalTestEntry, 'judge_scores' | 'judge_reasoning'> = {};
await runRecordedOfficeHoursAttempt({
collector: evalCollector, name: '/plan-eng-review-format-coverage', suite: 'Plan Format — Eng Coverage Issue',
model: resolveEvalModel('capture'), budgetMs: CAPTURE_MS,
judgeMetadata,
run: signal => runSkillTest({
signal,
prompt: `Read plan-eng-review/SKILL.md for the review workflow.
Read plan.md — that's the plan to review. This is a standalone plan document, not a codebase — skip any codebase exploration steps.
During your review (Section 3 Test Review is the natural place), generate ONE AskUserQuestion about test coverage depth where the options are clearly coverage-differentiated. For example:
A) Full coverage: happy path + edge cases + error paths (Completeness 10/10)
B) Happy path only (Completeness 7/10)
C) Smoke test (Completeness 3/10)
${captureInstruction(outFile)}
After writing the file with that ONE question, stop. Do not continue the review.`,
workingDirectory: planDir,
maxTurns: 10,
timeout: CAPTURE_MS,
testName: 'plan-eng-review-format-coverage',
runId,
model: resolveEvalModel('capture'),
}),
validate: async (result, signal) => {
logCost('/plan-eng-review format (coverage)', result);
expect(['success', 'error_max_turns']).toContain(result.exitReason);
expect(fs.existsSync(outFile)).toBe(true);
const captured = fs.readFileSync(outFile, 'utf-8');
expect(captured.length).toBeGreaterThan(100);
// Coverage-differentiated: Completeness: N/10 required. Recommendation
// presence checked by the judge.
expect(captured).toMatch(COMPLETENESS_RE);
await assertRecommendationQuality({
captured,
signal,
record: metadata => Object.assign(judgeMetadata, metadata),
evalCollector,
evalId: '/plan-eng-review-format-coverage',
evalTitle: 'Plan Format — Eng Coverage Issue',
result,
passed: ['success', 'error_max_turns'].includes(result.exitReason),
});
},
});
}, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS);
});
// --- Case 4: plan-eng-review kind-differentiated per-issue AskUserQuestion ---
describeIfSelected('Plan Format — Eng Kind Issue', ['plan-eng-review-format-kind'], () => {
let planDir: string;
let outFile: string;
beforeAll(() => {
planDir = setupPlanDir('skill-e2e-plan-format-eng-kind-', 'plan-eng-review');
outFile = path.join(planDir, 'ask-capture.md');
});
afterAll(() => {
try { fs.rmSync(planDir, { recursive: true, force: true }); } catch {}
});
testConcurrentIfSelected('plan-eng-review-format-kind', async () => {
const judgeMetadata: Pick<EvalTestEntry, 'judge_scores' | 'judge_reasoning'> = {};
await runRecordedOfficeHoursAttempt({
collector: evalCollector, name: '/plan-eng-review-format-kind', suite: 'Plan Format — Eng Kind Issue',
model: resolveEvalModel('capture'), budgetMs: CAPTURE_MS,
judgeMetadata,
run: signal => runSkillTest({
signal,
prompt: `Read plan-eng-review/SKILL.md for the review workflow.
Read plan.md — that's the plan to review. This is a standalone plan document, not a codebase — skip any codebase exploration steps.
During your review (Section 1 Architecture), generate ONE AskUserQuestion about an architectural choice where the options differ in kind, not in coverage. For example, "should we use Redis or Postgres for the cache layer?" — the options are different kinds of systems with different tradeoffs, not more-or-less-complete versions of the same thing.
${captureInstruction(outFile)}
After writing the file with that ONE question, stop. Do not continue the review.`,
workingDirectory: planDir,
maxTurns: 10,
timeout: CAPTURE_MS,
testName: 'plan-eng-review-format-kind',
runId,
model: resolveEvalModel('capture'),
}),
validate: async (result, signal) => {
logCost('/plan-eng-review format (kind)', result);
expect(['success', 'error_max_turns']).toContain(result.exitReason);
expect(fs.existsSync(outFile)).toBe(true);
const captured = fs.readFileSync(outFile, 'utf-8');
expect(captured.length).toBeGreaterThan(100);
// Kind-differentiated: Completeness: N/10 must NOT appear, and the brief
// says the options are not comparable on coverage (any wording).
// Recommendation presence checked by the judge.
expect(captured).not.toMatch(COMPLETENESS_RE);
expect(captured).toMatch(KIND_NOTE_RE);
await assertRecommendationQuality({
captured,
signal,
record: metadata => Object.assign(judgeMetadata, metadata),
evalCollector,
evalId: '/plan-eng-review-format-kind',
evalTitle: 'Plan Format — Eng Kind Issue',
result,
passed: ['success', 'error_max_turns'].includes(result.exitReason),
});
},
});
}, CAPTURE_MS + OFFICE_HOURS_BUN_GRACE_MS);
});
afterAll(async () => {
await finalizeEvalCollector(evalCollector);
});
test/skill-e2e-plan-mode-no-op.test.ts
+7 -6
View File
@@ -16,11 +16,12 @@
*
* Coverage grew with the scope-gate bypass (plan-mode auto-select B):
* - plan-ceo-review: original preamble-misfire regression.
* - plan-eng-review / plan-design-review: the bypass must NOT fire outside
* - plan-design-review: the bypass must NOT fire outside
* plan mode (scopeGateAutoSelectObserved stays false), and when the run
* ends in 'asked', the question that fired must be the scope gate itself
* (outside plan mode with no named target, the gate is the FIRST
* question by contract).
* question by contract). (plan-eng-review had the same two checks; they
* retired with that skill, GST-1698.)
* - plan-devex-review: no no-target scope gate, so only the terminal
* outcome and the absent plan-mode reminder are asserted. It reviews a
* pasted developer-facing plan: without one it reviews the checkout's
@@ -90,14 +91,14 @@ script against the task tracker. Prints to stdout; --out writes a file.
`;
describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
for (const skillName of ['plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'plan-devex-review'] as const) {
const hasScopeGate = skillName === 'plan-eng-review' || skillName === 'plan-design-review';
for (const skillName of ['plan-ceo-review', 'plan-design-review', 'plan-devex-review'] as const) {
const hasScopeGate = skillName === 'plan-design-review';
test(`${skillName} reaches a terminal outcome outside plan mode`, async () => {
const obs = await runPlanSkillObservation({
skillName,
inPlanMode: false,
timeoutMs: CAPTURE_MS,
// eng/design: force the prose-fallback path. The unconditional
// design: force the prose-fallback path. The unconditional
// gate-must-ask assert below pins the render shape the detector
// anchors on, and only the --disallowedTools prose fallback makes
// that shape CONTRACTUAL ("use exactly this shape" in the template);
@@ -164,7 +165,7 @@ describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
// review output), proving the target was used rather than the question
// merely skipped. Also the over-trigger guard for the tightened
// "explicit-only" exception wording.
for (const skillName of ['plan-eng-review', 'plan-design-review'] as const) {
for (const skillName of ['plan-design-review'] as const) {
test(`${skillName} skips the scope gate for an explicitly-pasted target`, async () => {
const obs = await runPlanSkillObservation({
skillName,
test/touchfiles.test.ts
+5 -3
View File
@@ -147,8 +147,7 @@ describe('selectTests', () => {
const actual = selectTests(['scripts/resolvers/testing.ts'], E2E_TOUCHFILES);
expect(actual.reason).toBe('diff');
expect(actual.selected.sort()).toEqual(expected);
for (const id of ['plan-eng-multi-finding-batching',
'plan-eng-review-format-coverage', 'ship-section-loading']) {
for (const id of ['plan-eng-multi-finding-batching', 'ship-section-loading']) {
expect(actual.selected).toContain(id);
expect(E2E_TIERS[id]).toBe('periodic');
}
@@ -633,7 +632,10 @@ describe('derived touchfile closure', () => {
};
const maps = [['E2E_TOUCHFILES', E2E_TOUCHFILES], ['LLM_JUDGE_TOUCHFILES', LLM_JUDGE_TOUCHFILES]] as const;
/** Paid files no key selects; they run only by tier or census. */
const KEYLESS_PAID: Record<string, string> = {};
const KEYLESS_PAID: Record<string, string> = {
// Template-literal ids (auq-matrix-${skill}) and no E2E map key; it runs on every periodic sweep. It only looked keyed while its plan-eng-review row quoted that SDK key (row retired with the skill, GST-1698).
'test/skill-e2e-auq-matrix.test.ts': 'periodic self-gated matrix with template-literal ids',
};
test('no free test file is a touchfile', () => {
const listed = maps.flatMap(([name, map]) => Object.entries(map).flatMap(([key, deps]) =>