I1: plan-eng-review-artifact runs the real interactive review in a PTY, replacing the composing fixture

The claude -p case told the model to skip the preamble and questions and go
straight to the review. In 37151477069, 37158847998, 37162480720 and a local
CI-image run it wrote the whole review into one file without executing the
workflow, so it measured composing, not /plan-eng-review. The interactive
batching PTY run in 37158847998 did write the artifact during Test review, so
there is no product defect.

New test/skill-e2e-plan-eng-artifact.test.ts keeps the case id, tier
(periodic), kind (rule) and budget (CAPTURE_LONG_MS): it drives the real
/plan-eng-review over the same dashboard plan and sources, answers each
question with its first option, stops once the QA test plan exists or the
report completes, and asserts the same requirement (exactly one new
eng-review test plan in the run's own state root, about the dashboard change).
The composing case is retired from skill-e2e-plan.test.ts in this commit.
runPlanSkillCounting reports engTestPlans (read before fixture cleanup) and
passes them to isCollectionComplete. One paid run in the CI image (Claude Code
2.1.284): pass, collection_complete at 498 s with one test plan.

The new file takes its own periodic runner (seeded at 498 s), so the periodic
census plans 26 slices: the planner's --max-parallel and both executors' caps
move from 25 to 26 so every slice still starts at once; the paid-file census
pin moves from 106 to 107.
This commit is contained in:
garrytan committed 2026-10-04 00:22:21 +00:00
1 parent 9244437ab2
commit dc09764b84
8 files changed
+142 -150

No files matched your search

+3 -3
View File
@@ -114,7 +114,7 @@ jobs:
- name: Emit run manifest (ALL periodic tests minus reasoned excludes)
env:
EVALS_ALL: "1"
run: EVALS_TIER=periodic bun --no-install run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slice-budget 540 --jobs 2 --max-parallel 25
run: EVALS_TIER=periodic bun --no-install run scripts/test-paid-shards.ts --tier periodic --emit-plan /tmp/paid-plan/manifest.json --slice-budget 540 --jobs 2 --max-parallel 26
# Codex-only slices run on the host-run eval-codex-slices job; the
# container executor runs every other slice.
@@ -172,7 +172,7 @@ jobs:
fail-fast: false
# Every planned slice starts at once; test/evals-workflow-wiring.test.ts
# fails when the live plan outgrows this cap.
max-parallel: 25
max-parallel: 26
matrix:
slice: ${{ fromJSON(needs.plan-slices.outputs.periodic_slices) }}
steps:
@@ -308,7 +308,7 @@ jobs:
packages: read
strategy:
fail-fast: false
max-parallel: 25
max-parallel: 26
matrix:
slice: ${{ fromJSON(needs.plan-slices.outputs.periodic_codex_slices) }}
steps: