Periodic-tier eval surfaced that Opus 4.7 writes "Recommendation: A) SCOPE
EXPANSION because..." (option label, no "Choose" prefix), which the
generate-ask-user-format.ts spec actually mandates — `Recommendation: <choice>
because <reason>` where <choice> is the bare option label. The legacy regex
`/[Rr]ecommendation:[*\s]*Choose/` pinned down a per-skill template-example
phrasing that the canonical spec doesn't require, so it false-failed on
correctly-formatted captures.
judgeRecommendation.present (deterministic regex over the canonical shape)
plus has_because and reason_substance >= 4 cover the recommendation surface
end-to-end. Drop the redundant strict regex from all five wired call sites
(four plan-format cases + new office-hours Phase 4 test).
Verified by re-reading the captured AUQs from both failing periodic runs:
both contained substantive Recommendation lines that the spec accepts and
the judge correctly grades at substance >= 4.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reproduces the production bug: agent in builder mode reaches Phase 4, presents
A/B/C alternatives, writes "Recommendation: C" in chat prose, and starts
editing the design doc immediately — never calls AskUserQuestion. The Phase 4
STOP-gate fix is the production-side change; this test traps regressions.
SDK + captureInstruction pattern (mirrors skill-e2e-plan-format). The PTY
harness can't seed builder mode + accept-premises to reach Phase 4
(runPlanSkillObservation only sends /skill\\r and waits), so we instruct the
agent to dump the verbatim Phase 4 AskUserQuestion to a file and assert on it
directly. The captured file IS the question — no false-pass risk on which
question got asked, since earlier-phase AUQs cannot satisfy the Phase-4-vocab
regex (approach / alternative / architecture / implementation).
Periodic-tier: Phase 4 requires the agent to invent 2-3 distinct architectures,
more open-ended than the 4 plan-format cases. Reclassify to gate if stable.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Garry Tan