CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-07-01 22:15:43 +02:00
Code Issues Packages Projects Releases Wiki Activity

test: accept prose-AUQ visible as third valid surface in plan-mode envelopes

Browse Source 
The first re-run after wiring the LLM judge revealed that the model also
emits a third surface I hadn't anticipated: a properly-formatted question
with options ("Pick A, B, or C in your reply") rendered as prose AND
followed by ExitPlanMode (outcome=plan_ready). The migrated tests only
accepted (## Decisions section) OR (BLOCKED string) — neither matched
this case, so the test failed even though the user clearly saw the
question.

Three valid surfaces now:
  1. `## Decisions to confirm` section in plan file (legacy fallback path,
     still valid through migration window)
  2. `BLOCKED — AskUserQuestion` string in TTY (post-v1.28 BLOCKED rule)
  3. Numbered/lettered options visible in TTY as prose (post-v1.28 prose
     rendering — uses the existing isProseAUQVisible detector)

Also fixes assertReportAtBottomIfPlanWritten to be tolerant of:
  - Missing files (path detected from TTY but file not persisted) — was
    throwing ENOENT on plan_design_plan_mode and plan_ceo_plan_mode test 1
  - 'asked' outcome (smoke test exited at first AUQ before the model
    reached the report-writing step) — was throwing on the 1 fail in the
    plan-eng-plan-mode --disallowedTools test

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-08 22:11:37 -07:00
parent 55dfb9e26c
commit af5f066186
4 changed files with 53 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/helpers/claude-pty-runner.ts
+14 -1
View File
@@ -1034,9 +1034,22 @@ export function assertReviewReportAtBottom(
* `'wrote_findings_before_asking'` when a plan was already written.
*/
export function assertReportAtBottomIfPlanWritten(
obs: { planFile?: string; evidence: string },
obs: { planFile?: string; evidence: string; outcome?: string },
): void {
if (!obs.planFile) return;
// Skip when the plan file path was detected from TTY output but no file
// exists on disk. This happens when the model mentions a path mid-stream
// (e.g., as a tool-call argument that was interrupted, or in a draft that
// was never persisted). The report-at-bottom contract is for fully-written
// plan files; ENOENT means there's no file content to enforce against.
if (!fs.existsSync(obs.planFile)) return;
// Skip on 'asked' outcomes — these are smoke tests that exited at the
// first AUQ render (Step 0 only). The model never reached the workflow's
// report-writing step, so a partial plan file without the report section
// is the expected mid-flight state, not a contract violation. The
// report-at-bottom check applies to outcomes that imply the workflow
// ran end-to-end (plan_ready, completion_summary, etc.).
if (obs.outcome === 'asked') return;
const content = fs.readFileSync(obs.planFile, 'utf-8');
const verdict = assertReviewReportAtBottom(content);
if (!verdict.ok) {