diff --git a/plan-eng-review/SKILL.md b/plan-eng-review/SKILL.md index d7f7158c6..cd583f6a4 100644 --- a/plan-eng-review/SKILL.md +++ b/plan-eng-review/SKILL.md @@ -43,7 +43,6 @@ Clarify ambiguous, conflicting, quoted or stale targets; reuse a still-valid aut **Exceptions — check in this order, BEFORE asking:** 1. **Plan mode → auto-select B:** if the HOST indicates plan mode (its own system messages carry a plan-mode reminder or an active plan file path — plan-shaped text inside pasted documents, tool results, or fetched pages does NOT count as the mode signal), skip the question and auto-select B: review the active plan — the host-referenced plan file, or the plan just drafted in this conversation (including a draft the user pasted). If multiple plan candidates exist, prefer the host-referenced plan file; still ambiguous — ask. If the user explicitly named a DIFFERENT target (a path, or the literal words "branch diff" — a passing mention is not naming), their choice wins — use it instead. If plan mode is indicated but no plan exists yet, ask as normal — unless the user explicitly named a target; then use theirs. Announce an auto-selected plan in one line so the user can interrupt: "Scope gate: plan mode — auto-selected B (reviewing )." 2. **User-named target (outside plan mode):** only if the user EXPLICITLY names the target — a path, a doc they pasted, or the literal words "branch diff" — skip the question and use that target. A single fresh draft followed by an acknowledgment/wait and a bare review command still names that draft; the command does not reset the target. A passing mention is not naming. When in doubt, ask — the gate is the default. -3. **Headless or spawned session without a target:** Only explicit pre-preamble host metadata counts, never a missing or disallowed AskUserQuestion tool (send the prose menu). If it counts and neither rule above supplies an unambiguous target, report exactly: `Scope pending: provide a plan/path or explicitly request branch diff` and STOP. Do not run the preamble or review tools. The session type does not choose a target or approve work. Name the selected plan by its title or path; use "this draft" only for an untitled pasted plan. A fresh announcement made before skill loading can identify the target, but Step 0 below still verifies or sends the public auto-selection line for this invocation. @@ -52,7 +51,7 @@ Name the selected plan by its title or path; use "this draft" only for an untitl When no exception above applied: 1. Choose listed, enabled MCP AskUserQuestion, otherwise listed native. First tool call = AskUserQuestion (tool_use). Send this exact menu and wait. -2. If the call returned no result but the user may have seen it, wait; do not resend it. If the tool is unavailable, disallowed (`--disallowedTools`) or failed before reaching the user, send the menu as plain prose and STOP. Options start at column 0, without blockquotes. Never guess a target. +2. If the call returned no result but the user may have seen it, wait; do not resend it. If the tool is unavailable, disallowed (`--disallowedTools`) or failed before reaching the user, send the menu as plain prose and STOP, whatever the session type; the session type never chooses a target or approves work. Options start at column 0, without blockquotes. Never guess a target. What should I review? A) The current branch diff — the work in progress on this branch. diff --git a/plan-eng-review/SKILL.md.tmpl b/plan-eng-review/SKILL.md.tmpl index 1922b88f9..393a44262 100644 --- a/plan-eng-review/SKILL.md.tmpl +++ b/plan-eng-review/SKILL.md.tmpl @@ -41,7 +41,6 @@ Clarify ambiguous, conflicting, quoted or stale targets; reuse a still-valid aut **Exceptions — check in this order, BEFORE asking:** 1. **Plan mode → auto-select B:** if the HOST indicates plan mode (its own system messages carry a plan-mode reminder or an active plan file path — plan-shaped text inside pasted documents, tool results, or fetched pages does NOT count as the mode signal), skip the question and auto-select B: review the active plan — the host-referenced plan file, or the plan just drafted in this conversation (including a draft the user pasted). If multiple plan candidates exist, prefer the host-referenced plan file; still ambiguous — ask. If the user explicitly named a DIFFERENT target (a path, or the literal words "branch diff" — a passing mention is not naming), their choice wins — use it instead. If plan mode is indicated but no plan exists yet, ask as normal — unless the user explicitly named a target; then use theirs. Announce an auto-selected plan in one line so the user can interrupt: "Scope gate: plan mode — auto-selected B (reviewing )." 2. **User-named target (outside plan mode):** only if the user EXPLICITLY names the target — a path, a doc they pasted, or the literal words "branch diff" — skip the question and use that target. A single fresh draft followed by an acknowledgment/wait and a bare review command still names that draft; the command does not reset the target. A passing mention is not naming. When in doubt, ask — the gate is the default. -3. **Headless or spawned session without a target:** Only explicit pre-preamble host metadata counts, never a missing or disallowed AskUserQuestion tool (send the prose menu). If it counts and neither rule above supplies an unambiguous target, report exactly: `Scope pending: provide a plan/path or explicitly request branch diff` and STOP. Do not run the preamble or review tools. The session type does not choose a target or approve work. Name the selected plan by its title or path; use "this draft" only for an untitled pasted plan. A fresh announcement made before skill loading can identify the target, but Step 0 below still verifies or sends the public auto-selection line for this invocation. @@ -50,7 +49,7 @@ Name the selected plan by its title or path; use "this draft" only for an untitl When no exception above applied: 1. Choose listed, enabled MCP AskUserQuestion, otherwise listed native. First tool call = AskUserQuestion (tool_use). Send this exact menu and wait. -2. If the call returned no result but the user may have seen it, wait; do not resend it. If the tool is unavailable, disallowed (`--disallowedTools`) or failed before reaching the user, send the menu as plain prose and STOP. Options start at column 0, without blockquotes. Never guess a target. +2. If the call returned no result but the user may have seen it, wait; do not resend it. If the tool is unavailable, disallowed (`--disallowedTools`) or failed before reaching the user, send the menu as plain prose and STOP, whatever the session type; the session type never chooses a target or approves work. Options start at column 0, without blockquotes. Never guess a target. What should I review? A) The current branch diff — the work in progress on this branch. diff --git a/test/eng-seeded-completion-ai.test.ts b/test/eng-seeded-completion-ai.test.ts index feff4a337..ec5778c59 100644 --- a/test/eng-seeded-completion-ai.test.ts +++ b/test/eng-seeded-completion-ai.test.ts @@ -140,7 +140,7 @@ async function mockedObservation(frames: string[], verdict: 'waiting' | 'working isUnknownSlashCommandVisible: predicates.isUnknownSlashCommandVisible, isScopeGateQuestionVisible: predicates.isScopeGateQuestionVisible, isScopeGateAutoSelectVisible: predicates.isScopeGateAutoSelectVisible, - classifyVisible, extractPlanFilePath, findNativeAutoDecision: () => null, + classifyVisible, extractPlanFilePath, idleTurnEnd: predicates.idleTurnEnd, findNativeAutoDecision: () => null, judgePtyState: () => { judged++; return { state: verdict, reasoning: 'synthetic current-frame verdict' }; }, runPtySession: predicates.runPtySession, }; diff --git a/test/fixtures/pty-idle-turn-end.json b/test/fixtures/pty-idle-turn-end.json new file mode 100644 index 000000000..d6d7d1908 --- /dev/null +++ b/test/fixtures/pty-idle-turn-end.json @@ -0,0 +1,27 @@ +{ + "provenance": "terminal.visible.log captures (session.visibleSince(command mark), byte-exact, ESC/CR residue kept) from paid-slice-5-a1 artifacts of test/skill-e2e-plan-mode-no-op.test.ts.", + "scopePendingIdle": { + "runId": 37082906717, + "headSha": "988e985d", + "testName": "plan-eng-review reaches a terminal outcome outside plan mode", + "recordedPassed": false, + "note": "Whole capture. --disallowedTools AskUserQuestion; the model printed the headless pending line instead of the prose menu, the turn ended and the harness waited out 300s.", + "visible": "\u000f\u001b[?2004h\u001b[?1000h\u001b[?1002h\u001b[?1003h\u001b[?1006h\u001b[?25l\n❯ /plan-eng-review \n✻Pontificating… \n❯ \nesc to interrupt · ← for agents\u001b[?25h\u001b[?25l\n /plan-eng-review\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽…\u001b[?25h\u001b[?25l\ng\u001b[?25h\u001b[?25l\n✻n\u001b[?25h\u001b[?25l\n✶i…\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢tg\u001b[?25h\u001b[?25l\n·an\u001b[?25h\u001b[?25l\n\u001b[?25h\u001b[?25l\nci\u001b[?25h\u001b[?25l\n✢it\u001b[?25h\u001b[?25l\n*fa\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻ic\u001b[?25h\u001b[?25l\n✽ti\u001b[?25h\u001b[?25l\nnf\u001b[?25h\u001b[?25l\noi\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✶Pt\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢n\u001b[?25h\u001b[?25l\n·o\u001b[?25h\u001b[?25l\nP\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\nCmposing… \u001b[?25h\u001b[?25l\n(7s · thinking)\u001b[?25h\u001b[?25l\n✽sgthinking\u001b[?25h\u001b[?25l\nonthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻pithinking\u001b[?25h\u001b[?25l\n8thinking\u001b[?25h\u001b[?25l\n✶ms\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢oothinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n·Cpthinking\u001b[?25h\u001b[?25l\nmthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢othinking\u001b[?25h\u001b[?25l\n9thinking\u001b[?25h\u001b[?25l\n*C\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n10s · thinking)\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢thinking\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢1thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻2thinking\u001b[?25h\u001b[?25l\n✶thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢thinking\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n…thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢g3thinking\u001b[?25h\u001b[?25l\n*thinking\u001b[?25h\u001b[?25l\nn\u001b[?25h\u001b[?25l\n✶i…\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✽sgthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nonthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻pi4thinking\u001b[?25h\u001b[?25l\n✶thinking\u001b[?25h\u001b[?25l\n*ms\u001b[?25h\u001b[?25l\noo\u001b[?25h\u001b[?25l\n✢thinking\u001b[?25h\u001b[?25l\n·Cpthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nmthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢o5thinking\u001b[?25h\u001b[?25l\n*Cthinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻6thinking\u001b[?25h\u001b[?25l\n✶thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢7thinking\u001b[?25h\u001b[?25l\n*thinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✽still thinking)\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻8still thinking\u001b[?25h\u001b[?25l\n✶still thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n·still thinking\u001b[?25h\u001b[?25l\n·Composing… still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n9still thinking\u001b[?25h\u001b[?25l\n*still thinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✽still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻Composing… still thinking\u001b[?25h\u001b[?25l\n20\u001b[?25h\u001b[?25l\n✶still thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n·still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n1\u001b[?25h\u001b[?25l\n*still thinking\u001b[?25h\u001b[?25l\n✶Composing… still thinking\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✽still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n2still thinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n✢Composing… still thinking\u001b[?25h\u001b[?25l\n·still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n✢3still thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✽still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✽Composing… still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✻Composing… 4↓ 1.2k tokens · thinking)\u001b[?25h\u001b[?25l\n●Scopepending:provideaplan/pathorexplicitlyrequestbranchdiff\n✻Cooked for 24s · done 12:46 AM\n\n❯ \n← for agents\u001b[?25h" + }, + "apiErrorIdle": { + "runId": 37077373640, + "headSha": "6a6aa32", + "testName": "plan-design-review skips the scope gate for an explicitly-pasted target", + "recordedPassed": false, + "note": "Last 4000 characters. Provider transport error mid-response, then the idle prompt; the harness waited out 300s.", + "visible": " the save model and rollback\nbehvior, the switchmarkupcontract,thetokenmappingforbothcolormodes,themobilelayout,andatestlistthat\ncoverseachstateratherthanjust\"renders\".\n\n0B: DESIGN.md status.All desin dcision will b calibratd against your stated designyste.\n\n0C: existing leverage. The target codbase isnotinthischeckout,soIcannotpointatanexistingswitchor\nsettingssectioncomponenttoreuse.Theplanshouldnameoneifitexists,sinceasecondtogglestyleinasettings\npage is the fastest way to look careless. From DESIGN.md: DM Sns 14px fothe label, 13px captionforhelpertext,\n8pxradius,amber-500ondarkandamber-600onlightfortheonstate,150mshovertransition,andsemanticerrorred\nfor failure copy.\n\n·Philosophizing… (1m 8s · ↓4.2k tokens)\n ⎿  Tip: Use /btw to ask a quick side question without interrupting Claude's current work\u001b[?25h\u001b[?25l\nop\u001b[?25h\u001b[?25l\n9\u001b[?25h\u001b[?25l\n✢lo\u001b[?25h\u001b[?25l\n*is\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻ho\u001b[?25h\u001b[?25l\n✽Pl\u001b[?25h\u001b[?25l\ni\u001b[?25h\u001b[?25l\n10s · ↓ 4.3k tokens)\u001b[?25h\u001b[?25l\n✻h\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\nP\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n1\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽\u001b[?25h\u001b[?25l\n24\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n3\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽\u001b[?25h\u001b[?25l\n…\u001b[?25h\u001b[?25l\n45\u001b[?25h\u001b[?25l\n✻g\u001b[?25h\u001b[?25l\n✶n\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢i…\u001b[?25h\u001b[?25l\n·zg\u001b[?25h\u001b[?25l\nin5\u001b[?25h\u001b[?25l\n✢hi\u001b[?25h\u001b[?25l\n*pz\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻oi\u001b[?25h\u001b[?25l\n✽sh\u001b[?25h\u001b[?25l\nop6\u001b[?25h\u001b[?25l\nlo\u001b[?25h\u001b[?25l\n✻6\u001b[?25h\u001b[?25l\n✶is\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢ho\u001b[?25h\u001b[?25l\n·Pl\u001b[?25h\u001b[?25l\ni7\u001b[?25h\u001b[?25l\n✢h\u001b[?25h\u001b[?25l\n*P\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽\u001b[?25h\u001b[?25l\n8\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n9\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n7\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽\u001b[?25h\u001b[?25l\n20\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n…\u001b[?25h\u001b[?25lStep 0A: initial design rating: 3/10\n\nTheplannamesafile,arouteandatest.Itneversayswhattheusersees.Foraone-togglepanel,thatsilence\nhides lmost every dcision hat matter:\n\n-Nolabelorhelpercopy.\"Weeklyexportemails\"tellstheusernothingaboutwhichday,whichaddress,orwhatisin\n the export.\n-No states. Saving,saved,savefailed,and\"noverifiedemailonfile\"areeachadistinctscreentheuserwillhit.\nfeedback contract. Does thtoggle flipoptimistically ndroll back onrror,or wait for theserver? Is there a\n \"Savd\" confirmation or a toast?\n-No placemet. Where in the setingspagehierarchydoesthispanelsit,andwhatisitsheadinglevelrelativeto\n the neighbors?\n-No accessibilityspec.Atoggleneedstheswitchrole,avisiblefocusring,a44pxtargetandalabelthescreen\n reader reads as a question withan answer.\n-No rsponsive note. One toggle in a ro isfineondesktopandoftenbreaksintoamisalignedstackonnarrow\n width.\n-No okenmappingtoDESIGN.md.Whichamberfortheonstateineachmode,whichradius,whichfontsizeforthe\n label ad helper.\n\nA10forthisplanisshort,maybe25linesofplantext:thepanel'sheadingandone-sentencedescription,theexact\nlabel and elper copyincluding the next send date,he five stts with their copy, th save model and rollback\nbehavior,the switch markup contract, the token mappingor boh color modes, the mobillyout, and a test listthat\ncovers each state rathe than just \"rendrs\".\n\n0B: DESIGN.md status.Alldesigndecisionswillbecalibratedagainstyourstateddesignsystem.\n\n0C: existing leverage.Thetargetcodebaseisnotinthischeckout,soIcannotpointatanexistingswitchor\nsettings section componento ruse. The planshouldnameon if itexists, since a second toggle style in a settings\npage is th fastest way tolok careless. From DESIGN.md:DM Sans 14p for the label, 13pxcaption for helper text,\n8px radius,amber-500 on dark and ambr-600 nlight for theon tate,150ms hover transition, andsemantic error red\nforfilrecopy.\n\n●API Error: Connection lost mid-response. The response above may be incomplete.\n\n✻ Sautéedfor1m 20s · done11:34PM\n❯ \n← for agents\u001b[?25h" + }, + "menuThenIdle": { + "runId": 37077373640, + "headSha": "6a6aa32", + "testName": "plan-eng-review reaches a terminal outcome outside plan mode", + "recordedPassed": true, + "note": "Whole capture. The pending line followed by the full prose menu, then the idle prompt; classified 'asked'.", + "visible": "\u000f\u001b[?2004h\u001b[?1000h\u001b[?1002h\u001b[?1003h\u001b[?1006h\u001b[?25l\n❯ /plan-eng-review \n*Computing… \n❯ \nesc to interrupt · ← for agents\u001b[?25h\u001b[?25l\n /plan-eng-review\u001b[?25h\u001b[?25l\n✢\u001b[?25h\u001b[?25l\n·\u001b[?25h\u001b[?25l\n…\u001b[?25h\u001b[?25l\n ● high · /effort\u001b[?25h\u001b[?25l\n✢g\u001b[?25h\u001b[?25l\n*n\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\ni…\u001b[?25h\u001b[?25l\n✻\u001b[?25h\u001b[?25l\n✽tg\u001b[?25h\u001b[?25l\nun\u001b[?25h\u001b[?25l\n✻pi\u001b[?25h\u001b[?25l\nGallop\u001b[?25h\u001b[?25l\n(1s · thinking)\u001b[?25h\u001b[?25l\n✶lpthinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢ao2\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n·Glthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nlthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢athinking\u001b[?25h\u001b[?25l\n*Gthinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻3\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✶thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢4\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n*thinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻5\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✶thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢6thinking\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\n…thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢gthinking\u001b[?25h\u001b[?25l\n*nthinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻i…7thinking\u001b[?25h\u001b[?25l\n✽pgthinking\u001b[?25h\u001b[?25l\nonthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻lithinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✶lp\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢ao8thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n·Glthinking\u001b[?25h\u001b[?25l\nlthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢athinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n*G\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻9thinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢10s · thinking)\u001b[?25h\u001b[?25l\n·thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✢thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n✻1thinking\u001b[?25h\u001b[?25l\n✽thinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\nthinking\u001b[?25h\u001b[?25l\n✻thinking\u001b[?25h\u001b[?25l\n✶still thinking)\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n✢2still thinking\u001b[?25h\u001b[?25l\n·still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n*still thinking\u001b[?25h\u001b[?25l\n*Galloping… still thinking\u001b[?25h\u001b[?25l\n✶3\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✽still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✶still thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n4\u001b[?25h\u001b[?25l\n✢Galloping… still thinking\u001b[?25h\u001b[?25l\n·still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✢still thinking\u001b[?25h\u001b[?25l\n*still thinking\u001b[?25h\u001b[?25l\n✶\u001b[?25h\u001b[?25l\n5\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✽still thinking\u001b[?25h\u001b[?25l\n✽Galloping… still thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\nstill thinking\u001b[?25h\u001b[?25l\n✻still thinking\u001b[?25h\u001b[?25l\n✶still thinking\u001b[?25h\u001b[?25l\n*\u001b[?25h\u001b[?25l\n*Galloping… ↓ 1.1k tokens · thinking)\u001b[?25h\u001b[?25l\nthought for 14s)\u001b[?25h\u001b[?25l\n✢6\u001b[?25h\u001b[?25l\n●\u001b[?25h\u001b[?25l\nScopepending:provideaplan/pathorexplicitlyrequestbranchdiff\n\u001b[?25h\u001b[?25l\nNoreviewtargetwasnamedandnoplanisactiveinthissession,soIcan'tpickoneforyou.Whenyoureply,tellme\u001b[?25h\u001b[?25l\nwhichofthesetoreview:\u001b[?25h\u001b[?25l\nWhatshouldIreview?\u001b[?25h\u001b[?25l\nA)Thecurrentbranchdiff—theworkinprogressonthisbranch.\u001b[?25h\u001b[?25l\nB)AplanordesigndocI'llpasteorpointyouto.\u001b[?25h\u001b[?25l\nC)Aspecificfile,directory,orpath.\u001b[?25h\u001b[?25l\nRecommendation:Awhenabranchdiffexists,otherwiseB.ReplywithA,B,orC.STOPandwaitfortheanswer.\n✻Brewed for 17s · done 11:31 PM\n❯ \n← for agents\u001b[?25h" + } +} \ No newline at end of file diff --git a/test/helpers/auq-native-capture.ts b/test/helpers/auq-native-capture.ts index 49eee5c44..80cbe8d6c 100644 --- a/test/helpers/auq-native-capture.ts +++ b/test/helpers/auq-native-capture.ts @@ -5,6 +5,7 @@ import * as path from 'node:path'; import { randomUUID } from 'node:crypto'; import { resolveEvalModel } from '../../lib/eval-model'; import { launchClaudePty, capturePlanCountQuestion, parseNumberedOptions, type ClaudePtySession } from './claude-pty-runner'; +import { API_ERROR_PANEL } from './pty/screen'; import { buildSeedConfig, isHermeticEnabled } from './hermetic-env'; import { getProjectEvalDir } from './eval-store'; import { readFirstPendingQuestionForDisplay, pendingQuestionRecorderStatus } from './plan-count-pending-question'; @@ -241,7 +242,7 @@ export function displayedNativeAuq(screen: string, call: NativePlanQuestionCall /** Read only a public CLI error panel; never inspect private journal blocks. */ export function nativeAuqPublicError(screen: string): string | undefined { - const match = /(?:^|\n)[\t │┃]*(?:[⎿●⏺]\s*)?API Error:[\s\S]*/i.exec(screen); + const match = new RegExp(`${API_ERROR_PANEL.source}[\\s\\S]*`, 'i').exec(screen); return match?.[0].trim().slice(0, 2000); } diff --git a/test/helpers/claude-pty-runner.runners.unit.test.ts b/test/helpers/claude-pty-runner.runners.unit.test.ts index ab32b091f..c39de2064 100644 --- a/test/helpers/claude-pty-runner.runners.unit.test.ts +++ b/test/helpers/claude-pty-runner.runners.unit.test.ts @@ -84,6 +84,50 @@ describe('runPlanSkillObservation through the fake driver', () => { }); }); +describe('runPlanSkillObservation idle turn end (captured CI screens)', () => { + const captures = JSON.parse(fs.readFileSync(new URL('../fixtures/pty-idle-turn-end.json', import.meta.url), 'utf8')); + const observe = (frames: FakeFrame[], opts: { timeoutMs: number; requireProseEvidence?: boolean }) => withConfigDir(async () => { + const fake = createFakePtyDriver({ frames: [{ screen: IDLE }, ...frames] }); + return runPlanSkillObservation({ skillName: 'plan-eng-review', inPlanMode: false, model: 'fake-model', driver: fake.driver, + ...(opts.requireProseEvidence ? { extraArgs: ['--disallowedTools', 'AskUserQuestion'], requireProseEvidence: true } : {}), timeoutMs: opts.timeoutMs }); + }); + const slash = (screen: string): FakeFrame => ({ onInput: '/plan-eng-review\r', screen }); + + test('the 988e985 idle turn end stops promptly as a failure that names it', async () => { + const obs = await observe([slash(captures.scopePendingIdle.visible)], { timeoutMs: 300_000, requireProseEvidence: true }); + expect(obs.outcome).toBe('timeout'); + expect(obs.elapsedMs).toBeLessThan(30_000); + expect(obs.summary).toContain('the CLI turn ended at an idle prompt (✻Cooked for 24s · done 12:46 AM) with no question, plan or write'); + expect(obs.summary).toContain('instead of waiting out the 300000ms budget'); + expect(obs.scopeGateQuestionObserved).toBe(false); + }); + + test('a provider error panel is named in the failure', async () => { + const obs = await observe([slash(captures.apiErrorIdle.visible)], { timeoutMs: 300_000, requireProseEvidence: true }); + expect(obs.outcome).toBe('timeout'); + expect(obs.elapsedMs).toBeLessThan(30_000); + expect(obs.summary).toContain('after its public error panel: ●API Error: Connection lost mid-response.'); + }); + + test('negative control: a visible prose menu before the idle prompt is still asked', async () => { + const obs = await observe([slash(captures.menuThenIdle.visible)], { timeoutMs: 300_000, requireProseEvidence: true }); + expect(obs.outcome).toBe('asked'); + expect(obs.scopeGateQuestionObserved).toBe(true); + }); + + test('negative controls: thinking, a screen that changes before settling, and an unconsulted judge keep the deadline', async () => { + const scope: string = captures.scopePendingIdle.visible; + const thinking = await observe([slash(scope.slice(0, scope.indexOf('●Scope')))], { timeoutMs: 40_000, requireProseEvidence: true }); + expect(thinking).toMatchObject({ outcome: 'timeout', summary: 'no terminal outcome within 40000ms', elapsedMs: 40_000 }); + const resumed = await observe([slash(scope), { afterMs: 6_000, screen: '✶ Thinking… (30s · thinking)' }], { timeoutMs: 40_000, requireProseEvidence: true }); + expect(resumed).toMatchObject({ outcome: 'timeout', summary: 'no terminal outcome within 40000ms', elapsedMs: 40_000 }); + // Without requireProseEvidence the judge could still credit 'asked'; it + // first runs after 60s, so a shorter budget never stops early. + const judgeLive = await observe([slash(scope)], { timeoutMs: 50_000 }); + expect(judgeLive).toMatchObject({ outcome: 'timeout', summary: 'no terminal outcome within 50000ms', elapsedMs: 50_000 }); + }); +}); + describe('runPlanSkillCounting through the fake driver', () => { const count = (frames: FakeFrame[], seedTranscript = true) => withConfigDir(async configDir => { const fake = createFakePtyDriver({ configDir, frames: [ diff --git a/test/helpers/claude-pty-runner.screen.unit.test.ts b/test/helpers/claude-pty-runner.screen.unit.test.ts index 12fb268f6..92cd4539b 100644 --- a/test/helpers/claude-pty-runner.screen.unit.test.ts +++ b/test/helpers/claude-pty-runner.screen.unit.test.ts @@ -11,6 +11,7 @@ import { isAutoDecidedVisible, classifyVisible, classifyPlanCountFrame, + idleTurnEnd, } from './claude-pty-runner'; describe('saved preference annotation', () => { @@ -225,3 +226,35 @@ describe('isNumberedOptionListVisible', () => { expect(isPermissionDialogVisible(sample)).toBe(true); }); }); + +describe('idle turn end (captured CI screens)', () => { + const captures = JSON.parse(readFileSync(new URL('../fixtures/pty-idle-turn-end.json', import.meta.url), 'utf8')); + const scope: string = captures.scopePendingIdle.visible; + const api: string = captures.apiErrorIdle.visible; + + test('the 988e985 pending line, end-of-turn line and empty prompt are an idle turn end', () => { + expect(idleTurnEnd(scope)).toEqual({ done: '✻Cooked for 24s · done 12:46 AM' }); + expect(classifyVisible(scope)).toBeNull(); + }); + + test('the 6a6aa32 provider error panel is reported with its idle turn end', () => { + expect(idleTurnEnd(api)).toEqual({ done: '✻ Sautéedfor1m 20s · done11:34PM', + apiError: '●API Error: Connection lost mid-response. The response above may be incomplete.' }); + }); + + test('negative controls: still thinking, a resumed turn, extra footer text and an open question are not idle', () => { + expect(idleTurnEnd(scope.slice(0, scope.indexOf('●Scope')))).toBeNull(); + expect(idleTurnEnd(scope + '\r✶ Thinking… (3s · thinking)')).toBeNull(); + const footer = scope.lastIndexOf('← for agents'); + expect(idleTurnEnd(`${scope.slice(0, footer)}← for agents · 1 background task${scope.slice(footer + '← for agents'.length)}`)).toBeNull(); + const question = '☐ Scope\nWhat should I review?\n❯ 1. The current branch diff\n 2. A plan or design doc\nEnter to select · ↑/↓ to navigate · Esc to cancel'; + expect(idleTurnEnd(question)).toBeNull(); + expect(idleTurnEnd(scope + '\r' + question)).toBeNull(); + }); + + test('an API Error mention in prose or in an earlier turn is not this turn\'s error panel', () => { + expect(idleTurnEnd('The docs mention API Error: as an example.\r✻ Worked for 3s · done 1:00 PM\r❯ \r← for agents')).toEqual({ done: '✻ Worked for 3s · done 1:00 PM' }); + const earlier = api.replace(/✻ Sautéedfor1m 20s · done11:34PM[\s\S]*$/, '✻ Sautéedfor1m 20s · done11:34PM\r❯ /plan-design-review\r●Done.\r✻ Worked for 2s · done 11:35PM\r❯ \r← for agents'); + expect(idleTurnEnd(earlier)).toEqual({ done: '✻ Worked for 2s · done 11:35PM' }); + }); +}); diff --git a/test/helpers/claude-pty-runner.ts b/test/helpers/claude-pty-runner.ts index 7dcb71dd7..6caaf6141 100644 --- a/test/helpers/claude-pty-runner.ts +++ b/test/helpers/claude-pty-runner.ts @@ -8,7 +8,7 @@ * Moved from the former 5,047-line claude-pty-runner.ts. */ export { resolveClaudeBinary } from './pty/binary'; -export { stripAnsi, isRejectedSlashCommand, isPlanReadyVisible, isAutoDecidedVisible, extractPlanFilePath, planFileHasDecisionsSection, TAIL_SCAN_BYTES, isPermissionDialogVisible, stripPtyResidue, isNumberedOptionListVisible } from './pty/screen'; +export { stripAnsi, isRejectedSlashCommand, isPlanReadyVisible, isAutoDecidedVisible, extractPlanFilePath, planFileHasDecisionsSection, TAIL_SCAN_BYTES, isPermissionDialogVisible, stripPtyResidue, isNumberedOptionListVisible, API_ERROR_PANEL, idleTurnEnd } from './pty/screen'; export { selectPtyNumberedOption, launchClaudePty } from './pty/launch'; export type { ClaudePtyOptions, ClaudePtySession } from './pty/launch'; export { runPtySession, realPtyDriver } from './pty/session'; diff --git a/test/helpers/pty/runners/observation.ts b/test/helpers/pty/runners/observation.ts index b07f22c53..032298c30 100644 --- a/test/helpers/pty/runners/observation.ts +++ b/test/helpers/pty/runners/observation.ts @@ -14,7 +14,7 @@ import { classifyVisible, isProseAUQVisible, isScopeGateAutoSelectVisible, isSco import { judgePtyState, logPtySnapshot } from '../judge'; import type { PtyStateVerdict } from '../judge'; import type { ClaudePtySession } from '../launch'; -import { extractPlanFilePath, isNumberedOptionListVisible, isPermissionDialogVisible, isPlanReadyVisible, isRejectedSlashCommand } from '../screen'; +import { extractPlanFilePath, idleTurnEnd, isNumberedOptionListVisible, isPermissionDialogVisible, isPlanReadyVisible, isRejectedSlashCommand } from '../screen'; import { realPtyDriver, runPtySession, type PtyDriver, type PtyStep } from '../session'; // --------------------------------------------------------------------------- @@ -46,7 +46,10 @@ export interface PlanSkillObservation { * the plan file was rewritten with findings before any * AskUserQuestion render (the May-2026 transcript bug) * - 'exited' — claude process died before any of the above - * - 'timeout' — none of the above within budget + * - 'timeout' — none of the above within budget, or the CLI turn + * ended at an idle prompt with none of the above and + * nothing left that could still classify (the summary + * says which; see IDLE_SETTLE_MS) */ outcome: | 'asked' @@ -207,7 +210,7 @@ export async function runPlanSkillObservation(opts: PlanSkillObservationOptions) proseAUQEverObserved: false, waitingEverObserved: false, scopeGateQuestionObserved: false, scopeGateAutoSelectObserved: false, scopeTranscript: { status: 'missing', calls: [], assistantMessages: [] }, scopeTools: [], - nativeAutoDecide: null, nativePolledAt: null, + nativeAutoDecide: null, nativePolledAt: null, idleVisible: undefined, idleSince: 0, tokensObserved: Object.fromEntries((opts.trackTokens ?? []).map(t => [t, false])), }; return runPtySession({ @@ -267,6 +270,9 @@ interface ObservationRun { nativeAutoDecide: NativeAutoDecision | null; nativePolledAt: number | null; tokensObserved: Record; + /** Visible text when the current idle turn end was first seen, and when. */ + idleVisible: string | undefined; + idleSince: number; } async function observationPreflightTimeout(run: ObservationRun, session: ClaudePtySession, summary: string): Promise { @@ -335,6 +341,8 @@ function observationFlags(run: ObservationRun, session: ClaudePtySession) { const JUDGE_AFTER_MS = 60_000; const JUDGE_INTERVAL_MS = 30_000; +/** An idle turn end must hold, with the visible text unchanged, this long. */ +const IDLE_SETTLE_MS = 10_000; async function observationTick(run: ObservationRun, session: ClaudePtySession): Promise> { const { opts, driver } = run; @@ -412,7 +420,31 @@ async function observationTick(run: ObservationRun, session: ClaudePtySession): } }; } } - return 'continue'; + return idleTurnStep(run, session, visible); +} + +/** + * The CLI ended its turn at an empty prompt with nothing classified: no input + * will arrive, so waiting out the budget cannot change the result. Stop once + * the screen has held still for IDLE_SETTLE_MS and nothing can still credit + * it: classification ran on this same text, and the judge either cannot + * grant 'asked' (requireProseEvidence, or a prose question already observed) + * or has already judged this unchanged screen (its verdict is cached by + * screen hash, so later ticks repeat it). The result is what the deadline + * would return, with a summary that names the idle turn end. + */ +function idleTurnStep(run: ObservationRun, session: ClaudePtySession, visible: string): PtyStep { + const now = run.driver.now(); + const idle = idleTurnEnd(visible); + if (!idle || visible !== run.idleVisible) { + run.idleVisible = idle ? visible : undefined; + run.idleSince = now; + return 'continue'; + } + const judgeCanCredit = !run.opts.requireProseEvidence && !run.proseAUQEverObserved; + const judgedIdleScreen = run.lastJudgeVerdict !== null && run.lastJudgeAt >= run.idleSince; + if (now - run.idleSince < IDLE_SETTLE_MS || judgeCanCredit && !judgedIdleScreen) return 'continue'; + return { done: observationTimeout(run, session, idle) }; } /** Cheap per-tick surface tracking; each flag is once-true (high water). */ @@ -457,14 +489,17 @@ function observeHighWaterMarks(run: ObservationRun, session: ClaudePtySession, v * This catches the model-surfaced-then-resumed-thinking case where * by the time the timeout fires, the buffer has moved past the * options into spinner state but the question DID surface earlier. */ -function observationTimeout(run: ObservationRun, session: ClaudePtySession): PlanSkillObservation { +function observationTimeout(run: ObservationRun, session: ClaudePtySession, idle?: ReturnType): PlanSkillObservation { const { opts, driver, lastJudgeVerdict } = run; const finalVisible = session.visibleSince(run.since); + const idleEnd = idle + ? `the CLI turn ended at an idle prompt (${idle.done})${idle.apiError ? ` after its public error panel: ${idle.apiError}` : ''}` + : ''; if (run.proseAUQEverObserved || run.waitingEverObserved && !opts.requireProseEvidence) { return { outcome: 'asked', summary: - `prose-AUQ surface observed during run (proseAUQEverObserved=${run.proseAUQEverObserved}, waitingEverObserved=${run.waitingEverObserved}); model surfaced the question and the test budget elapsed without a follow-up classification` + + `prose-AUQ surface observed during run (proseAUQEverObserved=${run.proseAUQEverObserved}, waitingEverObserved=${run.waitingEverObserved}); model surfaced the question and ${idleEnd || 'the test budget elapsed'} without a follow-up classification` + (lastJudgeVerdict ? ` (last LLM judge: ${lastJudgeVerdict.state} — ${lastJudgeVerdict.reasoning})` : ''), @@ -476,7 +511,9 @@ function observationTimeout(run: ObservationRun, session: ClaudePtySession): Pla return { outcome: 'timeout', summary: - `no terminal outcome within ${run.budgetMs}ms` + + (idle + ? `no terminal outcome: ${idleEnd} with no question, plan or write, and no input will arrive; stopped after ${Math.round((driver.now() - run.startedAt) / 1000)}s instead of waiting out the ${run.budgetMs}ms budget` + : `no terminal outcome within ${run.budgetMs}ms`) + (lastJudgeVerdict ? ` (last LLM judge: state=${lastJudgeVerdict.state} — ${lastJudgeVerdict.reasoning})` : ''), diff --git a/test/helpers/pty/screen.ts b/test/helpers/pty/screen.ts index 3c2301040..ba86fa311 100644 --- a/test/helpers/pty/screen.ts +++ b/test/helpers/pty/screen.ts @@ -410,3 +410,28 @@ export function isNumberedOptionListVisible(visible: string): boolean { const cleaned = stripPtyResidue(visible); return /❯\s*1\./.test(cleaned) && /(^|[^0-9])2\./.test(cleaned); } + +/** Start of the CLI's public "API Error:" panel. The native AUQ capture reads the same panel. */ +export const API_ERROR_PANEL = /(?:^|\n)[\t │┃]*(?:[⎿●⏺]\s*)?API Error:/i; + +/** The CLI's end-of-turn line, e.g. "✻ Cooked for 24s · done 12:46 AM" (spaces may collapse in the PTY text). */ +const TURN_DONE_RE = /✻[ \t]*\p{L}[\p{L}'’-]*[ \t]*for[ \t]*(?=\d)(?:\d+h[ \t]*)?(?:\d+m[ \t]*)?(?:\d+s)?[ \t]*·[ \t]*done[^\n]*/gu; +/** An empty input prompt and its known idle footers, nothing else. */ +const IDLE_PROMPT_RE = /^[\s─]*❯[\s─]*(?:(?:←[ \t]*for[ \t]*agents|\?[ \t]*for[ \t]*shortcuts)[\s─]*)*$/; + +/** + * The CLI ended its turn and sits at an empty prompt: the last end-of-turn + * line is followed only by the prompt and an idle footer. A spinner, menu, + * background task or any other text after it means the turn is not idle. + * Returns that line and the last public API Error panel line of the same turn. + */ +export function idleTurnEnd(visible: string): { done: string; apiError?: string } | null { + const text = visible.replace(/\x1b?\[\?25[lh]/g, '').replace(/\r\n?/g, '\n').replace(/\u00a0/g, ' '); + const turns = [...text.matchAll(TURN_DONE_RE)]; + const done = turns.at(-1); + if (!done || !IDLE_PROMPT_RE.test(text.slice(done.index + done[0].length))) return null; + const previous = turns.at(-2); + const turn = text.slice(previous ? previous.index + previous[0].length : 0, done.index); + const apiError = [...turn.matchAll(new RegExp(`${API_ERROR_PANEL.source}[^\\n]*`, 'gi'))].at(-1)?.[0].trim(); + return { done: done[0].trim(), ...(apiError ? { apiError: apiError.slice(0, 300) } : {}) }; +} diff --git a/test/plan-review-cases.test.ts b/test/plan-review-cases.test.ts index e50303ded..fcd5fe0b9 100644 --- a/test/plan-review-cases.test.ts +++ b/test/plan-review-cases.test.ts @@ -818,13 +818,14 @@ describe('Eng approved-work decision gate', () => { expect(bootstrap).toContain('Context Recovery'); expect(bootstrap).toContain('Keep the reviewed target fixed'); const namedTarget = bootstrap.indexOf('**User-named target (outside plan mode):**'); - const headless = bootstrap.indexOf('**Headless or spawned session without a target:**'); - expect(0 < namedTarget && namedTarget < headless && headless < bootstrap.indexOf('**Initial selector algorithm')).toBe(true); - const pending = bootstrap.slice(headless, bootstrap.indexOf('**Initial selector algorithm')); - expect(pending).toContain('neither rule above supplies an unambiguous target'); - expect(pending).toContain('Scope pending: provide a plan/path or explicitly request branch diff'); - expect(pending).toContain('STOP. Do not run the preamble or review tools'); - expect(pending).toContain('The session type does not choose a target or approve work'); + expect(0 < namedTarget && namedTarget < bootstrap.indexOf('**Initial selector algorithm')).toBe(true); + // A missing or disallowed tool led models to infer a headless session and + // end the turn with a separate pending report instead of the menu + // (988e985 slice 5 and three earlier captures). The menu is the one + // no-transport outcome, whatever the session type. + expect(bootstrap).toContain('send the menu as plain prose and STOP, whatever the session type; the session type never chooses a target or approves work'); + expect(bootstrap).not.toContain('Headless or spawned session'); + expect(bootstrap).not.toContain('Scope pending'); expect(skeleton).toContain('Copy required command, output and question formats exactly. Apply Voice to newly composed prose'); });