fix(evals): parse single-logical-line AskUserQuestions in the PTY runner

When the PTY reflows a boxed AUQ, ALL options land on ONE logical line
after stripAnsi — parseNumberedOptions parsed one option per line, found
only '1.', and the >=2 check failed forever while the correct question
sat on screen (plan-design-with-ui timed out this way twice, with the
rendered scope-gate AUQ visible in both failure buffers). The cursor
line is now parsed as a stream of ascending N. tokens; DEC cursor-
visibility residue is stripped before matching; plan-design-with-ui's
budgets grow to fit observed ~6min preamble+thinking latency. Pinned by
test/pty-auq-single-line.test.ts using the real failure buffers; all 142
existing parser-consumer unit tests still green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-13 10:34:25 -07:00
co-authored by Claude Fable 5
parent bd11416d80
commit 828b229900
3 changed files with 120 additions and 27 deletions
+65
View File
@@ -0,0 +1,65 @@
/**
* Pins single-logical-line AskUserQuestion detection (the plan-design-with-ui
* gate-timeout class). When the PTY reflows a boxed AUQ, ALL options land on
* ONE logical line after stripAnsi — the per-line option parser then finds
* only option 1 and the >= 2 check fails forever while the (correct) question
* sits on screen. Both fixture strings below are condensed from REAL observed
* failure buffers of test/skill-e2e-plan-design-with-ui.test.ts.
*/
import { describe, expect, it } from 'bun:test';
import {
isNumberedOptionListVisible,
parseNumberedOptions,
stripPtyResidue,
} from './helpers/claude-pty-runner';
// Condensed from the 2026-08-13 failure buffer: whole AUQ on one logical
// line — dividers, cursor option, options 2-5, footer. Note the missing
// spaces ("2.Planordesigndoc") from stripped cursor-positioning escapes.
const SINGLE_LINE_AUQ =
'────────────Planning: /tmp/x/.claude/plans/soft-questing-jellyfish.md──────────' +
' ☐ Review target What should I review? <gstack-qid:plan-design-review-scope-gate>' +
'1.Branch diff (current WIP) Review the design implications of what changed. ' +
'Recommendation: A when a branch diff exists.2.Planordesigndoc Paste or point me to a plan file. ' +
'3. Spcific page, file, r pah Name a specific file. 4. Type something.' +
'────────────5.ChataboutthisEnter to select · ↑/↓ o navigate · Escto cancel';
// The same AUQ with DEC cursor-visibility residue + spinner frames
// interleaved, as captured before stripPtyResidue existed.
const RESIDUE_AUQ =
'[?25l✻Sprouting…[?25h[?25l✶[?25h' + SINGLE_LINE_AUQ.slice(0, 200) +
'[?25l·still thinking[?25h' + SINGLE_LINE_AUQ.slice(200);
describe('single-logical-line AUQ detection', () => {
it('isNumberedOptionListVisible matches the reflowed one-line AUQ', () => {
expect(isNumberedOptionListVisible(SINGLE_LINE_AUQ)).toBe(true);
});
it('parseNumberedOptions finds the full ascending option run on one line', () => {
const options = parseNumberedOptions(SINGLE_LINE_AUQ);
expect(options.length).toBeGreaterThanOrEqual(2);
expect(options[0]).toEqual({ index: 1, label: expect.stringContaining('Branch diff') });
expect(options[1]?.index).toBe(2);
});
it('survives DEC residue + spinner interleave', () => {
expect(isNumberedOptionListVisible(RESIDUE_AUQ)).toBe(true);
expect(parseNumberedOptions(RESIDUE_AUQ).length).toBeGreaterThanOrEqual(2);
});
it('stripPtyResidue removes cursor-visibility fragments only', () => {
expect(stripPtyResidue('[?25la[?25hb')).toBe('ab');
expect(stripPtyResidue('keep [?250] this')).toBe('keep [?250] this');
});
it('still rejects prose numbered lists (no cursor sigil)', () => {
expect(parseNumberedOptions('steps: 1. Read the file 2. Edit it 3. Done')).toEqual([]);
});
it('still parses classic multi-line AUQs', () => {
const multiLine = 'What should I do?\n 1. First option\n 2. Second option\n 3. Third option\n';
const options = parseNumberedOptions(multiLine);
expect(options.map((o) => o.index)).toEqual([1, 2, 3]);
});
});