fix(pty-runner): positional floor exclusion, flag builder, outcome union, token tracking

Review-army + adversarial findings on the scope-gate observability work,
all verified before fixing:

- Floor check: acceptance scanned the CUMULATIVE buffer while the scope-gate
  exclusion scanned only the 1500-byte tail, so an early gate render satisfied
  the floor vacuously once ~1.5KB of output accumulated (found independently
  by 4 review passes; predicate reproduced). Acceptance now scans only content
  APPENDED after the first gate render (positional anchor), and the LLM-judge
  'waiting' shortcut no longer fires while the gate menu is the pending render.
- High-water flags are built once and spread at every return path — the
  hand-spread pattern had already drifted (judge-waiting return omitted two
  flags), which made must-stay-false asserts vacuous on those paths.
- isScopeGateAutoSelectVisible: tense-tolerant selected/selecting/selects
  token (must-be-TRUE asserts shouldn't fail semantically-perfect paraphrases)
  and quoted-occurrence rejection (a model verbatim-quoting the announcement
  while declining must not trip must-stay-FALSE asserts). Fixtures added for
  both directions.
- PlanSkillObservation outcome union gains 'wrote_findings_before_asking'
  (returned at runtime via classifyVisible but missing from the type).
- trackTokens/tokensObserved: cumulative-buffer token high-water for
  consumption asserts (the 2KB evidence tail is lossy and the plan-file
  fallback is unreachable outside plan mode).
- New scope-gate-floor unit pins (from the ship coverage audit): both gate
  render forms trip acceptance and exclusion; a genuine finding AUQ is not
  excluded; tail-scoping semantics pinned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-08-12 09:09:32 -07:00
co-authored by Claude Fable 5
parent 57c53e7913
commit 4e61233021
3 changed files with 236 additions and 30 deletions
@@ -0,0 +1,122 @@
/**
* Scope-gate floor-exclusion regression pins (free, static).
*
* runPlanSkillFloorCheck's acceptance condition changed with the plan-mode
* auto-select-B work: a render only satisfies the finding floor when
*
* (isNumberedOptionListVisible(visible) || isProseAUQVisible(visible))
* && !isPermissionDialogVisible(tail)
* && !isScopeGateQuestionVisible(tail) // <- new exclusion
*
* where tail = visible.slice(-TAIL_SCAN_BYTES). The composition lives inline
* in the paid PTY loop, so these tests pin the load-bearing behavior of each
* detector on the exact render shapes the floor passes them:
*
* 1. Both scope-gate render forms (native numbered UI, prose lettered
* fallback) trip the acceptance detectors — WITHOUT the exclusion the
* gate would trivially satisfy the floor inside the 3s pre-target
* window. The exclusion must catch both forms.
* 2. A genuine finding-driven AskUserQuestion must NOT trip the exclusion,
* or the floor becomes unsatisfiable.
* 3. The exclusion is TAIL-scoped by design: an early gate render that has
* scrolled past TAIL_SCAN_BYTES must not suppress a later real finding
* AskUserQuestion.
*
* Also closes the untested OR-branch of isScopeGateAutoSelectVisible: the
* fully-collapsed hyphen-less 'autoselectedb' form.
*/
import { describe, test, expect } from 'bun:test';
import {
TAIL_SCAN_BYTES,
isNumberedOptionListVisible,
isProseAUQVisible,
isPermissionDialogVisible,
isScopeGateQuestionVisible,
isScopeGateAutoSelectVisible,
} from './claude-pty-runner';
// The gate's native AskUserQuestion render (numbered options + cursor) —
// what fires inside the floor check's 3s window before the seed arrives.
const GATE_NATIVE_RENDER = `
What should I review?
1. The current branch diff — the work in progress on this branch.
2. A plan or design doc I'll paste or point you to.
3. A specific file, directory, or path.
`;
// The gate's prose fallback render (lettered options under --disallowedTools).
const GATE_PROSE_RENDER = `
What should I review?
A) The current branch diff — the work in progress on this branch.
B) A plan or design doc I'll paste or point you to.
C) A specific file, directory, or path.
Recommendation: A when a branch diff exists, otherwise B.
`;
// A genuine finding-driven AskUserQuestion — the render the floor MEASURES.
const FINDING_AUQ_RENDER = `
Finding 1: the plan reimplements test sharding that Bun provides natively.
1. Use Bun's native --shard flag (recommended)
2. Keep the custom scheduler as planned
3. Defer this decision to implementation
`;
describe('floor-check scope-gate exclusion (acceptance-condition regression)', () => {
test('native gate render trips the acceptance detector — the exclusion is load-bearing', () => {
// Pre-exclusion, this render satisfied the floor by itself.
expect(isNumberedOptionListVisible(GATE_NATIVE_RENDER)).toBe(true);
expect(isPermissionDialogVisible(GATE_NATIVE_RENDER)).toBe(false);
// The new exclusion catches it.
expect(isScopeGateQuestionVisible(GATE_NATIVE_RENDER)).toBe(true);
});
test('prose gate render trips the prose-AUQ arm — the exclusion catches that form too', () => {
expect(isProseAUQVisible(GATE_PROSE_RENDER)).toBe(true);
expect(isPermissionDialogVisible(GATE_PROSE_RENDER)).toBe(false);
expect(isScopeGateQuestionVisible(GATE_PROSE_RENDER)).toBe(true);
});
test('a genuine finding AskUserQuestion is NOT excluded — the floor stays satisfiable', () => {
expect(isNumberedOptionListVisible(FINDING_AUQ_RENDER)).toBe(true);
expect(isPermissionDialogVisible(FINDING_AUQ_RENDER)).toBe(false);
expect(isScopeGateQuestionVisible(FINDING_AUQ_RENDER)).toBe(false);
});
test('tail-scoping: an early gate render scrolled out of the tail does not suppress a later finding AUQ', () => {
// Gate render, then >TAIL_SCAN_BYTES of review output, then the real
// finding AskUserQuestion — the shape the TAIL-scoped exclusion exists for.
const filler = 'Reading the plan and auditing the design system.\n'.repeat(
Math.ceil(TAIL_SCAN_BYTES / 48) + 4,
);
const visible = GATE_NATIVE_RENDER + filler + FINDING_AUQ_RENDER;
const tail = visible.slice(-TAIL_SCAN_BYTES);
// Full buffer still remembers the gate (scrollback)…
expect(isScopeGateQuestionVisible(visible)).toBe(true);
// …but the floor's exclusion looks only at the tail, which is clean:
expect(isScopeGateQuestionVisible(tail)).toBe(false);
// and the acceptance arm (full-buffer scan) sees the finding AUQ.
expect(isNumberedOptionListVisible(visible)).toBe(true);
expect(isPermissionDialogVisible(tail)).toBe(false);
});
test('a gate render inside the tail IS suppressed (no false floor pass)', () => {
const tail = GATE_NATIVE_RENDER.slice(-TAIL_SCAN_BYTES);
expect(isScopeGateQuestionVisible(tail)).toBe(true);
});
});
describe('isScopeGateAutoSelectVisible collapsed hyphen-less branch', () => {
test("matches the fully-collapsed 'autoselectedb' form (hyphen lost in TTY reflow)", () => {
const sample = 'Scopegate:planmode—autoselectedB(reviewingPLAN.md).';
expect(isScopeGateAutoSelectVisible(sample)).toBe(true);
});
test('hyphen-less token without the announcement prefix stays false', () => {
const sample = 'The agent autoselectedB from the menu without announcing a scope gate decision.';
expect(isScopeGateAutoSelectVisible(sample)).toBe(false);
});
});