Files
gstack/test/helpers/claude-pty-runner.scope-gate-floor.unit.test.ts
T
Garry TanandClaude Fable 5 4e61233021 fix(pty-runner): positional floor exclusion, flag builder, outcome union, token tracking
Review-army + adversarial findings on the scope-gate observability work,
all verified before fixing:

- Floor check: acceptance scanned the CUMULATIVE buffer while the scope-gate
  exclusion scanned only the 1500-byte tail, so an early gate render satisfied
  the floor vacuously once ~1.5KB of output accumulated (found independently
  by 4 review passes; predicate reproduced). Acceptance now scans only content
  APPENDED after the first gate render (positional anchor), and the LLM-judge
  'waiting' shortcut no longer fires while the gate menu is the pending render.
- High-water flags are built once and spread at every return path — the
  hand-spread pattern had already drifted (judge-waiting return omitted two
  flags), which made must-stay-false asserts vacuous on those paths.
- isScopeGateAutoSelectVisible: tense-tolerant selected/selecting/selects
  token (must-be-TRUE asserts shouldn't fail semantically-perfect paraphrases)
  and quoted-occurrence rejection (a model verbatim-quoting the announcement
  while declining must not trip must-stay-FALSE asserts). Fixtures added for
  both directions.
- PlanSkillObservation outcome union gains 'wrote_findings_before_asking'
  (returned at runtime via classifyVisible but missing from the type).
- trackTokens/tokensObserved: cumulative-buffer token high-water for
  consumption asserts (the 2KB evidence tail is lossy and the plan-file
  fallback is unreachable outside plan mode).
- New scope-gate-floor unit pins (from the ship coverage audit): both gate
  render forms trip acceptance and exclusion; a genuine finding AUQ is not
  excluded; tail-scoping semantics pinned.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-12 09:09:32 -07:00

123 lines
5.4 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
/**
* Scope-gate floor-exclusion regression pins (free, static).
*
* runPlanSkillFloorCheck's acceptance condition changed with the plan-mode
* auto-select-B work: a render only satisfies the finding floor when
*
* (isNumberedOptionListVisible(visible) || isProseAUQVisible(visible))
* && !isPermissionDialogVisible(tail)
* && !isScopeGateQuestionVisible(tail) // <- new exclusion
*
* where tail = visible.slice(-TAIL_SCAN_BYTES). The composition lives inline
* in the paid PTY loop, so these tests pin the load-bearing behavior of each
* detector on the exact render shapes the floor passes them:
*
* 1. Both scope-gate render forms (native numbered UI, prose lettered
* fallback) trip the acceptance detectors — WITHOUT the exclusion the
* gate would trivially satisfy the floor inside the 3s pre-target
* window. The exclusion must catch both forms.
* 2. A genuine finding-driven AskUserQuestion must NOT trip the exclusion,
* or the floor becomes unsatisfiable.
* 3. The exclusion is TAIL-scoped by design: an early gate render that has
* scrolled past TAIL_SCAN_BYTES must not suppress a later real finding
* AskUserQuestion.
*
* Also closes the untested OR-branch of isScopeGateAutoSelectVisible: the
* fully-collapsed hyphen-less 'autoselectedb' form.
*/
import { describe, test, expect } from 'bun:test';
import {
TAIL_SCAN_BYTES,
isNumberedOptionListVisible,
isProseAUQVisible,
isPermissionDialogVisible,
isScopeGateQuestionVisible,
isScopeGateAutoSelectVisible,
} from './claude-pty-runner';
// The gate's native AskUserQuestion render (numbered options + cursor) —
// what fires inside the floor check's 3s window before the seed arrives.
const GATE_NATIVE_RENDER = `
What should I review?
1. The current branch diff — the work in progress on this branch.
2. A plan or design doc I'll paste or point you to.
3. A specific file, directory, or path.
`;
// The gate's prose fallback render (lettered options under --disallowedTools).
const GATE_PROSE_RENDER = `
What should I review?
A) The current branch diff — the work in progress on this branch.
B) A plan or design doc I'll paste or point you to.
C) A specific file, directory, or path.
Recommendation: A when a branch diff exists, otherwise B.
`;
// A genuine finding-driven AskUserQuestion — the render the floor MEASURES.
const FINDING_AUQ_RENDER = `
Finding 1: the plan reimplements test sharding that Bun provides natively.
1. Use Bun's native --shard flag (recommended)
2. Keep the custom scheduler as planned
3. Defer this decision to implementation
`;
describe('floor-check scope-gate exclusion (acceptance-condition regression)', () => {
test('native gate render trips the acceptance detector — the exclusion is load-bearing', () => {
// Pre-exclusion, this render satisfied the floor by itself.
expect(isNumberedOptionListVisible(GATE_NATIVE_RENDER)).toBe(true);
expect(isPermissionDialogVisible(GATE_NATIVE_RENDER)).toBe(false);
// The new exclusion catches it.
expect(isScopeGateQuestionVisible(GATE_NATIVE_RENDER)).toBe(true);
});
test('prose gate render trips the prose-AUQ arm — the exclusion catches that form too', () => {
expect(isProseAUQVisible(GATE_PROSE_RENDER)).toBe(true);
expect(isPermissionDialogVisible(GATE_PROSE_RENDER)).toBe(false);
expect(isScopeGateQuestionVisible(GATE_PROSE_RENDER)).toBe(true);
});
test('a genuine finding AskUserQuestion is NOT excluded — the floor stays satisfiable', () => {
expect(isNumberedOptionListVisible(FINDING_AUQ_RENDER)).toBe(true);
expect(isPermissionDialogVisible(FINDING_AUQ_RENDER)).toBe(false);
expect(isScopeGateQuestionVisible(FINDING_AUQ_RENDER)).toBe(false);
});
test('tail-scoping: an early gate render scrolled out of the tail does not suppress a later finding AUQ', () => {
// Gate render, then >TAIL_SCAN_BYTES of review output, then the real
// finding AskUserQuestion — the shape the TAIL-scoped exclusion exists for.
const filler = 'Reading the plan and auditing the design system.\n'.repeat(
Math.ceil(TAIL_SCAN_BYTES / 48) + 4,
);
const visible = GATE_NATIVE_RENDER + filler + FINDING_AUQ_RENDER;
const tail = visible.slice(-TAIL_SCAN_BYTES);
// Full buffer still remembers the gate (scrollback)…
expect(isScopeGateQuestionVisible(visible)).toBe(true);
// …but the floor's exclusion looks only at the tail, which is clean:
expect(isScopeGateQuestionVisible(tail)).toBe(false);
// and the acceptance arm (full-buffer scan) sees the finding AUQ.
expect(isNumberedOptionListVisible(visible)).toBe(true);
expect(isPermissionDialogVisible(tail)).toBe(false);
});
test('a gate render inside the tail IS suppressed (no false floor pass)', () => {
const tail = GATE_NATIVE_RENDER.slice(-TAIL_SCAN_BYTES);
expect(isScopeGateQuestionVisible(tail)).toBe(true);
});
});
describe('isScopeGateAutoSelectVisible collapsed hyphen-less branch', () => {
test("matches the fully-collapsed 'autoselectedb' form (hyphen lost in TTY reflow)", () => {
const sample = 'Scopegate:planmode—autoselectedB(reviewingPLAN.md).';
expect(isScopeGateAutoSelectVisible(sample)).toBe(true);
});
test('hyphen-less token without the announcement prefix stays false', () => {
const sample = 'The agent autoselectedB from the menu without announcing a scope gate decision.';
expect(isScopeGateAutoSelectVisible(sample)).toBe(false);
});
});