CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-06-28 20:50:05 +02:00
Code Issues Packages Projects Releases Wiki Activity

test: delete --disallowedTools AskUserQuestion-blocked test variants

Browse Source 
These tests simulated a fictional environment that doesn't exist in
production. Real Conductor sessions launch claude with
`--disallowedTools AskUserQuestion` AND register
`mcp__conductor__AskUserQuestion` — the model has the MCP variant. But
the tests passed `--disallowedTools` without standing up any MCP server,
so they tested "model behavior with NO AUQ available," which no real
user state produces.

Combined with bare `/plan-ceo-review` invocation (no follow-up content),
this forced the model into a 5+ minute deliberation loop trying to
prose-render a question with options it had to first invent. The result
was persistent flakes that consumed nine paid E2E runs trying to fix
"the model takes too long" — but the actual problem was the test
configuration, not the model.

Removals:
- test/skill-e2e-autoplan-auto-mode.test.ts (deleted; the entire file
  was a single AUQ-blocked test)
- test/skill-e2e-plan-ceo-plan-mode.test.ts test 2 (the migrated
  --disallowedTools test); test 1 (baseline plan-mode smoke) stays
- test/skill-e2e-plan-design-plan-mode.test.ts test 2 (same shape);
  test 1 stays
- test/skill-e2e-plan-eng-plan-mode.test.ts test 2 (same shape); test 1
  (baseline) and test 3 (STOP-gate with seeded plan, different
  contract) stay
- test/helpers/touchfiles.ts: autoplan-auto-mode entry removed
- test/touchfiles.test.ts: assertion count + commentary updated

Coverage retained: test 1 of each plan-mode file already verifies the
model fires AUQ; the periodic finding-count tests verify per-finding
AUQ cadence end-to-end. The harness improvements landed during this
debugging cycle (isProseAUQVisible regex, LLM judge, snapshot logging,
high-water-mark tracking, ENOENT-tolerant assertReportAtBottomIfPlanWritten)
all stay — they're useful for the remaining plan-mode tests that can
also encounter prose rendering and slow-thinking phases.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-05-08 23:43:37 -07:00
parent 4d577830a8
commit 558e195c46
6 changed files with 6 additions and 342 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-e2e-plan-eng-plan-mode.test.ts
-55
View File
@@ -10,7 +10,6 @@ import {
runPlanSkillObservation,
planFileHasDecisionsSection,
assertReportAtBottomIfPlanWritten,
isProseAUQVisible,
} from './helpers/claude-pty-runner';
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
@@ -66,60 +65,6 @@ describeE2E('plan-eng-review plan-mode smoke (gate)', () => {
assertReportAtBottomIfPlanWritten(obs);
}, 360_000);
// v1.21+ regression: see skill-e2e-plan-ceo-plan-mode.test.ts for the
// contract. Pass envelope is ['asked', 'plan_ready']; failure signals
// are 'auto_decided' (AUTO_DECIDE without opt-in) plus the standard
// silent_write/exited/timeout.
test('AskUserQuestion surfaces when --disallowedTools AskUserQuestion is set', async () => {
const obs = await runPlanSkillObservation({
skillName: 'plan-eng-review',
inPlanMode: true,
extraArgs: ['--disallowedTools', 'AskUserQuestion'],
timeoutMs: 300_000,
});
// Surface visibility check (consistent with plan-ceo / plan-design /
// autoplan migrations): user must SEE the question via a `## Decisions`
// section in the plan file (legacy) OR a BLOCKED string in TTY OR
// prose-rendered AUQ options in TTY.
const blockedVisible = /BLOCKED\s*[—-]\s*AskUserQuestion/i.test(obs.evidence);
const proseAUQVisible = isProseAUQVisible(obs.evidence) || obs.proseAUQEverObserved === true;
const surfaceVisible = blockedVisible || proseAUQVisible || obs.waitingEverObserved === true;
if (
obs.outcome === 'auto_decided' ||
obs.outcome === 'silent_write' ||
obs.outcome === 'timeout'
) {
throw new Error(
`plan-eng-review AskUserQuestion-blocked regression: outcome=${obs.outcome}\n` +
`summary: ${obs.summary}\n` +
`elapsed: ${obs.elapsedMs}ms\n` +
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
if (obs.outcome === 'exited' && !surfaceVisible) {
throw new Error(
`plan-eng-review AskUserQuestion-blocked regression: outcome=exited without any visible question surface (no BLOCKED string, no prose-rendered AUQ options). Model quit silently.\n` +
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
if (obs.outcome === 'plan_ready') {
const decisionsOk = obs.planFile && planFileHasDecisionsSection(obs.planFile);
if (!decisionsOk && !surfaceVisible) {
throw new Error(
`plan-eng-review AskUserQuestion-blocked regression: plan_ready without any visible question surface (no "## Decisions" section in ${obs.planFile ?? '<no plan file detected>'}, no BLOCKED string, no prose AUQ options) — Step 0 was silently skipped.\n` +
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
}
expect(['asked', 'plan_ready', 'exited']).toContain(obs.outcome);
// NOTE: assertReportAtBottomIfPlanWritten intentionally not called —
// see plan-ceo-plan-mode test 2 for the rationale. Under
// --disallowedTools the model can't run the full review, so the
// report-at-bottom contract doesn't apply here.
}, 360_000);
// D3-B / D4-B: when a plan with guaranteed-finding-triggering complexity
// is seeded, the skill MUST fire AskUserQuestion (or fall back to a
// Decisions section) before writing findings to the plan. The