mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-07 14:06:42 +02:00
Garry Tan
aaf2668103
Conductor launches Claude Code with --disallowedTools AskUserQuestion
--permission-mode default --permission-prompt-tool stdio (verified by
inspecting the live conductor claude process via ps -p ... -o args=).
Native AskUserQuestion is removed from the model's tool registry; without
fallback guidance the plan-mode skills (plan-ceo-review, plan-eng-review,
plan-design-review, plan-devex-review, autoplan, office-hours) silently
proceed and never surface decisions to the user.
Adds 6 gate-tier real-PTY regression cases:
- 4 inline test cases inside the existing plan-X-review-plan-mode.test
files, each exercising the same skill with extraArgs ['--disallowedTools',
'AskUserQuestion'] and asserting outcome === 'asked'. plan-design-review
keeps the ['asked', 'plan_ready'] envelope (legitimate short-circuit on
no-UI-scope) but explicitly fails on 'auto_decided'.
- 2 standalone test files for autoplan + office-hours (which had no prior
plan-mode test). autoplan asserts the FIRST non-auto-decided gate fires
(Phase 1 premise confirmation) — autoplan auto-decides intermediate
questions BY DESIGN.
Touchfile entries:
- autoplan-auto-mode + office-hours-auto-mode added to E2E_TOUCHFILES +
E2E_TIERS (gate)
- existing plan-X-review-plan-mode entries gain question-tuning.ts and
generate-ask-user-format.ts touchfile deps so AUTO_DECIDE-related
resolver changes correctly invalidate the regression tests
- touchfiles.test.ts count updated 18 -> 19 to cover the autoplan
touchfile dependency on plan-ceo-review/**
Filenames retain `auto-mode` for branch-history continuity. Auto-mode (the
AUTO_DECIDE preamble path when QUESTION_TUNING=true) is a related but
distinct silencing mechanism; both share the same fix surface in the
preamble.
These tests are expected to FAIL on this branch until the fix lands. The
failure is the receipt for the regression.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
45 lines
1.9 KiB
TypeScript
45 lines
1.9 KiB
TypeScript
/**
|
|
* office-hours AskUserQuestion-blocked regression (gate, paid, real-PTY).
|
|
*
|
|
* v1.21+ regression: Conductor launches Claude Code with
|
|
* `--disallowedTools AskUserQuestion --permission-mode default` (verified
|
|
* by inspecting the parent claude process via `ps`). office-hours' first
|
|
* step issues a startup-vs-builder mode AskUserQuestion
|
|
* (office-hours/SKILL.md.tmpl:69); when AskUserQuestion is disallowed at
|
|
* the tool-registry level the model cannot ask and silently picks one mode,
|
|
* breaking the whole interactive premise. This test asserts that question
|
|
* still surfaces — fix must route through mcp__conductor__AskUserQuestion
|
|
* (when present) or plan-file + ExitPlanMode flow.
|
|
*
|
|
* Filename keeps `auto-mode` for branch-history continuity. Auto-mode (the
|
|
* AUTO_DECIDE preamble path when QUESTION_TUNING=true) is a related but
|
|
* distinct silencing mechanism; both share the same fix surface.
|
|
*/
|
|
|
|
import { describe, test, expect } from 'bun:test';
|
|
import { runPlanSkillObservation } from './helpers/claude-pty-runner';
|
|
|
|
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
|
const describeE2E = shouldRun ? describe : describe.skip;
|
|
|
|
describeE2E('office-hours AskUserQuestion-blocked smoke (gate)', () => {
|
|
test('AskUserQuestion surfaces when --disallowedTools AskUserQuestion is set', async () => {
|
|
const obs = await runPlanSkillObservation({
|
|
skillName: 'office-hours',
|
|
inPlanMode: true,
|
|
extraArgs: ['--disallowedTools', 'AskUserQuestion'],
|
|
timeoutMs: 300_000,
|
|
});
|
|
|
|
if (obs.outcome !== 'asked') {
|
|
throw new Error(
|
|
`office-hours AskUserQuestion-blocked regression: outcome=${obs.outcome}\n` +
|
|
`summary: ${obs.summary}\n` +
|
|
`elapsed: ${obs.elapsedMs}ms\n` +
|
|
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
|
|
);
|
|
}
|
|
expect(obs.outcome).toEqual('asked');
|
|
}, 360_000);
|
|
});
|