mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
v1.11.1.0 fix: plan-mode handshake + canUseTool test harness (#1182)
* feat: plan-mode handshake for interactive review skills Add a preamble-level STOP-Ask handshake that fires when the user invokes any of the 4 interactive review skills (plan-ceo-review, plan-eng-review, plan-design-review, plan-devex-review) while their Claude Code session is in plan mode. Without this gate, plan mode's "this supercedes any other instructions" system-reminder outranked the skills' interactive STOP gates and the skills silently wrote plan files without any per-finding AskUserQuestion. The handshake offers 2 options (exit-and-rerun, cancel) — the original third "stay and batch" option was dropped after two independent reviewers flagged it as a silent bypass of the skills' anti-skip rule. Architecture decisions (CEO+Eng review): - Preamble-level resolver, not per-template injection (Codex finding #2) - Position 1 in preamble composition: after bash block (_SESSION_ID live), before onboarding AskUserQuestion gates (so fresh-install users see the handshake first, not drowned in telemetry/proactive/routing prompts) - Generator-only `interactive: true` frontmatter flag, following the `preamble-tier` precedent (no host-config frontmatter allowlist edits) - Host-scoped to Claude via `ctx.host === 'claude'` check inside the resolver (simpler than `suppressedResolvers` which only gates `{{}}` placeholders) - One-way-door classification in scripts/question-registry.ts for all 4 skills so question-tuning `never-ask` preferences can't suppress the gate - Synchronous telemetry write to ~/.gstack/analytics/skill-usage.jsonl on handshake fire (captures A-exit and C-cancel outcomes that terminate the skill before end-of-run telemetry runs) Also adds an explicit STOP block to plan-ceo-review Step 0C-bis so the approach-selection question can't silently skip to mode selection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: extend agent-sdk-runner with canUseTool for AskUserQuestion interception Test harness at test/helpers/agent-sdk-runner.ts gains an optional `canUseTool` callback parameter. When a test supplies it, the harness flips `permissionMode` from `bypassPermissions` (overlay-harness default) to `default` so the SDK actually invokes the callback on every tool use, and auto-adds `AskUserQuestion` to `allowedTools` so Claude can fire it at all. Exports a `passThroughNonAskUserQuestion` helper so tests that only want to intercept AskUserQuestion can auto-allow every other tool with one line: `return passThroughNonAskUserQuestion(toolName, input)`. This is the foundation for D14 — every future interactive-skill E2E test can now assert on AskUserQuestion shape and routing. Previous E2E tests at `test/skill-e2e.test.ts` explicitly instructed the model to skip AskUserQuestion ("non-interactive run") which meant no test could actually verify the question content or routing. 6 new unit tests in test/agent-sdk-runner.test.ts cover: - permissionMode flips to 'default' when canUseTool supplied - permissionMode stays 'bypassPermissions' when canUseTool absent - canUseTool callback reaches the SDK options - AskUserQuestion auto-added to allowedTools when canUseTool supplied - AskUserQuestion NOT added when canUseTool absent - passThroughNonAskUserQuestion helper returns allow+updatedInput Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: plan-mode handshake E2E coverage and unit assertions Adds 6 E2E test files and 8 new unit assertions to verify the plan-mode handshake works end-to-end and stays correct under regeneration. E2E tests (gate-tier, paid, EVALS=1 EVALS_TIER=gate): - test/skill-e2e-plan-ceo-plan-mode.test.ts — handshake fires before any Write/Edit when plan-mode distinctive phrase is present; 2-option shape (Exit/Cancel); option A routes to ExitPlanMode cleanly - test/skill-e2e-plan-eng-plan-mode.test.ts — same contract for plan-eng - test/skill-e2e-plan-design-plan-mode.test.ts — same contract for plan-design; exercises C-cancel branch instead of A-exit - test/skill-e2e-plan-devex-plan-mode.test.ts — same contract for plan-devex - test/skill-e2e-plan-mode-no-op.test.ts — negative regression: handshake must NOT fire when distinctive phrase is absent; skill proceeds normally through Step 0 (REGRESSION RULE guardrail against breaking existing interactive-review sessions) - test/e2e-harness-audit.test.ts — free unit test asserting every `interactive: true` skill has at least one canUseTool-using test file (prevents future drift where a skill opts in without coverage) Shared helper test/helpers/plan-mode-handshake-helpers.ts centralizes the canUseTool interceptor + distinctive-phrase injection so the 4 sibling E2E tests are thin wiring (~20 LOC each) and can't drift out of sync. Unit assertions added to test/gen-skill-docs.test.ts: - handshake section present in all 4 Claude-generated SKILL.md files - handshake section absent from non-interactive Claude skills (ship, review, qa, office-hours, codex, retro, cso) - handshake section absent from non-Claude host outputs (.agents, etc.) - 0C-bis STOP block present in plan-ceo-review/SKILL.md at correct position (between the "Present these approach options" line and "### 0D-prelude" header) - handshake resolver wired BEFORE generateUpgradeCheck in preamble composition order 6 new gate-tier entries added to test/helpers/touchfiles.ts so any change to the handshake resolver, preamble composition, skill templates, question registry, one-way-door classifier, or agent-sdk-runner fires the relevant E2E tests. test/touchfiles.test.ts updated for the new selection count (plan-ceo-review/** now triggers 15 tests, up from 8). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(v1.11.1.0): VERSION bump + CHANGELOG entry + TODOS follow-ups Bumps from main's v1.11.0.0 to v1.11.1.0 (PATCH — bug-fix release, no new user-facing artifacts). CHANGELOG entry covers the plan-mode handshake, agent-sdk-runner canUseTool extension, and the 2 follow-up TODOs. CHANGELOG order: v1.11.1.0 (this) → v1.11.0.0 (workspace-aware ship, merged from main) → v1.10.1.0 (overlay efficacy harness). No duplicate headers. Syncs package.json version to match VERSION per the Step 12 idempotency invariant (both files must agree or /ship halts). TODOS.md: - Preserves the Testing/security-bench-haiku-responses P1 added on main - Adds P1 "Structural STOP-Ask forcing function" — broader class of the bug this release fixes - Adds P2 "Apply interactive: true to non-review skills (office-hours, codex, investigate, qa, retro, cso)" Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -366,6 +366,101 @@ describe('runAgentSdkTest — options propagation', () => {
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// canUseTool extension (D10 CEO / D4 eng)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
describe('runAgentSdkTest — canUseTool extension', () => {
|
||||
test('permissionMode flips to "default" when canUseTool is supplied', async () => {
|
||||
freshSem();
|
||||
const stub: StubConfig = {
|
||||
streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]],
|
||||
calls: [],
|
||||
};
|
||||
await runAgentSdkTest({
|
||||
...BASE_OPTS,
|
||||
queryProvider: makeStubProvider(stub),
|
||||
canUseTool: async (_toolName, input) => ({ behavior: 'allow', updatedInput: input }),
|
||||
});
|
||||
const opts = stub.calls[0]!.options!;
|
||||
expect(opts.permissionMode).toBe('default');
|
||||
expect(opts.allowDangerouslySkipPermissions).toBe(false);
|
||||
});
|
||||
|
||||
test('permissionMode stays "bypassPermissions" when canUseTool is NOT supplied', async () => {
|
||||
freshSem();
|
||||
const stub: StubConfig = {
|
||||
streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]],
|
||||
calls: [],
|
||||
};
|
||||
await runAgentSdkTest({
|
||||
...BASE_OPTS,
|
||||
queryProvider: makeStubProvider(stub),
|
||||
});
|
||||
const opts = stub.calls[0]!.options!;
|
||||
expect(opts.permissionMode).toBe('bypassPermissions');
|
||||
expect(opts.allowDangerouslySkipPermissions).toBe(true);
|
||||
});
|
||||
|
||||
test('canUseTool callback reaches the SDK options', async () => {
|
||||
freshSem();
|
||||
const stub: StubConfig = {
|
||||
streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]],
|
||||
calls: [],
|
||||
};
|
||||
const cb = async (_toolName: string, input: Record<string, unknown>) => ({
|
||||
behavior: 'allow' as const,
|
||||
updatedInput: input,
|
||||
});
|
||||
await runAgentSdkTest({
|
||||
...BASE_OPTS,
|
||||
queryProvider: makeStubProvider(stub),
|
||||
canUseTool: cb,
|
||||
});
|
||||
const opts = stub.calls[0]!.options! as Options & { canUseTool?: unknown };
|
||||
expect(typeof opts.canUseTool).toBe('function');
|
||||
});
|
||||
|
||||
test('AskUserQuestion is auto-added to allowedTools when canUseTool is supplied', async () => {
|
||||
freshSem();
|
||||
const stub: StubConfig = {
|
||||
streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]],
|
||||
calls: [],
|
||||
};
|
||||
await runAgentSdkTest({
|
||||
...BASE_OPTS,
|
||||
allowedTools: ['Read', 'Grep'], // explicitly omits AskUserQuestion
|
||||
queryProvider: makeStubProvider(stub),
|
||||
canUseTool: async (_toolName, input) => ({ behavior: 'allow', updatedInput: input }),
|
||||
});
|
||||
const opts = stub.calls[0]!.options!;
|
||||
expect(opts.allowedTools).toContain('AskUserQuestion');
|
||||
expect(opts.tools).toContain('AskUserQuestion');
|
||||
});
|
||||
|
||||
test('AskUserQuestion is NOT auto-added when canUseTool is absent', async () => {
|
||||
freshSem();
|
||||
const stub: StubConfig = {
|
||||
streams: [[systemInit(), assistantTurn([{ type: 'text', text: 'ok' }]), resultSuccess()]],
|
||||
calls: [],
|
||||
};
|
||||
await runAgentSdkTest({
|
||||
...BASE_OPTS,
|
||||
allowedTools: ['Read', 'Grep'],
|
||||
queryProvider: makeStubProvider(stub),
|
||||
});
|
||||
const opts = stub.calls[0]!.options!;
|
||||
expect(opts.allowedTools).not.toContain('AskUserQuestion');
|
||||
});
|
||||
|
||||
test('passThroughNonAskUserQuestion helper returns allow+updatedInput', async () => {
|
||||
const { passThroughNonAskUserQuestion } = await import('../test/helpers/agent-sdk-runner');
|
||||
const result = passThroughNonAskUserQuestion('Read', { file_path: '/tmp/x' });
|
||||
expect(result.behavior).toBe('allow');
|
||||
expect(result.updatedInput).toEqual({ file_path: '/tmp/x' });
|
||||
});
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Rate-limit retry (three shapes)
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
/**
|
||||
* E2E harness audit — every skill with `interactive: true` in its frontmatter
|
||||
* must have at least one test file that uses `canUseTool` via the extended
|
||||
* agent-sdk-runner. This prevents future drift where a skill opts into the
|
||||
* handshake without adding real coverage.
|
||||
*
|
||||
* Runs as a free unit test (no API calls). Pure filesystem scan.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
|
||||
const ROOT = path.resolve(import.meta.dir, '..');
|
||||
const SKILL_GLOBS = [
|
||||
'plan-ceo-review',
|
||||
'plan-eng-review',
|
||||
'plan-design-review',
|
||||
'plan-devex-review',
|
||||
'office-hours',
|
||||
'codex',
|
||||
'investigate',
|
||||
'qa',
|
||||
'retro',
|
||||
'cso',
|
||||
'review',
|
||||
'ship',
|
||||
'design-review',
|
||||
'devex-review',
|
||||
'qa-only',
|
||||
'design-consultation',
|
||||
'design-shotgun',
|
||||
'autoplan',
|
||||
'land-and-deploy',
|
||||
'plan-tune',
|
||||
'document-release',
|
||||
'context-save',
|
||||
'context-restore',
|
||||
'health',
|
||||
'setup-deploy',
|
||||
'setup-browser-cookies',
|
||||
'canary',
|
||||
'learn',
|
||||
'benchmark',
|
||||
'benchmark-models',
|
||||
'make-pdf',
|
||||
'open-gstack-browser',
|
||||
'gstack-upgrade',
|
||||
'pair-agent',
|
||||
'design-html',
|
||||
'freeze',
|
||||
'unfreeze',
|
||||
'careful',
|
||||
'guard',
|
||||
];
|
||||
|
||||
/**
|
||||
* Load .tmpl files for each skill and return the names of those that have
|
||||
* `interactive: true` in frontmatter.
|
||||
*/
|
||||
function findInteractiveSkills(): string[] {
|
||||
const interactive: string[] = [];
|
||||
for (const skill of SKILL_GLOBS) {
|
||||
const tmplPath = path.join(ROOT, skill, 'SKILL.md.tmpl');
|
||||
if (!fs.existsSync(tmplPath)) continue;
|
||||
const content = fs.readFileSync(tmplPath, 'utf-8');
|
||||
// Frontmatter lives between the first '---' and the next '---'.
|
||||
const fmEnd = content.indexOf('\n---', 4);
|
||||
if (fmEnd < 0) continue;
|
||||
const frontmatter = content.slice(0, fmEnd);
|
||||
if (/^interactive:\s*true\s*$/m.test(frontmatter)) {
|
||||
interactive.push(skill);
|
||||
}
|
||||
}
|
||||
return interactive;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scan a test file's contents for the canUseTool-via-harness pattern.
|
||||
* Either: direct canUseTool usage in runAgentSdkTest, or usage of the
|
||||
* shared plan-mode-handshake-helpers that wrap it.
|
||||
*/
|
||||
function hasCanUseToolCoverage(testFile: string): boolean {
|
||||
const content = fs.readFileSync(testFile, 'utf-8');
|
||||
if (content.includes('canUseTool')) return true;
|
||||
if (content.includes('runPlanModeHandshakeTest')) return true;
|
||||
return false;
|
||||
}
|
||||
|
||||
describe('E2E harness audit — interactive skills must have canUseTool coverage', () => {
|
||||
test('every interactive: true skill has at least one canUseTool test', () => {
|
||||
const interactive = findInteractiveSkills();
|
||||
expect(interactive.length).toBeGreaterThan(0);
|
||||
|
||||
const testFiles = fs
|
||||
.readdirSync(path.join(ROOT, 'test'))
|
||||
.filter((f) => f.startsWith('skill-e2e-') && f.endsWith('.test.ts'))
|
||||
.map((f) => path.join(ROOT, 'test', f));
|
||||
|
||||
const filesWithCoverage = testFiles.filter(hasCanUseToolCoverage);
|
||||
|
||||
for (const skill of interactive) {
|
||||
// Match the skill name in any test file that uses canUseTool. File
|
||||
// naming convention is `skill-e2e-<skill>-*.test.ts` — either the full
|
||||
// name (plan-ceo-review) or a subset token.
|
||||
const hasDedicatedTest = filesWithCoverage.some((f) => {
|
||||
const base = path.basename(f, '.test.ts');
|
||||
return base.includes(skill) || base.includes(skill.replace(/-review$/, ''));
|
||||
});
|
||||
expect(hasDedicatedTest, `skill "${skill}" has interactive:true but no canUseTool-based E2E test`).toBe(true);
|
||||
}
|
||||
});
|
||||
});
|
||||
@@ -2774,3 +2774,93 @@ describe('voice-triggers processing', () => {
|
||||
expect(frontmatter).not.toContain('voice-triggers:');
|
||||
});
|
||||
});
|
||||
|
||||
describe('plan-mode handshake (interactive: true) resolver', () => {
|
||||
const INTERACTIVE_SKILLS = [
|
||||
'plan-ceo-review',
|
||||
'plan-eng-review',
|
||||
'plan-design-review',
|
||||
'plan-devex-review',
|
||||
];
|
||||
|
||||
const HANDSHAKE_MARKER = '## Plan Mode Handshake';
|
||||
|
||||
test.each(INTERACTIVE_SKILLS)(
|
||||
'%s (Claude host) SKILL.md contains the handshake section',
|
||||
(skill) => {
|
||||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||||
expect(content).toContain(HANDSHAKE_MARKER);
|
||||
expect(content).toContain(
|
||||
'Plan mode is active. The user indicated that they do not want you to execute yet',
|
||||
);
|
||||
},
|
||||
);
|
||||
|
||||
test('handshake is absent from non-interactive Claude skills', () => {
|
||||
const nonInteractive = ['ship', 'review', 'qa', 'office-hours', 'codex', 'retro', 'cso'];
|
||||
for (const skill of nonInteractive) {
|
||||
const content = fs.readFileSync(path.join(ROOT, skill, 'SKILL.md'), 'utf-8');
|
||||
expect(content).not.toContain(HANDSHAKE_MARKER);
|
||||
}
|
||||
});
|
||||
|
||||
test('handshake is absent from non-Claude host outputs when present on disk', () => {
|
||||
// Non-Claude hosts render to hostSubdirs (.agents/, .openclaw/, etc). The
|
||||
// handshake resolver returns '' when ctx.host !== 'claude', so those
|
||||
// outputs must not contain the marker. The current gen-skill-docs layout
|
||||
// prefixes skill names as `gstack-<skill>` under the hostSubdir; older
|
||||
// layouts used `gstack/<skill>` (no prefix). Only stable-present paths
|
||||
// are asserted — older ones may or may not exist per install history.
|
||||
const candidateOutputs = [
|
||||
// Current prefixed layout
|
||||
path.join(ROOT, '.agents', 'skills', 'gstack-plan-ceo-review', 'SKILL.md'),
|
||||
path.join(ROOT, '.openclaw', 'skills', 'gstack-plan-ceo-review', 'SKILL.md'),
|
||||
path.join(ROOT, '.opencode', 'skills', 'gstack-plan-ceo-review', 'SKILL.md'),
|
||||
path.join(ROOT, '.factory', 'skills', 'gstack-plan-ceo-review', 'SKILL.md'),
|
||||
path.join(ROOT, '.hermes', 'skills', 'gstack-plan-ceo-review', 'SKILL.md'),
|
||||
];
|
||||
let checked = 0;
|
||||
for (const out of candidateOutputs) {
|
||||
if (fs.existsSync(out)) {
|
||||
const content = fs.readFileSync(out, 'utf-8');
|
||||
expect(content).not.toContain(HANDSHAKE_MARKER);
|
||||
checked++;
|
||||
}
|
||||
}
|
||||
// At least one non-Claude host's output should exist after a full gen
|
||||
// run; this test is meaningful only if we checked something. If no
|
||||
// non-Claude outputs exist locally, the cross-host guarantee is still
|
||||
// enforced by the resolver's ctx.host check; this test is belt-and-
|
||||
// suspenders and becomes a no-op rather than a false positive.
|
||||
if (checked === 0) {
|
||||
// eslint-disable-next-line no-console
|
||||
console.warn(
|
||||
'plan-mode handshake: no non-Claude host outputs found for cross-host absence check — ' +
|
||||
'run `bun run gen:skill-docs --host all` to populate',
|
||||
);
|
||||
}
|
||||
});
|
||||
|
||||
test('0C-bis STOP block present in plan-ceo-review/SKILL.md', () => {
|
||||
const content = fs.readFileSync(path.join(ROOT, 'plan-ceo-review', 'SKILL.md'), 'utf-8');
|
||||
const presentIdx = content.indexOf('Present these approach options via AskUserQuestion');
|
||||
const preludeIdx = content.indexOf('### 0D-prelude');
|
||||
expect(presentIdx).toBeGreaterThan(0);
|
||||
expect(preludeIdx).toBeGreaterThan(presentIdx);
|
||||
const between = content.slice(presentIdx, preludeIdx);
|
||||
expect(between).toContain('**STOP.**');
|
||||
expect(between).toContain('Do NOT proceed to Step 0D or 0F until the user responds to 0C-bis');
|
||||
});
|
||||
|
||||
test('handshake resolver is wired BEFORE generateUpgradeCheck in preamble', () => {
|
||||
const content = fs.readFileSync(
|
||||
path.join(ROOT, 'plan-ceo-review', 'SKILL.md'),
|
||||
'utf-8',
|
||||
);
|
||||
const handshakeIdx = content.indexOf(HANDSHAKE_MARKER);
|
||||
const upgradeIdx = content.indexOf('UPGRADE_AVAILABLE');
|
||||
expect(handshakeIdx).toBeGreaterThan(0);
|
||||
expect(upgradeIdx).toBeGreaterThan(0);
|
||||
expect(handshakeIdx).toBeLessThan(upgradeIdx);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -31,6 +31,7 @@ import {
|
||||
type PermissionMode,
|
||||
type SettingSource,
|
||||
type Options,
|
||||
type CanUseTool,
|
||||
} from '@anthropic-ai/claude-agent-sdk';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
@@ -111,6 +112,43 @@ export interface RunAgentSdkOptions {
|
||||
* retries reuse the original workingDirectory (fine for read-only tests).
|
||||
*/
|
||||
onRetry?: (freshDir: string) => void;
|
||||
/**
|
||||
* Optional canUseTool callback. When supplied, the harness flips
|
||||
* permissionMode from 'bypassPermissions' to 'default' so the SDK actually
|
||||
* routes tool-use approval decisions through the callback. Without this
|
||||
* flip, bypassPermissions short-circuits the callback and tests that want
|
||||
* to assert on AskUserQuestion content silently pass without asserting.
|
||||
*
|
||||
* Callback contract matches the SDK: fires on every tool-use approval
|
||||
* request and on AskUserQuestion invocations. For non-AskUserQuestion
|
||||
* tools that tests don't care about, use `passThroughNonAskUserQuestion`
|
||||
* to auto-allow them.
|
||||
*/
|
||||
canUseTool?: CanUseTool;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pass-through helper: auto-allows any tool_use that isn't AskUserQuestion.
|
||||
* Most plan-mode handshake tests only care about the handshake AskUserQuestion;
|
||||
* every other tool (Read, Grep, Bash, Write, Edit, ExitPlanMode) should just
|
||||
* run. Compose with a test-specific AskUserQuestion handler:
|
||||
*
|
||||
* canUseTool: async (toolName, input, options) => {
|
||||
* if (toolName === 'AskUserQuestion') {
|
||||
* // custom assertions + canned answer
|
||||
* return { behavior: 'allow', updatedInput: { questions: input.questions, answers: {...} } };
|
||||
* }
|
||||
* return passThroughNonAskUserQuestion(toolName, input);
|
||||
* }
|
||||
*/
|
||||
export function passThroughNonAskUserQuestion(
|
||||
toolName: string,
|
||||
input: Record<string, unknown>,
|
||||
): { behavior: 'allow'; updatedInput: Record<string, unknown> } {
|
||||
// SDK requires an allow response to include updatedInput — pass the original
|
||||
// input through unchanged so the tool runs as the model intended.
|
||||
void toolName;
|
||||
return { behavior: 'allow', updatedInput: input };
|
||||
}
|
||||
|
||||
export class RateLimitExhaustedError extends Error {
|
||||
@@ -287,19 +325,37 @@ export async function runAgentSdkTest(
|
||||
let terminalResult: SDKResultMessage | null = null;
|
||||
|
||||
try {
|
||||
// When canUseTool is supplied, the SDK must route tool-use approval
|
||||
// decisions through the callback. bypassPermissions short-circuits
|
||||
// that. Flip to 'default' mode so canUseTool actually fires. Tests
|
||||
// that want AskUserQuestion interception without this flip would
|
||||
// silently auto-pass — the exact testability gap D14/D4-eng fix.
|
||||
const hasCanUseTool = typeof opts.canUseTool === 'function';
|
||||
const resolvedPermissionMode: PermissionMode =
|
||||
opts.permissionMode ?? (hasCanUseTool ? 'default' : 'bypassPermissions');
|
||||
|
||||
// When canUseTool is supplied, ensure AskUserQuestion is in the allowed
|
||||
// tools list. Without it, Claude can't invoke AskUserQuestion at all
|
||||
// and the callback never has a chance to fire on it.
|
||||
const baseTools = opts.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'];
|
||||
const resolvedTools =
|
||||
hasCanUseTool && !baseTools.includes('AskUserQuestion')
|
||||
? [...baseTools, 'AskUserQuestion']
|
||||
: baseTools;
|
||||
|
||||
const sdkOpts: Options = {
|
||||
model,
|
||||
cwd: opts.workingDirectory,
|
||||
maxTurns: opts.maxTurns ?? 5,
|
||||
tools: opts.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'],
|
||||
tools: resolvedTools,
|
||||
disallowedTools: opts.disallowedTools,
|
||||
allowedTools: opts.allowedTools ?? ['Read', 'Glob', 'Grep', 'Bash'],
|
||||
permissionMode: opts.permissionMode ?? 'bypassPermissions',
|
||||
allowDangerouslySkipPermissions:
|
||||
(opts.permissionMode ?? 'bypassPermissions') === 'bypassPermissions',
|
||||
allowedTools: resolvedTools,
|
||||
permissionMode: resolvedPermissionMode,
|
||||
allowDangerouslySkipPermissions: resolvedPermissionMode === 'bypassPermissions',
|
||||
settingSources: opts.settingSources ?? [],
|
||||
env: opts.env,
|
||||
pathToClaudeCodeExecutable: opts.pathToClaudeCodeExecutable,
|
||||
...(hasCanUseTool ? { canUseTool: opts.canUseTool } : {}),
|
||||
};
|
||||
// Empty bare string means "omit entirely" (SDK runs with no override).
|
||||
// Any object or non-empty string is passed through.
|
||||
|
||||
@@ -0,0 +1,166 @@
|
||||
/**
|
||||
* Shared helpers for plan-mode handshake E2E tests.
|
||||
*
|
||||
* Four sibling test files (plan-ceo, plan-eng, plan-design, plan-devex) exercise
|
||||
* the identical handshake contract against different skills. This helper
|
||||
* centralizes the canUseTool interceptor and the assertion shape so the four
|
||||
* test files are thin wiring (~40 LOC each) and can't drift out of sync.
|
||||
*
|
||||
* See scripts/resolvers/preamble/generate-plan-mode-handshake.ts for the
|
||||
* handshake prose that the tests below assert against.
|
||||
*/
|
||||
|
||||
import { expect } from 'bun:test';
|
||||
import * as fs from 'fs';
|
||||
import * as path from 'path';
|
||||
import * as os from 'os';
|
||||
import { execSync } from 'child_process';
|
||||
import {
|
||||
runAgentSdkTest,
|
||||
passThroughNonAskUserQuestion,
|
||||
resolveClaudeBinary,
|
||||
type AgentSdkResult,
|
||||
} from './agent-sdk-runner';
|
||||
|
||||
/** Distinctive phrase matching what Claude Code's harness actually injects. */
|
||||
export const PLAN_MODE_REMINDER =
|
||||
'Plan mode is active. The user indicated that they do not want you to execute yet';
|
||||
|
||||
export interface HandshakeCaptureResult {
|
||||
sdkResult: AgentSdkResult;
|
||||
/** Each AskUserQuestion that fired, with its input payload. */
|
||||
askUserQuestions: Array<{ input: Record<string, unknown>; orderIndex: number }>;
|
||||
/** Tool-use events in the order they fired (names only). */
|
||||
toolOrder: string[];
|
||||
/** Whether any Write or Edit tool fired BEFORE the first AskUserQuestion. */
|
||||
writeOrEditBeforeAsk: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Run a skill via the Agent SDK with canUseTool intercepting every tool use.
|
||||
* Inject the plan-mode distinctive phrase into the system prompt and auto-
|
||||
* answer the handshake with the given answerLabel ("Exit" or "Cancel"). Return
|
||||
* the captured events for assertion.
|
||||
*/
|
||||
export async function runPlanModeHandshakeTest(opts: {
|
||||
/** Skill name, e.g. 'plan-ceo-review'. */
|
||||
skillName: string;
|
||||
/** "Exit" to pick option A (exit-and-rerun) or "Cancel" for option C. */
|
||||
answerLabel: 'Exit' | 'Cancel';
|
||||
/** If true, DO NOT inject the reminder — used by the no-op regression test. */
|
||||
omitPlanModeReminder?: boolean;
|
||||
/** Max turns for the SDK call (default 4 — handshake + exit should fit easily). */
|
||||
maxTurns?: number;
|
||||
}): Promise<HandshakeCaptureResult> {
|
||||
const { skillName, answerLabel, omitPlanModeReminder, maxTurns } = opts;
|
||||
|
||||
const askUserQuestions: HandshakeCaptureResult['askUserQuestions'] = [];
|
||||
const toolOrder: string[] = [];
|
||||
let toolIndex = 0;
|
||||
let firstAskIndex = -1;
|
||||
|
||||
const workingDir = fs.mkdtempSync(
|
||||
path.join(os.tmpdir(), `plan-mode-handshake-${skillName}-`),
|
||||
);
|
||||
|
||||
// The SDK requires AskUserQuestion to be in the allowed tools list. The
|
||||
// harness auto-adds it when canUseTool is supplied, but we also want Read
|
||||
// so the skill can load its own file if it tries to.
|
||||
const binary = resolveClaudeBinary();
|
||||
|
||||
try {
|
||||
// Inject the distinctive phrase into the system prompt by appending it to
|
||||
// the default Claude Code preset. Claude Code's real plan mode uses an
|
||||
// injected system-reminder; in SDK tests we use systemPrompt.append which
|
||||
// the model treats as equally authoritative.
|
||||
const reminderAppend = omitPlanModeReminder
|
||||
? ''
|
||||
: `\n\n<system-reminder>\n${PLAN_MODE_REMINDER}. This supercedes any other instructions you have received.\n</system-reminder>\n`;
|
||||
|
||||
const sdkResult = await runAgentSdkTest({
|
||||
systemPrompt: {
|
||||
type: 'preset',
|
||||
preset: 'claude_code',
|
||||
append: reminderAppend,
|
||||
},
|
||||
userPrompt: `Read the skill file at ${path.resolve(
|
||||
import.meta.dir,
|
||||
'..',
|
||||
'..',
|
||||
skillName,
|
||||
'SKILL.md',
|
||||
)} and follow its instructions. There is no real plan to review — just start the skill and respond to any AskUserQuestion that fires.`,
|
||||
workingDirectory: workingDir,
|
||||
maxTurns: maxTurns ?? 4,
|
||||
allowedTools: ['Read', 'Grep', 'Glob', 'Bash'],
|
||||
...(binary ? { pathToClaudeCodeExecutable: binary } : {}),
|
||||
canUseTool: async (toolName, input) => {
|
||||
toolOrder.push(toolName);
|
||||
if (toolName === 'AskUserQuestion') {
|
||||
if (firstAskIndex === -1) firstAskIndex = toolIndex;
|
||||
askUserQuestions.push({ input, orderIndex: toolIndex });
|
||||
toolIndex++;
|
||||
// Auto-answer with the label the test specified.
|
||||
const q = (input.questions as Array<{ question: string; options: Array<{ label: string }> }>)[0];
|
||||
const matched = q.options.find((o) => o.label.includes(answerLabel));
|
||||
const answer = matched ? matched.label : q.options[0]!.label;
|
||||
return {
|
||||
behavior: 'allow',
|
||||
updatedInput: {
|
||||
questions: input.questions,
|
||||
answers: { [q.question]: answer },
|
||||
},
|
||||
};
|
||||
}
|
||||
toolIndex++;
|
||||
return passThroughNonAskUserQuestion(toolName, input);
|
||||
},
|
||||
});
|
||||
|
||||
const writeOrEditBeforeAsk =
|
||||
firstAskIndex > 0 &&
|
||||
toolOrder.slice(0, firstAskIndex).some((t) => t === 'Write' || t === 'Edit');
|
||||
|
||||
return { sdkResult, askUserQuestions, toolOrder, writeOrEditBeforeAsk };
|
||||
} finally {
|
||||
try {
|
||||
fs.rmSync(workingDir, { recursive: true, force: true });
|
||||
} catch { /* ignore cleanup errors */ }
|
||||
}
|
||||
}
|
||||
|
||||
/** Assert the shape of a fired handshake AskUserQuestion. */
|
||||
export function assertHandshakeShape(
|
||||
aq: { input: Record<string, unknown> },
|
||||
): void {
|
||||
const questions = aq.input.questions as Array<{
|
||||
question: string;
|
||||
options: Array<{ label: string }>;
|
||||
}>;
|
||||
expect(questions).toBeDefined();
|
||||
expect(questions.length).toBe(1);
|
||||
const q = questions[0]!;
|
||||
// D8 dropped Option B; handshake has exactly 2 options.
|
||||
expect(q.options.length).toBe(2);
|
||||
const labels = q.options.map((o) => o.label);
|
||||
expect(labels.some((l) => l.includes('Exit'))).toBe(true);
|
||||
expect(labels.some((l) => l.includes('Cancel'))).toBe(true);
|
||||
}
|
||||
|
||||
/** Read the skill-usage.jsonl log and return handshake entries. */
|
||||
export function readHandshakeLog(): Array<Record<string, unknown>> {
|
||||
const logPath = path.join(os.homedir(), '.gstack', 'analytics', 'skill-usage.jsonl');
|
||||
if (!fs.existsSync(logPath)) return [];
|
||||
const lines = fs.readFileSync(logPath, 'utf-8').split('\n').filter(Boolean);
|
||||
return lines
|
||||
.map((line) => {
|
||||
try {
|
||||
return JSON.parse(line);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})
|
||||
.filter((x): x is Record<string, unknown> => x !== null && x.event === 'plan_mode_handshake');
|
||||
}
|
||||
|
||||
export { execSync };
|
||||
@@ -82,6 +82,17 @@ export const E2E_TOUCHFILES: Record<string, string[]> = {
|
||||
'plan-eng-review-artifact': ['plan-eng-review/**'],
|
||||
'plan-review-report': ['plan-eng-review/**', 'scripts/gen-skill-docs.ts'],
|
||||
|
||||
// Plan-mode handshake (v1.10.2.0) — gate-tier safety regression tests.
|
||||
// Each fires when any of: the interactive skill's template, the resolver,
|
||||
// preamble composition, the Agent SDK harness, the question registry, or
|
||||
// the one-way-door classifier changes.
|
||||
'plan-ceo-review-plan-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
'plan-eng-review-plan-mode': ['plan-eng-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
'plan-design-review-plan-mode-handshake': ['plan-design-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
'plan-devex-review-plan-mode': ['plan-devex-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'scripts/question-registry.ts', 'scripts/one-way-doors.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
'plan-mode-no-op': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'scripts/resolvers/preamble.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
'e2e-harness-audit': ['plan-ceo-review/**', 'plan-eng-review/**', 'plan-design-review/**', 'plan-devex-review/**', 'scripts/resolvers/preamble/generate-plan-mode-handshake.ts', 'test/helpers/agent-sdk-runner.ts'],
|
||||
|
||||
// AskUserQuestion format regression (RECOMMENDATION + Completeness: N/10)
|
||||
// Fires when either template OR the two preamble resolvers change.
|
||||
'plan-ceo-review-format-mode': ['plan-ceo-review/**', 'scripts/resolvers/preamble/generate-ask-user-format.ts', 'scripts/resolvers/preamble/generate-completeness-section.ts', 'scripts/resolvers/preamble.ts', 'model-overlays/opus-4-7.md'],
|
||||
@@ -317,6 +328,14 @@ export const E2E_TIERS: Record<string, 'gate' | 'periodic'> = {
|
||||
'plan-eng-coverage-audit': 'gate',
|
||||
'plan-review-report': 'gate',
|
||||
|
||||
// Plan-mode handshake — deterministic safety regression, gate-tier
|
||||
'plan-ceo-review-plan-mode': 'gate',
|
||||
'plan-eng-review-plan-mode': 'gate',
|
||||
'plan-design-review-plan-mode-handshake': 'gate',
|
||||
'plan-devex-review-plan-mode': 'gate',
|
||||
'plan-mode-no-op': 'gate',
|
||||
'e2e-harness-audit': 'gate',
|
||||
|
||||
// AskUserQuestion format regression — periodic (Opus 4.7 non-deterministic benchmark)
|
||||
'plan-ceo-review-format-mode': 'periodic',
|
||||
'plan-ceo-review-format-approach': 'periodic',
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
/**
|
||||
* plan-ceo-review plan-mode handshake E2E (gate tier, paid).
|
||||
*
|
||||
* Asserts: when /plan-ceo-review is invoked with the plan-mode distinctive
|
||||
* phrase in the system reminder, the skill fires AskUserQuestion FIRST
|
||||
* (before any Write or Edit), the question has exactly 2 options (A exit,
|
||||
* C cancel), picking "Exit" leads to an orderly exit with no plan file
|
||||
* written.
|
||||
*
|
||||
* Cost: ~$0.50–$1.00 per run. Gated: EVALS=1 EVALS_TIER=gate.
|
||||
* Depends on: scripts/resolvers/preamble/generate-plan-mode-handshake.ts,
|
||||
* test/helpers/agent-sdk-runner.ts (canUseTool extension).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
assertHandshakeShape,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-ceo-review plan-mode handshake (gate)', () => {
|
||||
test('handshake fires before any Write/Edit when plan mode is detected', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-ceo-review',
|
||||
answerLabel: 'Exit',
|
||||
});
|
||||
|
||||
// Handshake must have fired at least once.
|
||||
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
||||
// Critically: no Write or Edit fired before the first AskUserQuestion.
|
||||
// This is the bug v1.10.2.0 fixes — plan mode used to allow silent
|
||||
// plan-file writes without any interactive gate.
|
||||
expect(result.writeOrEditBeforeAsk).toBe(false);
|
||||
// Handshake shape: 2 options (Exit/Cancel), Option B dropped per D8.
|
||||
assertHandshakeShape(result.askUserQuestions[0]!);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* plan-design-review plan-mode handshake E2E (gate tier, paid).
|
||||
*
|
||||
* See test/skill-e2e-plan-ceo-plan-mode.test.ts for the shared assertion
|
||||
* contract. This file exercises the same handshake against /plan-design-review.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
assertHandshakeShape,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-design-review plan-mode handshake (gate)', () => {
|
||||
test('handshake fires before any Write/Edit when plan mode is detected', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-design-review',
|
||||
answerLabel: 'Cancel', // exercise the C-cancel branch instead of A-exit
|
||||
});
|
||||
|
||||
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(result.writeOrEditBeforeAsk).toBe(false);
|
||||
assertHandshakeShape(result.askUserQuestions[0]!);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* plan-devex-review plan-mode handshake E2E (gate tier, paid).
|
||||
*
|
||||
* See test/skill-e2e-plan-ceo-plan-mode.test.ts for the shared assertion
|
||||
* contract. This file exercises the same handshake against /plan-devex-review.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
assertHandshakeShape,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-devex-review plan-mode handshake (gate)', () => {
|
||||
test('handshake fires before any Write/Edit when plan mode is detected', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-devex-review',
|
||||
answerLabel: 'Exit',
|
||||
});
|
||||
|
||||
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(result.writeOrEditBeforeAsk).toBe(false);
|
||||
assertHandshakeShape(result.askUserQuestions[0]!);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,28 @@
|
||||
/**
|
||||
* plan-eng-review plan-mode handshake E2E (gate tier, paid).
|
||||
*
|
||||
* See test/skill-e2e-plan-ceo-plan-mode.test.ts for the shared assertion
|
||||
* contract. This file exercises the same handshake against /plan-eng-review.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
assertHandshakeShape,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-eng-review plan-mode handshake (gate)', () => {
|
||||
test('handshake fires before any Write/Edit when plan mode is detected', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-eng-review',
|
||||
answerLabel: 'Exit',
|
||||
});
|
||||
|
||||
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
||||
expect(result.writeOrEditBeforeAsk).toBe(false);
|
||||
assertHandshakeShape(result.askUserQuestions[0]!);
|
||||
}, 120_000);
|
||||
});
|
||||
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* Plan-mode handshake negative regression (gate tier, paid).
|
||||
*
|
||||
* Asserts: when /plan-ceo-review is invoked WITHOUT the plan-mode distinctive
|
||||
* phrase in the system reminder, the handshake does NOT fire. The skill
|
||||
* should proceed to its normal Step 0 flow. This is the REGRESSION RULE
|
||||
* guardrail — the handshake must be a no-op outside plan mode or it breaks
|
||||
* every existing interactive-review session.
|
||||
*
|
||||
* Cost: ~$0.50 per run. Gated: EVALS=1 EVALS_TIER=gate.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
PLAN_MODE_REMINDER,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-mode handshake no-op outside plan mode (gate regression)', () => {
|
||||
test('handshake does NOT fire when distinctive phrase is absent', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-ceo-review',
|
||||
answerLabel: 'Exit', // ignored — handshake should never fire
|
||||
omitPlanModeReminder: true,
|
||||
maxTurns: 3, // enough to see Step 0 start, but bounded
|
||||
});
|
||||
|
||||
// The handshake AskUserQuestion should NOT have fired during Step 0 entry.
|
||||
// Other AskUserQuestions may fire later in the skill (e.g., Step 0C-bis),
|
||||
// but they will NOT have the handshake's question text.
|
||||
for (const aq of result.askUserQuestions) {
|
||||
const questions = aq.input.questions as Array<{ question: string }>;
|
||||
for (const q of questions) {
|
||||
// The handshake's question mentions the distinctive phrase in its
|
||||
// prose; a non-handshake AskUserQuestion won't.
|
||||
expect(q.question).not.toContain(PLAN_MODE_REMINDER);
|
||||
}
|
||||
}
|
||||
}, 120_000);
|
||||
});
|
||||
+10
-2
@@ -85,8 +85,16 @@ describe('selectTests', () => {
|
||||
expect(result.selected).toContain('codex-offered-ceo-review');
|
||||
expect(result.selected).toContain('plan-ceo-review-format-mode');
|
||||
expect(result.selected).toContain('plan-ceo-review-format-approach');
|
||||
expect(result.selected.length).toBe(8);
|
||||
expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 8);
|
||||
// v1.10.2.0 plan-mode handshake entries also depend on plan-ceo-review/**
|
||||
expect(result.selected).toContain('plan-ceo-review-plan-mode');
|
||||
expect(result.selected).toContain('plan-mode-no-op');
|
||||
expect(result.selected).toContain('e2e-harness-audit');
|
||||
expect(result.selected).toContain('plan-ceo-review-prosons-cadence');
|
||||
expect(result.selected).toContain('plan-review-prosons-format');
|
||||
expect(result.selected).toContain('plan-review-prosons-hardstop-neg');
|
||||
expect(result.selected).toContain('plan-review-prosons-neutral-neg');
|
||||
expect(result.selected.length).toBe(15);
|
||||
expect(result.skipped.length).toBe(Object.keys(E2E_TOUCHFILES).length - 15);
|
||||
});
|
||||
|
||||
test('global touchfile triggers ALL tests', () => {
|
||||
|
||||
Reference in New Issue
Block a user