mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-01 19:25:10 +02:00
v1.11.1.0 fix: plan-mode handshake + canUseTool test harness (#1182)
* feat: plan-mode handshake for interactive review skills Add a preamble-level STOP-Ask handshake that fires when the user invokes any of the 4 interactive review skills (plan-ceo-review, plan-eng-review, plan-design-review, plan-devex-review) while their Claude Code session is in plan mode. Without this gate, plan mode's "this supercedes any other instructions" system-reminder outranked the skills' interactive STOP gates and the skills silently wrote plan files without any per-finding AskUserQuestion. The handshake offers 2 options (exit-and-rerun, cancel) — the original third "stay and batch" option was dropped after two independent reviewers flagged it as a silent bypass of the skills' anti-skip rule. Architecture decisions (CEO+Eng review): - Preamble-level resolver, not per-template injection (Codex finding #2) - Position 1 in preamble composition: after bash block (_SESSION_ID live), before onboarding AskUserQuestion gates (so fresh-install users see the handshake first, not drowned in telemetry/proactive/routing prompts) - Generator-only `interactive: true` frontmatter flag, following the `preamble-tier` precedent (no host-config frontmatter allowlist edits) - Host-scoped to Claude via `ctx.host === 'claude'` check inside the resolver (simpler than `suppressedResolvers` which only gates `{{}}` placeholders) - One-way-door classification in scripts/question-registry.ts for all 4 skills so question-tuning `never-ask` preferences can't suppress the gate - Synchronous telemetry write to ~/.gstack/analytics/skill-usage.jsonl on handshake fire (captures A-exit and C-cancel outcomes that terminate the skill before end-of-run telemetry runs) Also adds an explicit STOP block to plan-ceo-review Step 0C-bis so the approach-selection question can't silently skip to mode selection. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * feat: extend agent-sdk-runner with canUseTool for AskUserQuestion interception Test harness at test/helpers/agent-sdk-runner.ts gains an optional `canUseTool` callback parameter. When a test supplies it, the harness flips `permissionMode` from `bypassPermissions` (overlay-harness default) to `default` so the SDK actually invokes the callback on every tool use, and auto-adds `AskUserQuestion` to `allowedTools` so Claude can fire it at all. Exports a `passThroughNonAskUserQuestion` helper so tests that only want to intercept AskUserQuestion can auto-allow every other tool with one line: `return passThroughNonAskUserQuestion(toolName, input)`. This is the foundation for D14 — every future interactive-skill E2E test can now assert on AskUserQuestion shape and routing. Previous E2E tests at `test/skill-e2e.test.ts` explicitly instructed the model to skip AskUserQuestion ("non-interactive run") which meant no test could actually verify the question content or routing. 6 new unit tests in test/agent-sdk-runner.test.ts cover: - permissionMode flips to 'default' when canUseTool supplied - permissionMode stays 'bypassPermissions' when canUseTool absent - canUseTool callback reaches the SDK options - AskUserQuestion auto-added to allowedTools when canUseTool supplied - AskUserQuestion NOT added when canUseTool absent - passThroughNonAskUserQuestion helper returns allow+updatedInput Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * test: plan-mode handshake E2E coverage and unit assertions Adds 6 E2E test files and 8 new unit assertions to verify the plan-mode handshake works end-to-end and stays correct under regeneration. E2E tests (gate-tier, paid, EVALS=1 EVALS_TIER=gate): - test/skill-e2e-plan-ceo-plan-mode.test.ts — handshake fires before any Write/Edit when plan-mode distinctive phrase is present; 2-option shape (Exit/Cancel); option A routes to ExitPlanMode cleanly - test/skill-e2e-plan-eng-plan-mode.test.ts — same contract for plan-eng - test/skill-e2e-plan-design-plan-mode.test.ts — same contract for plan-design; exercises C-cancel branch instead of A-exit - test/skill-e2e-plan-devex-plan-mode.test.ts — same contract for plan-devex - test/skill-e2e-plan-mode-no-op.test.ts — negative regression: handshake must NOT fire when distinctive phrase is absent; skill proceeds normally through Step 0 (REGRESSION RULE guardrail against breaking existing interactive-review sessions) - test/e2e-harness-audit.test.ts — free unit test asserting every `interactive: true` skill has at least one canUseTool-using test file (prevents future drift where a skill opts in without coverage) Shared helper test/helpers/plan-mode-handshake-helpers.ts centralizes the canUseTool interceptor + distinctive-phrase injection so the 4 sibling E2E tests are thin wiring (~20 LOC each) and can't drift out of sync. Unit assertions added to test/gen-skill-docs.test.ts: - handshake section present in all 4 Claude-generated SKILL.md files - handshake section absent from non-interactive Claude skills (ship, review, qa, office-hours, codex, retro, cso) - handshake section absent from non-Claude host outputs (.agents, etc.) - 0C-bis STOP block present in plan-ceo-review/SKILL.md at correct position (between the "Present these approach options" line and "### 0D-prelude" header) - handshake resolver wired BEFORE generateUpgradeCheck in preamble composition order 6 new gate-tier entries added to test/helpers/touchfiles.ts so any change to the handshake resolver, preamble composition, skill templates, question registry, one-way-door classifier, or agent-sdk-runner fires the relevant E2E tests. test/touchfiles.test.ts updated for the new selection count (plan-ceo-review/** now triggers 15 tests, up from 8). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> * chore(v1.11.1.0): VERSION bump + CHANGELOG entry + TODOS follow-ups Bumps from main's v1.11.0.0 to v1.11.1.0 (PATCH — bug-fix release, no new user-facing artifacts). CHANGELOG entry covers the plan-mode handshake, agent-sdk-runner canUseTool extension, and the 2 follow-up TODOs. CHANGELOG order: v1.11.1.0 (this) → v1.11.0.0 (workspace-aware ship, merged from main) → v1.10.1.0 (overlay efficacy harness). No duplicate headers. Syncs package.json version to match VERSION per the Step 12 idempotency invariant (both files must agree or /ship halts). TODOS.md: - Preserves the Testing/security-bench-haiku-responses P1 added on main - Adds P1 "Structural STOP-Ask forcing function" — broader class of the bug this release fixes - Adds P2 "Apply interactive: true to non-review skills (office-hours, codex, investigate, qa, retro, cso)" Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
@@ -425,7 +425,11 @@ function processTemplate(tmplPath: string, host: Host = 'claude'): { outputPath:
|
||||
const tierMatch = tmplContent.match(/^preamble-tier:\s*(\d+)$/m);
|
||||
const preambleTier = tierMatch ? parseInt(tierMatch[1], 10) : undefined;
|
||||
|
||||
const ctx: TemplateContext = { skillName, tmplPath, benefitsFrom, host, paths: HOST_PATHS[host], preambleTier, model: MODEL_ARG_VAL };
|
||||
// Extract interactive flag from frontmatter (generator-only; controls plan-mode handshake inclusion)
|
||||
const interactiveMatch = tmplContent.match(/^interactive:\s*(true|false)\s*$/m);
|
||||
const interactive = interactiveMatch ? interactiveMatch[1] === 'true' : undefined;
|
||||
|
||||
const ctx: TemplateContext = { skillName, tmplPath, benefitsFrom, host, paths: HOST_PATHS[host], preambleTier, model: MODEL_ARG_VAL, interactive };
|
||||
|
||||
// Replace placeholders (supports parameterized: {{NAME:arg1:arg2}})
|
||||
// Config-driven: suppressedResolvers return empty string for this host
|
||||
|
||||
@@ -261,6 +261,45 @@ export const QUESTIONS = {
|
||||
description: "Approve the design doc, revise sections, or start over?",
|
||||
},
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// Plan-mode handshake — fires at the top of any interactive review skill
|
||||
// when the user is in plan mode. Safety-critical, always asked regardless
|
||||
// of user's tuning preferences. See scripts/resolvers/preamble/generate-
|
||||
// plan-mode-handshake.ts.
|
||||
// -----------------------------------------------------------------------
|
||||
'plan-ceo-review-plan-mode-handshake': {
|
||||
id: 'plan-ceo-review-plan-mode-handshake',
|
||||
skill: 'plan-ceo-review',
|
||||
category: 'routing',
|
||||
door_type: 'one-way',
|
||||
options: ['exit-and-rerun', 'cancel'],
|
||||
description: "Plan mode detected — exit and rerun interactively, or cancel?",
|
||||
},
|
||||
'plan-eng-review-plan-mode-handshake': {
|
||||
id: 'plan-eng-review-plan-mode-handshake',
|
||||
skill: 'plan-eng-review',
|
||||
category: 'routing',
|
||||
door_type: 'one-way',
|
||||
options: ['exit-and-rerun', 'cancel'],
|
||||
description: "Plan mode detected — exit and rerun interactively, or cancel?",
|
||||
},
|
||||
'plan-design-review-plan-mode-handshake': {
|
||||
id: 'plan-design-review-plan-mode-handshake',
|
||||
skill: 'plan-design-review',
|
||||
category: 'routing',
|
||||
door_type: 'one-way',
|
||||
options: ['exit-and-rerun', 'cancel'],
|
||||
description: "Plan mode detected — exit and rerun interactively, or cancel?",
|
||||
},
|
||||
'plan-devex-review-plan-mode-handshake': {
|
||||
id: 'plan-devex-review-plan-mode-handshake',
|
||||
skill: 'plan-devex-review',
|
||||
category: 'routing',
|
||||
door_type: 'one-way',
|
||||
options: ['exit-and-rerun', 'cancel'],
|
||||
description: "Plan mode detected — exit and rerun interactively, or cancel?",
|
||||
},
|
||||
|
||||
// -----------------------------------------------------------------------
|
||||
// /plan-ceo-review — scope & strategy
|
||||
// -----------------------------------------------------------------------
|
||||
|
||||
@@ -22,6 +22,7 @@ import { generateQuestionTuning } from './question-tuning';
|
||||
|
||||
// Core bootstrap
|
||||
import { generatePreambleBash } from './preamble/generate-preamble-bash';
|
||||
import { generatePlanModeHandshake } from './preamble/generate-plan-mode-handshake';
|
||||
import { generateUpgradeCheck } from './preamble/generate-upgrade-check';
|
||||
import { generateCompletionStatus } from './preamble/generate-completion-status';
|
||||
|
||||
@@ -78,6 +79,13 @@ export function generatePreamble(ctx: TemplateContext): string {
|
||||
}
|
||||
const sections = [
|
||||
generatePreambleBash(ctx),
|
||||
// Plan-mode handshake at position 1: after bash (so _SESSION_ID / _BRANCH /
|
||||
// _TEL env vars are live for the synchronous telemetry write) and before
|
||||
// all onboarding AskUserQuestion gates (so fresh-install users in plan mode
|
||||
// see the handshake first, not drowned in telemetry / proactive / routing
|
||||
// prompts). Host-scoped to Claude + interactive-frontmatter-scoped inside
|
||||
// the resolver — no-op for every other skill/host combination.
|
||||
generatePlanModeHandshake(ctx),
|
||||
generateUpgradeCheck(ctx),
|
||||
generateWritingStyleMigration(ctx),
|
||||
generateLakeIntro(),
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* Plan-mode handshake resolver.
|
||||
*
|
||||
* Emits a STOP-Ask gate at the very top of the preamble that fires when a user
|
||||
* invokes an interactive review skill while their Claude Code session is in
|
||||
* plan mode. Without this gate, plan mode's "This supercedes any other
|
||||
* instructions you have received" system-reminder wins against the skill's
|
||||
* interactive STOP-Ask workflow and the skill silently writes a plan file
|
||||
* instead of running the per-finding AskUserQuestion loop (v1.10.2.0 bug fix).
|
||||
*
|
||||
* Host scope
|
||||
* ----------
|
||||
* Only renders for Claude host (ctx.host === 'claude'). Other hosts use
|
||||
* different plan-mode semantics (Codex, OpenClaw, etc.) and should not see
|
||||
* Claude-specific ExitPlanMode / esc-esc prose.
|
||||
*
|
||||
* Opt-in
|
||||
* ------
|
||||
* Only renders when the consuming skill's frontmatter has `interactive: true`.
|
||||
* That flag is a generator-only input parsed by scripts/gen-skill-docs.ts
|
||||
* from the skill's .tmpl frontmatter and passed through TemplateContext.
|
||||
* Currently used by: plan-ceo-review, plan-eng-review, plan-design-review,
|
||||
* plan-devex-review.
|
||||
*
|
||||
* Composition position
|
||||
* --------------------
|
||||
* Inserted at index 1 in scripts/resolvers/preamble.ts — after
|
||||
* generatePreambleBash (so _SESSION_ID, _BRANCH, _TEL env vars are live for
|
||||
* the synchronous telemetry write) and before generateUpgradeCheck and all
|
||||
* onboarding AskUserQuestion gates (so fresh-install users in plan mode see
|
||||
* the handshake first, not drowned in telemetry / proactive / routing
|
||||
* prompts).
|
||||
*
|
||||
* One-way door
|
||||
* ------------
|
||||
* The handshake question_id `plan-mode-handshake` is classified door_type
|
||||
* one-way in scripts/question-registry.ts. gstack-question-preference --check
|
||||
* always returns ASK_NORMALLY for it, so a user who set `never-ask` on
|
||||
* another question cannot accidentally suppress this safety gate.
|
||||
*/
|
||||
|
||||
import type { TemplateContext } from '../types';
|
||||
|
||||
export function generatePlanModeHandshake(ctx: TemplateContext): string {
|
||||
if (ctx.host !== 'claude') return '';
|
||||
if (!ctx.interactive) return '';
|
||||
|
||||
return `## Plan Mode Handshake — FIRST, BEFORE ANY ANALYSIS
|
||||
|
||||
**Check every \`<system-reminder>\` in this turn for the literal phrase:**
|
||||
|
||||
> \`Plan mode is active. The user indicated that they do not want you to execute yet\`
|
||||
|
||||
If that phrase is **absent**: proceed normally. This section is a no-op.
|
||||
|
||||
If that phrase is **present**, the user is in plan mode. Plan mode's system
|
||||
reminder says "This supercedes any other instructions you have received,"
|
||||
which conflicts with this skill's interactive STOP-Ask workflow. You MUST
|
||||
resolve the conflict via AskUserQuestion BEFORE reading any files, running
|
||||
any bash, or composing any plan content.
|
||||
|
||||
### What to do when plan mode is detected
|
||||
|
||||
Before emitting the AskUserQuestion, run this bash block synchronously to
|
||||
log that the handshake fired (captures A-exit and C-cancel outcomes that
|
||||
would terminate the skill before end-of-skill telemetry runs):
|
||||
|
||||
\`\`\`bash
|
||||
# PLAN MODE EXCEPTION — ALWAYS RUN (telemetry-only write to ~/.gstack/)
|
||||
mkdir -p ~/.gstack/analytics
|
||||
echo '{"skill":"'"\${_SKILL_NAME:-unknown}"'","event":"plan_mode_handshake","outcome":"fired","branch":"'"\${_BRANCH:-unknown}"'","session":"'"\${_SESSION_ID:-unknown}"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
\`\`\`
|
||||
|
||||
Then emit exactly **one** AskUserQuestion with \`question_id: "\${SKILL_NAME}-plan-mode-handshake"\`
|
||||
(e.g., \`plan-ceo-review-plan-mode-handshake\`, using the current skill's name)
|
||||
and these two options. The question is classified \`door_type: one-way\` in
|
||||
the question registry for every interactive skill, so question-tuning
|
||||
preferences (\`never-ask\`, \`always-ask\`) do NOT apply — this gate always fires.
|
||||
|
||||
**Question body (follow the AskUserQuestion Format section below):**
|
||||
|
||||
> This skill runs an interactive review that stops at every finding to ask
|
||||
> you a question. Plan mode's default workflow is "read files, write plan,
|
||||
> exit" — that silently bypasses every STOP gate in this skill. How do you
|
||||
> want to proceed?
|
||||
>
|
||||
> **Recommendation: A** because this skill was designed for back-and-forth.
|
||||
> Each scope call and each per-section finding needs your decision before it
|
||||
> lands in the plan. Exiting plan mode and running the skill normally is the
|
||||
> only path that preserves the interactive contract.
|
||||
>
|
||||
> *Note: options differ in kind (workflow shape), not coverage — no
|
||||
> completeness score.*
|
||||
>
|
||||
> **A) Exit plan mode and run interactively (recommended)**
|
||||
> ✅ Every STOP gate in this skill fires as designed — you approve each
|
||||
> scope call, each per-section finding, each cross-model tension before any
|
||||
> decision lands in the plan. No silent bypass.
|
||||
> ✅ Matches the skill's documented workflow. Each AskUserQuestion has a
|
||||
> clear recommendation, pros/cons, and net line you can skim in ~5 seconds.
|
||||
> ❌ Two-step: press esc-esc to exit plan mode, then rerun
|
||||
> \`/plan-{skill-name}\`. Slight context-switch friction, but the alternative
|
||||
> is shipping a rubber-stamp review.
|
||||
>
|
||||
> **C) Cancel — I meant to run something else**
|
||||
> ✅ Clean exit, no partial state, no plan file written, no findings
|
||||
> recorded. Use this if you invoked the skill by mistake.
|
||||
> ❌ No output at all — no review, no plan file. Fine if that's what you
|
||||
> want; otherwise pick A.
|
||||
>
|
||||
> **Net.** Plan mode is incompatible with this skill's per-finding STOP
|
||||
> gates. A is the right choice for any real review; C is the bail-out.
|
||||
|
||||
### Routing the user's answer
|
||||
|
||||
**If the user picks A (exit and rerun):**
|
||||
|
||||
1. Append the outcome to the telemetry log (synchronous, before ExitPlanMode):
|
||||
\`\`\`bash
|
||||
echo '{"skill":"'"\${_SKILL_NAME:-unknown}"'","event":"plan_mode_handshake","outcome":"A-exit","branch":"'"\${_BRANCH:-unknown}"'","session":"'"\${_SESSION_ID:-unknown}"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
\`\`\`
|
||||
2. Respond to the user: "Press **esc-esc** to exit plan mode, then rerun
|
||||
\`/{skill-name}\`. The skill will run interactively with every STOP gate
|
||||
firing as designed."
|
||||
3. Call \`ExitPlanMode\` with an empty plan body (plan mode requires
|
||||
turn-end via AskUserQuestion or ExitPlanMode; there is no plan to
|
||||
approve, so ExitPlanMode with an empty message is the correct exit).
|
||||
|
||||
**If the user picks C (cancel):**
|
||||
|
||||
1. Append the outcome:
|
||||
\`\`\`bash
|
||||
echo '{"skill":"'"\${_SKILL_NAME:-unknown}"'","event":"plan_mode_handshake","outcome":"C-cancel","branch":"'"\${_BRANCH:-unknown}"'","session":"'"\${_SESSION_ID:-unknown}"'","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
\`\`\`
|
||||
2. Tell the user: "Cancelled. No plan written."
|
||||
3. Call \`ExitPlanMode\` with an empty message noting the user cancelled.
|
||||
|
||||
**After the handshake completes (either A or C),** do NOT continue with the
|
||||
rest of this skill's workflow. The handshake is terminal for this turn.
|
||||
`;
|
||||
}
|
||||
@@ -61,6 +61,7 @@ export interface TemplateContext {
|
||||
paths: HostPaths;
|
||||
preambleTier?: number; // 1-4, controls which preamble sections are included
|
||||
model?: Model; // model family for behavioral overlay. Omitted/undefined → no overlay.
|
||||
interactive?: boolean; // true → emit plan-mode handshake in preamble. Generator-only, not written to SKILL.md.
|
||||
}
|
||||
|
||||
/** Resolver function signature. args is populated for parameterized placeholders like {{INVOKE_SKILL:name}}. */
|
||||
|
||||
Reference in New Issue
Block a user