v1.12.1.0 fix: remove vestigial plan-mode handshake (#1185)

* refactor: remove vestigial plan-mode handshake resolver

Delete scripts/resolvers/preamble/generate-plan-mode-handshake.ts and
its four question-registry entries. Split the authoritative
"Plan Mode Safe Operations" and "Skill Invocation During Plan Mode"
sections out of generate-completion-status.ts into a sibling
generatePlanModeInfo() export in the same module, wired at preamble
position 1 where the handshake used to live. Same text, new position.

The vestigial handshake told interactive review skills to emit an
A=exit-and-rerun / C=cancel AskUserQuestion before running their
interactive STOP-Ask workflow. That contradicted the authoritative
rule at the tail of completion-status.ts saying AskUserQuestion
satisfies plan mode's end-of-turn requirement. Skills now run
directly when invoked in plan mode, with each finding gated by
AskUserQuestion just like outside plan mode.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* test: rename plan-mode-handshake-helpers to plan-mode-helpers, strengthen smokes

Rename test/helpers/plan-mode-handshake-helpers.ts to
test/helpers/plan-mode-helpers.ts. Keep the write-guard helper that
asserts no Write/Edit tool call before the first AskUserQuestion
(this is what catches silent-bypass regressions the textual smoke
can't see). Rename the API: runPlanModeHandshakeTest to
runPlanModeSkillTest, assertHandshakeShape to assertNotHandshakeShape.
Extend the capture struct with exitPlanModeBeforeAsk.

Rewrite the four per-skill E2E tests (plan-ceo, plan-eng, plan-design,
plan-devex) as smoke tests that assert the skill's Step 0 question
fires first, not an A/C handshake. Each test picks a cheap first
answer (HOLD, TRIAGE, numeric score) so the run terminates quickly.

Keep test/skill-e2e-plan-mode-no-op.test.ts as the outside-plan-mode
non-interference regression, per codex outside-voice review: deleting
it would lose coverage for "the hoisted section stays quiet when plan
mode is absent."

Replace the gen-skill-docs.test.ts handshake describe block (lines
2778+) with a plan-mode-info describe block that:
- scans every generated SKILL.md under the repo root + every host
  subdir (.agents, .openclaw, .opencode, .factory, .hermes, .kiro,
  .cursor, .slate) and asserts "## Plan Mode Handshake" is absent
- asserts "## Skill Invocation During Plan Mode" lands in the first
  15KB of each of the four review skills' generated SKILL.md

Both assertions run on every bun test. A PR that re-introduces the
handshake resolver fails CI immediately.

Update test/e2e-harness-audit.test.ts to reference the renamed
runPlanModeSkillTest. Update test/helpers/touchfiles.ts entries to
point at the new resolver owner (generate-completion-status.ts) and
the renamed helper, and align per-skill touchfile keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: regenerate SKILL.md across all hosts + refresh golden fixtures

Run bun run gen:skill-docs for every host to flush the vestigial
"## Plan Mode Handshake" section from every generated SKILL.md and
emit the hoisted "## Skill Invocation During Plan Mode" section at
preamble position 1 instead. Refresh the three golden-fixture
snapshots (claude, codex, factory) to match the new position.

No behavior change beyond the resolver swap in the prior commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: bump version and changelog (v1.12.1.0)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-24 02:11:24 -07:00
committed by GitHub
parent 2014557e7f
commit aeea57f96a
58 changed files with 917 additions and 1391 deletions
+18 -20
View File
@@ -1,40 +1,38 @@
/**
* plan-ceo-review plan-mode handshake E2E (gate tier, paid).
* plan-ceo-review plan-mode smoke test (gate tier, paid).
*
* Asserts: when /plan-ceo-review is invoked with the plan-mode distinctive
* phrase in the system reminder, the skill fires AskUserQuestion FIRST
* (before any Write or Edit), the question has exactly 2 options (A exit,
* C cancel), picking "Exit" leads to an orderly exit with no plan file
* written.
* phrase in the system reminder, the skill goes STRAIGHT to its Step 0
* scope-mode AskUserQuestion. Specifically:
* 1. First AskUserQuestion is NOT the old vestigial handshake
* (A=exit-and-rerun / C=cancel).
* 2. No Write or Edit tool fires before the first AskUserQuestion
* (catches silent plan-file-write bypass).
* 3. ExitPlanMode does not fire before the first AskUserQuestion.
*
* Cost: ~$0.50$1.00 per run. Gated: EVALS=1 EVALS_TIER=gate.
* Depends on: scripts/resolvers/preamble/generate-plan-mode-handshake.ts,
* test/helpers/agent-sdk-runner.ts (canUseTool extension).
*/
import { describe, test, expect } from 'bun:test';
import {
runPlanModeHandshakeTest,
assertHandshakeShape,
} from './helpers/plan-mode-handshake-helpers';
runPlanModeSkillTest,
assertNotHandshakeShape,
} from './helpers/plan-mode-helpers';
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
const describeE2E = shouldRun ? describe : describe.skip;
describeE2E('plan-ceo-review plan-mode handshake (gate)', () => {
test('handshake fires before any Write/Edit when plan mode is detected', async () => {
const result = await runPlanModeHandshakeTest({
describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
test('goes straight to scope-mode question, no handshake, no silent writes', async () => {
const result = await runPlanModeSkillTest({
skillName: 'plan-ceo-review',
answerLabel: 'Exit',
// Step 0 asks for review mode; HOLD is the cheapest, most-neutral answer.
firstAnswerSubstring: 'HOLD',
});
// Handshake must have fired at least once.
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
// Critically: no Write or Edit fired before the first AskUserQuestion.
// This is the bug v1.10.2.0 fixes — plan mode used to allow silent
// plan-file writes without any interactive gate.
assertNotHandshakeShape(result.askUserQuestions[0]!);
expect(result.writeOrEditBeforeAsk).toBe(false);
// Handshake shape: 2 options (Exit/Cancel), Option B dropped per D8.
assertHandshakeShape(result.askUserQuestions[0]!);
expect(result.exitPlanModeBeforeAsk).toBe(false);
}, 120_000);
});