mirror of
https://github.com/garrytan/gstack.git
synced 2026-06-10 20:07:49 +02:00
test: plan-mode handshake E2E coverage and unit assertions
Adds 6 E2E test files and 8 new unit assertions to verify the plan-mode handshake works end-to-end and stays correct under regeneration. E2E tests (gate-tier, paid, EVALS=1 EVALS_TIER=gate): - test/skill-e2e-plan-ceo-plan-mode.test.ts — handshake fires before any Write/Edit when plan-mode distinctive phrase is present; 2-option shape (Exit/Cancel); option A routes to ExitPlanMode cleanly - test/skill-e2e-plan-eng-plan-mode.test.ts — same contract for plan-eng - test/skill-e2e-plan-design-plan-mode.test.ts — same contract for plan-design; exercises C-cancel branch instead of A-exit - test/skill-e2e-plan-devex-plan-mode.test.ts — same contract for plan-devex - test/skill-e2e-plan-mode-no-op.test.ts — negative regression: handshake must NOT fire when distinctive phrase is absent; skill proceeds normally through Step 0 (REGRESSION RULE guardrail against breaking existing interactive-review sessions) - test/e2e-harness-audit.test.ts — free unit test asserting every `interactive: true` skill has at least one canUseTool-using test file (prevents future drift where a skill opts in without coverage) Shared helper test/helpers/plan-mode-handshake-helpers.ts centralizes the canUseTool interceptor + distinctive-phrase injection so the 4 sibling E2E tests are thin wiring (~20 LOC each) and can't drift out of sync. Unit assertions added to test/gen-skill-docs.test.ts: - handshake section present in all 4 Claude-generated SKILL.md files - handshake section absent from non-interactive Claude skills (ship, review, qa, office-hours, codex, retro, cso) - handshake section absent from non-Claude host outputs (.agents, etc.) - 0C-bis STOP block present in plan-ceo-review/SKILL.md at correct position (between the "Present these approach options" line and "### 0D-prelude" header) - handshake resolver wired BEFORE generateUpgradeCheck in preamble composition order 6 new gate-tier entries added to test/helpers/touchfiles.ts so any change to the handshake resolver, preamble composition, skill templates, question registry, one-way-door classifier, or agent-sdk-runner fires the relevant E2E tests. test/touchfiles.test.ts updated for the new selection count (plan-ceo-review/** now triggers 15 tests, up from 8). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* Plan-mode handshake negative regression (gate tier, paid).
|
||||
*
|
||||
* Asserts: when /plan-ceo-review is invoked WITHOUT the plan-mode distinctive
|
||||
* phrase in the system reminder, the handshake does NOT fire. The skill
|
||||
* should proceed to its normal Step 0 flow. This is the REGRESSION RULE
|
||||
* guardrail — the handshake must be a no-op outside plan mode or it breaks
|
||||
* every existing interactive-review session.
|
||||
*
|
||||
* Cost: ~$0.50 per run. Gated: EVALS=1 EVALS_TIER=gate.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
runPlanModeHandshakeTest,
|
||||
PLAN_MODE_REMINDER,
|
||||
} from './helpers/plan-mode-handshake-helpers';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-mode handshake no-op outside plan mode (gate regression)', () => {
|
||||
test('handshake does NOT fire when distinctive phrase is absent', async () => {
|
||||
const result = await runPlanModeHandshakeTest({
|
||||
skillName: 'plan-ceo-review',
|
||||
answerLabel: 'Exit', // ignored — handshake should never fire
|
||||
omitPlanModeReminder: true,
|
||||
maxTurns: 3, // enough to see Step 0 start, but bounded
|
||||
});
|
||||
|
||||
// The handshake AskUserQuestion should NOT have fired during Step 0 entry.
|
||||
// Other AskUserQuestions may fire later in the skill (e.g., Step 0C-bis),
|
||||
// but they will NOT have the handshake's question text.
|
||||
for (const aq of result.askUserQuestions) {
|
||||
const questions = aq.input.questions as Array<{ question: string }>;
|
||||
for (const q of questions) {
|
||||
// The handshake's question mentions the distinctive phrase in its
|
||||
// prose; a non-handshake AskUserQuestion won't.
|
||||
expect(q.question).not.toContain(PLAN_MODE_REMINDER);
|
||||
}
|
||||
}
|
||||
}, 120_000);
|
||||
});
|
||||
Reference in New Issue
Block a user