mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
Garry Tan
473c8717d8
Rename test/helpers/plan-mode-handshake-helpers.ts to test/helpers/plan-mode-helpers.ts. Keep the write-guard helper that asserts no Write/Edit tool call before the first AskUserQuestion (this is what catches silent-bypass regressions the textual smoke can't see). Rename the API: runPlanModeHandshakeTest to runPlanModeSkillTest, assertHandshakeShape to assertNotHandshakeShape. Extend the capture struct with exitPlanModeBeforeAsk. Rewrite the four per-skill E2E tests (plan-ceo, plan-eng, plan-design, plan-devex) as smoke tests that assert the skill's Step 0 question fires first, not an A/C handshake. Each test picks a cheap first answer (HOLD, TRIAGE, numeric score) so the run terminates quickly. Keep test/skill-e2e-plan-mode-no-op.test.ts as the outside-plan-mode non-interference regression, per codex outside-voice review: deleting it would lose coverage for "the hoisted section stays quiet when plan mode is absent." Replace the gen-skill-docs.test.ts handshake describe block (lines 2778+) with a plan-mode-info describe block that: - scans every generated SKILL.md under the repo root + every host subdir (.agents, .openclaw, .opencode, .factory, .hermes, .kiro, .cursor, .slate) and asserts "## Plan Mode Handshake" is absent - asserts "## Skill Invocation During Plan Mode" lands in the first 15KB of each of the four review skills' generated SKILL.md Both assertions run on every bun test. A PR that re-introduces the handshake resolver fails CI immediately. Update test/e2e-harness-audit.test.ts to reference the renamed runPlanModeSkillTest. Update test/helpers/touchfiles.ts entries to point at the new resolver owner (generate-completion-status.ts) and the renamed helper, and align per-skill touchfile keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
39 lines
1.5 KiB
TypeScript
39 lines
1.5 KiB
TypeScript
/**
|
||
* plan-ceo-review plan-mode smoke test (gate tier, paid).
|
||
*
|
||
* Asserts: when /plan-ceo-review is invoked with the plan-mode distinctive
|
||
* phrase in the system reminder, the skill goes STRAIGHT to its Step 0
|
||
* scope-mode AskUserQuestion. Specifically:
|
||
* 1. First AskUserQuestion is NOT the old vestigial handshake
|
||
* (A=exit-and-rerun / C=cancel).
|
||
* 2. No Write or Edit tool fires before the first AskUserQuestion
|
||
* (catches silent plan-file-write bypass).
|
||
* 3. ExitPlanMode does not fire before the first AskUserQuestion.
|
||
*
|
||
* Cost: ~$0.50–$1.00 per run. Gated: EVALS=1 EVALS_TIER=gate.
|
||
*/
|
||
|
||
import { describe, test, expect } from 'bun:test';
|
||
import {
|
||
runPlanModeSkillTest,
|
||
assertNotHandshakeShape,
|
||
} from './helpers/plan-mode-helpers';
|
||
|
||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||
const describeE2E = shouldRun ? describe : describe.skip;
|
||
|
||
describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
|
||
test('goes straight to scope-mode question, no handshake, no silent writes', async () => {
|
||
const result = await runPlanModeSkillTest({
|
||
skillName: 'plan-ceo-review',
|
||
// Step 0 asks for review mode; HOLD is the cheapest, most-neutral answer.
|
||
firstAnswerSubstring: 'HOLD',
|
||
});
|
||
|
||
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
||
assertNotHandshakeShape(result.askUserQuestions[0]!);
|
||
expect(result.writeOrEditBeforeAsk).toBe(false);
|
||
expect(result.exitPlanModeBeforeAsk).toBe(false);
|
||
}, 120_000);
|
||
});
|