mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 11:45:20 +02:00
Garry Tan
473c8717d8
Rename test/helpers/plan-mode-handshake-helpers.ts to test/helpers/plan-mode-helpers.ts. Keep the write-guard helper that asserts no Write/Edit tool call before the first AskUserQuestion (this is what catches silent-bypass regressions the textual smoke can't see). Rename the API: runPlanModeHandshakeTest to runPlanModeSkillTest, assertHandshakeShape to assertNotHandshakeShape. Extend the capture struct with exitPlanModeBeforeAsk. Rewrite the four per-skill E2E tests (plan-ceo, plan-eng, plan-design, plan-devex) as smoke tests that assert the skill's Step 0 question fires first, not an A/C handshake. Each test picks a cheap first answer (HOLD, TRIAGE, numeric score) so the run terminates quickly. Keep test/skill-e2e-plan-mode-no-op.test.ts as the outside-plan-mode non-interference regression, per codex outside-voice review: deleting it would lose coverage for "the hoisted section stays quiet when plan mode is absent." Replace the gen-skill-docs.test.ts handshake describe block (lines 2778+) with a plan-mode-info describe block that: - scans every generated SKILL.md under the repo root + every host subdir (.agents, .openclaw, .opencode, .factory, .hermes, .kiro, .cursor, .slate) and asserts "## Plan Mode Handshake" is absent - asserts "## Skill Invocation During Plan Mode" lands in the first 15KB of each of the four review skills' generated SKILL.md Both assertions run on every bun test. A PR that re-introduces the handshake resolver fails CI immediately. Update test/e2e-harness-audit.test.ts to reference the renamed runPlanModeSkillTest. Update test/helpers/touchfiles.ts entries to point at the new resolver owner (generate-completion-status.ts) and the renamed helper, and align per-skill touchfile keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
48 lines
1.8 KiB
TypeScript
48 lines
1.8 KiB
TypeScript
/**
|
|
* Plan-mode-info no-op regression (gate tier, paid).
|
|
*
|
|
* Asserts: when /plan-ceo-review is invoked WITHOUT the plan-mode distinctive
|
|
* phrase in the system reminder, the plan-mode-info preamble section is a
|
|
* no-op. The skill should proceed to its normal Step 0 flow with no
|
|
* AskUserQuestion echoing or referencing the plan-mode reminder text.
|
|
*
|
|
* This guardrails the "outside plan mode, this block doesn't interfere"
|
|
* case — a different coverage case from the per-skill in-plan-mode smokes.
|
|
* If the plan-mode-info section ever starts misfiring for non-plan-mode
|
|
* sessions, this test catches it.
|
|
*
|
|
* Cost: ~$0.50 per run. Gated: EVALS=1 EVALS_TIER=gate.
|
|
*/
|
|
|
|
import { describe, test, expect } from 'bun:test';
|
|
import {
|
|
runPlanModeSkillTest,
|
|
PLAN_MODE_REMINDER,
|
|
} from './helpers/plan-mode-helpers';
|
|
|
|
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
|
const describeE2E = shouldRun ? describe : describe.skip;
|
|
|
|
describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
|
|
test('no AskUserQuestion echoes the plan-mode reminder when absent', async () => {
|
|
const result = await runPlanModeSkillTest({
|
|
skillName: 'plan-ceo-review',
|
|
firstAnswerSubstring: 'HOLD',
|
|
omitPlanModeReminder: true,
|
|
maxTurns: 3,
|
|
});
|
|
|
|
// Skill should still hit Step 0 normally outside plan mode.
|
|
expect(result.askUserQuestions.length).toBeGreaterThanOrEqual(1);
|
|
|
|
// No AskUserQuestion should echo the plan-mode distinctive phrase.
|
|
// If one does, the plan-mode-info section is leaking outside plan mode.
|
|
for (const aq of result.askUserQuestions) {
|
|
const questions = aq.input.questions as Array<{ question: string }>;
|
|
for (const q of questions) {
|
|
expect(q.question).not.toContain(PLAN_MODE_REMINDER);
|
|
}
|
|
}
|
|
}, 120_000);
|
|
});
|