mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-13 16:38:56 +02:00
feat(plan-eng/design-review): auto-select B in plan mode at the scope gate
In plan mode the scope gate's "What should I review? A/B/C" question is pure
friction: there is no branch diff and the target is the plan being drafted.
Both gates gain an ordered exceptions block, checked BEFORE asking:
1. Plan mode → auto-select B: review the active plan (in context or pasted),
announce it in one line ("Scope gate: plan mode — auto-selected B
(reviewing <target>)") so the user can interrupt; an explicitly different
user-named target still wins; no plan drafted yet → ask as normal.
2. User-named target (outside plan mode): explicit-only — a path, a pasted
doc, or the literal words "branch diff". A passing mention is not naming;
when in doubt, ask.
Outside plan mode with no explicitly-named target, nothing changes. Plan-mode
is checked FIRST because the PTY harness seeds drafts as pasted user messages
(claude-pty-runner.ts:1600) — ordering makes the seeded smokes deterministic.
Pinning: seeded plan-mode smokes assert no gate render + announcement rendered
(eng test 2; new design seeded test); plan-mode-no-op extends to eng/design
(bypass must not misfire outside plan mode; first question must be the gate)
plus a named-target case proving the pasted target is consumed; a drift-guard
asserts the two hand-duplicated exceptions blocks stay identical modulo the
two variant slots and carry the announcement string the detectors pin.
Skeleton ceilings ratcheted with comments (eng 68k, design 89k; eng union
ratio 1.08→1.09) — measured 67,006 B / 88,226 B after regen.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
44cd3037f0
commit
2009f89283
@@ -1,48 +1,138 @@
|
||||
/**
|
||||
* Plan-mode-info no-op regression (gate tier, paid, real-PTY).
|
||||
*
|
||||
* Asserts: when /plan-ceo-review is invoked OUTSIDE plan mode (no
|
||||
* Asserts: when a plan-review skill is invoked OUTSIDE plan mode (no
|
||||
* --permission-mode plan flag, no plan-mode reminder injected), the skill
|
||||
* still reaches a terminal outcome ('asked' or 'plan_ready'). This is the
|
||||
* negative coverage to the per-skill plan-mode smokes — if the
|
||||
* plan-mode-info preamble section ever starts misfiring for non-plan-mode
|
||||
* sessions (e.g., gating questions on a phrase that isn't there), this
|
||||
* test catches it.
|
||||
* negative coverage to the per-skill plan-mode smokes — if plan-mode-keyed
|
||||
* behavior ever starts misfiring for non-plan-mode sessions (e.g., gating
|
||||
* questions on a phrase that isn't there, or the plan-eng/plan-design
|
||||
* scope-gate auto-select-B bypass firing without plan mode), this test
|
||||
* catches it.
|
||||
*
|
||||
* Why this matters: outside plan mode, claude doesn't render a native
|
||||
* confirmation UI. The skill must drive its own AskUserQuestion. Same
|
||||
* runner, same outcome contract — just `inPlanMode: false`.
|
||||
*
|
||||
* Coverage grew with the scope-gate bypass (plan-mode auto-select B):
|
||||
* - plan-ceo-review: original preamble-misfire regression.
|
||||
* - plan-eng-review / plan-design-review: the bypass must NOT fire outside
|
||||
* plan mode (scopeGateAutoSelectObserved stays false), and when the run
|
||||
* ends in 'asked', the question that fired must be the scope gate itself
|
||||
* (outside plan mode with no named target, the gate is the FIRST
|
||||
* question by contract).
|
||||
* - named-target case: a pasted draft (initialPlanContent) IS an
|
||||
* explicitly-named target, so the gate must NOT ask — and the review
|
||||
* must actually consume the pasted content.
|
||||
*
|
||||
* Cost note: 4 sequential PTY runs (~3-5 min each) in the gate lane, up
|
||||
* from 1 pre-bypass. Selected only when plan-ceo/eng/design or the runner
|
||||
* change (see 'plan-mode-no-op' in touchfiles.ts).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { runPlanSkillObservation } from './helpers/claude-pty-runner';
|
||||
|
||||
const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
const PLAN_MODE_REMINDER =
|
||||
'Plan mode is active. The user indicated that they do not want you to execute yet';
|
||||
|
||||
// Distinctive token proves the pasted target was consumed by the review —
|
||||
// not just that no question fired. Nonsense-unique so it can't appear by
|
||||
// coincidence in skill output.
|
||||
const SEED_TOKEN = 'ZephyrLedgerWidget';
|
||||
const NAMED_TARGET_SEED = `
|
||||
# Plan: ${SEED_TOKEN} settings panel
|
||||
|
||||
## Scope
|
||||
Add a ${SEED_TOKEN} settings panel with a single toggle that enables
|
||||
weekly export emails. One new component, one route, one test file.
|
||||
|
||||
## Files
|
||||
- src/components/${SEED_TOKEN}.tsx (new)
|
||||
- src/routes/settings.tsx (add panel)
|
||||
- test/${SEED_TOKEN}.test.tsx (new)
|
||||
`;
|
||||
|
||||
describeE2E('plan-mode-info no-op outside plan mode (gate regression)', () => {
|
||||
test('skill reaches a terminal outcome outside plan mode', async () => {
|
||||
for (const skillName of ['plan-ceo-review', 'plan-eng-review', 'plan-design-review'] as const) {
|
||||
test(`${skillName} reaches a terminal outcome outside plan mode`, async () => {
|
||||
const obs = await runPlanSkillObservation({
|
||||
skillName,
|
||||
inPlanMode: false,
|
||||
timeoutMs: 300_000,
|
||||
});
|
||||
|
||||
if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') {
|
||||
throw new Error(
|
||||
`plan-mode no-op regression FAILED (${skillName}): outcome=${obs.outcome}\n` +
|
||||
`summary: ${obs.summary}\n` +
|
||||
`elapsed: ${obs.elapsedMs}ms\n` +
|
||||
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
|
||||
);
|
||||
}
|
||||
expect(['asked', 'plan_ready']).toContain(obs.outcome);
|
||||
|
||||
// Negative regression: the rendered output must NOT echo the plan-mode
|
||||
// distinctive reminder phrase. If it does, the plan-mode preamble
|
||||
// section is leaking outside plan mode.
|
||||
expect(obs.evidence).not.toContain(PLAN_MODE_REMINDER);
|
||||
|
||||
if (skillName !== 'plan-ceo-review') {
|
||||
// Scope-gate bypass must not misfire: no auto-select announcement
|
||||
// outside plan mode.
|
||||
expect(obs.scopeGateAutoSelectObserved ?? false).toBe(false);
|
||||
// And when a question fired, it must have been the scope gate —
|
||||
// outside plan mode with no named target, the gate is the FIRST
|
||||
// question by contract (hard STOP before any tool call).
|
||||
if (obs.outcome === 'asked') {
|
||||
expect(obs.scopeGateQuestionObserved ?? false).toBe(true);
|
||||
}
|
||||
}
|
||||
}, 360_000);
|
||||
}
|
||||
|
||||
// Named-target exception (outside plan mode): a pasted draft IS an
|
||||
// explicitly-named target, so the scope gate must NOT ask — and the
|
||||
// review must consume the pasted content (seed token visible in the
|
||||
// review output), proving the target was used rather than the question
|
||||
// merely skipped. Also the over-trigger guard for the tightened
|
||||
// "explicit-only" exception wording.
|
||||
test('plan-eng-review skips the scope gate for an explicitly-pasted target', async () => {
|
||||
const obs = await runPlanSkillObservation({
|
||||
skillName: 'plan-ceo-review',
|
||||
skillName: 'plan-eng-review',
|
||||
inPlanMode: false,
|
||||
initialPlanContent: NAMED_TARGET_SEED,
|
||||
timeoutMs: 300_000,
|
||||
});
|
||||
|
||||
if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') {
|
||||
throw new Error(
|
||||
`plan-mode no-op regression FAILED: outcome=${obs.outcome}\n` +
|
||||
`named-target no-op FAILED: outcome=${obs.outcome}\n` +
|
||||
`summary: ${obs.summary}\n` +
|
||||
`elapsed: ${obs.elapsedMs}ms\n` +
|
||||
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
|
||||
);
|
||||
}
|
||||
expect(['asked', 'plan_ready']).toContain(obs.outcome);
|
||||
|
||||
// Negative regression: the rendered output must NOT echo the plan-mode
|
||||
// distinctive reminder phrase. If it does, the plan-mode preamble
|
||||
// section is leaking outside plan mode.
|
||||
const PLAN_MODE_REMINDER =
|
||||
'Plan mode is active. The user indicated that they do not want you to execute yet';
|
||||
expect(obs.evidence).not.toContain(PLAN_MODE_REMINDER);
|
||||
|
||||
// The pasted doc is the named target: gate question must not render,
|
||||
// no plan-mode announcement either (we are NOT in plan mode).
|
||||
expect(obs.scopeGateQuestionObserved ?? false).toBe(false);
|
||||
expect(obs.scopeGateAutoSelectObserved ?? false).toBe(false);
|
||||
|
||||
// Target consumption: the review must reference the seeded feature.
|
||||
// evidence is the lossy 2KB tail, so also accept the token appearing in
|
||||
// the plan file the review wrote — the token names the component, route,
|
||||
// and test file every finding has to discuss, so absence from BOTH means
|
||||
// the pasted target was not actually reviewed.
|
||||
const inEvidence = obs.evidence.includes(SEED_TOKEN);
|
||||
const inPlanFile =
|
||||
!!obs.planFile && existsSync(obs.planFile) && readFileSync(obs.planFile, 'utf-8').includes(SEED_TOKEN);
|
||||
expect(inEvidence || inPlanFile).toBe(true);
|
||||
}, 360_000);
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user