mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-07 22:16:52 +02:00
merge: resolve conflicts with origin/main, bump to v1.22.0.0
Main shipped v1.21.0.0 (browser-skills) and v1.21.1.0 (plan-ceo smoke
tightening + classifyVisible extraction) while this branch was in flight.
Resolves all four merge conflicts:
- VERSION 1.22.0.0 (was claiming v1.21.0.0; queue advanced past v1.21.1.0;
MINOR scale on top per scale-aware bumps)
- package.json synced to 1.22.0.0
- claude-pty-runner.ts: integrated 'auto_decided' outcome into main's new
classifyVisible() pure function. Branch order is now silent_write →
auto_decided → plan_ready → asked. Kept BOTH new options on the runner
(`extraArgs` from this branch, `env` from main). isPlanReadyVisible and
isAutoDecidedVisible remain whitespace-tolerant.
- touchfiles.test.ts: count of plan-ceo-review/SKILL.md selections updates
to 21 (main added plan-ceo-finding-count; this branch added
autoplan-auto-mode + auto-decide-preserved).
- CHANGELOG.md: dropped this branch's stale v1.21.0.0 draft, kept main's
v1.21.1.0 entry intact, wrote v1.22.0.0 entry on top covering the
AskUserQuestion-blocked regression fix.
plan-ceo-plan-mode test auto-merged: main's stricter baseline ('asked'
only) + this branch's AskUserQuestion-blocked regression case both
present and run sequentially. SKILL.md goldens regenerated via
`bun run gen:skill-docs --host all` and the 3 host-ship goldens copied
into test/fixtures/golden/.
Free test suite: 470/470 pass post-merge.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -1,22 +1,34 @@
|
||||
/**
|
||||
* plan-ceo-review plan-mode smoke (gate, paid, real-PTY).
|
||||
*
|
||||
* Asserts: when /plan-ceo-review is invoked in plan mode, the skill reaches
|
||||
* a terminal outcome that is either:
|
||||
* - 'asked' — skill emitted its Step 0 numbered prompt (scope mode
|
||||
* selection, or the routing-injection prompt that runs
|
||||
* before Step 0)
|
||||
* - 'plan_ready' — skill ran end-to-end and surfaced claude's native
|
||||
* "Ready to execute" confirmation
|
||||
* Asserts: when /plan-ceo-review is invoked in plan mode, the FIRST terminal
|
||||
* outcome is 'asked' — a skill-question numbered list. Permission dialogs
|
||||
* (which also render numbered lists) are filtered out by `runPlanSkillObservation`
|
||||
* via its `isPermissionDialogVisible(visible.slice(-1500))` short-circuit.
|
||||
*
|
||||
* FAIL conditions: silent Write/Edit before any prompt, claude crash,
|
||||
* timeout.
|
||||
* Reaching 'plan_ready' first IS the regression we want to catch: the agent
|
||||
* skipped Step 0 entirely and went straight to ExitPlanMode. The original
|
||||
* failure had the assistant read a diff, write a plan with two issues, and
|
||||
* call ExitPlanMode without ever firing AskUserQuestion — the user had to
|
||||
* manually call out the missing per-issue questions.
|
||||
*
|
||||
* Replaces the SDK-based test that never worked: the SDK's canUseTool
|
||||
* interceptor on AskUserQuestion never fires in plan mode because plan
|
||||
* mode renders its native confirmation as TTY UI, not via the
|
||||
* AskUserQuestion tool. The real PTY harness observes the rendered
|
||||
* terminal output directly.
|
||||
* Why this skill is special: unlike plan-eng-review / plan-design-review /
|
||||
* plan-devex-review (whose smokes accept either 'asked' or 'plan_ready'),
|
||||
* plan-ceo-review's template mandates Step 0A premise challenge (3 baked-in
|
||||
* questions) AND Step 0F mode selection BEFORE any plan write. There is no
|
||||
* legitimate path to plan_ready that does not first emit a skill-question
|
||||
* numbered prompt.
|
||||
*
|
||||
* Env passthrough: passes `QUESTION_TUNING=false` and `EXPLAIN_LEVEL=default`
|
||||
* via the runner's env option. Today these are advisory — `gstack-config`
|
||||
* reads `~/.gstack/config.yaml`, not env vars, so a contributor with
|
||||
* `question_tuning: true` set in their YAML config can still see AUTO_DECIDE
|
||||
* masking. The env passthrough is wired so a future gstack-config change to
|
||||
* honor env overrides will make this test hermetic without further edits.
|
||||
* Tracked as a post-merge follow-up.
|
||||
*
|
||||
* FAIL conditions: 'plan_ready' first, silent Write/Edit before any prompt,
|
||||
* claude crash, timeout.
|
||||
*
|
||||
* See test/helpers/claude-pty-runner.ts for runner internals.
|
||||
*/
|
||||
@@ -28,22 +40,34 @@ const shouldRun = !!process.env.EVALS && process.env.EVALS_TIER === 'gate';
|
||||
const describeE2E = shouldRun ? describe : describe.skip;
|
||||
|
||||
describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
|
||||
test('reaches a terminal outcome (asked or plan_ready) without silent writes', async () => {
|
||||
test('first terminal outcome is asked (Step 0 fires before any plan write)', async () => {
|
||||
const obs = await runPlanSkillObservation({
|
||||
skillName: 'plan-ceo-review',
|
||||
inPlanMode: true,
|
||||
timeoutMs: 300_000,
|
||||
env: { QUESTION_TUNING: 'false', EXPLAIN_LEVEL: 'default' },
|
||||
});
|
||||
|
||||
if (obs.outcome === 'silent_write' || obs.outcome === 'exited' || obs.outcome === 'timeout') {
|
||||
if (obs.outcome !== 'asked') {
|
||||
const diagnosis =
|
||||
obs.outcome === 'plan_ready'
|
||||
? `'plan_ready' first means the agent skipped Step 0 entirely and went straight to ExitPlanMode without asking.`
|
||||
: obs.outcome === 'timeout'
|
||||
? `Timeout means the agent neither asked nor completed within the budget — likely hung mid-question or stuck on a permission dialog.`
|
||||
: obs.outcome === 'silent_write'
|
||||
? `Silent Write/Edit fired to an unsanctioned path before any AskUserQuestion — also a Step 0 skip.`
|
||||
: `Outcome '${obs.outcome}' is unexpected; investigate the evidence below.`;
|
||||
throw new Error(
|
||||
`plan-ceo-review plan-mode smoke FAILED: outcome=${obs.outcome}\n` +
|
||||
`plan-ceo-review smoke FAILED: outcome=${obs.outcome}\n` +
|
||||
`${diagnosis}\n` +
|
||||
`Expected 'asked'. See plan-ceo-review/SKILL.md.tmpl: the Step 0 STOP rules ` +
|
||||
`and the "One issue = one AskUserQuestion call" rule under "CRITICAL RULE — ` +
|
||||
`How to ask questions".\n` +
|
||||
`summary: ${obs.summary}\n` +
|
||||
`elapsed: ${obs.elapsedMs}ms\n` +
|
||||
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
|
||||
);
|
||||
}
|
||||
expect(['asked', 'plan_ready']).toContain(obs.outcome);
|
||||
}, 360_000);
|
||||
|
||||
// v1.21+ regression: Conductor launches Claude Code with
|
||||
|
||||
Reference in New Issue
Block a user