CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-11 23:32:36 +02:00
Code Issues Packages Projects Releases Wiki Activity

test(harness): fix detection order + whitespace-tolerant pattern matching

Browse Source 
Two bugs surfaced when validating the v1.21 fix end-to-end:

1. PlanSkillObservation outcome detection ran 'asked' (any numbered
   options list) BEFORE 'plan_ready'. Plan-mode's "Ready to execute?"
   confirmation IS a numbered options list (1=auto, 2=manual, ...), so
   any skill that successfully reached the native confirmation got
   misclassified as 'asked'. Reorder: 'auto_decided' (most specific,
   requires AUTO_DECIDE annotation) > 'plan_ready' (next, requires the
   "ready to execute" stem) > 'asked' (any remaining numbered list).

2. isPlanReadyVisible and isAutoDecidedVisible regexes only matched
   spaced forms ("ready to execute", "(your preference)"). stripAnsi
   removes cursor-positioning escapes (`\x1b[40C`) entirely instead of
   replacing them with spaces, so the same text can render as
   "readytoexecute" or "(yourpreference)". Both detectors now test the
   spaced form first, fall through to a whitespace-collapsed comparison.
   Inline unit smoke confirms both forms match.

Updates to the 5 strict 'asked' regression test cases (plan-ceo,
plan-eng, plan-devex, autoplan, office-hours): with the detection order
corrected, the model's plan-file fallback flow legitimately lands at
'plan_ready' instead of 'asked'. Pass envelope expanded to ['asked',
'plan_ready'] (matching plan-design-review's existing pattern). Failure
signals tightened to include 'auto_decided' (catches AUTO_DECIDE without
opt-in) plus the standard silent_write/exited/timeout. plan-design was
already on this contract from v1.21's first commit, no change needed.

The expanded envelope is correct: under --disallowedTools AskUserQuestion
the Tool resolution preamble routes the question through plan-mode's
native "Ready to execute?" surface — the user still sees the decision,
just via the plan-file flow rather than a numbered prompt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-30 22:27:33 -07:00
parent a7cfbeba4c
commit 78e4b770fa
6 changed files with 103 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-e2e-plan-ceo-plan-mode.test.ts
+22 -6
View File
@@ -50,10 +50,21 @@ describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
// `--disallowedTools AskUserQuestion --permission-mode default` (verified
// via `ps` on the live Conductor claude process). Native AskUserQuestion
// is removed from the model's tool registry; without fallback guidance
// the model can't ask and silently proceeds. plan-ceo-review's Step 0
// always asks a scope-mode question, so 'asked' is the only pass — the
// fix must route through mcp__conductor__AskUserQuestion (when present)
// or the plan-file + ExitPlanMode flow.
// the model can't ask and silently proceeds.
//
// The fix (Tool resolution preamble) accepts two surface paths under
// --disallowedTools:
// - 'asked' — model emits a numbered-option prompt as prose (with
// the same D<N> + Pros/cons format as a real AUQ)
// - 'plan_ready' — model writes the question into the plan file as a
// "## Decisions to confirm" section + ExitPlanMode;
// the native plan-mode "Ready to execute?" surfaces
// it through the TTY confirmation
//
// Both let the user see the decision. Failure signals are
// silent_write/exited/timeout (model never surfaced the question) and
// 'auto_decided' (the AUTO_DECIDE preamble fired without a /plan-tune
// opt-in — caught explicitly).
test('AskUserQuestion surfaces when --disallowedTools AskUserQuestion is set', async () => {
const obs = await runPlanSkillObservation({
skillName: 'plan-ceo-review',
@@ -62,7 +73,12 @@ describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
timeoutMs: 300_000,
});
if (obs.outcome !== 'asked') {
if (
obs.outcome === 'auto_decided' ||
obs.outcome === 'silent_write' ||
obs.outcome === 'exited' ||
obs.outcome === 'timeout'
) {
throw new Error(
`plan-ceo-review AskUserQuestion-blocked regression: outcome=${obs.outcome}\n` +
`summary: ${obs.summary}\n` +
@@ -70,6 +86,6 @@ describeE2E('plan-ceo-review plan-mode smoke (gate)', () => {
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
expect(obs.outcome).toEqual('asked');
expect(['asked', 'plan_ready']).toContain(obs.outcome);
}, 360_000);
});