CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-05-07 14:06:42 +02:00
Code Issues Packages Projects Releases Wiki Activity

test(harness): fix detection order + whitespace-tolerant pattern matching

Browse Source 
Two bugs surfaced when validating the v1.21 fix end-to-end:

1. PlanSkillObservation outcome detection ran 'asked' (any numbered
   options list) BEFORE 'plan_ready'. Plan-mode's "Ready to execute?"
   confirmation IS a numbered options list (1=auto, 2=manual, ...), so
   any skill that successfully reached the native confirmation got
   misclassified as 'asked'. Reorder: 'auto_decided' (most specific,
   requires AUTO_DECIDE annotation) > 'plan_ready' (next, requires the
   "ready to execute" stem) > 'asked' (any remaining numbered list).

2. isPlanReadyVisible and isAutoDecidedVisible regexes only matched
   spaced forms ("ready to execute", "(your preference)"). stripAnsi
   removes cursor-positioning escapes (`\x1b[40C`) entirely instead of
   replacing them with spaces, so the same text can render as
   "readytoexecute" or "(yourpreference)". Both detectors now test the
   spaced form first, fall through to a whitespace-collapsed comparison.
   Inline unit smoke confirms both forms match.

Updates to the 5 strict 'asked' regression test cases (plan-ceo,
plan-eng, plan-devex, autoplan, office-hours): with the detection order
corrected, the model's plan-file fallback flow legitimately lands at
'plan_ready' instead of 'asked'. Pass envelope expanded to ['asked',
'plan_ready'] (matching plan-design-review's existing pattern). Failure
signals tightened to include 'auto_decided' (catches AUTO_DECIDE without
opt-in) plus the standard silent_write/exited/timeout. plan-design was
already on this contract from v1.21's first commit, no change needed.

The expanded envelope is correct: under --disallowedTools AskUserQuestion
the Tool resolution preamble routes the question through plan-mode's
native "Ready to execute?" surface — the user still sees the decision,
just via the plan-file flow rather than a numbered prompt.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-30 22:27:33 -07:00
parent a7cfbeba4c
commit 78e4b770fa
6 changed files with 103 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-e2e-plan-eng-plan-mode.test.ts
+10 -5
View File
@@ -31,9 +31,9 @@ describeE2E('plan-eng-review plan-mode smoke (gate)', () => {
}, 360_000);
// v1.21+ regression: see skill-e2e-plan-ceo-plan-mode.test.ts for the
// contract. plan-eng-review's Step 0 always issues a scope-challenge
// AskUserQuestion (and per-section STOPs after that), so 'asked' is the
// only pass when AskUserQuestion is --disallowedTools.
// contract. Pass envelope is ['asked', 'plan_ready']; failure signals
// are 'auto_decided' (AUTO_DECIDE without opt-in) plus the standard
// silent_write/exited/timeout.
test('AskUserQuestion surfaces when --disallowedTools AskUserQuestion is set', async () => {
const obs = await runPlanSkillObservation({
skillName: 'plan-eng-review',
@@ -42,7 +42,12 @@ describeE2E('plan-eng-review plan-mode smoke (gate)', () => {
timeoutMs: 300_000,
});
if (obs.outcome !== 'asked') {
if (
obs.outcome === 'auto_decided' ||
obs.outcome === 'silent_write' ||
obs.outcome === 'exited' ||
obs.outcome === 'timeout'
) {
throw new Error(
`plan-eng-review AskUserQuestion-blocked regression: outcome=${obs.outcome}\n` +
`summary: ${obs.summary}\n` +
@@ -50,6 +55,6 @@ describeE2E('plan-eng-review plan-mode smoke (gate)', () => {
`--- evidence (last 2KB visible) ---\n${obs.evidence}`,
);
}
expect(obs.outcome).toEqual('asked');
expect(['asked', 'plan_ready']).toContain(obs.outcome);
}, 360_000);
});