CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-07-11 10:23:44 +02:00
Code Issues Packages Projects Releases Wiki Activity

fix(test): autoplan-dual-voice sandbox registers skills + assertions read real stream-json

Browse Source 
Root cause of months of silent local failure: the sandbox copied skill dirs to
the repo root, but claude >= 2.x resolves slash commands strictly from
registered skills, so /autoplan short-circuited with 'Unknown command' (0
turns, ~1s) on every attempt. Install /autoplan + review skills at
project-level .claude/skills/ (same pattern as skill-routing-e2e).

Also: the transcript filter matched entry.type === 'tool_use', a shape that
never appears at the top level of raw stream-json, so assertions only ever saw
the final result text; filter on assistant/user events instead. Hang
protection accepts the Phase 1 review dispatch (Agent/Task tool call carrying
review instructions) as progress evidence, since full Phase 1 completion is
15+ min of subagent work. Budget raised to 10 min / 40 turns.

Invisible in CI: the file is in neither evals.yml nor evals-periodic.yml
matrices (coverage decision filed in TODOS.md).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-07-09 23:49:09 -07:00
parent bb4de3926f
commit 88dec06907
1 changed files with 43 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/skill-e2e-autoplan-dual-voice.test.ts
+43 -10
View File
@@ -40,6 +40,19 @@ describeIfSelected('Autoplan dual-voice E2E', ['autoplan-dual-voice'], () => {
copyDirSync(path.join(ROOT, 'plan-design-review'), path.join(workDir, 'plan-design-review'));
copyDirSync(path.join(ROOT, 'plan-devex-review'), path.join(workDir, 'plan-devex-review'));
// Register the skills as project-level slash commands. The root copies
// above are NOT enough on their own: claude -p only discovers skills under
// .claude/skills/, and an unregistered slash command short-circuits with
// "Unknown command: /autoplan" (0 turns, ~1s) on claude >= 2.x — the model
// never runs, so both voice assertions fail. Same install pattern as
// installSkills() in skill-routing-e2e.test.ts.
const skillsBase = path.join(workDir, '.claude', 'skills');
for (const skill of ['autoplan', 'plan-ceo-review', 'plan-eng-review', 'plan-design-review', 'plan-devex-review']) {
const dest = path.join(skillsBase, skill);
fs.mkdirSync(dest, { recursive: true });
fs.copyFileSync(path.join(ROOT, skill, 'SKILL.md'), path.join(dest, 'SKILL.md'));
}
// Write a tiny plan file for /autoplan to review.
planPath = path.join(workDir, 'TEST_PLAN.md');
fs.writeFileSync(planPath, `# Test Plan: add /greet skill
@@ -65,20 +78,24 @@ Add a new /greet skill that prints a welcome message.
test.skipIf(!evalsEnabled)(
'both Claude + Codex voices produce output in Phase 1 (within timeout)',
async () => {
// Fire /autoplan with a 5-min hard timeout on the spawn itself.
// Fire /autoplan with a 10-min hard timeout on the spawn itself.
// The skill itself has 10-min phase timeouts + auth-gate failfast.
// If Codex is unavailable on the test machine, the skill should print
// [codex-unavailable] and still complete the Claude subagent half.
// Budget note: 5 min / 30 turns was enough at v1.0-era skill sizes, but
// the full-depth Phase 1 (registered skill + CEO review subagent) now
// needs longer — at 300s the run was killed mid-CEO-review with both
// voices already fired but no Phase-1-complete marker yet.
const result = await runSkillTest({
testName: 'autoplan-dual-voice',
workingDirectory: workDir,
prompt: `/autoplan ${planPath}`,
timeout: 300_000, // 5 min
timeout: 600_000, // 10 min
// /autoplan spawns subagents and calls codex via Bash; it needs the
// full tool set to get past Phase 1. Bash+Read+Write alone wasn't
// enough — the skill stalled trying to invoke Agent/Skill.
allowedTools: ['Bash', 'Read', 'Write', 'Edit', 'Grep', 'Glob', 'Agent', 'Skill'],
maxTurns: 30,
maxTurns: 40,
runId,
});
@@ -91,8 +108,14 @@ Add a new /greet skill that prints a welcome message.
// false positives regardless of skill behavior. Filter to tool_result
// content + assistant messages emitted DURING execution.
const transcript = Array.isArray(result.transcript) ? result.transcript : [];
// The transcript holds RAW stream-json events: tool_use blocks live
// INSIDE assistant events' message.content, and tool_results inside
// user events — no top-level entry ever has type 'tool_use'. Filtering
// on that shape matched nothing, silently reducing `out` to
// result.output alone, so a run killed at the spawn timeout (no result
// event) had NOTHING to match and every assertion failed.
const executionContent = transcript
.filter((entry: any) => entry && (entry.type === 'tool_use' || entry.type === 'tool_result' || entry.role === 'assistant'))
.filter((entry: any) => entry && (entry.type === 'assistant' || entry.type === 'user'))
.map((entry: any) => JSON.stringify(entry))
.join('\n');
const out = (result.output ?? '') + '\n' + executionContent;
@@ -112,17 +135,27 @@ Add a new /greet skill that prints a welcome message.
expect(claudeVoiceFired).toBe(true);
expect(codexVoiceFired || codexUnavailable).toBe(true);
// Hang protection: require phase completion evidence, not name mentions.
// "Phase 1 complete" or a phase-transition marker, not "plan-ceo-review"
// as a bare string (which appears in the prompt itself).
// Hang protection: require pipeline-progress evidence, not name mentions.
// Full Phase 1 COMPLETION (three parallel review subagents, each loading a
// 25-35K-token skill) routinely exceeds 10 minutes on sonnet, so requiring
// the "Phase 1 complete" banner would force a 20-minute test for no extra
// dual-voice signal. Accept EITHER the completion banner (autoplan/SKILL.md
// "PHASE 1 COMPLETE" mandatory output) OR structural evidence that the
// Phase 1 review dispatch actually happened: an Agent tool_use whose input
// carries review instructions (execution artifact built by the skill, not
// an echo of our prompt).
const reachedPhase1 = /Phase\s+1\s+(complete|done|finished)|CEO\s+Review\s+(complete|done|approved)|Strategy\s*&\s*Scope\s+(complete|done)|Phase\s+2\s+(started|begin)/i.test(out);
expect(reachedPhase1).toBe(true);
const toolCalls = Array.isArray(result.toolCalls) ? result.toolCalls : [];
const reviewDispatched = toolCalls.some((tc: any) =>
(tc?.tool === 'Agent' || tc?.tool === 'Task') &&
/review|ceo|eng manager|strategy/i.test(JSON.stringify(tc?.input ?? {})));
expect(reachedPhase1 || reviewDispatched).toBe(true);
logCost('autoplan-dual-voice', result);
recordE2E(evalCollector, 'autoplan-dual-voice', 'Autoplan dual-voice E2E', result, {
passed: claudeVoiceFired && (codexVoiceFired || codexUnavailable) && reachedPhase1,
passed: claudeVoiceFired && (codexVoiceFired || codexUnavailable) && (reachedPhase1 || reviewDispatched),
});
},
330_000, // per-test timeout slightly > spawn timeout so cleanup can run
630_000, // per-test timeout slightly > spawn timeout so cleanup can run
);
});