mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
Garry Tan
6f67406e01
The test shipped on main in v0.18.4.0 used wrong option names and
wrong result fields throughout. It could not have passed in any
environment:
Broken API calls:
- `workdir` → should be `workingDirectory`
The fixture setup (git init, copy autoplan + plan-*-review dirs,
write TEST_PLAN.md) was completely ignored. claude -p spawned with
undefined cwd instead of the tmp workdir.
- `timeoutMs: 300_000` → should be `timeout: 300_000`
Fell back to default 120s. Explains the observed ~170s failure
(test harness overhead + retry startup).
- `name: 'autoplan-dual-voice'` → should be `testName: 'autoplan-dual-voice'`
No per-test run directory was created.
- `evalCollector` → not a recognized `runSkillTest` option at all.
Broken result access:
- `result.stdout + result.stderr` → SkillTestResult has neither
field. `out` was literally "undefinedundefined" every time.
- Every regex match fired false. All 3 assertions (claudeVoiceFired,
codex-or-unavailable, reachedPhase1) failed on every attempt.
- `logCost(result)` → signature is `logCost(label, result)`.
- `recordE2E('autoplan-dual-voice', result)` → signature is
`recordE2E(evalCollector, name, suite, result, extra)`.
Fixes:
- Renamed all 4 broken options in the runSkillTest call.
- Changed assertion source to `result.output` plus JSON-serialized
`result.transcript` (broader net for voice fingerprints in tool
inputs/outputs).
- Widened regex alternatives: codex voice now matches "CODEX SAYS"
and "codex-plan-review"; Claude voice now matches subagent_type;
unavailable matches CODEX_NOT_AVAILABLE.
- Added Agent + Skill + Edit + Grep + Glob to allowedTools. Without
Agent, /autoplan can't spawn subagents and never reaches Phase 1.
- Raised maxTurns 15 → 30 (autoplan is a long multi-phase skill).
- Fixed logCost + recordE2E signatures, passing `passed:` flag into
recordE2E per the neighboring context-save pattern.