CalvinBackup/gstack
1
0
Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-08-31 02:10:51 +02:00
Code Issues Packages Projects Releases Wiki Activity

test: rewrite 5 plan-mode E2E tests on the real-PTY harness

Browse Source 
Replaces SDK-based assertions with runPlanSkillObservation contract. Each
test launches real claude --permission-mode plan, invokes the skill, and
asserts the outcome reaches 'asked' or 'plan_ready' within a 300s budget
(no silent Write/Edit, no crash, no timeout).

Affected:
- test/skill-e2e-plan-ceo-plan-mode.test.ts
- test/skill-e2e-plan-eng-plan-mode.test.ts
- test/skill-e2e-plan-design-plan-mode.test.ts
- test/skill-e2e-plan-devex-plan-mode.test.ts
- test/skill-e2e-plan-mode-no-op.test.ts (inPlanMode: false; tests the
  preamble plan-mode-info no-op path)

test/e2e-harness-audit.test.ts — recognize runPlanSkillObservation as a
valid coverage path alongside the legacy canUseTool / runPlanModeSkillTest.

test/helpers/touchfiles.ts — point the 5 plan-mode test selections and
the e2e-harness-audit selection at test/helpers/claude-pty-runner.ts
instead of the deleted plan-mode-helpers.ts.

Proof: bun test EVALS=1 EVALS_TIER=gate on these 5 files runs sequentially
in 790s and passes 5/5. Same tests were 0/5 on origin/main, on v1.0.0.0,
and on this branch with the SDK harness.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
2026-04-25 21:23:54 -07:00
co-authored by Claude Opus 4.7
parent 1b1fd30ec7
commit 38f31e3b1d
7 changed files with 140 additions and 113 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/e2e-harness-audit.test.ts
+11 -6
View File
@@ -1,8 +1,11 @@
/**
* E2E harness audit — every skill with `interactive: true` in its frontmatter
* must have at least one test file that uses `canUseTool` via the extended
* agent-sdk-runner. This prevents future drift where a skill opts into the
* handshake without adding real coverage.
* must have at least one test file that drives a real interactive session.
* Two valid coverage paths:
* 1. `canUseTool` via the agent-sdk-runner (legacy SDK-based path)
* 2. `runPlanSkillObservation` via the claude-pty-runner (real-PTY path
* added when the SDK harness was found unable to observe plan mode's
* native confirmation UI — see test/helpers/claude-pty-runner.ts)
*
* Runs as a free unit test (no API calls). Pure filesystem scan.
*/
@@ -76,14 +79,16 @@ function findInteractiveSkills(): string[] {
}
/**
* Scan a test file's contents for the canUseTool-via-harness pattern.
* Either: direct canUseTool usage in runAgentSdkTest, or usage of the
* shared plan-mode-helpers that wrap it.
* Scan a test file's contents for any of the supported real-interactive
* coverage patterns. Either: direct canUseTool usage in runAgentSdkTest,
* the legacy plan-mode-helpers wrapper, or the new real-PTY observation
* helper.
*/
function hasCanUseToolCoverage(testFile: string): boolean {
const content = fs.readFileSync(testFile, 'utf-8');
if (content.includes('canUseTool')) return true;
if (content.includes('runPlanModeSkillTest')) return true;
if (content.includes('runPlanSkillObservation')) return true;
return false;
}