mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
00bc482fe1
* feat: add /canary, /benchmark, /land-and-deploy skills (v0.7.0) Three new skills that close the deploy loop: - /canary: standalone post-deploy monitoring with browse daemon - /benchmark: performance regression detection with Web Vitals - /land-and-deploy: merge PR, wait for deploy, canary verify production Incorporates patterns from community PR #151. Co-Authored-By: HMAKT99 <HMAKT99@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add Performance & Bundle Impact category to review checklist New Pass 2 (INFORMATIONAL) category catching heavy dependencies (moment.js, lodash full), missing lazy loading, synchronous scripts, CSS @import blocking, fetch waterfalls, and tree-shaking breaks. Both /review and /ship automatically pick this up via checklist.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add {{DEPLOY_BOOTSTRAP}} resolver + deployed row in dashboard - New generateDeployBootstrap() resolver auto-detects deploy platform (Vercel, Netlify, Fly.io, GH Actions, etc.), production URL, and merge method. Persists to CLAUDE.md like test bootstrap. - Review Readiness Dashboard now shows a "Deployed" row from /land-and-deploy JSONL entries (informational, never gates shipping). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: mark 3 TODOs completed, bump v0.7.0, update CHANGELOG Superseded by /land-and-deploy: - /merge skill — review-gated PR merge - Deploy-verify skill - Post-deploy verification (ship + browse) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: /setup-deploy skill + platform-specific deploy verification - New /setup-deploy skill: interactive guided setup for deploy configuration. Detects Fly.io, Render, Vercel, Netlify, Heroku, Railway, GitHub Actions, and custom deploy scripts. Writes config to CLAUDE.md with custom hooks section for non-standard setups. - Enhanced deploy bootstrap: platform-specific URL resolution (fly.toml app → {app}.fly.dev, render.yaml → {service}.onrender.com, etc.), deploy status commands (fly status, heroku releases), and custom deploy hooks section in CLAUDE.md for manual/scripted deploys. - Platform-specific deploy verification in /land-and-deploy Step 6: Strategy A (GitHub Actions polling), Strategy B (platform CLI: fly/render/heroku), Strategy C (auto-deploy: vercel/netlify), Strategy D (custom hooks from CLAUDE.md). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: E2E + LLM-judge evals for deploy skills - 4 E2E tests: land-and-deploy (Fly.io detection + deploy report), canary (monitoring report structure), benchmark (perf report schema), setup-deploy (platform detection → CLAUDE.md config) - 4 LLM-judge evals: workflow quality for all 4 new skills - Touchfile entries for diff-based test selection (E2E + LLM-judge) - 460 free tests pass, 0 fail Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: harden E2E tests — server lifecycle, timeouts, preamble budget, skip flaky Cross-cutting fixes: - Pre-seed ~/.gstack/.completeness-intro-seen and ~/.gstack/.telemetry-prompted so preamble doesn't burn 3-7 turns on lake intro + telemetry in every test - Each describe block creates its own test server instance instead of sharing a global that dies between suites Test fixes (5 tests): - /qa quick: own server instance + preamble skip - /review SQL injection: timeout 90→180s, maxTurns 15→20, added assertion that review output actually mentions SQL injection - /review design-lite: maxTurns 25→35 + preamble skip (now detects 7/7) - ship-base-branch: both timeouts 90→150/180s + preamble skip - plan-eng artifact: clean stale state in beforeAll, maxTurns 20→25 Skipped (4 flaky/redundant tests): - contributor-mode: tests prompt compliance, not skill functionality - design-consultation-research: WebSearch-dependent, redundant with core - design-consultation-preview: redundant with core test - /qa bootstrap: too ambitious (65 turns, installs vitest) Also: preamble skip added to qa-only, qa-fix-loop, design-consultation-core, and design-consultation-existing prompts. Updated touchfiles entries and touchfiles.test.ts. Added honest comment to codex-review-findings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: redesign 6 skipped/todo E2E tests + add test.concurrent support Redesigned tests (previously skipped/todo): - contributor-mode: pre-fail approach, 5 turns/30s (was 10 turns/90s) - design-consultation-research: WebSearch-only, 8 turns/90s (was 45/480s) - design-consultation-preview: preview HTML only, 8 turns/90s (was 30/480s) - qa-bootstrap: bootstrap-only, 12 turns/90s (was 65/420s) - /ship workflow: local bare remote, 15 turns/120s (was test.todo) - /setup-browser-cookies: browser detection smoke, 5 turns/45s (was test.todo) Added testConcurrentIfSelected() helper for future parallelization. Updated touchfiles entries for all 6 re-enabled tests. Target: 0 skip, 0 todo, 0 fail across all E2E tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: relax contributor-mode assertions — test structure not exact phrasing * perf: enable test.concurrent for 31 independent E2E tests Convert 18 skill-e2e, 11 routing, and 2 codex tests from sequential to test.concurrent. Only design-consultation tests (4) remain sequential due to shared designDir state. Expected ~6x speedup on Teams high-burst. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add --concurrent flag to bun test + convert remaining 4 sequential tests bun's test.concurrent only works within a describe block, not across describe blocks. Adding --concurrent to the CLI command makes ALL tests concurrent regardless of describe boundaries. Also converted the 4 design-consultation tests to concurrent (each already independent). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: split monolithic E2E test into 8 parallel files Split test/skill-e2e.test.ts (3442 lines) into 8 category files: - skill-e2e-browse.test.ts (7 tests) - skill-e2e-review.test.ts (7 tests) - skill-e2e-qa-bugs.test.ts (3 tests) - skill-e2e-qa-workflow.test.ts (4 tests) - skill-e2e-plan.test.ts (6 tests) - skill-e2e-design.test.ts (7 tests) - skill-e2e-workflow.test.ts (6 tests) - skill-e2e-deploy.test.ts (4 tests) Bun runs each file in its own worker = 10 parallel workers (8 split + routing + codex). Expected: 78 min → ~12 min. Extracted shared helpers to test/helpers/e2e-helpers.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: bump default E2E concurrency to 15 * perf: add model pinning infrastructure + rate-limit telemetry to E2E runner Default E2E model changed from Opus to Sonnet (5x faster, 5x cheaper). Session runner now accepts `model` option with EVALS_MODEL env var override. Added timing telemetry (first_response_ms, max_inter_turn_ms) and wall_clock_ms to eval-store for diagnosing rate-limit impact. Added EVALS_FAST test filtering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve 3 E2E test failures — tmpdir race, wasted turns, brittle assertions plan-design-review-plan-mode: give each test its own tmpdir to eliminate race condition where concurrent tests pollute each other's working directory. ship-local-workflow: inline ship workflow steps in prompt instead of having agent read 700+ line SKILL.md (was wasting 6 of 15 turns on file I/O). design-consultation-core: replace exact section name matching with fuzzy synonym-based matching (e.g. "Colors" matches "Color", "Type System" matches "Typography"). All 7 sections still required, LLM judge still hard fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: pin quality tests to Opus, add --retry 2 and test:e2e:fast tier ~10 quality-sensitive tests (planted-bug detection, design quality judge, strategic review, retro analysis) explicitly pinned to Opus. ~30 structure tests default to Sonnet for 5x speed improvement. Added --retry 2 to all E2E scripts for flaky test resilience. Added test:e2e:fast script that excludes 8 slowest tests for quick feedback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: mark E2E model pinning TODO as shipped Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add SKILL.md merge conflict directive to CLAUDE.md When resolving merge conflicts on generated SKILL.md files, always merge the .tmpl templates first, then regenerate — never accept either side's generated output directly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add DEPLOY_BOOTSTRAP resolver to gen-skill-docs The land-and-deploy template referenced {{DEPLOY_BOOTSTRAP}} but no resolver existed, causing gen-skill-docs to fail. Added generateDeployBootstrap() that generates the deploy config detection bash block (check CLAUDE.md for persisted config, auto-detect platform from config files, detect deploy workflows). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate SKILL.md files after DEPLOY_BOOTSTRAP fix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move prompt temp file outside workingDirectory to prevent race condition The .prompt-tmp file was written inside workingDirectory, which gets deleted by afterAll cleanup. With --concurrent --retry, afterAll can interleave with retries, causing "No such file or directory" crashes at 0s (seen in review-design-lite and office-hours-spec-review). Fix: write prompt file to os.tmpdir() with a unique suffix so it survives directory cleanup. Also convert review-design-lite from describeE2E to describeIfSelected for proper diff-based test selection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add --retry 2 --concurrent flags to test:evals scripts for consistency test:evals and test:evals:all were missing the retry and concurrency flags that test:e2e already had, causing inconsistent behavior between the two script families. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: HMAKT99 <HMAKT99@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
587 lines
23 KiB
TypeScript
587 lines
23 KiB
TypeScript
import { describe, test, expect, beforeAll, afterAll } from 'bun:test';
|
|
import { runSkillTest } from './helpers/session-runner';
|
|
import {
|
|
ROOT, browseBin, runId, evalsEnabled,
|
|
describeIfSelected, testConcurrentIfSelected,
|
|
copyDirSync, setupBrowseShims, logCost, recordE2E,
|
|
createEvalCollector, finalizeEvalCollector,
|
|
} from './helpers/e2e-helpers';
|
|
import { spawnSync } from 'child_process';
|
|
import * as fs from 'fs';
|
|
import * as path from 'path';
|
|
import * as os from 'os';
|
|
|
|
const evalCollector = createEvalCollector('e2e-workflow');
|
|
|
|
// --- Document-Release skill E2E ---
|
|
|
|
describeIfSelected('Document-Release skill E2E', ['document-release'], () => {
|
|
let docReleaseDir: string;
|
|
|
|
beforeAll(() => {
|
|
docReleaseDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-doc-release-'));
|
|
|
|
// Copy document-release skill files
|
|
copyDirSync(path.join(ROOT, 'document-release'), path.join(docReleaseDir, 'document-release'));
|
|
|
|
// Init git repo with initial docs
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: docReleaseDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Create initial README with a features list
|
|
fs.writeFileSync(path.join(docReleaseDir, 'README.md'),
|
|
'# Test Project\n\n## Features\n\n- Feature A\n- Feature B\n\n## Install\n\n```bash\nnpm install\n```\n');
|
|
|
|
// Create initial CHANGELOG that must NOT be clobbered
|
|
fs.writeFileSync(path.join(docReleaseDir, 'CHANGELOG.md'),
|
|
'# Changelog\n\n## 1.0.0 — 2026-03-01\n\n- Initial release with Feature A and Feature B\n- Setup CI pipeline\n');
|
|
|
|
// Create VERSION file (already bumped)
|
|
fs.writeFileSync(path.join(docReleaseDir, 'VERSION'), '1.1.0\n');
|
|
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial']);
|
|
|
|
// Create feature branch with a code change
|
|
run('git', ['checkout', '-b', 'feat/add-feature-c']);
|
|
fs.writeFileSync(path.join(docReleaseDir, 'feature-c.ts'), 'export function featureC() { return "C"; }\n');
|
|
fs.writeFileSync(path.join(docReleaseDir, 'VERSION'), '1.1.1\n');
|
|
fs.writeFileSync(path.join(docReleaseDir, 'CHANGELOG.md'),
|
|
'# Changelog\n\n## 1.1.1 — 2026-03-16\n\n- Added Feature C\n\n## 1.0.0 — 2026-03-01\n\n- Initial release with Feature A and Feature B\n- Setup CI pipeline\n');
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'feat: add feature C']);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(docReleaseDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
test('/document-release updates docs without clobbering CHANGELOG', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read the file document-release/SKILL.md for the document-release workflow instructions.
|
|
|
|
Run the /document-release workflow on this repo. The base branch is "main".
|
|
|
|
IMPORTANT:
|
|
- Do NOT use AskUserQuestion — auto-approve everything or skip if unsure.
|
|
- Do NOT push or create PRs (there is no remote).
|
|
- Do NOT run gh commands (no remote).
|
|
- Focus on updating README.md to reflect the new Feature C.
|
|
- Do NOT overwrite or regenerate CHANGELOG entries.
|
|
- Skip VERSION bump (it's already bumped).
|
|
- After editing, just commit the changes locally.`,
|
|
workingDirectory: docReleaseDir,
|
|
maxTurns: 30,
|
|
allowedTools: ['Bash', 'Read', 'Write', 'Edit', 'Grep', 'Glob'],
|
|
timeout: 180_000,
|
|
testName: 'document-release',
|
|
runId,
|
|
});
|
|
|
|
logCost('/document-release', result);
|
|
|
|
// Read CHANGELOG to verify it was NOT clobbered
|
|
const changelog = fs.readFileSync(path.join(docReleaseDir, 'CHANGELOG.md'), 'utf-8');
|
|
const hasOriginalEntries = changelog.includes('Initial release with Feature A and Feature B')
|
|
&& changelog.includes('Setup CI pipeline')
|
|
&& changelog.includes('1.0.0');
|
|
if (!hasOriginalEntries) {
|
|
console.warn('CHANGELOG CLOBBERED — original entries missing!');
|
|
}
|
|
|
|
// Check if README was updated
|
|
const readme = fs.readFileSync(path.join(docReleaseDir, 'README.md'), 'utf-8');
|
|
const readmeUpdated = readme.includes('Feature C') || readme.includes('feature-c') || readme.includes('feature C');
|
|
|
|
const exitOk = ['success', 'error_max_turns'].includes(result.exitReason);
|
|
recordE2E(evalCollector, '/document-release', 'Document-Release skill E2E', result, {
|
|
passed: exitOk && hasOriginalEntries,
|
|
});
|
|
|
|
// Critical guardrail: CHANGELOG must not be clobbered
|
|
expect(hasOriginalEntries).toBe(true);
|
|
|
|
// Accept error_max_turns — thorough doc review is not a failure
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
|
|
// Informational: did it update README?
|
|
if (readmeUpdated) {
|
|
console.log('README updated to include Feature C');
|
|
} else {
|
|
console.warn('README was NOT updated — agent may not have found the feature');
|
|
}
|
|
}, 240_000);
|
|
});
|
|
|
|
// --- Ship workflow with local bare remote ---
|
|
|
|
describeIfSelected('Ship workflow E2E', ['ship-local-workflow'], () => {
|
|
let shipWorkDir: string;
|
|
let shipRemoteDir: string;
|
|
|
|
beforeAll(() => {
|
|
shipRemoteDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-ship-remote-'));
|
|
shipWorkDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-ship-work-'));
|
|
|
|
// Create bare remote
|
|
spawnSync('git', ['init', '--bare'], { cwd: shipRemoteDir, stdio: 'pipe' });
|
|
|
|
// Clone it as working repo
|
|
spawnSync('git', ['clone', shipRemoteDir, shipWorkDir], { stdio: 'pipe' });
|
|
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: shipWorkDir, stdio: 'pipe', timeout: 5000 });
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Initial commit on main
|
|
fs.writeFileSync(path.join(shipWorkDir, 'app.ts'), 'console.log("v1");\n');
|
|
fs.writeFileSync(path.join(shipWorkDir, 'VERSION'), '0.1.0.0\n');
|
|
fs.writeFileSync(path.join(shipWorkDir, 'CHANGELOG.md'), '# Changelog\n');
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial']);
|
|
run('git', ['push', '-u', 'origin', 'main']);
|
|
|
|
// Feature branch
|
|
run('git', ['checkout', '-b', 'feature/ship-test']);
|
|
fs.writeFileSync(path.join(shipWorkDir, 'app.ts'), 'console.log("v2");\n');
|
|
run('git', ['add', 'app.ts']);
|
|
run('git', ['commit', '-m', 'feat: update to v2']);
|
|
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(shipWorkDir, { recursive: true, force: true }); } catch {}
|
|
try { fs.rmSync(shipRemoteDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('ship-local-workflow', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `You are running a ship workflow. This is fully automated — do NOT ask for confirmation at any step. Run straight through.
|
|
|
|
Step 0 — Detect base branch:
|
|
Try: gh pr view --json baseRefName -q .baseRefName
|
|
If that fails, try: gh repo view --json defaultBranchRef -q .defaultBranchRef.name
|
|
If both fail, fall back to "main". Use the detected branch as <base> in all subsequent steps.
|
|
|
|
Step 2 — Merge base branch:
|
|
git fetch origin <base> && git merge origin/<base> --no-edit
|
|
If already up to date, continue silently.
|
|
|
|
Step 4 — Version bump:
|
|
Read the VERSION file (4-digit format: MAJOR.MINOR.PATCH.MICRO).
|
|
Auto-pick MICRO bump (increment the 4th digit). Write the new version to VERSION.
|
|
|
|
Step 5 — CHANGELOG:
|
|
Read CHANGELOG.md. Auto-generate an entry from the branch commits:
|
|
- git log <base>..HEAD --oneline
|
|
- git diff <base>...HEAD
|
|
Format: ## [X.Y.Z.W] - YYYY-MM-DD with bullet points. Prepend after the header.
|
|
|
|
Step 6 — Commit:
|
|
Stage all changes. Commit with message: "chore: bump version and changelog (vX.Y.Z.W)"
|
|
|
|
Step 7 — Push:
|
|
git push -u origin <branch-name>
|
|
|
|
Finally, write ship-summary.md with the version and branch.`,
|
|
workingDirectory: shipWorkDir,
|
|
maxTurns: 15,
|
|
timeout: 120_000,
|
|
testName: 'ship-local-workflow',
|
|
runId,
|
|
});
|
|
|
|
logCost('/ship local workflow', result);
|
|
|
|
// Check push succeeded
|
|
const remoteLog = spawnSync('git', ['log', '--oneline'], { cwd: shipRemoteDir, stdio: 'pipe' });
|
|
const remoteCommits = remoteLog.stdout.toString().trim().split('\n').length;
|
|
|
|
// Check VERSION was bumped
|
|
const versionContent = fs.existsSync(path.join(shipWorkDir, 'VERSION'))
|
|
? fs.readFileSync(path.join(shipWorkDir, 'VERSION'), 'utf-8').trim() : '';
|
|
const versionBumped = versionContent !== '0.1.0.0';
|
|
|
|
recordE2E(evalCollector, '/ship local workflow', 'Ship workflow E2E', result, {
|
|
passed: remoteCommits > 1 && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(remoteCommits).toBeGreaterThan(1);
|
|
console.log(`Remote commits: ${remoteCommits}, VERSION: ${versionContent}, bumped: ${versionBumped}`);
|
|
}, 150_000);
|
|
});
|
|
|
|
// --- Browser cookie detection smoke test ---
|
|
|
|
describeIfSelected('Setup Browser Cookies E2E', ['setup-cookies-detect'], () => {
|
|
let cookieDir: string;
|
|
|
|
beforeAll(() => {
|
|
cookieDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-cookies-'));
|
|
// Copy skill files
|
|
fs.mkdirSync(path.join(cookieDir, 'setup-browser-cookies'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'setup-browser-cookies', 'SKILL.md'),
|
|
path.join(cookieDir, 'setup-browser-cookies', 'SKILL.md'),
|
|
);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(cookieDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('setup-cookies-detect', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read setup-browser-cookies/SKILL.md for the cookie import workflow.
|
|
|
|
This is a test environment. List which browsers you can detect on this system by checking for their cookie database files.
|
|
Write the detected browsers to ${cookieDir}/detected-browsers.md.
|
|
Do NOT launch the cookie picker UI — just detect and report.`,
|
|
workingDirectory: cookieDir,
|
|
maxTurns: 5,
|
|
timeout: 45_000,
|
|
testName: 'setup-cookies-detect',
|
|
runId,
|
|
});
|
|
|
|
logCost('/setup-browser-cookies detect', result);
|
|
|
|
const detectPath = path.join(cookieDir, 'detected-browsers.md');
|
|
const detectExists = fs.existsSync(detectPath);
|
|
const detectContent = detectExists ? fs.readFileSync(detectPath, 'utf-8') : '';
|
|
const hasBrowserName = /chrome|arc|brave|edge|comet|safari|firefox/i.test(detectContent);
|
|
|
|
recordE2E(evalCollector, '/setup-browser-cookies detect', 'Setup Browser Cookies E2E', result, {
|
|
passed: detectExists && hasBrowserName && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(detectExists).toBe(true);
|
|
if (detectExists) {
|
|
expect(hasBrowserName).toBe(true);
|
|
}
|
|
}, 60_000);
|
|
});
|
|
|
|
// --- gstack-upgrade E2E ---
|
|
|
|
describeIfSelected('gstack-upgrade E2E', ['gstack-upgrade-happy-path'], () => {
|
|
let upgradeDir: string;
|
|
let remoteDir: string;
|
|
|
|
beforeAll(() => {
|
|
upgradeDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-upgrade-'));
|
|
remoteDir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-remote-'));
|
|
|
|
const run = (cmd: string, args: string[], cwd: string) =>
|
|
spawnSync(cmd, args, { cwd, stdio: 'pipe', timeout: 5000 });
|
|
|
|
// Init the "project" repo
|
|
run('git', ['init'], upgradeDir);
|
|
run('git', ['config', 'user.email', 'test@test.com'], upgradeDir);
|
|
run('git', ['config', 'user.name', 'Test'], upgradeDir);
|
|
|
|
// Create mock gstack install directory (local-git type)
|
|
const mockGstack = path.join(upgradeDir, '.claude', 'skills', 'gstack');
|
|
fs.mkdirSync(mockGstack, { recursive: true });
|
|
|
|
// Init as a git repo
|
|
run('git', ['init'], mockGstack);
|
|
run('git', ['config', 'user.email', 'test@test.com'], mockGstack);
|
|
run('git', ['config', 'user.name', 'Test'], mockGstack);
|
|
|
|
// Create bare remote
|
|
run('git', ['init', '--bare'], remoteDir);
|
|
run('git', ['remote', 'add', 'origin', remoteDir], mockGstack);
|
|
|
|
// Write old version files
|
|
fs.writeFileSync(path.join(mockGstack, 'VERSION'), '0.5.0\n');
|
|
fs.writeFileSync(path.join(mockGstack, 'CHANGELOG.md'),
|
|
'# Changelog\n\n## 0.5.0 — 2026-03-01\n\n- Initial release\n');
|
|
fs.writeFileSync(path.join(mockGstack, 'setup'),
|
|
'#!/bin/bash\necho "Setup completed"\n', { mode: 0o755 });
|
|
|
|
// Initial commit + push
|
|
run('git', ['add', '.'], mockGstack);
|
|
run('git', ['commit', '-m', 'initial'], mockGstack);
|
|
run('git', ['push', '-u', 'origin', 'HEAD:main'], mockGstack);
|
|
|
|
// Create new version (simulate upstream release)
|
|
fs.writeFileSync(path.join(mockGstack, 'VERSION'), '0.6.0\n');
|
|
fs.writeFileSync(path.join(mockGstack, 'CHANGELOG.md'),
|
|
'# Changelog\n\n## 0.6.0 — 2026-03-15\n\n- New feature: interactive design review\n- Fix: snapshot flag validation\n\n## 0.5.0 — 2026-03-01\n\n- Initial release\n');
|
|
run('git', ['add', '.'], mockGstack);
|
|
run('git', ['commit', '-m', 'release 0.6.0'], mockGstack);
|
|
run('git', ['push', 'origin', 'HEAD:main'], mockGstack);
|
|
|
|
// Reset working copy back to old version
|
|
run('git', ['reset', '--hard', 'HEAD~1'], mockGstack);
|
|
|
|
// Copy gstack-upgrade skill
|
|
fs.mkdirSync(path.join(upgradeDir, 'gstack-upgrade'), { recursive: true });
|
|
fs.copyFileSync(
|
|
path.join(ROOT, 'gstack-upgrade', 'SKILL.md'),
|
|
path.join(upgradeDir, 'gstack-upgrade', 'SKILL.md'),
|
|
);
|
|
|
|
// Commit so git repo is clean
|
|
run('git', ['add', '.'], upgradeDir);
|
|
run('git', ['commit', '-m', 'initial project'], upgradeDir);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(upgradeDir, { recursive: true, force: true }); } catch {}
|
|
try { fs.rmSync(remoteDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
testConcurrentIfSelected('gstack-upgrade-happy-path', async () => {
|
|
const mockGstack = path.join(upgradeDir, '.claude', 'skills', 'gstack');
|
|
const result = await runSkillTest({
|
|
prompt: `Read gstack-upgrade/SKILL.md for the upgrade workflow.
|
|
|
|
You are running /gstack-upgrade standalone. The gstack installation is at ./.claude/skills/gstack (local-git type — it has a .git directory with an origin remote).
|
|
|
|
Current version: 0.5.0. A new version 0.6.0 is available on origin/main.
|
|
|
|
Follow the standalone upgrade flow:
|
|
1. Detect install type (local-git)
|
|
2. Run git fetch origin && git reset --hard origin/main in the install directory
|
|
3. Run the setup script
|
|
4. Show what's new from CHANGELOG
|
|
|
|
Skip any AskUserQuestion calls — auto-approve the upgrade. Write a summary of what you did to stdout.
|
|
|
|
IMPORTANT: The install directory is at ./.claude/skills/gstack — use that exact path.`,
|
|
workingDirectory: upgradeDir,
|
|
maxTurns: 20,
|
|
timeout: 180_000,
|
|
testName: 'gstack-upgrade-happy-path',
|
|
runId,
|
|
});
|
|
|
|
logCost('/gstack-upgrade happy path', result);
|
|
|
|
// Check that the version was updated
|
|
const versionAfter = fs.readFileSync(path.join(mockGstack, 'VERSION'), 'utf-8').trim();
|
|
const output = result.output || '';
|
|
const mentionsUpgrade = output.toLowerCase().includes('0.6.0') ||
|
|
output.toLowerCase().includes('upgrade') ||
|
|
output.toLowerCase().includes('updated');
|
|
|
|
recordE2E(evalCollector, '/gstack-upgrade happy path', 'gstack-upgrade E2E', result, {
|
|
passed: versionAfter === '0.6.0' && ['success', 'error_max_turns'].includes(result.exitReason),
|
|
});
|
|
|
|
expect(['success', 'error_max_turns']).toContain(result.exitReason);
|
|
expect(versionAfter).toBe('0.6.0');
|
|
}, 240_000);
|
|
});
|
|
|
|
// --- Test Coverage Audit E2E ---
|
|
|
|
describeIfSelected('Test Coverage Audit E2E', ['ship-coverage-audit'], () => {
|
|
let coverageDir: string;
|
|
|
|
beforeAll(() => {
|
|
coverageDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-coverage-'));
|
|
|
|
// Copy ship skill files
|
|
copyDirSync(path.join(ROOT, 'ship'), path.join(coverageDir, 'ship'));
|
|
copyDirSync(path.join(ROOT, 'review'), path.join(coverageDir, 'review'));
|
|
|
|
// Create a Node.js project WITH test framework but coverage gaps
|
|
fs.writeFileSync(path.join(coverageDir, 'package.json'), JSON.stringify({
|
|
name: 'test-coverage-app',
|
|
version: '1.0.0',
|
|
type: 'module',
|
|
scripts: { test: 'echo "no tests yet"' },
|
|
devDependencies: { vitest: '^1.0.0' },
|
|
}, null, 2));
|
|
|
|
// Create vitest config
|
|
fs.writeFileSync(path.join(coverageDir, 'vitest.config.ts'),
|
|
`import { defineConfig } from 'vitest/config';\nexport default defineConfig({ test: {} });\n`);
|
|
|
|
fs.writeFileSync(path.join(coverageDir, 'VERSION'), '0.1.0.0\n');
|
|
fs.writeFileSync(path.join(coverageDir, 'CHANGELOG.md'), '# Changelog\n');
|
|
|
|
// Create source file with multiple code paths
|
|
fs.mkdirSync(path.join(coverageDir, 'src'), { recursive: true });
|
|
fs.writeFileSync(path.join(coverageDir, 'src', 'billing.ts'), `
|
|
export function processPayment(amount: number, currency: string) {
|
|
if (amount <= 0) throw new Error('Invalid amount');
|
|
if (currency !== 'USD' && currency !== 'EUR') throw new Error('Unsupported currency');
|
|
return { status: 'success', amount, currency };
|
|
}
|
|
|
|
export function refundPayment(paymentId: string, reason: string) {
|
|
if (!paymentId) throw new Error('Payment ID required');
|
|
if (!reason) throw new Error('Reason required');
|
|
return { status: 'refunded', paymentId, reason };
|
|
}
|
|
`);
|
|
|
|
// Create a test directory with ONE test (partial coverage)
|
|
fs.mkdirSync(path.join(coverageDir, 'test'), { recursive: true });
|
|
fs.writeFileSync(path.join(coverageDir, 'test', 'billing.test.ts'), `
|
|
import { describe, test, expect } from 'vitest';
|
|
import { processPayment } from '../src/billing';
|
|
|
|
describe('processPayment', () => {
|
|
test('processes valid payment', () => {
|
|
const result = processPayment(100, 'USD');
|
|
expect(result.status).toBe('success');
|
|
});
|
|
// GAP: no test for invalid amount
|
|
// GAP: no test for unsupported currency
|
|
// GAP: refundPayment not tested at all
|
|
});
|
|
`);
|
|
|
|
// Init git repo with main branch
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: coverageDir, stdio: 'pipe', timeout: 5000 });
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
run('git', ['add', '.']);
|
|
run('git', ['commit', '-m', 'initial commit']);
|
|
|
|
// Create feature branch
|
|
run('git', ['checkout', '-b', 'feature/billing']);
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(coverageDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
test('/ship Step 3.4 produces coverage diagram', async () => {
|
|
const result = await runSkillTest({
|
|
prompt: `Read the file ship/SKILL.md for the ship workflow instructions.
|
|
|
|
You are on the feature/billing branch. The base branch is main.
|
|
This is a test project — there is no remote, no PR to create.
|
|
|
|
ONLY run Step 3.4 (Test Coverage Audit) from the ship workflow.
|
|
Skip all other steps (tests, evals, review, version, changelog, commit, push, PR).
|
|
|
|
The source code is in ${coverageDir}/src/billing.ts.
|
|
Existing tests are in ${coverageDir}/test/billing.test.ts.
|
|
The test command is: echo "tests pass" (mocked — just pretend tests pass).
|
|
|
|
Produce the ASCII coverage diagram showing which code paths are tested and which have gaps.
|
|
Do NOT generate new tests — just produce the diagram and coverage summary.
|
|
Output the diagram directly.`,
|
|
workingDirectory: coverageDir,
|
|
maxTurns: 15,
|
|
allowedTools: ['Bash', 'Read', 'Write', 'Edit', 'Glob', 'Grep'],
|
|
timeout: 120_000,
|
|
testName: 'ship-coverage-audit',
|
|
runId,
|
|
});
|
|
|
|
logCost('/ship coverage audit', result);
|
|
recordE2E(evalCollector, '/ship Step 3.4 coverage audit', 'Test Coverage Audit E2E', result, {
|
|
passed: result.exitReason === 'success',
|
|
});
|
|
|
|
expect(result.exitReason).toBe('success');
|
|
|
|
// Check output contains coverage diagram elements
|
|
const output = result.output || '';
|
|
const hasGap = output.includes('GAP') || output.includes('gap') || output.includes('NO TEST');
|
|
const hasTested = output.includes('TESTED') || output.includes('tested') || output.includes('✓');
|
|
const hasCoverage = output.includes('COVERAGE') || output.includes('coverage') || output.includes('paths tested');
|
|
|
|
console.log(`Output has GAP markers: ${hasGap}`);
|
|
console.log(`Output has TESTED markers: ${hasTested}`);
|
|
console.log(`Output has coverage summary: ${hasCoverage}`);
|
|
|
|
// At minimum, the agent should have read the source and test files
|
|
const readCalls = result.toolCalls.filter(tc => tc.tool === 'Read');
|
|
expect(readCalls.length).toBeGreaterThan(0);
|
|
}, 180_000);
|
|
});
|
|
|
|
// --- Codex skill E2E ---
|
|
|
|
describeIfSelected('Codex skill E2E', ['codex-review'], () => {
|
|
let codexDir: string;
|
|
|
|
beforeAll(() => {
|
|
codexDir = fs.mkdtempSync(path.join(os.tmpdir(), 'skill-e2e-codex-'));
|
|
|
|
const run = (cmd: string, args: string[]) =>
|
|
spawnSync(cmd, args, { cwd: codexDir, stdio: 'pipe', timeout: 5000 });
|
|
|
|
run('git', ['init', '-b', 'main']);
|
|
run('git', ['config', 'user.email', 'test@test.com']);
|
|
run('git', ['config', 'user.name', 'Test']);
|
|
|
|
// Commit a clean base on main
|
|
fs.writeFileSync(path.join(codexDir, 'app.rb'), '# clean base\nclass App\nend\n');
|
|
run('git', ['add', 'app.rb']);
|
|
run('git', ['commit', '-m', 'initial commit']);
|
|
|
|
// Create feature branch with vulnerable code (reuse review fixture)
|
|
run('git', ['checkout', '-b', 'feature/add-vuln']);
|
|
const vulnContent = fs.readFileSync(path.join(ROOT, 'test', 'fixtures', 'review-eval-vuln.rb'), 'utf-8');
|
|
fs.writeFileSync(path.join(codexDir, 'user_controller.rb'), vulnContent);
|
|
run('git', ['add', 'user_controller.rb']);
|
|
run('git', ['commit', '-m', 'add vulnerable controller']);
|
|
|
|
// Copy the codex skill file
|
|
fs.copyFileSync(path.join(ROOT, 'codex', 'SKILL.md'), path.join(codexDir, 'codex-SKILL.md'));
|
|
});
|
|
|
|
afterAll(() => {
|
|
try { fs.rmSync(codexDir, { recursive: true, force: true }); } catch {}
|
|
});
|
|
|
|
test('/codex review produces findings and GATE verdict', async () => {
|
|
// Check codex is available — skip if not installed
|
|
const codexCheck = spawnSync('which', ['codex'], { stdio: 'pipe', timeout: 3000 });
|
|
if (codexCheck.status !== 0) {
|
|
console.warn('codex CLI not installed — skipping E2E test');
|
|
return;
|
|
}
|
|
|
|
const result = await runSkillTest({
|
|
prompt: `You are in a git repo on branch feature/add-vuln with changes against main.
|
|
Read codex-SKILL.md for the /codex skill instructions.
|
|
Run /codex review to review the current diff against main.
|
|
Write the full output (including the GATE verdict) to ${codexDir}/codex-output.md`,
|
|
workingDirectory: codexDir,
|
|
maxTurns: 15,
|
|
timeout: 300_000,
|
|
testName: 'codex-review',
|
|
runId,
|
|
model: 'claude-opus-4-6',
|
|
});
|
|
|
|
logCost('/codex review', result);
|
|
recordE2E(evalCollector, '/codex review', 'Codex skill E2E', result);
|
|
expect(result.exitReason).toBe('success');
|
|
|
|
// Check that output file was created with review content
|
|
const outputPath = path.join(codexDir, 'codex-output.md');
|
|
if (fs.existsSync(outputPath)) {
|
|
const output = fs.readFileSync(outputPath, 'utf-8');
|
|
// Should contain the CODEX SAYS header or GATE verdict
|
|
const hasCodexOutput = output.includes('CODEX') || output.includes('GATE') || output.includes('codex');
|
|
expect(hasCodexOutput).toBe(true);
|
|
}
|
|
}, 360_000);
|
|
});
|
|
|
|
// Module-level afterAll — finalize eval collector after all tests complete
|
|
afterAll(async () => {
|
|
await finalizeEvalCollector(evalCollector);
|
|
});
|