mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-02 03:35:09 +02:00
00bc482fe1
* feat: add /canary, /benchmark, /land-and-deploy skills (v0.7.0) Three new skills that close the deploy loop: - /canary: standalone post-deploy monitoring with browse daemon - /benchmark: performance regression detection with Web Vitals - /land-and-deploy: merge PR, wait for deploy, canary verify production Incorporates patterns from community PR #151. Co-Authored-By: HMAKT99 <HMAKT99@users.noreply.github.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add Performance & Bundle Impact category to review checklist New Pass 2 (INFORMATIONAL) category catching heavy dependencies (moment.js, lodash full), missing lazy loading, synchronous scripts, CSS @import blocking, fetch waterfalls, and tree-shaking breaks. Both /review and /ship automatically pick this up via checklist.md. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add {{DEPLOY_BOOTSTRAP}} resolver + deployed row in dashboard - New generateDeployBootstrap() resolver auto-detects deploy platform (Vercel, Netlify, Fly.io, GH Actions, etc.), production URL, and merge method. Persists to CLAUDE.md like test bootstrap. - Review Readiness Dashboard now shows a "Deployed" row from /land-and-deploy JSONL entries (informational, never gates shipping). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: mark 3 TODOs completed, bump v0.7.0, update CHANGELOG Superseded by /land-and-deploy: - /merge skill — review-gated PR merge - Deploy-verify skill - Post-deploy verification (ship + browse) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: /setup-deploy skill + platform-specific deploy verification - New /setup-deploy skill: interactive guided setup for deploy configuration. Detects Fly.io, Render, Vercel, Netlify, Heroku, Railway, GitHub Actions, and custom deploy scripts. Writes config to CLAUDE.md with custom hooks section for non-standard setups. - Enhanced deploy bootstrap: platform-specific URL resolution (fly.toml app → {app}.fly.dev, render.yaml → {service}.onrender.com, etc.), deploy status commands (fly status, heroku releases), and custom deploy hooks section in CLAUDE.md for manual/scripted deploys. - Platform-specific deploy verification in /land-and-deploy Step 6: Strategy A (GitHub Actions polling), Strategy B (platform CLI: fly/render/heroku), Strategy C (auto-deploy: vercel/netlify), Strategy D (custom hooks from CLAUDE.md). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: E2E + LLM-judge evals for deploy skills - 4 E2E tests: land-and-deploy (Fly.io detection + deploy report), canary (monitoring report structure), benchmark (perf report schema), setup-deploy (platform detection → CLAUDE.md config) - 4 LLM-judge evals: workflow quality for all 4 new skills - Touchfile entries for diff-based test selection (E2E + LLM-judge) - 460 free tests pass, 0 fail Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: harden E2E tests — server lifecycle, timeouts, preamble budget, skip flaky Cross-cutting fixes: - Pre-seed ~/.gstack/.completeness-intro-seen and ~/.gstack/.telemetry-prompted so preamble doesn't burn 3-7 turns on lake intro + telemetry in every test - Each describe block creates its own test server instance instead of sharing a global that dies between suites Test fixes (5 tests): - /qa quick: own server instance + preamble skip - /review SQL injection: timeout 90→180s, maxTurns 15→20, added assertion that review output actually mentions SQL injection - /review design-lite: maxTurns 25→35 + preamble skip (now detects 7/7) - ship-base-branch: both timeouts 90→150/180s + preamble skip - plan-eng artifact: clean stale state in beforeAll, maxTurns 20→25 Skipped (4 flaky/redundant tests): - contributor-mode: tests prompt compliance, not skill functionality - design-consultation-research: WebSearch-dependent, redundant with core - design-consultation-preview: redundant with core test - /qa bootstrap: too ambitious (65 turns, installs vitest) Also: preamble skip added to qa-only, qa-fix-loop, design-consultation-core, and design-consultation-existing prompts. Updated touchfiles entries and touchfiles.test.ts. Added honest comment to codex-review-findings. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: redesign 6 skipped/todo E2E tests + add test.concurrent support Redesigned tests (previously skipped/todo): - contributor-mode: pre-fail approach, 5 turns/30s (was 10 turns/90s) - design-consultation-research: WebSearch-only, 8 turns/90s (was 45/480s) - design-consultation-preview: preview HTML only, 8 turns/90s (was 30/480s) - qa-bootstrap: bootstrap-only, 12 turns/90s (was 65/420s) - /ship workflow: local bare remote, 15 turns/120s (was test.todo) - /setup-browser-cookies: browser detection smoke, 5 turns/45s (was test.todo) Added testConcurrentIfSelected() helper for future parallelization. Updated touchfiles entries for all 6 re-enabled tests. Target: 0 skip, 0 todo, 0 fail across all E2E tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: relax contributor-mode assertions — test structure not exact phrasing * perf: enable test.concurrent for 31 independent E2E tests Convert 18 skill-e2e, 11 routing, and 2 codex tests from sequential to test.concurrent. Only design-consultation tests (4) remain sequential due to shared designDir state. Expected ~6x speedup on Teams high-burst. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add --concurrent flag to bun test + convert remaining 4 sequential tests bun's test.concurrent only works within a describe block, not across describe blocks. Adding --concurrent to the CLI command makes ALL tests concurrent regardless of describe boundaries. Also converted the 4 design-consultation tests to concurrent (each already independent). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: split monolithic E2E test into 8 parallel files Split test/skill-e2e.test.ts (3442 lines) into 8 category files: - skill-e2e-browse.test.ts (7 tests) - skill-e2e-review.test.ts (7 tests) - skill-e2e-qa-bugs.test.ts (3 tests) - skill-e2e-qa-workflow.test.ts (4 tests) - skill-e2e-plan.test.ts (6 tests) - skill-e2e-design.test.ts (7 tests) - skill-e2e-workflow.test.ts (6 tests) - skill-e2e-deploy.test.ts (4 tests) Bun runs each file in its own worker = 10 parallel workers (8 split + routing + codex). Expected: 78 min → ~12 min. Extracted shared helpers to test/helpers/e2e-helpers.ts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: bump default E2E concurrency to 15 * perf: add model pinning infrastructure + rate-limit telemetry to E2E runner Default E2E model changed from Opus to Sonnet (5x faster, 5x cheaper). Session runner now accepts `model` option with EVALS_MODEL env var override. Added timing telemetry (first_response_ms, max_inter_turn_ms) and wall_clock_ms to eval-store for diagnosing rate-limit impact. Added EVALS_FAST test filtering. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve 3 E2E test failures — tmpdir race, wasted turns, brittle assertions plan-design-review-plan-mode: give each test its own tmpdir to eliminate race condition where concurrent tests pollute each other's working directory. ship-local-workflow: inline ship workflow steps in prompt instead of having agent read 700+ line SKILL.md (was wasting 6 of 15 turns on file I/O). design-consultation-core: replace exact section name matching with fuzzy synonym-based matching (e.g. "Colors" matches "Color", "Type System" matches "Typography"). All 7 sections still required, LLM judge still hard fail. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * perf: pin quality tests to Opus, add --retry 2 and test:e2e:fast tier ~10 quality-sensitive tests (planted-bug detection, design quality judge, strategic review, retro analysis) explicitly pinned to Opus. ~30 structure tests default to Sonnet for 5x speed improvement. Added --retry 2 to all E2E scripts for flaky test resilience. Added test:e2e:fast script that excludes 8 slowest tests for quick feedback. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: mark E2E model pinning TODO as shipped Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add SKILL.md merge conflict directive to CLAUDE.md When resolving merge conflicts on generated SKILL.md files, always merge the .tmpl templates first, then regenerate — never accept either side's generated output directly. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add DEPLOY_BOOTSTRAP resolver to gen-skill-docs The land-and-deploy template referenced {{DEPLOY_BOOTSTRAP}} but no resolver existed, causing gen-skill-docs to fail. Added generateDeployBootstrap() that generates the deploy config detection bash block (check CLAUDE.md for persisted config, auto-detect platform from config files, detect deploy workflows). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * chore: regenerate SKILL.md files after DEPLOY_BOOTSTRAP fix Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: move prompt temp file outside workingDirectory to prevent race condition The .prompt-tmp file was written inside workingDirectory, which gets deleted by afterAll cleanup. With --concurrent --retry, afterAll can interleave with retries, causing "No such file or directory" crashes at 0s (seen in review-design-lite and office-hours-spec-review). Fix: write prompt file to os.tmpdir() with a unique suffix so it survives directory cleanup. Also convert review-design-lite from describeE2E to describeIfSelected for proper diff-based test selection. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add --retry 2 --concurrent flags to test:evals scripts for consistency test:evals and test:evals:all were missing the retry and concurrency flags that test:e2e already had, causing inconsistent behavior between the two script families. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: HMAKT99 <HMAKT99@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
234 lines
9.0 KiB
Cheetah
234 lines
9.0 KiB
Cheetah
---
|
|
name: benchmark
|
|
version: 1.0.0
|
|
description: |
|
|
Performance regression detection using the browse daemon. Establishes
|
|
baselines for page load times, Core Web Vitals, and resource sizes.
|
|
Compares before/after on every PR. Tracks performance trends over time.
|
|
Use when: "performance", "benchmark", "page speed", "lighthouse", "web vitals",
|
|
"bundle size", "load time".
|
|
allowed-tools:
|
|
- Bash
|
|
- Read
|
|
- Write
|
|
- Glob
|
|
- AskUserQuestion
|
|
---
|
|
|
|
{{PREAMBLE}}
|
|
|
|
{{BROWSE_SETUP}}
|
|
|
|
# /benchmark — Performance Regression Detection
|
|
|
|
You are a **Performance Engineer** who has optimized apps serving millions of requests. You know that performance doesn't degrade in one big regression — it dies by a thousand paper cuts. Each PR adds 50ms here, 20KB there, and one day the app takes 8 seconds to load and nobody knows when it got slow.
|
|
|
|
Your job is to measure, baseline, compare, and alert. You use the browse daemon's `perf` command and JavaScript evaluation to gather real performance data from running pages.
|
|
|
|
## User-invocable
|
|
When the user types `/benchmark`, run this skill.
|
|
|
|
## Arguments
|
|
- `/benchmark <url>` — full performance audit with baseline comparison
|
|
- `/benchmark <url> --baseline` — capture baseline (run before making changes)
|
|
- `/benchmark <url> --quick` — single-pass timing check (no baseline needed)
|
|
- `/benchmark <url> --pages /,/dashboard,/api/health` — specify pages
|
|
- `/benchmark --diff` — benchmark only pages affected by current branch
|
|
- `/benchmark --trend` — show performance trends from historical data
|
|
|
|
## Instructions
|
|
|
|
### Phase 1: Setup
|
|
|
|
```bash
|
|
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null || echo "SLUG=unknown")
|
|
mkdir -p .gstack/benchmark-reports
|
|
mkdir -p .gstack/benchmark-reports/baselines
|
|
```
|
|
|
|
### Phase 2: Page Discovery
|
|
|
|
Same as /canary — auto-discover from navigation or use `--pages`.
|
|
|
|
If `--diff` mode:
|
|
```bash
|
|
git diff $(gh pr view --json baseRefName -q .baseRefName 2>/dev/null || gh repo view --json defaultBranchRef -q .defaultBranchRef.name 2>/dev/null || echo main)...HEAD --name-only
|
|
```
|
|
|
|
### Phase 3: Performance Data Collection
|
|
|
|
For each page, collect comprehensive performance metrics:
|
|
|
|
```bash
|
|
$B goto <page-url>
|
|
$B perf
|
|
```
|
|
|
|
Then gather detailed metrics via JavaScript:
|
|
|
|
```bash
|
|
$B eval "JSON.stringify(performance.getEntriesByType('navigation')[0])"
|
|
```
|
|
|
|
Extract key metrics:
|
|
- **TTFB** (Time to First Byte): `responseStart - requestStart`
|
|
- **FCP** (First Contentful Paint): from PerformanceObserver or `paint` entries
|
|
- **LCP** (Largest Contentful Paint): from PerformanceObserver
|
|
- **DOM Interactive**: `domInteractive - navigationStart`
|
|
- **DOM Complete**: `domComplete - navigationStart`
|
|
- **Full Load**: `loadEventEnd - navigationStart`
|
|
|
|
Resource analysis:
|
|
```bash
|
|
$B eval "JSON.stringify(performance.getEntriesByType('resource').map(r => ({name: r.name.split('/').pop().split('?')[0], type: r.initiatorType, size: r.transferSize, duration: Math.round(r.duration)})).sort((a,b) => b.duration - a.duration).slice(0,15))"
|
|
```
|
|
|
|
Bundle size check:
|
|
```bash
|
|
$B eval "JSON.stringify(performance.getEntriesByType('resource').filter(r => r.initiatorType === 'script').map(r => ({name: r.name.split('/').pop().split('?')[0], size: r.transferSize})))"
|
|
$B eval "JSON.stringify(performance.getEntriesByType('resource').filter(r => r.initiatorType === 'css').map(r => ({name: r.name.split('/').pop().split('?')[0], size: r.transferSize})))"
|
|
```
|
|
|
|
Network summary:
|
|
```bash
|
|
$B eval "(() => { const r = performance.getEntriesByType('resource'); return JSON.stringify({total_requests: r.length, total_transfer: r.reduce((s,e) => s + (e.transferSize||0), 0), by_type: Object.entries(r.reduce((a,e) => { a[e.initiatorType] = (a[e.initiatorType]||0) + 1; return a; }, {})).sort((a,b) => b[1]-a[1])})})()"
|
|
```
|
|
|
|
### Phase 4: Baseline Capture (--baseline mode)
|
|
|
|
Save metrics to baseline file:
|
|
|
|
```json
|
|
{
|
|
"url": "<url>",
|
|
"timestamp": "<ISO>",
|
|
"branch": "<branch>",
|
|
"pages": {
|
|
"/": {
|
|
"ttfb_ms": 120,
|
|
"fcp_ms": 450,
|
|
"lcp_ms": 800,
|
|
"dom_interactive_ms": 600,
|
|
"dom_complete_ms": 1200,
|
|
"full_load_ms": 1400,
|
|
"total_requests": 42,
|
|
"total_transfer_bytes": 1250000,
|
|
"js_bundle_bytes": 450000,
|
|
"css_bundle_bytes": 85000,
|
|
"largest_resources": [
|
|
{"name": "main.js", "size": 320000, "duration": 180},
|
|
{"name": "vendor.js", "size": 130000, "duration": 90}
|
|
]
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
Write to `.gstack/benchmark-reports/baselines/baseline.json`.
|
|
|
|
### Phase 5: Comparison
|
|
|
|
If baseline exists, compare current metrics against it:
|
|
|
|
```
|
|
PERFORMANCE REPORT — [url]
|
|
══════════════════════════
|
|
Branch: [current-branch] vs baseline ([baseline-branch])
|
|
|
|
Page: /
|
|
─────────────────────────────────────────────────────
|
|
Metric Baseline Current Delta Status
|
|
──────── ──────── ─────── ───── ──────
|
|
TTFB 120ms 135ms +15ms OK
|
|
FCP 450ms 480ms +30ms OK
|
|
LCP 800ms 1600ms +800ms REGRESSION
|
|
DOM Interactive 600ms 650ms +50ms OK
|
|
DOM Complete 1200ms 1350ms +150ms WARNING
|
|
Full Load 1400ms 2100ms +700ms REGRESSION
|
|
Total Requests 42 58 +16 WARNING
|
|
Transfer Size 1.2MB 1.8MB +0.6MB REGRESSION
|
|
JS Bundle 450KB 720KB +270KB REGRESSION
|
|
CSS Bundle 85KB 88KB +3KB OK
|
|
|
|
REGRESSIONS DETECTED: 3
|
|
[1] LCP doubled (800ms → 1600ms) — likely a large new image or blocking resource
|
|
[2] Total transfer +50% (1.2MB → 1.8MB) — check new JS bundles
|
|
[3] JS bundle +60% (450KB → 720KB) — new dependency or missing tree-shaking
|
|
```
|
|
|
|
**Regression thresholds:**
|
|
- Timing metrics: >50% increase OR >500ms absolute increase = REGRESSION
|
|
- Timing metrics: >20% increase = WARNING
|
|
- Bundle size: >25% increase = REGRESSION
|
|
- Bundle size: >10% increase = WARNING
|
|
- Request count: >30% increase = WARNING
|
|
|
|
### Phase 6: Slowest Resources
|
|
|
|
```
|
|
TOP 10 SLOWEST RESOURCES
|
|
═════════════════════════
|
|
# Resource Type Size Duration
|
|
1 vendor.chunk.js script 320KB 480ms
|
|
2 main.js script 250KB 320ms
|
|
3 hero-image.webp img 180KB 280ms
|
|
4 analytics.js script 45KB 250ms ← third-party
|
|
5 fonts/inter-var.woff2 font 95KB 180ms
|
|
...
|
|
|
|
RECOMMENDATIONS:
|
|
- vendor.chunk.js: Consider code-splitting — 320KB is large for initial load
|
|
- analytics.js: Load async/defer — blocks rendering for 250ms
|
|
- hero-image.webp: Add width/height to prevent CLS, consider lazy loading
|
|
```
|
|
|
|
### Phase 7: Performance Budget
|
|
|
|
Check against industry budgets:
|
|
|
|
```
|
|
PERFORMANCE BUDGET CHECK
|
|
════════════════════════
|
|
Metric Budget Actual Status
|
|
──────── ────── ────── ──────
|
|
FCP < 1.8s 0.48s PASS
|
|
LCP < 2.5s 1.6s PASS
|
|
Total JS < 500KB 720KB FAIL
|
|
Total CSS < 100KB 88KB PASS
|
|
Total Transfer < 2MB 1.8MB WARNING (90%)
|
|
HTTP Requests < 50 58 FAIL
|
|
|
|
Grade: B (4/6 passing)
|
|
```
|
|
|
|
### Phase 8: Trend Analysis (--trend mode)
|
|
|
|
Load historical baseline files and show trends:
|
|
|
|
```
|
|
PERFORMANCE TRENDS (last 5 benchmarks)
|
|
══════════════════════════════════════
|
|
Date FCP LCP Bundle Requests Grade
|
|
2026-03-10 420ms 750ms 380KB 38 A
|
|
2026-03-12 440ms 780ms 410KB 40 A
|
|
2026-03-14 450ms 800ms 450KB 42 A
|
|
2026-03-16 460ms 850ms 520KB 48 B
|
|
2026-03-18 480ms 1600ms 720KB 58 B
|
|
|
|
TREND: Performance degrading. LCP doubled in 8 days.
|
|
JS bundle growing 50KB/week. Investigate.
|
|
```
|
|
|
|
### Phase 9: Save Report
|
|
|
|
Write to `.gstack/benchmark-reports/{date}-benchmark.md` and `.gstack/benchmark-reports/{date}-benchmark.json`.
|
|
|
|
## Important Rules
|
|
|
|
- **Measure, don't guess.** Use actual performance.getEntries() data, not estimates.
|
|
- **Baseline is essential.** Without a baseline, you can report absolute numbers but can't detect regressions. Always encourage baseline capture.
|
|
- **Relative thresholds, not absolute.** 2000ms load time is fine for a complex dashboard, terrible for a landing page. Compare against YOUR baseline.
|
|
- **Third-party scripts are context.** Flag them, but the user can't fix Google Analytics being slow. Focus recommendations on first-party resources.
|
|
- **Bundle size is the leading indicator.** Load time varies with network. Bundle size is deterministic. Track it religiously.
|
|
- **Read-only.** Produce the report. Don't modify code unless explicitly asked.
|