mirror of
https://github.com/garrytan/gstack.git
synced 2026-08-07 14:58:37 +02:00
feat: QA restructure, browser ref staleness, eval efficiency metrics (v0.4.0) (#83)
* feat: browser ref staleness detection via async count() validation resolveRef() now checks element count to detect stale refs after page mutations (e.g. SPA navigation). RefEntry stores role+name metadata for better diagnostics. 3 new snapshot tests for staleness detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: qa-only skill, qa fix loop, plan-to-QA artifact flow Add /qa-only (report-only, Edit tool blocked), restructure /qa with find-fix-verify cycle, add {{QA_METHODOLOGY}} DRY placeholder for shared methodology. /plan-eng-review now writes test-plan artifacts to ~/.gstack/projects/<slug>/ for QA consumption. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: eval efficiency metrics — turns, duration, commentary across all surfaces Add generateCommentary() for natural-language delta interpretation, per-test turns/duration in comparison and summary output, judgePassed unit tests, 3 new E2E tests (qa-only, qa fix loop, plan artifact). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * chore: bump version and changelog (v0.4.0) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: update ARCHITECTURE, BROWSER, CONTRIBUTING, README for v0.4.0 - ARCHITECTURE: add ref staleness detection section, update RefEntry type - BROWSER: add ref staleness paragraph to snapshot system docs - CONTRIBUTING: update eval tool descriptions with commentary feature - README: fix missing qa-only in project-local uninstall command Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * docs: add user-facing benefit descriptions to v0.4.0 changelog Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Garry Tan
co-authored by
Claude Opus 4.6
Claude Opus 4.6
parent
bb46ca6b21
commit
f3ee0ee28a
+189
-10
@@ -1,16 +1,20 @@
|
||||
---
|
||||
name: qa
|
||||
version: 1.0.0
|
||||
version: 2.0.0
|
||||
description: |
|
||||
Systematically QA test a web application. Use when asked to "qa", "QA", "test this site",
|
||||
"find bugs", "dogfood", or review quality. Four modes: diff-aware (automatic on feature
|
||||
branches — analyzes git diff, identifies affected pages, tests them), full (systematic
|
||||
exploration), quick (30-second smoke test), regression (compare against baseline). Produces
|
||||
structured report with health score, screenshots, and repro steps.
|
||||
Systematically QA test a web application and fix bugs found. Runs QA testing,
|
||||
then iteratively fixes bugs in source code, committing each fix atomically and
|
||||
re-verifying. Use when asked to "qa", "QA", "test this site", "find bugs",
|
||||
"test and fix", or "fix what's broken". Three tiers: Quick (critical/high only),
|
||||
Standard (+ medium), Exhaustive (+ cosmetic). Produces before/after health scores,
|
||||
fix evidence, and a ship-readiness summary. For report-only mode, use /qa-only.
|
||||
allowed-tools:
|
||||
- Bash
|
||||
- Read
|
||||
- Write
|
||||
- Edit
|
||||
- Glob
|
||||
- Grep
|
||||
- AskUserQuestion
|
||||
---
|
||||
<!-- AUTO-GENERATED from SKILL.md.tmpl — do not edit directly -->
|
||||
@@ -25,24 +29,38 @@ _UPD=$(~/.claude/skills/gstack/bin/gstack-update-check 2>/dev/null || .claude/sk
|
||||
|
||||
If output shows `UPGRADE_AVAILABLE <old> <new>`: read `~/.claude/skills/gstack/gstack-upgrade/SKILL.md` and follow the "Inline upgrade flow" (auto-upgrade if configured, otherwise AskUserQuestion with 4 options, write snooze state if declined). If `JUST_UPGRADED <from> <to>`: tell user "Running gstack v{to} (just updated!)" and continue.
|
||||
|
||||
# /qa: Systematic QA Testing
|
||||
# /qa: Test → Fix → Verify
|
||||
|
||||
You are a QA engineer. Test web applications like a real user — click everything, fill every form, check every state. Produce a structured report with evidence.
|
||||
You are a QA engineer AND a bug-fix engineer. Test web applications like a real user — click everything, fill every form, check every state. When you find bugs, fix them in source code with atomic commits, then re-verify. Produce a structured report with before/after evidence.
|
||||
|
||||
## Setup
|
||||
|
||||
**Parse the user's request for these parameters:**
|
||||
|
||||
| Parameter | Default | Override example |
|
||||
|-----------|---------|-----------------|
|
||||
|-----------|---------|-----------------:|
|
||||
| Target URL | (auto-detect or required) | `https://myapp.com`, `http://localhost:3000` |
|
||||
| Mode | full | `--quick`, `--regression .gstack/qa-reports/baseline.json` |
|
||||
| Tier | Standard | `--quick`, `--exhaustive` |
|
||||
| Mode | full | `--regression .gstack/qa-reports/baseline.json` |
|
||||
| Output dir | `.gstack/qa-reports/` | `Output to /tmp/qa` |
|
||||
| Scope | Full app (or diff-scoped) | `Focus on the billing page` |
|
||||
| Auth | None | `Sign in to user@example.com`, `Import cookies from cookies.json` |
|
||||
|
||||
**Tiers determine which issues get fixed:**
|
||||
- **Quick:** Fix critical + high severity only
|
||||
- **Standard:** + medium severity (default)
|
||||
- **Exhaustive:** + low/cosmetic severity
|
||||
|
||||
**If no URL is given and you're on a feature branch:** Automatically enter **diff-aware mode** (see Modes below). This is the most common case — the user just shipped code on a branch and wants to verify it works.
|
||||
|
||||
**Require clean working tree before starting:**
|
||||
```bash
|
||||
if [ -n "$(git status --porcelain)" ]; then
|
||||
echo "ERROR: Working tree is dirty. Commit or stash changes before running /qa."
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
**Find the browse binary:**
|
||||
|
||||
## SETUP (run this check BEFORE any browse command)
|
||||
@@ -73,6 +91,22 @@ mkdir -p "$REPORT_DIR/screenshots"
|
||||
|
||||
---
|
||||
|
||||
## Test Plan Context
|
||||
|
||||
Before falling back to git diff heuristics, check for richer test plan sources:
|
||||
|
||||
1. **Project-scoped test plans:** Check `~/.gstack/projects/` for recent `*-test-plan-*.md` files for this repo
|
||||
```bash
|
||||
SLUG=$(git remote get-url origin 2>/dev/null | sed 's|.*[:/]\([^/]*/[^/]*\)\.git$|\1|;s|.*[:/]\([^/]*/[^/]*\)$|\1|' | tr '/' '-')
|
||||
ls -t ~/.gstack/projects/$SLUG/*-test-plan-*.md 2>/dev/null | head -1
|
||||
```
|
||||
2. **Conversation context:** Check if a prior `/plan-eng-review` or `/plan-ceo-review` produced test plan output in this conversation
|
||||
3. **Use whichever source is richer.** Fall back to git diff analysis only if neither is available.
|
||||
|
||||
---
|
||||
|
||||
## Phases 1-6: QA Baseline
|
||||
|
||||
## Modes
|
||||
|
||||
### Diff-aware (automatic when on a feature branch with no URL)
|
||||
@@ -347,6 +381,8 @@ Minimum 0 per category.
|
||||
9. **Never delete output files.** Screenshots and reports accumulate — that's intentional.
|
||||
10. **Use `snapshot -C` for tricky UIs.** Finds clickable divs that the accessibility tree misses.
|
||||
|
||||
Record baseline health score at end of Phase 6.
|
||||
|
||||
---
|
||||
|
||||
## Output Structure
|
||||
@@ -358,8 +394,151 @@ Minimum 0 per category.
|
||||
│ ├── initial.png # Landing page annotated screenshot
|
||||
│ ├── issue-001-step-1.png # Per-issue evidence
|
||||
│ ├── issue-001-result.png
|
||||
│ ├── issue-001-before.png # Before fix (if fixed)
|
||||
│ ├── issue-001-after.png # After fix (if fixed)
|
||||
│ └── ...
|
||||
└── baseline.json # For regression mode
|
||||
```
|
||||
|
||||
Report filenames use the domain and date: `qa-report-myapp-com-2026-03-12.md`
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: Triage
|
||||
|
||||
Sort all discovered issues by severity, then decide which to fix based on the selected tier:
|
||||
|
||||
- **Quick:** Fix critical + high only. Mark medium/low as "deferred."
|
||||
- **Standard:** Fix critical + high + medium. Mark low as "deferred."
|
||||
- **Exhaustive:** Fix all, including cosmetic/low severity.
|
||||
|
||||
Mark issues that cannot be fixed from source code (e.g., third-party widget bugs, infrastructure issues) as "deferred" regardless of tier.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: Fix Loop
|
||||
|
||||
For each fixable issue, in severity order:
|
||||
|
||||
### 8a. Locate source
|
||||
|
||||
```bash
|
||||
# Grep for error messages, component names, route definitions
|
||||
# Glob for file patterns matching the affected page
|
||||
```
|
||||
|
||||
- Find the source file(s) responsible for the bug
|
||||
- ONLY modify files directly related to the issue
|
||||
|
||||
### 8b. Fix
|
||||
|
||||
- Read the source code, understand the context
|
||||
- Make the **minimal fix** — smallest change that resolves the issue
|
||||
- Do NOT refactor surrounding code, add features, or "improve" unrelated things
|
||||
|
||||
### 8c. Commit
|
||||
|
||||
```bash
|
||||
git add <only-changed-files>
|
||||
git commit -m "fix(qa): ISSUE-NNN — short description"
|
||||
```
|
||||
|
||||
- One commit per fix. Never bundle multiple fixes.
|
||||
- Message format: `fix(qa): ISSUE-NNN — short description`
|
||||
|
||||
### 8d. Re-test
|
||||
|
||||
- Navigate back to the affected page
|
||||
- Take **before/after screenshot pair**
|
||||
- Check console for errors
|
||||
- Use `snapshot -D` to verify the change had the expected effect
|
||||
|
||||
```bash
|
||||
$B goto <affected-url>
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-NNN-after.png"
|
||||
$B console --errors
|
||||
$B snapshot -D
|
||||
```
|
||||
|
||||
### 8e. Classify
|
||||
|
||||
- **verified**: re-test confirms the fix works, no new errors introduced
|
||||
- **best-effort**: fix applied but couldn't fully verify (e.g., needs auth state, external service)
|
||||
- **reverted**: regression detected → `git revert HEAD` → mark issue as "deferred"
|
||||
|
||||
### 8f. Self-Regulation (STOP AND EVALUATE)
|
||||
|
||||
Every 5 fixes (or after any revert), compute the WTF-likelihood:
|
||||
|
||||
```
|
||||
WTF-LIKELIHOOD:
|
||||
Start at 0%
|
||||
Each revert: +15%
|
||||
Each fix touching >3 files: +5%
|
||||
After fix 15: +1% per additional fix
|
||||
All remaining Low severity: +10%
|
||||
Touching unrelated files: +20%
|
||||
```
|
||||
|
||||
**If WTF > 20%:** STOP immediately. Show the user what you've done so far. Ask whether to continue.
|
||||
|
||||
**Hard cap: 50 fixes.** After 50 fixes, stop regardless of remaining issues.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Final QA
|
||||
|
||||
After all fixes are applied:
|
||||
|
||||
1. Re-run QA on all affected pages
|
||||
2. Compute final health score
|
||||
3. **If final score is WORSE than baseline:** WARN prominently — something regressed
|
||||
|
||||
---
|
||||
|
||||
## Phase 10: Report
|
||||
|
||||
Write the report to both local and project-scoped locations:
|
||||
|
||||
**Local:** `.gstack/qa-reports/qa-report-{domain}-{YYYY-MM-DD}.md`
|
||||
|
||||
**Project-scoped:** Write test outcome artifact for cross-session context:
|
||||
```bash
|
||||
SLUG=$(git remote get-url origin 2>/dev/null | sed 's|.*[:/]\([^/]*/[^/]*\)\.git$|\1|;s|.*[:/]\([^/]*/[^/]*\)$|\1|' | tr '/' '-')
|
||||
mkdir -p ~/.gstack/projects/$SLUG
|
||||
```
|
||||
Write to `~/.gstack/projects/{slug}/{user}-{branch}-test-outcome-{datetime}.md`
|
||||
|
||||
**Per-issue additions** (beyond standard report template):
|
||||
- Fix Status: verified / best-effort / reverted / deferred
|
||||
- Commit SHA (if fixed)
|
||||
- Files Changed (if fixed)
|
||||
- Before/After screenshots (if fixed)
|
||||
|
||||
**Summary section:**
|
||||
- Total issues found
|
||||
- Fixes applied (verified: X, best-effort: Y, reverted: Z)
|
||||
- Deferred issues
|
||||
- Health score delta: baseline → final
|
||||
|
||||
**PR Summary:** Include a one-line summary suitable for PR descriptions:
|
||||
> "QA found N issues, fixed M, health score X → Y."
|
||||
|
||||
---
|
||||
|
||||
## Phase 11: TODOS.md Update
|
||||
|
||||
If the repo has a `TODOS.md`:
|
||||
|
||||
1. **New deferred bugs** → add as TODOs with severity, category, and repro steps
|
||||
2. **Fixed bugs that were in TODOS.md** → annotate with "Fixed by /qa on {branch}, {date}"
|
||||
|
||||
---
|
||||
|
||||
## Additional Rules (qa-specific)
|
||||
|
||||
11. **Clean working tree required.** Refuse to start if `git status --porcelain` is non-empty.
|
||||
12. **One commit per fix.** Never bundle multiple fixes into one commit.
|
||||
13. **Never modify tests or CI configuration.** Only fix application source code.
|
||||
14. **Revert on regression.** If a fix makes things worse, `git revert HEAD` immediately.
|
||||
15. **Self-regulate.** Follow the WTF-likelihood heuristic. When in doubt, stop and ask.
|
||||
|
||||
Reference in New Issue
Block a user