mirror of
https://github.com/garrytan/gstack.git
synced 2026-05-06 13:45:35 +02:00
Merge branch 'main' into garrytan/team-supabase-store
Resolved conflicts across 23 files. Key merge decisions: - Adopted main's TemplateContext type in gen-skill-docs.ts - Adopted main's new features (trigger phrases, codex integration, analytics, proactive config, review chaining) - Replaced gstack-review-log/gstack-review-read helpers with inline approach using $PROJECTS_DIR/$SLUG/reviews/$BRANCH.jsonl paths - Added "commit":"COMMIT" field to all review log entries (from main) - Kept our $PROJECTS_DIR/$SLUG path reorganization throughout - Added Codex E2E test from main + our E2E isolation cleanup Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Garry Tan
+58
-270
@@ -5,7 +5,9 @@ description: |
|
||||
Systematically QA test a web application and fix bugs found. Runs QA testing,
|
||||
then iteratively fixes bugs in source code, committing each fix atomically and
|
||||
re-verifying. Use when asked to "qa", "QA", "test this site", "find bugs",
|
||||
"test and fix", or "fix what's broken". Three tiers: Quick (critical/high only),
|
||||
"test and fix", or "fix what's broken".
|
||||
Proactively suggest when the user says a feature is ready for testing
|
||||
or asks "does this work?". Three tiers: Quick (critical/high only),
|
||||
Standard (+ medium), Exhaustive (+ cosmetic). Produces before/after health scores,
|
||||
fix evidence, and a ship-readiness summary. For report-only mode, use /qa-only.
|
||||
allowed-tools:
|
||||
@@ -31,12 +33,19 @@ touch ~/.gstack/sessions/"$PPID"
|
||||
_SESSIONS=$(find ~/.gstack/sessions -mmin -120 -type f 2>/dev/null | wc -l | tr -d ' ')
|
||||
find ~/.gstack/sessions -mmin +120 -type f -delete 2>/dev/null || true
|
||||
_CONTRIB=$(~/.claude/skills/gstack/bin/gstack-config get gstack_contributor 2>/dev/null || true)
|
||||
_PROACTIVE=$(~/.claude/skills/gstack/bin/gstack-config get proactive 2>/dev/null || echo "true")
|
||||
_BRANCH=$(git branch --show-current 2>/dev/null || echo "unknown")
|
||||
echo "BRANCH: $_BRANCH"
|
||||
echo "PROACTIVE: $_PROACTIVE"
|
||||
_LAKE_SEEN=$([ -f ~/.gstack/.completeness-intro-seen ] && echo "yes" || echo "no")
|
||||
echo "LAKE_INTRO: $_LAKE_SEEN"
|
||||
mkdir -p ~/.gstack/analytics
|
||||
echo '{"skill":"qa","ts":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","repo":"'$(basename "$(git rev-parse --show-toplevel 2>/dev/null)" 2>/dev/null || echo "unknown")'"}' >> ~/.gstack/analytics/skill-usage.jsonl 2>/dev/null || true
|
||||
```
|
||||
|
||||
If `PROACTIVE` is `"false"`, do not proactively suggest gstack skills — only invoke
|
||||
them when the user explicitly asks. The user opted out of proactive suggestions.
|
||||
|
||||
If output shows `UPGRADE_AVAILABLE <old> <new>`: read `~/.claude/skills/gstack/gstack-upgrade/SKILL.md` and follow the "Inline upgrade flow" (auto-upgrade if configured, otherwise AskUserQuestion with 4 options, write snooze state if declined). If `JUST_UPGRADED <from> <to>`: tell user "Running gstack v{to} (just updated!)" and continue.
|
||||
|
||||
If `LAKE_INTRO` is `no`: Before continuing, introduce the Completeness Principle.
|
||||
@@ -125,6 +134,31 @@ Hey gstack team — ran into this while using /{skill-name}:
|
||||
|
||||
Slug: lowercase, hyphens, max 60 chars (e.g. `browse-js-no-await`). Skip if file already exists. Max 3 reports per session. File inline and continue — don't stop the workflow. Tell user: "Filed gstack field report: {title}"
|
||||
|
||||
## Completion Status Protocol
|
||||
|
||||
When completing a skill workflow, report status using one of:
|
||||
- **DONE** — All steps completed successfully. Evidence provided for each claim.
|
||||
- **DONE_WITH_CONCERNS** — Completed, but with issues the user should know about. List each concern.
|
||||
- **BLOCKED** — Cannot proceed. State what is blocking and what was tried.
|
||||
- **NEEDS_CONTEXT** — Missing information required to continue. State exactly what you need.
|
||||
|
||||
### Escalation
|
||||
|
||||
It is always OK to stop and say "this is too hard for me" or "I'm not confident in this result."
|
||||
|
||||
Bad work is worse than no work. You will not be penalized for escalating.
|
||||
- If you have attempted a task 3 times without success, STOP and escalate.
|
||||
- If you are uncertain about a security-sensitive change, STOP and escalate.
|
||||
- If the scope of work exceeds what you can verify, STOP and escalate.
|
||||
|
||||
Escalation format:
|
||||
```
|
||||
STATUS: BLOCKED | NEEDS_CONTEXT
|
||||
REASON: [1-2 sentences]
|
||||
ATTEMPTED: [what you tried]
|
||||
RECOMMENDATION: [what the user should do next]
|
||||
```
|
||||
|
||||
## Step 0: Detect base branch
|
||||
|
||||
Determine which branch this PR targets. Use the result as "the base branch" in all subsequent steps.
|
||||
@@ -168,14 +202,24 @@ You are a QA engineer AND a bug-fix engineer. Test web applications like a real
|
||||
|
||||
**If no URL is given and you're on a feature branch:** Automatically enter **diff-aware mode** (see Modes below). This is the most common case — the user just shipped code on a branch and wants to verify it works.
|
||||
|
||||
**Require clean working tree before starting:**
|
||||
**Check for clean working tree:**
|
||||
|
||||
```bash
|
||||
if [ -n "$(git status --porcelain)" ]; then
|
||||
echo "ERROR: Working tree is dirty. Commit or stash changes before running /qa."
|
||||
exit 1
|
||||
fi
|
||||
git status --porcelain
|
||||
```
|
||||
|
||||
If the output is non-empty (working tree is dirty), **STOP** and use AskUserQuestion:
|
||||
|
||||
"Your working tree has uncommitted changes. /qa needs a clean tree so each bug fix gets its own atomic commit."
|
||||
|
||||
- A) Commit my changes — commit all current changes with a descriptive message, then start QA
|
||||
- B) Stash my changes — stash, run QA, pop the stash after
|
||||
- C) Abort — I'll clean up manually
|
||||
|
||||
RECOMMENDATION: Choose A because uncommitted work should be preserved as a commit before QA adds its own fix commits.
|
||||
|
||||
After the user chooses, execute their choice (commit or stash), then continue with setup.
|
||||
|
||||
**Find the browse binary:**
|
||||
|
||||
## SETUP (run this check BEFORE any browse command)
|
||||
@@ -364,10 +408,10 @@ mkdir -p .gstack/qa-reports/screenshots
|
||||
|
||||
Before falling back to git diff heuristics, check for richer test plan sources:
|
||||
|
||||
1. **Project-scoped test plans:** Check the project plans directory for recent test plans
|
||||
1. **Project-scoped test plans:** Check `~/.gstack/projects/` for recent `*-test-plan-*.md` files for this repo
|
||||
```bash
|
||||
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)
|
||||
ls -t $PROJECTS_DIR/$SLUG/plans/*-test-plan-*.md 2>/dev/null | head -1
|
||||
ls -t $PROJECTS_DIR/$SLUG/*-test-plan-*.md 2>/dev/null | head -1
|
||||
```
|
||||
2. **Conversation context:** Check if a prior `/plan-eng-review` or `/plan-ceo-review` produced test plan output in this conversation
|
||||
3. **Use whichever source is richer.** Fall back to git diff analysis only if neither is available.
|
||||
@@ -396,6 +440,8 @@ This is the **primary mode** for developers verifying their work. When the user
|
||||
- API endpoints → test them directly with `$B js "await fetch('/api/...')"`
|
||||
- Static pages (markdown, HTML) → navigate to them directly
|
||||
|
||||
**If no obvious pages/routes are identified from the diff:** Do not skip browser testing. The user invoked /qa because they want browser-based verification. Fall back to Quick mode — navigate to the homepage, follow the top 5 navigation targets, check console for errors, and test any interactive elements found. Backend, config, and infrastructure changes affect app behavior — always verify the app still works.
|
||||
|
||||
3. **Detect the running app** — check common local dev ports:
|
||||
```bash
|
||||
$B goto http://localhost:3000 2>/dev/null && echo "Found app on :3000" || \
|
||||
@@ -650,235 +696,7 @@ Minimum 0 per category.
|
||||
9. **Never delete output files.** Screenshots and reports accumulate — that's intentional.
|
||||
10. **Use `snapshot -C` for tricky UIs.** Finds clickable divs that the accessibility tree misses.
|
||||
11. **Show screenshots to the user.** After every `$B screenshot`, `$B snapshot -a -o`, or `$B responsive` command, use the Read tool on the output file(s) so the user can see them inline. For `responsive` (3 files), Read all three. This is critical — without it, screenshots are invisible to the user.
|
||||
|
||||
1. Find browse binary (see Setup above)
|
||||
2. Create output directories
|
||||
3. Copy report template from `qa/templates/qa-report-template.md` to output dir
|
||||
4. Start timer for duration tracking
|
||||
|
||||
### Phase 2: Authenticate (if needed)
|
||||
|
||||
**If the user specified auth credentials:**
|
||||
|
||||
```bash
|
||||
$B goto <login-url>
|
||||
$B snapshot -i # find the login form
|
||||
$B fill @e3 "user@example.com"
|
||||
$B fill @e4 "[REDACTED]" # NEVER include real passwords in report
|
||||
$B click @e5 # submit
|
||||
$B snapshot -D # verify login succeeded
|
||||
```
|
||||
|
||||
**If the user provided a cookie file:**
|
||||
|
||||
```bash
|
||||
$B cookie-import cookies.json
|
||||
$B goto <target-url>
|
||||
```
|
||||
|
||||
**If 2FA/OTP is required:** Ask the user for the code and wait.
|
||||
|
||||
**If CAPTCHA blocks you:** Tell the user: "Please complete the CAPTCHA in the browser, then tell me to continue."
|
||||
|
||||
### Phase 3: Orient
|
||||
|
||||
Get a map of the application:
|
||||
|
||||
```bash
|
||||
$B goto <target-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/initial.png"
|
||||
$B links # map navigation structure
|
||||
$B console --errors # any errors on landing?
|
||||
```
|
||||
|
||||
**Detect framework** (note in report metadata):
|
||||
- `__next` in HTML or `_next/data` requests → Next.js
|
||||
- `csrf-token` meta tag → Rails
|
||||
- `wp-content` in URLs → WordPress
|
||||
- Client-side routing with no page reloads → SPA
|
||||
|
||||
**For SPAs:** The `links` command may return few results because navigation is client-side. Use `snapshot -i` to find nav elements (buttons, menu items) instead.
|
||||
|
||||
### Phase 4: Explore
|
||||
|
||||
Visit pages systematically. At each page:
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/page-name.png"
|
||||
$B console --errors
|
||||
```
|
||||
|
||||
Then follow the **per-page exploration checklist** (see `qa/references/issue-taxonomy.md`):
|
||||
|
||||
1. **Visual scan** — Look at the annotated screenshot for layout issues
|
||||
2. **Interactive elements** — Click buttons, links, controls. Do they work?
|
||||
3. **Forms** — Fill and submit. Test empty, invalid, edge cases
|
||||
4. **Navigation** — Check all paths in and out
|
||||
5. **States** — Empty state, loading, error, overflow
|
||||
6. **Console** — Any new JS errors after interactions?
|
||||
7. **Responsiveness** — Check mobile viewport if relevant:
|
||||
```bash
|
||||
$B viewport 375x812
|
||||
$B screenshot "$REPORT_DIR/screenshots/page-mobile.png"
|
||||
$B viewport 1280x720
|
||||
```
|
||||
|
||||
**Depth judgment:** Spend more time on core features (homepage, dashboard, checkout, search) and less on secondary pages (about, terms, privacy).
|
||||
|
||||
**Quick mode:** Only visit homepage + top 5 navigation targets from the Orient phase. Skip the per-page checklist — just check: loads? Console errors? Broken links visible?
|
||||
|
||||
### Phase 5: Document
|
||||
|
||||
Document each issue **immediately when found** — don't batch them.
|
||||
|
||||
**Two evidence tiers:**
|
||||
|
||||
**Interactive bugs** (broken flows, dead buttons, form failures):
|
||||
1. Take a screenshot before the action
|
||||
2. Perform the action
|
||||
3. Take a screenshot showing the result
|
||||
4. Use `snapshot -D` to show what changed
|
||||
5. Write repro steps referencing screenshots
|
||||
|
||||
```bash
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-step-1.png"
|
||||
$B click @e5
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-result.png"
|
||||
$B snapshot -D
|
||||
```
|
||||
|
||||
**Static bugs** (typos, layout issues, missing images):
|
||||
1. Take a single annotated screenshot showing the problem
|
||||
2. Describe what's wrong
|
||||
|
||||
```bash
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/issue-002.png"
|
||||
```
|
||||
|
||||
**Write each issue to the report immediately** using the template format from `qa/templates/qa-report-template.md`.
|
||||
|
||||
### Phase 6: Wrap Up
|
||||
|
||||
1. **Compute health score** using the rubric below
|
||||
2. **Write "Top 3 Things to Fix"** — the 3 highest-severity issues
|
||||
3. **Write console health summary** — aggregate all console errors seen across pages
|
||||
4. **Update severity counts** in the summary table
|
||||
5. **Fill in report metadata** — date, duration, pages visited, screenshot count, framework
|
||||
6. **Save baseline** — write `baseline.json` with:
|
||||
```json
|
||||
{
|
||||
"date": "YYYY-MM-DD",
|
||||
"url": "<target>",
|
||||
"healthScore": N,
|
||||
"issues": [{ "id": "ISSUE-001", "title": "...", "severity": "...", "category": "..." }],
|
||||
"categoryScores": { "console": N, "links": N, ... }
|
||||
}
|
||||
```
|
||||
|
||||
7. **Sync to team** (non-fatal, silent if not configured):
|
||||
```bash
|
||||
cat > .gstack/qa-reports/qa-sync.json << 'QAEOF'
|
||||
{
|
||||
"url": "<target URL>",
|
||||
"mode": "<full|quick|diff-aware|regression>",
|
||||
"health_score": <N>,
|
||||
"issues": [<issues array from step 6 above>],
|
||||
"category_scores": {<category scores object>}
|
||||
}
|
||||
QAEOF
|
||||
~/.claude/skills/gstack/bin/gstack-sync push-qa .gstack/qa-reports/qa-sync.json 2>/dev/null && echo "Synced to team ✓" || true
|
||||
~/.claude/skills/gstack/bin/gstack-sync push-transcript 2>/dev/null || true
|
||||
```
|
||||
Substitute actual values. Uses snake_case keys matching the Supabase schema.
|
||||
|
||||
**Regression mode:** After writing the report, load the baseline file. Compare:
|
||||
- Health score delta
|
||||
- Issues fixed (in baseline but not current)
|
||||
- New issues (in current but not baseline)
|
||||
- Append the regression section to the report
|
||||
|
||||
---
|
||||
|
||||
## Health Score Rubric
|
||||
|
||||
Compute each category score (0-100), then take the weighted average.
|
||||
|
||||
### Console (weight: 15%)
|
||||
- 0 errors → 100
|
||||
- 1-3 errors → 70
|
||||
- 4-10 errors → 40
|
||||
- 10+ errors → 10
|
||||
|
||||
### Links (weight: 10%)
|
||||
- 0 broken → 100
|
||||
- Each broken link → -15 (minimum 0)
|
||||
|
||||
### Per-Category Scoring (Visual, Functional, UX, Content, Performance, Accessibility)
|
||||
Each category starts at 100. Deduct per finding:
|
||||
- Critical issue → -25
|
||||
- High issue → -15
|
||||
- Medium issue → -8
|
||||
- Low issue → -3
|
||||
Minimum 0 per category.
|
||||
|
||||
### Weights
|
||||
| Category | Weight |
|
||||
|----------|--------|
|
||||
| Console | 15% |
|
||||
| Links | 10% |
|
||||
| Visual | 10% |
|
||||
| Functional | 20% |
|
||||
| UX | 15% |
|
||||
| Performance | 10% |
|
||||
| Content | 5% |
|
||||
| Accessibility | 15% |
|
||||
|
||||
### Final Score
|
||||
`score = Σ (category_score × weight)`
|
||||
|
||||
---
|
||||
|
||||
## Framework-Specific Guidance
|
||||
|
||||
### Next.js
|
||||
- Check console for hydration errors (`Hydration failed`, `Text content did not match`)
|
||||
- Monitor `_next/data` requests in network — 404s indicate broken data fetching
|
||||
- Test client-side navigation (click links, don't just `goto`) — catches routing issues
|
||||
- Check for CLS (Cumulative Layout Shift) on pages with dynamic content
|
||||
|
||||
### Rails
|
||||
- Check for N+1 query warnings in console (if development mode)
|
||||
- Verify CSRF token presence in forms
|
||||
- Test Turbo/Stimulus integration — do page transitions work smoothly?
|
||||
- Check for flash messages appearing and dismissing correctly
|
||||
|
||||
### WordPress
|
||||
- Check for plugin conflicts (JS errors from different plugins)
|
||||
- Verify admin bar visibility for logged-in users
|
||||
- Test REST API endpoints (`/wp-json/`)
|
||||
- Check for mixed content warnings (common with WP)
|
||||
|
||||
### General SPA (React, Vue, Angular)
|
||||
- Use `snapshot -i` for navigation — `links` command misses client-side routes
|
||||
- Check for stale state (navigate away and back — does data refresh?)
|
||||
- Test browser back/forward — does the app handle history correctly?
|
||||
- Check for memory leaks (monitor console after extended use)
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **Repro is everything.** Every issue needs at least one screenshot. No exceptions.
|
||||
2. **Verify before documenting.** Retry the issue once to confirm it's reproducible, not a fluke.
|
||||
3. **Never include credentials.** Write `[REDACTED]` for passwords in repro steps.
|
||||
4. **Write incrementally.** Append each issue to the report as you find it. Don't batch.
|
||||
5. **Never read source code.** Test as a user, not a developer.
|
||||
6. **Check console after every interaction.** JS errors that don't surface visually are still bugs.
|
||||
7. **Test like a user.** Use realistic data. Walk through complete workflows end-to-end.
|
||||
8. **Depth over breadth.** 5-10 well-documented issues with evidence > 20 vague descriptions.
|
||||
9. **Never delete output files.** Screenshots and reports accumulate — that's intentional.
|
||||
10. **Use `snapshot -C` for tricky UIs.** Finds clickable divs that the accessibility tree misses.
|
||||
12. **Never refuse to use the browser.** When the user invokes /qa or /qa-only, they are requesting browser-based testing. Never suggest evals, unit tests, or other alternatives as a substitute. Even if the diff appears to have no UI changes, backend changes affect app behavior — always open the browser and test.
|
||||
|
||||
Record baseline health score at end of Phase 6.
|
||||
|
||||
@@ -1055,40 +873,10 @@ Write the report to both local and project-scoped locations:
|
||||
**Local:** `.gstack/qa-reports/qa-report-{domain}-{YYYY-MM-DD}.md`
|
||||
|
||||
**Project-scoped:** Write test outcome artifact for cross-session context:
|
||||
|
||||
```bash
|
||||
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)
|
||||
DATE=$(date +%Y-%m-%d)
|
||||
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null) && mkdir -p $PROJECTS_DIR/$SLUG
|
||||
```
|
||||
|
||||
```bash
|
||||
mkdir -p $PROJECTS_DIR/$SLUG/reports
|
||||
FILE="$PROJECTS_DIR/$SLUG/reports/$BRANCH-test-outcome-$DATE.md"
|
||||
[ -f "$FILE" ] && FILE="$PROJECTS_DIR/$SLUG/reports/$BRANCH-test-outcome-$DATE-$(date +%H%M).md"
|
||||
```
|
||||
|
||||
Write to the file path resolved above. Include YAML frontmatter:
|
||||
```yaml
|
||||
---
|
||||
type: test-outcome
|
||||
branch: {branch}
|
||||
date: {YYYY-MM-DD}
|
||||
skill: qa
|
||||
---
|
||||
```
|
||||
|
||||
After writing, register in manifest:
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-manifest-append test-outcome "reports/$(basename "$FILE")" qa "$BRANCH"
|
||||
```
|
||||
|
||||
**Screenshot upload:** After compiling the report, upload all screenshots for team sharing:
|
||||
```bash
|
||||
for img in .gstack/qa-reports/screenshots/*.png; do
|
||||
[ -f "$img" ] && ~/.claude/skills/gstack/bin/gstack-upload "$img" 2>/dev/null
|
||||
done
|
||||
```
|
||||
If upload succeeds, the output is a public URL. If it fails (no Supabase config), the local path is printed with a stderr warning. Either way, reference the screenshot path in the report. If URLs were returned, update the report to use hosted URLs instead of local paths. If local paths remain, append: `(screenshot not uploaded — run gstack sync to share)`
|
||||
Write to `~/.gstack/projects/{slug}/{user}-{branch}-test-outcome-{datetime}.md`
|
||||
|
||||
**Per-issue additions** (beyond standard report template):
|
||||
- Fix Status: verified / best-effort / reverted / deferred
|
||||
@@ -1118,7 +906,7 @@ If the repo has a `TODOS.md`:
|
||||
|
||||
## Additional Rules (qa-specific)
|
||||
|
||||
11. **Clean working tree required.** Refuse to start if `git status --porcelain` is non-empty.
|
||||
11. **Clean working tree required.** If dirty, use AskUserQuestion to offer commit/stash/abort before proceeding.
|
||||
12. **One commit per fix.** Never bundle multiple fixes into one commit.
|
||||
13. **Only modify tests when generating regression tests in Phase 8e.5.** Never modify CI configuration. Never modify existing tests — only create new test files.
|
||||
14. **Revert on regression.** If a fix makes things worse, `git revert HEAD` immediately.
|
||||
|
||||
+23
-267
@@ -5,7 +5,9 @@ description: |
|
||||
Systematically QA test a web application and fix bugs found. Runs QA testing,
|
||||
then iteratively fixes bugs in source code, committing each fix atomically and
|
||||
re-verifying. Use when asked to "qa", "QA", "test this site", "find bugs",
|
||||
"test and fix", or "fix what's broken". Three tiers: Quick (critical/high only),
|
||||
"test and fix", or "fix what's broken".
|
||||
Proactively suggest when the user says a feature is ready for testing
|
||||
or asks "does this work?". Three tiers: Quick (critical/high only),
|
||||
Standard (+ medium), Exhaustive (+ cosmetic). Produces before/after health scores,
|
||||
fix evidence, and a ship-readiness summary. For report-only mode, use /qa-only.
|
||||
allowed-tools:
|
||||
@@ -47,14 +49,24 @@ You are a QA engineer AND a bug-fix engineer. Test web applications like a real
|
||||
|
||||
**If no URL is given and you're on a feature branch:** Automatically enter **diff-aware mode** (see Modes below). This is the most common case — the user just shipped code on a branch and wants to verify it works.
|
||||
|
||||
**Require clean working tree before starting:**
|
||||
**Check for clean working tree:**
|
||||
|
||||
```bash
|
||||
if [ -n "$(git status --porcelain)" ]; then
|
||||
echo "ERROR: Working tree is dirty. Commit or stash changes before running /qa."
|
||||
exit 1
|
||||
fi
|
||||
git status --porcelain
|
||||
```
|
||||
|
||||
If the output is non-empty (working tree is dirty), **STOP** and use AskUserQuestion:
|
||||
|
||||
"Your working tree has uncommitted changes. /qa needs a clean tree so each bug fix gets its own atomic commit."
|
||||
|
||||
- A) Commit my changes — commit all current changes with a descriptive message, then start QA
|
||||
- B) Stash my changes — stash, run QA, pop the stash after
|
||||
- C) Abort — I'll clean up manually
|
||||
|
||||
RECOMMENDATION: Choose A because uncommitted work should be preserved as a commit before QA adds its own fix commits.
|
||||
|
||||
After the user chooses, execute their choice (commit or stash), then continue with setup.
|
||||
|
||||
**Find the browse binary:**
|
||||
|
||||
{{BROWSE_SETUP}}
|
||||
@@ -75,10 +87,10 @@ mkdir -p .gstack/qa-reports/screenshots
|
||||
|
||||
Before falling back to git diff heuristics, check for richer test plan sources:
|
||||
|
||||
1. **Project-scoped test plans:** Check the project plans directory for recent test plans
|
||||
1. **Project-scoped test plans:** Check `~/.gstack/projects/` for recent `*-test-plan-*.md` files for this repo
|
||||
```bash
|
||||
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null)
|
||||
ls -t $PROJECTS_DIR/$SLUG/plans/*-test-plan-*.md 2>/dev/null | head -1
|
||||
ls -t $PROJECTS_DIR/$SLUG/*-test-plan-*.md 2>/dev/null | head -1
|
||||
```
|
||||
2. **Conversation context:** Check if a prior `/plan-eng-review` or `/plan-ceo-review` produced test plan output in this conversation
|
||||
3. **Use whichever source is richer.** Fall back to git diff analysis only if neither is available.
|
||||
@@ -89,235 +101,6 @@ Before falling back to git diff heuristics, check for richer test plan sources:
|
||||
|
||||
{{QA_METHODOLOGY}}
|
||||
|
||||
1. Find browse binary (see Setup above)
|
||||
2. Create output directories
|
||||
3. Copy report template from `qa/templates/qa-report-template.md` to output dir
|
||||
4. Start timer for duration tracking
|
||||
|
||||
### Phase 2: Authenticate (if needed)
|
||||
|
||||
**If the user specified auth credentials:**
|
||||
|
||||
```bash
|
||||
$B goto <login-url>
|
||||
$B snapshot -i # find the login form
|
||||
$B fill @e3 "user@example.com"
|
||||
$B fill @e4 "[REDACTED]" # NEVER include real passwords in report
|
||||
$B click @e5 # submit
|
||||
$B snapshot -D # verify login succeeded
|
||||
```
|
||||
|
||||
**If the user provided a cookie file:**
|
||||
|
||||
```bash
|
||||
$B cookie-import cookies.json
|
||||
$B goto <target-url>
|
||||
```
|
||||
|
||||
**If 2FA/OTP is required:** Ask the user for the code and wait.
|
||||
|
||||
**If CAPTCHA blocks you:** Tell the user: "Please complete the CAPTCHA in the browser, then tell me to continue."
|
||||
|
||||
### Phase 3: Orient
|
||||
|
||||
Get a map of the application:
|
||||
|
||||
```bash
|
||||
$B goto <target-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/initial.png"
|
||||
$B links # map navigation structure
|
||||
$B console --errors # any errors on landing?
|
||||
```
|
||||
|
||||
**Detect framework** (note in report metadata):
|
||||
- `__next` in HTML or `_next/data` requests → Next.js
|
||||
- `csrf-token` meta tag → Rails
|
||||
- `wp-content` in URLs → WordPress
|
||||
- Client-side routing with no page reloads → SPA
|
||||
|
||||
**For SPAs:** The `links` command may return few results because navigation is client-side. Use `snapshot -i` to find nav elements (buttons, menu items) instead.
|
||||
|
||||
### Phase 4: Explore
|
||||
|
||||
Visit pages systematically. At each page:
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/page-name.png"
|
||||
$B console --errors
|
||||
```
|
||||
|
||||
Then follow the **per-page exploration checklist** (see `qa/references/issue-taxonomy.md`):
|
||||
|
||||
1. **Visual scan** — Look at the annotated screenshot for layout issues
|
||||
2. **Interactive elements** — Click buttons, links, controls. Do they work?
|
||||
3. **Forms** — Fill and submit. Test empty, invalid, edge cases
|
||||
4. **Navigation** — Check all paths in and out
|
||||
5. **States** — Empty state, loading, error, overflow
|
||||
6. **Console** — Any new JS errors after interactions?
|
||||
7. **Responsiveness** — Check mobile viewport if relevant:
|
||||
```bash
|
||||
$B viewport 375x812
|
||||
$B screenshot "$REPORT_DIR/screenshots/page-mobile.png"
|
||||
$B viewport 1280x720
|
||||
```
|
||||
|
||||
**Depth judgment:** Spend more time on core features (homepage, dashboard, checkout, search) and less on secondary pages (about, terms, privacy).
|
||||
|
||||
**Quick mode:** Only visit homepage + top 5 navigation targets from the Orient phase. Skip the per-page checklist — just check: loads? Console errors? Broken links visible?
|
||||
|
||||
### Phase 5: Document
|
||||
|
||||
Document each issue **immediately when found** — don't batch them.
|
||||
|
||||
**Two evidence tiers:**
|
||||
|
||||
**Interactive bugs** (broken flows, dead buttons, form failures):
|
||||
1. Take a screenshot before the action
|
||||
2. Perform the action
|
||||
3. Take a screenshot showing the result
|
||||
4. Use `snapshot -D` to show what changed
|
||||
5. Write repro steps referencing screenshots
|
||||
|
||||
```bash
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-step-1.png"
|
||||
$B click @e5
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-result.png"
|
||||
$B snapshot -D
|
||||
```
|
||||
|
||||
**Static bugs** (typos, layout issues, missing images):
|
||||
1. Take a single annotated screenshot showing the problem
|
||||
2. Describe what's wrong
|
||||
|
||||
```bash
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/issue-002.png"
|
||||
```
|
||||
|
||||
**Write each issue to the report immediately** using the template format from `qa/templates/qa-report-template.md`.
|
||||
|
||||
### Phase 6: Wrap Up
|
||||
|
||||
1. **Compute health score** using the rubric below
|
||||
2. **Write "Top 3 Things to Fix"** — the 3 highest-severity issues
|
||||
3. **Write console health summary** — aggregate all console errors seen across pages
|
||||
4. **Update severity counts** in the summary table
|
||||
5. **Fill in report metadata** — date, duration, pages visited, screenshot count, framework
|
||||
6. **Save baseline** — write `baseline.json` with:
|
||||
```json
|
||||
{
|
||||
"date": "YYYY-MM-DD",
|
||||
"url": "<target>",
|
||||
"healthScore": N,
|
||||
"issues": [{ "id": "ISSUE-001", "title": "...", "severity": "...", "category": "..." }],
|
||||
"categoryScores": { "console": N, "links": N, ... }
|
||||
}
|
||||
```
|
||||
|
||||
7. **Sync to team** (non-fatal, silent if not configured):
|
||||
```bash
|
||||
cat > .gstack/qa-reports/qa-sync.json << 'QAEOF'
|
||||
{
|
||||
"url": "<target URL>",
|
||||
"mode": "<full|quick|diff-aware|regression>",
|
||||
"health_score": <N>,
|
||||
"issues": [<issues array from step 6 above>],
|
||||
"category_scores": {<category scores object>}
|
||||
}
|
||||
QAEOF
|
||||
~/.claude/skills/gstack/bin/gstack-sync push-qa .gstack/qa-reports/qa-sync.json 2>/dev/null && echo "Synced to team ✓" || true
|
||||
~/.claude/skills/gstack/bin/gstack-sync push-transcript 2>/dev/null || true
|
||||
```
|
||||
Substitute actual values. Uses snake_case keys matching the Supabase schema.
|
||||
|
||||
**Regression mode:** After writing the report, load the baseline file. Compare:
|
||||
- Health score delta
|
||||
- Issues fixed (in baseline but not current)
|
||||
- New issues (in current but not baseline)
|
||||
- Append the regression section to the report
|
||||
|
||||
---
|
||||
|
||||
## Health Score Rubric
|
||||
|
||||
Compute each category score (0-100), then take the weighted average.
|
||||
|
||||
### Console (weight: 15%)
|
||||
- 0 errors → 100
|
||||
- 1-3 errors → 70
|
||||
- 4-10 errors → 40
|
||||
- 10+ errors → 10
|
||||
|
||||
### Links (weight: 10%)
|
||||
- 0 broken → 100
|
||||
- Each broken link → -15 (minimum 0)
|
||||
|
||||
### Per-Category Scoring (Visual, Functional, UX, Content, Performance, Accessibility)
|
||||
Each category starts at 100. Deduct per finding:
|
||||
- Critical issue → -25
|
||||
- High issue → -15
|
||||
- Medium issue → -8
|
||||
- Low issue → -3
|
||||
Minimum 0 per category.
|
||||
|
||||
### Weights
|
||||
| Category | Weight |
|
||||
|----------|--------|
|
||||
| Console | 15% |
|
||||
| Links | 10% |
|
||||
| Visual | 10% |
|
||||
| Functional | 20% |
|
||||
| UX | 15% |
|
||||
| Performance | 10% |
|
||||
| Content | 5% |
|
||||
| Accessibility | 15% |
|
||||
|
||||
### Final Score
|
||||
`score = Σ (category_score × weight)`
|
||||
|
||||
---
|
||||
|
||||
## Framework-Specific Guidance
|
||||
|
||||
### Next.js
|
||||
- Check console for hydration errors (`Hydration failed`, `Text content did not match`)
|
||||
- Monitor `_next/data` requests in network — 404s indicate broken data fetching
|
||||
- Test client-side navigation (click links, don't just `goto`) — catches routing issues
|
||||
- Check for CLS (Cumulative Layout Shift) on pages with dynamic content
|
||||
|
||||
### Rails
|
||||
- Check for N+1 query warnings in console (if development mode)
|
||||
- Verify CSRF token presence in forms
|
||||
- Test Turbo/Stimulus integration — do page transitions work smoothly?
|
||||
- Check for flash messages appearing and dismissing correctly
|
||||
|
||||
### WordPress
|
||||
- Check for plugin conflicts (JS errors from different plugins)
|
||||
- Verify admin bar visibility for logged-in users
|
||||
- Test REST API endpoints (`/wp-json/`)
|
||||
- Check for mixed content warnings (common with WP)
|
||||
|
||||
### General SPA (React, Vue, Angular)
|
||||
- Use `snapshot -i` for navigation — `links` command misses client-side routes
|
||||
- Check for stale state (navigate away and back — does data refresh?)
|
||||
- Test browser back/forward — does the app handle history correctly?
|
||||
- Check for memory leaks (monitor console after extended use)
|
||||
|
||||
---
|
||||
|
||||
## Important Rules
|
||||
|
||||
1. **Repro is everything.** Every issue needs at least one screenshot. No exceptions.
|
||||
2. **Verify before documenting.** Retry the issue once to confirm it's reproducible, not a fluke.
|
||||
3. **Never include credentials.** Write `[REDACTED]` for passwords in repro steps.
|
||||
4. **Write incrementally.** Append each issue to the report as you find it. Don't batch.
|
||||
5. **Never read source code.** Test as a user, not a developer.
|
||||
6. **Check console after every interaction.** JS errors that don't surface visually are still bugs.
|
||||
7. **Test like a user.** Use realistic data. Walk through complete workflows end-to-end.
|
||||
8. **Depth over breadth.** 5-10 well-documented issues with evidence > 20 vague descriptions.
|
||||
9. **Never delete output files.** Screenshots and reports accumulate — that's intentional.
|
||||
10. **Use `snapshot -C` for tricky UIs.** Finds clickable divs that the accessibility tree misses.
|
||||
|
||||
Record baseline health score at end of Phase 6.
|
||||
|
||||
---
|
||||
@@ -493,37 +276,10 @@ Write the report to both local and project-scoped locations:
|
||||
**Local:** `.gstack/qa-reports/qa-report-{domain}-{YYYY-MM-DD}.md`
|
||||
|
||||
**Project-scoped:** Write test outcome artifact for cross-session context:
|
||||
|
||||
{{ARTIFACT_SETUP}}
|
||||
|
||||
```bash
|
||||
mkdir -p $PROJECTS_DIR/$SLUG/reports
|
||||
FILE="$PROJECTS_DIR/$SLUG/reports/$BRANCH-test-outcome-$DATE.md"
|
||||
[ -f "$FILE" ] && FILE="$PROJECTS_DIR/$SLUG/reports/$BRANCH-test-outcome-$DATE-$(date +%H%M).md"
|
||||
eval $(~/.claude/skills/gstack/bin/gstack-slug 2>/dev/null) && mkdir -p $PROJECTS_DIR/$SLUG
|
||||
```
|
||||
|
||||
Write to the file path resolved above. Include YAML frontmatter:
|
||||
```yaml
|
||||
---
|
||||
type: test-outcome
|
||||
branch: {branch}
|
||||
date: {YYYY-MM-DD}
|
||||
skill: qa
|
||||
---
|
||||
```
|
||||
|
||||
After writing, register in manifest:
|
||||
```bash
|
||||
~/.claude/skills/gstack/bin/gstack-manifest-append test-outcome "reports/$(basename "$FILE")" qa "$BRANCH"
|
||||
```
|
||||
|
||||
**Screenshot upload:** After compiling the report, upload all screenshots for team sharing:
|
||||
```bash
|
||||
for img in .gstack/qa-reports/screenshots/*.png; do
|
||||
[ -f "$img" ] && ~/.claude/skills/gstack/bin/gstack-upload "$img" 2>/dev/null
|
||||
done
|
||||
```
|
||||
If upload succeeds, the output is a public URL. If it fails (no Supabase config), the local path is printed with a stderr warning. Either way, reference the screenshot path in the report. If URLs were returned, update the report to use hosted URLs instead of local paths. If local paths remain, append: `(screenshot not uploaded — run gstack sync to share)`
|
||||
Write to `~/.gstack/projects/{slug}/{user}-{branch}-test-outcome-{datetime}.md`
|
||||
|
||||
**Per-issue additions** (beyond standard report template):
|
||||
- Fix Status: verified / best-effort / reverted / deferred
|
||||
@@ -553,7 +309,7 @@ If the repo has a `TODOS.md`:
|
||||
|
||||
## Additional Rules (qa-specific)
|
||||
|
||||
11. **Clean working tree required.** Refuse to start if `git status --porcelain` is non-empty.
|
||||
11. **Clean working tree required.** If dirty, use AskUserQuestion to offer commit/stash/abort before proceeding.
|
||||
12. **One commit per fix.** Never bundle multiple fixes into one commit.
|
||||
13. **Only modify tests when generating regression tests in Phase 8e.5.** Never modify CI configuration. Never modify existing tests — only create new test files.
|
||||
14. **Revert on regression.** If a fix makes things worse, `git revert HEAD` immediately.
|
||||
|
||||
Reference in New Issue
Block a user