mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-11 23:49:01 +02:00
refactor(qa): /qa and /qa-only drive Aside, fall back to $B
QA_METHODOLOGY runs every phase as Aside scripts (orient, explore, document, re-test, mobile viewport via CDP emulation, links via HEAD fetch); the authenticate phase is 'you are already signed in'; a 13th rule requires consent before mutating actions on non-local targets; the fallback section translates each step onto $B. The qa E2E tests run on whichever engine is present. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
@@ -117,25 +117,32 @@ This is the **primary mode** for developers verifying their work. When the user
|
||||
- View/template/component files → which pages render them
|
||||
- Model/service files → which pages use those models (check controllers that reference them)
|
||||
- CSS/style files → which pages include those stylesheets
|
||||
- API endpoints → test them directly with \`$B js "await fetch('/api/...')"\`
|
||||
- API endpoints → call them with the session's own cookies from one \`aside repl\` script:
|
||||
\`\`\`bash
|
||||
aside repl '
|
||||
const pg = await openTab("<base-url>");
|
||||
const r = await fetch("<base-url>/api/...", { method: "GET" });
|
||||
console.log("API_STATUS=" + r.status);
|
||||
console.log("API_BODY_START"); console.log((await r.text()).slice(0, 4000)); console.log("API_BODY_END");
|
||||
await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
- Static pages (markdown, HTML) → navigate to them directly
|
||||
|
||||
**If no obvious pages/routes are identified from the diff:** Do not skip browser testing. The user invoked /qa because they want browser-based verification. Fall back to Quick mode — navigate to the homepage, follow the top 5 navigation targets, check console for errors, and test any interactive elements found. Backend, config, and infrastructure changes affect app behavior — always verify the app still works.
|
||||
|
||||
3. **Detect the running app** — check common local dev ports:
|
||||
3. **Detect the running app** — probe common local dev ports (no browser needed to find a port):
|
||||
\`\`\`bash
|
||||
$B goto http://localhost:3000 2>/dev/null && echo "Found app on :3000" || \\
|
||||
$B goto http://localhost:4000 2>/dev/null && echo "Found app on :4000" || \\
|
||||
$B goto http://localhost:8080 2>/dev/null && echo "Found app on :8080"
|
||||
for p in 3000 4000 8080; do curl -sI --max-time 3 "http://localhost:$p" >/dev/null 2>&1 && echo "Found app on :$p"; done
|
||||
\`\`\`
|
||||
If no local app is found, check for a staging/preview URL in the PR or environment. If nothing works, ask the user for the URL.
|
||||
Open the first URL that answers in Aside. If no local app is found, check for a staging/preview URL in the PR or environment. If nothing works, ask the user for the URL.
|
||||
|
||||
4. **Test each affected page/route:**
|
||||
- Navigate to the page
|
||||
- Navigate to the page (the Read-a-page script in Phase 3)
|
||||
- Take a screenshot
|
||||
- Check console for errors
|
||||
- Check console for errors (the \`CONSOLE_ERRORS=\` line)
|
||||
- If the change was interactive (forms, buttons, flows), test the interaction end-to-end
|
||||
- Use \`snapshot -D\` before and after actions to verify the change had the expected effect
|
||||
- Snapshot before acting and print the diff after (the Drive-a-flow script in Phase 5) to verify the change had the expected effect
|
||||
|
||||
5. **Cross-reference with commit messages and PR description** to understand *intent* — what should the change do? Verify it actually does that.
|
||||
|
||||
@@ -163,77 +170,87 @@ Run full mode, then load \`baseline.json\` from a previous run. Diff: which issu
|
||||
|
||||
### Phase 1: Initialize
|
||||
|
||||
1. Find browse binary (see Setup above)
|
||||
1. Confirm Aside is READY (see BROWSER SETUP above). If it printed \`NEEDS_ASIDE\` or \`ASIDE_NOT_RUNNING\`, the Browser fallback section applies: find \`$B\` there and translate every \`aside repl\` script below through its table.
|
||||
2. Create output directories
|
||||
3. Copy report template from \`qa/templates/qa-report-template.md\` to output dir
|
||||
4. Start timer for duration tracking
|
||||
|
||||
### Phase 2: Authenticate (if needed)
|
||||
|
||||
**If the user specified auth credentials:**
|
||||
Aside is the user's real browser, so the session is already signed in wherever the user is signed in. You never authenticate — the user does. In the fallback browser there is no session to inherit: import one with /setup-browser-cookies, or \`$B handoff\` for a human sign-in and \`$B resume\` when they're done.
|
||||
|
||||
\`\`\`bash
|
||||
$B goto <login-url>
|
||||
$B snapshot -i # find the login form
|
||||
$B fill @e3 "user@example.com"
|
||||
$B fill @e4 "[REDACTED]" # NEVER include real passwords in report
|
||||
$B click @e5 # submit
|
||||
$B snapshot -D # verify login succeeded
|
||||
\`\`\`
|
||||
**If a sign-in wall appears:** stop and tell the user: "Sign in to <origin> in Aside yourself (open it in a new Aside tab), then tell me you're done." Then re-run the step — the browser's cookies now apply. Never type passwords, one-time codes, or payment details, and never read or print cookies, tokens, or localStorage.
|
||||
|
||||
**If the user provided a cookie file:**
|
||||
**If 2FA/OTP is required:** The user completes it in the Aside window, then tells you to continue.
|
||||
|
||||
\`\`\`bash
|
||||
$B cookie-import cookies.json
|
||||
$B goto <target-url>
|
||||
\`\`\`
|
||||
|
||||
**If 2FA/OTP is required:** Ask the user for the code and wait.
|
||||
|
||||
**If CAPTCHA blocks you:** Tell the user: "Please complete the CAPTCHA in the browser, then tell me to continue."
|
||||
**If CAPTCHA blocks you:** Tell the user: "Please complete the CAPTCHA in Aside, then tell me to continue."
|
||||
|
||||
### Phase 3: Orient
|
||||
|
||||
Get a map of the application:
|
||||
Get a map of the application. One script reads the landing page — console errors from load, the interactive snapshot tree, the visible text, and a screenshot:
|
||||
|
||||
\`\`\`bash
|
||||
$B goto <target-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/initial.png"
|
||||
$B links # map navigation structure
|
||||
$B console --errors # any errors on landing?
|
||||
aside repl '
|
||||
const HOOK = \`(() => { window.__gstackErrs = window.__gstackErrs || []; const oe = console.error; console.error = (...a) => { window.__gstackErrs.push(a.map(String).join(" ")); oe.apply(console, a); }; window.addEventListener("error", e => window.__gstackErrs.push("uncaught: " + e.message)); window.addEventListener("unhandledrejection", e => window.__gstackErrs.push("unhandledrejection: " + (e.reason && e.reason.message || e.reason))); })()\`;
|
||||
const pg = await openTab("about:blank");
|
||||
await pg._sendToTarget("Page.addScriptToEvaluateOnNewDocument", { source: HOOK });
|
||||
await pg.goto("<target-url>");
|
||||
const s = await snapshot(pg, { interactive: true });
|
||||
console.log(s.tree);
|
||||
console.log("CONSOLE_ERRORS=" + JSON.stringify(await pg.evaluate(() => window.__gstackErrs)));
|
||||
console.log("TEXT_START"); console.log((await pg.evaluate(() => document.body.innerText)).slice(0, 20000)); console.log("TEXT_END");
|
||||
await pg.screenshot({ path: "initial.jpg", type: "jpeg", quality: 60, fullPage: true });
|
||||
console.log("ASIDE_DIR=" + pwd);
|
||||
await closeTab(pg);
|
||||
console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
|
||||
Then copy the screenshot out of the printed directory and show it: \`cp "<ASIDE_DIR>/initial.jpg" "$REPORT_DIR/screenshots/initial.jpg"\`, then Read it.
|
||||
|
||||
Map the navigation structure with the links script (same-origin, read-only HEAD requests with the user's cookies):
|
||||
|
||||
\`\`\`bash
|
||||
aside repl '
|
||||
const pg = await openTab("<target-url>");
|
||||
const links = await pg.evaluate(() => [...new Set([...document.querySelectorAll("a[href]")].map(a => a.href))].filter(h => h.startsWith(location.origin) && !/logout|signout|delete|remove|cancel|unsubscribe/i.test(h)));
|
||||
for (const l of links) { const r = await fetch(l, { method: "HEAD" }).catch(e => ({ status: "ERR " + e.message })); console.log("LINK", r.status, l); }
|
||||
await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
|
||||
Every \`LINK\` line with a 4xx/5xx or \`ERR\` status is a broken link for the Links score.
|
||||
|
||||
**Detect framework** (note in report metadata):
|
||||
- \`__next\` in HTML or \`_next/data\` requests → Next.js
|
||||
- \`csrf-token\` meta tag → Rails
|
||||
- \`wp-content\` in URLs → WordPress
|
||||
- Client-side routing with no page reloads → SPA
|
||||
|
||||
**For SPAs:** The \`links\` command may return few results because navigation is client-side. Use \`snapshot -i\` to find nav elements (buttons, menu items) instead.
|
||||
**For SPAs:** The links script may return few results because navigation is client-side. Use \`snapshot(pg, { interactive: true })\` to find nav elements (buttons, menu items) instead.
|
||||
|
||||
### Phase 4: Explore
|
||||
|
||||
Visit pages systematically. At each page:
|
||||
|
||||
\`\`\`bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/page-name.png"
|
||||
$B console --errors
|
||||
\`\`\`
|
||||
Visit pages systematically. At each page, run the Read-a-page script from Phase 3 against the page URL with \`page-<name>.jpg\` as the screenshot path, copy it into \`$REPORT_DIR/screenshots/\`, and Read it.
|
||||
|
||||
Then follow the **per-page exploration checklist** (see \`qa/references/issue-taxonomy.md\`):
|
||||
|
||||
1. **Visual scan** — Look at the annotated screenshot for layout issues
|
||||
1. **Visual scan** — Look at the screenshot for layout issues (use the annotated-screenshot script when you need ref labels on the page)
|
||||
2. **Interactive elements** — Click buttons, links, controls. Do they work?
|
||||
3. **Forms** — Fill and submit. Test empty, invalid, edge cases
|
||||
4. **Navigation** — Check all paths in and out
|
||||
5. **States** — Empty state, loading, error, overflow
|
||||
6. **Console** — Any new JS errors after interactions?
|
||||
7. **Responsiveness** — Check mobile viewport if relevant:
|
||||
6. **Console** — Any new JS errors after interactions? Print \`CONSOLE_ERRORS=\` after every action
|
||||
7. **Responsiveness** — Check the mobile viewport if relevant:
|
||||
\`\`\`bash
|
||||
$B viewport 375x812
|
||||
$B screenshot "$REPORT_DIR/screenshots/page-mobile.png"
|
||||
$B viewport 1280x720
|
||||
aside repl '
|
||||
const pg = await openTab("<page-url>");
|
||||
await pg._sendToTarget("Emulation.setDeviceMetricsOverride", { width: 375, height: 812, deviceScaleFactor: 2, mobile: true });
|
||||
await sleep(300);
|
||||
await pg.screenshot({ path: "page-mobile.jpg", type: "jpeg", quality: 60, fullPage: true });
|
||||
await pg._sendToTarget("Emulation.clearDeviceMetricsOverride", {});
|
||||
console.log("ASIDE_DIR=" + pwd); await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
|
||||
**Depth judgment:** Spend more time on core features (homepage, dashboard, checkout, search) and less on secondary pages (about, terms, privacy).
|
||||
@@ -246,26 +263,47 @@ Document each issue **immediately when found** — don't batch them.
|
||||
|
||||
**Two evidence tiers:**
|
||||
|
||||
**Interactive bugs** (broken flows, dead buttons, form failures):
|
||||
**Interactive bugs** (broken flows, dead buttons, form failures) — one script per flow, because tabs close when the script ends:
|
||||
1. Take a screenshot before the action
|
||||
2. Perform the action
|
||||
3. Take a screenshot showing the result
|
||||
4. Use \`snapshot -D\` to show what changed
|
||||
4. Print the snapshot diff to show what changed
|
||||
5. Write repro steps referencing screenshots
|
||||
|
||||
\`\`\`bash
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-step-1.png"
|
||||
$B click @e5
|
||||
$B screenshot "$REPORT_DIR/screenshots/issue-001-result.png"
|
||||
$B snapshot -D
|
||||
aside repl '
|
||||
const HOOK = \`(() => { window.__gstackErrs = window.__gstackErrs || []; const oe = console.error; console.error = (...a) => { window.__gstackErrs.push(a.map(String).join(" ")); oe.apply(console, a); }; window.addEventListener("error", e => window.__gstackErrs.push("uncaught: " + e.message)); })()\`;
|
||||
const pg = await openTab("about:blank");
|
||||
await pg._sendToTarget("Page.addScriptToEvaluateOnNewDocument", { source: HOOK });
|
||||
await pg.goto("<page-url>");
|
||||
await snapshot(pg, { interactive: true }); // baseline for .diff; refs like e12 name the elements
|
||||
await pg.screenshot({ path: "issue-001-step-1.jpg", type: "jpeg", quality: 60 });
|
||||
await pg.locator("e12").click(); // or pg.fill("#email", "qa@example.com"), pg.getByRole("button", { name: "Save" }).click()
|
||||
await sleep(500); // or await pg.waitForSelector("#done"); await pg.waitForURL(/dashboard/)
|
||||
const s = await snapshot(pg);
|
||||
console.log("DIFF_START"); console.log(s.diff); console.log("DIFF_END");
|
||||
console.log("URL=" + pg.url());
|
||||
console.log("CONSOLE_ERRORS=" + JSON.stringify(await pg.evaluate(() => window.__gstackErrs)));
|
||||
await pg.screenshot({ path: "issue-001-result.jpg", type: "jpeg", quality: 60 });
|
||||
console.log("ASIDE_DIR=" + pwd);
|
||||
await closeTab(pg);
|
||||
console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
|
||||
Copy both screenshots out of the printed \`ASIDE_DIR\` into \`$REPORT_DIR/screenshots/\` and Read them.
|
||||
|
||||
**Static bugs** (typos, layout issues, missing images):
|
||||
1. Take a single annotated screenshot showing the problem
|
||||
2. Describe what's wrong
|
||||
|
||||
\`\`\`bash
|
||||
$B snapshot -i -a -o "$REPORT_DIR/screenshots/issue-002.png"
|
||||
aside repl '
|
||||
const pg = await openTab("<page-url>");
|
||||
const a = await annotatedScreenshot(pg);
|
||||
await fs.writeFile(path.join(pwd, "issue-002.png"), Buffer.from(a.base64Image, "base64"));
|
||||
console.log("ASIDE_DIR=" + pwd); await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
\`\`\`
|
||||
|
||||
**Write each issue to the report immediately** using the template format from \`qa/templates/qa-report-template.md\`.
|
||||
@@ -356,7 +394,7 @@ Minimum 0 per category.
|
||||
- Check for mixed content warnings (common with WP)
|
||||
|
||||
### General SPA (React, Vue, Angular)
|
||||
- Use \`snapshot -i\` for navigation — \`links\` command misses client-side routes
|
||||
- Use \`snapshot(pg, { interactive: true })\` for navigation — the links script misses client-side routes
|
||||
- Check for stale state (navigate away and back — does data refresh?)
|
||||
- Test browser back/forward — does the app handle history correctly?
|
||||
- Check for memory leaks (monitor console after extended use)
|
||||
@@ -367,16 +405,17 @@ Minimum 0 per category.
|
||||
|
||||
1. **Repro is everything.** Every issue needs at least one screenshot. No exceptions.
|
||||
2. **Verify before documenting.** Retry the issue once to confirm it's reproducible, not a fluke.
|
||||
3. **Never include credentials.** Write \`[REDACTED]\` for passwords in repro steps.
|
||||
3. **Never include credentials.** You never type them — the user signs in inside Aside. Write \`[REDACTED]\` if a repro step has to mention one.
|
||||
4. **Write incrementally.** Append each issue to the report as you find it. Don't batch.
|
||||
5. **Never read source code.** Test as a user, not a developer.
|
||||
6. **Check console after every interaction.** JS errors that don't surface visually are still bugs.
|
||||
7. **Test like a user.** Use realistic data. Walk through complete workflows end-to-end.
|
||||
8. **Depth over breadth.** 5-10 well-documented issues with evidence > 20 vague descriptions.
|
||||
9. **Never delete output files.** Screenshots and reports accumulate — that's intentional.
|
||||
10. **Use \`snapshot -C\` for tricky UIs.** Finds clickable divs that the accessibility tree misses.
|
||||
11. **Show screenshots to the user.** After every \`$B screenshot\`, \`$B snapshot -a -o\`, or \`$B responsive\` command, use the Read tool on the output file(s) so the user can see them inline. For \`responsive\` (3 files), Read all three. This is critical — without it, screenshots are invisible to the user.
|
||||
12. **Never refuse to use the browser.** When the user invokes /qa or /qa-only, they are requesting browser-based testing. Never suggest evals, unit tests, or other alternatives as a substitute. Even if the diff appears to have no UI changes, backend changes affect app behavior — always open the browser and test.`;
|
||||
10. **Use \`annotatedScreenshot(pg)\` when the tree misses a clickable element.** Ref labels drawn on the page find clickable divs the accessibility tree skips; then click by ref or CSS selector.
|
||||
11. **Show screenshots to the user.** After every script that saves a screenshot, \`cp\` it out of the printed \`ASIDE_DIR\` into \`$REPORT_DIR/screenshots/\` and use the Read tool on the copied file so the user can see it inline. This is critical — without it, screenshots are invisible to the user.
|
||||
12. **Never refuse to use the browser.** When the user invokes /qa or /qa-only, they are requesting browser-based testing in Aside. Never suggest evals, unit tests, curl, or other alternatives as a substitute. Even if the diff appears to have no UI changes, backend changes affect app behavior — always open the app in the browser and test.
|
||||
13. **Mutating actions on a non-local target need consent.** Submitting, creating, deleting, purchasing, or changing settings on anything that is not LOCAL follows the "Invocation is consent to LOOK, not to ACT" rule in BROWSER SETUP — one AskUserQuestion per run, before the first such action.`;
|
||||
}
|
||||
|
||||
export function generateCoAuthorTrailer(ctx: TemplateContext): string {
|
||||
|
||||
Reference in New Issue
Block a user