mirror of
https://github.com/garrytan/gstack.git
synced 2026-09-09 06:28:59 +02:00
refactor(deploy): benchmark, canary, land-and-deploy Step 7, devex-review drive Aside
One aside repl script per page prints NAV/PAINT/LCP/RESOURCES/SCRIPTS/CSS/SUMMARY (benchmark), CONSOLE_ERRORS/NAV/TEXT + screenshot (canary, re-run every 60s), and the post-deploy check reads responseStatus from the navigation entry; each carries the $B fallback. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Sina
+23
-29
@@ -3,7 +3,7 @@ name: benchmark
|
||||
preamble-tier: 1
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Performance regression detection using the browse daemon. Establishes
|
||||
Performance regression detection. Establishes
|
||||
baselines for page load times, Core Web Vitals, and resource sizes.
|
||||
Compares before/after on every PR. Tracks performance trends over time.
|
||||
Use when: "performance", "benchmark", "page speed", "lighthouse", "web vitals",
|
||||
@@ -25,13 +25,15 @@ allowed-tools:
|
||||
|
||||
{{PREAMBLE}}
|
||||
|
||||
{{BROWSE_SETUP}}
|
||||
{{ASIDE_SETUP}}
|
||||
|
||||
{{BROWSE_FALLBACK}}
|
||||
|
||||
# /benchmark — Performance Regression Detection
|
||||
|
||||
You are a **Performance Engineer** who has optimized apps serving millions of requests. You know that performance doesn't degrade in one big regression — it dies by a thousand paper cuts. Each PR adds 50ms here, 20KB there, and one day the app takes 8 seconds to load and nobody knows when it got slow.
|
||||
|
||||
Your job is to measure, baseline, compare, and alert. You use the browse daemon's `perf` command and JavaScript evaluation to gather real performance data from running pages.
|
||||
Your job is to measure, baseline, compare, and alert. You drive the Aside browser and read `performance.getEntries()` straight from the live page — real numbers from a real browser, not estimates.
|
||||
|
||||
## User-invocable
|
||||
When the user types `/benchmark`, run this skill.
|
||||
@@ -65,42 +67,34 @@ git diff $(gh pr view --json baseRefName -q .baseRefName 2>/dev/null || gh repo
|
||||
|
||||
### Phase 3: Performance Data Collection
|
||||
|
||||
For each page, collect comprehensive performance metrics:
|
||||
For each page, ONE `aside repl` script opens the page and prints every metric as a labelled line. Tabs die when the script ends, so nothing carries over between pages — each page gets its own run:
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B perf
|
||||
aside repl '
|
||||
const pg = await openTab("<page-url>");
|
||||
await pg.waitForLoadState("load");
|
||||
console.log("NAV=" + await pg.evaluate(() => JSON.stringify(performance.getEntriesByType("navigation")[0]))); // stringify IN the page: PerformanceEntry fields are getters and serialize to {} across the bridge
|
||||
console.log("PAINT=" + await pg.evaluate(() => JSON.stringify(performance.getEntriesByType("paint").map(p => ({ name: p.name, start: Math.round(p.startTime) })))));
|
||||
console.log("LCP=" + await pg.evaluate(() => new Promise(res => { const po = new PerformanceObserver(l => { const e = l.getEntries().pop(); if (e) res(Math.round(e.startTime)); }); po.observe({ type: "largest-contentful-paint", buffered: true }); setTimeout(() => res(null), 3000); })));
|
||||
console.log("RESOURCES=" + JSON.stringify(await pg.evaluate(() => performance.getEntriesByType("resource").map(r => ({ name: r.name.split("/").pop().split("?")[0], type: r.initiatorType, size: r.transferSize, duration: Math.round(r.duration) })).sort((a, b) => b.duration - a.duration).slice(0, 15))));
|
||||
console.log("SCRIPTS=" + JSON.stringify(await pg.evaluate(() => performance.getEntriesByType("resource").filter(r => r.initiatorType === "script").map(r => ({ name: r.name.split("/").pop().split("?")[0], size: r.transferSize })))));
|
||||
console.log("CSS=" + JSON.stringify(await pg.evaluate(() => performance.getEntriesByType("resource").filter(r => r.initiatorType === "css").map(r => ({ name: r.name.split("/").pop().split("?")[0], size: r.transferSize })))));
|
||||
console.log("SUMMARY=" + JSON.stringify(await pg.evaluate(() => { const r = performance.getEntriesByType("resource"); return { total_requests: r.length, total_transfer: r.reduce((s, e) => s + (e.transferSize || 0), 0), by_type: Object.entries(r.reduce((a, e) => { a[e.initiatorType] = (a[e.initiatorType] || 0) + 1; return a; }, {})).sort((a, b) => b[1] - a[1]) }; })));
|
||||
await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
```
|
||||
|
||||
Then gather detailed metrics via JavaScript:
|
||||
`NAV=` is the navigation timing entry, `PAINT=` the paint entries (FCP lives here), `LCP=` the largest-contentful-paint start time (`null` if the page emitted no LCP entry within 3s), `RESOURCES=` the 15 slowest resources, `SCRIPTS=` / `CSS=` the bundle inventory, `SUMMARY=` request count, total transfer, and requests by type. A missing `GSTACK_STEP_OK` or a line starting with `[error` means the page did not load — record it as a failure, not a slow page.
|
||||
|
||||
```bash
|
||||
$B eval "JSON.stringify(performance.getEntriesByType('navigation')[0])"
|
||||
```
|
||||
|
||||
Extract key metrics:
|
||||
Extract key metrics from `NAV=`:
|
||||
- **TTFB** (Time to First Byte): `responseStart - requestStart`
|
||||
- **FCP** (First Contentful Paint): from PerformanceObserver or `paint` entries
|
||||
- **LCP** (Largest Contentful Paint): from PerformanceObserver
|
||||
- **FCP** (First Contentful Paint): the `first-contentful-paint` entry in `PAINT=`
|
||||
- **LCP** (Largest Contentful Paint): the `LCP=` line (`null` if the page emitted no LCP entry — record it as missing, not 0)
|
||||
- **DOM Interactive**: `domInteractive - navigationStart`
|
||||
- **DOM Complete**: `domComplete - navigationStart`
|
||||
- **Full Load**: `loadEventEnd - navigationStart`
|
||||
|
||||
Resource analysis:
|
||||
```bash
|
||||
$B eval "JSON.stringify(performance.getEntriesByType('resource').map(r => ({name: r.name.split('/').pop().split('?')[0], type: r.initiatorType, size: r.transferSize, duration: Math.round(r.duration)})).sort((a,b) => b.duration - a.duration).slice(0,15))"
|
||||
```
|
||||
|
||||
Bundle size check:
|
||||
```bash
|
||||
$B eval "JSON.stringify(performance.getEntriesByType('resource').filter(r => r.initiatorType === 'script').map(r => ({name: r.name.split('/').pop().split('?')[0], size: r.transferSize})))"
|
||||
$B eval "JSON.stringify(performance.getEntriesByType('resource').filter(r => r.initiatorType === 'css').map(r => ({name: r.name.split('/').pop().split('?')[0], size: r.transferSize})))"
|
||||
```
|
||||
|
||||
Network summary:
|
||||
```bash
|
||||
$B eval "(() => { const r = performance.getEntriesByType('resource'); return JSON.stringify({total_requests: r.length, total_transfer: r.reduce((s,e) => s + (e.transferSize||0), 0), by_type: Object.entries(r.reduce((a,e) => { a[e.initiatorType] = (a[e.initiatorType]||0) + 1; return a; }, {})).sort((a,b) => b[1]-a[1])})})()"
|
||||
```
|
||||
Load times jitter with the network. If the user wants stable numbers, run the script 3 times per page and take the median of each metric.
|
||||
|
||||
### Phase 4: Baseline Capture (--baseline mode)
|
||||
|
||||
|
||||
+47
-26
@@ -4,7 +4,7 @@ preamble-tier: 2
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Post-deploy canary monitoring. Watches the live app for console errors,
|
||||
performance regressions, and page failures using the browse daemon. Takes
|
||||
performance regressions, and page failures. Takes
|
||||
periodic screenshots, compares against pre-deploy baselines, and alerts
|
||||
on anomalies. Use when: "monitor deploy", "canary", "post-deploy check",
|
||||
"watch production", "verify deploy". (gstack)
|
||||
@@ -22,7 +22,9 @@ triggers:
|
||||
|
||||
{{PREAMBLE}}
|
||||
|
||||
{{BROWSE_SETUP}}
|
||||
{{ASIDE_SETUP}}
|
||||
|
||||
{{BROWSE_FALLBACK}}
|
||||
|
||||
{{BASE_BRANCH_DETECT}}
|
||||
|
||||
@@ -30,7 +32,7 @@ triggers:
|
||||
|
||||
You are a **Release Reliability Engineer** watching production after a deploy. You've seen deploys that pass CI but break in production — a missing environment variable, a CDN cache serving stale assets, a database migration that's slower than expected on real data. Your job is to catch these in the first 10 minutes, not 10 hours.
|
||||
|
||||
You use the browse daemon to watch the live app, take screenshots, check console errors, and compare against baselines. You are the safety net between "shipped" and "verified."
|
||||
You drive the Aside browser to watch the live app, take screenshots, check console errors, and compare against baselines. You are the safety net between "shipped" and "verified."
|
||||
|
||||
## User-invocable
|
||||
When the user types `/canary`, run this skill.
|
||||
@@ -62,14 +64,24 @@ If the user passed `--baseline`, capture the current state BEFORE deploying.
|
||||
For each page (either from `--pages` or the homepage):
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o ".gstack/canary-reports/baselines/<page-name>.png"
|
||||
$B console --errors
|
||||
$B perf
|
||||
$B text
|
||||
aside repl '
|
||||
const HOOK = `(() => { window.__gstackErrs = window.__gstackErrs || []; const oe = console.error; console.error = (...a) => { window.__gstackErrs.push(a.map(String).join(" ")); oe.apply(console, a); }; window.addEventListener("error", e => window.__gstackErrs.push("uncaught: " + e.message)); window.addEventListener("unhandledrejection", e => window.__gstackErrs.push("unhandledrejection: " + (e.reason && e.reason.message || e.reason))); })()`;
|
||||
const pg = await openTab("about:blank");
|
||||
await pg._sendToTarget("Page.addScriptToEvaluateOnNewDocument", { source: HOOK });
|
||||
await pg.goto("<page-url>");
|
||||
console.log("CONSOLE_ERRORS=" + JSON.stringify(await pg.evaluate(() => window.__gstackErrs)));
|
||||
console.log("NAV=" + await pg.evaluate(() => JSON.stringify(performance.getEntriesByType("navigation")[0])));
|
||||
console.log("TEXT_START"); console.log((await pg.evaluate(() => document.body.innerText)).slice(0, 20000)); console.log("TEXT_END");
|
||||
await pg.screenshot({ path: "<page-name>.jpg", type: "jpeg", quality: 60, fullPage: true });
|
||||
console.log("ASIDE_DIR=" + pwd);
|
||||
await closeTab(pg);
|
||||
console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
```
|
||||
|
||||
Collect for each page: screenshot path, console error count, page load time from `perf`, and a text content snapshot.
|
||||
Then copy the screenshot out of the printed session directory: `cp "<ASIDE_DIR>/<page-name>.jpg" .gstack/canary-reports/baselines/<page-name>.jpg`
|
||||
|
||||
Collect for each page: screenshot path, console error count (`CONSOLE_ERRORS=`), load time (`loadEventEnd` in `NAV=`), and the text snapshot between `TEXT_START` / `TEXT_END`.
|
||||
|
||||
Save the baseline manifest to `.gstack/canary-reports/baseline.json`:
|
||||
|
||||
@@ -80,7 +92,7 @@ Save the baseline manifest to `.gstack/canary-reports/baseline.json`:
|
||||
"branch": "<current branch>",
|
||||
"pages": {
|
||||
"/": {
|
||||
"screenshot": "baselines/home.png",
|
||||
"screenshot": "baselines/home.jpg",
|
||||
"console_errors": 0,
|
||||
"load_time_ms": 450
|
||||
}
|
||||
@@ -95,12 +107,15 @@ Then STOP and tell the user: "Baseline captured. Deploy your changes, then run `
|
||||
If no `--pages` were specified, auto-discover pages to monitor:
|
||||
|
||||
```bash
|
||||
$B goto <url>
|
||||
$B links
|
||||
$B snapshot -i
|
||||
aside repl '
|
||||
const pg = await openTab("<url>");
|
||||
const links = await pg.evaluate(() => [...new Set([...document.querySelectorAll("a[href]")].map(a => a.href))].filter(h => h.startsWith(location.origin) && !/logout|signout|delete|remove|cancel|unsubscribe/i.test(h)));
|
||||
for (const l of links) { const r = await fetch(l, { method: "HEAD" }).catch(e => ({ status: "ERR " + e.message })); console.log("LINK", r.status, l); }
|
||||
await closeTab(pg); console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
```
|
||||
|
||||
Extract the top 5 internal navigation links from the `links` output. Always include the homepage. Present the page list via AskUserQuestion:
|
||||
Extract the top 5 internal navigation links from the `LINK` lines (same-origin only — the script already filters). Always include the homepage. Present the page list via AskUserQuestion:
|
||||
|
||||
- **Context:** Monitoring the production site at the given URL after a deploy.
|
||||
- **Question:** Which pages should the canary monitor?
|
||||
@@ -115,29 +130,35 @@ If no `baseline.json` exists, take a quick snapshot now as a reference point.
|
||||
|
||||
For each page to monitor:
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o ".gstack/canary-reports/screenshots/pre-<page-name>.png"
|
||||
$B console --errors
|
||||
$B perf
|
||||
```
|
||||
Run the Phase 2 read script for each page with the screenshot saved as `pre-<page-name>.jpg`, then `cp "<ASIDE_DIR>/pre-<page-name>.jpg" .gstack/canary-reports/screenshots/`.
|
||||
|
||||
Record the console error count and load time for each page. These become the reference for detecting regressions during monitoring.
|
||||
|
||||
### Phase 5: Continuous Monitoring Loop
|
||||
|
||||
Monitor for the specified duration. Every 60 seconds, check each page:
|
||||
Monitor for the specified duration. Every 60 seconds, check each page. Nothing persists between scripts — every check re-opens the page from its URL and captures fresh evidence:
|
||||
|
||||
```bash
|
||||
$B goto <page-url>
|
||||
$B snapshot -i -a -o ".gstack/canary-reports/screenshots/<page-name>-<check-number>.png"
|
||||
$B console --errors
|
||||
$B perf
|
||||
aside repl '
|
||||
const HOOK = `(() => { window.__gstackErrs = window.__gstackErrs || []; const oe = console.error; console.error = (...a) => { window.__gstackErrs.push(a.map(String).join(" ")); oe.apply(console, a); }; window.addEventListener("error", e => window.__gstackErrs.push("uncaught: " + e.message)); window.addEventListener("unhandledrejection", e => window.__gstackErrs.push("unhandledrejection: " + (e.reason && e.reason.message || e.reason))); })()`;
|
||||
const pg = await openTab("about:blank");
|
||||
await pg._sendToTarget("Page.addScriptToEvaluateOnNewDocument", { source: HOOK });
|
||||
await pg.goto("<page-url>");
|
||||
console.log("CONSOLE_ERRORS=" + JSON.stringify(await pg.evaluate(() => window.__gstackErrs)));
|
||||
console.log("NAV=" + await pg.evaluate(() => JSON.stringify(performance.getEntriesByType("navigation")[0])));
|
||||
console.log("TEXT_START"); console.log((await pg.evaluate(() => document.body.innerText)).slice(0, 20000)); console.log("TEXT_END");
|
||||
await pg.screenshot({ path: "<page-name>-<check-number>.jpg", type: "jpeg", quality: 60, fullPage: true });
|
||||
console.log("ASIDE_DIR=" + pwd);
|
||||
await closeTab(pg);
|
||||
console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
```
|
||||
|
||||
Then `cp "<ASIDE_DIR>/<page-name>-<check-number>.jpg" .gstack/canary-reports/screenshots/`.
|
||||
|
||||
After each check, compare results against the baseline (or pre-deploy snapshot):
|
||||
|
||||
1. **Page load failure** — `goto` returns error or timeout → CRITICAL ALERT
|
||||
1. **Page load failure** — the script prints a line starting with `[error` or never prints `GSTACK_STEP_OK` → CRITICAL ALERT
|
||||
2. **New console errors** — errors not present in baseline → HIGH ALERT
|
||||
3. **Performance regression** — load time exceeds 2x baseline → MEDIUM ALERT
|
||||
4. **Broken links** — new 404s not in baseline → LOW ALERT
|
||||
|
||||
+28
-16
@@ -3,8 +3,8 @@ name: devex-review
|
||||
preamble-tier: 3
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Live developer experience audit. Uses the browse tool to actually TEST the
|
||||
developer experience: navigates docs, tries the getting started flow, times
|
||||
Live developer experience audit. Actually TESTS the developer experience
|
||||
in the Aside browser: navigates docs, tries the getting started flow, times
|
||||
TTHW, screenshots error messages, evaluates CLI help text. Produces a DX
|
||||
scorecard with evidence. Compares against /plan-devex-review scores if they
|
||||
exist (the boomerang: plan said 3 minutes, reality says 8). Use when asked to
|
||||
@@ -33,26 +33,31 @@ allowed-tools:
|
||||
|
||||
{{BASE_BRANCH_DETECT}}
|
||||
|
||||
{{BROWSE_SETUP}}
|
||||
{{ASIDE_SETUP}}
|
||||
|
||||
{{BROWSE_FALLBACK}}
|
||||
|
||||
{{ASIDE_COOKBOOK}}
|
||||
|
||||
# /devex-review: Live Developer Experience Audit
|
||||
|
||||
You are a DX engineer dogfooding a live developer product. Not reviewing a plan.
|
||||
Not reading about the experience. TESTING it.
|
||||
|
||||
Use the browse tool to navigate docs, try the getting started flow, and screenshot
|
||||
what developers actually see. Use bash to try CLI commands. Measure, don't guess.
|
||||
Drive the Aside browser to navigate docs, try the getting started flow, and screenshot
|
||||
what developers actually see. One `aside repl` script per flow, each re-opening from the URL. Use bash to try CLI commands. Measure, don't guess.
|
||||
|
||||
{{DX_FRAMEWORK}}
|
||||
|
||||
## Scope Declaration
|
||||
|
||||
Browse can test web-accessible surfaces: docs pages, API playgrounds, web dashboards,
|
||||
signup flows, interactive tutorials, error pages.
|
||||
Aside can test web-accessible surfaces: docs pages, API playgrounds, web dashboards,
|
||||
signup flows, interactive tutorials, error pages — with the user's real logged-in
|
||||
sessions.
|
||||
|
||||
Browse CANNOT test: CLI install friction, terminal output quality, local environment
|
||||
setup, email verification flows, auth requiring real credentials, offline behavior,
|
||||
build times, IDE integration.
|
||||
Aside CANNOT test: CLI install friction, terminal output quality, local environment
|
||||
setup, email verification flows, credential entry (the user signs in themselves; you
|
||||
never type passwords), offline behavior, build times, IDE integration.
|
||||
|
||||
For untestable dimensions, use bash (for CLI --help, README, CHANGELOG) or mark as
|
||||
INFERRED from artifacts. Never guess. State your evidence source for every score.
|
||||
@@ -78,7 +83,8 @@ If prior scores exist, display them. These are your baseline for the boomerang c
|
||||
|
||||
## Step 1: Getting Started Audit
|
||||
|
||||
Navigate to the docs/landing page via browse. Screenshot it.
|
||||
Open the docs/landing page with the Aside read script from the cookbook (console errors,
|
||||
snapshot, screenshot, text). Copy the screenshot out of the printed ASIDE_DIR and Read it.
|
||||
|
||||
```
|
||||
GETTING STARTED AUDIT
|
||||
@@ -95,7 +101,7 @@ Score 0-10. Load "## Pass 1" from dx-hall-of-fame.md for calibration.
|
||||
|
||||
Test what you can:
|
||||
- CLI: Run `--help` via bash. Evaluate output quality, flag design, discoverability.
|
||||
- API playground: Navigate via browse if one exists. Screenshot.
|
||||
- API playground: Open it in Aside if one exists. Screenshot.
|
||||
- Naming: Check consistency across the API surface.
|
||||
|
||||
Score 0-10. Load "## Pass 2" from dx-hall-of-fame.md for calibration.
|
||||
@@ -103,7 +109,8 @@ Score 0-10. Load "## Pass 2" from dx-hall-of-fame.md for calibration.
|
||||
## Step 3: Error Message Audit
|
||||
|
||||
Trigger common error scenarios:
|
||||
- Browse: Navigate to 404 pages, submit invalid forms, try unauthenticated access
|
||||
- Aside: Open a 404 URL, submit an invalid form (on a non-LOCAL target that is a mutating
|
||||
action — one AskUserQuestion per run first, per the browser rules), open a protected URL
|
||||
- CLI: Run with missing args, invalid flags, bad input
|
||||
|
||||
Screenshot each error. Score against the Elm/Rust/Stripe three-tier model.
|
||||
@@ -112,7 +119,9 @@ Score 0-10. Load "## Pass 3" from dx-hall-of-fame.md for calibration.
|
||||
|
||||
## Step 4: Documentation Audit
|
||||
|
||||
Navigate the docs structure via browse:
|
||||
Navigate the docs structure in Aside (search is `pg.fill(<search selector>, <query>)`,
|
||||
then `pg.locator(<search selector>).press("Enter")` — or `pg.getByRole("searchbox").press("Enter")`,
|
||||
or a click — then `snapshot`):
|
||||
- Check search functionality (try 3 common queries)
|
||||
- Verify code examples are copy-paste-complete
|
||||
- Check language switcher behavior
|
||||
@@ -141,12 +150,15 @@ Score 0-10. Evidence: INFERRED from files. Load "## Pass 6" from dx-hall-of-fame
|
||||
|
||||
## Step 7: Community & Ecosystem Audit
|
||||
|
||||
Browse:
|
||||
Check the community links the docs point to. Aside stays on the docs origin (browser
|
||||
rule 2): confirm the links are PRESENT in the Step 1 snapshot or with the same-origin links
|
||||
script from the cookbook, and audit GitHub via `gh` in bash. Do not open Discord, Stack
|
||||
Overflow, or any other third-party site — mark those INFERRED (link present, not followed):
|
||||
- Community links (GitHub Discussions, Discord, Stack Overflow)
|
||||
- GitHub issues (response time, templates, labels)
|
||||
- Contributing guide
|
||||
|
||||
Score 0-10. Evidence: TESTED where web-accessible, INFERRED otherwise.
|
||||
Score 0-10. Evidence: TESTED for the docs page and GitHub, INFERRED otherwise.
|
||||
|
||||
## Step 8: DX Measurement Audit
|
||||
|
||||
|
||||
@@ -24,7 +24,9 @@ triggers:
|
||||
|
||||
{{THIRD_PARTY_ACTIONS}}
|
||||
|
||||
{{BROWSE_SETUP}}
|
||||
{{ASIDE_SETUP}}
|
||||
|
||||
{{BROWSE_FALLBACK}}
|
||||
|
||||
{{BASE_BRANCH_DETECT}}
|
||||
|
||||
@@ -308,45 +310,48 @@ Use the diff-scope classification from Step 5 to determine canary depth:
|
||||
| Diff Scope | Canary Depth |
|
||||
|------------|-------------|
|
||||
| SCOPE_DOCS only | Already skipped in Step 5 |
|
||||
| SCOPE_CONFIG only | Smoke: `$B goto` + verify 200 status |
|
||||
| SCOPE_CONFIG only | Smoke: the Aside script below; `responseStatus` in `NAV=` must be 200 |
|
||||
| SCOPE_BACKEND only | Console errors + perf check |
|
||||
| SCOPE_FRONTEND (any) | Full: console + perf + screenshot |
|
||||
| Mixed scopes | Full canary |
|
||||
|
||||
**Full canary sequence:**
|
||||
**Full canary sequence** — one `aside repl` script does the whole check (console hook first, then load, then evidence):
|
||||
|
||||
```bash
|
||||
$B goto <url>
|
||||
aside repl '
|
||||
const HOOK = `(() => { window.__gstackErrs = window.__gstackErrs || []; const oe = console.error; console.error = (...a) => { window.__gstackErrs.push(a.map(String).join(" ")); oe.apply(console, a); }; window.addEventListener("error", e => window.__gstackErrs.push("uncaught: " + e.message)); window.addEventListener("unhandledrejection", e => window.__gstackErrs.push("unhandledrejection: " + (e.reason && e.reason.message || e.reason))); })()`;
|
||||
const pg = await openTab("about:blank");
|
||||
await pg._sendToTarget("Page.addScriptToEvaluateOnNewDocument", { source: HOOK });
|
||||
await pg.goto("<url>");
|
||||
console.log("URL=" + pg.url());
|
||||
console.log("CONSOLE_ERRORS=" + JSON.stringify(await pg.evaluate(() => window.__gstackErrs)));
|
||||
console.log("NAV=" + await pg.evaluate(() => JSON.stringify(performance.getEntriesByType("navigation")[0])));
|
||||
console.log("TEXT_START"); console.log((await pg.evaluate(() => document.body.innerText)).slice(0, 20000)); console.log("TEXT_END");
|
||||
await pg.screenshot({ path: "post-deploy.jpg", type: "jpeg", quality: 60, fullPage: true });
|
||||
const a = await annotatedScreenshot(pg);
|
||||
await fs.writeFile(path.join(pwd, "post-deploy-annotated.png"), Buffer.from(a.base64Image, "base64"));
|
||||
console.log("ASIDE_DIR=" + pwd);
|
||||
await closeTab(pg);
|
||||
console.log("GSTACK_STEP_OK");
|
||||
'
|
||||
```
|
||||
|
||||
Check that the page loaded successfully (200, not an error page).
|
||||
Then copy the evidence out of the printed session directory:
|
||||
|
||||
```bash
|
||||
$B console --errors
|
||||
mkdir -p .gstack/deploy-reports && cp "<ASIDE_DIR>/post-deploy.jpg" "<ASIDE_DIR>/post-deploy-annotated.png" .gstack/deploy-reports/
|
||||
```
|
||||
|
||||
Check for critical console errors: lines containing `Error`, `Uncaught`, `Failed to load`, `TypeError`, `ReferenceError`. Ignore warnings.
|
||||
Read the output line by line:
|
||||
|
||||
```bash
|
||||
$B perf
|
||||
```
|
||||
|
||||
Check that page load time is under 10 seconds.
|
||||
|
||||
```bash
|
||||
$B text
|
||||
```
|
||||
|
||||
Verify the page has content (not blank, not a generic error page).
|
||||
|
||||
```bash
|
||||
$B snapshot -i -a -o ".gstack/deploy-reports/post-deploy.png"
|
||||
```
|
||||
|
||||
Take an annotated screenshot as evidence.
|
||||
- `URL=` — the page loaded and stayed on the site (not a redirect to an error page). A line starting with `[error` or a missing `GSTACK_STEP_OK` means the load failed.
|
||||
- `CONSOLE_ERRORS=` — check for critical errors: entries containing `Error`, `Uncaught`, `Failed to load`, `TypeError`, `ReferenceError`. Ignore warnings.
|
||||
- `NAV=` — `responseStatus` is the HTTP status of the document (Chromium PerformanceNavigationTiming) — must be 200. `loadEventEnd` is the page load time. Check that it is under 10 seconds.
|
||||
- `TEXT_START` / `TEXT_END` — verify the page has real content (not blank, not a generic error page).
|
||||
- `post-deploy.jpg` and the annotated `post-deploy-annotated.png` are the evidence. Read the copied screenshot so the user sees it.
|
||||
|
||||
**Health assessment:**
|
||||
- Page loads successfully with 200 status → PASS
|
||||
- Page loads successfully with 200 status (`responseStatus` in `NAV=`) → PASS
|
||||
- No critical console errors → PASS
|
||||
- Page has real content (not blank or error screen) → PASS
|
||||
- Loads in under 10 seconds → PASS
|
||||
|
||||
@@ -279,7 +279,7 @@ describeIfSelected('Canary skill E2E', ['canary-workflow'], () => {
|
||||
const result = await runSkillTest({
|
||||
prompt: `Read canary/SKILL.md for the /canary skill instructions.
|
||||
|
||||
You are simulating a canary check. There is NO browse daemon available and NO production URL.
|
||||
You are simulating a canary check. No browser is available on this machine (no Aside, no browse daemon) and there is NO production URL.
|
||||
|
||||
Instead, demonstrate you understand the workflow:
|
||||
1. Create the .gstack/canary-reports/ directory structure
|
||||
@@ -290,7 +290,7 @@ Instead, demonstrate you understand the workflow:
|
||||
the Phase 6 Health Report format (CANARY REPORT header, duration, pages, status,
|
||||
per-page results table, verdict)
|
||||
|
||||
Do NOT use AskUserQuestion. Do NOT run browse ($B) commands.
|
||||
Do NOT use AskUserQuestion. Do NOT run aside or browse ($B) commands.
|
||||
Just create the directory structure and report files showing the correct schema.`,
|
||||
workingDirectory: canaryDir,
|
||||
maxTurns: 15,
|
||||
@@ -340,7 +340,7 @@ describeIfSelected('Benchmark skill E2E', ['benchmark-workflow'], () => {
|
||||
const result = await runSkillTest({
|
||||
prompt: `Read benchmark/SKILL.md for the /benchmark skill instructions.
|
||||
|
||||
You are simulating a benchmark run. There is NO browse daemon available and NO production URL.
|
||||
You are simulating a benchmark run. No browser is available on this machine (no Aside, no browse daemon) and there is NO production URL.
|
||||
|
||||
Instead, demonstrate you understand the workflow:
|
||||
1. Create the .gstack/benchmark-reports/ directory structure including baselines/
|
||||
@@ -353,7 +353,7 @@ Instead, demonstrate you understand the workflow:
|
||||
table with Baseline/Current/Delta/Status columns, regression thresholds applied)
|
||||
4. Include the Phase 7 Performance Budget section in the report
|
||||
|
||||
Do NOT use AskUserQuestion. Do NOT run browse ($B) commands.
|
||||
Do NOT use AskUserQuestion. Do NOT run aside or browse ($B) commands.
|
||||
Just create the files showing the correct schema and report format.`,
|
||||
workingDirectory: benchDir,
|
||||
maxTurns: 15,
|
||||
|
||||
Reference in New Issue
Block a user