refactor(deploy): benchmark, canary, land-and-deploy Step 7, devex-review drive Aside

One aside repl script per page prints NAV/PAINT/LCP/RESOURCES/SCRIPTS/CSS/SUMMARY (benchmark), CONSOLE_ERRORS/NAV/TEXT + screenshot (canary, re-run every 60s), and the post-deploy check reads responseStatus from the navigation entry; each carries the $B fallback.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
Sina
2026-09-05 16:48:37 -04:00
co-authored by Claude Fable 5.1
parent 27e38cbeec
commit 0953aedbb1
5 changed files with 132 additions and 100 deletions
+28 -16
View File
@@ -3,8 +3,8 @@ name: devex-review
preamble-tier: 3
version: 1.0.0
description: |
Live developer experience audit. Uses the browse tool to actually TEST the
developer experience: navigates docs, tries the getting started flow, times
Live developer experience audit. Actually TESTS the developer experience
in the Aside browser: navigates docs, tries the getting started flow, times
TTHW, screenshots error messages, evaluates CLI help text. Produces a DX
scorecard with evidence. Compares against /plan-devex-review scores if they
exist (the boomerang: plan said 3 minutes, reality says 8). Use when asked to
@@ -33,26 +33,31 @@ allowed-tools:
{{BASE_BRANCH_DETECT}}
{{BROWSE_SETUP}}
{{ASIDE_SETUP}}
{{BROWSE_FALLBACK}}
{{ASIDE_COOKBOOK}}
# /devex-review: Live Developer Experience Audit
You are a DX engineer dogfooding a live developer product. Not reviewing a plan.
Not reading about the experience. TESTING it.
Use the browse tool to navigate docs, try the getting started flow, and screenshot
what developers actually see. Use bash to try CLI commands. Measure, don't guess.
Drive the Aside browser to navigate docs, try the getting started flow, and screenshot
what developers actually see. One `aside repl` script per flow, each re-opening from the URL. Use bash to try CLI commands. Measure, don't guess.
{{DX_FRAMEWORK}}
## Scope Declaration
Browse can test web-accessible surfaces: docs pages, API playgrounds, web dashboards,
signup flows, interactive tutorials, error pages.
Aside can test web-accessible surfaces: docs pages, API playgrounds, web dashboards,
signup flows, interactive tutorials, error pages — with the user's real logged-in
sessions.
Browse CANNOT test: CLI install friction, terminal output quality, local environment
setup, email verification flows, auth requiring real credentials, offline behavior,
build times, IDE integration.
Aside CANNOT test: CLI install friction, terminal output quality, local environment
setup, email verification flows, credential entry (the user signs in themselves; you
never type passwords), offline behavior, build times, IDE integration.
For untestable dimensions, use bash (for CLI --help, README, CHANGELOG) or mark as
INFERRED from artifacts. Never guess. State your evidence source for every score.
@@ -78,7 +83,8 @@ If prior scores exist, display them. These are your baseline for the boomerang c
## Step 1: Getting Started Audit
Navigate to the docs/landing page via browse. Screenshot it.
Open the docs/landing page with the Aside read script from the cookbook (console errors,
snapshot, screenshot, text). Copy the screenshot out of the printed ASIDE_DIR and Read it.
```
GETTING STARTED AUDIT
@@ -95,7 +101,7 @@ Score 0-10. Load "## Pass 1" from dx-hall-of-fame.md for calibration.
Test what you can:
- CLI: Run `--help` via bash. Evaluate output quality, flag design, discoverability.
- API playground: Navigate via browse if one exists. Screenshot.
- API playground: Open it in Aside if one exists. Screenshot.
- Naming: Check consistency across the API surface.
Score 0-10. Load "## Pass 2" from dx-hall-of-fame.md for calibration.
@@ -103,7 +109,8 @@ Score 0-10. Load "## Pass 2" from dx-hall-of-fame.md for calibration.
## Step 3: Error Message Audit
Trigger common error scenarios:
- Browse: Navigate to 404 pages, submit invalid forms, try unauthenticated access
- Aside: Open a 404 URL, submit an invalid form (on a non-LOCAL target that is a mutating
action — one AskUserQuestion per run first, per the browser rules), open a protected URL
- CLI: Run with missing args, invalid flags, bad input
Screenshot each error. Score against the Elm/Rust/Stripe three-tier model.
@@ -112,7 +119,9 @@ Score 0-10. Load "## Pass 3" from dx-hall-of-fame.md for calibration.
## Step 4: Documentation Audit
Navigate the docs structure via browse:
Navigate the docs structure in Aside (search is `pg.fill(<search selector>, <query>)`,
then `pg.locator(<search selector>).press("Enter")` — or `pg.getByRole("searchbox").press("Enter")`,
or a click — then `snapshot`):
- Check search functionality (try 3 common queries)
- Verify code examples are copy-paste-complete
- Check language switcher behavior
@@ -141,12 +150,15 @@ Score 0-10. Evidence: INFERRED from files. Load "## Pass 6" from dx-hall-of-fame
## Step 7: Community & Ecosystem Audit
Browse:
Check the community links the docs point to. Aside stays on the docs origin (browser
rule 2): confirm the links are PRESENT in the Step 1 snapshot or with the same-origin links
script from the cookbook, and audit GitHub via `gh` in bash. Do not open Discord, Stack
Overflow, or any other third-party site — mark those INFERRED (link present, not followed):
- Community links (GitHub Discussions, Discord, Stack Overflow)
- GitHub issues (response time, templates, labels)
- Contributing guide
Score 0-10. Evidence: TESTED where web-accessible, INFERRED otherwise.
Score 0-10. Evidence: TESTED for the docs page and GitHub, INFERRED otherwise.
## Step 8: DX Measurement Audit