mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-12 05:57:13 +02:00
v1.91.7.0 feat: add functional QA and pre-publication docs checks (#2983)
* feat: add surface-aware exploratory QA and ship documentation gates * test: preserve delegated QA setup authority after main integration * fix(qa): clarify exploration order and preserve report artifacts * test(qa): follow the shared setup reference directly * refactor(ship): make verification and recovery routes explicit * test(ship): align evidence and review guards with explicit routes * fix(workflows): clarify ship recovery and functional QA evidence * fix(workflows): clarify approval recovery and full QA coverage * refactor(workflows): order review transactions and clarify ship state * fix(ship): clarify final verification and fail closed at publication * fix(evals): attribute native atomic documentation writes * fix(ship): clarify recovery and documentation lifecycle guidance * fix(test): preserve observed native placeholder styling in CI * fix(codex): report watchdog timeouts without a process-exit race * Checkpoint functional QA implementation and workflow validation repairs * Fix documentation and shared-review fixture contracts * docs: clarify judge reuse and evaluation supervision * test: align review evidence and selected case contracts * test: verify append-only documentation checkpoints and recovery * fix: qualify QA workflows and CI validation repairs * fix: launch shared-libs fixture scripts on Windows * fix: qualify QA deadlines, fixture isolation, and shard cleanup * fix: preserve qualified QA and cancellation repairs * fix: enforce functional fixture authority and share strict event decoding * fix: retain free-test evidence and explain recovery * fix: reject malformed native evidence after decoder consolidation * test: use reliable capture for telemetry privacy filters * test: refresh measured quick coverage and document validation costs * Fix native fixture receipts and preserve VM validation evidence * Align negative judge controls with upstream clarity policy * Fix report-only QA preparation and public evidence handling * Clarify QA-only preparation and current-report preservation * Stream Ship quality judgments with an explicit 64k response contract * Validate compact judge reasoning locally with supported wire schema * Align functional QA fixture instructions with evidence acceptance * Bind native browser diagnostics to execution evidence and align review verdicts * Preserve native diagnostic line boundaries * Serialize functional QA evidence from native captures * Keep large QA evidence fixture payload out of Windows argv
This commit is contained in:
Garry Tan authored and GitHub committed
1 parent
65bfb0ce49
commit
dcaea52800
333 files changed
+41755
-7357
No files matched your search
+151
-58
@@ -3,8 +3,8 @@ name: qa-only
|
||||
preamble-tier: 4
|
||||
version: 1.0.0
|
||||
description: |
|
||||
Report-only QA testing. Systematically tests a web application and produces a
|
||||
structured report with health score, screenshots, and repro steps — but never
|
||||
Report browser/API/CLI/job/worker/webhook bugs. Produces a
|
||||
structured report with contract evidence or browser scores and repro steps — but never
|
||||
fixes anything. Use when asked to "just report bugs", "qa report only", or
|
||||
"test but don't fix". For the full test-fix-verify loop, use /qa instead.
|
||||
Proactively suggest when the user wants a bug report without any code changes. (gstack)
|
||||
@@ -27,91 +27,184 @@ triggers:
|
||||
|
||||
# /qa-only: Report-Only QA Testing
|
||||
|
||||
You are a QA engineer. Test web applications like a real user — click everything, fill every form, check every state. Produce a structured report with evidence. **NEVER fix anything.**
|
||||
Explore the selected surfaces and report reproducible behavior with evidence.
|
||||
**NEVER fix anything or change product tests.** Write only reports, evidence and
|
||||
owned temporary fixtures; the Additional Rules below define these limits.
|
||||
|
||||
## Setup
|
||||
In shared sections, **caller** means this /qa-only workflow. The user sets its
|
||||
permissions; an invoking workflow may restrict them further. **Owned** means created
|
||||
for this run or explicitly assigned to it, not merely writable. Neither term permits repairs.
|
||||
|
||||
{{SECTION_INDEX:qa-only}}
|
||||
|
||||
Start at Request Parameters, then follow the sections below in order.
|
||||
|
||||
## Request Parameters
|
||||
|
||||
**Parse the user's request for these parameters:**
|
||||
|
||||
| Parameter | Default | Override example |
|
||||
|-----------|---------|-----------------:|
|
||||
| Target URL | (auto-detect or required) | `https://myapp.com`, `http://localhost:3000` |
|
||||
| Mode | full | `--quick`, `--regression .gstack/qa-reports/baseline.json` |
|
||||
| Target | (infer from request/repository or ask) | Browser URL, API route, CLI command, job, worker or webhook |
|
||||
| Mode | full | `--quick`, `--regression <previous-report-or-baseline>` |
|
||||
| Output dir | `.gstack/qa-reports/` | `Output to /tmp/qa` |
|
||||
| Scope | Full app (or diff-scoped) | `Focus on the billing page` |
|
||||
| Auth | Your Aside session (already signed in) | If a sign-in wall appears, you sign in yourself in Aside — no credentials in chat (see BROWSER SETUP). Fallback browser only: /setup-browser-cookies or `$B handoff` |
|
||||
| Scope | Selected target (or diff-scoped) | `Focus on duplicate webhook delivery` |
|
||||
|
||||
**If no URL is given and you're on a feature branch:** Automatically enter **diff-aware mode** (see Modes below). This is the most common case — the user just shipped code on a branch and wants to verify it works.
|
||||
Use an isolated synthetic identity for functional probes. For browser sessions,
|
||||
follow Browser Setup; never request credentials in chat.
|
||||
|
||||
**Browser: Aside**
|
||||
Parsing records the request; it does not start browser setup. If both `--quick` and
|
||||
`--regression` are supplied, ask the user to choose one mode before setup or probes.
|
||||
|
||||
{{ASIDE_SETUP}}
|
||||
|
||||
{{BROWSE_FALLBACK}}
|
||||
|
||||
**Create output directories:**
|
||||
|
||||
```bash
|
||||
REPORT_DIR=".gstack/qa-reports"
|
||||
mkdir -p "$REPORT_DIR/screenshots"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
{{LEARNINGS_SEARCH}}
|
||||
**On a feature branch without an explicit scope:** Use diff-aware testing of changed
|
||||
and adjacent behavior. Do not discover a browser merely because no URL was supplied.
|
||||
|
||||
## Test Plan Context
|
||||
|
||||
Before falling back to git diff heuristics, check for richer test plan sources:
|
||||
Look for a test plan in this conversation. If this session already knows the
|
||||
project's state directory, also Read its newest `*-test-plan-*.md` when permitted.
|
||||
Do not create state or run bookkeeping helpers just to find optional context.
|
||||
Prefer the plan covering more requested contracts; break ties by recency.
|
||||
If neither exists, use git diff analysis.
|
||||
|
||||
1. **Project-scoped test plans:** Check `~/.gstack/projects/` for recent `*-test-plan-*.md` files for this repo
|
||||
```bash
|
||||
setopt +o nomatch 2>/dev/null || true # zsh compat
|
||||
{{SLUG_EVAL}}
|
||||
ls -t ~/.gstack/projects/$SLUG/*-test-plan-*.md 2>/dev/null | head -1
|
||||
```
|
||||
2. **Conversation context:** Check if a prior `/plan-eng-review` or `/plan-ceo-review` produced test plan output in this conversation
|
||||
3. **Use whichever source is richer.** Fall back to git diff analysis only if neither is available.
|
||||
{{LEARNINGS_SEARCH}}
|
||||
|
||||
## Select Surfaces and Isolation
|
||||
|
||||
Load the shared preparation gate now: complete its scope and selected-method Reads,
|
||||
await their results, and select the surfaces. Defer charters, clocks and probes to
|
||||
Run the Selected Checks, after report ownership and conditional browser setup below.
|
||||
|
||||
{{SECTION:exploratory}}
|
||||
|
||||
Each surface's method defines Full, Quick and Regression. A mode flag applies to all
|
||||
selected surfaces unless the request names one surface; the others default to Full.
|
||||
For mixed Regression, the argument is the prior combined report. Resolve its functional
|
||||
replay evidence and browser baseline links first, then give each method its own baseline.
|
||||
A missing baseline blocks that surface's regression coverage, not independent checks.
|
||||
In mixed runs, use the user's surface order, defaulting to functional then browser.
|
||||
Finish one surface's probes before starting the next surface's clock; any supplied
|
||||
absolute deadline still applies to both. Do not reset a clock when switching surfaces.
|
||||
|
||||
## Prepare Report Artifacts
|
||||
|
||||
Resolve and preserve supplied prior report/baseline paths and their evidence links
|
||||
before writing. Select the requested output dir or `.gstack/qa-reports`; create it if absent.
|
||||
Use that directory as `REPORT_DIR` only when it is empty; otherwise choose a fresh owned run subdirectory.
|
||||
Use `run-YYYYMMDDTHHMMSSZ` in UTC, adding a suffix on collision.
|
||||
All local reports, baselines and evidence use this directory.
|
||||
Never overwrite artifacts from earlier runs. Preserve this run's baselines,
|
||||
screenshots and exploration notes when finalizing its report.
|
||||
|
||||
A caller's fixed artifact paths and permissions take precedence. An existing empty
|
||||
directory already established as owned by the caller needs no new shell commands
|
||||
to revalidate it; use the caller's supported interface and fixed destinations.
|
||||
If safe preservation is impossible within those permissions, report an output blocker;
|
||||
do not expand write authority or silently redirect required artifacts.
|
||||
|
||||
For `{target}`, use the browser hostname, CLI executable basename, or named
|
||||
API service/job/worker/webhook. Replace characters other than letters, digits and
|
||||
hyphens with hyphens. For mixed targets, use
|
||||
`mixed-{project-label}`, sanitizing the repository name the same way; use `mixed-target`
|
||||
when no repository name is available. List the individual targets in the report.
|
||||
|
||||
Set `REPORT_FILE` to the caller's final report filename, otherwise
|
||||
`$REPORT_DIR/qa-report-{target}-{YYYY-MM-DD}.md`. Charters and final findings use this
|
||||
same file, not a sidecar.
|
||||
|
||||
## Browser Setup (conditional)
|
||||
|
||||
**Browser surface only:** load its setup; functional-only runs skip this section.
|
||||
|
||||
{{QA_RESOURCE:browser-setup}}
|
||||
|
||||
---
|
||||
|
||||
{{QA_METHODOLOGY}}
|
||||
## Run the Selected Checks
|
||||
|
||||
Use the shared section already loaded above; do not restart its preparation.
|
||||
With its required Reads complete and report ownership resolved, Write the charters
|
||||
into the owned report and wait for the successful
|
||||
Write result before starting any probe clock or baseline. Use `REPORT_FILE`. State each expected result,
|
||||
risk, entrypoint, isolation and exit condition before probing; never invent the plan later.
|
||||
A failed baseline contract stays failed. Before browser probes, source/diff reads only
|
||||
map changes to pages and flows; read `TODOS.md` if present to identify known bugs.
|
||||
During browser discovery, observe behavior without reading source to diagnose it.
|
||||
|
||||
---
|
||||
|
||||
## Output
|
||||
|
||||
Write the report to both local and project-scoped locations:
|
||||
### Assemble the report
|
||||
|
||||
**Local:** `.gstack/qa-reports/qa-report-{domain}-{YYYY-MM-DD}.md`
|
||||
After probing stops, load the finalization procedure below. Use retained evidence;
|
||||
this step does not authorize more probes or restart an expired clock.
|
||||
Do not preload reporting. To recover from an accidental early Read:
|
||||
If already read, issue another Read now and await its
|
||||
acknowledgement, even if the tool reports unchanged content.
|
||||
The no-repeat rule covers preparation Reads, not this finalization Read.
|
||||
|
||||
**Project-scoped:** Write test outcome artifact for cross-session context:
|
||||
```bash
|
||||
{{SLUG_SETUP}}
|
||||
```
|
||||
Write to `~/.gstack/projects/{slug}/{user}-{branch}-test-outcome-{datetime}.md`
|
||||
{{SECTION:reporting}}
|
||||
|
||||
Use templates from this host's installed QA directory. For mixed runs, use separate browser and functional sections in this same report.
|
||||
Keep common metadata once: date, branch/revision, caller/authority, mode, scope and timing/stop reason.
|
||||
Preserve the initial charters under **Charters** after that metadata, before findings.
|
||||
|
||||
- **Browser:** `templates/qa-report-template.md`: targets, URL, framework,
|
||||
page/screenshot counts, findings, health/category scores and regression comparison.
|
||||
- **Functional:** `templates/functional-report-template.md`: native tools/runtime,
|
||||
fixture ownership, contracts, findings, discoveries/proposed tests and cleanup.
|
||||
|
||||
Nest remaining headings per surface, without duplicating the shared title or metadata.
|
||||
Preserve surface-specific scope, timing and coverage limits.
|
||||
Browser scores apply only to browser coverage; never combine them with functional
|
||||
outcomes. In each section link the current baseline or replay evidence and checkpoints;
|
||||
for functional regression the report plus replay evidence is the baseline. Regression
|
||||
also links the prior input baseline/report; missing required replay inputs block affected
|
||||
coverage. Prior baselines are not applicable to Full/Quick. Report-only
|
||||
repair/test fields contain proposals or not-run status, never claims of edits.
|
||||
|
||||
### Write the checked report
|
||||
|
||||
After the reporting procedure's consistency check, write `REPORT_FILE` and the
|
||||
project copy below. These are the default report destinations; a caller's narrower
|
||||
permissions or fixed paths override them. Do not create a forbidden second copy.
|
||||
|
||||
Use this session's existing project slug and state directory for the project copy.
|
||||
If unknown or not writable within the supplied permissions, report that copy as
|
||||
blocked; still write the permitted local report. Do not run state-setup helpers.
|
||||
Write identical content to `~/.gstack/projects/{slug}/{user}-{branch}-test-outcome-{datetime}.md`.
|
||||
Get `{user}`/`{branch}` from `git config user.name`/`git branch --show-current`
|
||||
(fallbacks: `unknown-user`/`detached`); sanitize like `{target}`. Use UTC `YYYYMMDDTHHMMSSZ`.
|
||||
If that destination exists, choose a fresh suffixed filename; never replace a prior report.
|
||||
|
||||
### Output Structure
|
||||
|
||||
```
|
||||
.gstack/qa-reports/
|
||||
├── qa-report-{domain}-{YYYY-MM-DD}.md # Structured report
|
||||
├── screenshots/
|
||||
│ ├── initial.jpg # Landing page screenshot
|
||||
│ ├── issue-001-step-1.jpg # Per-issue evidence
|
||||
│ ├── issue-001-result.jpg
|
||||
│ ├── issue-002.png # Annotated screenshot (static bugs)
|
||||
│ └── ...
|
||||
└── baseline.json # For regression mode
|
||||
```
|
||||
`REPORT_DIR` stays the report root throughout the run. For browser-only and mixed
|
||||
runs, keep screenshots in `$REPORT_DIR/screenshots/` and the browser baseline in
|
||||
`$REPORT_DIR/baseline.json`.
|
||||
The shared loop's mixed-surface split applies only to clocks and checkpoints:
|
||||
|
||||
Report filenames use the domain and date: `qa-report-myapp-com-2026-03-12.md`
|
||||
| Run | Clock/checkpoint directory |
|
||||
|-----|----------------------------|
|
||||
| One surface (browser or functional) | `$REPORT_DIR` |
|
||||
| Mixed: browser probes | `$REPORT_DIR/browser` |
|
||||
| Mixed: functional probes | `$REPORT_DIR/functional` |
|
||||
|
||||
---
|
||||
|
||||
{{LEARNINGS_LOG}}
|
||||
Each probe directory holds its own `exploration-NNN.json` sequence and, only when
|
||||
timed, `deadline.json`. Caller-fixed paths override this layout. Do not reassign
|
||||
`REPORT_DIR` to a surface directory or move the shared browser artifact paths.
|
||||
|
||||
## Additional Rules (qa-only specific)
|
||||
|
||||
11. **Never fix bugs.** Find and document only. Do not read source code, edit files, or suggest fixes in the report. Your job is to report what's broken, not to fix it. Use `/qa` for the test-fix-verify loop.
|
||||
12. **No test framework detected?** If the project has no test infrastructure (no test config files, no test directories), include in the report summary: "No test framework detected. Run `/qa` to bootstrap one and enable regression test generation."
|
||||
1. **Never fix bugs or write product tests.** Find and document only. Necessary read-only
|
||||
source discovery is allowed for functional targets, while browser discovery stays
|
||||
black-box. Do not edit product code, tests, dependencies, config or tracked state
|
||||
through any tool, including shell writes, renames, deletions and edit-then-restore.
|
||||
Never commit, stash or bootstrap. Proposed regressions belong in report artifacts.
|
||||
2. **During preflight, check documented native commands and test infrastructure.** For browser targets, inspect documentation only for this framework check, before discovery. If absent,
|
||||
report missing coverage and proposed cases without installing anything. An unavailable
|
||||
command/service is not a product defect. Never invoke /qa or another skill from this report-only run.
|
||||
When the browser app's repository is available and no framework is documented, say
|
||||
"No test framework detected. Run `/qa` to bootstrap in a separate, user-authorized repair session."
|
||||
Functional targets keep the gap without a new framework.
|
||||
Reference in new issue
Block a user