mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-05 02:47:17 +02:00
fix(qa-only,qa): name the exploratory read point and finalization order; judge qa with its browser assets
qa-only judges cited 'next section' pointing at the wrong heading, an exploratory trigger that contradicted its read point, clock ownership in mixed runs and the unstated order of exploratory section 4 vs reporting. The qa judge penalized the absent qa-report-template and issue-taxonomy that qa-patterns loads; with them in, it found issue-taxonomy's dangling 'rule 13' (the consent rule is browser rule 3).
This commit is contained in:
1 parent
23e3636e4c
commit
4dfed0b783
5 files changed
+12
-11
No files matched your search
+5
-5
@@ -418,7 +418,7 @@ Read sections in full when directed; do not work from memory.
|
|||||||
|
|
||||||
| When | Read this section |
|
| When | Read this section |
|
||||||
|------|-------------------|
|
|------|-------------------|
|
||||||
| running selected report-only baseline and exploratory probes without product or test writes | `sections/exploratory.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory |
|
| selecting surfaces and later running the selected report-only probes without product or test writes (one Read covers both) | `sections/exploratory.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory |
|
||||||
| finalizing the report after probing stops | `sections/reporting.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory |
|
| finalizing the report after probing stops | `sections/reporting.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory |
|
||||||
|
|
||||||
Start at Request Parameters, then follow the sections below in order.
|
Start at Request Parameters, then follow the sections below in order.
|
||||||
@@ -464,11 +464,11 @@ the current behavior. Reading old notes never requires writing new ones.
|
|||||||
|
|
||||||
## Select Surfaces and Isolation
|
## Select Surfaces and Isolation
|
||||||
|
|
||||||
Load the shared preparation gate now (next section): complete its scope and selected-method Reads,
|
Load the shared preparation gate now (the exploratory STOP just below): complete its scope and selected-method Reads,
|
||||||
await their results, and select the surfaces. Defer charters, clocks and probes to
|
await their results, and select the surfaces. Defer charters, clocks and probes to
|
||||||
Run the Selected Checks, after report ownership and conditional browser setup below.
|
Run the Selected Checks, after report ownership and conditional browser setup below.
|
||||||
|
|
||||||
> **STOP.** Before running selected report-only baseline and exploratory probes without product or test writes, Read `sections/exploratory.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory in full and follow it.
|
> **STOP.** Before selecting surfaces and later running the selected report-only probes without product or test writes (one Read covers both), Read `sections/exploratory.md` relative to the installed `qa-only`/`gstack-qa-only` SKILL.md directory in full and follow it.
|
||||||
> Use this host's installed path, never the product working directory or another host's assets.
|
> Use this host's installed path, never the product working directory or another host's assets.
|
||||||
> If missing or unreadable, report a QA setup blocker and its affected probes as blocked; continue other safe probes (independent functional/static checks). Missing/unreadable assets block required QA.
|
> If missing or unreadable, report a QA setup blocker and its affected probes as blocked; continue other safe probes (independent functional/static checks). Missing/unreadable assets block required QA.
|
||||||
|
|
||||||
@@ -478,7 +478,7 @@ For mixed Regression, the argument is the prior combined report. Resolve its fun
|
|||||||
replay evidence and browser baseline links first, then give each method its own baseline.
|
replay evidence and browser baseline links first, then give each method its own baseline.
|
||||||
A missing baseline blocks that surface's regression coverage, not independent checks.
|
A missing baseline blocks that surface's regression coverage, not independent checks.
|
||||||
In mixed runs, use the user's surface order, defaulting to functional then browser.
|
In mixed runs, use the user's surface order, defaulting to functional then browser.
|
||||||
Finish one surface's probes before starting the next surface's clock; any supplied
|
Each surface keeps its own clock in its own directory. Finish one surface's probes before starting the next surface's clock; any supplied
|
||||||
absolute deadline still applies to both. Do not reset a clock when switching surfaces.
|
absolute deadline still applies to both. Do not reset a clock when switching surfaces.
|
||||||
|
|
||||||
## Prepare Report Artifacts
|
## Prepare Report Artifacts
|
||||||
@@ -532,7 +532,7 @@ During browser discovery, observe behavior without reading source to diagnose it
|
|||||||
|
|
||||||
### Assemble the report
|
### Assemble the report
|
||||||
|
|
||||||
After probing stops, load the finalization procedure below. Use retained evidence;
|
After probing stops, load the finalization procedure below. Order: exploratory §4 annotations and materialize (browser-only runs pass an empty evidence list), then this procedure, then the final report Write. Use retained evidence;
|
||||||
this step does not authorize more probes or restart an expired clock.
|
this step does not authorize more probes or restart an expired clock.
|
||||||
Do not preload reporting. To recover from an accidental early Read:
|
Do not preload reporting. To recover from an accidental early Read:
|
||||||
If already read, issue another Read now and await its
|
If already read, issue another Read now and await its
|
||||||
|
|||||||
@@ -71,7 +71,7 @@ If neither exists, use git diff analysis.
|
|||||||
|
|
||||||
## Select Surfaces and Isolation
|
## Select Surfaces and Isolation
|
||||||
|
|
||||||
Load the shared preparation gate now (next section): complete its scope and selected-method Reads,
|
Load the shared preparation gate now (the exploratory STOP just below): complete its scope and selected-method Reads,
|
||||||
await their results, and select the surfaces. Defer charters, clocks and probes to
|
await their results, and select the surfaces. Defer charters, clocks and probes to
|
||||||
Run the Selected Checks, after report ownership and conditional browser setup below.
|
Run the Selected Checks, after report ownership and conditional browser setup below.
|
||||||
|
|
||||||
@@ -83,7 +83,7 @@ For mixed Regression, the argument is the prior combined report. Resolve its fun
|
|||||||
replay evidence and browser baseline links first, then give each method its own baseline.
|
replay evidence and browser baseline links first, then give each method its own baseline.
|
||||||
A missing baseline blocks that surface's regression coverage, not independent checks.
|
A missing baseline blocks that surface's regression coverage, not independent checks.
|
||||||
In mixed runs, use the user's surface order, defaulting to functional then browser.
|
In mixed runs, use the user's surface order, defaulting to functional then browser.
|
||||||
Finish one surface's probes before starting the next surface's clock; any supplied
|
Each surface keeps its own clock in its own directory. Finish one surface's probes before starting the next surface's clock; any supplied
|
||||||
absolute deadline still applies to both. Do not reset a clock when switching surfaces.
|
absolute deadline still applies to both. Do not reset a clock when switching surfaces.
|
||||||
|
|
||||||
## Prepare Report Artifacts
|
## Prepare Report Artifacts
|
||||||
@@ -137,7 +137,7 @@ During browser discovery, observe behavior without reading source to diagnose it
|
|||||||
|
|
||||||
### Assemble the report
|
### Assemble the report
|
||||||
|
|
||||||
After probing stops, load the finalization procedure below. Use retained evidence;
|
After probing stops, load the finalization procedure below. Order: exploratory §4 annotations and materialize (browser-only runs pass an empty evidence list), then this procedure, then the final report Write. Use retained evidence;
|
||||||
this step does not authorize more probes or restart an expired clock.
|
this step does not authorize more probes or restart an expired clock.
|
||||||
Do not preload reporting. To recover from an accidental early Read:
|
Do not preload reporting. To recover from an accidental early Read:
|
||||||
If already read, issue another Read now and await its
|
If already read, issue another Read now and await its
|
||||||
|
|||||||
@@ -7,7 +7,7 @@
|
|||||||
"id": "exploratory",
|
"id": "exploratory",
|
||||||
"file": "exploratory.md",
|
"file": "exploratory.md",
|
||||||
"title": "Report-only exploratory QA",
|
"title": "Report-only exploratory QA",
|
||||||
"trigger": "running selected report-only baseline and exploratory probes without product or test writes"
|
"trigger": "selecting surfaces and later running the selected report-only probes without product or test writes (one Read covers both)"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "reporting",
|
"id": "reporting",
|
||||||
|
|||||||
@@ -77,7 +77,7 @@ For each page visited during a QA session:
|
|||||||
|
|
||||||
1. **Visual scan** — Take a screenshot (the Read-a-page script; `annotatedScreenshot(pg)` when you need ref labels). Look for layout issues, broken images, alignment.
|
1. **Visual scan** — Take a screenshot (the Read-a-page script; `annotatedScreenshot(pg)` when you need ref labels). Look for layout issues, broken images, alignment.
|
||||||
2. **Interactive elements** — Click every button, link, and control. Does each do what it says?
|
2. **Interactive elements** — Click every button, link, and control. Does each do what it says?
|
||||||
3. **Forms** — Fill and submit (non-local target: consent first — rule 13). Test empty submission, invalid data, edge cases (long text, special characters).
|
3. **Forms** — Fill and submit (non-local target: consent first — browser rule 3). Test empty submission, invalid data, edge cases (long text, special characters).
|
||||||
4. **Navigation** — Check all paths in/out. Breadcrumbs, back button, deep links, mobile menu.
|
4. **Navigation** — Check all paths in/out. Breadcrumbs, back button, deep links, mobile menu.
|
||||||
5. **States** — Check empty state, loading state, error state, full/overflow state.
|
5. **States** — Check empty state, loading state, error state, full/overflow state.
|
||||||
6. **Console** — Print `CONSOLE_ERRORS=` after interactions. Any new JS errors or failed requests?
|
6. **Console** — Print `CONSOLE_ERRORS=` after interactions. Any new JS errors or failed requests?
|
||||||
|
|||||||
@@ -203,7 +203,8 @@ describeIfSelected('QA skill quality evals', ['qa/SKILL.md workflow', 'qa/SKILL.
|
|||||||
const t0 = Date.now();
|
const t0 = Date.now();
|
||||||
const section = readWorkflowJudgeInput({ root: ROOT, skillPath: 'qa/SKILL.md',
|
const section = readWorkflowJudgeInput({ root: ROOT, skillPath: 'qa/SKILL.md',
|
||||||
startMarker: '# /qa: Test', endMarker: null,
|
startMarker: '# /qa: Test', endMarker: null,
|
||||||
references: ['qa/templates/functional-report-template.md'] }).text;
|
// qa-patterns.md loads both browser assets; judges penalized their absence.
|
||||||
|
references: ['qa/templates/functional-report-template.md', 'qa/templates/qa-report-template.md', 'qa/references/issue-taxonomy.md'] }).text;
|
||||||
|
|
||||||
const samples = await judgePanel(() => callJudge<JudgeScore>(`You are evaluating the quality of a QA testing workflow document for an AI coding agent.
|
const samples = await judgePanel(() => callJudge<JudgeScore>(`You are evaluating the quality of a QA testing workflow document for an AI coding agent.
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user