W8 T11: descriptive QA probe names (PROBE_DIR, DEADLINE_FILE, DEADLINE_TOOL, EVIDENCE_TOOL) plus a three-line probe summary; structural name checks replace alias byte pins

This commit is contained in:
garrytan committed 2026-10-03 20:53:29 +00:00
1 parent dfb7330a8c
commit 5b88eb798a
6 files changed
+85 -36

No files matched your search

+18 -15
View File
@@ -76,30 +76,33 @@ For /qa and /qa-only:
- Browser Quick: SECONDS=30. Browser Full/Regression: SECONDS=900.
- Functional Full, Quick and Regression have no default total timer.
Set SECONDS to the shorter mode/caller limit; an unlimited mode uses the caller's bound.
Without a total time limit, do not start D; announce finite command timeouts.
Without a total time limit, do not create DEADLINE_FILE; announce finite command timeouts.
Stop when scoped contracts are tested or blocked.
Clocks/checkpoints use REPORT_DIR; mixed standalone runs use REPORT_DIR/browser and REPORT_DIR/functional, with one final report at REPORT_DIR. Caller paths win.
Names used below (quote every path):
- R: this surface's owned probe directory, chosen above. D: \`R/deadline.json\`, which exists only for a bounded run.
- PROBE_DIR: this surface's owned probe directory, chosen above. DEADLINE_FILE: \`PROBE_DIR/deadline.json\`, which exists only for a bounded run.
- SECONDS: the total probe budget set above. NNN: a fresh three-digit ID (001, 002, ...) for each capture or checkpoint; never reuse one.
- G = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`, the deadline guard: \`start\` creates D, \`status\` prints \`remainingMs\` and \`expired\`, and \`run\` executes one command, stopping it at the deadline.
- Q = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`, the functional evidence recorder: \`capture\` runs one command, stores its exit code, stdout and stderr under \`R/.qa-evidence/NNN/\`, and prints status \`complete\`, \`incomplete\` or \`sensitive\`.
- DEADLINE_TOOL = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`, the deadline guard: \`start\` creates DEADLINE_FILE, \`status\` prints \`remainingMs\` and \`expired\`, and \`run\` executes one command, stopping it at the deadline.
- EVIDENCE_TOOL = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`, the functional evidence recorder: \`capture\` runs one command, stores its exit code, stdout and stderr under \`PROBE_DIR/.qa-evidence/NNN/\`, and prints status \`complete\`, \`incomplete\` or \`sensitive\`.
Start once before baseline: \`bun G start D SECONDS [EARLIER_UTC]\` if bounded.
Start once before baseline: \`bun DEADLINE_TOOL start DEADLINE_FILE SECONDS [EARLIER_UTC]\` if bounded.
EARLIER_UTC = caller's absolute deadline, if set.
Functional: \`bun Q capture R NNN [--public] --deadline D -- COMMAND ARGS\`.
Unbounded: use \`--timeout-ms MS\` instead of \`--deadline D\`.
--public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of \`R/.qa-evidence/NNN/observation.json\` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
Bounded browsers: \`bun G run D -- COMMAND ARGS\`. No detached probes.
Never reset D/bypass G. Expiry or invalid/missing D stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
Functional: \`bun EVIDENCE_TOOL capture PROBE_DIR NNN [--public] --deadline DEADLINE_FILE -- COMMAND ARGS\`.
Unbounded: use \`--timeout-ms MS\` instead of \`--deadline DEADLINE_FILE\`.
--public requires approved public/synthetic output; EVIDENCE_TOOL screens credentials. For complete private captures, await a safe Read of \`PROBE_DIR/.qa-evidence/NNN/observation.json\` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
Bounded browsers: \`bun DEADLINE_TOOL run DEADLINE_FILE -- COMMAND ARGS\`. No detached probes.
Never reset DEADLINE_FILE/bypass DEADLINE_TOOL. Expiry or an invalid/missing DEADLINE_FILE stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
## 2. Probe loop
Each probe is one native command/interaction plus checks, excluding bookkeeping.
Never batch probes.
How one probe works: (1) if bounded, check the clock and choose the next command;
(2) publish a checkpoint naming it (functional: \`capture --after\` publishes it);
(3) run exactly that command via DEADLINE_TOOL or EVIDENCE_TOOL, read its result, repeat.
1. First demonstrate success: output AND durable effects. Guard if bounded; await completion.
2. **Decide whether another probe is needed.** If bounded, run \`bun G status D\`.
2. **Decide whether another probe is needed.** If bounded, run \`bun DEADLINE_TOOL status DEADLINE_FILE\`.
If expired or no safe next probe remains, STOP exploration; write the report, not a checkpoint.
${reportOnly ? ` **Classify the last result before copying it.** For public or synthetic observations,
retain the entire result unchanged, including owned fixture paths, IDs, hashes and
@@ -119,11 +122,11 @@ ${reportOnly ? ` For guarded text, copy the complete span between the guard's
Preserve every safe program-JSON key/value and identity hash unchanged.
Withhold unsafe values, disclose limits and stop that chain.
Check fields before publication. No drafts/placeholders or invented safe-path redactions; corrections cannot repair published notes.
Functional: do not write this file; the next capture publishes it: \`bun Q capture R NNN --deadline D --after PREV --hypothesis 'why' -- CMD\` (PREV: the last complete capture's ID). Q supplies observed; never transcribe it.
Functional: do not write this file; the next capture publishes it: \`bun EVIDENCE_TOOL capture PROBE_DIR NNN --deadline DEADLINE_FILE --after PREV --hypothesis 'why' -- CMD\` (PREV: the last complete capture's ID). EVIDENCE_TOOL supplies observed; never transcribe it.
Browser checkpoints use Write: create the four-field file yourself.
Wait for successful checkpoint publication before dispatch.
Never backfill or overwrite notes.
3. Run that exact probe; G enforces the deadline when bounded.
3. Run that exact probe; DEADLINE_TOOL enforces the deadline when bounded.
Report refusals as not-run; retain initial state/inputs/results. Repeat from step 2.
4. Replay the exact failing command/request from the same initial fixture state via steps 2–3 (same native command, fresh capture ID)
${reportOnly ? 'to confirm it' : 'before repair'}, then minimize via those gates. Expiry leaves confirmation/minimization incomplete.
@@ -147,7 +150,7 @@ Never freeze buggy output, weaken tests or delete valid red tests.`}
## 4. Final report
Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests.
Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun Q materialize R annotations.json\` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence.
Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json\` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence.
Evidence is invocation-local${reportOnly ? '.' : '; /ship reruns once per invocation.'}
Missing prerequisites/expectations/observations, timeouts and refusal never pass.
Pass requires all required current-input contracts to pass with no required remainder.
@@ -260,7 +263,7 @@ Never install, import cookies or bootstrap tests. Functional-only skips browser
**3. Run smoke and plan checks.**
Follow the shared Probe loop for smoke checks and replays until the smoke limit.
Then run required plan checks and revalidation, even after smoke expires, using the same procedure but no smoke guard; never reset the clock. Their checkpoints sit beside D; they skip \`G status D\` and use \`--timeout-ms\`, not \`--deadline D\`. Post-expiry smoke rechecks are not-run.
Then run required plan checks and revalidation, even after smoke expires, using the same procedure but no smoke guard; never reset the clock. Their checkpoints sit beside DEADLINE_FILE; they skip \`DEADLINE_TOOL status DEADLINE_FILE\` and use \`--timeout-ms\`, not \`--deadline DEADLINE_FILE\`. Post-expiry smoke rechecks are not-run.
Use finite command timeouts, capped at the caller's remaining time if it has a deadline.${ship ? '' : ' /review sets none; only an invoker-supplied EARLIER_UTC counts.'}
Await clock/guard results before acting. When the caller's deadline expires, mark unfinished checks not-run.