mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-08 09:48:20 +01:00
W8 T11: descriptive QA probe names (PROBE_DIR, DEADLINE_FILE, DEADLINE_TOOL, EVIDENCE_TOOL) plus a three-line probe summary; structural name checks replace alias byte pins
This commit is contained in:
1 parent
dfb7330a8c
commit
5b88eb798a
6 files changed
+85
-36
No files matched your search
+18
-15
@@ -76,30 +76,33 @@ For /qa and /qa-only:
|
||||
- Browser Quick: SECONDS=30. Browser Full/Regression: SECONDS=900.
|
||||
- Functional Full, Quick and Regression have no default total timer.
|
||||
Set SECONDS to the shorter mode/caller limit; an unlimited mode uses the caller's bound.
|
||||
Without a total time limit, do not start D; announce finite command timeouts.
|
||||
Without a total time limit, do not create DEADLINE_FILE; announce finite command timeouts.
|
||||
Stop when scoped contracts are tested or blocked.
|
||||
Clocks/checkpoints use REPORT_DIR; mixed standalone runs use REPORT_DIR/browser and REPORT_DIR/functional, with one final report at REPORT_DIR. Caller paths win.
|
||||
Names used below (quote every path):
|
||||
- R: this surface's owned probe directory, chosen above. D: \`R/deadline.json\`, which exists only for a bounded run.
|
||||
- PROBE_DIR: this surface's owned probe directory, chosen above. DEADLINE_FILE: \`PROBE_DIR/deadline.json\`, which exists only for a bounded run.
|
||||
- SECONDS: the total probe budget set above. NNN: a fresh three-digit ID (001, 002, ...) for each capture or checkpoint; never reuse one.
|
||||
- G = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`, the deadline guard: \`start\` creates D, \`status\` prints \`remainingMs\` and \`expired\`, and \`run\` executes one command, stopping it at the deadline.
|
||||
- Q = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`, the functional evidence recorder: \`capture\` runs one command, stores its exit code, stdout and stderr under \`R/.qa-evidence/NNN/\`, and prints status \`complete\`, \`incomplete\` or \`sensitive\`.
|
||||
- DEADLINE_TOOL = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`, the deadline guard: \`start\` creates DEADLINE_FILE, \`status\` prints \`remainingMs\` and \`expired\`, and \`run\` executes one command, stopping it at the deadline.
|
||||
- EVIDENCE_TOOL = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`, the functional evidence recorder: \`capture\` runs one command, stores its exit code, stdout and stderr under \`PROBE_DIR/.qa-evidence/NNN/\`, and prints status \`complete\`, \`incomplete\` or \`sensitive\`.
|
||||
|
||||
Start once before baseline: \`bun G start D SECONDS [EARLIER_UTC]\` if bounded.
|
||||
Start once before baseline: \`bun DEADLINE_TOOL start DEADLINE_FILE SECONDS [EARLIER_UTC]\` if bounded.
|
||||
EARLIER_UTC = caller's absolute deadline, if set.
|
||||
Functional: \`bun Q capture R NNN [--public] --deadline D -- COMMAND ARGS\`.
|
||||
Unbounded: use \`--timeout-ms MS\` instead of \`--deadline D\`.
|
||||
--public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of \`R/.qa-evidence/NNN/observation.json\` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
|
||||
Bounded browsers: \`bun G run D -- COMMAND ARGS\`. No detached probes.
|
||||
Never reset D/bypass G. Expiry or invalid/missing D stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
|
||||
Functional: \`bun EVIDENCE_TOOL capture PROBE_DIR NNN [--public] --deadline DEADLINE_FILE -- COMMAND ARGS\`.
|
||||
Unbounded: use \`--timeout-ms MS\` instead of \`--deadline DEADLINE_FILE\`.
|
||||
--public requires approved public/synthetic output; EVIDENCE_TOOL screens credentials. For complete private captures, await a safe Read of \`PROBE_DIR/.qa-evidence/NNN/observation.json\` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
|
||||
Bounded browsers: \`bun DEADLINE_TOOL run DEADLINE_FILE -- COMMAND ARGS\`. No detached probes.
|
||||
Never reset DEADLINE_FILE/bypass DEADLINE_TOOL. Expiry or an invalid/missing DEADLINE_FILE stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
|
||||
|
||||
## 2. Probe loop
|
||||
|
||||
Each probe is one native command/interaction plus checks, excluding bookkeeping.
|
||||
Never batch probes.
|
||||
How one probe works: (1) if bounded, check the clock and choose the next command;
|
||||
(2) publish a checkpoint naming it (functional: \`capture --after\` publishes it);
|
||||
(3) run exactly that command via DEADLINE_TOOL or EVIDENCE_TOOL, read its result, repeat.
|
||||
|
||||
1. First demonstrate success: output AND durable effects. Guard if bounded; await completion.
|
||||
2. **Decide whether another probe is needed.** If bounded, run \`bun G status D\`.
|
||||
2. **Decide whether another probe is needed.** If bounded, run \`bun DEADLINE_TOOL status DEADLINE_FILE\`.
|
||||
If expired or no safe next probe remains, STOP exploration; write the report, not a checkpoint.
|
||||
${reportOnly ? ` **Classify the last result before copying it.** For public or synthetic observations,
|
||||
retain the entire result unchanged, including owned fixture paths, IDs, hashes and
|
||||
@@ -119,11 +122,11 @@ ${reportOnly ? ` For guarded text, copy the complete span between the guard's
|
||||
Preserve every safe program-JSON key/value and identity hash unchanged.
|
||||
Withhold unsafe values, disclose limits and stop that chain.
|
||||
Check fields before publication. No drafts/placeholders or invented safe-path redactions; corrections cannot repair published notes.
|
||||
Functional: do not write this file; the next capture publishes it: \`bun Q capture R NNN --deadline D --after PREV --hypothesis 'why' -- CMD\` (PREV: the last complete capture's ID). Q supplies observed; never transcribe it.
|
||||
Functional: do not write this file; the next capture publishes it: \`bun EVIDENCE_TOOL capture PROBE_DIR NNN --deadline DEADLINE_FILE --after PREV --hypothesis 'why' -- CMD\` (PREV: the last complete capture's ID). EVIDENCE_TOOL supplies observed; never transcribe it.
|
||||
Browser checkpoints use Write: create the four-field file yourself.
|
||||
Wait for successful checkpoint publication before dispatch.
|
||||
Never backfill or overwrite notes.
|
||||
3. Run that exact probe; G enforces the deadline when bounded.
|
||||
3. Run that exact probe; DEADLINE_TOOL enforces the deadline when bounded.
|
||||
Report refusals as not-run; retain initial state/inputs/results. Repeat from step 2.
|
||||
4. Replay the exact failing command/request from the same initial fixture state via steps 2–3 (same native command, fresh capture ID)
|
||||
${reportOnly ? 'to confirm it' : 'before repair'}, then minimize via those gates. Expiry leaves confirmation/minimization incomplete.
|
||||
@@ -147,7 +150,7 @@ Never freeze buggy output, weaken tests or delete valid red tests.`}
|
||||
## 4. Final report
|
||||
|
||||
Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests.
|
||||
Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun Q materialize R annotations.json\` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence.
|
||||
Write PROBE_DIR/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown \`bun EVIDENCE_TOOL materialize PROBE_DIR annotations.json\` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence.
|
||||
Evidence is invocation-local${reportOnly ? '.' : '; /ship reruns once per invocation.'}
|
||||
Missing prerequisites/expectations/observations, timeouts and refusal never pass.
|
||||
Pass requires all required current-input contracts to pass with no required remainder.
|
||||
@@ -260,7 +263,7 @@ Never install, import cookies or bootstrap tests. Functional-only skips browser
|
||||
|
||||
**3. Run smoke and plan checks.**
|
||||
Follow the shared Probe loop for smoke checks and replays until the smoke limit.
|
||||
Then run required plan checks and revalidation, even after smoke expires, using the same procedure but no smoke guard; never reset the clock. Their checkpoints sit beside D; they skip \`G status D\` and use \`--timeout-ms\`, not \`--deadline D\`. Post-expiry smoke rechecks are not-run.
|
||||
Then run required plan checks and revalidation, even after smoke expires, using the same procedure but no smoke guard; never reset the clock. Their checkpoints sit beside DEADLINE_FILE; they skip \`DEADLINE_TOOL status DEADLINE_FILE\` and use \`--timeout-ms\`, not \`--deadline DEADLINE_FILE\`. Post-expiry smoke rechecks are not-run.
|
||||
Use finite command timeouts, capped at the caller's remaining time if it has a deadline.${ship ? '' : ' /review sets none; only an invoker-supplied EARLIER_UTC counts.'}
|
||||
Await clock/guard results before acting. When the caller's deadline expires, mark unfinished checks not-run.
|
||||
|
||||
|
||||
Reference in new issue
Block a user