mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-15 21:07:20 +01:00
fix(evals): qa protocol names defined for the judge, async PTY judge so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier
This commit is contained in:
1 parent
6e9667067c
commit
b24bb7a49c
11 files changed
+86
-40
No files matched your search
+10
-6
@@ -79,13 +79,17 @@ Set SECONDS to the shorter mode/caller limit; an unlimited mode uses the caller'
|
||||
Without a total time limit, do not start D; announce finite command timeouts.
|
||||
Stop when scoped contracts are tested or blocked.
|
||||
Clocks/checkpoints use REPORT_DIR; mixed standalone runs use REPORT_DIR/browser and REPORT_DIR/functional, with one final report at REPORT_DIR. Caller paths win.
|
||||
R = owned probe directory; D = R/deadline.json. Quote paths.
|
||||
G = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`; Q = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`.
|
||||
Names used below (quote every path):
|
||||
- R: this surface's owned probe directory, chosen above. D: \`R/deadline.json\`, which exists only for a bounded run.
|
||||
- SECONDS: the total probe budget set above. NNN: a fresh three-digit ID (001, 002, ...) for each capture or checkpoint; never reuse one.
|
||||
- G = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-deadline\`, the deadline guard: \`start\` creates D, \`status\` prints \`remainingMs\` and \`expired\`, and \`run\` executes one command, stopping it at the deadline.
|
||||
- Q = \`${quoteSafePath(ctx.paths.binDir)}/gstack-qa-evidence\`, the functional evidence recorder: \`capture\` runs one command, stores its exit code, stdout and stderr under \`R/.qa-evidence/NNN/\`, and prints status \`complete\`, \`incomplete\` or \`sensitive\`.
|
||||
|
||||
Start once before baseline: \`bun G start D SECONDS [EARLIER_UTC]\` if bounded.
|
||||
EARLIER_UTC = caller's absolute deadline, if set.
|
||||
Functional: \`bun Q capture R NNN [--public] --deadline D -- COMMAND ARGS\`.
|
||||
Unbounded: use \`--timeout-ms MS\` instead. Use fresh three-digit IDs.
|
||||
--public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of \`R/.qa-evidence/NNN/observation.json\`. Sensitive/incomplete captures cannot anchor checkpoints.
|
||||
Unbounded: use \`--timeout-ms MS\` instead of \`--deadline D\`.
|
||||
--public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of \`R/.qa-evidence/NNN/observation.json\` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
|
||||
Bounded browsers: \`bun G run D -- COMMAND ARGS\`. No detached probes.
|
||||
Never reset D/bypass G. Expiry or invalid/missing D stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
|
||||
|
||||
@@ -115,8 +119,8 @@ ${reportOnly ? ` For guarded text, copy the complete span between the guard's
|
||||
Preserve every safe program-JSON key/value and identity hash unchanged.
|
||||
Withhold unsafe values, disclose limits and stop that chain.
|
||||
Check fields before publication. No drafts/placeholders or invented safe-path redactions; corrections cannot repair published notes.
|
||||
Functional: the next capture publishes it: \`... --after PREV --hypothesis 'why' -- CMD\` (PREV: last complete capture). Q supplies observed; never transcribe it.
|
||||
Browser checkpoints use Write.
|
||||
Functional: do not write this file; the next capture publishes it: \`bun Q capture R NNN --deadline D --after PREV --hypothesis 'why' -- CMD\` (PREV: the last complete capture's ID). Q supplies observed; never transcribe it.
|
||||
Browser checkpoints use Write: create the four-field file yourself.
|
||||
Wait for successful checkpoint publication before dispatch.
|
||||
Never backfill or overwrite notes.
|
||||
3. Run that exact probe; G enforces the deadline when bounded.
|
||||
|
||||
@@ -173,6 +173,7 @@ Visit every reachable page (5-15 minutes). Score health; document 5-10 evidenced
|
||||
|
||||
### Regression (\`--regression <baseline>\`)
|
||||
Run Full; append fixed/new issues and score delta. Preserve the supplied prior baseline.
|
||||
A missing or unreadable baseline is a missing prerequisite: it blocks the comparison, not the Full run.
|
||||
|
||||
## Workflow
|
||||
|
||||
|
||||
Reference in new issue
Block a user