mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-01 00:49:39 +02:00
Garry Tan
dcaea52800
* feat: add surface-aware exploratory QA and ship documentation gates * test: preserve delegated QA setup authority after main integration * fix(qa): clarify exploration order and preserve report artifacts * test(qa): follow the shared setup reference directly * refactor(ship): make verification and recovery routes explicit * test(ship): align evidence and review guards with explicit routes * fix(workflows): clarify ship recovery and functional QA evidence * fix(workflows): clarify approval recovery and full QA coverage * refactor(workflows): order review transactions and clarify ship state * fix(ship): clarify final verification and fail closed at publication * fix(evals): attribute native atomic documentation writes * fix(ship): clarify recovery and documentation lifecycle guidance * fix(test): preserve observed native placeholder styling in CI * fix(codex): report watchdog timeouts without a process-exit race * Checkpoint functional QA implementation and workflow validation repairs * Fix documentation and shared-review fixture contracts * docs: clarify judge reuse and evaluation supervision * test: align review evidence and selected case contracts * test: verify append-only documentation checkpoints and recovery * fix: qualify QA workflows and CI validation repairs * fix: launch shared-libs fixture scripts on Windows * fix: qualify QA deadlines, fixture isolation, and shard cleanup * fix: preserve qualified QA and cancellation repairs * fix: enforce functional fixture authority and share strict event decoding * fix: retain free-test evidence and explain recovery * fix: reject malformed native evidence after decoder consolidation * test: use reliable capture for telemetry privacy filters * test: refresh measured quick coverage and document validation costs * Fix native fixture receipts and preserve VM validation evidence * Align negative judge controls with upstream clarity policy * Fix report-only QA preparation and public evidence handling * Clarify QA-only preparation and current-report preservation * Stream Ship quality judgments with an explicit 64k response contract * Validate compact judge reasoning locally with supported wire schema * Align functional QA fixture instructions with evidence acceptance * Bind native browser diagnostics to execution evidence and align review verdicts * Preserve native diagnostic line boundaries * Serialize functional QA evidence from native captures * Keep large QA evidence fixture payload out of Windows argv
86 lines
5.0 KiB
Markdown
86 lines
5.0 KiB
Markdown
# QA deadlines
|
||
|
||
Bounded exploratory QA uses `bin/gstack-qa-deadline` from the installed gstack
|
||
runtime. Browser Quick keeps its 30-second limit; browser Full/Regression uses the
|
||
15-minute maximum of its 5–15-minute exploration window. Review/ship smoke keeps its
|
||
5-minute or 12-probe limit, whichever comes first. The workflow enforces the probe
|
||
count; the helper enforces elapsed time. Required plan checks are outside the smoke
|
||
guard: run them after smoke with the same checkpoint sequence and finite command
|
||
timeouts capped by the caller's remaining deadline. An expired caller deadline
|
||
leaves checks not-run; never restart the smoke clock to run them.
|
||
Functional Full/Quick/Regression has no default total exploration deadline; Quick
|
||
limits scope to success plus the highest-risk changed edge. A caller's stricter
|
||
duration or absolute deadline still bounds the run. Standalone mixed runs use
|
||
owned `REPORT_DIR/browser` and `REPORT_DIR/functional` directories for their clocks
|
||
and checkpoints, with one final report at `REPORT_DIR`. Create those directories
|
||
before starting their clocks. Single-surface runs and review/ship smoke keep their
|
||
clock and checkpoints at `REPORT_DIR`; fixed caller paths take precedence.
|
||
|
||
## Command interface
|
||
|
||
Run the helper with Bun. `FILE` is `deadline.json` inside the invocation-owned,
|
||
canonical probe directory; its parent must already exist. Use quoted absolute
|
||
paths in place of `GUARD` and `FILE` below.
|
||
|
||
```text
|
||
bun GUARD start FILE SECONDS [EARLIER_UTC]
|
||
bun GUARD status FILE
|
||
bun GUARD run FILE -- COMMAND ARGS...
|
||
```
|
||
|
||
`start` runs once, immediately before the baseline. It exclusively creates a
|
||
versioned, read-only receipt and clamps the selected duration to an earlier caller
|
||
deadline when supplied. It does not replace an existing file. `status` reads the
|
||
actual clock. `run` checks the same receipt again before launching, then supervises
|
||
the command for the remaining time. Replays and minimization use that same deadline;
|
||
never replace the receipt or restart the timer to finish more work.
|
||
|
||
Arguments are passed directly, without shell evaluation. For a permitted script,
|
||
the child command is `bash -c 'script'`; keep every probe inside that child rather
|
||
than appending an unguarded command after the helper. Missing or malformed state,
|
||
symlinked paths and unavailable process containment block dispatch.
|
||
|
||
The child's stdout/stderr remain its evidence. Guard-owned lines begin with
|
||
`QA_DEADLINE ` and contain separate JSON bookkeeping; do not copy them into the
|
||
checkpoint's observed program JSON. Mark a refused next probe not-run in the report;
|
||
preserve its original checkpoint rather than rewriting it as an observation.
|
||
For JSON-emitting probes, `observed` is the decoded child JSON itself, not a `child`
|
||
envelope or a mixture of results and guard metadata. For other output, retain the
|
||
full child text. Command fields retain the complete outer command, including the
|
||
guard invocation; guard diagnostics and interpretations belong in the report.
|
||
|
||
## Reporting measurements
|
||
|
||
The configured probe budget is not the total session duration. Child launch/finish
|
||
receipts measure guarded command spans; gaps between calls do not measure individual
|
||
tool costs or establish how many probes can fit in another run.
|
||
|
||
`/qa-only` loads its reporting section after probing stops and checks every repeated
|
||
finding against the retained evidence before writing. Its in-progress report marks
|
||
total session elapsed as unmeasured: the final Write and cleanup have not finished.
|
||
Initial charters and final findings use the caller's same report file. Learning notes
|
||
and automatic memory obey the same caller-authorized write destinations.
|
||
An optional measured interval names its actual start/end receipts and excluded work,
|
||
including later report Writes and cleanup; it is not a completed-session measurement.
|
||
|
||
## Exit and cleanup behavior
|
||
|
||
- Guard expiry or timeout returns 124. A child can independently return 124 too;
|
||
use the guard receipt's event and `timedOut` field to distinguish those cases.
|
||
- Guard errors return 2; a missing executable returns 127. Otherwise the child's
|
||
status is preserved.
|
||
- On Linux/macOS, cleanup covers the command's inherited process group. Detached
|
||
or new-session descendants are outside that guarantee, so detached probes are
|
||
unsupported. Force-killing the guard itself with SIGKILL also prevents its POSIX
|
||
cleanup handler from running.
|
||
- On Windows, dedicated nested Jobs contain the probe worker and its descendants,
|
||
including children whose immediate parent exits. Failure to initialize this
|
||
containment prevents the command from starting.
|
||
- Receipt flushing happens after probe cleanup and can take up to five seconds;
|
||
blocked or broken output returns 2. This allowance does not extend probe work.
|
||
|
||
Terminating a probe client does not undo a request already accepted by a service
|
||
or stop an already-running browser. Preserve any known partial effects and report
|
||
uncertain completion instead of assuming cancellation meant no effect. Report
|
||
writing may finish after the exploration deadline, but new probes may not start.
|