Files
gstack/docs/reference-qa-deadlines.md
T
Garry Tan dcaea52800 v1.91.7.0 feat: add functional QA and pre-publication docs checks (#2983)
* feat: add surface-aware exploratory QA and ship documentation gates

* test: preserve delegated QA setup authority after main integration

* fix(qa): clarify exploration order and preserve report artifacts

* test(qa): follow the shared setup reference directly

* refactor(ship): make verification and recovery routes explicit

* test(ship): align evidence and review guards with explicit routes

* fix(workflows): clarify ship recovery and functional QA evidence

* fix(workflows): clarify approval recovery and full QA coverage

* refactor(workflows): order review transactions and clarify ship state

* fix(ship): clarify final verification and fail closed at publication

* fix(evals): attribute native atomic documentation writes

* fix(ship): clarify recovery and documentation lifecycle guidance

* fix(test): preserve observed native placeholder styling in CI

* fix(codex): report watchdog timeouts without a process-exit race

* Checkpoint functional QA implementation and workflow validation repairs

* Fix documentation and shared-review fixture contracts

* docs: clarify judge reuse and evaluation supervision

* test: align review evidence and selected case contracts

* test: verify append-only documentation checkpoints and recovery

* fix: qualify QA workflows and CI validation repairs

* fix: launch shared-libs fixture scripts on Windows

* fix: qualify QA deadlines, fixture isolation, and shard cleanup

* fix: preserve qualified QA and cancellation repairs

* fix: enforce functional fixture authority and share strict event decoding

* fix: retain free-test evidence and explain recovery

* fix: reject malformed native evidence after decoder consolidation

* test: use reliable capture for telemetry privacy filters

* test: refresh measured quick coverage and document validation costs

* Fix native fixture receipts and preserve VM validation evidence

* Align negative judge controls with upstream clarity policy

* Fix report-only QA preparation and public evidence handling

* Clarify QA-only preparation and current-report preservation

* Stream Ship quality judgments with an explicit 64k response contract

* Validate compact judge reasoning locally with supported wire schema

* Align functional QA fixture instructions with evidence acceptance

* Bind native browser diagnostics to execution evidence and align review verdicts

* Preserve native diagnostic line boundaries

* Serialize functional QA evidence from native captures

* Keep large QA evidence fixture payload out of Windows argv
2026-09-29 06:07:35 -07:00

5.0 KiB
Raw Blame History

QA deadlines

Bounded exploratory QA uses bin/gstack-qa-deadline from the installed gstack runtime. Browser Quick keeps its 30-second limit; browser Full/Regression uses the 15-minute maximum of its 5–15-minute exploration window. Review/ship smoke keeps its 5-minute or 12-probe limit, whichever comes first. The workflow enforces the probe count; the helper enforces elapsed time. Required plan checks are outside the smoke guard: run them after smoke with the same checkpoint sequence and finite command timeouts capped by the caller's remaining deadline. An expired caller deadline leaves checks not-run; never restart the smoke clock to run them. Functional Full/Quick/Regression has no default total exploration deadline; Quick limits scope to success plus the highest-risk changed edge. A caller's stricter duration or absolute deadline still bounds the run. Standalone mixed runs use owned REPORT_DIR/browser and REPORT_DIR/functional directories for their clocks and checkpoints, with one final report at REPORT_DIR. Create those directories before starting their clocks. Single-surface runs and review/ship smoke keep their clock and checkpoints at REPORT_DIR; fixed caller paths take precedence.

Command interface

Run the helper with Bun. FILE is deadline.json inside the invocation-owned, canonical probe directory; its parent must already exist. Use quoted absolute paths in place of GUARD and FILE below.

bun GUARD start FILE SECONDS [EARLIER_UTC]
bun GUARD status FILE
bun GUARD run FILE -- COMMAND ARGS...

start runs once, immediately before the baseline. It exclusively creates a versioned, read-only receipt and clamps the selected duration to an earlier caller deadline when supplied. It does not replace an existing file. status reads the actual clock. run checks the same receipt again before launching, then supervises the command for the remaining time. Replays and minimization use that same deadline; never replace the receipt or restart the timer to finish more work.

Arguments are passed directly, without shell evaluation. For a permitted script, the child command is bash -c 'script'; keep every probe inside that child rather than appending an unguarded command after the helper. Missing or malformed state, symlinked paths and unavailable process containment block dispatch.

The child's stdout/stderr remain its evidence. Guard-owned lines begin with QA_DEADLINE and contain separate JSON bookkeeping; do not copy them into the checkpoint's observed program JSON. Mark a refused next probe not-run in the report; preserve its original checkpoint rather than rewriting it as an observation. For JSON-emitting probes, observed is the decoded child JSON itself, not a child envelope or a mixture of results and guard metadata. For other output, retain the full child text. Command fields retain the complete outer command, including the guard invocation; guard diagnostics and interpretations belong in the report.

Reporting measurements

The configured probe budget is not the total session duration. Child launch/finish receipts measure guarded command spans; gaps between calls do not measure individual tool costs or establish how many probes can fit in another run.

/qa-only loads its reporting section after probing stops and checks every repeated finding against the retained evidence before writing. Its in-progress report marks total session elapsed as unmeasured: the final Write and cleanup have not finished. Initial charters and final findings use the caller's same report file. Learning notes and automatic memory obey the same caller-authorized write destinations. An optional measured interval names its actual start/end receipts and excluded work, including later report Writes and cleanup; it is not a completed-session measurement.

Exit and cleanup behavior

  • Guard expiry or timeout returns 124. A child can independently return 124 too; use the guard receipt's event and timedOut field to distinguish those cases.
  • Guard errors return 2; a missing executable returns 127. Otherwise the child's status is preserved.
  • On Linux/macOS, cleanup covers the command's inherited process group. Detached or new-session descendants are outside that guarantee, so detached probes are unsupported. Force-killing the guard itself with SIGKILL also prevents its POSIX cleanup handler from running.
  • On Windows, dedicated nested Jobs contain the probe worker and its descendants, including children whose immediate parent exits. Failure to initialize this containment prevents the command from starting.
  • Receipt flushing happens after probe cleanup and can take up to five seconds; blocked or broken output returns 2. This allowance does not extend probe work.

Terminating a probe client does not undo a request already accepted by a service or stop an already-running browser. Preserve any known partial effects and report uncertain completion instead of assuming cancellation meant no effect. Report writing may finish after the exploration deadline, but new probes may not start.