mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-01 17:09:39 +02:00
v1.91.7.0 feat: add functional QA and pre-publication docs checks (#2983)
* feat: add surface-aware exploratory QA and ship documentation gates * test: preserve delegated QA setup authority after main integration * fix(qa): clarify exploration order and preserve report artifacts * test(qa): follow the shared setup reference directly * refactor(ship): make verification and recovery routes explicit * test(ship): align evidence and review guards with explicit routes * fix(workflows): clarify ship recovery and functional QA evidence * fix(workflows): clarify approval recovery and full QA coverage * refactor(workflows): order review transactions and clarify ship state * fix(ship): clarify final verification and fail closed at publication * fix(evals): attribute native atomic documentation writes * fix(ship): clarify recovery and documentation lifecycle guidance * fix(test): preserve observed native placeholder styling in CI * fix(codex): report watchdog timeouts without a process-exit race * Checkpoint functional QA implementation and workflow validation repairs * Fix documentation and shared-review fixture contracts * docs: clarify judge reuse and evaluation supervision * test: align review evidence and selected case contracts * test: verify append-only documentation checkpoints and recovery * fix: qualify QA workflows and CI validation repairs * fix: launch shared-libs fixture scripts on Windows * fix: qualify QA deadlines, fixture isolation, and shard cleanup * fix: preserve qualified QA and cancellation repairs * fix: enforce functional fixture authority and share strict event decoding * fix: retain free-test evidence and explain recovery * fix: reject malformed native evidence after decoder consolidation * test: use reliable capture for telemetry privacy filters * test: refresh measured quick coverage and document validation costs * Fix native fixture receipts and preserve VM validation evidence * Align negative judge controls with upstream clarity policy * Fix report-only QA preparation and public evidence handling * Clarify QA-only preparation and current-report preservation * Stream Ship quality judgments with an explicit 64k response contract * Validate compact judge reasoning locally with supported wire schema * Align functional QA fixture instructions with evidence acceptance * Bind native browser diagnostics to execution evidence and align review verdicts * Preserve native diagnostic line boundaries * Serialize functional QA evidence from native captures * Keep large QA evidence fixture payload out of Windows argv
This commit is contained in:
1 parent
65bfb0ce49
commit
dcaea52800
333 files changed
+41755
-7357
No files matched your search
@@ -0,0 +1,85 @@
|
||||
# QA deadlines
|
||||
|
||||
Bounded exploratory QA uses `bin/gstack-qa-deadline` from the installed gstack
|
||||
runtime. Browser Quick keeps its 30-second limit; browser Full/Regression uses the
|
||||
15-minute maximum of its 5–15-minute exploration window. Review/ship smoke keeps its
|
||||
5-minute or 12-probe limit, whichever comes first. The workflow enforces the probe
|
||||
count; the helper enforces elapsed time. Required plan checks are outside the smoke
|
||||
guard: run them after smoke with the same checkpoint sequence and finite command
|
||||
timeouts capped by the caller's remaining deadline. An expired caller deadline
|
||||
leaves checks not-run; never restart the smoke clock to run them.
|
||||
Functional Full/Quick/Regression has no default total exploration deadline; Quick
|
||||
limits scope to success plus the highest-risk changed edge. A caller's stricter
|
||||
duration or absolute deadline still bounds the run. Standalone mixed runs use
|
||||
owned `REPORT_DIR/browser` and `REPORT_DIR/functional` directories for their clocks
|
||||
and checkpoints, with one final report at `REPORT_DIR`. Create those directories
|
||||
before starting their clocks. Single-surface runs and review/ship smoke keep their
|
||||
clock and checkpoints at `REPORT_DIR`; fixed caller paths take precedence.
|
||||
|
||||
## Command interface
|
||||
|
||||
Run the helper with Bun. `FILE` is `deadline.json` inside the invocation-owned,
|
||||
canonical probe directory; its parent must already exist. Use quoted absolute
|
||||
paths in place of `GUARD` and `FILE` below.
|
||||
|
||||
```text
|
||||
bun GUARD start FILE SECONDS [EARLIER_UTC]
|
||||
bun GUARD status FILE
|
||||
bun GUARD run FILE -- COMMAND ARGS...
|
||||
```
|
||||
|
||||
`start` runs once, immediately before the baseline. It exclusively creates a
|
||||
versioned, read-only receipt and clamps the selected duration to an earlier caller
|
||||
deadline when supplied. It does not replace an existing file. `status` reads the
|
||||
actual clock. `run` checks the same receipt again before launching, then supervises
|
||||
the command for the remaining time. Replays and minimization use that same deadline;
|
||||
never replace the receipt or restart the timer to finish more work.
|
||||
|
||||
Arguments are passed directly, without shell evaluation. For a permitted script,
|
||||
the child command is `bash -c 'script'`; keep every probe inside that child rather
|
||||
than appending an unguarded command after the helper. Missing or malformed state,
|
||||
symlinked paths and unavailable process containment block dispatch.
|
||||
|
||||
The child's stdout/stderr remain its evidence. Guard-owned lines begin with
|
||||
`QA_DEADLINE ` and contain separate JSON bookkeeping; do not copy them into the
|
||||
checkpoint's observed program JSON. Mark a refused next probe not-run in the report;
|
||||
preserve its original checkpoint rather than rewriting it as an observation.
|
||||
For JSON-emitting probes, `observed` is the decoded child JSON itself, not a `child`
|
||||
envelope or a mixture of results and guard metadata. For other output, retain the
|
||||
full child text. Command fields retain the complete outer command, including the
|
||||
guard invocation; guard diagnostics and interpretations belong in the report.
|
||||
|
||||
## Reporting measurements
|
||||
|
||||
The configured probe budget is not the total session duration. Child launch/finish
|
||||
receipts measure guarded command spans; gaps between calls do not measure individual
|
||||
tool costs or establish how many probes can fit in another run.
|
||||
|
||||
`/qa-only` loads its reporting section after probing stops and checks every repeated
|
||||
finding against the retained evidence before writing. Its in-progress report marks
|
||||
total session elapsed as unmeasured: the final Write and cleanup have not finished.
|
||||
Initial charters and final findings use the caller's same report file. Learning notes
|
||||
and automatic memory obey the same caller-authorized write destinations.
|
||||
An optional measured interval names its actual start/end receipts and excluded work,
|
||||
including later report Writes and cleanup; it is not a completed-session measurement.
|
||||
|
||||
## Exit and cleanup behavior
|
||||
|
||||
- Guard expiry or timeout returns 124. A child can independently return 124 too;
|
||||
use the guard receipt's event and `timedOut` field to distinguish those cases.
|
||||
- Guard errors return 2; a missing executable returns 127. Otherwise the child's
|
||||
status is preserved.
|
||||
- On Linux/macOS, cleanup covers the command's inherited process group. Detached
|
||||
or new-session descendants are outside that guarantee, so detached probes are
|
||||
unsupported. Force-killing the guard itself with SIGKILL also prevents its POSIX
|
||||
cleanup handler from running.
|
||||
- On Windows, dedicated nested Jobs contain the probe worker and its descendants,
|
||||
including children whose immediate parent exits. Failure to initialize this
|
||||
containment prevents the command from starting.
|
||||
- Receipt flushing happens after probe cleanup and can take up to five seconds;
|
||||
blocked or broken output returns 2. This allowance does not extend probe work.
|
||||
|
||||
Terminating a probe client does not undo a request already accepted by a service
|
||||
or stop an already-running browser. Preserve any known partial effects and report
|
||||
uncertain completion instead of assuming cancellation meant no effect. Report
|
||||
writing may finish after the exploration deadline, but new probes may not start.
|
||||
Reference in new issue
Block a user