Commit Graph
6 Commits
Author SHA1 Message Date
garrytan ee929ff710 feat(qa): helpers answer --help, and the QA eval interfaces declare it
Approved by Garry: asking gstack-qa-evidence or gstack-qa-deadline for usage
is read-only, so both helpers print usage and exit 0 on --help (the evidence
usage now names the annotation shape), and the functional and caller command
allowlists accept exactly 'bun <path>/bin/gstack-qa-{evidence,deadline} --help'.
Two CI runs failed only on that call.
2026-09-30 19:14:59 +00:00
garrytan 1643cd94de fix(qa-evidence,observer): reject placeholder metadata and replay-only learning; declare the docs atomic-write target
- materialize measures revision, runtime and cwd itself and rejects supplied
  values that differ (CI run wrote revision "HEAD" and runtime "bun"), and
  refuses learning checkpoints that replay the same probe, naming the fix.
- The docs write observer treats Claude Code's atomic temp for the authorized
  doc target as transient, so a temp renamed before its per-file watch no
  longer marks the observation incomplete (ship-docsync-completion flake).
  Per-file monitoring outside declared targets stays fail-closed.
2026-09-30 18:24:46 +00:00
garrytan a7872aaae0 feat(qa-evidence): enforce the checkpoint sequence and fill report bookkeeping in code
- capture refuses to run another probe until a checkpoint anchored on the
  latest complete capture names this capture as its next command, and every
  complete capture prints that requirement.
- materialize fills revision, runtime, cwd and learning (checkpoints whose next
  native command differs) when omitted and prints the reportLinks the report
  must include; the QA section shrinks accordingly.
2026-09-30 17:40:33 +00:00
garrytan 79092b22ea fix(evals): cut path variance at its measured sources
- gstack-qa-evidence capture prints startedAt/completedAt/durationMs and, for
  --deadline captures, remainingMs; the functional report takes durations from
  them. The section clock notice asks for one clock read up front instead of one
  after every checkpoint (QA runs spent 7-14% of tool calls on date -u).
- ship plan-completion: skip the audit dispatch when discovery already found no
  plan (the dispatch-vs-skip conflict produced an optional 60-100 s subagent).
- materialize/checkpoint validation errors state the expected schema, so a
  rejected annotations file is fixable in one call instead of blocking the phase.
- session-runner counts turns from the transcript when a run times out, so
  timeouts stop reporting 'turn 0'.
2026-09-30 15:59:58 +00:00
garrytan 8dd3c44fc6 fix(qa): checkpoint receipts print the report link for their exploration file
qa-functional-webhook-report failed in two of three censuses because the
report linked .qa-evidence/NNN capture folders as "checkpoints" and never
linked exploration-NNN.json. The checkpoint receipt now prints
link: [checkpoint NNN](exploration-NNN.json), and the functional report
template says capture folders are not checkpoints.
2026-09-29 21:49:46 +00:00
Garry Tan dcaea52800 v1.91.7.0 feat: add functional QA and pre-publication docs checks (#2983)
* feat: add surface-aware exploratory QA and ship documentation gates

* test: preserve delegated QA setup authority after main integration

* fix(qa): clarify exploration order and preserve report artifacts

* test(qa): follow the shared setup reference directly

* refactor(ship): make verification and recovery routes explicit

* test(ship): align evidence and review guards with explicit routes

* fix(workflows): clarify ship recovery and functional QA evidence

* fix(workflows): clarify approval recovery and full QA coverage

* refactor(workflows): order review transactions and clarify ship state

* fix(ship): clarify final verification and fail closed at publication

* fix(evals): attribute native atomic documentation writes

* fix(ship): clarify recovery and documentation lifecycle guidance

* fix(test): preserve observed native placeholder styling in CI

* fix(codex): report watchdog timeouts without a process-exit race

* Checkpoint functional QA implementation and workflow validation repairs

* Fix documentation and shared-review fixture contracts

* docs: clarify judge reuse and evaluation supervision

* test: align review evidence and selected case contracts

* test: verify append-only documentation checkpoints and recovery

* fix: qualify QA workflows and CI validation repairs

* fix: launch shared-libs fixture scripts on Windows

* fix: qualify QA deadlines, fixture isolation, and shard cleanup

* fix: preserve qualified QA and cancellation repairs

* fix: enforce functional fixture authority and share strict event decoding

* fix: retain free-test evidence and explain recovery

* fix: reject malformed native evidence after decoder consolidation

* test: use reliable capture for telemetry privacy filters

* test: refresh measured quick coverage and document validation costs

* Fix native fixture receipts and preserve VM validation evidence

* Align negative judge controls with upstream clarity policy

* Fix report-only QA preparation and public evidence handling

* Clarify QA-only preparation and current-report preservation

* Stream Ship quality judgments with an explicit 64k response contract

* Validate compact judge reasoning locally with supported wire schema

* Align functional QA fixture instructions with evidence acceptance

* Bind native browser diagnostics to execution evidence and align review verdicts

* Preserve native diagnostic line boundaries

* Serialize functional QA evidence from native captures

* Keep large QA evidence fixture payload out of Windows argv
2026-09-29 06:07:35 -07:00