Files
gstack/qa-only/sections/exploratory.md
T
ac20ef1e13 v1.91.16.0 fix: safe installs and upgrades, skills start everywhere, truthful completion (parallel wave) (#3014)
* fix(make-pdf,design): real TOC page numbers and anchors; design models move to gpt-5.5 + gpt-image-2 (#2903, #2807)

* ci: pin every action to a SHA, Scorecard, checksummed Bun, scoped Codex CLI in the CI image, macOS named-regressions job (#1948, #1706, #1997)

* fix(cso,browse): Windows/macOS lease identity, retention skips bad runs, truthful 'not assessed' reports, env.kv false positive, Windows browse spawn fixes (#2894, #2912, #2048, #2637, #2805)

* fix(outside-voice): one Codex model precedence for probe and dispatch, bounded timeouts with KILL escalation, probe cache identity, guarded mktemp, skills hidden in nested Codex (#2776, #2914, #2787, #2881, #2847)

* feat(setup): install registry, ./setup --status, staged activation, per-scope migrations, host tiers and conformance kit, OpenCode/Hermes fixes, Copilot CLI (experimental), disabled_skills, slimmer CLAUDE.md (#1925, #2879, #2347, #2338, #2626, #2629, #2825, #2015, #393, #1206, #2096)

Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com>
Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com>
Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>

* fix(preamble): skills start in worktree-isolated Claude Code sessions — one literal skill-start line, context-recovery helper, no eval in skill blocks, $N token tripwire, honest plan-mode precedence (#2763, #2896, #2851)

* fix(setup): per-install renders for renamed, vendored and custom-config installs, space-safe alias roots, per-host /cso launcher path (#1882, #2906)

* test: free tests never write the real HOME (home-write tripwire), no orphaned browse daemons, symlinked-TMPDIR (macOS) fixes, gbrain-on-PATH isolation (#2895, #2816, #2829, #2841, #2982, #2911, #2909, #2908, #2910)

* chore: integrate wave follow-ups — regenerate all hosts, README model docs, TODOS, touchfiles, duration seeds, Bun 1.3 telemetry test fix, replay fixtures pinned to their recorded bytes

* v1.91.14.0: release metadata

* test(browse): group-kill cleanup falls back to the leader when macOS returns EPERM for a live group

* ci: Codex login steps run under bash (the CI container's default sh rejects pipefail)

* fix: full-suite repairs — private state roots for 7 real-HOME writers, truthful home-guard attribution with --attribute-home, gstack-paths shim in ship-hook fixtures, Copilot in team-init probes, test-of-test detector, cookie review fixture pinned to the approved excerpt

* fix(windows): LF for Dockerfiles, normalized home-guard exclusions, path.relative fixture containment, ship push test spawn timeouts

* test(design-detect): widen the whole-scan budget test's timing margins so a loaded macOS runner can't time out the single-batch call

* fix(evals): qa protocol names defined for the judge, async PTY judge so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier

* test(pty): the non-blocking judge case is POSIX-only (its fake judge is a shebang script, like the PTY harness it covers)

* test(evals): ship-coverage-audit names absolute skill paths, uses Read, and skips the Step 7 artifact nothing consumes (removes the mechanism behind 3 of 4 historical timeouts)

* test: deflake terminal-agent concurrent startup (write the placeholder record before spawning, like the product) and qa-evidence resumed pipe (barrier on a full high-water mark instead of a sleep)

* fix(design): daemon log stream handles open errors instead of crashing the daemon; shutdown test awaits the stubbed exit

* test(evals): skillify checks the product's resolved state root (no GSTACK_HOME bypass), negative cases check every root; tpa shims live outside the agent's cwd and answer consistently

* test(evals): coverage-audit detector splits parallel columns at │ rails and [+] groups; coverage prompts name absolute skill paths; TODOs from the gate eval census

* fix(plan-eng-review): scope gate always sends the menu (no headless 'Scope pending' exit); PTY harness ends idle turn-ends and API-error panels promptly with a precise diagnostic

* test(evals): one position-bound coverage-read rule credits read-only && chains (labels, context ranges, git displays in either order) and still rejects forged output

* fix(ship): documentation section has one saved source inserted by cat (never retyped); spawned-marker evidence is the child output; refreshed codex/factory ship goldens

* test(docsync): the byte-exact insert positive is POSIX-only (the bash-command observer refuses backslash paths by design)

* test(evals): plan-design-with-ui focus-title rule accepts the comma-less 'dimensions or focus' wording (replayed on the captured CI run, negative controls kept)

* fix(qa-callers): exploratory preflight says the scope/method Reads come after exploratory.md; design-review scan check credits an executed scan proven by one engine run over the changed files

---------

Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com>
Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com>
Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>
2026-10-03 07:54:09 -07:00

7.7 KiB
Raw Blame History

Shared exploratory QA

The caller (/qa, /qa-only, /review or /ship) owns decisions, tests, fixes and publication. Discovery writes only reports/evidence and owned fixture state; no workflows, framework installs or publication.

0. Preparation gate

Complete these Reads in order before writing charters or probing:

  1. Read sections/scope.md relative to the installed qa/gstack-qa SKILL.md directory in full and select the surfaces.
  2. Read the selected surface methods below in full.

Use this host's installed qa/gstack-qa SKILL.md directory for these reads:

Functional surfaces: Read sections/system-functional.md in full.

Browser surfaces only: Read sections/qa-patterns.md in full.

Await each successful Read result before continuing. A supplied target, isolation description, section index or remembered method is not a completed instruction Read. Do not repeat a Read already completed in this invocation; reuse only its acknowledged full contents. If either required Read is missing, complete it now before Charter and preflight. Missing or unreadable assets, prerequisites or permission block affected probes, not independent safe checks. Report QA setup blockers.

1. Charter and preflight

Reuse resolved REPORT_DIR; otherwise own a fresh .gstack/qa-reports subdirectory. Write a charter per behavior: contract, risk, entrypoint, isolation, exit condition, source, commands and inputs. Save charters as Markdown in the report.

For /qa and /qa-only:

  • Browser Quick: SECONDS=30. Browser Full/Regression: SECONDS=900.
  • Functional Full, Quick and Regression have no default total timer. Set SECONDS to the shorter mode/caller limit; an unlimited mode uses the caller's bound. Without a total time limit, do not start D; announce finite command timeouts. Stop when scoped contracts are tested or blocked. Clocks/checkpoints use REPORT_DIR; mixed standalone runs use REPORT_DIR/browser and REPORT_DIR/functional, with one final report at REPORT_DIR. Caller paths win. Names used below (quote every path):
  • R: this surface's owned probe directory, chosen above. D: R/deadline.json, which exists only for a bounded run.
  • SECONDS: the total probe budget set above. NNN: a fresh three-digit ID (001, 002, ...) for each capture or checkpoint; never reuse one.
  • G = $HOME/.claude/skills/gstack/bin/gstack-qa-deadline, the deadline guard: start creates D, status prints remainingMs and expired, and run executes one command, stopping it at the deadline.
  • Q = $HOME/.claude/skills/gstack/bin/gstack-qa-evidence, the functional evidence recorder: capture runs one command, stores its exit code, stdout and stderr under R/.qa-evidence/NNN/, and prints status complete, incomplete or sensitive.

Start once before baseline: bun G start D SECONDS [EARLIER_UTC] if bounded. EARLIER_UTC = caller's absolute deadline, if set. Functional: bun Q capture R NNN [--public] --deadline D -- COMMAND ARGS. Unbounded: use --timeout-ms MS instead of --deadline D. --public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of R/.qa-evidence/NNN/observation.json (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints. Bounded browsers: bun G run D -- COMMAND ARGS. No detached probes. Never reset D/bypass G. Expiry or invalid/missing D stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.

2. Probe loop

Each probe is one native command/interaction plus checks, excluding bookkeeping. Never batch probes.

  1. First demonstrate success: output AND durable effects. Guard if bounded; await completion.
  2. Decide whether another probe is needed. If bounded, run bun G status D. If expired or no safe next probe remains, STOP exploration; write the report, not a checkpoint. Classify the last result before copying it. For public or synthetic observations, retain the entire result unchanged, including owned fixture paths, IDs, hashes and existing credential placeholders. An absolute state path is not itself a secret. For actual secrets/private payloads, withhold those values and disclose the redaction and replay limits in the report. If no safe exact observation can be retained, stop the affected probe chain; never invent a substitute path, identity or state. Publish before probing. Create exploration-NNN.json in the probe directory, beside its deadline if bounded, with exactly four top-level fields: observationCommand: last completed probe's full outer command, including guard. observed: its exact decoded child JSON (no wrapper/extra keys), or its full non-JSON text. For guarded text, copy the complete span between the guard's started and finished receipt lines. Keep its whitespace and content fences verbatim. Do not summarize, relabel or add timing text. The guard adds one newline before its finished receipt; that separator is not child text. For unguarded text, copy the complete result instead. If capture is incomplete, report that limit instead of reconstructing it. hypothesis: why nextCommand. nextCommand: exact command/request, guarded if bounded. Preserve every safe program-JSON key/value and identity hash unchanged. Withhold unsafe values, disclose limits and stop that chain. Check fields before publication. No drafts/placeholders or invented safe-path redactions; corrections cannot repair published notes. Functional: do not write this file; the next capture publishes it: bun Q capture R NNN --deadline D --after PREV --hypothesis 'why' -- CMD (PREV: the last complete capture's ID). Q supplies observed; never transcribe it. Browser checkpoints use Write: create the four-field file yourself. Wait for successful checkpoint publication before dispatch. Never backfill or overwrite notes.
  3. Run that exact probe; G enforces the deadline when bounded. Report refusals as not-run; retain initial state/inputs/results. Repeat from step 2.
  4. Replay the exact failing command/request from the same initial fixture state via steps 2–3 (same native command, fresh capture ID) to confirm it, then minimize via those gates. Expiry leaves confirmation/minimization incomplete. Another input or a regression test is not that replay.
  5. If the user or another process changes source, commands or fixtures, review the affected contracts and return to step 2 for each affected revalidation (unproven=affected). Do not make product changes yourself. Keep the original limits/notes; update outcomes only from fresh evidence.

3. Parent handoff

Never change product code, tests, configuration, dependencies or Git through any tool, including shell, rename, deletion, commit, stash or edit-then-restore. Return test_stub proposals with their failing contract and expected assertion; never create tests or freeze buggy output.

4. Final report

Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests. Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown bun Q materialize R annotations.json (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence. Evidence is invocation-local. Missing prerequisites/expectations/observations, timeouts and refusal never pass. Pass requires all required current-input contracts to pass with no required remainder. Report blocked, inconclusive and not-run coverage without claiming success.