mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-01 06:24:41 +01:00
ac20ef1e13
* fix(make-pdf,design): real TOC page numbers and anchors; design models move to gpt-5.5 + gpt-image-2 (#2903, #2807) * ci: pin every action to a SHA, Scorecard, checksummed Bun, scoped Codex CLI in the CI image, macOS named-regressions job (#1948, #1706, #1997) * fix(cso,browse): Windows/macOS lease identity, retention skips bad runs, truthful 'not assessed' reports, env.kv false positive, Windows browse spawn fixes (#2894, #2912, #2048, #2637, #2805) * fix(outside-voice): one Codex model precedence for probe and dispatch, bounded timeouts with KILL escalation, probe cache identity, guarded mktemp, skills hidden in nested Codex (#2776, #2914, #2787, #2881, #2847) * feat(setup): install registry, ./setup --status, staged activation, per-scope migrations, host tiers and conformance kit, OpenCode/Hermes fixes, Copilot CLI (experimental), disabled_skills, slimmer CLAUDE.md (#1925, #2879, #2347, #2338, #2626, #2629, #2825, #2015, #393, #1206, #2096) Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com> Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com> Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com> * fix(preamble): skills start in worktree-isolated Claude Code sessions — one literal skill-start line, context-recovery helper, no eval in skill blocks, $N token tripwire, honest plan-mode precedence (#2763, #2896, #2851) * fix(setup): per-install renders for renamed, vendored and custom-config installs, space-safe alias roots, per-host /cso launcher path (#1882, #2906) * test: free tests never write the real HOME (home-write tripwire), no orphaned browse daemons, symlinked-TMPDIR (macOS) fixes, gbrain-on-PATH isolation (#2895, #2816, #2829, #2841, #2982, #2911, #2909, #2908, #2910) * chore: integrate wave follow-ups — regenerate all hosts, README model docs, TODOS, touchfiles, duration seeds, Bun 1.3 telemetry test fix, replay fixtures pinned to their recorded bytes * v1.91.14.0: release metadata * test(browse): group-kill cleanup falls back to the leader when macOS returns EPERM for a live group * ci: Codex login steps run under bash (the CI container's default sh rejects pipefail) * fix: full-suite repairs — private state roots for 7 real-HOME writers, truthful home-guard attribution with --attribute-home, gstack-paths shim in ship-hook fixtures, Copilot in team-init probes, test-of-test detector, cookie review fixture pinned to the approved excerpt * fix(windows): LF for Dockerfiles, normalized home-guard exclusions, path.relative fixture containment, ship push test spawn timeouts * test(design-detect): widen the whole-scan budget test's timing margins so a loaded macOS runner can't time out the single-batch call * fix(evals): qa protocol names defined for the judge, async PTY judge so overlapping sessions keep their cleanup deadline, bws setup cases use the standard JUDGE_MS tier * test(pty): the non-blocking judge case is POSIX-only (its fake judge is a shebang script, like the PTY harness it covers) * test(evals): ship-coverage-audit names absolute skill paths, uses Read, and skips the Step 7 artifact nothing consumes (removes the mechanism behind 3 of 4 historical timeouts) * test: deflake terminal-agent concurrent startup (write the placeholder record before spawning, like the product) and qa-evidence resumed pipe (barrier on a full high-water mark instead of a sleep) * fix(design): daemon log stream handles open errors instead of crashing the daemon; shutdown test awaits the stubbed exit * test(evals): skillify checks the product's resolved state root (no GSTACK_HOME bypass), negative cases check every root; tpa shims live outside the agent's cwd and answer consistently * test(evals): coverage-audit detector splits parallel columns at │ rails and [+] groups; coverage prompts name absolute skill paths; TODOs from the gate eval census * fix(plan-eng-review): scope gate always sends the menu (no headless 'Scope pending' exit); PTY harness ends idle turn-ends and API-error panels promptly with a precise diagnostic * test(evals): one position-bound coverage-read rule credits read-only && chains (labels, context ranges, git displays in either order) and still rejects forged output * fix(ship): documentation section has one saved source inserted by cat (never retyped); spawned-marker evidence is the child output; refreshed codex/factory ship goldens * test(docsync): the byte-exact insert positive is POSIX-only (the bash-command observer refuses backslash paths by design) * test(evals): plan-design-with-ui focus-title rule accepts the comma-less 'dimensions or focus' wording (replayed on the captured CI run, negative controls kept) * fix(qa-callers): exploratory preflight says the scope/method Reads come after exploratory.md; design-review scan check credits an executed scan proven by one engine run over the changed files --------- Co-authored-by: Andrey Esipov <61357035+andrey-esipov@users.noreply.github.com> Co-authored-by: Matthew Williams <5863886+ridermw@users.noreply.github.com> Co-authored-by: Rocky Wu <92764750+lolisaigao1234@users.noreply.github.com>
93 lines
6.9 KiB
Markdown
93 lines
6.9 KiB
Markdown
<!-- AUTO-GENERATED from exploratory.md.tmpl — do not edit directly -->
|
||
<!-- Regenerate: bun run gen:skill-docs -->
|
||
# Shared exploratory QA
|
||
|
||
The **caller** (/qa, /qa-only, /review or /ship) owns decisions, tests, fixes and publication. Discovery writes only reports/evidence
|
||
and owned fixture state; no workflows, framework installs or publication.
|
||
|
||
Complete these Reads in order before writing charters or probing. Await their results before the first probe, never in the same response. Do not repeat a Read already completed in this invocation.
|
||
1. Read `sections/scope.md` relative to the installed `qa`/`gstack-qa` SKILL.md directory in full and select the surfaces.
|
||
2. Read the selected surface methods below in full.
|
||
|
||
**Functional surfaces:**
|
||
Read `sections/system-functional.md` in full.
|
||
|
||
**Browser surfaces only:**
|
||
Read `sections/qa-patterns.md` in full.
|
||
|
||
Missing or unreadable assets, prerequisites or permission block affected probes, not independent safe checks. Report QA setup blockers.
|
||
|
||
## 1. Charter and preflight
|
||
|
||
Reuse resolved REPORT_DIR; otherwise own a fresh `.gstack/qa-reports` subdirectory.
|
||
Write a **charter** per behavior: contract, risk, entrypoint, isolation, exit condition, source, commands and inputs. Save charters as Markdown in the report.
|
||
|
||
For /review and /ship, no plan/server is required.
|
||
Stop after 5 minutes or 12 probes, whichever comes first (SECONDS=300 across surfaces).
|
||
Explicit plan checks and revalidation remain required beyond this smoke budget.
|
||
For /qa and /qa-only:
|
||
- Browser Quick: SECONDS=30. Browser Full/Regression: SECONDS=900.
|
||
- Functional Full, Quick and Regression have no default total timer.
|
||
Set SECONDS to the shorter mode/caller limit; an unlimited mode uses the caller's bound.
|
||
Without a total time limit, do not start D; announce finite command timeouts.
|
||
Stop when scoped contracts are tested or blocked.
|
||
Clocks/checkpoints use REPORT_DIR; mixed standalone runs use REPORT_DIR/browser and REPORT_DIR/functional, with one final report at REPORT_DIR. Caller paths win.
|
||
Names used below (quote every path):
|
||
- R: this surface's owned probe directory, chosen above. D: `R/deadline.json`, which exists only for a bounded run.
|
||
- SECONDS: the total probe budget set above. NNN: a fresh three-digit ID (001, 002, ...) for each capture or checkpoint; never reuse one.
|
||
- G = `$HOME/.claude/skills/gstack/bin/gstack-qa-deadline`, the deadline guard: `start` creates D, `status` prints `remainingMs` and `expired`, and `run` executes one command, stopping it at the deadline.
|
||
- Q = `$HOME/.claude/skills/gstack/bin/gstack-qa-evidence`, the functional evidence recorder: `capture` runs one command, stores its exit code, stdout and stderr under `R/.qa-evidence/NNN/`, and prints status `complete`, `incomplete` or `sensitive`.
|
||
|
||
Start once before baseline: `bun G start D SECONDS [EARLIER_UTC]` if bounded.
|
||
EARLIER_UTC = caller's absolute deadline, if set.
|
||
Functional: `bun Q capture R NNN [--public] --deadline D -- COMMAND ARGS`.
|
||
Unbounded: use `--timeout-ms MS` instead of `--deadline D`.
|
||
--public requires approved public/synthetic output; Q screens credentials. For complete private captures, await a safe Read of `R/.qa-evidence/NNN/observation.json` (the decoded stdout). Sensitive/incomplete captures cannot anchor checkpoints.
|
||
Bounded browsers: `bun G run D -- COMMAND ARGS`. No detached probes.
|
||
Never reset D/bypass G. Expiry or invalid/missing D stops probes; report unfinished coverage. QA_DEADLINE receipts are not observations.
|
||
|
||
## 2. Probe loop
|
||
|
||
Each probe is one native command/interaction plus checks, excluding bookkeeping.
|
||
Never batch probes.
|
||
|
||
1. First demonstrate success: output AND durable effects. Guard if bounded; await completion.
|
||
2. **Decide whether another probe is needed.** If bounded, run `bun G status D`.
|
||
If expired or no safe next probe remains, STOP exploration; write the report, not a checkpoint.
|
||
**Publish before probing.** Create `exploration-NNN.json` in the probe directory, beside its deadline if bounded, with exactly four top-level fields:
|
||
observationCommand: last completed probe's full outer command, including guard.
|
||
observed: its exact decoded child JSON (no wrapper/extra keys), or its full non-JSON text.
|
||
hypothesis: why nextCommand. nextCommand: exact command/request, guarded if bounded.
|
||
Preserve every safe program-JSON key/value and identity hash unchanged.
|
||
Withhold unsafe values, disclose limits and stop that chain.
|
||
Check fields before publication. No drafts/placeholders or invented safe-path redactions; corrections cannot repair published notes.
|
||
Functional: do not write this file; the next capture publishes it: `bun Q capture R NNN --deadline D --after PREV --hypothesis 'why' -- CMD` (PREV: the last complete capture's ID). Q supplies observed; never transcribe it.
|
||
Browser checkpoints use Write: create the four-field file yourself.
|
||
Wait for successful checkpoint publication before dispatch.
|
||
Never backfill or overwrite notes.
|
||
3. Run that exact probe; G enforces the deadline when bounded.
|
||
Report refusals as not-run; retain initial state/inputs/results. Repeat from step 2.
|
||
4. Replay the exact failing command/request from the same initial fixture state via steps 2–3 (same native command, fresh capture ID)
|
||
before repair, then minimize via those gates. Expiry leaves confirmation/minimization incomplete.
|
||
Another input or a regression test is not that replay.
|
||
5. After source/commands/fixtures change, re-review and return to step 2 for each affected revalidation (unproven=affected). Keep limits/notes; status requires fresh evidence.
|
||
|
||
## 3. Parent handoff
|
||
|
||
- **/qa:** parent owns severity, root-cause and Phase 8 regression gates before verified repair.
|
||
- **/review:** return before Fix-First; test_stub proposals require ASK approval.
|
||
- **Planning:** propose charters only; no execution.
|
||
|
||
Choose the smallest native test: unit for logic, integration for state/requests; E2E only if smaller tests miss the journey, not automatically both.
|
||
Mock only unrelated services.
|
||
Never freeze buggy output, weaken tests or delete valid red tests.
|
||
|
||
## 4. Final report
|
||
|
||
Use the surface report template; link each checkpoint. Separate browser scores, functional outcomes and proposed/executed tests.
|
||
Write R/annotations.json {evidence: [{capture, command, contract, expected, classification}], limits} (browser-only: evidence [], checkpoints in limits); before Markdown `bun Q materialize R annotations.json` (fills observed/metadata; prints reportLinks); you classify. Retain all safe probes, including failures/replays; disclose withheld/incomplete evidence.
|
||
Evidence is invocation-local; /ship reruns once per invocation.
|
||
Missing prerequisites/expectations/observations, timeouts and refusal never pass.
|
||
Pass requires all required current-input contracts to pass with no required remainder.
|
||
Required failure leaves /review incomplete and /ship blocked unless the user explicitly accepts that named risk; noninteractive runs return blocked. Only nonbehavioral diffs may be not applicable (give a reason); prompts/templates are behavioral.
|