v1.91.372.0 feat(flaky): gstack-flaky reruns a failed test command once and ledgers the flake (GSTA-158 P1) (#3612)

* feat(flaky): gstack-flaky run reruns a failed JUnit test once and ledgers the flake (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(flaky): stream output, keep unreported runner failures red, fail-closed duplicates and nonzero reruns

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes

gstack-flaky never greens an incomplete run: a targeted rerun needs the
runner's summary proving the first run complete, otherwise (and always for
generic runners) the rerun repeats the whole command. A rerun clears a test
only when it exits 0 and the test passes once on its first attempt. Killed
runners exit 3 (FLAKY_COMMAND_KILLED) and signals are forwarded. The first
report is restored at --junit with reruns kept beside it; ids are redacted at
every output; the ledger gets its own GSTACK_FLAKE_LEDGER_JUNIT; fixtures are
registered in FREE_FIXTURES.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: latch cancellation and check whole-command rerun reports

A cancel forwarded to a runner that traps the signal now counts as killed for
every later attempt, so a cancelled job is never rerun or greened. A
whole-command rerun clears nothing when its report has a failed test, misses a
first-run test, or its runner counted failures outside the report.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: whole-command rerun when coverage or runner-flaky tests can fail the run

A coverage threshold or a fail-on-flaky policy fails a run for a reason a
narrowed rerun cannot reproduce, so either forces a whole-command rerun.
--junit refuses an existing file that is not a JUnit report instead of
deleting it.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* feat(flaky): rerun the whole command by default; targeted rerun is opt-in

A rerun counts only on the runner's own full green verdict: exit 0, no failed
or missing test, nothing counted outside the report. That covers every way a
run can fail, which a narrowed rerun cannot (a coverage threshold with an lcov
reporter, a fail-on-flaky policy, a test never reached). --rerun-scope test
reruns the failed test alone, with its preconditions documented and the
complete-run checks kept as a safety net. An unreadable existing --junit file
is a usage error, not a crash.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: gstack-flaky CHANGELOG entry and standalone-binaries row (GSTA-158)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

docs/TESTING_INTERNALS.md

@@ -1435,6 +1435,13 @@ prints a run's ledgers as a table. A flaky file is fixed at source; the retry
 never masks it. `bun run eval:pass-rates` lists only the local ledger
 (`flakeLedgerPath()`, overridden by `GSTACK_FLAKE_LEDGER`).
 
+Other repositories get the same rule per test with `bin/gstack-flaky run`
+([docs/flaky.md](flaky.md)): it reads a JUnit report from any runner, reruns
+the command once when tests failed (counting only the runner's full green
+verdict), and writes `flaky-pass` and `rerun-fail` lines to its
+own ledger (`flake-ledger-junit.jsonl`, or `GSTACK_FLAKE_LEDGER_JUNIT`), kept apart
+from this file-level ledger.
+
 The aggregate job of `free-tests.yml` downloads every `flake-ledger-*` artifact
 of the run and appends `bun run scripts/test-health-report.ts flake-summary <dir>`
 to its job summary. The Windows lane uploads `flake-ledger-windows-<shard>`.

docs/flaky.md

@@ -0,0 +1,137 @@
+# Flaky tests: rerun once and record it (`gstack-flaky`)
+
+`gstack-flaky run` runs your test command, reads the JUnit report it writes,
+and when tests failed, runs the command once more. A test that passes on its
+rerun is a **flaky pass**: the job stays green and the flake is recorded on a ledger, so
+the daily flaky pass can fix it at the source. A test that fails twice is red.
+This is CREW rule 16 ("run the failed test again and note that it is flaky")
+for any runner that writes JUnit: Playwright, `bun test`, jest, vitest and
+pytest. gstack's own free suite keeps its file-level retry
+(`GSTACK_FREE_RETRY_FLAKY`, [Flake ledger](TESTING_INTERNALS.md#flake-ledger)).
+
+## Get started in GitHub Actions
+
+```yaml
+- uses: oven-sh/setup-bun@v2
+  with: { bun-version: 1.4.4 }
+- name: Install gstack-flaky (pinned)
+  env: { GSTACK_REF: <a gstack commit sha you have tested> }
+  run: |
+    git clone --quiet --filter=blob:none https://github.com/garrytan/gstack.git "$RUNNER_TEMP/gstack"
+    git -C "$RUNNER_TEMP/gstack" checkout --quiet "$GSTACK_REF"
+- name: E2E
+  env:
+    PLAYWRIGHT_JUNIT_OUTPUT_NAME: results/junit.xml
+    GSTACK_FLAKE_LEDGER_JUNIT: ${{ runner.temp }}/flaky-junit-ledger.jsonl
+  run: |
+    "$RUNNER_TEMP/gstack/bin/gstack-flaky" run --junit results/junit.xml -- npx playwright test --reporter=line,junit
+- if: always()
+  uses: actions/upload-artifact@v4
+  with: { name: flaky-junit-ledger-e2e, path: "${{ runner.temp }}/flaky-junit-ledger.jsonl", if-no-files-found: ignore, retention-days: 90 }
+```
+
+Replace `GSTACK_REF` with a gstack commit you have tested; the bin needs only
+Bun and git. In Actions it also writes a step-summary table of the flaky
+passes and one `::warning` annotation per flaky test, so they are visible on
+the run before anyone reads the ledger.
+
+## Runners
+
+| Runner | Command after `--` | Notes |
+|---|---|---|
+| Playwright | `npx playwright test --reporter=line,junit` with `PLAYWRIGHT_JUNIT_OUTPUT_NAME=<file>` | `gstack-flaky` sets `PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1` so Playwright's own `retries` are reported; with retries configured, those attempts stand and nothing is rerun again. |
+| bun | `bun test --reporter=junit --reporter-outfile=<file>` | |
+| jest, vitest, pytest (`--runner generic`) | your command, writing JUnit | Always reruns the whole command, even with `--rerun-scope test`. |
+
+The command after `--` runs without a shell. When it needs one, wrap it as
+`bash -c '<script> "$@"' _` so a targeted rerun's selection arguments reach
+the runner.
+
+## Rerun scope
+
+**Default: the whole command.** The rerun repeats your command exactly, and
+counts only on the runner's own full green verdict: it exits 0, no test fails,
+every test of the first run is in its report, and the runner counts no failure
+outside the report. That verdict covers every reason a run can fail, including
+the ones a narrowed rerun cannot see: a test the first run never reached
+(`--bail`, `--max-failures`, `-x`), a coverage threshold (however coverage is
+configured, `bunfig.toml` and lcov included), a fail-on-flaky policy, a
+global setup error. A test that failed in the first run and passes once in this
+rerun is a flaky pass. It costs one more run of the suite, which is still far
+cheaper than rerunning the CI job by hand.
+
+**Opt-in: `--rerun-scope test`.** Reruns the failed test alone (Playwright:
+`<file> --grep <title> --project <project>`; bun: `<file> --test-name-pattern <title>`)
+and clears it when that rerun exits 0. Use it only when the first run's exit
+status can come from failed tests alone: no coverage threshold in any form, no
+`--fail-on-flaky-tests`, no other step inside the command that can fail. As a
+safety net, `gstack-flaky` still reruns the whole command
+(`FLAKY_RERUN: scope=command (<why>)`) when the runner exited anything but
+1, its summary is missing, it stopped early (bun `Bailed out`, Playwright
+`did not run`, `interrupted`, `Testing stopped early`, a global timeout), it
+printed a coverage table or got `--coverage`, or it retried flaky tests itself.
+It cannot see a coverage threshold configured without console output (for
+example `coverageReporter = ["lcov"]` in `bunfig.toml`), which is why this
+scope is opt-in.
+
+## Rules
+
+- **One rerun, small failures only.** A failed test gets one rerun when the
+  runner made no retries and at most `--max-reruns` (3) tests failed. More
+  failures than that is a real red, and nothing is rerun. When the run is
+  already red for a test that cannot be rerun, a whole-command rerun is
+  skipped.
+- **A rerun clears a test only cleanly.** The rerun's command must exit 0 and
+  the test must run once and pass on its first attempt; a rerun the runner
+  retried again does not count.
+- **Never green by accident.** The report is deleted before every attempt, and
+  a report that attempt did not write counts as missing. A missing, empty,
+  zero-test or malformed report fails, as does a non-zero exit with no failed
+  test (a crash or a non-test step) and a runner killed by a signal. Cancelling
+  `gstack-flaky` (SIGINT, SIGTERM) is passed to the runner and latched: nothing
+  is rerun after it, whatever exit the runner chooses. When the runner's
+  summary counts failures its JUnit omits (a bun file that fails to load, a
+  Playwright error outside any test), the run is red and nothing is rerun.
+- **Reports.** After the run, `--junit` holds the first run's report, so a
+  later upload or test reporter sees what failed. Each rerun's report is kept
+  beside it as `<name>.rerun-<n>.xml`. Other reporters' outputs (Playwright's
+  `test-results/` and HTML report) are rewritten by a rerun; upload the JUnit
+  report as the record. An existing `--junit` file that is not a JUnit report,
+  or cannot be read, is refused (usage, exit 2), never deleted.
+- **Ids.** A test is `<file>::<project>::<describe > title>`, with the project
+  empty outside Playwright, so a test in two browser projects is two tests.
+  `gstack-flaky ids --junit <file>` prints them.
+- **Ledger.** One JSON line per flaky pass (`kind: flaky-pass`) and per test
+  that failed its rerun (`rerun-fail`), with the tree, commit, branch, run URL,
+  attempts, the rerun's scope (`test` or `command`) and the first lines of
+  the error. Ids, paths and failure text go through gstack's redaction engine
+  first; when the scan cannot run, they are withheld. The path is `--ledger`,
+  else `GSTACK_FLAKE_LEDGER_JUNIT`, else
+  `<state root>/flake-ledger-junit.jsonl`, and `FLAKY_RESULT` names it. It is
+  kept apart from the free suite's file-level ledger (`GSTACK_FLAKE_LEDGER`).
+  A ledger write error is a warning and never changes the verdict.
+
+## Output and exit codes
+
+One `FLAKY: <flaky-pass|fail|rerun-fail> <id> attempts=<…> (<reason>)` line per
+test that failed at least once, `FLAKY_RERUN: scope=command (<why>)` when the
+rerun repeated the whole command, then
+`FLAKY_RESULT: tests=<n> failed=<n> flaky=<n> reruns=<n> command_exit=<n> verdict=pass|fail ledger=<file|none>`.
+
+| Exit | Meaning |
+|---|---|
+| 0 | Every test passed, flaky passes included |
+| 1 | A test failed (after its rerun, or with no rerun), including `FLAKY_UNREPORTED_FAILURES` |
+| 2 | Usage error, including a `--junit` path that is not a `.xml` file, or an existing file there that is not a readable JUnit report |
+| 3 | No usable report, a crash, or a killed runner (a code below on stderr) |
+
+| Code | Exit | Cause | Fix |
+|---|---|---|---|
+| `FLAKY_NO_RESULTS` | 3 | No report at `--junit`, an empty one, or one with no testcase | Make the command write JUnit to that path |
+| `FLAKY_JUNIT_INVALID` | 3 | The report is not well-formed XML | Make sure one process writes it and nothing appends to it |
+| `FLAKY_RETRIES_HIDDEN` | 3 | Playwright retried but its report omits the retries | Set `PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1` where the report is written |
+| `FLAKY_COMMAND_FAILED` | 3 | Non-zero exit with no failed test | Read the command output: a real red |
+| `FLAKY_COMMAND_KILLED` | 3 | The runner was killed by a signal (a cancelled job, an OOM kill) | Nothing was rerun; rerun the job |
+| `FLAKY_COMMAND_NOT_STARTED` | 3 | The command could not be started | Check the argv after `--` |
+| `FLAKY_UNREPORTED_FAILURES` | 1 | The runner counted failures or errors its JUnit omits | Fix the load error or the error outside any test: a real red |
+| `FLAKY_LEDGER_WRITE_FAILED` | — | The ledger could not be appended (warning) | Check the path; the verdict stands |

docs/standalone-binaries.md

@@ -18,6 +18,7 @@ Beyond the slash-command skills, gstack ships standalone CLIs for workflows that
 | `gstack-review-log` | **Review-pass receipts** — `--start <skill>` captures the working-tree fingerprint before a diff review; `'<JSON>' --finish <token>` consumes that single-use, repository/branch/skill-scoped receipt. Binding requires matching start/end content and reviewer-reported `completed:true` and `converged:true`; it is not independent proof that a model read the source. `--check-shared-libs <token>` reads a current finding as JSON on stdin and checks prior Skip decisions against the actual branch, capture and eligible raw source blobs. It returns `reusable:false` when proof is missing or unsafe, including older records without logger-versioned coverage. Final review logging computes shared-code fingerprints and coverage rather than trusting supplied proof. |
 | `gstack-review-read` | **Review freshness** — emits review records with computed `review_freshness.status` and `reason`: CURRENT, STALE, or UNVERIFIED for diff reviews. `/ship` and `/land-and-deploy` use the same grade; a matching commit alone never certifies a diff review. [Dashboard rules](skills.md#review-readiness-dashboard). |
 | `gstack-evidence` | **Verification-evidence ledger** — `run --label <lane> -- <cmd>` transparently wraps any test command (the child's exit code always passes through) and records what ran against which working-tree fingerprint (tracked and untracked files that are not gitignored, minus an untracked `.bun/` folder at the repo root, which Bun can create during a run); a run whose files changed while it ran is recorded without a fingerprint and with up to five of the changed paths, which `check` names in its STALE reason. `check` grades each label FRESH/STALE/MISSING with `--expect-cmd`, `--max-age`, and `--allow-paths` binding. /ship and /land-and-deploy cite fresh evidence instead of re-running suites. Per-run logs are 0600, capped at 2MB, pruned after 30 days; the ledger and logs stay machine-local by design. |
+| `gstack-flaky` | **Flaky-test rerun for any JUnit runner** — `run --junit <file.xml> -- <command>` runs your test command (Playwright, `bun test`, jest, vitest, pytest) and, when at most `--max-reruns` (3) tests failed and the runner made no retries of its own, reruns the whole command once (`--rerun-scope test` opts in to rerunning only the failed tests). A test that fails and then passes on that rerun is a flaky pass: the run stays green and one redacted line per flaky pass or failed rerun goes on a ledger (`GSTACK_FLAKE_LEDGER_JUNIT`, default `<state root>/flake-ledger-junit.jsonl`). A missing or malformed report, a crash or a killed runner never passes. `ids --junit <file>` prints a report's test ids. Exit 0 pass, 1 a test failed, 2 usage, 3 no usable report. [Guide](flaky.md). |
 | `gstack-pr-wait` | **PR event watcher** — `gstack-pr-wait <pr> [--repo owner/name] [--until green\|ready\|any-change] [--interval 20] [--timeout 60] [--exclude CHECK,...]` polls one pull request through `gh` and prints one stdout line per real event (a check failed with its link, all checks finished, a comment or review from someone other than the author, a review bot finished on the head, the base moved and its new commits touch this PR's files, BEHIND or DIRTY, merged or closed), with progress on stderr, so a monitor wakes the agent once per event instead of on a timer. `green` gates on every check on the head (an advisory bot's review is never waited for, its checks gate like any other), as `gstack-ci-gate` does; a check the base branch requires (rulesets and classic branch protection) is pending until it appears, never green by omission; two identical polls confirm a check set is complete; `--exclude` leaves a named non-required check out of the verdict as `gstack-ci-gate --exclude` does (a required name is refused and gates). Exit 0 when the condition is met, 1 when it needs action or `--timeout` passes (`PR_WAIT_BLOCKED`), 2 on bad input, a second watcher for the same PR or a local error (`PR_WAIT_REFUSED`). State under `<state root>/pr-wait/`, baseline discarded after 6 hours. Review bots are configuration: `gstack-config set pr_wait_gating_bots greptile,macroscope`, `pr_wait_advisory_bots coderabbit` (advisory bots are reported, never waited on). `/land-and-deploy` Step 3 is one call of it. Design ported from tstack's pr-wait (MIT, Theo Browne; NOTICE.md). [Exit codes](troubleshooting.md#gstack-pr-wait). |
 | `gstack-issue-guard` | **Tracker-text trust envelope** — fetches GitHub issue/PR text (`issue <n>`, `pr-body`, `pr-comments`, or `--stdin`) and wraps it in a labeled envelope so agents treat it as data: injection-shaped lines get labeled even through fullwidth and invisible-character evasion, and forged envelope banners are defused. Every tracker-text ingress in gstack routes through it, enforced by a CI scanner. |
 | `gstack-ios-qa-daemon` | **iOS QA daemon** — Mac-side broker between an agent and a connected iPhone or iPad over USB CoreDevice. Loopback by default; `--tailnet` opens a Tailscale-facing listener with identity-gated capability tiers. Single-instance via flock on `~/.gstack/ios-qa-daemon.pid`. See [docs/howto-ios-testing-with-gstack.md](howto-ios-testing-with-gstack.md). |