## Flake recipe
A flake is a test whose verdict depends on something other than the code it
guards. The Iron Law holds here: no rerun, no wider timeout, no retry and no
skip until the cause is named and probed.
1. **Reproduce on main.** Check out `origin/` in a worktree
(`git worktree add
origin/`) and run the failing file alone N
times (N ≥ 10). On gstack:
`bun run scripts/ship-measure.ts free --files --reruns N`; elsewhere a
loop over the project's single-file test command. Record `failed/N` on main,
then the same on the branch. A failure that reproduces on main is outside the
PR; one that appears only on the branch is inside it.
2. **Cause inside or outside the PR.** Diff the two runs' logs at the first
divergence and name the dependency: timing, file order, shared state, network,
clock, a resource limit. "Flaky" is a symptom, never a cause.
3. **Compare the metric with the last N green runs.** Once for `` and once
for the branch (`--branch `), against the workflow that ran the
failing shard; read the duration or retry token next to the test:
```bash
gh run list --branch --workflow free-tests.yml --status success --limit 20 --json databaseId,headSha,createdAt,updatedAt
gh run view --log | grep -F -- ""
```
The run where the duration, retry count or failing test first moved dates the
regression; cite its `headSha`.
4. **Forced probe.** Build the smallest change that makes the test fail
deterministically for the stated cause (inject the delay, pin the file order,
freeze the clock, exhaust the resource). Show the probe output failing before
the fix and passing after. No probe, no cause.
5. **Never loosen.** A wider timeout, a retry, a looser assertion or a skip is a
gate relaxation (review tag RH-15); the fix lives in the code or in the test's
isolation, not in its threshold.
**The flake-fix number.** A fix that touches only the test (a wider timeout, a retry,
a looser assertion, a skip) must state `P(fail | regression)`: the probability the test
still fails when the behavior it guards regresses, before and after the change. Derive
it from the forced probe (break the behavior on purpose, run the test N times, count
failures) and write both numbers in `## Flake evidence`. A number that drops is a gate
relaxation (review tag RH-15) and needs the owner's decision, not a merge.
6. **A rerun only when the cause is known and outside the PR** (infrastructure,
a polluter already fixed on main, a provider outage), stated in the PR body.
Reruns follow `ship-measure`'s discipline: bounded, counted, logged,
`flaky retry off`; a free rerun is diagnostic, never the green. Paid evals
are never retried (`EVAL_POLICY` in `test/helpers/periodic-exclude-data.ts`).
7. **`## Flake evidence` in the PR body (required for any flake fix).** The
probe's before and after output, `failed/N` on main and on the branch, the
`gh run list` comparison with the first-moved `headSha`, and a routing note
for the owner: `fix in this PR` | `rerun; cause outside the PR: ` |
`owner decision needed: `.