mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-04 15:57:33 +01:00
Garry Tanandcapy-ai[bot]
ab79e55119
* v1.91.70.0 feat(unattended): artifact contract, unattended session kind, findings-derived review log, gate list Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.72.0 feat(autoplan): unattended runner next|submit, pluggable outside voice, review checklists, spec-loop cap Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(b-runner): atomic lock creation in spend-ledger, judge bundle skips runner-only sections, fixture env spelling - lib/spend-ledger.ts: create the lock by linking a complete temp file so a contender never reads an empty lock and breaks a live owner (CI admitted 20 of a cap of 8 under contention); unreadable locks break only by mtime age. - test/helpers/workflow-judge-input.ts: exclude manifest runner_only sections (the B4 checklists) from the judge bundle; the judge rates the interactive workflow. test/workflow-judge-input.test.ts expects the same set. - test/helpers/autoplan-run-fixture.ts: spell the runner's session kind without the literal the free-suite environment scan rejects. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.74.0 feat(review): plan reality rows, sideways review and cso sweep, investigate recipes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: register the investigate carve case and map Reality rows to real headings in the review checklists Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test: pin the corrected Reality rows obligation rows Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(investigate): use allowlisted placeholders in the flake and polluter fences; select the investigate carve case on PTY changes Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
44 lines
2.7 KiB
Cheetah
44 lines
2.7 KiB
Cheetah
## Flake recipe
|
|
|
|
A flake is a test whose verdict depends on something other than the code it
|
|
guards. The Iron Law holds here: no rerun, no wider timeout, no retry and no
|
|
skip until the cause is named and probed.
|
|
|
|
1. **Reproduce on main.** Check out `origin/<base>` in a worktree
|
|
(`git worktree add <dir> origin/<base>`) and run the failing file alone N
|
|
times (N ≥ 10). On gstack:
|
|
`bun run scripts/ship-measure.ts free --files <file> --reruns N`; elsewhere a
|
|
loop over the project's single-file test command. Record `failed/N` on main,
|
|
then the same on the branch. A failure that reproduces on main is outside the
|
|
PR; one that appears only on the branch is inside it.
|
|
2. **Cause inside or outside the PR.** Diff the two runs' logs at the first
|
|
divergence and name the dependency: timing, file order, shared state, network,
|
|
clock, a resource limit. "Flaky" is a symptom, never a cause.
|
|
3. **Compare the metric with the last N green runs.** Once for `<base>` and once
|
|
for the branch (`--branch <branch-name>`), against the workflow that ran the
|
|
failing shard; read the duration or retry token next to the test:
|
|
```bash
|
|
gh run list --branch <base> --workflow free-tests.yml --status success --limit 20 --json databaseId,headSha,createdAt,updatedAt
|
|
gh run view <run-id> --log | grep -F -- "<failing-test-file>"
|
|
```
|
|
The run where the duration, retry count or failing test first moved dates the
|
|
regression; cite its `headSha`.
|
|
4. **Forced probe.** Build the smallest change that makes the test fail
|
|
deterministically for the stated cause (inject the delay, pin the file order,
|
|
freeze the clock, exhaust the resource). Show the probe output failing before
|
|
the fix and passing after. No probe, no cause.
|
|
5. **Never loosen.** A wider timeout, a retry, a looser assertion or a skip is a
|
|
gate relaxation (review tag RH-15); the fix lives in the code or in the test's
|
|
isolation, not in its threshold.
|
|
{{FLAKE_FIX_NUMBER:investigate}}
|
|
6. **A rerun only when the cause is known and outside the PR** (infrastructure,
|
|
a polluter already fixed on main, a provider outage), stated in the PR body.
|
|
Reruns follow `ship-measure`'s discipline: bounded, counted, logged,
|
|
`flaky retry off`; a free rerun is diagnostic, never the green. Paid evals
|
|
are never retried (`EVAL_POLICY` in `test/helpers/periodic-exclude-data.ts`).
|
|
7. **`## Flake evidence` in the PR body (required for any flake fix).** The
|
|
probe's before and after output, `failed/N` on main and on the branch, the
|
|
`gh run list` comparison with the first-moved `headSha`, and a routing note
|
|
for the owner: `fix in this PR` | `rerun; cause outside the PR: <what>` |
|
|
`owner decision needed: <the relaxation and its P(fail | regression) before/after>`.
|