diff --git a/docs/ship-policy.md b/docs/ship-policy.md index eada39397..3fb80cf1e 100644 --- a/docs/ship-policy.md +++ b/docs/ship-policy.md @@ -195,14 +195,15 @@ of them with the run URL (`PREGATE_CLEARED: `; a lane of another workflow stays uncleared and is named in the refusal), and then reads the head as usual, which passes because the dispatched check runs attach to the commit; the `ci-hint:` line then says those non-pull_request runs count. A red run -clears no lane and refuses (`QUEUE_NOT_READY`); a fork head is never dispatched -here (`QUEUE_TRAIN_UNTRUSTED`). By hand the same four steps are `gh workflow +clears no lane and refuses (`QUEUE_NOT_READY`); a run still going at `--timeout` +(default 2700 s) is `QUEUE_TRAIN_PENDING`, and rerunning the command keeps +waiting; a fork head is never dispatched here (`QUEUE_TRAIN_UNTRUSTED`). By hand the same four steps are `gh workflow run --ref `, wait, `gstack-pregate clear --remote --run-url `, rerun `ready`. The `ci-hint:` line is advisory text for the lane; nothing parses it. `ready --screen` on such a PR has no merge ref to screen, so it builds the scratch merge described below (`SCREEN_MERGE:`, pushed as `screen/-`) and screens that; only a real conflict with the base -is `QUEUE_SCREEN_STALE_REF`. A CHANGELOG-only conflict is screenable, and a +(or a remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A CHANGELOG-only conflict is screenable, and a lane whose skill's own paid cases ran focused may still READY with `--no-screen`. `ci:` counts the latest run per check (one app's check name), as GitHub's @@ -420,8 +421,8 @@ lagging or unfetchable merge ref (GitHub stops recomputing it while the PR conflicts with the base, CHANGELOG-only conflicts included) is replaced by a scratch merge of the head onto the base, pushed as `screen/-` (the `SCREEN_MERGE:` line says which); a screen dispatched on a stale ref would -plan no cases and report success. Only a real conflict with the base is -`QUEUE_SCREEN_STALE_REF`. A finished screen that planned no cases (no +plan no cases and report success. Only a real conflict with the base (or a +remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A finished screen that planned no cases (no `paid-plan` artifact, which a superseded run never uploads, or a plan manifest with nothing planned) is never followed as a screen: `ready --screen` dispatches a fresh one and `train build` prints `SCREEN: # run planned @@ -554,13 +555,15 @@ repair. When verify confirms the repair merged, it prints `FREEZE: cleared` and `NEXT:` builds a train of the PRs still unmerged. **5b. Coordinator: after the merge.** `gstack-ship-queue verify --post-merge - [--timeout ]` watches every check run attached to the merged commit -(the push workflows: free tests, the Windows subset, the main backstop) until -none is still running, prints one `CHECK: ` per check and + [--timeout ]` (the merged commit's full 40-character SHA) watches +every check run attached to it (the push workflows: free tests, the Windows +subset, the main backstop) and every commit status until none is still +running, prints one `CHECK: ` per check and `POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)`. Green needs at least one completed check and no failure; `neutral` and -`skipped` (the backstop superseding itself) count as skipped. A failure is -`QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is +`skipped` (the backstop superseding itself) count as skipped; a pending +commit status keeps the commit pending and a failed one makes it red. A +failure is `QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is `QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. It keeps no process state, so rerunning it after a machine restart continues the watch. @@ -569,9 +572,10 @@ deletes the `train/` branches of merged, superseded and dropped trains (`WOULD-DELETE:`, `DELETED:`, `GONE:`). `train prune --stale` also settles a `merging` or `green` train whose every PR is MERGED or CLOSED on GitHub (its PRs landed through another train or by hand): it logs each PR's state on a -`STALE:` line, marks the record `stale`, deletes its branch and any -`screen/-*` scratch branches of those PRs; a train with an OPEN PR is kept -(`KEEP: … still needs the train`). `train forget --train | --receipt +`STALE:` line, marks the record `stale` and deletes its branch; a train with an +OPEN PR is kept (`KEEP: … still needs the train`). The same run deletes every +`screen/-*` scratch branch whose PR is MERGED or CLOSED and keeps one whose +PR is still OPEN. `train forget --train | --receipt @ [--dry-run]` removes one record (a settled train with its attempts, or one READY receipt) from local state and from `refs/notes/gstack-trains` with an exact lease on the notes ref, printing `FORGET:` then `FORGOTTEN:`; a @@ -588,14 +592,19 @@ dispatch), waits for the pending run by polling gh, reads the verdict and prints the last step's `NEXT:`. It never dispatches anything the record would not dispatch on its own, so rerunning it after an interruption continues from the record (`WATCH:` lines say what it did; a timeout, default 7200 s, is -`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`). +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`; `--free-timeout `, +default 2700, bounds the wait for the free tests). **Screens are read from the run.** `ready --screen` and `verify --pr --screen` read a dispatched screen's verdict from the run's own artifacts (`gate-binding-a`, `report-verdict-a`), never from a lane's prose, and record it in the READY receipt (`screen.verdict`): `SCREEN_VERDICT: # run GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one -`FLAKY:` line per rule-16 flaky case. A RED screen refuses `ready` under +`FLAKY:` line per rule-16 flaky case. `PENDING` is a run still going, +`UNKNOWN` one the queue cannot read, and `INCOMPLETE` a finished run that bound +no verdict (its report job did not finish): `ready --screen` dispatches afresh +instead of following it, `verify --screen` refuses it (`QUEUE_PROOF_MISSING`) +and `train build` seats the pin unscreened, as for `NOT-RUN`. A RED screen refuses `ready` under either choice and is never re-dispatched at the same merge SHA (`QUEUE_SCREEN_RED`); a run that concluded success with no slice (its route job found the head superseded or the plan selected nothing) is `NOT-RUN`: @@ -605,13 +614,14 @@ unscreened with a `SCREEN: … counted as not-run` line. When GitHub's `refs/pull//merge` lags the PR head or is missing, `ready --screen` plans the screen on a scratch merge of the head onto the base, pushed as `screen/-` (`SCREEN_MERGE:` names it; `prune --stale` deletes it -once the PR merged); only a real conflict with the base is -`QUEUE_SCREEN_STALE_REF`. +once the PR merged or closed); only a real conflict with the base (or a remote +whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. **Stacked tests before the gate.** `train build --prs … --dry-run --stack-tests [--stack-tests-command ""]` runs each PR's new test files (added relative -to the base, under a test directory or named `*.test.*`/`*.spec.*`) on the -integrated prefix that includes it, T, in a scratch worktree, and prints +to the base, with a name ending in `.test`, `_test`, `.spec` or `_spec` and a +JS or TS extension, bun's own rule, so a helper or fixture under `test/` is not +one) on the integrated prefix that includes it, T, in a scratch worktree, and prints `STACK_TESTS: # T new test file(s) pass|FAIL ()` or `no new test files`. A failure is `QUEUE_TRAIN_STACK_RED` (nothing written, pushed or dispatched): the tests pass on the PR head alone only because nothing before diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index e9605e9ee..138f1b9db 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -3818,7 +3818,9 @@ time; `gstack-ship-policy validate` prints `MIN_READER:` for exactly that. coordinator could misread: the working tree has uncommitted tracked changes, HEAD is not the PR's pushed head (read through `gh pr view`), CI at that exact head is not green (failed, pending or unreadable), or the pregate result -is missing, failed or recorded on another tree (`PREGATE_STALE`). +is missing, failed or recorded on another tree (`PREGATE_STALE`). Under +`--dispatch-ci` it is also a dispatched workflow run that concluded red (a red +run clears no pregate lane) or a `gh workflow run` that failed. **Fix.** Commit and push, wait for CI on that head, rerun `gstack-pregate run --allow-repo-commands` when the tree changed, then rerun @@ -3908,7 +3910,8 @@ a `train` block with `min_gstack_version` on the base branch first **Meaning.** A pinned PR's head repository is not the base repository. The train branch and its dispatches run with repository secrets, so fork work -never rides a train. +never rides a train. `ready --dispatch-ci` refuses a fork head for the same +reason and dispatches nothing. **Fix.** Run the printed `train build --prs `; the fork PR stays on the trusted-branch path. @@ -3972,7 +3975,11 @@ one-PR train. ### `QUEUE_TRAIN_PENDING` **Meaning.** The train's run (free tests, the full gate or the bisect) has -not finished, so there is no verdict yet. +not finished, so there is no verdict yet. The same code covers a `train watch` +that reached its timeout, `verify --pr --screen` on a screen run still +going, `verify --post-merge` on checks that have not settled, and a +`ready --dispatch-ci` workflow run still going at its `--timeout`; each prints +the command to rerun as `NEXT:`. **Fix.** Rerun the printed `NEXT:` command later. Nothing is dispatched again while a run is pending. @@ -4110,10 +4117,14 @@ to fix this: the stamp comes only when the slot is called. ### `QUEUE_SCREEN_STALE_REF` -**Meaning.** `gstack-ship-queue ready --screen` dispatches the screen against -GitHub's `refs/pull//merge`, and that ref is missing or still merges an -older head than the PR's pushed head. GitHub stops recomputing the merge ref -while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp +**Meaning.** `ready --screen` refuses with this code when the PR conflicts with +the base beyond the CHANGELOG and the stamp files (the message lists the +conflicting paths) or when it cannot list the remote's refs. A missing or +lagging merge ref alone no longer refuses (see Fix). The rest of this +paragraph is why such a ref cannot be screened directly: `ready --screen` +dispatches the screen against GitHub's `refs/pull//merge`, and that ref can +be missing or still merge an older head than the PR's pushed head. GitHub +stops recomputing the merge ref while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp counts), so a screen dispatched there checks out the old merge, sees itself superseded, plans 0 cases and reports success. GSTA-95 hit this on #3431: `ready` printed `SCREEN_DISPATCH:` as if a screen ran. `ready` now compares @@ -4129,10 +4140,10 @@ read (`… paid-plan artifact has expired`, `… manifest.json is unreadable`, a unreadable run or artifact list, an unreadable repository slug, or a READY whose run id is not a number) keeps the slot screened and prints `SCREEN: # ; counted as screened, unverified`; `ready --screen` -follows such a run. When `ready` cannot list the remote's refs or fetch the -merge ref at all, it refuses with the same code and a `could not list …` or -`could not fetch …` message whose NEXT line says to check the remote, not to -resolve a conflict. +follows such a run. When `ready` cannot list the remote's refs, it refuses with +the same code and a `could not list …` message whose NEXT line says to check +the remote, not to resolve a conflict; a merge ref that lists but cannot be +fetched is replaced by the scratch merge described under Fix. **Fix.** Since the scratch-merge screen landed, a lagging merge ref is no longer a refusal: `ready --screen` builds the merge of the PR head onto @@ -4142,8 +4153,10 @@ base's side, the CHANGELOG merges by section), pushes it as `SCREEN_MERGE: scratch merge …`. The code now means a real conflict: the message lists the conflicting paths. Resolve those with the base branch, push, and run `gstack-ship-queue ready --pr --screen` again. A -CHANGELOG-only or stamp-file-only conflict never refuses. `train prune ---stale` deletes the scratch branches of merged or closed PRs. +CHANGELOG-only or stamp-file-only conflict does not refuse, unless the PR's +CHANGELOG drops or renames a released heading (the message then names the +CHANGELOG). `train prune --stale` deletes the scratch branches of merged or +closed PRs. ### `QUEUE_SCREEN_RED` @@ -4172,8 +4185,9 @@ the flaky cases are listed for the ledger and need no rerun. ### `QUEUE_TRAIN_STACK_RED` **Meaning.** `gstack-ship-queue train build --dry-run --stack-tests` ran a PR's -new test files (files the PR adds relative to the base under a test directory -or named `*.test.*` / `*.spec.*`) on the integrated prefix that includes it, +new test files (files the PR adds relative to the base whose name ends in +`.test`, `_test`, `.spec` or `_spec` plus a JS or TS extension, bun's own +rule; a helper or fixture under `test/` is not one) on the integrated prefix that includes it, T (the base plus every slot up to and including that PR), in a scratch worktree, and they failed. The `STACK_TESTS: # T new test file(s) FAIL (): ` line names the files and the runner's @@ -4191,11 +4205,13 @@ other than `bun test`; the files are appended to the command. ### `QUEUE_POST_MERGE_RED` -**Meaning.** `gstack-ship-queue verify --post-merge ` watched the check -runs attached to a commit the train merged onto the base branch (the push -workflows: free tests, the Windows subset, the main backstop) until none was -still running, and at least one concluded `failure`, `timed_out`, -`cancelled` or another non-success. A `neutral` or `skipped` check (the +**Meaning.** `gstack-ship-queue verify --post-merge ` (the full 40-character +SHA) watched the check runs attached to a commit the train merged onto the +base branch (the push workflows: free tests, the Windows subset, the main +backstop) and its commit statuses until none was still running, and at least +one check concluded `failure`, `timed_out`, `cancelled` or another non-success, +or a commit status failed (the message then says a commit status, not a check +run). A `neutral` or `skipped` check (the backstop superseding itself when main moved on) is counted as skipped, never red. The command keeps no process state: it polls gh until the checks settle or `--timeout` (default 2700 s) passes, prints `POST_MERGE: