diff --git a/changelog.d/gsta157-train-any-repo.md b/changelog.d/capy-ship-queue-merge-trains.md similarity index 100% rename from changelog.d/gsta157-train-any-repo.md rename to changelog.d/capy-ship-queue-merge-trains.md diff --git a/docs/captain.md b/docs/captain.md index 4db6bbaed..b6fe0bb29 100644 --- a/docs/captain.md +++ b/docs/captain.md @@ -248,7 +248,8 @@ a push included. The sequence that works unattended on Capy: ([base_red](ship-policy.md#base-red)): the lane is waiting, not failed. It registers `gstack-base-red wait --pause ` as a background operation, tells the controller which cases hold it, and on `BASE_GREEN` runs - `/ship --resume `; no READY goes out on the conditional receipt. + `/ship --resume `; no READY goes out on the conditional receipt, + except on a train with `gates` ([Merge trains for other repos](#merge-trains-for-other-repos)). 9. READY: `gstack-ship-queue ready --pr --pregate --screen|--no-screen`; send the printed block. 10. After the slot call, the stamp commit keeps the pin's receipt; re-READY with the same @@ -338,7 +339,7 @@ the verdict and lands each landable slot in order. Lanes READY with `ready --pr --no-screen`; there is no screen and no slot call. A lane whose /ship paused on a red base READYs with `--base-red --fixer `, its fixer with `--fixes-base-red `, and the train seats the fixer at slot 1. The `update` commit `train land` pushes to a PR -branch (the gated tree on [pin, main]) is the one exception to CREW rule 19's "never merge +branch (the gated tree on [pin, main]) is the one exception to CREW rule 14's "never merge the base into a PR": it is tool-made, recorded before the push, and pushed under the pin's lease, like a stamped train's stamp commit. A peer crew adopting this starts from diff --git a/docs/ship-policy.md b/docs/ship-policy.md index dc0e17b3a..ec393a3a6 100644 --- a/docs/ship-policy.md +++ b/docs/ship-policy.md @@ -60,7 +60,7 @@ refuses the file (`POLICY_GSTACK_TOO_OLD`). `validate` ends with | `review_scope` | `frozen` \| `diff-plus-blocking` \| `all` | `frozen` | The finding scope of `/ship`'s review loop; see [review_scope](#review-scope). `frozen` fixes the diff on pass 1, files blocking findings in unchanged code as issues, and later passes read only the range since the previous pass. `diff-plus-blocking` is the same but fixes a blocking finding in unchanged code in this PR instead of filing it. `all` is the earlier loop: every finding is fixed on every pass. Needs `min_gstack_version` when not `frozen`. | | `min_gstack_version` | version or `null` | `null` | The oldest gstack that may read the file; see the compatibility rule above. | | `docs` | `{ caps: [{ path, max_bytes }] }` or `null` | `null` | Documentation files with a byte cap. `/ship` Step 14.5 runs `gstack-ship-receipt docs-caps --base ` (the caps come from `origin/`'s policy, never the branch's copy; `--policy ` is for a repository bootstrapping its policy) before the verification gate and prints one `DOCS_CAP: / ok\|over` line per cap; any `over` (an absent file counts) exits 1 and blocks the ship until the file is trimmed or the cap raised on the base branch (`DOCS_AUDIT_BLOCKED`). At most 64 caps, each path contained. See [the ship receipt](#ship-receipt). Needs `min_gstack_version`. | -| `train` | `{ gate_workflow, free_workflow, full_gate_check, record_artifact, max_size, fill_hours, screen_selection, trial_reservation_usd }` or `null` | `null` | Turns on [merge trains](#run-a-merge-train), read from the base revision only. Every field is required and an unknown field is `POLICY_INVALID`: `max_size` 1 to 6, `fill_hours` above 0 and at most 24, `screen_selection` `gstack-paid` \| `none`, `trial_reservation_usd` above 0 and at most 100. `null` keeps the per-PR queue exactly as it is. Needs `min_gstack_version`. | +| `train` | `{ gate_workflow, free_workflow, full_gate_check, record_artifact, max_size, fill_hours, screen_selection, trial_reservation_usd, land? }`, `{ gates, land, max_size, fill_hours }` or `null` | `null` | Turns on [merge trains](#run-a-merge-train), read from the base revision only. Every field is required and an unknown field is `POLICY_INVALID`: `max_size` 1 to 6, `fill_hours` above 0 and at most 24, `screen_selection` `gstack-paid` \| `none`, `trial_reservation_usd` above 0 and at most 100. A block with `gates` ([any repository](#train-gates)) takes `max_size` 1 to 5 and needs `min_gstack_version` at least `TRAIN_GATES_KEY_VERSION`. `null` keeps the per-PR queue exactly as it is. Needs `min_gstack_version`. | | `base_red` | `{ setup, select, timeout_sec }` or `null` | `null` | How `/ship` Step 5 proves a failure is the base branch's own before pausing on it; see [base_red](#base-red). Read from the base revision only. Needs `min_gstack_version`. | | `changelog_fragments` | `{ dir, required_checks? }` or `null` | `null` | PRs carry unversioned [CHANGELOG fragments](#changelog-fragments) in `dir` (one flat, repo-relative directory, no `..`) and the merge step ([`land`](#land)) assigns the version. `required_checks` (up to 50 distinct names) are the checks [`train land`](#stamped-trains-under-changelog_fragments-train-land) requires present and green on a stamp commit. `null` keeps `[Unreleased]` sections and per-PR stamps. Needs `min_gstack_version`. | @@ -479,7 +479,9 @@ prefixes. rerun is GREEN with a `FLAKY:` line, red again is real, and both attempts' jobs stay in the record. `post_merge_checks` names checks that must succeed on main after each land before the next slot lands (`verify --post-merge - --checks `; `train watch --land` runs it between slots). A workflow whose concurrency + --checks `; `train land` checks the previous slot's merge and + refuses the next slot until they are green, and `train watch --land` runs it + between slots). A workflow whose concurrency group is not keyed by `github.ref` (or the nonce) gets one prefix at a time. - **`command` gates.** Each prefix is checked out clean and detached and runs @@ -487,8 +489,8 @@ prefixes. (standard-16, `UBI_OWNER` required). The VM's `git write-tree` must equal the prefix tree, and exit 0 is GREEN. Only names in `pass_env` reach the VM (`UBICLOUD_*` never). Each run reserves its timeout plus 10 minutes at the - list price, and one `spend_cap_usd` covers every run of the train, reruns - included (3 dollars covers six prefixes and one rerun at 1800 s; `train init` + list price, and each command gate's `spend_cap_usd` covers every run of that + gate in the train, reruns included (3 dollars covers six prefixes and one rerun at 1800 s; `train init` prints that default and `train doctor` flags a cap below a full train). The next run past the cap refuses (`QUEUE_TRAIN_SPEND_CAP`); raising the cap is a policy commit that moves main, so it is followed by a new train. @@ -545,7 +547,8 @@ declares fragments nor stamps at merge. After the block lands on the base, `train doctor` re-checks it: a missing workflow, nonce input or run-name, a required job no workflow defines, a concurrency group not keyed by ref, a push trigger that matches `train-prefix/` refs, a workflow never green on the base, -and a merge method GitHub has turned off (`QUEUE_TRAIN_DOCTOR_FINDINGS`). +a command gate cap below what a full train reserves, a merge method GitHub has +turned off, and `rebase` as the merge method (`QUEUE_TRAIN_DOCTOR_FINDINGS`). Neither command writes anything. @@ -883,7 +886,7 @@ repair. When verify confirms the repair merged, it prints `FREEZE: cleared` and `NEXT:` builds a train of the PRs still unmerged. **5b. Coordinator: after the merge.** `gstack-ship-queue verify --post-merge - [--timeout ]` (the merged commit's full 40-character SHA) watches + [--timeout ] [--checks ]` (the merged commit's full 40-character SHA) watches every check run attached to it (the push workflows: free tests, the Windows subset, the main backstop) and every commit status until none is still running, prints one `CHECK: ` per check and @@ -892,7 +895,10 @@ Green needs at least one completed check and no failure; `neutral` and `skipped` (the backstop superseding itself) count as skipped; a pending commit status keeps the commit pending and a failed one makes it red. A failure is `QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is -`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. It keeps no process +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. With `--checks` (a +workflows gate's `post_merge_checks`), each named check must also run and +succeed: one that did not succeed is `QUEUE_POST_MERGE_RED`, and one that +never ran is `QUEUE_TRAIN_PENDING` at the timeout. It keeps no process state, so rerunning it after a machine restart continues the watch. **6. Clean up.** `gstack-ship-queue train prune [--train ] [--dry-run]` @@ -1189,7 +1195,9 @@ expiry), finishes every other step, and writes its receipt with `--base-red `expires_at`) makes it **conditional**. `gstack-ship-receipt result` prints `SHIP_PAUSED: base-red …` and `status=incomplete`; `ready`, `train build`, `verify --train`, `carry` and every other receipt consumer refuse a conditional -receipt with `QUEUE_RECEIPT_CONDITIONAL`. +receipt with `QUEUE_RECEIPT_CONDITIONAL`. The one exception is a train with +`gates`: `ready --base-red --fixer ` READYs the PR to ride behind its fixer +([base-red PRs](#train-gates)). To resume, a parent runs `gstack-base-red wait --pause ` as a background operation. On each new base head (`git ls-remote`) it reruns `setup` and diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index ee0fab2d6..9d0732bab 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -4555,9 +4555,12 @@ one-PR train. **Meaning.** The train's run (free tests, the full gate or the bisect) has not finished, so there is no verdict yet. The same code covers a `train watch` that reached its timeout, `verify --pr --screen` on a screen run still -going, `verify --post-merge` on checks that have not settled, and a -`ready --dispatch-ci` workflow run still going at its `--timeout`; each prints -the command to rerun as `NEXT:`. +going, `verify --post-merge` on checks that have not settled, a +`ready --dispatch-ci` workflow run still going at its `--timeout`, a `train +build` or `train verdict` while another process holds the train's lease or +runs one of its command gates, and a `train land` whose previous slot's +post-merge checks have not settled; each prints the command to rerun as +`NEXT:`. **Fix.** Rerun the printed `NEXT:` command later. Nothing is dispatched again while a run is pending. @@ -4574,6 +4577,10 @@ record. An incomplete gate is never green. one re-dispatch EVAL_POLICY.infraRedispatch allows (`train rebuild --train --infra`) and report both runs. Otherwise diagnose the run. +On a train with `gates` it means a prefix gate is incomplete (a required job +that was skipped or never ran, for example); the `PREFIX:` line names the +run. `train rebuild` refuses, so diagnose that run and build a new train. + ### `QUEUE_TRAIN_RED` @@ -4584,6 +4591,11 @@ verdict. **Fix.** Bisect the red cases (`train bisect --train --spend-cap `). After a rebuild's second red, route the cases to a repair lane. +On a train with `gates` there is no bisect: T is the first red prefix after +a green one and every later prefix is red, so slot i is the culprit. The slots +before it land (`train land`, or `train watch --land`), the culprit goes back +to its lane, and `NEXT:` builds the rest into a new train. + ### `QUEUE_TRAIN_RED_ON_MAIN` @@ -4593,6 +4605,9 @@ included: main is broken, not the train's PRs. **Fix.** Fix forward on main first (a one-PR train), then rebuild this train unchanged. +On a train with `gates`, no bisect runs: T0 (main) is red on its own gate run, +or red on anything other than exactly slot 1's `FIXES_BASE_RED` cases. + ### `QUEUE_TRAIN_UNATTRIBUTED` @@ -4603,6 +4618,10 @@ monotone. One trial per prefix cannot tell a flake from a break. `), reporting both runs. Any other pattern: a repair lane, then a changed train once the repair merges. +On a train with `gates` it means a red prefix is followed by a green one. +Repair the red cases or seat the PRs in another order (`--order`); `train +rebuild` refuses, so this train never runs again unchanged. + ### `QUEUE_TRAIN_BISECT_CAPPED` @@ -4687,8 +4706,8 @@ again by its lane ([ship-policy.md](ship-policy.md#train-gates)). ### `QUEUE_TRAIN_SPEND_CAP` -**Meaning.** A command gate's next Ubicloud run would pass the train's one -spend cap, or `UBI_OWNER` is not set. Nothing more ran; finished runs keep +**Meaning.** A command gate's next Ubicloud run would pass that gate's spend +cap for the train, or `UBI_OWNER` is not set. Nothing more ran; finished runs keep their outcomes. **Fix.** Set `UBI_OWNER` to your thread code and rerun `train build --train @@ -4725,7 +4744,8 @@ stamp-at-merge` on the base branch, then run `train init` again. `FINDING:` line names one problem: a gate workflow missing, without the `nonce` input or run-name, a required job it does not define, a global concurrency group, a push trigger on `train-prefix/` refs, a workflow never -green on the base, or a merge method GitHub refuses. +green on the base, a command gate cap below what a full train reserves, a +merge method GitHub refuses, or `rebase` as the merge method. **Fix.** Fix each finding on the base branch (`train init` prints the snippet), then run `train doctor` again. @@ -4813,7 +4833,9 @@ to fix this: the stamp comes only when the slot is called. **Meaning.** The PR body's ship receipt carries a `base_red` block: /ship's Step 5 failed on cases `gstack-base-red` proved are the base branch's own, so /ship paused before READY ([base_red](ship-policy.md#base-red)). Nothing -READYs, trains, carries or merges on it. +READYs, trains, carries or merges on it, except on a train with `gates`, where +`ready --base-red --fixer ` READYs it to ride behind its fixer at slot 1 +([base-red PRs](ship-policy.md#train-gates)). **Fix.** Wait for the base to pass those cases (`gstack-base-red wait --pause `), then run `/ship --resume `, which reruns Step 5 and rewrites @@ -5005,7 +5027,10 @@ or `--timeout` (default 2700 s) passes, prints `POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)` and one `CHECK: ` line per check, and a timeout is `QUEUE_TRAIN_PENDING` with the same command as `NEXT:`, so rerunning it after -an interruption continues the watch. +an interruption continues the watch. With `--checks ` (a workflows +gate's `post_merge_checks`, which `train land` and `train watch --land` pass), +each named check must also succeed; one that never ran is `QUEUE_TRAIN_PENDING` +at the timeout. **Fix.** Read the red run named on its `CHECK:` line. Repair on a new PR or roll back the merge; `gstack-ship-queue train status` names the freeze if the