CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-11-01 06:24:41 +01:00
Code Issues Packages Projects Releases Wiki Activity

v1.91.146.0 feat(ship-queue): merge trains with pattern bisect and train-record verify (GSTA-49 T1) (#3272)

Browse Source 
* feat(ship-queue): merge trains with pattern bisect and train-record verify (GSTA-49 T1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test(windows): curate ship-queue-train.test.ts out of the Windows subset (gh shim not resolved on windows-latest)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.145.0 feat(ship-queue): merge trains with pattern bisect and train-record verify (GSTA-49 T1)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.146.0 restamp: #3295 takes v1.91.145.0

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-19 10:29:44 +00:00
1 parent a09b3b8532
commit 6a4ab69dcd
25 files changed
+3047 -36
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/ship-policy.md
+184 -1
View File
@@ -18,7 +18,7 @@ introduces or changes the policy runs under the reviewed bootstrap path
Tools: `gstack-ship-policy init|validate|show-effective|check-history`,
`gstack-restamp`, `gstack-tree-receipt`, `gstack-ship-receipt`,
`gstack-ship-queue` ([shared merge queue](#run-a-shared-merge-queue)),
`gstack-ship-queue` ([shared merge queue](#run-a-shared-merge-queue), [merge trains](#run-a-merge-train)),
`gstack-evidence bundle|ancestor`. They share the exit table (`0` ok, `1`
fail, `2` usage, `3` refused or needs a flag) and end every error line with a
code from [troubleshooting.md](troubleshooting.md#queue-aware-ship-policy-restamp-tree-receipt-ship-receipt).
@@ -58,6 +58,7 @@ refuses the file (`POLICY_GSTACK_TOO_OLD`). `validate` ends with
| `gate_command` | command or `null` | `null` | The repo's full pre-merge gate. `gstack-ship-queue gate-ahead` and `ready` print it for the caller to run (in the background or in CI); gstack never executes it. Needs `min_gstack_version`. |
| `full_e2e` | `true` \| `false` | `false` | `true`: every merge needs a passing full-gate result bound to a tree equal to the merging one modulo stamps; `gstack-ship-queue ready`, `stamp` and `verify` refuse without one (`QUEUE_GATE_MISSING`): a gate-ahead record, or the lane's `--gate-exit 0 --gate-log <file>` from its run on the gated head. Needs `min_gstack_version`. |
| `min_gstack_version` | version or `null` | `null` | The oldest gstack that may read the file; see the compatibility rule above. |
| `train` | `{ gate_workflow, free_workflow, full_gate_check, record_artifact, max_size, fill_hours, screen_selection, trial_reservation_usd }` or `null` | `null` | Turns on [merge trains](#run-a-merge-train), read from the base revision only. Every field is required and an unknown field is `POLICY_INVALID`: `max_size` 1 to 6, `fill_hours` above 0 and at most 24, `screen_selection` `gstack-paid` \| `none`, `trial_reservation_usd` above 0 and at most 100. `null` keeps the per-PR queue exactly as it is. Needs `min_gstack_version`. |
Containment, checked by `validate` and again at write time: paths and globs are
repo-relative with no `..` segment, under 256 characters, at most 64 per list,
@@ -254,6 +255,188 @@ prints) only after every lane's gstack is at least that version, the crew's
install floor (`GSTACK_MIN_REVISION`). A lane on an older gstack refuses the
policy by name rather than reading it differently.
<a id="run-a-merge-train"></a>
## Run a merge train
A merge train gates several READY PRs with one full gate instead of one gate
each. The coordinator merges the pinned, unstamped PR heads in slot order onto
main in a scratch worktree, records each prefix tree (T0 is main, Tk is the
train head), runs the free tests and then one full gate on the train head, and
merges the PRs one at a time, each only while main's tree equals the prefix
the gate covered modulo stamps. A red train is bisected on its red cases only.
The surface is off unless the base revision's `.gstack/ship-policy.json`
declares a `train` block. Without one, `train …` and `verify --train` refuse
with `QUEUE_TRAIN_OFF`, and `ready --pr <n>` prints exactly the per-PR READY
block above (`--screen`, `--no-screen` and `--broad` are ignored with a
warning on stderr). A block for gstack itself would read:
```json
"train": {
"gate_workflow": "evals.yml", "free_workflow": "free-tests.yml",
"full_gate_check": "gstack/full-gate", "record_artifact": "gstack-train-record",
"max_size": 5, "fill_hours": 3, "screen_selection": "gstack-paid",
"trial_reservation_usd": 1.5
},
"min_gstack_version": "<the version that introduced train>"
```
gstack's own policy does not carry it yet: trains go live for gstack only
after the workflow side (the train-nonce record, the `cases` and
`prefix_shas` inputs, the dispatched screen) has merged and one dry-run train
of two already-merged heads has passed. `gate_workflow` is dispatched with
`evals_all`, `expected_sha` and `nonce` (bisects add `cases` and
`prefix_shas`, screens add `screen_pr`, `screen_merge_sha` and
`screen_policy`). That run publishes `full_gate_check` as a check run on the
train head and the train record as the `record_artifact` artifact only when
its census is complete and green; gstack reads both and never writes a
check. `screen_selection: gstack-paid` computes gstack's paid case selection
for the screen recommendation; `none` leaves the choice to the lane.
`trial_reservation_usd` is what the bisect reserves for a trial without cost
data.
Every command prints its lines, the coded error line on stderr, then
`NEXT: <command>` and `GSTACK_RESULT: skill=ship-queue status=… run=<record>`.
Records live under `$GSTACK_STATE_ROOT/ship-queue/` (`trains/<nonce>/train.json`
with one attempt record per dispatch, `ready/` receipts and `freeze/`) and
are mirrored to origin as git notes on `refs/notes/gstack-trains` before any
remote effect, so a lost machine loses nothing: `train status` on a clean
state root rebuilds them. `GSTACK_SHIP_QUEUE_NOW` pins the clock for tests.
**1. Lane: READY with a screen choice.** Under a train policy READY needs an
explicit choice:
```bash
gstack-ship-queue ready --pr 42 # prints SCREEN: lines and the recommendation, refuses
gstack-ship-queue ready --pr 42 --no-screen # READY receipt; the PR rides unscreened
gstack-ship-queue ready --pr 42 --screen [--broad] # dispatches one paid screen on the PR's merge SHA
```
Without a choice it prints the `SCREEN:` selection (with
`screen_selection: gstack-paid`, the skill's own cases and `recommend
--screen` or `--no-screen`: `--screen` when the screen selects a skill's own
cases) and refuses with `READY_SCREEN_CHOICE_REQUIRED`. The price line says
the dollar figure is pending the cost replay; until it lands, `SCREEN:` shows
case counts and the recommendation only. A lane that already ran focused
local cases says so in the READY block and may pass `--no-screen`. `--screen`
dispatches `gate_workflow` with `screen_pr`, `screen_merge_sha` (the
`refs/pull/<n>/merge` SHA) and `screen_policy` (`narrow`, or `broad` with
`--broad`, today's full PR selection). The nonce key carries the purpose and
the policy, so a repeat follows the same run and a broad screen never adopts
a narrow one. The READY block gains `screen: …`, and the receipt is written
locally and to the notes ref. `--dry-run` prints the dispatch without sending
it.
**2. Coordinator: build.** Name the pins exactly as READY reported them:
```bash
gstack-ship-queue train build --prs 41@<sha>,42@<sha>,43@<sha> --dry-run
gstack-ship-queue train build --prs 41@<sha>,42@<sha>,43@<sha>
```
It prints `ORDER:` (the slot order to announce: screened PRs first in READY
order, then unscreened; when `max_size` or more unscreened PRs wait they form
their own train and `HELD:` lists them), one `PREFIX: T<i> <commit>
tree=<tree>` per prefix and one `SLOT:` per PR. `--dry-run` adds the
`ESTIMATE:` line and writes, pushes and dispatches nothing. Otherwise it
writes `train.json`, pushes `train/<nonce>`, dispatches `free_workflow` there
and waits (`--free-timeout <s>`, default 2700) for a green
`workflow_dispatch` run on that exact SHA, then dispatches the full gate
through eval-bg's nonce. A train under `max_size` waits until its oldest READY
has waited `fill_hours` (`QUEUE_TRAIN_WAITING`); `--urgent` dispatches a
one-PR train at once. The same pins on the same base return the live train
instead of a second one; `train build --train <nonce>` resumes a train from
its record. Refusals, all before any spend: `QUEUE_TRAIN_UNTRUSTED` (a fork
head), `QUEUE_TRAIN_MOVED` (a head moved from its pin), `QUEUE_NOT_READY` (no
READY receipt at that head), `QUEUE_TRAIN_CONFLICT` (the printed
`QUEUE_TRAIN_CONFLICT: train <id> stopped at #<pr> (conflicts: <files>) after
merging #<a>,#<b>; nothing dispatched, no branch pushed; fix: …` line names
the build without that PR) and `QUEUE_TRAIN_FREE_RED` (free tests red,
missing, cancelled or on another SHA). In a merge, version files take the
prefix's side and the CHANGELOG merges by section, since the stamp at merge
rewrites both.
**3. Coordinator: status and verdict.**
```bash
gstack-ship-queue train status [--train <nonce>]
gstack-ship-queue train verdict --train <nonce>
```
`status` never spends. When a dispatch was attempted but its run never showed
up (`QUEUE_TRAIN_DISPATCH_UNKNOWN`), it lists the workflow's dispatch runs and
either adopts the run carrying the nonce or marks the dispatch `absent`; only
an absent dispatch is sent again, by `train build --train <nonce>`, under the
same nonce. `verdict` reads the run, its record and the `full_gate_check`
check: `GREEN` only for a complete census bound to the train nonce, head and
tree; `QUEUE_TRAIN_PENDING` while it runs; `QUEUE_TRAIN_INCOMPLETE` for INFRA
or INCOMPLETE panels, unbound, partial or cancelled shards, a slice killed
before upload, or no record (never green); `QUEUE_TRAIN_RED` with the red
cases.
**4. A red train: bisect, then at most one rebuild.**
```bash
gstack-ship-queue train bisect --train <nonce> --spend-cap 5 --dry-run
gstack-ship-queue train bisect --train <nonce> --spend-cap 5
gstack-ship-queue train rebuild --train <nonce>
```
The bisect dispatches the red cases (`cases`) on every prefix T0..Tk
(`prefix_shas`), one trial each, after checking every prefix is an ancestor
of `train/<nonce>` (`QUEUE_TRAIN_PREFIX_INVALID`). It reserves every
case×prefix trial (unknown costs at `trial_reservation_usd`) and launches only
the prefixes that fit the cap (`QUEUE_TRAIN_BISECT_CAPPED`; completed prefixes
are kept and a rerun with a higher cap runs only the rest). Rerun it to read
the results. The pattern decides: `ATTRIBUTED #<pr>` only when monotone
(green before, red from that PR on), and `NEXT:` builds the train without it;
red on every prefix with T0 included is `QUEUE_TRAIN_RED_ON_MAIN` (fix forward
on main first); anything else is `QUEUE_TRAIN_UNATTRIBUTED`. An all-green
bisect earns exactly one unchanged rebuild (`train rebuild`, reporting both
runs); a non-monotone pattern earns none and goes to a repair lane. Every
other rebuild is `QUEUE_TRAIN_REBUILD_REFUSED`, and a second red stays red.
An INFRA/INCOMPLETE-only census earns the single re-dispatch
EVAL_POLICY.infraRedispatch allows: `train rebuild --train <nonce> --infra`.
The rebuild-once rule lives in the train module beside that constant until
EVAL_POLICY records it. Bisect results never turn a train green.
**5. Coordinator: verify, merge, verify again.** For each slot in order:
```bash
gstack-ship-queue verify --pr 41 --train <nonce> # MERGE_CONDITION: sha=<head>
# merge #41 with that expected head; gstack-restamp stamps it at merge
gstack-ship-queue verify --pr 41 --train <nonce> # VERIFY_TRAIN: #41 merged; main's tree is T1 …
```
Before a merge, main's tree must equal the prefix before the PR modulo stamps
and the PR must still be at its pin (stamps aside); after it, main's tree
must equal the prefix after it. Anything else is `QUEUE_TRAIN_TREE_MISMATCH`
(or `QUEUE_TRAIN_MOVED`). When nothing of the train has merged, that is a plain
refusal: build a new train on current main. When earlier slots already
merged, main holds a prefix no gate covered on its own, so verify freezes the
queue: the freeze goes to local state and the notes ref first, then the full
gate is dispatched on main's SHA at once (the backstop), and the
`QUEUE_TRAIN_FROZEN:` line lists the merged PRs, main's tree and the backstop
run. While frozen, ordinary trains and other trains' merges are refused
(`QUEUE_TRAIN_FROZEN`). Admit one repair PR at its exact head:
```bash
gstack-ship-queue train build --repair --prs 44@<sha>
```
It runs the normal free tests, full gate and verification on main plus the
repair. When verify confirms the repair merged, it prints `FREEZE: cleared`
and `NEXT:` builds a train of the PRs still unmerged.
**6. Clean up.** `gstack-ship-queue train prune [--train <nonce>] [--dry-run]`
deletes the `train/<nonce>` branches of merged, superseded and dropped trains
(`WOULD-DELETE:`, `DELETED:`, `GONE:`).
**Escape hatches.** Remove the `train` block on the base branch to return
every lane to the per-PR queue at once. `--urgent` ships one PR alone,
`--no-screen` skips the paid screen, `--broad` widens it to today's selection,
and `train build --prs <list without it>` drops a PR at any step.
<a id="ship"></a>
## What `/ship` reads from it
docs/troubleshooting.md
+192
View File
@@ -3684,6 +3684,198 @@ and the current one. Record the live attempt, or prepare a new one with
`gate-ahead --after <pr>` and run the gate there without editing tracked
files.
<a id="queue-train-off"></a>
### `QUEUE_TRAIN_OFF`
**Meaning.** A merge-train command (`train …` or `verify --train`) ran
against a base revision whose `.gstack/ship-policy.json` declares no `train`
block, so the train surface is off and the per-PR queue applies. Nothing was
written.
**Fix.** Use the per-PR queue (`ready --pr <n>`, `stamp`, `verify`), or land
a `train` block with `min_gstack_version` on the base branch first
([Run a merge train](ship-policy.md#run-a-merge-train)).
<a id="queue-train-untrusted"></a>
### `QUEUE_TRAIN_UNTRUSTED`
**Meaning.** A pinned PR's head repository is not the base repository. The
train branch and its dispatches run with repository secrets, so fork work
never rides a train.
**Fix.** Run the printed `train build --prs <list without it>`; the fork PR
stays on the trusted-branch path.
<a id="queue-train-moved"></a>
### `QUEUE_TRAIN_MOVED`
**Meaning.** A PR's head is no longer the SHA the train pinned (at build, or
past its pin beyond stamps at verify), so the train would not test what
merges. At verify after earlier slots merged, the queue also freezes
(`QUEUE_TRAIN_FROZEN`).
**Fix.** Ask the lane for a fresh READY on the new head, then build the train
with the new pin (`train build --prs <n>@<new sha>,…`).
<a id="queue-train-conflict"></a>
### `QUEUE_TRAIN_CONFLICT`
**Meaning.** Merging a PR onto the prefix before it conflicts outside the
version files and the CHANGELOG. The `QUEUE_TRAIN_CONFLICT:` line names the
PR, the conflicting files and the PRs merged before it. Nothing was
dispatched and no branch was pushed.
**Fix.** Run the printed `train build --prs <list without that PR>`. The PR's
lane merges main (with the earlier PRs once they land) and reports READY
again.
<a id="queue-train-free-red"></a>
### `QUEUE_TRAIN_FREE_RED`
**Meaning.** The free tests dispatched on `train/<nonce>` are red, cancelled,
missing (no `workflow_dispatch` run appeared before `--free-timeout`) or ran
on another SHA, so the paid gate was not dispatched.
**Fix.** Read the free-tests run. A PR-caused red drops that PR (`train build
--prs <list without it>`); a flaky test goes to its owner before the next
build.
<a id="queue-train-dispatch-unknown"></a>
### `QUEUE_TRAIN_DISPATCH_UNKNOWN`
**Meaning.** A train dispatch was attempted, but no run carrying its nonce
appeared, so whether it started is unknown (`gh workflow run` can fail after
GitHub accepted the dispatch). It is never dispatched again blindly.
**Fix.** Run `gstack-ship-queue train status --train <nonce>`. It adopts the
run carrying the nonce, or marks the dispatch absent; only then does
`train build --train <nonce>` dispatch it again, under the same nonce.
<a id="queue-train-waiting"></a>
### `QUEUE_TRAIN_WAITING`
**Meaning.** The train has fewer PRs than `train.max_size` and its oldest
READY PR has waited less than `train.fill_hours`, so it does not dispatch
yet. The message prints when it will.
**Fix.** Add READY PRs, rerun after the printed time, or pass `--urgent` for a
one-PR train.
<a id="queue-train-pending"></a>
### `QUEUE_TRAIN_PENDING`
**Meaning.** The train's run (free tests, the full gate or the bisect) has
not finished, so there is no verdict yet.
**Fix.** Rerun the printed `NEXT:` command later. Nothing is dispatched again
while a run is pending.
<a id="queue-train-incomplete"></a>
### `QUEUE_TRAIN_INCOMPLETE`
**Meaning.** The train gate did not complete its census: INFRA or INCOMPLETE
panels, an unbound, partial or cancelled shard, a slice killed before its
upload, a record bound to another nonce, head or tree, or no published
record. An incomplete gate is never green.
**Fix.** When every gap is machine-classified INFRA or INCOMPLETE, take the
one re-dispatch EVAL_POLICY.infraRedispatch allows (`train rebuild --train
<nonce> --infra`) and report both runs. Otherwise diagnose the run.
<a id="queue-train-red"></a>
### `QUEUE_TRAIN_RED`
**Meaning.** The train's full gate is red on the train tree; the message names
the red cases. Its PRs do not merge, and nothing is retried to change the
verdict.
**Fix.** Bisect the red cases (`train bisect --train <nonce> --spend-cap
<usd>`). After a rebuild's second red, route the cases to a repair lane.
<a id="queue-train-red-on-main"></a>
### `QUEUE_TRAIN_RED_ON_MAIN`
**Meaning.** The bisect found the red cases red on every prefix, main (T0)
included: main is broken, not the train's PRs.
**Fix.** Fix forward on main first (a one-PR train), then rebuild this train
unchanged.
<a id="queue-train-unattributed"></a>
### `QUEUE_TRAIN_UNATTRIBUTED`
**Meaning.** The bisect pattern blames no PR and not main: all green, or not
monotone. One trial per prefix cannot tell a flake from a break.
**Fix.** All green: the one preregistered rebuild (`train rebuild --train
<nonce>`), reporting both runs. Any other pattern: a repair lane, then a
changed train once the repair merges.
<a id="queue-train-bisect-capped"></a>
### `QUEUE_TRAIN_BISECT_CAPPED`
**Meaning.** The bisect's reserved price (every case×prefix trial, trials
without cost data at `train.trial_reservation_usd`) passes `--spend-cap`.
Prefixes that fit were launched, completed prefixes are kept, and the
outcome stays unattributed; it is never green.
**Fix.** Resume with a higher cap (`train bisect --train <nonce> --spend-cap
<usd>`). Only prefixes without a result run.
<a id="queue-train-prefix-invalid"></a>
### `QUEUE_TRAIN_PREFIX_INVALID`
**Meaning.** A bisect prefix is not an ancestor of the train head on
`train/<nonce>` (the branch moved), or a red case is not in the gate's plan,
so the bisect inputs are refused before any dispatch.
**Fix.** Check that `train/<nonce>` still points at the recorded head
(`train status`). A moved branch needs a new train.
<a id="queue-train-rebuild-refused"></a>
### `QUEUE_TRAIN_REBUILD_REFUSED`
**Meaning.** `train rebuild` is not allowed: there is no bisect record (or,
with `--infra`, no INFRA/INCOMPLETE-only census), the bisect was not all
green, or the train was already rebuilt or re-dispatched once.
**Fix.** Route the red cases to a repair lane; the train is rebuilt only once
the repair merges (a changed train).
<a id="queue-train-tree-mismatch"></a>
### `QUEUE_TRAIN_TREE_MISMATCH`
**Meaning.** Main's tree is neither the train's prefix before this PR nor the
one after it, modulo stamps, so the merge would not publish what the gate
tested.
**Fix.** Stop merging. If earlier PRs of the train merged, the queue is now
frozen (`QUEUE_TRAIN_FROZEN`); otherwise build a new train on current main.
<a id="queue-train-frozen"></a>
### `QUEUE_TRAIN_FROZEN`
**Meaning.** A train stopped partway through publication, so main holds a
prefix no gate covered on its own. The `QUEUE_TRAIN_FROZEN:` line lists the
merged PRs, main's tree and the backstop run dispatched on main's SHA. The
freeze lives in local state and on `refs/notes/gstack-trains`. Ordinary
trains and other trains' merges are refused until a repair lands.
**Fix.** Admit one repair PR at its exact head (`train build --repair --prs
<n>@<sha>`). The freeze clears when verify finds main's tree after that merge
in a green record; `NEXT:` then builds a train of the PRs still unmerged.
<a id="ready-screen-choice-required"></a>
### `READY_SCREEN_CHOICE_REQUIRED`
**Meaning.** The base policy turns trains on, so READY needs an explicit
screen choice: `--screen` (pay for a screen on the PR's merge SHA now) or
`--no-screen`. The `SCREEN:` lines show the selection and the recommendation.
**Fix.** Rerun `gstack-ship-queue ready` with the printed recommendation. A
lane that already ran focused local cases says so in the READY block and may
pass `--no-screen`.
<a id="pregate-timing-assertion"></a>
### `PREGATE_TIMING_ASSERTION`