diff --git a/CHANGELOG.md b/CHANGELOG.md index c81ff0298..304353838 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## [1.91.263.0] - 2026-10-27 + +**Merge trains read their screens from the run, follow themselves without a daemon, and refuse what would have cost a gate.** Operating trains on 2026-10-26 showed the gaps: two screens were VERDICT RED while their lanes reported green and the heads were READY'd anyway; 18 of 41 PR screens concluded success with no slice; a PR whose merge ref lagged main could not be screened at all (#3465); two trains whose PRs merged elsewhere sat in `merging`/`green` forever; a background `gh run watch` chain died with every machine restart; a PR's new tests passed on its branch and failed once stacked behind another PR; and the four-step hand procedure for a conflicting PR's CI was typed by hand each time. + +### Added +- **`lib/ship-queue-screen.ts`**: a dispatched screen's verdict is read from the run's own artifacts (`gate-binding-a`: verdict, reconcile exit, flaky cases; `report-verdict-a/paid-report/collector-outcomes.json`: the panels that failed the lane and the rule-16 flaky lines; a confirmed verdict reuse in `paid-plan/gate-reuse.json`), never from a lane's prose. `ready --screen`, `verify --pr --screen` and `train build` print `SCREEN_VERDICT: # run GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one `FLAKY:` line per flaky case, and record the verdict in the READY receipt (`screen.verdict`, locally and in `refs/notes/gstack-trains`). +- **`QUEUE_SCREEN_RED`**: a RED screen refuses `ready` under `--screen` and `--no-screen` alike, is never re-dispatched at the same merge SHA, and `train build` refuses to seat the pin. **`QUEUE_SCREEN_NOT_RUN`**: a screen run that concluded success with no slice (its route job found the head superseded by a push, or the plan selected nothing) is `NOT-RUN`: `verify --screen` refuses, the receipt records `NOT_RUN`, `ready --screen` dispatches afresh instead of following it, and `train build` seats the pin unscreened with a `SCREEN: … counted as not-run` line. +- **Scratch-merge screens (#3465)**: when GitHub's `refs/pull//merge` lags the PR head, is missing or cannot be fetched, `ready --screen` plans the screen on a scratch merge of the head onto the base, pushed as `screen/-` (`SCREEN_MERGE:` names it); only a real conflict with the base (or a remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`, whose text now says so, and a base that cannot be fetched is `POLICY_SOURCE_UNAVAILABLE`. +- **`train watch --train [--timeout ]`** (`lib/ship-queue-train-watch.ts`): the one resumable follow command. It reads the record, runs the step its state asks for (`train build --train` to push, run the free tests and dispatch the gate; `train status` to settle an unconfirmed dispatch), waits for the pending run by polling gh, reads the verdict and prints the last step's `NEXT:`. No process state: rerun it after an interruption and it continues from the record; it never dispatches anything the record would not dispatch on its own. +- **`train prune --stale`**: a `merging` or `green` train whose every PR is MERGED or CLOSED on GitHub is marked `stale` (a new terminal state), its branch deleted, each PR's state logged on a `STALE:` line; a train with an OPEN PR is kept and says why. The same run deletes every `screen/-*` scratch branch whose PR is MERGED or CLOSED. **`train forget --train | --receipt @ [--dry-run]`** (`lib/ship-queue-train-forget.ts`) removes one settled train (attempts included) or one READY receipt from local state and from the notes ref with an exact `--force-with-lease` on `refs/notes/gstack-trains`; a train still building, running or merging is refused, and a rejected lease removes nothing. +- **`verify --post-merge [--timeout ]`** (`lib/ship-queue-post-merge.ts`): after a merge, waits for every check run and commit status attached to the merged commit (its full 40-hex SHA) to settle, prints one `CHECK: ` per check and `POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)`; green needs a completed check and no failure, `neutral`/`skipped` count as skipped, a pending commit status keeps it pending, a failed check or status is **`QUEUE_POST_MERGE_RED`**, a timeout is `QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. +- **`ready … --dispatch-ci [--timeout ]`** (`lib/ship-queue-dispatch-ci.ts`): for a PR that conflicts with the base and so has no pull_request CI, dispatches each named workflow on the branch (skipped when a run at this head already concluded), waits for the runs (`CI_DISPATCH:` lines), clears every requires-remote pregate lane whose workflow is one of them with the run URL (`PREGATE_CLEARED:`), then reads the head as usual. A red run clears no lane and refuses; a fork head is `QUEUE_TRAIN_UNTRUSTED`. This replaces the four-step hand procedure in docs/ship-policy.md. +- **`train build --dry-run --stack-tests [--stack-tests-command ""]`** (`lib/ship-queue-stack-tests.ts`): runs each PR's new test files on the integrated prefix that includes it (T) in a scratch worktree and prints `STACK_TESTS: # T new test file(s) pass|FAIL ()` or `no new test files`; a failure is **`QUEUE_TRAIN_STACK_RED`** with nothing written, pushed or dispatched. + +### Changed +- **`lib/ship-queue-train-github.ts`**: `downloadJson` finds a named file anywhere in a multi-path artifact (`report-verdict-a1` keeps its `paid-report/` directory). **`lib/ship-queue.ts`**: `ciAtHead` is exported and returns the per-check list it already read. **`scripts/lib/windows-curation.ts`**: `test/ship-queue-train-ops.test.ts` is excluded from the Windows subset like the other gh-shim drivers. +- docs/ship-policy.md's merge-train section documents every command above; docs/troubleshooting.md gains `QUEUE_SCREEN_RED`, `QUEUE_SCREEN_NOT_RUN`, `QUEUE_TRAIN_STACK_RED` and `QUEUE_POST_MERGE_RED`, and `QUEUE_SCREEN_STALE_REF` now means a real conflict only. + +### Tests +- `test/ship-queue-train-ops.test.ts` (new; the gh shim in `test/helpers/ship-queue-train-fixture.ts` gained `binding`, `outcomes`, `reuse`, `plan: null` and `ci` rules with the real artifact layout): a RED screen refuses ready under either choice, is printed by verify --screen with its red and flaky cases and refuses the pin in train build; a GREEN screen with flaky cases is followed and recorded; a NOT-RUN screen refuses verify with `QUEUE_SCREEN_NOT_RUN`, records `NOT_RUN` and is re-dispatched; a lagging or missing merge ref gets a scratch merge as `screen/-` and a real conflict stays `QUEUE_SCREEN_STALE_REF`; prune --stale settles only an all-merged-or-closed train and keeps one with an OPEN PR; train forget removes a train or a receipt and refuses a merging train; watch resumes from free-pending through the gate to the verdict and never dispatches twice; verify --post-merge waits, is red with the failed names, green when settled, resumable after a timeout; ready --dispatch-ci dispatches, waits, clears the matching lanes, refuses a red run and a fork head; --stack-tests catches a PR whose new test fails on T2 and passes after `--order` moves it first. Each was red before its change. +- `test/ship-queue-train.test.ts`: the stale-ref test now proves the scratch merge path; the planned-0-cases test expects the SCREEN: verdict lines. + +### Gate integrity +- No threshold, rubric, trial count, case kind or judge prompt changed. A screen's verdict is read as evals.yml bound it; nothing here changes what a case must pass. + ## [1.91.262.0] - 2026-10-27 **The weekly release accepts a commit a green train merged, and five lane frictions seen in operation are gone.** Trains merge faster than the main backstop settles, so most main commits ended with a neutral (`superseded`) `main-backstop` check run and the weekly cut returned `NO_GREEN_COMMIT` for a week of commits every one of which merged through a green train. The evidence already existed in `refs/notes/gstack-trains`; the gate now reads it. Alongside: `gstack-issue-links` no longer turns another repository's `(name #N, #M)` into this repository's Refs lines, `gstack-post` no longer asks for a confirmation on run ids written next to run words, `gstack-post pr-body` no longer wipes a PR body when a lane posts one section, and `gstack-ship-queue ready` finds the pregate verdict a lane recorded from a separate clone. diff --git a/VERSION b/VERSION index 89fd084b4..54b95e6c9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.262.0 +1.91.263.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 2b94af0ff..8fedef666 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.262.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.263.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index bc9fe7537..dca89bec9 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -286,6 +286,10 @@ gstack_result_anchor() { QUEUE_RECEIPT_MISSING) printf '%s' 'queue-receipt-missing' ;; QUEUE_RECEIPT_STALE) printf '%s' 'queue-receipt-stale' ;; QUEUE_SCREEN_STALE_REF) printf '%s' 'queue-screen-stale-ref' ;; + QUEUE_SCREEN_NOT_RUN) printf '%s' 'queue-screen-not-run' ;; + QUEUE_TRAIN_STACK_RED) printf '%s' 'queue-train-stack-red' ;; + QUEUE_POST_MERGE_RED) printf '%s' 'queue-post-merge-red' ;; + QUEUE_SCREEN_RED) printf '%s' 'queue-screen-red' ;; READY_SCREEN_CHOICE_REQUIRED) printf '%s' 'ready-screen-choice-required' ;; PREGATE_TIMING_ASSERTION) printf '%s' 'pregate-timing-assertion' ;; PREGATE_NO_POLICY) printf '%s' 'pregate-no-policy' ;; diff --git a/bin/gstack-ship-queue b/bin/gstack-ship-queue index a5f852d10..b130a7133 100755 --- a/bin/gstack-ship-queue +++ b/bin/gstack-ship-queue @@ -12,7 +12,9 @@ * gstack-ship-queue gate-ahead --after --pr [--predecessor-ref ] [--allow-repo-commands] * gstack-ship-queue gate-ahead --record --pr --attempt --exit --log * gstack-ship-queue gate-ahead --list --pr - * gstack-ship-queue train build|status|verdict|bisect|rebuild|prune ... (merge trains; train --help) + * gstack-ship-queue verify --pr --screen (train policy: the head's screen verdict from its run) + * gstack-ship-queue verify --post-merge [--timeout ] (after a merge: main's push workflows at that commit until they settle) + * gstack-ship-queue train build|status|verdict|bisect|rebuild|watch|prune ... (merge trains; train --help) * * Every subcommand takes --base (default origin/HEAD, else main) and * ends with GSTACK_RESULT: skill=ship-queue status=complete|refused run=. @@ -47,7 +49,10 @@ const HELP = `gstack-ship-queue — the shared merge queue's protocol steps (doc gate-ahead --record --pr --attempt --exit --log bind the gate result to that attempt's gated tree; removes its worktree gate-ahead --list --pr attempts for a PR, newest last (the current one) - train build|status|verdict|bisect|rebuild|prune ... + verify --pr --screen (train policy) the PR head's screen verdict read from its run (SCREEN_VERDICT) + verify --post-merge [--timeout ] wait for every check run at a merged commit to settle: POST_MERGE: green|red|pending + and one CHECK: line per workflow; resumable, default timeout 2700 s + train build|status|verdict|bisect|rebuild|watch|prune ... merge trains, when the base policy declares a train block (train --help; docs/ship-policy.md#run-a-merge-train); ready then takes --screen|--no-screen @@ -146,6 +151,7 @@ async function main(argv: string[]): Promise { if (!args[0] || args[0].startsWith('-')) { process.stdout.write(train.TRAIN_HELP + '\n'); return args.includes('--help') || args.includes('-h') ? EXIT.ok : EXIT.usage; } return train.trainMain(ctx, args); } + if (cmd === 'verify' && flag(args, '--post-merge') !== undefined) return (await import('../lib/ship-queue-train-cli')).postMergeMain(ctx, args); if (!pr || !/^\d+$/.test(pr)) return usage(`${cmd} needs --pr `); const gateExit = flag(args, '--gate-exit'); if (gateExit !== undefined && !/^-?\d+$/.test(gateExit)) return usage('--gate-exit needs an exit code'); @@ -168,7 +174,7 @@ async function main(argv: string[]): Promise { })); } if (cmd === 'verify') { - if (flag(args, '--train')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr); + if (flag(args, '--train') || args.includes('--screen')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr); const record = flag(args, '--record'); return finish('verify', verify(ctx, { pr, gated: flag(args, '--gated'), record: record ? path.resolve(record) : undefined, receipt: flag(args, '--receipt'), diff --git a/docs/ship-policy.md b/docs/ship-policy.md index 12a042f7e..789e8d65e 100644 --- a/docs/ship-policy.md +++ b/docs/ship-policy.md @@ -187,18 +187,25 @@ READY PR as conflicting each time main stamps; the train merges the CHANGELOG by section, so the lane never merges main for it. The conflict does mean GitHub has no merge ref for the PR, so no `pull_request` CI runs on the head and `ready` prints `ci: none at ` with a `ci-hint:` line. The lane then -dispatches the free workflows on the branch (`gh workflow run free-tests.yml ---ref && gh workflow run windows-free-tests.yml --ref `), -clears the requires-remote pregate lanes with those runs -(`gstack-pregate clear --remote --run-url `) and reruns `ready`, -which passes because the dispatched check runs attach to the commit; the -`ci-hint:` line then says those non-pull_request runs count. The `ci-hint:` line is -advisory text for the lane; nothing parses it, and a PR whose head is on a fork -gets a hint that leaves the dispatch to the coordinator. `ready --screen` on such a -PR refuses, because there is no merge ref to screen (`QUEUE_SCREEN_STALE_REF`, -` has no refs/pull//merge`); that refusal's "merge into the -branch" advice does not apply to a CHANGELOG-only conflict, so the lane READYs -with `--no-screen` after the skill's own paid cases ran focused. +runs `ready … --dispatch-ci free-tests.yml,windows-free-tests.yml [--timeout +]` (under a train policy): it dispatches each named workflow on the branch +(`gh workflow run --ref `, skipped when a run at this head already +concluded), waits for the runs (`CI_DISPATCH: --ref run +`), clears every requires-remote pregate lane whose workflow is one +of them with the run URL (`PREGATE_CLEARED: `; a lane of another +workflow stays uncleared and is named in the refusal), and then reads the head +as usual, which passes because the dispatched check runs attach to the commit; +the `ci-hint:` line then says those non-pull_request runs count. A red run +clears no lane and refuses (`QUEUE_NOT_READY`); a run still going at `--timeout` +(default 2700 s) is `QUEUE_TRAIN_PENDING`, and rerunning the command keeps +waiting; a fork head is never dispatched here (`QUEUE_TRAIN_UNTRUSTED`). By hand the same four steps are `gh workflow +run --ref `, wait, `gstack-pregate clear --remote --run-url +`, rerun `ready`. The `ci-hint:` line is advisory text for the lane; +nothing parses it. `ready --screen` on such a PR has no merge ref to screen, +so it builds the scratch merge described below (`SCREEN_MERGE:`, pushed as +`screen/-`) and screens that; only a real conflict with the base +(or a remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A CHANGELOG-only conflict is screenable, and a +lane whose skill's own paid cases ran focused may still READY with `--no-screen`. `ci:` counts the latest run per check (one app's check name), as GitHub's rollup does: a check that was re-run or superseded on the same commit (a @@ -393,7 +400,7 @@ explicit choice: ```bash gstack-ship-queue ready --pr 42 # prints SCREEN: lines and the recommendation, refuses gstack-ship-queue ready --pr 42 --no-screen # READY receipt; the PR rides unscreened -gstack-ship-queue ready --pr 42 --screen [--broad] # dispatches one paid screen on the PR's merge SHA (QUEUE_SCREEN_STALE_REF when GitHub's merge ref lags the head) +gstack-ship-queue ready --pr 42 --screen [--broad] # dispatches one paid screen on the PR's merge SHA (a scratch merge when GitHub's merge ref lags the head; QUEUE_SCREEN_STALE_REF only on a real conflict) ``` Without a choice it prints the `SCREEN:` selection (with @@ -410,10 +417,13 @@ the policy, so a repeat follows the same run and a broad screen never adopts a narrow one. The READY block gains `screen: …`, and the receipt is written locally and to the notes ref. `--dry-run` prints the dispatch without sending it. Before dispatching, `ready` fetches the merge ref and requires its second -parent to be the PR head: a missing or lagging merge ref (GitHub stops -recomputing it while the PR conflicts with the base, CHANGELOG-only conflicts -included) is `QUEUE_SCREEN_STALE_REF`, because a screen dispatched there plans -no cases and reports success. A finished screen that planned no cases (no +parent to be the PR head and its first parent the current base: a missing, +lagging or unfetchable merge ref (GitHub stops recomputing it while the PR +conflicts with the base, CHANGELOG-only conflicts included) is replaced by a +scratch merge of the head onto the base, pushed as `screen/-` +(the `SCREEN_MERGE:` line says which); a screen dispatched on a stale ref would +plan no cases and report success. Only a real conflict with the base (or a +remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A finished screen that planned no cases (no `paid-plan` artifact, which a superseded run never uploads, or a plan manifest with nothing planned) is never followed as a screen: `ready --screen` dispatches a fresh one and `train build` prints `SCREEN: # run planned @@ -547,9 +557,81 @@ It runs the normal free tests, full gate and verification on main plus the repair. When verify confirms the repair merged, it prints `FREEZE: cleared` and `NEXT:` builds a train of the PRs still unmerged. +**5b. Coordinator: after the merge.** `gstack-ship-queue verify --post-merge + [--timeout ]` (the merged commit's full 40-character SHA) watches +every check run attached to it (the push workflows: free tests, the Windows +subset, the main backstop) and every commit status until none is still +running, prints one `CHECK: ` per check and +`POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)`. +Green needs at least one completed check and no failure; `neutral` and +`skipped` (the backstop superseding itself) count as skipped; a pending +commit status keeps the commit pending and a failed one makes it red. A +failure is `QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. It keeps no process +state, so rerunning it after a machine restart continues the watch. + **6. Clean up.** `gstack-ship-queue train prune [--train ] [--dry-run]` deletes the `train/` branches of merged, superseded and dropped trains -(`WOULD-DELETE:`, `DELETED:`, `GONE:`). +(`WOULD-DELETE:`, `DELETED:`, `GONE:`). `train prune --stale` also settles a +`merging` or `green` train whose every PR is MERGED or CLOSED on GitHub (its +PRs landed through another train or by hand): it logs each PR's state on a +`STALE:` line, marks the record `stale` and deletes its branch; a train with an +OPEN PR is kept (`KEEP: … still needs the train`). The same run deletes every +`screen/-*` scratch branch whose PR is MERGED or CLOSED and keeps one whose +PR is still OPEN. `train forget --train | --receipt +@ [--dry-run]` removes one record (a settled train with its attempts, +or one READY receipt) from local state and from `refs/notes/gstack-trains` +with an exact lease on the notes ref, printing `FORGET:` then `FORGOTTEN:`; a +train that is still building, running or merging (`built`, `pushed`, +`free-pending`, `free-green`, `dispatch-unknown`, `dispatch-absent`, +`gate-pending`, `merging`) is refused; a rejected lease removes nothing and +restores the local notes ref. + +**Following a train without a daemon.** `gstack-ship-queue train watch --train + [--timeout ]` is the one resumable follow command: it reads the +record, runs the step its state asks for (`train build --train` to push, run +the free tests and dispatch the gate; `train status` to settle an unconfirmed +dispatch), waits for the pending run by polling gh, reads the verdict and +prints the last step's `NEXT:`. It never dispatches anything the record would +not dispatch on its own, so rerunning it after an interruption continues from +the record (`WATCH:` lines say what it did; a timeout, default 7200 s, is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`; `--free-timeout `, +default 2700, bounds the wait for the free tests). + +**Screens are read from the run.** `ready --screen` and `verify --pr +--screen` read a dispatched screen's verdict from the run's own artifacts +(`gate-binding-a`, `report-verdict-a`), never from a lane's +prose, and record it in the READY receipt (`screen.verdict`): `SCREEN_VERDICT: +# run GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one +`FLAKY:` line per rule-16 flaky case. `PENDING` is a run still going, +`UNKNOWN` one the queue cannot read, and `INCOMPLETE` a finished run that bound +no verdict (its report job did not finish): `ready --screen` dispatches afresh +instead of following it, `verify --screen` refuses it (`QUEUE_PROOF_MISSING`) +and `train build` seats the pin unscreened, as for `NOT-RUN`. A RED screen refuses `ready` under +either choice, is never re-dispatched at the same merge SHA and is not seated +by `train build`, whose `NEXT:` names the build without that PR +(`QUEUE_SCREEN_RED`); a run that concluded success with no slice (its route +job found the head superseded or the plan selected nothing) is `NOT-RUN`: +`verify --screen` refuses with `QUEUE_SCREEN_NOT_RUN`, `ready --screen` +dispatches afresh instead of following it, and `train build` seats the pin +unscreened with a `SCREEN: … counted as not-run` line. When GitHub's +`refs/pull//merge` lags the PR head or is missing, `ready --screen` plans +the screen on a scratch merge of the head onto the base, pushed as +`screen/-` (`SCREEN_MERGE:` names it; `prune --stale` deletes it +once the PR merged or closed); only a real conflict with the base (or a remote +whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. + +**Stacked tests before the gate.** `train build --prs … --dry-run --stack-tests +[--stack-tests-command ""]` runs each PR's new test files (added relative +to the base, with a name ending in `.test`, `_test`, `.spec` or `_spec` and a +JS or TS extension, bun's own rule, so a helper or fixture under `test/` is not +one) on the integrated prefix that includes it, T, in a scratch worktree, and prints +`STACK_TESTS: # T new test file(s) pass|FAIL ()` or `no new +test files`. A failure is `QUEUE_TRAIN_STACK_RED` (nothing written, pushed or +dispatched): the tests pass on the PR head alone only because nothing before +it in the train had changed what they read, so fix them against that prefix or +reorder with `--order`. The default command is `bun --no-env-file +--config=/dev/null test` with the files appended. **Escape hatches.** Remove the `train` block on the base branch to return every lane to the per-PR queue at once. `--urgent` ships one PR alone, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 289b5013a..acbbc22dc 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2643,7 +2643,9 @@ or corrupt object) that `git ls-tree` cannot confirm is absent; it never falls through to the user-owned file. It also refuses when Bun loaded a `.env` file from the working directory (a bare `bun gstack-pregate`), since a branch could then pick the user policy: run `gstack-pregate` directly or as -`bun --no-env-file`. +`bun --no-env-file`. `gstack-ship-queue ready` under a train policy fetches +`origin/` first and refuses with this code, quoting git's error, when +that fetch fails. **Fix.** `git fetch origin --depth=1`, then rerun. To run against the branch's own file deliberately, pass `--policy-from head --allow-repo-commands`; @@ -4051,7 +4053,9 @@ time; `gstack-ship-policy validate` prints `MIN_READER:` for exactly that. coordinator could misread: the working tree has uncommitted tracked changes, HEAD is not the PR's pushed head (read through `gh pr view`), CI at that exact head is not green (failed, pending or unreadable), or the pregate result -is missing, failed or recorded on another tree (`PREGATE_STALE`). +is missing, failed or recorded on another tree (`PREGATE_STALE`). Under +`--dispatch-ci` it is also a dispatched workflow run that concluded red (a red +run clears no pregate lane) or a `gh workflow run` that failed. **Fix.** Commit and push, wait for CI on that head, rerun `gstack-pregate run --allow-repo-commands` when the tree changed, then rerun @@ -4065,8 +4069,13 @@ which line is not yet true. input does not match: the gated head is not a commit here, HEAD differs from the gated head beyond stamps (the gate never covered this tree), an earlier stamp with other inputs is unfinished, the stamp journal's head moved, or a -gate-ahead merge could not be recomputed with `git merge-tree`. Nothing was -written unless the `QUEUE_STAMP_REFUSED:` line says `committed=yes`. +gate-ahead merge could not be recomputed with `git merge-tree`. The train +commands use it too: `train forget` on a record that does not exist or on a +train still building, running or merging, `train watch` on an unknown record, +`verify --pr --screen` on a run with no verdict to read (an INCOMPLETE or +unreadable run), and `ready --screen` when the scratch merge branch cannot be +pushed. Nothing was written unless the `QUEUE_STAMP_REFUSED:` line says +`committed=yes`. **Fix.** Supply the named input (fetch the gated head, pass the current base sha), or regate the tree you mean to stamp. For an unfinished stamp, rerun the @@ -4141,7 +4150,8 @@ a `train` block with `min_gstack_version` on the base branch first **Meaning.** A pinned PR's head repository is not the base repository. The train branch and its dispatches run with repository secrets, so fork work -never rides a train. +never rides a train. `ready --dispatch-ci` refuses a fork head for the same +reason and dispatches nothing. **Fix.** Run the printed `train build --prs `; the fork PR stays on the trusted-branch path. @@ -4205,7 +4215,11 @@ one-PR train. ### `QUEUE_TRAIN_PENDING` **Meaning.** The train's run (free tests, the full gate or the bisect) has -not finished, so there is no verdict yet. +not finished, so there is no verdict yet. The same code covers a `train watch` +that reached its timeout, `verify --pr --screen` on a screen run still +going, `verify --post-merge` on checks that have not settled, and a +`ready --dispatch-ci` workflow run still going at its `--timeout`; each prints +the command to rerun as `NEXT:`. **Fix.** Rerun the printed `NEXT:` command later. Nothing is dispatched again while a run is pending. @@ -4343,10 +4357,14 @@ to fix this: the stamp comes only when the slot is called. ### `QUEUE_SCREEN_STALE_REF` -**Meaning.** `gstack-ship-queue ready --screen` dispatches the screen against -GitHub's `refs/pull//merge`, and that ref is missing or still merges an -older head than the PR's pushed head. GitHub stops recomputing the merge ref -while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp +**Meaning.** `ready --screen` refuses with this code when the PR conflicts with +the base beyond the CHANGELOG and the stamp files (the message lists the +conflicting paths) or when it cannot list the remote's refs. A missing or +lagging merge ref alone no longer refuses (see Fix). The rest of this +paragraph is why such a ref cannot be screened directly: `ready --screen` +dispatches the screen against GitHub's `refs/pull//merge`, and that ref can +be missing or still merge an older head than the PR's pushed head. GitHub +stops recomputing the merge ref while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp counts), so a screen dispatched there checks out the old merge, sees itself superseded, plans 0 cases and reports success. GSTA-95 hit this on #3431: `ready` printed `SCREEN_DISPATCH:` as if a screen ran. `ready` now compares @@ -4362,16 +4380,113 @@ read (`… paid-plan artifact has expired`, `… manifest.json is unreadable`, a unreadable run or artifact list, an unreadable repository slug, or a READY whose run id is not a number) keeps the slot screened and prints `SCREEN: # ; counted as screened, unverified`; `ready --screen` -follows such a run. When `ready` cannot list the remote's refs or fetch the -merge ref at all, it refuses with the same code and a `could not list …` or -`could not fetch …` message whose NEXT line says to check the remote, not to -resolve a conflict. +follows such a run. When `ready` cannot list the remote's refs, it refuses with +the same code and a `could not list …` message whose NEXT line says to check +the remote, not to resolve a conflict; a merge ref that lists but cannot be +fetched is replaced by the scratch merge described under Fix. -**Fix.** Resolve the conflict with the base branch (or wait for GitHub to -recompute the merge ref), push, and run `gstack-ship-queue ready --pr ---screen` again. Under crew rule 14 a CHANGELOG-only conflict is left to the -train, so a lane that cannot get a fresh merge ref READYs `--no-screen` only -after its own focused paid cases passed (rule 6). +**Fix.** Since the scratch-merge screen landed, a lagging merge ref is no +longer a refusal: `ready --screen` builds the merge of the PR head onto +`origin/` itself with the train's merge rules (version files take the +base's side, the CHANGELOG merges by section), pushes it as +`screen/-` and dispatches the screen on that commit, printing +`SCREEN_MERGE: scratch merge …`. The code now means a real conflict: the +message lists the conflicting paths. Resolve those with the base branch, +push, and run `gstack-ship-queue ready --pr --screen` again. A +CHANGELOG-only or stamp-file-only conflict does not refuse, unless the PR's +CHANGELOG drops or renames a released heading (the message then names the +CHANGELOG). `train prune --stale` deletes the scratch branches of merged or +closed PRs. + + +### `QUEUE_SCREEN_RED` + +**Meaning.** The screen run dispatched for this PR head finished RED. The +verdict is read from the run itself, never from a lane's report: the +`gate-binding-a` artifact evals.yml's report job uploads (verdict, +reconcile exit, flaky cases) and the `report-verdict-a` artifact's +`collector-outcomes.json` (the panels that failed the lane and the rule-16 +flaky lines). A case the binding calls red already failed twice inside the +run (rule 16 reruns a first failure once), so the red is real. `ready` +refuses with this code when the head's recorded screen run is red, whichever +of `--screen` or `--no-screen` it is given, and never dispatches another +screen at the same merge SHA; `train build` refuses to seat the pin and its +`NEXT:` names the build without it; `verify --pr --screen` prints +`SCREEN_VERDICT: # run RED red= flaky=` and one +`FLAKY:` line per flaky case. The verdict is recorded in the READY receipt +(`screen.verdict`), locally and in `refs/notes/gstack-trains`. + +**Fix.** Read the red cases from the `SCREEN_VERDICT:` line, repair, push a +new head, run /ship and `gstack-ship-queue ready --pr --screen` again +(the new head gets its own screen). A GREEN screen with flaky cases is green: +the flaky cases are listed for the ledger and need no rerun. + + +### `QUEUE_TRAIN_STACK_RED` + +**Meaning.** `gstack-ship-queue train build --dry-run --stack-tests` ran a PR's +new test files (files the PR adds relative to the base whose name ends in +`.test`, `_test`, `.spec` or `_spec` plus a JS or TS extension, bun's own +rule; a helper or fixture under `test/` is not one) on the integrated prefix that includes it, +T (the base plus every slot up to and including that PR), in a scratch +worktree, and they failed. The `STACK_TESTS: # T new test file(s) +FAIL (): ` line names the files and the runner's +failing lines. Those tests pass on the PR head alone only because nothing +before it in the train had changed what they read (a PR stacked on another +PR's test files or fixtures hit this on 2026-10-26). Nothing was written, +pushed or dispatched; the ordinary dry run (without `--stack-tests`) does not +run them. + +**Fix.** Fix the PR's tests against that prefix, or reorder the train with +`--order` so the PR they depend on comes first, then rerun the dry run. Pass +`--stack-tests-command ""` when the repo's tests run under something +other than `bun test`; the files are appended to the command. + + +### `QUEUE_POST_MERGE_RED` + +**Meaning.** `gstack-ship-queue verify --post-merge ` (the full 40-character +SHA) watched the check runs attached to a commit the train merged onto the +base branch (the push workflows: free tests, the Windows subset, the main +backstop) and its commit statuses until none was still running, and at least +one check concluded `failure`, `timed_out`, `cancelled` or another non-success, +or a commit status failed (the message then says a commit status, not a check +run). A `neutral` or `skipped` check (the +backstop superseding itself when main moved on) is counted as skipped, never +red. The command keeps no process state: it polls gh until the checks settle +or `--timeout` (default 2700 s) passes, prints `POST_MERGE: +green|red|pending|none (passed= failed= pending= skipped=)` and one +`CHECK: ` line per check, and a timeout is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`, so rerunning it after +an interruption continues the watch. + +**Fix.** Read the red run named on its `CHECK:` line. Repair on a new PR or +roll back the merge; `gstack-ship-queue train status` names the freeze if the +backstop filed one. + + +### `QUEUE_SCREEN_NOT_RUN` + +**Meaning.** The screen run dispatched for this PR concluded `success` but +ran no slice. evals.yml's `route` job plans nothing when +`scripts/eval-readiness.ts wait` finds the PR head superseded (a push landed +between `ready --screen` and the plan) or when the plan selects no case; the +run then uploads no `paid-plan` artifact (or an empty manifest), every slice +job is skipped and GitHub shows a green check. On 2026-10-26, 18 of 41 PR +screens ended this way. The verdict is read from the run (`lib/ship-queue-screen.ts`), +so such a run is `NOT-RUN`: `verify --pr --screen` prints +`SCREEN_VERDICT: # run NOT-RUN ()` and refuses with this code, the +READY receipt records `screen.verdict.outcome: NOT_RUN`, `ready --screen` never +follows the run (its `SCREEN_DISPATCH:` line says `not following … a screen +that planned nothing is not a screen`) and `train build` seats the pin unscreened +with `SCREEN: # run planned no cases …; counted as not-run, the slot +rides unscreened`. + +**Fix.** `gstack-ship-queue ready --pr --screen` at the current head +dispatches a fresh screen. A head that moved since the dispatch needs a fresh +READY anyway (`QUEUE_TRAIN_MOVED`); a head that did not move and still plans +nothing has nothing diff-relevant to screen, and `--no-screen` with the +focused cases named in the READY block is the honest choice. ### `READY_SCREEN_CHOICE_REQUIRED` diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index d4551c814..5c25f026e 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -148,6 +148,26 @@ export const RELEASE_RESULT_CODES = { summary: 'GitHub’s refs/pull//merge is missing or still merges an older head than the PR’s: the PR conflicts with the base (a CHANGELOG-only conflict counts), so a screen dispatched on it would plan no cases and report success', fix: 'resolve the conflict with the base branch (or wait for GitHub to recompute the merge ref), push, then gstack-ship-queue ready --pr --screen again; a lane never READYs on a screen that planned nothing', }, + QUEUE_SCREEN_NOT_RUN: { + anchor: 'queue-screen-not-run', + summary: 'the PR’s dispatched screen run concluded success but ran no slice (the run planned no cases: its route job found the PR head superseded by a push, or the plan selected nothing), so the head is not screened and the run is counted as not-run, never as green', + fix: 'gstack-ship-queue ready --pr --screen dispatches a fresh screen at the current head (a not-run screen is never followed); train build seats the pin unscreened and says so on its SCREEN: line', + }, + QUEUE_TRAIN_STACK_RED: { + anchor: 'queue-train-stack-red', + summary: 'train build --dry-run --stack-tests ran a PR’s new test files on the integrated prefix that includes it (T: main plus every slot up to that PR) and they failed; the tests pass on the PR head alone only because nothing before it in the train had changed what they read', + fix: 'fix the PR’s tests against that prefix (read the STACK_TESTS: line for the files and the failing assertion), or reorder the train with --order so the PR they depend on comes first, then rerun the dry run; nothing was written, pushed or dispatched', + }, + QUEUE_POST_MERGE_RED: { + anchor: 'queue-post-merge-red', + summary: 'a push workflow on the merged commit (free tests, the Windows subset, the main backstop) concluded failure, timed out or was cancelled after the train merged it; a neutral or skipped check is not red', + fix: 'read the red run (gstack-ship-queue verify --post-merge prints one CHECK: line per workflow), repair on a new PR or roll back the merge; gstack-ship-queue train status names the freeze if the backstop filed one', + }, + QUEUE_SCREEN_RED: { + anchor: 'queue-screen-red', + summary: 'the PR head’s dispatched screen run finished RED (its gate binding says so; a case that failed twice is real), so the head is not READY and a train will not seat it', + fix: 'read the SCREEN_VERDICT line (gstack-ship-queue verify --pr --screen prints the red and flaky cases), repair, push a new head, run /ship and ready again; a red screen is never re-dispatched at the same merge SHA', + }, READY_SCREEN_CHOICE_REQUIRED: { anchor: 'ready-screen-choice-required', summary: 'the base policy turns trains on, so READY needs an explicit screen choice: --screen (pay for a diff-relevant screen now) or --no-screen', diff --git a/lib/ship-queue-dispatch-ci.ts b/lib/ship-queue-dispatch-ci.ts new file mode 100644 index 000000000..0c496ab8c --- /dev/null +++ b/lib/ship-queue-dispatch-ci.ts @@ -0,0 +1,80 @@ +/** + * `ready --dispatch-ci [--timeout ]`: a PR that + * conflicts with the base (a CHANGELOG-only conflict after every stamp + * counts) has no merge ref, so GitHub runs no pull_request CI on its head and + * `ready` prints `ci: none`. The lane used to dispatch the free workflows by + * hand, wait, clear the requires-remote pregate lanes with the run URLs and + * rerun ready. This does those four steps before `ready` reads the head: one + * `gh workflow run --ref ` per named workflow (skipped when a + * run at this head already concluded), a wait for each run, and one + * `PREGATE_CLEARED:` per requires-remote lane whose workflow is one of them + * and whose run concluded success. A red run clears no lane and refuses. A + * fork head is never dispatched here (QUEUE_TRAIN_UNTRUSTED). + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import type { PregateFile } from './headless-artifacts'; +import { PREGATE_OUT_REL, clearObligation, readPregateFile, writePregateFile } from './pregate/index'; +import type { QueueContext } from './ship-queue'; +import type { QueueFailure } from './ship-queue-gate'; +import { prInfo, repoSlug, runView } from './ship-queue-train-github'; + +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface DispatchCiOptions { pr: string; workflows: string[]; pregate?: string; timeoutSec: number; pollMs: number } +export type DispatchCiResult = { lines: string[] } | { lines: string[]; failure: QueueFailure }; + +interface RunRow { databaseId: number; url: string; status: string; conclusion: string | null; headSha: string } + +function runsAt(ctx: QueueContext, workflow: string, branch: string, head: string): RunRow[] { + const r = ctx.gh(['run', 'list', '--workflow', workflow, '--branch', branch, '--event', 'workflow_dispatch', '--limit', '20', '--json', 'databaseId,url,status,conclusion,headSha']); + try { return r.ok ? (JSON.parse(r.out) as RunRow[]).filter(x => x.headSha === head) : []; } catch { return []; } +} + +export async function dispatchCi(ctx: QueueContext, o: DispatchCiOptions): Promise { + const info = prInfo(ctx, o.pr); + if ('error' in info) return { lines: [], failure: { code: 'QUEUE_NOT_READY', message: info.error } }; + if (!info.headRefName) return { lines: [], failure: { code: 'QUEUE_NOT_READY', message: `cannot read PR #${o.pr}'s branch through gh (no headRefName)` } }; + if (info.isCrossRepository) return { lines: [], failure: { code: 'QUEUE_TRAIN_UNTRUSTED', message: `#${o.pr}'s head is on a fork; its workflows are not dispatched here (the coordinator decides)` } }; + const head = info.headRefOid; + const branch = info.headRefName; + const slug = repoSlug(ctx); + const deadline = Date.now() + o.timeoutSec * 1000; + const lines: string[] = []; + const concluded = new Map(); + for (const wf of o.workflows) { + const atHead = runsAt(ctx, wf, branch, head); + const prior = atHead.find(r => r.status === 'completed'); + if (prior) { lines.push(`CI_DISPATCH: ${wf} not dispatched (run ${prior.databaseId} already concluded ${prior.conclusion ?? 'unknown'} at this head)`); concluded.set(wf, prior); continue; } + const live = atHead[0]; + if (!live) { + const d = ctx.gh(['workflow', 'run', wf, '--ref', branch]); + if (!d.ok) return { lines, failure: { code: 'QUEUE_NOT_READY', message: `gh workflow run ${wf} --ref ${branch} failed (${(d.err ?? d.out).trim().replace(/\s+/g, ' ').slice(0, 120) || 'no output'})` } }; + } + let run: RunRow | undefined = live; + for (;;) { + run = run && slug ? runView(ctx, slug, run.databaseId) ?? run : runsAt(ctx, wf, branch, head)[0]; + if (run?.status === 'completed') break; + if (Date.now() >= deadline) return { lines: [...lines, `CI_DISPATCH: ${wf} --ref ${branch} ${run ? `run ${run.databaseId} ${run.url} ${run.status}` : 'no run appeared'}`], failure: { code: 'QUEUE_TRAIN_PENDING', message: `${wf} ${run ? `run ${run.databaseId} is still ${run.status}` : 'has no workflow_dispatch run at this head yet'} at the --dispatch-ci timeout; rerun ready --dispatch-ci to continue waiting` } }; + await sleep(o.pollMs); + } + lines.push(`CI_DISPATCH: ${wf} --ref ${branch} run ${run.databaseId} ${run.url} ${run.conclusion ?? 'unknown'}`); + concluded.set(wf, run); + } + const red = [...concluded.entries()].filter(([, r]) => r.conclusion !== 'success'); + if (red.length) return { lines, failure: { code: 'QUEUE_NOT_READY', message: red.map(([wf, r]) => `${wf} run ${r.databaseId} concluded ${r.conclusion ?? 'unknown'} at ${head.slice(0, 12)}`).join('; ') + '; a red run clears no lane' } }; + const pregateFile = path.resolve(ctx.repoRoot, o.pregate ?? PREGATE_OUT_REL); + if (!fs.existsSync(pregateFile)) return { lines }; + const read = readPregateFile(pregateFile); + if (!read.file) return { lines }; + let file: PregateFile = read.file; + let changed = false; + for (const lane of (file.requires_remote ?? []).filter(l => !l.receipt && concluded.has(l.workflow))) { + const run = concluded.get(lane.workflow)!; + file = clearObligation(file, lane.lane, run.url, `gstack-ship-queue ready --dispatch-ci ${new Date().toISOString()}`); + lines.push(`PREGATE_CLEARED: ${lane.lane} ${run.url}`); + changed = true; + } + if (changed) writePregateFile(ctx.repoRoot, pregateFile, file); + return { lines }; +} diff --git a/lib/ship-queue-post-merge.ts b/lib/ship-queue-post-merge.ts new file mode 100644 index 000000000..05a5fc56a --- /dev/null +++ b/lib/ship-queue-post-merge.ts @@ -0,0 +1,45 @@ +/** + * `verify --post-merge [--timeout ]`: the coordinator's follow + * command after a train merged a PR onto the base branch. The push workflows + * run on the merged commit (free tests, the Windows subset, the main backstop + * that supersedes itself when main moves on), and a background `gh run + * watch` chain dies with the machine that started it. This command keeps no + * state: it reads every check run attached to the sha, waits (polling gh) + * until none is still running or the timeout passes, prints one `CHECK:` + * line per check, and a `POST_MERGE:` summary that is green only when at + * least one check completed and none failed; a neutral or skipped check is + * skipped, never red. Commit statuses count too: a pending status keeps the + * commit pending, a failed one makes it red (named as a commit status when no + * check run failed). Rerun it after an interruption and it continues. + */ +import { ciAtHead, type QueueContext } from './ship-queue'; +import { outFailed, outOk, outPending, type TrainOut } from './ship-queue-train'; + +const Q = 'gstack-ship-queue'; +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface PostMergeOptions { sha: string; timeoutSec: number; pollMs: number } + +export async function verifyPostMerge(ctx: QueueContext, o: PostMergeOptions): Promise { + const deadline = Date.now() + o.timeoutSec * 1000; + const at = o.sha.slice(0, 12); + const again = `${Q} verify --post-merge ${o.sha} --timeout ${o.timeoutSec}`; + for (;;) { + const ci = ciAtHead(ctx, o.sha); + const lines = [`POST_MERGE: ${ci.line.replace(/^(\S+) at ([0-9a-f]+)/, '$2 $1')}`, ...ci.checks.map(c => `CHECK: ${c.name} ${c.status === 'completed' ? c.conclusion ?? 'unknown' : c.status}`)]; + const settled = !ci.none && !/^(pending|unknown) /.test(ci.line) && !ci.checks.some(c => c.status !== 'completed'); + if (settled && ci.green) return outOk(lines, 'none', 'none'); + if (settled) { + const red = ci.checks.filter(c => c.status === 'completed' && !['success', 'skipped', 'neutral'].includes(c.conclusion ?? '')).map(c => c.name); + return outFailed({ code: 'QUEUE_POST_MERGE_RED', message: `main's push workflows at ${at} are red: ${red.join(', ') || 'a commit status (no check run failed)'}` }, lines, + `read the red run, repair on a new PR or roll back; ${Q} train status names the freeze if the backstop filed one`, 'none'); + } + if (Date.now() >= deadline) { + const pending = ci.checks.filter(c => c.status !== 'completed').map(c => c.name); + const message = ci.line.startsWith('unknown') ? `the check runs at ${at} are unreadable (${ci.line.replace(/^unknown at [0-9a-f]+ /, '')})` + : ci.none ? `no check run has attached to ${at} yet` : `main's push workflows at ${at} have not settled at the timeout: ${pending.join(', ') || 'a commit status'} still running`; + return outPending({ code: 'QUEUE_TRAIN_PENDING', message }, lines, again, 'none'); + } + await sleep(o.pollMs); + } +} diff --git a/lib/ship-queue-screen.ts b/lib/ship-queue-screen.ts new file mode 100644 index 000000000..febce6a21 --- /dev/null +++ b/lib/ship-queue-screen.ts @@ -0,0 +1,114 @@ +/** + * A dispatched screen's verdict, read from the run itself: + * never from a lane's prose. evals.yml's report job publishes + * `gate-binding-a` (scripts/gate-reuse.ts: verdict, reconcile exit, + * flaky cases) and `report-verdict-a` (collector-outcomes.json: the + * panels and the rule-16 flaky lines) on every executed screen; a superseded + * screen plans nothing and uploads neither (hollowScreen): it concluded + * success with no slice, so its outcome is `not-run`, never green. The outcome is + * recorded in the READY receipt (ScreenChoice.verdict) by `ready`, + * `verify --screen` and `train build`, so the controller reads one line. + */ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { REUSE_FILE, parseBinding, parseReuse } from '../scripts/gate-reuse'; +import type { QueueContext } from './ship-queue'; +import { downloadJson, hollowScreen, runView } from './ship-queue-train-github'; +import { nowIso, type ScreenChoice, type ScreenVerdictRecord } from './ship-queue-train-record'; + +export type ScreenOutcome = + | { kind: 'pending'; detail: string } + | { kind: 'not-run'; detail: string } + | { kind: 'unknown'; detail: string } + | { kind: 'incomplete'; detail: string; flaky: string[] } + | { kind: 'green'; detail: string; flaky: string[]; flakyLines: string[] } + | { kind: 'red'; detail: string; red: string[]; flaky: string[]; flakyLines: string[] }; + +const BINDING = 'gate-binding'; +const REPORT = 'report-verdict'; +/** The plan artifact; a confirmed verdict reuse (R2a/R2) leaves its record there and binds nothing new. */ +const PLAN = 'paid-plan'; + +/** The newest attempt's artifact of a name family (`-a`); null when the run uploaded none or the list is unreadable. */ +function newestArtifact(ctx: QueueContext, slug: string, run: number, name: string): string | null | 'unreadable' { + const artifacts: Array<{ name?: string; expired?: boolean }> = []; + for (let page = 1; page <= 20; page++) { + const r = ctx.gh(['api', `repos/${slug}/actions/runs/${run}/artifacts?per_page=100&page=${page}`]); + let list: { artifacts?: Array<{ name?: string; expired?: boolean }> } | null = null; + try { list = r.ok ? JSON.parse(r.out) : null; } catch {} + if (!Array.isArray(list?.artifacts)) return 'unreadable'; + artifacts.push(...list.artifacts); + if (list.artifacts.length < 100) break; + } + const hits = artifacts.map(a => ({ name: a.name ?? '', expired: !!a.expired, attempt: Number(new RegExp(`^${name}-a(\\d+)$`).exec(a.name ?? '')?.[1]) })) + .filter(a => Number.isSafeInteger(a.attempt)).sort((a, b) => b.attempt - a.attempt); + if (!hits.length) return null; + return hits[0]!.expired ? 'unreadable' : hits[0]!.name; +} + +interface Outcomes { panels?: Array<{ case?: string; status?: string; failsLane?: boolean; redClass?: string | null }>; flakyLines?: string[] } + +/** Read one screen run: status, plan, gate binding, panels. */ +export function screenVerdict(ctx: QueueContext, slug: string, run: number): ScreenOutcome { + const view = runView(ctx, slug, run); + if (!view) return { kind: 'unknown', detail: `run ${run} is unreadable` }; + if (view.status !== 'completed') return { kind: 'pending', detail: `run ${run} is ${view.status}` }; + const plan = hollowScreen(ctx, slug, run); + if (plan.kind === 'hollow') return { kind: 'not-run', detail: plan.reason }; + if (plan.kind === 'unknown') return { kind: 'unknown', detail: plan.reason }; + const bindingName = newestArtifact(ctx, slug, run, BINDING); + if (bindingName === 'unreadable') return { kind: 'unknown', detail: `run ${run}'s artifact list is unreadable or its ${BINDING} artifact has expired` }; + const scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-screen-verdict-')); + let binding: ReturnType = null; + let outcomes: Outcomes | null = null; + try { + if (!bindingName) { + const reuse = parseReuse(downloadJson(ctx, slug, run, PLAN, scratch, REUSE_FILE)); + if (reuse && view.conclusion === 'success') { + const flaky = [...(reuse.reused_flaky ?? [])].sort(); + return { kind: 'green', detail: `reused the GREEN verdict of run ${reuse.reused_run_id} (${reuse.kind}): ${reuse.why}${flaky.length ? `, flaky ${flaky.length}` : ''}`, flaky, flakyLines: [] }; + } + return { kind: 'incomplete', detail: `run ${run} concluded ${view.conclusion ?? 'unknown'} and bound no verdict (no ${BINDING} artifact: the report job did not finish)`, flaky: [] }; + } + binding = parseBinding(downloadJson(ctx, slug, run, bindingName, scratch, 'gate-binding.json')); + const reportName = newestArtifact(ctx, slug, run, REPORT); + outcomes = typeof reportName === 'string' ? downloadJson(ctx, slug, run, reportName, scratch, 'collector-outcomes.json') as Outcomes | null : null; + } finally { fs.rmSync(scratch, { recursive: true, force: true }); } + if (!binding) return { kind: 'unknown', detail: `run ${run}'s ${bindingName} artifact is unreadable` }; + const flaky = [...(binding.flaky_cases ?? [])].sort(); + const flakyLines = Array.isArray(outcomes?.flakyLines) ? outcomes!.flakyLines!.filter((l): l is string => typeof l === 'string') : []; + const red = (outcomes?.panels ?? []).filter(p => typeof p.case === 'string' && (p.failsLane === true || (p.status === 'FAIL' && p.redClass === 'VERDICT'))).map(p => p.case!).sort(); + const base = `verdict ${binding.verdict}, reconcile exit ${binding.reconcile_exit}, run concluded ${view.conclusion ?? 'unknown'}${flaky.length ? `, flaky ${flaky.length}` : ''}`; + if (binding.verdict === 'GREEN' && binding.reconcile_exit === 0 && view.conclusion === 'success') return { kind: 'green', detail: base, flaky, flakyLines }; + if (binding.verdict === 'RED' || binding.reconcile_exit !== 0 || red.length) return { kind: 'red', detail: base, red, flaky, flakyLines }; + return { kind: 'incomplete', detail: base, flaky }; +} + +/** The receipt field for an outcome that is settled (a pending or unknown run records nothing). */ +export function verdictRecord(o: ScreenOutcome, run: number): ScreenVerdictRecord | null { + if (o.kind === 'pending' || o.kind === 'unknown') return null; + const outcome = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind === 'not-run' ? 'NOT_RUN' : 'INCOMPLETE'; + return { outcome, run, red: o.kind === 'red' ? o.red : [], flaky: 'flaky' in o ? o.flaky : [], detail: o.detail, read_at: nowIso() }; +} + +/** `SCREEN_VERDICT:` and the per-case `FLAKY:` lines for one PR's screen. */ +export function verdictLines(pr: string, run: number, o: ScreenOutcome): string[] { + const word = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind.toUpperCase(); + const red = o.kind === 'red' && o.red.length ? ` red=${o.red.join(',')}` : ''; + const flaky = 'flaky' in o && o.flaky.length ? ` flaky=${o.flaky.join(',')}` : ''; + const lines = [`SCREEN_VERDICT: #${pr} run ${run} ${word}${red}${flaky} (${o.detail})`]; + if ('flakyLines' in o) for (const l of o.flakyLines) lines.push(`FLAKY: ${l}`); + return lines; +} + +/** Apply a read outcome to a receipt's screen choice; returns whether the receipt changed. */ +export function recordVerdict(screen: ScreenChoice, o: ScreenOutcome): boolean { + if (!screen.run) return false; + const rec = verdictRecord(o, screen.run); + if (!rec) return false; + const same = screen.verdict && screen.verdict.outcome === rec.outcome && screen.verdict.run === rec.run && screen.verdict.red.join() === rec.red.join() && screen.verdict.flaky.join() === rec.flaky.join(); + if (same) return false; + screen.verdict = rec; + return true; +} diff --git a/lib/ship-queue-stack-tests.ts b/lib/ship-queue-stack-tests.ts new file mode 100644 index 000000000..514ba01f6 --- /dev/null +++ b/lib/ship-queue-stack-tests.ts @@ -0,0 +1,63 @@ +/** + * `train build --dry-run --stack-tests`: before a train spends a gate, run + * each PR's new test files on the integrated prefix that includes it (T, + * the merge of main and every slot up to and including the PR), not on the + * PR head alone. A test that passes on its branch and fails once an earlier + * slot changed what it reads is a stack red, caught here for the price of a + * few test files instead of a gate. Each prefix is checked out in a scratch + * worktree (the repo's node_modules linked in when it has one), the files run + * with the test command (`bun --no-env-file --config=/dev/null test` by + * default; `--stack-tests-command ""` for another runner, the files + * appended), and the worktree is removed. A PR that adds no test file is + * reported as such and never fails. + */ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { bunIsolationArgs } from './bun-isolation'; +import { git } from './ship-queue-gate'; +import type { TrainPrefix, TrainSlot } from './ship-queue-train-record'; + +export const DEFAULT_STACK_TEST_COMMAND = ['bun', ...bunIsolationArgs(), 'test']; +/** Bun's own test-file rule: the name ends in `.test`, `_test`, `.spec` or `_spec` before its extension; a helper or fixture under test/ is not a test file and would make `bun test ` exit 1 for matching nothing. */ +const TEST_FILE = /[._](?:test|spec)\.[cm]?[jt]sx?$/; + +export interface StackTestResult { pr: string; prefix: number; files: string[]; ok: boolean; detail: string } + +/** Test files the PR adds relative to the base (added, not modified). */ +export function newTestFiles(repoRoot: string, baseSha: string, head: string): string[] { + const r = git(repoRoot, ['diff', '--name-only', '--diff-filter=A', `${baseSha}...${head}`]); + return r.ok ? r.out.split('\n').map(s => s.trim()).filter(f => f && TEST_FILE.test(f)) : []; +} + +export function stackTests(repoRoot: string, baseSha: string, slots: TrainSlot[], prefixes: TrainPrefix[], command: string[] = DEFAULT_STACK_TEST_COMMAND, timeoutMs = 600_000): StackTestResult[] { + const results: StackTestResult[] = []; + for (const [i, slot] of slots.entries()) { + const prefix = prefixes[i + 1]; + if (!prefix) break; + const files = newTestFiles(repoRoot, baseSha, slot.head); + if (!files.length) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: true, detail: 'no new test files' }); continue; } + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-stack-tests-')); + try { + const add = git(repoRoot, ['worktree', 'add', '--detach', '-q', dir, prefix.commit]); + if (!add.ok) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: false, detail: `cannot check out T${prefix.index}: ${add.err.slice(0, 160)}` }); continue; } + const modules = path.join(repoRoot, 'node_modules'); + if (fs.existsSync(modules) && !fs.existsSync(path.join(dir, 'node_modules'))) fs.symlinkSync(modules, path.join(dir, 'node_modules'), 'dir'); + const [bin, ...args] = command; + const run = spawnSync(bin!, [...args, ...files.map(f => `./${f}`)], { cwd: dir, encoding: 'utf8', timeout: timeoutMs, env: { ...process.env, CI: process.env.CI ?? '1' }, maxBuffer: 64 * 1024 * 1024 }); + const tail = `${run.stdout ?? ''}\n${run.stderr ?? ''}`.split('\n').filter(l => /^\(fail\)|^\s*\d+ (pass|fail)$|^error:/.test(l)).slice(-6).join('; '); + results.push({ pr: slot.pr, prefix: prefix.index, files, ok: run.status === 0, detail: run.status === 0 ? 'pass' : run.error ? `${run.error.message}` : tail || `exit ${run.status}` }); + } finally { + git(repoRoot, ['worktree', 'remove', '--force', dir]); + fs.rmSync(dir, { recursive: true, force: true }); + } + } + return results; +} + +/** One `STACK_TESTS:` line per slot. */ +export function stackTestLine(r: StackTestResult): string { + if (!r.files.length) return `STACK_TESTS: #${r.pr} T${r.prefix} no new test files`; + return `STACK_TESTS: #${r.pr} T${r.prefix} ${r.files.length} new test file(s) ${r.ok ? 'pass' : 'FAIL'} (${r.files.join(', ')})${r.ok ? '' : `: ${r.detail}`}`; +} diff --git a/lib/ship-queue-train-cli.ts b/lib/ship-queue-train-cli.ts index cc41f5988..01fb58477 100644 --- a/lib/ship-queue-train-cli.ts +++ b/lib/ship-queue-train-cli.ts @@ -10,12 +10,15 @@ import { EXIT, formatResult } from './headless-artifacts'; import { RESULT_CODES } from './result-codes'; import type { QueueContext } from './ship-queue'; import { buildTrain, pruneTrains, trainStatus, type TrainOut } from './ship-queue-train'; -import { trainReady } from './ship-queue-train-ready'; +import { trainReady, verifyScreen } from './ship-queue-train-ready'; import { trainBisect, trainRebuild, trainVerdict } from './ship-queue-train-verdict'; import { verifyTrain } from './ship-queue-train-verify'; +import { trainWatch } from './ship-queue-train-watch'; +import { verifyPostMerge } from './ship-queue-post-merge'; +import { trainForget } from './ship-queue-train-forget'; import { redactCause } from './ship-queue-train-record'; -export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] [--urgent | --repair] [--dry-run] [--free-timeout ] +export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] [--urgent | --repair] [--dry-run [--stack-tests [--stack-tests-command ""]]] [--free-timeout ] --order: explicit slot order, every PR in --prs once (default: screened first, then READY order) train build --train resume a train (push, free tests, dispatch) from its record @@ -27,12 +30,27 @@ export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] train bisect --train --spend-cap [--dry-run] red cases on every prefix T0..Tk, resumable and capped train rebuild --train [--infra] the one rebuild an all-green bisect earns (or the one INFRA re-dispatch) - train prune [--train ] [--dry-run] - delete train/ branches of merged, superseded or dropped trains - ready --pr --screen [--broad] | --no-screen [--dry-run] - (train policy) READY plus the screen choice, recorded as a receipt + train watch --train [--timeout ] + resumable follow: runs the step the record asks for (build, status), waits for the + pending run, reads the verdict, prints NEXT; rerun it after a restart (no daemon) + train forget --train | --receipt @ [--dry-run] + remove a settled train record (attempts, bisect) or a READY receipt from + refs/notes/gstack-trains under an exact lease, then delete the local files + train prune [--train ] [--dry-run] [--stale] + delete train/ branches of merged, superseded or dropped trains; --stale also + settles merging or green trains whose every PR is merged or closed on GitHub, and + deletes screen/-* branches of merged or closed PRs + ready --pr --screen [--broad] | --no-screen [--dry-run] [--dispatch-ci [--timeout ]] + (train policy) READY plus the screen choice, recorded as a receipt; a head whose + screen run is RED is refused (QUEUE_SCREEN_RED). --dispatch-ci runs the named + workflows on the branch first (a conflicting PR has no pull_request CI), waits for + them and clears the matching requires-remote pregate lanes with the run URLs verify --pr --train [--no-fetch] (train policy) the merge check against a green train record + verify --pr --screen (train policy) the PR head's screen verdict read from its run: SCREEN_VERDICT and + FLAKY lines, recorded in the READY receipt + verify --post-merge [--timeout ] after a merge: wait for every check run at the merged commit to settle; + POST_MERGE: green|red|pending plus one CHECK: line per workflow (resumable) Merge trains need a "train" block in the base revision's ship policy (docs/ship-policy.md#run-a-merge-train).`; function flag(args: string[], name: string): string | undefined { @@ -80,14 +98,30 @@ export async function trainMain(ctx: QueueContext, args: string[]): Promise,... orders a new train from --prs; a resumed train keeps its recorded order'); + const stackCommand = flag(rest, '--stack-tests-command'); + if ((rest.includes('--stack-tests') || stackCommand !== undefined) && !dryRun) return usage('--stack-tests runs each PR\'s new test files on its integrated prefix; it needs --dry-run'); + const stackWords = stackCommand?.split(/\s+/).filter(Boolean); + if (stackCommand !== undefined && !stackWords?.length) return usage('--stack-tests-command needs a command'); return guarded('train build', () => buildTrain(ctx, { pins: flag(rest, '--prs'), order: flag(rest, '--order'), resume: nonce, dryRun, urgent: rest.includes('--urgent'), repair: rest.includes('--repair'), freeTimeoutSec, - pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, + pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, stackTests: stackWords?.length ? stackWords : rest.includes('--stack-tests') ? true : undefined, })); } if (sub === 'status') return guarded('train status', () => trainStatus(ctx, { train: nonce })); - if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce })); + if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce, stale: rest.includes('--stale') })); + if (sub === 'forget') { + const receipt = flag(rest, '--receipt'); + if (!nonce && !receipt) return usage('train forget needs --train or --receipt @'); + return guarded('train forget', () => trainForget(ctx, { train: nonce, receipt, dryRun })); + } if (!nonce) return usage(`train ${sub ?? ''} needs --train `); + if (sub === 'watch') { + const timeoutSec = positive(flag(rest, '--timeout'), 7200); + if (Number.isNaN(timeoutSec)) return usage('--timeout needs seconds'); + const freeTimeoutSec = positive(flag(rest, '--free-timeout'), 2700); + if (Number.isNaN(freeTimeoutSec)) return usage('--free-timeout needs seconds'); + return guarded('train watch', () => trainWatch(ctx, { train: nonce, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, freeTimeoutSec })); + } if (sub === 'verdict') return guarded('train verdict', () => trainVerdict(ctx, { train: nonce })); if (sub === 'rebuild') return guarded('train rebuild', () => trainRebuild(ctx, { train: nonce, infra: rest.includes('--infra') })); if (sub === 'bisect') { @@ -103,9 +137,24 @@ export async function readyMain(ctx: QueueContext, args: string[], base: { pr: s const noScreen = args.includes('--no-screen'); if (screen && noScreen) return usage('pass one of --screen or --no-screen'); if (args.includes('--broad') && !screen) return usage('--broad selects the screen policy; it needs --screen'); - return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run') })); + const dispatchCi = flag(args, '--dispatch-ci'); + const workflows = dispatchCi === undefined ? [] : dispatchCi.split(',').map(s => s.trim()).filter(Boolean); + if (dispatchCi !== undefined && (!workflows.length || workflows.some(w => !/^[\w.-]+\.ya?ml$/.test(w)))) return usage('--dispatch-ci names workflow files under .github/workflows, comma-separated (free-tests.yml,windows-free-tests.yml)'); + const timeoutSec = positive(flag(args, '--timeout'), 2700); + if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds'); + return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run'), + dispatchCi: workflows.length ? { workflows, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 } : undefined })); +} + +export async function postMergeMain(ctx: QueueContext, args: string[]): Promise { + const sha = flag(args, '--post-merge'); + if (!sha || !/^[0-9a-f]{40}$/.test(sha)) return usage('verify --post-merge needs the merged commit\'s full 40-hex SHA'); + const timeoutSec = positive(flag(args, '--timeout'), 2700); + if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds'); + return guarded('verify', () => verifyPostMerge(ctx, { sha, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 })); } export async function verifyMain(ctx: QueueContext, args: string[], pr: string): Promise { + if (args.includes('--screen')) return guarded('verify', () => verifyScreen(ctx, { pr })); return guarded('verify', () => verifyTrain(ctx, { pr, train: flag(args, '--train')!, noFetch: args.includes('--no-fetch') })); } diff --git a/lib/ship-queue-train-forget.ts b/lib/ship-queue-train-forget.ts new file mode 100644 index 000000000..99895e6b9 --- /dev/null +++ b/lib/ship-queue-train-forget.ts @@ -0,0 +1,56 @@ +/** + * `train forget --train | --receipt @ [--dry-run]`: remove a + * train record (its attempts and bisect ledger) or a READY receipt from + * refs/notes/gstack-trains under an exact lease, then delete the local files + * (until now a stale record needed a hand-run + * `git notes remove` plus a file hunt on every machine that ran the train). + * A rejected lease removes nothing (the local notes ref is restored too, so a + * later write cannot carry the removal without a lease). Other machines that + * still hold the local files can push the record back with their next write; + * the output says so. + */ +import * as fs from 'node:fs'; +import type { QueueContext } from './ship-queue'; +import { outOk, outRefused, trainEnv, type TrainOut } from './ship-queue-train'; +import { attemptsOf, forgetNotes, loadBisect, loadReady, loadTrain, noteName, readyPath, rebuildFromNotes, trainDir, type AnyRecord, type TrainState } from './ship-queue-train-record'; + +const Q = 'gstack-ship-queue'; +/** States with work still in flight (a branch being built or pushed, free tests or a gate running, a merge under way): forget never settles them. */ +const LIVE_STATES: readonly TrainState[] = ['built', 'pushed', 'free-pending', 'free-green', 'dispatch-unknown', 'dispatch-absent', 'gate-pending', 'merging']; + +export function trainForget(ctx: QueueContext, o: { train?: string; receipt?: string; dryRun: boolean }): TrainOut { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + rebuildFromNotes(ctx.repoRoot, env.remote); + const records: AnyRecord[] = []; + const files: string[] = []; + let what = ''; + if (o.train) { + const t = loadTrain(o.train); + if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + if (LIVE_STATES.includes(t.state)) { + return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `train ${t.nonce} is ${t.state}; forget settles nothing that is still building, running or merging (train prune --stale for a train whose PRs merged elsewhere)` }, [], `${Q} train status --train ${t.nonce}`, trainDir(t.nonce)); + } + const bisect = loadBisect(t.nonce); + records.push(t, ...attemptsOf(t), ...(bisect ? [bisect] : [])); + files.push(trainDir(t.nonce)); + what = `train ${t.nonce} (${t.state}; ${t.attempts.length} attempt(s)${bisect ? ', a bisect ledger' : ''})`; + } else { + const m = /^#?(\d+)@([0-9a-f]{40,64})$/.exec(o.receipt ?? ''); + if (!m) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: '--receipt takes @' }, [], `${Q} train forget --receipt @`, 'none'); + const r = loadReady(env.repo.keys, m[1]!, m[2]!); + if (!r) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no READY receipt for #${m[1]} at ${m[2]!.slice(0, 12)} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + records.push(r); + files.push(readyPath(r.repo_key, r.pr, r.head)); + what = `READY receipt #${r.pr} at ${r.head.slice(0, 12)} (screen ${r.screen.choice}${r.screen.run ? `, run ${r.screen.run}` : ''})`; + } + const names = records.map(noteName); + const lines = [`FORGET: ${what}`, ...names.map(n => ` note ${n}`), ...files.map(f => ` local ${f}`)]; + if (o.dryRun) return outOk([...lines, 'DRY-RUN: nothing removed'], `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`}`, 'none'); + const f = forgetNotes(ctx.repoRoot, names, env.remote); + if (!f.pushed) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the notes ref was not updated (${f.detail}); nothing deleted locally` }, lines, `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`} (the lease is re-read each run)`, 'none'); + for (const file of files) fs.rmSync(file, { recursive: true, force: true }); + lines.push(`FORGOTTEN: ${f.removed.length} note(s) removed${f.missing.length ? `, ${f.missing.length} already absent` : ''}; ${f.detail}; ${files.length} local path(s) deleted`, + 'note: a machine that still holds these files locally can push the record back with its next write; run train forget there too, or clear its $GSTACK_STATE_ROOT/ship-queue copy'); + return outOk(lines, `${Q} train status`, 'none'); +} diff --git a/lib/ship-queue-train-github.ts b/lib/ship-queue-train-github.ts index 952f0a101..592a125b5 100644 --- a/lib/ship-queue-train-github.ts +++ b/lib/ship-queue-train-github.ts @@ -57,14 +57,21 @@ export function asRun(summary: RunSummary): { id: number; url: string } { return { id: summary.databaseId, url: summary.url }; } -/** One JSON artifact of a run (the named file, else the first .json file it holds); null when it is missing or unreadable. */ +/** + * One JSON artifact of a run: the named file wherever the artifact keeps it + * (a multi-path upload keeps its directories, so report-verdict-a1 holds + * paid-report/collector-outcomes.json), else the first .json file at its + * root; null when it is missing or unreadable. + */ export function downloadJson(ctx: QueueContext, slug: string, run: number, name: string, scratch: string, file?: string): unknown { const dir = path.join(scratch, `${run}-${name}`); fs.rmSync(dir, { recursive: true, force: true }); fs.mkdirSync(dir, { recursive: true }); try { if (!ctx.gh(['run', 'download', String(run), '--repo', slug, '-n', name, '-D', dir]).ok) return null; - const pick = file ?? fs.readdirSync(dir).find(f => f.endsWith('.json')); + const pick = file + ? (fs.readdirSync(dir, { recursive: true }) as string[]).filter(f => path.basename(f) === file).sort((a, b) => a.length - b.length)[0] + : fs.readdirSync(dir).find(f => f.endsWith('.json')); return pick && fs.existsSync(path.join(dir, pick)) ? JSON.parse(fs.readFileSync(path.join(dir, pick), 'utf8')) : null; } catch { return null; } finally { fs.rmSync(dir, { recursive: true, force: true }); } } diff --git a/lib/ship-queue-train-ready.ts b/lib/ship-queue-train-ready.ts index 66730e266..736b74809 100644 --- a/lib/ship-queue-train-ready.ts +++ b/lib/ship-queue-train-ready.ts @@ -2,13 +2,21 @@ * `ready --pr --screen | --no-screen` under a ship policy with a `train` * block (docs/ship-policy.md#run-a-merge-train; plan GSTA-49 T1). The * READY block is today's (lib/ship-queue.ts ready()); this adds the screen - * selection and Garry's default recommendation (screen when the screen + * selection and the default recommendation (screen when the screen * selects a skill's own cases), requires an explicit choice * (READY_SCREEN_CHOICE_REQUIRED), dispatches a chosen screen through * eval-bg's nonce with its purpose and policy in the key, and records the * choice as a READY receipt (local state and refs/notes/gstack-trains) that * `train build` requires. The screen's dollar price is left to the T0 cost * replay (lib/gate-cost.ts) and printed as pending until it lands. + * + * The screen's verdict comes from its run (lib/ship-queue-screen.ts): a head + * whose recorded screen is RED is refused (QUEUE_SCREEN_RED) under either + * choice, and a red run is never re-dispatched at its merge SHA. The screen + * is planned on GitHub's refs/pull//merge when that merges the head, else + * on a scratch merge of the head onto origin/ built with the train's + * merge rules and pushed as screen/- (#3465); only a real + * conflict is QUEUE_SCREEN_STALE_REF. */ import { randomBytes } from 'node:crypto'; import { DISPATCH_INPUTS, awaitNonceRun, dedupeKey, dispatchWorkflow, findDuplicateRun, listDispatchRuns, makeNonce, type WorkflowDispatch } from '../scripts/eval-bg'; @@ -16,13 +24,19 @@ import { EXIT } from './headless-artifacts'; import { ready, type QueueContext } from './ship-queue'; import { classifyTrainHead } from './version-gate-train'; import { git } from './ship-queue-gate'; -import { execFor, hollowScreen, repoSlug } from './ship-queue-train-github'; -import { loadReady, nowIso, persist, readyPath, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record'; -import { outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train'; +import { dispatchCi } from './ship-queue-dispatch-ci'; +import { recordVerdict, screenVerdict, verdictLines, type ScreenOutcome } from './ship-queue-screen'; +import { execFor, repoSlug } from './ship-queue-train-github'; +import { loadReady, nowIso, persist, readyPath, rebuildFromNotes, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record'; +import { fetchBase, mergeCommit, outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train'; const Q = 'gstack-ship-queue'; -export interface ReadyOptions { pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean } +export interface ReadyOptions { + pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean; + /** `--dispatch-ci`: run these workflows on the PR branch, wait, and clear the matching requires-remote pregate lanes before reading the head. */ + dispatchCi?: { workflows: string[]; timeoutSec: number; pollMs: number }; +} export interface ScreenSelection { lines: string[]; recommend: 'screen' | 'no-screen' | null } /** Skill directories a diff touches: top-level directories that hold a SKILL.md.tmpl at the head. */ @@ -70,8 +84,46 @@ async function screenSelection(env: TrainEnv, head: string, baseSha: string): Pr return { ...s, lines: [...s.lines, price] }; } +/** The scratch branch a screen's merge commit is pushed to when GitHub's merge ref lags the head. */ +export const screenBranch = (pr: string, head: string) => `screen/${pr}-${head.slice(0, 12)}`; + +/** + * The merge commit the screen tests: GitHub's refs/pull//merge when it + * merges the PR head, else a scratch merge of the head onto origin/ + * (the train's merge rules) pushed as screen/-, so evals.yml can + * check it out and read its parents. A conflict beyond the CHANGELOG and the + * stamp files is the only QUEUE_SCREEN_STALE_REF left. + */ +function screenMerge(env: TrainEnv, o: ReadyOptions, head: string, baseSha: string): { mergeSha: string; branch?: string; line: string } | TrainOut { + const { ctx } = env; + const ls = git(ctx.repoRoot, ['ls-remote', env.remote, `refs/pull/${o.pr}/merge`]); + if (!ls.ok) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `could not list ${env.remote}'s refs: ${(ls.err || ls.out).trim().replace(/\s+/g, ' ').slice(0, 200) || 'git ls-remote failed'}` }, [], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + const github = ls.out.split(/\s/)[0] ?? ''; + let lag = `${env.remote} has no refs/pull/${o.pr}/merge`; + if (/^[0-9a-f]{40,64}$/.test(github)) { + const fetched = git(ctx.repoRoot, ['fetch', '-q', env.remote, `refs/pull/${o.pr}/merge`]); + const parent = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^2`]).out : ''; + const onto = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^1`]).out : ''; + if (parent === head && onto === baseSha) return { mergeSha: github, line: `SCREEN_MERGE: GitHub's refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges the head ${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)}` }; + lag = !fetched.ok ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} could not be fetched` + : parent !== head ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges ${parent ? parent.slice(0, 12) : 'no PR head'}, not ${head.slice(0, 12)}` + : `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges onto ${onto ? onto.slice(0, 12) : 'no base'}, not ${ctx.base}@${baseSha.slice(0, 12)}`; + } + const m = mergeCommit(ctx.repoRoot, baseSha, head, env.policy.policy, `screen #${o.pr}: merge ${head.slice(0, 12)} onto ${ctx.base} ${baseSha.slice(0, 12)}`); + if ('conflicts' in m) { + return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `#${o.pr} conflicts with ${ctx.base}@${baseSha.slice(0, 12)} beyond the CHANGELOG and the stamp files (${m.conflicts.join(', ')}); GitHub's merge ref lags too (${lag})` }, [], + `resolve the conflict with ${ctx.base}, push, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + } + const branch = screenBranch(o.pr, head); + const line = `SCREEN_MERGE: scratch merge ${m.commit.slice(0, 12)} of #${o.pr}@${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)} (${lag})`; + if (o.dryRun) return { mergeSha: m.commit, branch, line: `${line}; would push ${branch} (dry run)` }; + const p = git(ctx.repoRoot, ['push', '-q', '-f', env.remote, `${m.commit}:refs/heads/${branch}`]); + if (!p.ok) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `could not push the screen's scratch merge as ${branch}: ${p.err.trim().replace(/\s+/g, ' ').slice(0, 200)}` }, [line], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + return { mergeSha: m.commit, branch, line: `${line}; pushed ${branch}` }; +} + /** Dispatch (or adopt) the PR's screen on the base branch: inputs name the PR, its merge SHA and the screen policy. */ -async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; line: string } | TrainOut> { +async function dispatchScreen(env: TrainEnv, o: ReadyOptions, slug: string | null, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; lines: string[] } | TrainOut> { const policy = o.broad ? 'broad' : 'narrow'; const scope = { purpose: 'screen' as const, policy: policy as 'narrow' | 'broad' }; const nonce = makeNonce('pr', mergeSha, baseSha, randomBytes(3).toString('hex'), scope); @@ -80,23 +132,45 @@ async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string, [DISPATCH_INPUTS.baseSha]: baseSha, [DISPATCH_INPUTS.screenPr]: o.pr, [DISPATCH_INPUTS.screenMergeSha]: mergeSha, [DISPATCH_INPUTS.screenPolicy]: policy, } }; const inputs = Object.entries(dispatch.inputs).map(([k, v]) => `${k}=${v}`).join(' '); - if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)` }; - const slug = repoSlug(env.ctx); + if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)`] }; if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, [], `${Q} ready --pr ${o.pr} --screen`, 'none'); const exec = execFor(env.ctx); const existing = findDuplicateRun(listDispatchRuns(dispatch.workflow, exec, slug) ?? [], dedupeKey('pr', mergeSha, baseSha, scope)); - const plan = existing ? hollowScreen(env.ctx, slug, existing.databaseId) : null; - const hollow = plan?.kind === 'hollow' ? plan.reason : null; - if (existing && !hollow) return { choice: { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)` }; + const prior: ScreenOutcome | null = existing ? screenVerdict(env.ctx, slug, existing.databaseId) : null; + if (existing && prior) { + const choice: ScreenChoice = { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha }; + recordVerdict(choice, prior); + const lines = verdictLines(o.pr, existing.databaseId, prior); + if (prior.kind === 'red') return outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at merge ${mergeSha.slice(0, 12)} (run ${existing.databaseId}) is RED on ${prior.red.join(', ') || 'its gate binding'}; a red screen is not re-dispatched at the same merge SHA` }, + lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + if (prior.kind !== 'not-run' && prior.kind !== 'incomplete') { + return { choice, lines: [...lines, `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)`] }; + } + } + const notFollowing = existing && prior ? ` (not following ${prior.detail}: a screen that ${prior.kind === 'not-run' ? 'planned nothing' : 'bound no verdict'} is not a screen)` : ''; const found = dispatchWorkflow(dispatch, env.ctx.base, exec, slug) ? await awaitNonceRun(dispatch.workflow, nonce, exec, slug) : null; if (!found) return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `no ${dispatch.workflow} run named ${nonce} appeared after the screen dispatch` }, [], `${Q} ready --pr ${o.pr} --screen${o.broad ? ' --broad' : ''} (adopts the run by its key once it appears)`, 'none'); - return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${hollow ? ` (not following ${hollow}: a screen that planned nothing is not a screen)` : ''}` }; + return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${notFollowing}`] }; } +const screenLine = (c: ScreenChoice) => `screen: ${c.choice}${c.policy ? ` policy=${c.policy} nonce=${c.nonce} run=${c.run ?? '-'}${c.verdict ? ` verdict=${c.verdict.outcome}` : ''}` : ''}`; + export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise { + const fetchErr = fetchBase(ctx); const env = trainEnv(ctx); if ('code' in env) return outRefused(env, [], 'none', 'none'); + if (fetchErr) return outRefused({ code: 'POLICY_SOURCE_UNAVAILABLE', message: fetchErr }, [], 'none', 'none'); + const ciLines: string[] = []; + if (o.dispatchCi && !o.dryRun) { + const d = await dispatchCi(ctx, { pr: o.pr, pregate: o.pregate, ...o.dispatchCi }); + ciLines.push(...d.lines); + if ('failure' in d) { + const again = `${Q} ready --pr ${o.pr} ${o.screen === 'screen' ? '--screen' : '--no-screen'} --dispatch-ci ${o.dispatchCi.workflows.join(',')} --timeout ${o.dispatchCi.timeoutSec}`; + return (d.failure.code === 'QUEUE_TRAIN_PENDING' ? outPending : outRefused)(d.failure, ciLines, d.failure.code === 'QUEUE_TRAIN_PENDING' ? again : `read the run, repair, push, run /ship, then ${again}`, 'none'); + } + } const base = ready(ctx, o); + base.lines.unshift(...ciLines); if (!base.ok) { const out = outRefused(base.failure!, base.lines, `fix the refusal, then ${Q} ready --pr ${o.pr} --screen|--no-screen`, base.run); return ['QUEUE_VERIFY_MISMATCH', 'QUEUE_STAMP_NOT_STAMP_ONLY'].includes(base.failure!.code) ? { ...out, exit: EXIT.fail } : out; @@ -107,6 +181,18 @@ export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise l.startsWith('READY ')) + 1, 0, screenLine(choice)); const now = nowIso(); - const readyAt = loadReady(env.repo.keys, o.pr, head)?.ready_at ?? now; - const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: readyAt, screen: choice, updated_at: now }; + const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: prior?.ready_at ?? now, screen: choice, updated_at: now }; if (o.dryRun) return outOk([...lines, 'DRY-RUN: no READY receipt written, nothing dispatched'], `${Q} ready --pr ${o.pr} --${o.screen}${o.broad ? ' --broad' : ''}`, 'none'); const p = persist(ctx.repoRoot, [receipt], { remote: env.remote }); lines.push(`READY_RECEIPT: ${readyPath(env.repo.key, o.pr, head)} (${p.detail})`); return outOk(lines, `send the READY block; the coordinator builds the train: ${Q} train build --prs ${o.pr}@${head},...`, readyPath(env.repo.key, o.pr, head)); } + +/** + * `verify --pr --screen`: the screen verdict of the PR's pushed head from + * its run, recorded in the READY receipt; never a lane's prose. + */ +export function verifyScreen(ctx: QueueContext, o: { pr: string }): TrainOut { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + const view = ctx.gh(['pr', 'view', o.pr, '--json', 'headRefOid']); + let head = ''; + try { head = view.ok ? String(JSON.parse(view.out).headRefOid ?? '') : ''; } catch {} + if (!/^[0-9a-f]{40,64}$/.test(head)) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `cannot read PR #${o.pr}'s head through gh (${(view.err ?? view.out).trim().slice(0, 120) || 'no headRefOid'})` }, [], `${Q} verify --pr ${o.pr} --screen`, 'none'); + if (!loadReady(env.repo.keys, o.pr, head)) rebuildFromNotes(ctx.repoRoot, env.remote); + const receipt = loadReady(env.repo.keys, o.pr, head); + const run = receipt ? readyPath(receipt.repo_key, o.pr, head) : 'none'; + if (!receipt) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${o.pr} at ${head.slice(0, 12)} (local state or refs/notes/gstack-trains)` }, [], `the lane runs ${Q} ready --pr ${o.pr} --screen|--no-screen`, 'none'); + const lines = [`READY: #${o.pr} ${head} ready_at=${receipt.ready_at}`, screenLine(receipt.screen)]; + if (receipt.screen.choice !== 'screen' || !receipt.screen.run) return outOk([...lines, `SCREEN_VERDICT: #${o.pr} none (${receipt.screen.choice === 'screen' ? 'no run recorded' : 'READY --no-screen'})`], `${Q} train build --prs ${o.pr}@${head},...`, run); + const slug = repoSlug(ctx); + if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, lines, `${Q} verify --pr ${o.pr} --screen`, run); + const read = screenVerdict(ctx, slug, receipt.screen.run); + lines.push(...verdictLines(o.pr, receipt.screen.run, read)); + if (recordVerdict(receipt.screen, read)) { receipt.updated_at = nowIso(); lines.push(`READY_RECEIPT: ${run} (${persist(ctx.repoRoot, [receipt], { remote: env.remote }).detail}; verdict recorded)`); } + const again = `${Q} verify --pr ${o.pr} --screen`; + switch (read.kind) { + case 'green': return outOk(lines, `${Q} train build --prs ${o.pr}@${head},...`, run); + case 'pending': return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `the screen run ${receipt.screen.run} has not finished (${read.detail})` }, lines, again, run); + case 'red': return { ...outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at ${head.slice(0, 12)} (run ${receipt.screen.run}) is RED on ${read.red.join(', ') || 'its gate binding'}` }, lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, run), exit: EXIT.fail }; + case 'not-run': return outRefused({ code: 'QUEUE_SCREEN_NOT_RUN', message: `the screen of #${o.pr} (run ${receipt.screen.run}) concluded success with no slice: ${read.detail}; it is not a screen, and this head is not screened` }, lines, `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)`, run); + default: return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the screen of #${o.pr} (run ${receipt.screen.run}) has no verdict to read: ${read.detail}` }, lines, read.kind === 'incomplete' ? `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)` : again, run); + } +} diff --git a/lib/ship-queue-train-record.ts b/lib/ship-queue-train-record.ts index 16b4ac547..07e356791 100644 --- a/lib/ship-queue-train-record.ts +++ b/lib/ship-queue-train-record.ts @@ -20,11 +20,18 @@ const NOTES_REMOTE_REF = 'refs/notes/gstack-trains-origin'; export type TrainState = | 'built' | 'pushed' | 'free-pending' | 'free-red' | 'free-green' | 'dispatch-unknown' | 'dispatch-absent' | 'gate-pending' - | 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded'; -/** States after which a train never dispatches again; prune deletes their branches. */ -export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded']; + | 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded' | 'stale'; +/** States after which a train never dispatches again; prune deletes their branches (`stale`: every PR merged or closed outside the train, `train prune --stale`). */ +export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded', 'stale']; -export interface ScreenChoice { choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null } +/** What the screen run itself said (lib/ship-queue-screen.ts), recorded when it was read; absent while the run is pending or unread. */ +export interface ScreenVerdictRecord { outcome: 'GREEN' | 'RED' | 'NOT_RUN' | 'INCOMPLETE'; run: number; red: string[]; flaky: string[]; detail: string; read_at: string } +export interface ScreenChoice { + choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null; + /** The scratch branch `ready --screen` pushed when GitHub's merge ref lagged the head (docs/ship-policy.md#run-a-merge-train). */ + branch?: string; + verdict?: ScreenVerdictRecord; +} export interface TrainSlot { pr: string; head: string; ready_at: string; screen: ScreenChoice } export interface TrainPrefix { index: number; commit: string; tree: string; pr: string | null } export interface FreeRun { run: number | null; url: string | null; head_sha: string | null; conclusion: string | null; verdict: 'green' | 'red' | 'missing' | 'stale' | 'cancelled' | 'pending' } @@ -154,7 +161,7 @@ export function recordPath(r: AnyRecord): string { return readyPath(r.repo_key, r.pr, r.head); } -function noteName(r: AnyRecord): string { +export function noteName(r: AnyRecord): string { if (r.kind === 'train') return `gstack-trains:train:${r.nonce}`; if (r.kind === 'train-attempt') return `gstack-trains:attempt:${r.train}:${r.id}`; if (r.kind === 'train-bisect') return `gstack-trains:bisect:${r.train}`; @@ -227,6 +234,35 @@ function parseNote(text: string): AnyRecord | null { return null; } +/** + * Remove records from the notes ref (`train forget`): each named note is + * removed locally, then the ref is pushed under an exact lease on the remote + * tip this call fetched, so a concurrent writer's push is never overwritten + * (a rejected lease returns `pushed: false` and nothing is deleted locally). + * Names are note names (noteName); a name with no note is reported missing. + */ +export function forgetNotes(repoRoot: string, names: string[], remote = 'origin'): { removed: string[]; missing: string[]; failed: string[]; pushed: boolean; lease: string | null; detail: string } { + const f = fetchNotes(repoRoot, remote); + if (!f.ok) return { removed: [], missing: [], failed: [], pushed: false, lease: null, detail: f.detail }; + const lease = git(repoRoot, ['rev-parse', '--verify', '--quiet', NOTES_REMOTE_REF]).out || null; + const before = git(repoRoot, ['rev-parse', '--verify', '--quiet', NOTES_REF]).out || null; + const listed = git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'list']).out.split('\n').map(l => l.split(' ')[1]).filter(Boolean); + const removed: string[] = []; + const missing: string[] = []; + const failed: string[] = []; + const restore = () => { if (before) git(repoRoot, ['update-ref', NOTES_REF, before]); }; + for (const name of names) { + const blob = git(repoRoot, ['hash-object', '--stdin'], undefined, name).out; + if (!listed.includes(blob)) { missing.push(name); continue; } + if (git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'remove', blob], notesEnv()).ok) removed.push(name); else failed.push(name); + } + if (failed.length) { restore(); return { removed: [], missing, failed, pushed: false, lease, detail: `git notes remove failed for ${failed.join(', ')}; the local notes ref was restored` }; } + if (!removed.length) return { removed, missing, failed, pushed: false, lease, detail: 'no note to remove' }; + const p = git(repoRoot, ['push', '-q', lease ? `--force-with-lease=${NOTES_REF}:${lease}` : `--force-with-lease=${NOTES_REF}`, remote, `${NOTES_REF}:${NOTES_REF}`]); + if (!p.ok) restore(); + return { removed, missing, failed, pushed: p.ok, lease, detail: p.ok ? `notes pushed to ${remote} ${NOTES_REF} (lease ${lease?.slice(0, 12) ?? 'new ref'})` : `git push ${NOTES_REF} rejected: ${p.err.replace(/\s+/g, ' ').trim().slice(0, 160)}; the local notes ref was restored` }; +} + /** Rebuild local records from the notes ref: a note whose record is missing locally, or newer than the local copy, is written. */ export function rebuildFromNotes(repoRoot: string, remote = 'origin'): { restored: number; read: number; detail: string } { const f = fetchNotes(repoRoot, remote); diff --git a/lib/ship-queue-train-watch.ts b/lib/ship-queue-train-watch.ts new file mode 100644 index 000000000..160cb3dba --- /dev/null +++ b/lib/ship-queue-train-watch.ts @@ -0,0 +1,68 @@ +/** + * `train watch --train [--timeout ]`: the controller's one + * resumable follow command. Machines restart and kill + * background `gh run watch` chains, so watch keeps no process state: it + * reads the train record, runs the step the state asks for (train build + * --train to push, run the free tests and dispatch the gate; train status to + * settle an unconfirmed dispatch), waits for the pending run to finish by + * polling gh, then reads the verdict, and prints the last step's NEXT. Run it + * again after an interruption and it continues from the record. It never + * dispatches anything the record would not dispatch on its own. + */ +import type { QueueContext } from './ship-queue'; +import { buildTrain, findTrain, nextFor, outOk, outPending, outRefused, trainEnv, trainStatus, type TrainOut } from './ship-queue-train'; +import { repoSlug, runView } from './ship-queue-train-github'; +import { attemptsOf, trainFile, type TrainRecord } from './ship-queue-train-record'; +import { trainVerdict } from './ship-queue-train-verdict'; + +const Q = 'gstack-ship-queue'; +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface WatchOptions { train: string; timeoutSec: number; pollMs: number; freeTimeoutSec: number } + +/** Wait for the train's current gate run to complete; null when it did, else why the wait stopped. */ +async function awaitGate(ctx: QueueContext, t: TrainRecord, deadline: number, pollMs: number): Promise { + const a = attemptsOf(t).filter(x => x.purpose !== 'bisect').at(-1); + if (!a?.run) return `no gate run is recorded for ${t.nonce}`; + const slug = repoSlug(ctx); + if (!slug) return 'gh repo view could not name the repository'; + for (;;) { + const view = runView(ctx, slug, a.run.id); + if (view?.status === 'completed') return null; + if (Date.now() >= deadline) return `run ${a.run.id} is still ${view?.status ?? 'unreadable'} at the watch timeout`; + await sleep(pollMs); + } +} + +export async function trainWatch(ctx: QueueContext, o: WatchOptions): Promise { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + let t = findTrain(env, o.train); + if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + const run = trainFile(t.nonce); + const deadline = Date.now() + o.timeoutSec * 1000; + const lines: string[] = [`WATCH: train ${t.nonce} state=${t.state} (resumable; rerun this command after an interruption)`]; + const again = `${Q} train watch --train ${t.nonce} --timeout ${o.timeoutSec}`; + const step = (label: string, out: TrainOut): TrainOut => ({ ...out, lines: [...lines, `WATCH: ${label}`, ...out.lines] }); + const reload = () => { t = findTrain(env, o.train) ?? t; }; + if (t.state === 'dispatch-unknown') { + const s = trainStatus(ctx, { train: t.nonce }); + lines.push(`WATCH: train status settled the unconfirmed dispatch`, ...s.lines.filter(l => l.startsWith('ATTEMPT:'))); + reload(); + } + if (['built', 'pushed', 'free-pending', 'free-green', 'dispatch-absent'].includes(t.state)) { + const remaining = Math.max(1, Math.floor((deadline - Date.now()) / 1000)); + const b = await buildTrain(ctx, { resume: t.nonce, dryRun: false, urgent: false, repair: false, freeTimeoutSec: Math.min(o.freeTimeoutSec, remaining), pollMs: o.pollMs }); + reload(); + if (b.status !== 'complete') return step(`train build --train ${t.nonce} stopped (${b.failure?.code ?? b.status})`, b.failure?.code === 'QUEUE_TRAIN_PENDING' ? { ...b, next: again } : b); + lines.push(`WATCH: train build --train ${t.nonce}: free tests green, gate dispatched`); + } + if (t.state === 'gate-pending') { + const why = await awaitGate(ctx, t, deadline, o.pollMs); + if (why) return outPending({ code: 'QUEUE_TRAIN_PENDING', message: why }, lines, again, run); + lines.push('WATCH: gate run completed; reading the verdict'); + return step(`train verdict --train ${t.nonce}`, trainVerdict(ctx, { train: t.nonce })); + } + if (t.state === 'dispatch-unknown') return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `the gate dispatch for ${t.nonce} is still unconfirmed` }, lines, again, run); + return outOk([...lines, `WATCH: nothing to wait for in state ${t.state}`], nextFor(t), run); +} diff --git a/lib/ship-queue-train.ts b/lib/ship-queue-train.ts index 698b78817..48c13cd80 100644 --- a/lib/ship-queue-train.ts +++ b/lib/ship-queue-train.ts @@ -23,7 +23,9 @@ import type { LoadedPolicy, ShipPolicy, TrainPolicy } from './ship-policy'; import { loadBasePolicy, type QueueContext } from './ship-queue'; import { bindReceipt } from './ship-receipt'; import { git, sha256, withIndex, type QueueFailure } from './ship-queue-gate'; -import { asRun, execFor, hollowScreen, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github'; +import { recordVerdict, screenVerdict, verdictLines } from './ship-queue-screen'; +import { stackTestLine, stackTests } from './ship-queue-stack-tests'; +import { asRun, execFor, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github'; import { TERMINAL_STATES, activeFreeze, attemptsOf, listTrains, loadBisect, loadReady, loadTrain, nowIso, persist, rebuildFromNotes, repoIdentity, trainFile, type AttemptPurpose, type AttemptRecord, type TrainRecord, type TrainSlot, @@ -51,7 +53,7 @@ export function trainEnv(ctx: QueueContext): TrainEnv | QueueFailure { export function fetchBase(ctx: QueueContext): string | null { const remote = ctx.remote ?? 'origin'; const f = git(ctx.repoRoot, ['fetch', '--no-tags', '-q', remote, `+refs/heads/${ctx.base}:refs/remotes/${remote}/${ctx.base}`]); - return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.slice(0, 160)}`; + return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.replace(/\s+/g, ' ').trim().slice(0, 160)}`; } /** Make the train's prefix commits local (another machine built them): fetch train/. */ @@ -195,17 +197,28 @@ export function mergeStep(repoRoot: string, pred: string, head: string, policy: const TRAIN_IDENTITY = { GIT_AUTHOR_NAME: 'gstack-ship-queue', GIT_AUTHOR_EMAIL: 'ship-queue@gstack.local', GIT_COMMITTER_NAME: 'gstack-ship-queue', GIT_COMMITTER_EMAIL: 'ship-queue@gstack.local', GIT_AUTHOR_DATE: '2000-01-01T00:00:00Z', GIT_COMMITTER_DATE: '2000-01-01T00:00:00Z' }; +/** + * One train merge as a commit with a fixed identity and date: parents are the + * predecessor (first, as GitHub's merge ref) and the PR head. Prefix commits + * and the scratch merge a screen is planned on (`ready --screen` when + * GitHub's refs/pull//merge lags the head) are both built here. + */ +export function mergeCommit(repoRoot: string, pred: string, head: string, policy: ShipPolicy, message: string): { commit: string; tree: string } | { conflicts: string[] } { + const m = mergeStep(repoRoot, pred, head, policy); + if ('conflicts' in m) return m; + const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', head, '-m', message], { ...process.env, ...TRAIN_IDENTITY }); + return c.ok ? { commit: c.out, tree: m.tree } : { conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`] }; +} + /** Build T1..Tk on T0 = base; a conflict stops at the PR that conflicts. */ export function buildChain(repoRoot: string, baseSha: string, slots: TrainSlot[], policy: ShipPolicy): { prefixes: TrainRecord['prefixes'] } | { stoppedAt: string; conflicts: string[]; merged: string[] } { const prefixes: TrainRecord['prefixes'] = [{ index: 0, commit: baseSha, tree: git(repoRoot, ['rev-parse', `${baseSha}^{tree}`]).out, pr: null }]; for (const [i, slot] of slots.entries()) { const pred = prefixes[i]!.commit; - const m = mergeStep(repoRoot, pred, slot.head, policy); + const m = mergeCommit(repoRoot, pred, slot.head, policy, `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`); if ('conflicts' in m) return { stoppedAt: slot.pr, conflicts: m.conflicts, merged: slots.slice(0, i).map(s => s.pr) }; - const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', slot.head, '-m', `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`], { ...process.env, ...TRAIN_IDENTITY }); - if (!c.ok) return { stoppedAt: slot.pr, conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`], merged: slots.slice(0, i).map(s => s.pr) }; - prefixes.push({ index: i + 1, commit: c.out, tree: m.tree, pr: slot.pr }); + prefixes.push({ index: i + 1, commit: m.commit, tree: m.tree, pr: slot.pr }); } return { prefixes }; } @@ -266,7 +279,11 @@ export function gateDispatch(env: TrainEnv, train: TrainRecord, nonce: string): return { workflow: env.train.gate_workflow, inputs: { [DISPATCH_INPUTS.evalsAll]: 'true', [DISPATCH_INPUTS.expectedSha]: train.head, [DISPATCH_INPUTS.nonce]: nonce } }; } -export interface BuildOptions { pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number } +export interface BuildOptions { + pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number; + /** `--dry-run --stack-tests`: run each PR's new test files on its integrated prefix (lib/ship-queue-stack-tests.ts); the command's words when `--stack-tests-command` names one. */ + stackTests?: string[] | true; +} /** `--order ,,...`: every pinned PR exactly once (a leading # allowed). */ export function parseOrder(spec: string | undefined, pins: Array<{ pr: string }>): string[] | undefined | { error: string } { @@ -279,7 +296,7 @@ export function parseOrder(spec: string | undefined, pins: Array<{ pr: string }> } /** The slots a new train would carry: trusted, unmoved, READY PRs, in slot order. */ -function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>): { slots: TrainSlot[]; lines: string[] } | TrainOut { +function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>, dryRun: boolean): { slots: TrainSlot[]; lines: string[] } | TrainOut { const { ctx } = env; const specs = pins.map(p => `+refs/pull/${p.pr}/head:refs/gstack/train/pr-${p.pr}`); const f = git(ctx.repoRoot, ['fetch', '--no-tags', '-q', env.remote, ...specs]); @@ -302,9 +319,16 @@ function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>): { if (!ready) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${p.pr} at ${info.headRefOid.slice(0, 12)} (local state or ${'refs/notes/gstack-trains'})` }, [], `the lane runs ${Q} ready --pr ${p.pr} --screen|--no-screen`, 'none'); let screen = ready.screen; if (screen.choice === 'screen' && screen.run && slug === undefined) slug = repoSlug(ctx); - const plan = screen.choice === 'screen' && screen.run ? (slug ? hollowScreen(ctx, slug, screen.run) : { kind: 'unknown' as const, reason: 'the repository slug is unreadable' }) : null; - if (plan?.kind === 'hollow') { lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; } - else if (plan?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as screened, unverified`); + const read = screen.choice === 'screen' && screen.run ? (slug ? screenVerdict(ctx, slug, screen.run) : { kind: 'unknown' as const, detail: 'the repository slug is unreadable' }) : null; + if (read && screen.run && recordVerdict(ready.screen, read) && !dryRun) { ready.updated_at = nowIso(); persist(ctx.repoRoot, [ready], { remote: env.remote }); } + if (read?.kind === 'red') { + return outRefused({ code: 'QUEUE_SCREEN_RED', message: `#${p.pr}'s screen run ${screen.run} is RED on ${read.red.join(', ') || 'its gate binding'} (${read.detail}); a red head is not seated` }, + [...lines, ...verdictLines(p.pr, screen.run!, read)], without ? `${Q} train build --prs ${without}` : 'none', 'none'); + } + if (read?.kind === 'not-run' || read?.kind === 'incomplete') { lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; } + else if (read?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as screened, unverified`); + else if (read?.kind === 'green') lines.push(`SCREEN: #${p.pr} run ${screen.run} GREEN${read.flaky.length ? ` (flaky: ${read.flaky.join(', ')})` : ''}`); + else if (read?.kind === 'pending') lines.push(`SCREEN: #${p.pr} run ${screen.run} still running; counted as screened`); slots.push({ pr: p.pr, head: info.headRefOid, ready_at: ready.ready_at, screen }); } return { slots, lines }; @@ -321,7 +345,7 @@ async function newTrain(env: TrainEnv, o: BuildOptions): Promise<{ t: TrainRecor frozenLines(freeze), `${Q} train build --repair --prs @`, 'none'); const explicit = parseOrder(o.order, pins); if (explicit && 'error' in explicit) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: explicit.error }, [], 'none', 'none'); - const admitted = admitSlots(env, pins); + const admitted = admitSlots(env, pins, o.dryRun); if ('exit' in admitted) return admitted; const order = orderSlots(admitted.slots, o.repair ? 1 : env.train.max_size, explicit); const lines = [ @@ -460,8 +484,18 @@ export async function buildTrain(ctx: QueueContext, o: BuildOptions): Promise !r.ok); + if (red.length) { + const again = `${Q} train build ${prsArgs(t.slots, t.explicit_order)} --dry-run --stack-tests`; + return outFailed({ code: 'QUEUE_TRAIN_STACK_RED', message: red.map(r => `#${r.pr}'s new tests fail on the integrated prefix T${r.prefix} (${r.files.join(', ')})`).join('; ') + '; they pass on the PR head alone only if nothing before it in the train changes what they read' }, + [...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `fix ${red.map(r => `#${r.pr}'s tests against T${r.prefix}`).join(' and ')}, or reorder with --order, then ${again}`, 'none'); + } + } + return outOk([...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `${Q} train build ${prsArgs(t.slots, t.explicit_order)}`, 'none'); } if (t.state === 'built' && !t.attempts.length && !loadTrain(t.nonce)) { effect(t, 'train.json written'); @@ -528,21 +562,64 @@ export function trainStatus(ctx: QueueContext, o: { train?: string }): TrainOut return outOk(lines, next, single ? trainFile(single.nonce) : 'none'); } -/** `train prune`: delete train/ on the remote for every finished (merged, superseded, dropped) train. */ -export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string }): TrainOut { +/** A PR's state on GitHub for the stale check: MERGED, CLOSED, OPEN, or unreadable. */ +function prState(ctx: QueueContext, pr: string): string { + const info = prInfo(ctx, pr); + return 'error' in info ? 'unreadable' : info.state.toUpperCase(); +} + +/** + * `--stale`: a merging or green train whose every PR is MERGED or CLOSED on + * GitHub (merged elsewhere, closed, or merged outside the train) never + * finishes on its own; it becomes `stale` (terminal) and is pruned, with + * each PR's state in the log. Any OPEN or unreadable PR keeps the train. + */ +function staleCheck(ctx: QueueContext, t: TrainRecord): { stale: true; detail: string } | { stale: false; detail: string } { + const unmerged = t.slots.filter(s => !t.merged.some(m => m.pr === s.pr)); + const states = unmerged.map(s => `#${s.pr} ${prState(ctx, s.pr)}`); + const open = states.filter(s => !/ (MERGED|CLOSED)$/.test(s)); + const detail = `${t.merged.map(m => `#${m.pr} merged by the train`).concat(states).join(', ')}`; + return open.length ? { stale: false, detail: `${open.join(', ')} still ${open.length === 1 ? 'needs' : 'need'} the train` } : { stale: true, detail }; +} + +/** `train prune`: delete train/ on the remote for every finished (merged, superseded, dropped, stale) train; `--stale` first settles merging and green trains whose PRs all merged or closed elsewhere, and the screen/-* branches of merged or closed PRs. */ +export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string; stale?: boolean }): TrainOut { const env = trainEnv(ctx); if ('code' in env) return outRefused(env, [], 'none', 'none'); rebuildFromNotes(ctx.repoRoot, env.remote); const lines: string[] = []; const rgit = (args: string[]) => git(ctx.repoRoot, args); + const remove = (branch: string, label: string, after?: () => void) => { + if (!remoteBranchSha(rgit, env.remote, branch)) { lines.push(`GONE: ${branch} (${label})`); return; } + if (o.dryRun) { lines.push(`WOULD-DELETE: ${branch} (${label})`); return; } + const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${branch}`]); + lines.push(d.ok ? `DELETED: ${branch} (${label})` : `FAILED: ${branch}: ${d.err.slice(0, 120)}`); + if (d.ok) after?.(); + }; for (const t of listTrains(env.repo.keys).filter(x => !o.train || x.nonce === o.train)) { + if (o.stale && (t.state === 'merging' || t.state === 'green')) { + const s = staleCheck(ctx, t); + if (s.stale) { + lines.push(`STALE: ${t.nonce} (${t.state}): ${s.detail}${o.dryRun ? '; would mark stale' : ''}`); + if (!o.dryRun) { t.state = 'stale'; effect(t, `marked stale by train prune --stale: ${s.detail}`); save(env, t); } + remove(t.branch, o.dryRun ? 'stale' : t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); }); + continue; + } + lines.push(`KEEP: ${t.branch} (${t.state}; ${s.detail})`); + continue; + } if (!TERMINAL_STATES.includes(t.state)) { lines.push(`KEEP: ${t.branch} (${t.state})`); continue; } - if (!remoteBranchSha(rgit, env.remote, t.branch)) { lines.push(`GONE: ${t.branch} (${t.state})`); continue; } - if (o.dryRun) { lines.push(`WOULD-DELETE: ${t.branch} (${t.state})`); continue; } - const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${t.branch}`]); - lines.push(d.ok ? `DELETED: ${t.branch} (${t.state})` : `FAILED: ${t.branch}: ${d.err.slice(0, 120)}`); - if (d.ok) { effect(t, `branch deleted: ${t.branch}`); save(env, t); } + remove(t.branch, t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); }); + } + if (o.stale && !o.train) { + const refs = git(ctx.repoRoot, ['ls-remote', env.remote, 'refs/heads/screen/*']); + for (const ref of refs.ok ? refs.out.split('\n') : []) { + const m = /refs\/heads\/(screen\/(\d+)-[0-9a-f]{12})$/.exec(ref.split(/\s/)[1] ?? ''); + if (!m) continue; + const state = prState(ctx, m[2]!); + if (state === 'MERGED' || state === 'CLOSED') remove(m[1]!, `#${m[2]} ${state}`); else lines.push(`KEEP: ${m[1]} (#${m[2]} ${state})`); + } } if (!lines.length) lines.push('PRUNE: no trains'); - return outOk(lines, o.dryRun ? `${Q} train prune` : 'none', 'none'); + return outOk(lines, o.dryRun ? `${Q} train prune${o.stale ? ' --stale' : ''}` : 'none', 'none'); } diff --git a/lib/ship-queue.ts b/lib/ship-queue.ts index e4b80b6b0..166a9a24b 100644 --- a/lib/ship-queue.ts +++ b/lib/ship-queue.ts @@ -146,20 +146,21 @@ export function latestRunPerName } { const at = sha.slice(0, 12); const runs = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/check-runs`, 'check_runs'); - if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false }; + if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false, checks: [] }; const statuses = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/status`, 'statuses'); - if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false }; + if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false, checks: [] }; + const checks = latestRunPerName(runs.items).map(c => ({ name: String(c.name ?? ''), status: String(c.status ?? ''), conclusion: c.conclusion === undefined || c.conclusion === null ? null : String(c.conclusion) })); const verdicts = [ - ...latestRunPerName(runs.items).map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'), + ...checks.map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'), ...statuses.items.map(s => s.state === 'success' ? 'passed' : s.state === 'pending' ? 'pending' : 'failed'), ]; const count = (v: string) => verdicts.filter(x => x === v).length; const [passed, failed, pending, skipped] = [count('passed'), count('failed'), count('pending'), count('skipped')]; const green = verdicts.length > 0 && pending === 0 && failed === 0; - return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0 }; + return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0, checks }; } /** diff --git a/package.json b/package.json index dee1c57e9..2ef96a72b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.262", + "version": "1.91.263", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/lib/windows-curation.ts b/scripts/lib/windows-curation.ts index 5547efe53..772a4d663 100644 --- a/scripts/lib/windows-curation.ts +++ b/scripts/lib/windows-curation.ts @@ -295,6 +295,10 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }> file: 'test/ship-queue-train-cost.test.ts', reason: 'drives bin/gstack-ship-queue train verdict against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts; on windows-latest spawnSync(\'gh\') resolves the runner\'s gh.exe instead of the shim (windows-free-shard 2 on #3478 at d5e6a7d)', }, + { + file: 'test/ship-queue-train-ops.test.ts', + reason: 'drives bin/gstack-ship-queue (screen verdicts, scratch-merge screens, prune --stale, watch, forget, verify --post-merge, ready --dispatch-ci, --stack-tests) against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts', + }, { file: 'test/ship-policy-compat.test.ts', reason: 'spawns bin/gstack-ship-policy, bin/gstack-restamp and bin/gstack-tree-receipt (shebangs) through test/helpers/restamp-fixture.ts', diff --git a/test/helpers/ship-queue-train-fixture.ts b/test/helpers/ship-queue-train-fixture.ts index 87ac2286c..04d85ef65 100644 --- a/test/helpers/ship-queue-train-fixture.ts +++ b/test/helpers/ship-queue-train-fixture.ts @@ -31,13 +31,22 @@ export interface Rule { plan?: { planned: number } | { manifest: unknown } | null; /** Marks every artifact of runs dispatched under this rule expired in the artifacts listing. */ expired?: boolean; + /** A screen run's gate binding (gate-binding-a1/gate-binding.json, scripts/gate-reuse.ts) and its report's collector-outcomes.json (report-verdict-a1). */ + binding?: { verdict: 'GREEN' | 'RED'; reconcile_exit?: number; flaky?: string[] }; + outcomes?: { red?: string[]; flakyLines?: string[] }; + /** The run confirmed a verdict reuse (R2a/R2): paid-plan carries a valid gate-reuse.json naming this run and no binding is uploaded. */ + reuse?: { run: number; flaky?: string[] }; check?: string | null; + /** The check-run name a run dispatched under this rule attaches to its head (a push or workflow_dispatch run's job), with the run's status and conclusion. */ + ci?: string; bisect?: { prefixes: string[]; patterns: Record; costUsd?: number }; } export interface ShimData { repo: string; origin: string; nextId: number; prs: Record; runs: Array>; artifacts: Record>; checks: Record>; ci: Record; rules: Record; artifact: string; log: Array>; + /** Commit statuses (the legacy status API) at a sha, beside the check runs in `ci`. */ + statuses?: Record>; } export interface TrainFixture { repo: FixtureRepo; state: string; dataFile: string; env: Record; read: () => ShimData; write: (fn: (d: ShimData) => void) => void } @@ -163,6 +172,13 @@ function shimSource(): string { " if (rule.record && !inputs.prefix_shas) d.artifacts[id][d.artifact] = { schema: 1, kind: 'train-gate', nonce: rule.record.nonce || inputs.nonce, head_sha: head, tree: rev(head + '^{tree}'),", " verdict: rule.record.verdict || 'GREEN', census: rule.record.census, panels: rule.record.panels };", " if (rule.check && !inputs.prefix_shas) (d.checks[head] = d.checks[head] || []).push({ name: 'gstack/full-gate', status: 'completed', conclusion: rule.check });", + " if (rule.ci) (d.ci[head] = d.ci[head] || []).push({ name: rule.ci, status: run.status, conclusion: run.conclusion, id: id, completed_at: '2026-10-26T00:00:00Z' });", + " if (rule.binding) d.artifacts[id]['gate-binding-a1'] = { schema: 1, repository: d.repo, pr: Number(inputs.screen_pr) || 0, run_id: id, run_attempt: 1, head_sha: rev(head + '^2') || head, base_sha: rev(head + '^1') || head,", + " profile: 'pr', reconcile_exit: rule.binding.reconcile_exit === undefined ? (rule.binding.verdict === 'GREEN' ? 0 : 1) : rule.binding.reconcile_exit, verdict: rule.binding.verdict, manual_accepted: 0, selection_sha256: 'a'.repeat(64),", + " flaky_cases: rule.binding.flaky || [], purpose: 'screen', tested_sha: head, screen_policy: inputs.screen_policy || 'narrow' };", + " if (rule.reuse && d.artifacts[id]['paid-plan']) d.artifacts[id]['paid-plan'].__reuse = { schema: 1, kind: 'main-moved', pr: Number(inputs.screen_pr) || 0, reused_run_id: rule.reuse.run, reused_run_attempt: 1, gated_head: 'b'.repeat(40), current_head: 'c'.repeat(40),", + " gated_base: 'd'.repeat(40), base_sha: 'e'.repeat(40), selection_sha256: 'a'.repeat(64), why: 'same own diff on a moved main', gated_merge_base: 'f'.repeat(40), merge_base: '1'.repeat(40), own_diff_sha256: 'b'.repeat(64), reused_screen_policy: 'narrow', reused_flaky: rule.reuse.flaky || [] };", + " if (rule.outcomes) d.artifacts[id]['report-verdict-a1'] = { version: 2, panels: (rule.outcomes.red || []).map(c => ({ case: c, kind: 'judge', status: 'FAIL', failsLane: true, redClass: 'VERDICT' })), flakyLines: rule.outcomes.flakyLines || [] };", " if (rule.bisect && inputs.prefix_shas) for (const sha of inputs.prefix_shas.split(',')) {", " const at = rule.bisect.prefixes.indexOf(sha);", " d.artifacts[id][d.artifact + '-bisect-' + sha.slice(0, 12)] = { schema: 1, kind: 'train-bisect-prefix', nonce: inputs.nonce, prefix_sha: sha, checked_out_sha: sha,", @@ -178,8 +194,9 @@ function shimSource(): string { "if (a[0] === 'run' && a[1] === 'view') { const r = d.runs.find(x => String(x.databaseId) === a[2]); if (!r) fail('run not found'); out(r); }", "if (a[0] === 'run' && a[1] === 'download') {", " const art = (d.artifacts[a[2]] || {})[opt('-n')]; if (!art) fail('no artifact matches');", - " fs.mkdirSync(opt('-D'), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), opt('-n') === 'paid-plan' ? 'manifest.json' : 'record.json'), JSON.stringify(art));", - " if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);", + " const fileFor = n => n === 'paid-plan' ? 'manifest.json' : n.startsWith('gate-binding-') ? 'gate-binding.json' : n.startsWith('report-verdict-') ? 'paid-report/collector-outcomes.json' : 'record.json';", + " fs.mkdirSync(path.dirname(path.join(opt('-D'), fileFor(opt('-n')))), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), fileFor(opt('-n'))), JSON.stringify(art));", + " if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), art.__reuse ? JSON.stringify(art.__reuse) : '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);", "}", "if (a[0] === 'api') {", " const arts = /actions\\/runs\\/(\\d+)\\/artifacts(?:\\?name=([^&]+))?/.exec(a[1] || '');", @@ -187,7 +204,7 @@ function shimSource(): string { " const named = /commits\\/([0-9a-f]+)\\/check-runs\\?check_name=([^&]+)/.exec(a[1] || '');", " if (named) { const list = (d.checks[named[1]] || []).filter(c => c.name === decodeURIComponent(named[2])); out({ total_count: list.length, check_runs: list }); }", " const paged = /commits\\/([0-9a-f]+)\\/(check-runs|status)\\?.*page=(\\d+)/.exec(a[1] || '');", - " if (paged) { const list = paged[2] === 'check-runs' ? (d.ci[paged[1]] || []) : []; const key = paged[2] === 'check-runs' ? 'check_runs' : 'statuses';", + " if (paged) { const list = paged[2] === 'check-runs' ? (d.ci[paged[1]] || []) : ((d.statuses || {})[paged[1]] || []); const key = paged[2] === 'check-runs' ? 'check_runs' : 'statuses';", " out(paged[3] === '1' ? { total_count: list.length, state: 'success', [key]: list } : { total_count: list.length, [key]: [] }); }", "}", "fail('gh shim: unsupported ' + a.join(' '));", diff --git a/test/ship-queue-train-ops.test.ts b/test/ship-queue-train-ops.test.ts new file mode 100644 index 000000000..e2af535fe --- /dev/null +++ b/test/ship-queue-train-ops.test.ts @@ -0,0 +1,474 @@ +/** + * Operational gaps seen on live trains, through the real + * bin/gstack-ship-queue against the train fixture: the screen verdict is read + * from the run (ready refuses a red head, verify --screen prints it, train + * build refuses the pin), a lagging GitHub merge ref gets a scratch merge + * (#3465), `train prune --stale` settles trains whose PRs merged elsewhere, + * and `train watch` resumes from the record after an interruption. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { cleanup, runBin } from './helpers/restamp-fixture'; +import { + GREEN_GATE, TRAIN_POLICY, dispatches, line, nonceOf, openPr, queue, readyReceipt, repoKey, trainJson, trainFixture, type Rule, type TrainFixture, +} from './helpers/ship-queue-train-fixture'; + +const fixtures: TrainFixture[] = []; +const make = (policy: Record = TRAIN_POLICY) => { const f = trainFixture(policy); fixtures.push(f); return f; }; +afterEach(() => { while (fixtures.length) cleanup(fixtures.pop()!.repo); }); + +const next = (out: string) => line(out, 'NEXT:'); +const result = (out: string) => line(out, 'GSTACK_RESULT:'); + +function readyFixture() { + const fx = make(); + const head = openPr(fx, '1'); + fx.write(d => { d.ci[head] = [{ status: 'completed', conclusion: 'success' }]; }); + const tree = runBin(fx.repo, 'gstack-wtree', []).out.trim(); + const pregate = path.join(fx.repo.root, 'pregate.json'); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + return { fx, head, pregate }; +} +const receiptOf = (fx: TrainFixture, pr: string, head: string) => JSON.parse(fs.readFileSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `${pr}-${head}.json`), 'utf8')); + +const RED_SCREEN: Rule = { status: 'completed', conclusion: 'failure', plan: { planned: 3 }, binding: { verdict: 'RED', flaky: ['c-flaky'] }, outcomes: { red: ['judge-voice-tone'], flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } }; +const GREEN_SCREEN: Rule = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN', flaky: ['c-flaky'] }, outcomes: { flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } }; + +describe('the screen verdict comes from the run', () => { + // Value: protects=a RED screen run (runs 40269992077 and 40275443393 on 2026-10-26 were VERDICT RED while the lane reported green) can never be READY'd, followed or seated; + // fails_when=ready or train build trust the lane's choice without reading the run's gate binding; why_new=no test read a screen's verdict; seam=the gh shim's gate-binding and report-verdict artifacts; no_claim=evals.yml's real upload step + test('a RED screen refuses ready under either choice and is never re-dispatched; verify --screen prints the red and flaky cases; train build refuses the pin', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = RED_SCREEN; }); + const first = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([first.status, line(first.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, expect.stringMatching(/^SCREEN_DISPATCH: evals\.yml run 100 /)]); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(again.status).toBe(EXIT.refused); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)'); + expect(line(again.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'); + expect(again.err).toMatch(/the screen recorded for [0-9a-f]{12} \(run 100\) is RED on judge-voice-tone; this head is not READY under any screen choice; fix: .*\(QUEUE_SCREEN_RED\)/); + expect(next(again.out)).toBe('NEXT: repair, push a new head, run /ship, then gstack-ship-queue ready --pr 1 --screen'); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'RED', run: 100, red: ['judge-voice-tone'], flaky: ['c-flaky'] }); + const noScreen = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']); + expect([noScreen.status, noScreen.err]).toEqual([EXIT.refused, expect.stringContaining('(QUEUE_SCREEN_RED)')]); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(verify.status).toBe(EXIT.fail); + expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)'); + expect(verify.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=pr\.screen\.narrow-\S+ run=100 verdict=RED$/); + expect(result(verify.out)).toMatch(/status=refused run=.*1-/); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect(build.status).toBe(EXIT.refused); + expect(build.err).toMatch(/#1's screen run 100 is RED on judge-voice-tone .*a red head is not seated.*\(QUEUE_SCREEN_RED\)/); + expect(next(build.out)).toBe('NEXT: none'); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'trains'))).toBe(false); + }, 120_000); + + test('a GREEN screen with flaky cases is followed, recorded and printed by verify --screen and train build; a run that bound no verdict is re-dispatched', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(again.status).toBe(EXIT.ok); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 GREEN flaky=c-flaky (verdict GREEN, reconcile exit 0, run concluded success, flaky 1)'); + expect(line(again.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)'); + expect(again.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=\S+ run=100 verdict=GREEN$/); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'GREEN', run: 100, red: [], flaky: ['c-flaky'] }); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([verify.status, verify.err]).toEqual([EXIT.ok, '']); + expect(line(verify.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'); + expect(next(verify.out)).toBe(`NEXT: gstack-ship-queue train build --prs 1@${head},...`); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect([build.status, line(build.out, 'SCREEN:'), line(build.out, 'ORDER:')]).toEqual([EXIT.ok, 'SCREEN: #1 run 100 GREEN (flaky: c-flaky)', 'ORDER: 1:#1(screened)']); + fx.write(d => { delete d.artifacts['100']['gate-binding-a1']; }); + const unbound = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(unbound.status).toBe(EXIT.refused); + expect(unbound.err).toMatch(/has no verdict to read: run 100 concluded success and bound no verdict \(no gate-binding artifact.*\(QUEUE_PROOF_MISSING\)/); + expect(next(unbound.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)'); + const redispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(redispatch.status).toBe(EXIT.ok); + expect(line(redispatch.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 101 .*\(not following run 100 concluded success and bound no verdict .*: a screen that bound no verdict is not a screen\)$/); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, reuse: { run: 77, flaky: ['c-reused'] } }; }); + const reusedDispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--broad', '--screen']); + expect(reusedDispatch.status).toBe(EXIT.ok); + const reused = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([reused.status, line(reused.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #1 run 102 GREEN flaky=c-reused (reused the GREEN verdict of run 77 (main-moved): same own diff on a moved main, flaky 1)']); + const noRun = queue(fx, ['verify', '--pr', '2', '--screen']); + expect(noRun.err).toMatch(/cannot read PR #2's head through gh.*\(QUEUE_PROOF_MISSING\)/); + const unscreened = openPr(fx, '3'); + readyReceipt(fx, '3', unscreened); + const none = queue(fx, ['verify', '--pr', '3', '--screen']); + expect([none.status, line(none.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #3 none (READY --no-screen)']); + }, 120_000); + + // Value: protects=a screen that concluded success with no slice (18 of 41 PR screens on 2026-10-26, superseded by a push before the plan) is NOT-RUN at verdict, not a screen and not a stale ref; + // fails_when=verify --screen calls it GREEN, HOLLOW or QUEUE_SCREEN_STALE_REF, or ready follows it; why_new=the hollow outcome reused the stale-ref code; seam=the gh shim's run with no plan artifact; no_claim=why evals.yml planned nothing + // Value: protects=train build --dry-run (with or without --stack-tests) writes nothing: the screen verdict it reads is printed, not recorded in the READY receipt or pushed to refs/notes/gstack-trains, as its DRY-RUN line says; + // fails_when=admitSlots persists the read verdict under --dry-run (the notes ref moves and the receipt gains a verdict before any real build); why_new=the other dry-run tests only check the trains dir; seam=none; no_claim=a verdict that was already recorded by ready or verify + test('train build --dry-run prints the screen verdict it read without recording it in the receipt or pushing the notes ref', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + expect(receiptOf(fx, '1', head).screen.verdict).toBeUndefined(); + const notes = fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains'); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect([build.status, line(build.out, 'SCREEN:'), line(build.out, 'DRY-RUN:')]).toEqual([EXIT.ok, 'SCREEN: #1 run 100 GREEN (flaky: c-flaky)', 'DRY-RUN: nothing written, pushed or dispatched']); + expect(receiptOf(fx, '1', head).screen.verdict).toBeUndefined(); + expect(fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains')).toBe(notes); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([verify.status, receiptOf(fx, '1', head).screen.verdict?.outcome]).toEqual([EXIT.ok, 'GREEN']); + expect(fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains')).not.toBe(notes); + }, 120_000); + + test('a screen run that concluded success with no slice is NOT-RUN: verify --screen refuses with QUEUE_SCREEN_NOT_RUN, the receipt records NOT_RUN, and ready --screen dispatches afresh', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(verify.status).toBe(EXIT.refused); + expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))'); + expect(verify.err).toMatch(/the screen of #1 \(run 100\) concluded success with no slice: run 100 planned no cases .*; it is not a screen, and this head is not screened; fix: .*\(QUEUE_SCREEN_NOT_RUN\)/); + expect(next(verify.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)'); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'NOT_RUN', run: 100 }); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.ok, 2]); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))'); + expect(line(again.out, 'SCREEN_DISPATCH:')).toMatch(/not following run 100 planned no cases .*: a screen that planned nothing is not a screen\)$/); + }, 120_000); +}); + +describe('#3465: the screen is planned on a scratch merge when GitHub\'s merge ref lags', () => { + // Value: protects=a CHANGELOG-only conflict (every main stamp leaves one) no longer blocks --screen: the screen tests the head merged onto current main by the train's rules; + // fails_when=ready still refuses on the lagging ref, pushes nothing, or the scratch merge's parents are not (base, head); why_new=the stale-ref test only asserted the refusal; seam=none; no_claim=evals.yml's checkout of the pushed branch + test('a lagging or missing merge ref gets a scratch merge pushed as screen/-; a real conflict is QUEUE_SCREEN_STALE_REF; prune --stale removes the branch once the PR merged', () => { + const { fx, head, pregate } = readyFixture(); + const { repo } = fx; + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + repo.git('checkout', '-q', '-B', 'mainsim', 'origin/main'); + const cl = fs.readFileSync(path.join(repo.work, 'CHANGELOG.md'), 'utf8'); + fs.writeFileSync(path.join(repo.work, 'CHANGELOG.md'), cl.replace('# Changelog\n', '# Changelog\n\n## [1.2.4.0] - 2026-10-18\n\n- another PR, stamped\n')); + fs.writeFileSync(path.join(repo.work, 'VERSION'), '1.2.4.0\n'); + repo.git('commit', '-qam', 'v1.2.4.0 stamp'); + repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main'); + repo.git('checkout', '-q', 'pr-1'); + const dry = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen', '--dry-run']); + expect([dry.status, dry.err]).toEqual([EXIT.ok, '']); + const main = repo.git('rev-parse', 'origin/main'); + expect(line(dry.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${head.slice(0, 12)} onto main@${main.slice(0, 12)} \\(refs/pull/1/merge [0-9a-f]{12} merges onto [0-9a-f]{12}, not main@${main.slice(0, 12)}\\); would push screen/1-${head.slice(0, 12)} \\(dry run\\)$`)); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); + const real = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([real.status, real.err]).toEqual([EXIT.ok, '']); + const branch = `screen/1-${head.slice(0, 12)}`; + expect(line(real.out, 'SCREEN_MERGE:')).toContain(`; pushed ${branch}`); + const scratch = repo.git('ls-remote', 'origin', `refs/heads/${branch}`).split(/\s/)[0]!; + expect(scratch).toMatch(/^[0-9a-f]{40}$/); + repo.git('fetch', '-q', 'origin', `refs/heads/${branch}`); + expect([repo.git('rev-parse', `${scratch}^1`), repo.git('rev-parse', `${scratch}^2`)]).toEqual([repo.git('rev-parse', 'origin/main'), head]); + expect(repo.git('show', `${scratch}:VERSION`)).toBe('1.2.4.0'); + expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- #1 change'); + expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- another PR, stamped'); + const d = dispatches(fx, 'evals.yml'); + expect(d.length).toBe(1); + expect((d[0]!.inputs as any)).toMatchObject({ screen_merge_sha: scratch, expected_sha: scratch, screen_pr: '1' }); + expect(receiptOf(fx, '1', head).screen).toMatchObject({ choice: 'screen', merge_sha: scratch, branch }); + repo.git('push', '-q', 'origin', ':refs/pull/1/merge'); + const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([missing.status, line(missing.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, 'SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)']); + expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge)'); + const kept = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(kept.out, 'KEEP:')).toBe(`KEEP: ${branch} (#1 OPEN)`); + fx.write(data => { (data.prs['1'] as any).state = 'MERGED'; }); + const pruned = queue(fx, ['train', 'prune', '--stale']); + expect(line(pruned.out, 'DELETED:')).toBe(`DELETED: ${branch} (#1 MERGED)`); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); + fx.write(data => { (data.prs['1'] as any).state = 'OPEN'; }); + repo.git('checkout', '-q', '-B', 'mainsim2', 'origin/main'); + fs.writeFileSync(path.join(repo.work, 'pr-1.txt'), 'main added the same file\n'); + repo.git('add', 'pr-1.txt'); + repo.git('commit', '-qm', 'main adds pr-1.txt'); + repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main'); + repo.git('checkout', '-q', 'pr-1'); + const conflict = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(conflict.status).toBe(EXIT.refused); + expect(conflict.err).toMatch(/#1 conflicts with main@[0-9a-f]{12} beyond the CHANGELOG and the stamp files \(pr-1\.txt\); GitHub's merge ref lags too \(origin has no refs\/pull\/1\/merge\).*\(QUEUE_SCREEN_STALE_REF\)/); + expect(next(conflict.out)).toBe('NEXT: resolve the conflict with main, push, run /ship, then gstack-ship-queue ready --pr 1 --screen'); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + }, 120_000); +}); + +describe('train prune --stale and train watch', () => { + function greenTrain(fx: TrainFixture, gate: Rule = GREEN_GATE) { + const h1 = openPr(fx, '1'); + const h2 = openPr(fx, '2'); + readyReceipt(fx, '1', h1); + readyReceipt(fx, '2', h2); + fx.write(d => { d.rules['evals.yml'] = gate; }); + const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2}`, '--free-timeout', '1']); + return { h1, h2, r, nonce: nonceOf(r.out) }; + } + + // Value: protects=a merging or green train whose PRs all merged elsewhere (gate.train-23af94b3c5e9 and -6755ab829acf on 2026-10-26) can be pruned without hand-editing refs/notes/gstack-trains; + // fails_when=prune skips merging/green trains, prunes one with an OPEN PR, or prunes without --stale; why_new=prune only knew terminal states; seam=the gh shim's PR states; no_claim=GitHub's real PR state + test('--stale settles a merging or green train whose every PR is MERGED or CLOSED on GitHub, logging each PR; an OPEN PR keeps it; without --stale nothing changes', () => { + const fx = make(); + const { nonce } = greenTrain(fx); + expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok); + expect(trainJson(fx, nonce).state).toBe('green'); + const plain = queue(fx, ['train', 'prune']); + expect(line(plain.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green)`); + const open = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(open.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #1 OPEN, #2 OPEN still need the train)`); + fx.write(d => { (d.prs['1'] as any).state = 'MERGED'; }); + const half = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(half.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #2 OPEN still needs the train)`); + fx.write(d => { (d.prs['2'] as any).state = 'CLOSED'; }); + const dry = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(dry.out, 'STALE:')).toBe(`STALE: ${nonce} (green): #1 MERGED, #2 CLOSED; would mark stale`); + expect(line(dry.out, 'WOULD-DELETE:')).toBe(`WOULD-DELETE: train/${nonce} (stale)`); + expect(trainJson(fx, nonce).state).toBe('green'); + const pruned = queue(fx, ['train', 'prune', '--stale']); + expect([pruned.status, line(pruned.out, 'STALE:'), line(pruned.out, 'DELETED:')]).toEqual([EXIT.ok, `STALE: ${nonce} (green): #1 MERGED, #2 CLOSED`, `DELETED: train/${nonce} (stale)`]); + const t = trainJson(fx, nonce); + expect(t.state).toBe('stale'); + expect(t.effects.map((e: any) => e.what)).toContain('marked stale by train prune --stale: #1 MERGED, #2 CLOSED'); + expect(fx.repo.git('ls-remote', 'origin', `refs/heads/train/${nonce}`)).toBe(''); + const status = queue(fx, ['train', 'status']); + expect(status.out).toContain('TRAIN: none live'); + }, 120_000); + + // Value: protects=a stale train record or READY receipt leaves refs/notes/gstack-trains through one command under an exact lease, with the local files, instead of a hand-run git notes remove on every machine; + // fails_when=forget removes the wrong notes, deletes locally when the push failed, or forgets a merging train; why_new=no CLI removed notes (merge-train skill, known gaps); seam=none; no_claim=a concurrent writer's race (the lease is exercised only as a fast-forward) + test('train forget removes a settled train (attempts included) or a READY receipt from the notes ref and the state root, leaves other records alone, keeps everything (the local notes ref too) when the lease is rejected, and refuses a train still building, running or merging', () => { + const fx = make(); + const { nonce, h1 } = greenTrain(fx); + expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok); + const trainDir = path.join(fx.state, 'ship-queue', 'trains', nonce); + expect(fs.existsSync(path.join(trainDir, 'attempts', 'a1.json'))).toBe(true); + const dry = queue(fx, ['train', 'forget', '--train', nonce, '--dry-run']); + expect([dry.status, line(dry.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: train ${nonce} (green; 1 attempt(s))`]); + expect(dry.out).toContain(` note gstack-trains:train:${nonce}`); + expect(dry.out).toContain(` note gstack-trains:attempt:${nonce}:a1`); + expect(fs.existsSync(trainDir)).toBe(true); + const notesBefore = fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains').split(/\s/)[0]!; + const forgot = queue(fx, ['train', 'forget', '--train', nonce]); + expect([forgot.status, forgot.err]).toEqual([EXIT.ok, '']); + expect(line(forgot.out, 'FORGOTTEN:')).toMatch(new RegExp(`^FORGOTTEN: 2 note\\(s\\) removed; notes pushed to origin refs/notes/gstack-trains \\(lease ${notesBefore.slice(0, 12)}\\); 1 local path\\(s\\) deleted$`)); + expect(fs.existsSync(trainDir)).toBe(false); + const clean = path.join(fx.repo.root, 'clean-state'); + const status = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: clean }); + expect(status.out).toContain('TRAIN: none live'); + expect(fs.existsSync(path.join(clean, 'ship-queue', 'trains', nonce))).toBe(false); + expect(fs.existsSync(path.join(clean, 'ship-queue', 'ready', repoKey(fx), `1-${h1}.json`))).toBe(false); + const again = queue(fx, ['train', 'forget', '--train', nonce]); + expect(again.err).toMatch(new RegExp(`no train record ${nonce} here or in refs/notes/gstack-trains.*\\(QUEUE_PROOF_MISSING\\)`)); + const h3 = openPr(fx, '3'); + fx.write(d => { d.ci[h3] = [{ status: 'completed', conclusion: 'success' }]; }); + const pregate = path.join(fx.repo.root, 'pregate3.json'); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h3, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + expect(queue(fx, ['ready', '--pr', '3', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok); + const h4 = openPr(fx, '4'); + fx.write(d => { d.ci[h4] = [{ status: 'completed', conclusion: 'success' }]; }); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h4, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + expect(queue(fx, ['ready', '--pr', '4', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok); + const receipt = queue(fx, ['train', 'forget', '--receipt', `3@${h3}`]); + expect([receipt.status, line(receipt.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: READY receipt #3 at ${h3.slice(0, 12)} (screen no-screen)`]); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false); + const rebuilt = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: path.join(fx.repo.root, 'clean-state-2') }); + expect(rebuilt.status).toBe(EXIT.ok); + expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `4-${h4}.json`))).toBe(true); + expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false); + expect(queue(fx, ['train', 'forget', '--receipt', '3@abc']).err).toMatch(/--receipt takes @.*\(QUEUE_PROOF_MISSING\)/); + expect(queue(fx, ['train', 'forget']).status).toBe(EXIT.usage); + const { nonce: second } = (() => { readyReceipt(fx, '3', h3); readyReceipt(fx, '4', h4); const r = queue(fx, ['train', 'build', '--prs', `3@${h3},4@${h4}`, '--free-timeout', '1']); return { nonce: nonceOf(r.out) }; })(); + expect(queue(fx, ['train', 'verdict', '--train', second]).status).toBe(EXIT.ok); + const hook = path.join(fx.repo.origin, 'hooks', 'pre-receive'); + fs.mkdirSync(path.dirname(hook), { recursive: true }); + fs.writeFileSync(hook, '#!/bin/sh\nwhile read o n r; do case "$r" in refs/notes/*) echo "notes frozen" >&2; exit 1;; esac; done\nexit 0\n', { mode: 0o755 }); + const localNotes = fx.repo.git('rev-parse', 'refs/notes/gstack-trains'); + const rejected = queue(fx, ['train', 'forget', '--train', second]); + expect(rejected.status).toBe(EXIT.refused); + expect(rejected.err).toMatch(/the notes ref was not updated \(git push refs\/notes\/gstack-trains rejected: .*notes frozen.*; the local notes ref was restored\); nothing deleted locally.*\(QUEUE_PROOF_MISSING\)/); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'))).toBe(true); + expect(fx.repo.git('rev-parse', 'refs/notes/gstack-trains')).toBe(localNotes); + expect(fx.repo.git('notes', '--ref=refs/notes/gstack-trains', 'list').split('\n').length).toBeGreaterThan(0); + fs.rmSync(hook); + const trainFile = path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'); + for (const state of ['dispatch-unknown', 'merging']) { + const t = JSON.parse(fs.readFileSync(trainFile, 'utf8')); + t.state = state; + fs.writeFileSync(trainFile, JSON.stringify(t)); + const live = queue(fx, ['train', 'forget', '--train', second]); + expect(live.status).toBe(EXIT.refused); + expect(live.err).toMatch(new RegExp(`is ${state}; forget settles nothing that is still building, running or merging .*\\(QUEUE_PROOF_MISSING\\)`)); + expect(fs.existsSync(trainFile)).toBe(true); + } + }, 120_000); + + // Value: protects=a restarted controller re-arms with one command: watch continues from the record (free tests, gate dispatch, the pending run, the verdict) and prints NEXT; + // fails_when=watch dispatches twice, blocks past --timeout, silently replaces a malformed --free-timeout with the default, or does not reach the verdict after a restart; why_new=no resumable follow existed (hand-run gh run watch chains died with the machine); seam=GSTACK_SHIP_QUEUE_POLL_MS; no_claim=real GitHub run timing + test('watch resumes from free-pending through the gate dispatch to the verdict, times out with its own NEXT, and never dispatches twice', () => { + const fx = make(); + fx.write(d => { d.rules['free-tests.yml'] = { status: 'in_progress', conclusion: null }; }); + const { nonce, r } = greenTrain(fx, { ...GREEN_GATE, status: 'in_progress', conclusion: null }); + expect(r.status).toBe(EXIT.refused); + expect(trainJson(fx, nonce).state).toBe('free-pending'); + const waiting = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect([waiting.status, result(waiting.out)]).toEqual([EXIT.refused, expect.stringContaining('status=incomplete')]); + expect(waiting.out).toContain(`WATCH: train ${nonce} state=free-pending (resumable; rerun this command after an interruption)`); + expect(next(waiting.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`); + expect(dispatches(fx, 'evals.yml')).toEqual([]); + fx.write(d => { for (const run of d.runs) if (run.workflow === 'free-tests.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); }); + const gate = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect(gate.status).toBe(EXIT.refused); + expect(gate.out).toContain(`WATCH: train build --train ${nonce}: free tests green, gate dispatched`); + expect(gate.err).toMatch(/is still in_progress at the watch timeout.*\(QUEUE_TRAIN_PENDING\)/); + expect(next(gate.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + expect(trainJson(fx, nonce).state).toBe('gate-pending'); + const again = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.refused, 1]); + fx.write(d => { for (const run of d.runs) if (run.workflow === 'evals.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); }); + const done = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '5']); + expect([done.status, done.err]).toEqual([EXIT.ok, '']); + expect(done.out).toContain('WATCH: gate run completed; reading the verdict'); + expect(line(done.out, 'VERDICT:')).toMatch(new RegExp(`^VERDICT: GREEN train ${nonce} run 10[0-9] `)); + expect(next(done.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`); + const idle = queue(fx, ['train', 'watch', '--train', nonce]); + expect([idle.status, line(idle.out, 'WATCH: nothing')]).toEqual([EXIT.ok, 'WATCH: nothing to wait for in state green']); + expect(next(idle.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`); + expect(queue(fx, ['train', 'watch']).status).toBe(EXIT.usage); + expect(queue(fx, ['train', 'watch', '--train', nonce, '--free-timeout', 'abc']).status).toBe(EXIT.usage); + expect(queue(fx, ['train', 'watch', '--train', 'no-such-train']).err).toMatch(/no train record no-such-train.*\(QUEUE_PROOF_MISSING\)/); + }, 120_000); +}); + +describe('verify --post-merge', () => { + // Value: protects=after a train merges, the coordinator watches main's push workflows at the merged sha until they settle instead of a background gh chain a restart kills (GBrain practice 8); + // fails_when=a pending check is called settled, a red check is called green, a pending or failed commit status (the legacy status API, which ciAtHead counts) is reported as red with no name or as green, or the command needs process state; why_new=no post-merge watch existed; seam=the gh shim's check runs and commit statuses at a sha; no_claim=which workflows main runs + test('waits for every check run at the sha to complete, prints each, is green only when none failed, red with the failed names, pending or red on a commit status, and resumable after a timeout', () => { + const fx = make(); + const sha = fx.repo.git('rev-parse', 'HEAD'); + fx.write(d => { d.ci[sha] = [{ name: 'free-tests', status: 'completed', conclusion: 'success' }, { name: 'windows-free-tests', status: 'in_progress', conclusion: null }, { name: 'main-backstop', status: 'completed', conclusion: 'neutral' }]; }); + const pending = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect(pending.status).toBe(EXIT.refused); + expect(line(pending.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} pending (passed=1 failed=0 pending=1 skipped=1)`); + expect(pending.out).toContain('CHECK: windows-free-tests in_progress'); + expect(pending.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} have not settled .*windows-free-tests.*\\(QUEUE_TRAIN_PENDING\\)`)); + expect(next(pending.out)).toBe(`NEXT: gstack-ship-queue verify --post-merge ${sha} --timeout 1`); + fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'failure' }; }); + const red = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect(red.status).toBe(EXIT.fail); + expect(line(red.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} red (passed=1 failed=1 pending=0 skipped=1)`); + expect(red.out).toContain('CHECK: windows-free-tests failure'); + expect(red.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} are red: windows-free-tests.*\\(QUEUE_POST_MERGE_RED\\)`)); + expect(next(red.out)).toBe('NEXT: read the red run, repair on a new PR or roll back; gstack-ship-queue train status names the freeze if the backstop filed one'); + fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'success' }; }); + const green = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([green.status, green.err]).toEqual([EXIT.ok, '']); + expect(line(green.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} green (passed=2 failed=0 pending=0 skipped=1)`); + expect(green.out.split('\n').filter(l => l.startsWith('CHECK: ')).sort()).toEqual(['CHECK: free-tests success', 'CHECK: main-backstop neutral', 'CHECK: windows-free-tests success']); + expect(next(green.out)).toBe('NEXT: none'); + fx.write(d => { d.statuses = { [sha]: [{ state: 'pending', context: 'ci/vendor' }] }; }); + const statusPending = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([statusPending.status, line(statusPending.out, 'POST_MERGE:')]).toEqual([EXIT.refused, `POST_MERGE: ${sha.slice(0, 12)} pending (passed=2 failed=0 pending=1 skipped=1)`]); + expect(statusPending.err).toMatch(/have not settled at the timeout: a commit status still running.*\(QUEUE_TRAIN_PENDING\)/); + fx.write(d => { d.statuses = { [sha]: [{ state: 'failure', context: 'ci/vendor' }] }; }); + const statusRed = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([statusRed.status, line(statusRed.out, 'POST_MERGE:')]).toEqual([EXIT.fail, `POST_MERGE: ${sha.slice(0, 12)} red (passed=2 failed=1 pending=0 skipped=1)`]); + expect(statusRed.err).toMatch(/are red: a commit status \(no check run failed\).*\(QUEUE_POST_MERGE_RED\)/); + fx.write(d => { d.statuses = {}; d.ci[sha] = []; }); + const none = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([none.status, line(none.out, 'POST_MERGE:')]).toEqual([EXIT.refused, `POST_MERGE: ${sha.slice(0, 12)} none (passed=0 failed=0 pending=0 skipped=0)`]); + expect(none.err).toMatch(new RegExp(`no check run has attached to ${sha.slice(0, 12)} yet.*\\(QUEUE_TRAIN_PENDING\\)`)); + expect(queue(fx, ['verify', '--post-merge', 'not-a-sha']).status).toBe(EXIT.usage); + }, 120_000); +}); + +describe('ready --dispatch-ci', () => { + // Value: protects=a PR that conflicts with the base (a CHANGELOG-only conflict after every stamp) gets its free workflows dispatched on the branch, waited for, and its requires-remote pregate lanes cleared with the run URLs in one command, replacing the four-step hand procedure; + // fails_when=the dispatch happens on a fork head, a red run clears a lane, a lane whose workflow was not dispatched is cleared, ready still says ci: none, or the CI_DISPATCH lines push the screen: line in front of READY (identifiers first); why_new=the ci-hint was advisory prose; seam=the gh shim's workflow run, run list and check runs; no_claim=GitHub's real dispatch latency + test('dispatches the named workflows on a conflicting PR, waits for their runs, clears the matching pregate lanes with the run URLs and then READYs; a fork head and a red run are refused', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { + (d.prs['1'] as any).mergeable = 'CONFLICTING'; + d.ci[head] = []; + d.rules['free-tests.yml'] = { ci: 'free-tests', headSha: head }; + d.rules['windows-free-tests.yml'] = { ci: 'windows-free-tests', headSha: head }; + }); + const tree = JSON.parse(fs.readFileSync(pregate, 'utf8')).tree; + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'requires-remote', checks: [{ id: 'unit', status: 'pass' }], requires_remote: [ + { lane: 'free-tests.yml/free-tests', platform: 'linux', workflow: 'free-tests.yml', job: 'free-tests', tests: ['test/a.test.ts'], receipt: null }, + { lane: 'windows-free-tests.yml/windows-free-tests', platform: 'windows', workflow: 'windows-free-tests.yml', job: 'windows-free-tests', tests: ['test/a.test.ts'], receipt: null }, + { lane: 'macos-free-tests.yml/macos', platform: 'macos', workflow: 'macos-free-tests.yml', job: 'macos', tests: ['test/b.test.ts'], receipt: null }] })); + const plain = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']); + expect([plain.status, line(plain.out, 'ci:')]).toEqual([EXIT.refused, `ci: none at ${head.slice(0, 12)} (passed=0 failed=0 pending=0 skipped=0)`]); + const r = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml,windows-free-tests.yml', '--timeout', '5']); + expect(r.out.split('\n').filter(l => l.startsWith('CI_DISPATCH:'))).toEqual([ + 'CI_DISPATCH: free-tests.yml --ref pr-1 run 100 https://example.invalid/runs/100 success', + 'CI_DISPATCH: windows-free-tests.yml --ref pr-1 run 101 https://example.invalid/runs/101 success']); + expect(r.out.split('\n').filter(l => l.startsWith('PREGATE_CLEARED:'))).toEqual([ + 'PREGATE_CLEARED: free-tests.yml/free-tests https://example.invalid/runs/100', + 'PREGATE_CLEARED: windows-free-tests.yml/windows-free-tests https://example.invalid/runs/101']); + expect(line(r.out, 'ci:')).toBe(`ci: green at ${head.slice(0, 12)} (passed=2 failed=0 pending=0 skipped=0)`); + expect(line(r.out, 'PREGATE:')).toMatch(/^PREGATE: requires-remote .*/); + expect(r.err).toMatch(/1 requires-remote obligation\(s\) without a receipt: macos-free-tests\.yml\/macos.*\(QUEUE_NOT_READY\)/); + const saved = JSON.parse(fs.readFileSync(pregate, 'utf8')); + expect(saved.requires_remote.map((l: any) => l.receipt)).toEqual(['https://example.invalid/runs/100', 'https://example.invalid/runs/101', null]); + expect(dispatches(fx).map(l => l.workflow)).toEqual(['free-tests.yml', 'windows-free-tests.yml']); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect(dispatches(fx).length).toBe(2); + expect(line(again.out, 'CI_DISPATCH:')).toBe('CI_DISPATCH: free-tests.yml not dispatched (run 100 already concluded success at this head)'); + const cleared = JSON.parse(fs.readFileSync(pregate, 'utf8')); + cleared.requires_remote[2].receipt = 'https://example.invalid/runs/7'; + fs.writeFileSync(pregate, JSON.stringify(cleared)); + const ok = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect([ok.status, ok.err]).toEqual([EXIT.ok, '']); + const okLines = ok.out.split('\n'); + expect(okLines[0]).toBe('CI_DISPATCH: free-tests.yml not dispatched (run 100 already concluded success at this head)'); + expect(okLines[okLines.findIndex(l => l.startsWith('READY #1 ')) + 1]).toBe('screen: no-screen'); + fx.write(d => { d.rules['free-tests.yml'] = { ci: 'free-tests', conclusion: 'failure', headSha: head }; d.ci[head] = []; d.runs.length = 0; }); + const red = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect([red.status, line(red.out, 'CI_DISPATCH:')]).toEqual([EXIT.refused, 'CI_DISPATCH: free-tests.yml --ref pr-1 run 102 https://example.invalid/runs/102 failure']); + expect(red.err).toMatch(new RegExp(`free-tests\\.yml run 102 concluded failure at ${head.slice(0, 12)}; a red run clears no lane.*\\(QUEUE_NOT_READY\\)`)); + expect(JSON.parse(fs.readFileSync(pregate, 'utf8')).requires_remote[0].receipt).toBe('https://example.invalid/runs/100'); + fx.write(d => { (d.prs['1'] as any).isCrossRepository = true; }); + const fork = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml']); + expect([fork.status, fork.err]).toEqual([EXIT.refused, expect.stringMatching(/#1's head is on a fork.*not dispatched.*\(QUEUE_TRAIN_UNTRUSTED\)/)]); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', '../evil.yml']).status).toBe(EXIT.usage); + }, 120_000); +}); + +describe('train build --dry-run --stack-tests', () => { + // Value: protects=a PR whose new tests pass on its own branch but fail once stacked behind another PR's change is caught before the train spends a gate (#3460 on #3451, 2026-10-26); + // fails_when=the new test files run against the PR head instead of the integrated prefix, a failing prefix still dry-runs green, or a PR without new tests (a helper under test/ included: bun exits 1 when a positional filter matches no test file) is misreported; why_new=no pre-dispatch stack check existed; seam=bun test in a scratch worktree; no_claim=the repo's full suite on the prefix + test('runs each PR\'s new test files on its integrated prefix T: a pass and a helper-only PR are reported, a failure refuses the dry run and names the prefix, and the ordinary dry run skips it', () => { + const fx = make(); + const h1 = openPr(fx, '1', { file: 'test/pr-1_test.ts', text: "import { expect, test } from 'bun:test';\ntest('pr-1', () => { expect(1).toBe(1); });\n" }); + const h2 = openPr(fx, '2', { file: 'test/pr-2.test.ts', text: "import { existsSync } from 'node:fs';\nimport { expect, test } from 'bun:test';\ntest('pr-2 wants no pr-1 test beside it', () => { expect(existsSync('test/pr-1_test.ts')).toBe(false); });\n" }); + const h3 = openPr(fx, '3', { file: 'test/helpers/pr-3-helper.ts', text: 'export const three = 3;\n' }); + readyReceipt(fx, '1', h1); + readyReceipt(fx, '2', h2); + readyReceipt(fx, '3', h3); + const plain = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run']); + expect([plain.status, plain.out.includes('STACK_TESTS:')]).toEqual([EXIT.ok, false]); + const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run', '--stack-tests']); + expect(r.status).toBe(EXIT.fail); + expect(r.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([ + 'STACK_TESTS: #1 T1 1 new test file(s) pass (test/pr-1_test.ts)', + expect.stringMatching(/^STACK_TESTS: #2 T2 1 new test file\(s\) FAIL \(test\/pr-2\.test\.ts\): .*\(fail\) pr-2 wants no pr-1 test beside it.*1 fail$/), + 'STACK_TESTS: #3 T3 no new test files']); + expect(r.err).toMatch(/#2's new tests fail on the integrated prefix T2 \(test\/pr-2\.test\.ts\); they pass on the PR head alone only if nothing before it in the train changes what they read.*\(QUEUE_TRAIN_STACK_RED\)/); + expect(next(r.out)).toBe(`NEXT: fix #2's tests against T2, or reorder with --order, then gstack-ship-queue train build --prs 1@${h1},2@${h2},3@${h3} --dry-run --stack-tests`); + expect(line(r.out, 'DRY-RUN:')).toBe('DRY-RUN: nothing written, pushed or dispatched'); + const reordered = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--order', '2,1,3', '--dry-run', '--stack-tests']); + expect(reordered.status).toBe(EXIT.ok); + expect(reordered.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([ + 'STACK_TESTS: #2 T1 1 new test file(s) pass (test/pr-2.test.ts)', + 'STACK_TESTS: #1 T2 1 new test file(s) pass (test/pr-1_test.ts)', + 'STACK_TESTS: #3 T3 no new test files']); + expect(fx.repo.git('worktree', 'list', '--porcelain')).not.toContain('gstack-stack-tests-'); + expect(queue(fx, ['train', 'build', '--prs', `1@${h1}`, '--stack-tests']).status).toBe(EXIT.usage); + }, 180_000); +}); diff --git a/test/ship-queue-train.test.ts b/test/ship-queue-train.test.ts index 97ae85475..318a25657 100644 --- a/test/ship-queue-train.test.ts +++ b/test/ship-queue-train.test.ts @@ -196,9 +196,9 @@ describe('ready under a train policy: the screen choice', () => { expect(queue(fx, ['ready', '--pr', '1', '--broad']).status).toBe(EXIT.usage); }, 120_000); - // Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success); - // fails_when=the merge ref's second parent is not compared with the PR head, or a missing merge ref keeps the old QUEUE_NOT_READY code; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in) - test('--screen refuses QUEUE_SCREEN_STALE_REF when refs/pull//merge lags the PR head or is missing, dispatching nothing; a recomputed merge ref admits it', () => { + // Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success): a lagging, missing or unfetchable ref is replaced by a scratch merge of the head onto current main (#3465), and an unreachable remote is still refused; + // fails_when=the merge ref's parents are not compared with the PR head and current main, or the scratch merge is skipped; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in) + test('--screen plans the screen on a scratch merge when refs/pull//merge lags the PR head, is missing or cannot be fetched; an unreachable remote is POLICY_SOURCE_UNAVAILABLE; a recomputed merge ref is used as is', () => { const { fx, head, pregate } = readyFixture(); const { repo } = fx; fs.appendFileSync(path.join(repo.work, 'pr-1.txt'), 'pushed after the merge ref was computed\n'); @@ -208,41 +208,41 @@ describe('ready under a train policy: the screen choice', () => { fx.write(d => { (d.prs['1'] as any).headRefOid = moved; (d.prs['1'] as any).body = shippedBody('1', moved); d.ci[moved] = [{ status: 'completed', conclusion: 'success' }]; }); const pregate2 = path.join(repo.root, 'pregate2.json'); fs.writeFileSync(pregate2, JSON.stringify({ schema_version: 1, tree: runBin(repo, 'gstack-wtree', []).out.trim(), head: moved, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); - const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(stale.status).toBe(EXIT.refused); - expect(stale.err).toMatch(new RegExp(`refs/pull/1/merge \\([0-9a-f]{12}\\) merges ${head.slice(0, 12)}, not the PR head ${moved.slice(0, 12)}.*\\(QUEUE_SCREEN_STALE_REF\\)`)); - expect(line(stale.out, 'NEXT:')).toBe('NEXT: resolve the conflict with main (or wait for GitHub to recompute refs/pull/1/merge), push, then gstack-ship-queue ready --pr 1 --screen; a screen dispatched on a stale merge ref plans no cases'); + const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([stale.status, stale.err]).toEqual([EXIT.ok, '']); + expect(line(stale.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${moved.slice(0, 12)} onto main@[0-9a-f]{12} \\(refs/pull/1/merge [0-9a-f]{12} merges ${head.slice(0, 12)}, not ${moved.slice(0, 12)}\\); would push screen/1-${moved.slice(0, 12)} \\(dry run\\)$`)); repo.git('push', '-q', 'origin', ':refs/pull/1/merge'); - const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(missing.status).toBe(EXIT.refused); - expect(missing.err).toMatch(/has no refs\/pull\/1\/merge.*\(QUEUE_SCREEN_STALE_REF\)/); + const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([missing.status, missing.err]).toEqual([EXIT.ok, '']); + expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge); would push'); const tree = repo.git('rev-parse', `${moved}^{tree}`); const main = repo.git('rev-parse', 'origin/main'); const badMerge = spawnSync('git', ['-C', repo.origin, 'hash-object', '-t', 'commit', '-w', '--stdin', '--literally'], { input: `tree ${tree}\nparent ${main}\nparent ${moved}\nauthor A 1700000000 +0000\ncommitter B 1700000000 +0000\n\nmerge without a committer email\n`, encoding: 'utf8', timeout: 30_000 }).stdout.trim(); spawnSync('git', ['-C', repo.origin, 'update-ref', 'refs/pull/1/merge', badMerge], { timeout: 30_000 }); repo.git('config', 'fetch.fsckObjects', 'true'); - const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(unfetchable.status).toBe(EXIT.refused); - expect(unfetchable.err).toMatch(/could not fetch refs\/pull\/1\/merge from origin: .*\(QUEUE_SCREEN_STALE_REF\)/); - expect(line(unfetchable.out, 'NEXT:')).toBe('NEXT: check the remote, then gstack-ship-queue ready --pr 1 --screen'); + const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([unfetchable.status, unfetchable.err]).toEqual([EXIT.ok, '']); + expect(line(unfetchable.out, 'SCREEN_MERGE:')).toMatch(/\(refs\/pull\/1\/merge [0-9a-f]{12} could not be fetched\); would push/); repo.git('config', '--unset', 'fetch.fsckObjects'); const originUrl = repo.git('remote', 'get-url', 'origin'); repo.git('remote', 'set-url', 'origin', path.join(repo.root, 'no-such-origin.git')); const unlisted = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); repo.git('remote', 'set-url', 'origin', originUrl); expect(unlisted.status).toBe(EXIT.refused); - expect(unlisted.err).toMatch(/could not list origin's refs: .*\(QUEUE_SCREEN_STALE_REF\)/); + expect(unlisted.err).toMatch(/git fetch origin main failed: .*\(POLICY_SOURCE_UNAVAILABLE\)/); expect(unlisted.err.split('\n').filter(Boolean).length).toBe(1); expect(dispatches(fx)).toEqual([]); expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready'))).toBe(false); repo.git('push', '-q', '-f', 'origin', `${mergeRefCommit(fx, moved)}:refs/pull/1/merge`); const fresh = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); expect([fresh.status, fresh.err]).toEqual([EXIT.ok, '']); + expect(line(fresh.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: GitHub's refs/pull/1/merge [0-9a-f]{12} merges the head ${moved.slice(0, 12)} onto main@[0-9a-f]{12}$`)); expect(line(fresh.out, 'SCREEN_DISPATCH:')).toContain('(dry run: not dispatched)'); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); }, 120_000); // Value: protects=a finished screen that planned 0 cases never counts as a screen: ready dispatches a fresh one instead of adopting it and train build seats the slot unscreened with a SCREEN: line; - // fails_when=the census is not read (a hollow run is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in) + // fails_when=the census is not read (a not-run screen is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in) test('a screen run that planned 0 cases is not followed by ready and is counted as not-run by train build', () => { const { fx, head, pregate } = readyFixture(); fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; }); @@ -260,7 +260,7 @@ describe('ready under a train policy: the screen choice', () => { fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 0 } }; }); const emptyPlan = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); expect(line(emptyPlan.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 102 .*not following run 101 planned no cases/); - fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 } }; }); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN' } }; }); const live = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); expect(live.status).toBe(EXIT.ok); expect(line(live.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 103 .*not following run 102 planned no cases \(an empty plan manifest; concluded success\)/); @@ -269,10 +269,11 @@ describe('ready under a train policy: the screen choice', () => { expect(line(followed.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 103 (same merge SHA, base and policy; no duplicate)'); expect(dispatches(fx, 'evals.yml').length).toBe(4); const liveBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); - expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']); - fx.write(d => { d.rules['evals.yml'] = { status: 'in_progress', conclusion: null }; }); + expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 GREEN']); + fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'in_progress', conclusion: null }); }); const runningBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); - expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']); + expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 still running; counted as screened']); + fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'completed', conclusion: 'success' }); }); fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success' }; d.artifacts['103']['paid-plan'] = { reuse: 'confirmed' }; }); const unreadable = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); expect([unreadable.status, line(unreadable.out, 'ORDER:'), line(unreadable.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', "SCREEN: #1 run 103's paid-plan manifest.json is unreadable; counted as screened, unverified"]);