From f375b873de3c7935bfc3422aa10463fd7b660779 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Mon, 26 Oct 2026 19:48:59 -0700 Subject: [PATCH] v1.91.263.0 feat(ship-queue): screens read from the run, train watch/forget/prune --stale, verify --post-merge, ready --dispatch-ci, --stack-tests (#3484) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * feat(ship-queue): screens read from the run, train watch/forget/prune --stale, verify --post-merge, ready --dispatch-ci, --stack-tests Operating merge trains on 2026-10-26 showed the gaps this closes. A dispatched screen's verdict is read from the run's gate-binding and report-verdict artifacts (lib/ship-queue-screen.ts) and recorded in the READY receipt: a RED screen refuses ready under either choice and is never re-dispatched at the same merge SHA (QUEUE_SCREEN_RED); a run that concluded success with no slice is NOT-RUN (QUEUE_SCREEN_NOT_RUN), never followed, and counted as unscreened by train build. When refs/pull//merge lags or is missing the screen is planned on a scratch merge pushed as screen/- (#3465); QUEUE_SCREEN_STALE_REF means a real conflict only. train watch --train is the resumable follow command; train prune --stale settles a merging/green train whose PRs all merged or closed elsewhere; train forget removes one record with an exact lease on refs/notes/gstack-trains; verify --post-merge watches main's push workflows until they settle; ready --dispatch-ci dispatches the free workflows on a conflicting PR, waits, and clears the matching requires-remote pregate lanes with the run URLs; train build --dry-run --stack-tests runs each PR's new test files on its integrated prefix before anything is dispatched (QUEUE_TRAIN_STACK_RED). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Regenerate gstack-doctor-components.sh for the four new QUEUE_* result codes The committed Bash render was four case lines behind its generator, so test/doctor-components.test.ts failed at c75ec3a. Output of bun run gen:skill-docs; no hand edits. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Restore the exact stderr line count in the POLICY_SOURCE_UNAVAILABLE test RH-15: toBeGreaterThan(0) after a toMatch is vacuous; the refusal is one deterministic line, so toBe(1) holds and is restored. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(ship-policy): a lagging merge ref is a scratch merge, not QUEUE_SCREEN_STALE_REF Three passages still described the pre-scratch-merge refusal; they now match screenMerge: a missing or lagging refs/pull//merge gets a screen/- scratch merge, and QUEUE_SCREEN_STALE_REF is a real conflict with the base only. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * stack-tests: drop the dead detail clause and assert the worktree is gone r.detail === 'pass' can never hold when r.ok is false; the dry-run test now checks git worktree list instead of a state-dir name that was never written. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * ready --dispatch-ci: fail closed on fork or unreadable PRs, surface gh errors, keep screen: after READY dispatchCi now uses prInfo (a fork head or an unreadable PR is refused instead of dispatched), reads gh's error text when the workflow run call fails, and calls runsAt once. The screen: line is inserted after the READY line instead of at index 1, which with --dispatch-ci put it before READY and after the first CI_DISPATCH line. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * verify --post-merge: honour commit statuses in the settled and red verdicts settled and red were computed from check runs only while ci.green also counted commit statuses, so a pending status was called settled and a failed status was reported as 'red: ' with no name. Pending statuses now keep the verdict pending; a red status with no failed check run is named as such. The fixture gains a statuses knob to drive both. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * train watch: reject a malformed --free-timeout as usage instead of taking the default Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * train forget: restore the local notes ref on a rejected lease and refuse every in-flight state forgetNotes removed the note locally before the leased push, so a rejected lease left a local notes ref without the note that the next plain persist pushed without a lease. It now snapshots the ref and restores it on rejection, reporting failed removals separately. forget refused only merging/gate-pending/free-pending; built, pushed, free-green, dispatch-unknown and dispatch-absent are in flight too and are refused the same way. CHANGELOG and ship-policy say so. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * train build --dry-run: never persist or push the read screen verdict admitSlots recorded a verdict that differed from the receipt and pushed refs/notes/gstack-trains before the dryRun branch printed 'nothing written, pushed or dispatched'. It now takes dryRun and skips the persist, matching trainReady's guard. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * stack-tests: only bun test files count as new tests, passed as paths TEST_FILE matched every .ts under test/, so a PR whose only added file was a helper or fixture ran 'bun test test/helpers/x.ts', which exits 1 (the filter matches no test file) and was reported QUEUE_TRAIN_STACK_RED. A bare positional is also a substring filter that picks up unrelated files. The rule is now bun's own (.test, _test_, .spec, _spec_ in the name) and each file is passed as ./, which bun treats as the exact file. The fixture PR 3 now adds a helper and is still 'no new test files'. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * screen: page the run's artifact list before picking the newest binding One page of 100 artifacts can miss gate-binding/report-verdict on a broad or rerun screen (three artifacts per slice plus reruns), which read as 'bound no verdict' and bought a fresh screen. newestArtifact now follows page= until a short page. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * stack-tests: match bun's _test and _spec suffixes too Bun discovers a file whose name ends in .test, _test, .spec or _spec before the extension. The previous rule missed the underscore forms (b_test.ts, e_spec.ts), so a PR adding only such files got no stack check, and it matched names bun ignores. PR 1 in the fixture now adds test/pr-1_test.ts to prove the underscore form is run. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: ship-policy and troubleshooting match the shipped train commands Step 14.5 documentation audit: the stack-tests naming rule is bun's own (.test, _test, .spec, _spec); prune --stale deletes every merged or closed PR's screen/-* branch; verify --post-merge takes the full SHA and counts commit statuses; train watch --free-timeout and ready --dispatch-ci --timeout defaults; PENDING/UNKNOWN/INCOMPLETE screen verdicts; QUEUE_SCREEN_STALE_REF's Meaning no longer contradicts its Fix; one sentence each for the new QUEUE_NOT_READY, QUEUE_TRAIN_UNTRUSTED and QUEUE_TRAIN_PENDING triggers. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * changelog: prune --stale scope, post-merge commit statuses, STALE_REF vs POLICY_SOURCE_UNAVAILABLE Three details the documentation audit found wrong against the code; the entry stays under [Unreleased] (stamp-at-merge). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs: RED screens are not seated; QUEUE_PROOF_MISSING and POLICY_SOURCE_UNAVAILABLE name the train-command causes Step 14.5 re-audit: ship-policy.md says train build does not seat a RED screen and its NEXT names the build without that PR; troubleshooting.md lists train forget, train watch, verify --screen and the scratch-branch push under QUEUE_PROOF_MISSING, and the origin/ fetch under POLICY_SOURCE_UNAVAILABLE. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * test(host-conformance): delete the named scope.md in the C3 broken-link case A test repair the remote gate requires, not a review finding. The case deleted the first .md readdirSync listed under gstack-qa/sections, and on the remote-gate VM's ext4 that was a file setup --status does not count, so the case was deterministically red there (#3479) while green locally and on Actions. The same one-line change ships in #3471 (fd468eb); this copy carries no CHANGELOG line so the two merge cleanly. Refs #3479 Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * v1.91.263.0 stamp Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 25 ++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- bin/gstack-doctor-components.sh | 4 + bin/gstack-ship-queue | 12 +- docs/ship-policy.md | 118 +++++- docs/troubleshooting.md | 153 +++++++- lib/result-codes-release.ts | 20 + lib/ship-queue-dispatch-ci.ts | 80 ++++ lib/ship-queue-post-merge.ts | 45 +++ lib/ship-queue-screen.ts | 114 ++++++ lib/ship-queue-stack-tests.ts | 63 +++ lib/ship-queue-train-cli.ts | 67 +++- lib/ship-queue-train-forget.ts | 56 +++ lib/ship-queue-train-github.ts | 11 +- lib/ship-queue-train-ready.ts | 165 ++++++-- lib/ship-queue-train-record.ts | 46 ++- lib/ship-queue-train-watch.ts | 68 ++++ lib/ship-queue-train.ts | 121 ++++-- lib/ship-queue.ts | 11 +- package.json | 2 +- scripts/lib/windows-curation.ts | 4 + test/helpers/ship-queue-train-fixture.ts | 23 +- test/ship-queue-train-ops.test.ts | 474 +++++++++++++++++++++++ test/ship-queue-train.test.ts | 41 +- 25 files changed, 1591 insertions(+), 136 deletions(-) create mode 100644 lib/ship-queue-dispatch-ci.ts create mode 100644 lib/ship-queue-post-merge.ts create mode 100644 lib/ship-queue-screen.ts create mode 100644 lib/ship-queue-stack-tests.ts create mode 100644 lib/ship-queue-train-forget.ts create mode 100644 lib/ship-queue-train-watch.ts create mode 100644 test/ship-queue-train-ops.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index c81ff0298..304353838 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,30 @@ # Changelog +## [1.91.263.0] - 2026-10-27 + +**Merge trains read their screens from the run, follow themselves without a daemon, and refuse what would have cost a gate.** Operating trains on 2026-10-26 showed the gaps: two screens were VERDICT RED while their lanes reported green and the heads were READY'd anyway; 18 of 41 PR screens concluded success with no slice; a PR whose merge ref lagged main could not be screened at all (#3465); two trains whose PRs merged elsewhere sat in `merging`/`green` forever; a background `gh run watch` chain died with every machine restart; a PR's new tests passed on its branch and failed once stacked behind another PR; and the four-step hand procedure for a conflicting PR's CI was typed by hand each time. + +### Added +- **`lib/ship-queue-screen.ts`**: a dispatched screen's verdict is read from the run's own artifacts (`gate-binding-a`: verdict, reconcile exit, flaky cases; `report-verdict-a/paid-report/collector-outcomes.json`: the panels that failed the lane and the rule-16 flaky lines; a confirmed verdict reuse in `paid-plan/gate-reuse.json`), never from a lane's prose. `ready --screen`, `verify --pr --screen` and `train build` print `SCREEN_VERDICT: # run GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one `FLAKY:` line per flaky case, and record the verdict in the READY receipt (`screen.verdict`, locally and in `refs/notes/gstack-trains`). +- **`QUEUE_SCREEN_RED`**: a RED screen refuses `ready` under `--screen` and `--no-screen` alike, is never re-dispatched at the same merge SHA, and `train build` refuses to seat the pin. **`QUEUE_SCREEN_NOT_RUN`**: a screen run that concluded success with no slice (its route job found the head superseded by a push, or the plan selected nothing) is `NOT-RUN`: `verify --screen` refuses, the receipt records `NOT_RUN`, `ready --screen` dispatches afresh instead of following it, and `train build` seats the pin unscreened with a `SCREEN: … counted as not-run` line. +- **Scratch-merge screens (#3465)**: when GitHub's `refs/pull//merge` lags the PR head, is missing or cannot be fetched, `ready --screen` plans the screen on a scratch merge of the head onto the base, pushed as `screen/-` (`SCREEN_MERGE:` names it); only a real conflict with the base (or a remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`, whose text now says so, and a base that cannot be fetched is `POLICY_SOURCE_UNAVAILABLE`. +- **`train watch --train [--timeout ]`** (`lib/ship-queue-train-watch.ts`): the one resumable follow command. It reads the record, runs the step its state asks for (`train build --train` to push, run the free tests and dispatch the gate; `train status` to settle an unconfirmed dispatch), waits for the pending run by polling gh, reads the verdict and prints the last step's `NEXT:`. No process state: rerun it after an interruption and it continues from the record; it never dispatches anything the record would not dispatch on its own. +- **`train prune --stale`**: a `merging` or `green` train whose every PR is MERGED or CLOSED on GitHub is marked `stale` (a new terminal state), its branch deleted, each PR's state logged on a `STALE:` line; a train with an OPEN PR is kept and says why. The same run deletes every `screen/-*` scratch branch whose PR is MERGED or CLOSED. **`train forget --train | --receipt @ [--dry-run]`** (`lib/ship-queue-train-forget.ts`) removes one settled train (attempts included) or one READY receipt from local state and from the notes ref with an exact `--force-with-lease` on `refs/notes/gstack-trains`; a train still building, running or merging is refused, and a rejected lease removes nothing. +- **`verify --post-merge [--timeout ]`** (`lib/ship-queue-post-merge.ts`): after a merge, waits for every check run and commit status attached to the merged commit (its full 40-hex SHA) to settle, prints one `CHECK: ` per check and `POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)`; green needs a completed check and no failure, `neutral`/`skipped` count as skipped, a pending commit status keeps it pending, a failed check or status is **`QUEUE_POST_MERGE_RED`**, a timeout is `QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. +- **`ready … --dispatch-ci [--timeout ]`** (`lib/ship-queue-dispatch-ci.ts`): for a PR that conflicts with the base and so has no pull_request CI, dispatches each named workflow on the branch (skipped when a run at this head already concluded), waits for the runs (`CI_DISPATCH:` lines), clears every requires-remote pregate lane whose workflow is one of them with the run URL (`PREGATE_CLEARED:`), then reads the head as usual. A red run clears no lane and refuses; a fork head is `QUEUE_TRAIN_UNTRUSTED`. This replaces the four-step hand procedure in docs/ship-policy.md. +- **`train build --dry-run --stack-tests [--stack-tests-command ""]`** (`lib/ship-queue-stack-tests.ts`): runs each PR's new test files on the integrated prefix that includes it (T) in a scratch worktree and prints `STACK_TESTS: # T new test file(s) pass|FAIL ()` or `no new test files`; a failure is **`QUEUE_TRAIN_STACK_RED`** with nothing written, pushed or dispatched. + +### Changed +- **`lib/ship-queue-train-github.ts`**: `downloadJson` finds a named file anywhere in a multi-path artifact (`report-verdict-a1` keeps its `paid-report/` directory). **`lib/ship-queue.ts`**: `ciAtHead` is exported and returns the per-check list it already read. **`scripts/lib/windows-curation.ts`**: `test/ship-queue-train-ops.test.ts` is excluded from the Windows subset like the other gh-shim drivers. +- docs/ship-policy.md's merge-train section documents every command above; docs/troubleshooting.md gains `QUEUE_SCREEN_RED`, `QUEUE_SCREEN_NOT_RUN`, `QUEUE_TRAIN_STACK_RED` and `QUEUE_POST_MERGE_RED`, and `QUEUE_SCREEN_STALE_REF` now means a real conflict only. + +### Tests +- `test/ship-queue-train-ops.test.ts` (new; the gh shim in `test/helpers/ship-queue-train-fixture.ts` gained `binding`, `outcomes`, `reuse`, `plan: null` and `ci` rules with the real artifact layout): a RED screen refuses ready under either choice, is printed by verify --screen with its red and flaky cases and refuses the pin in train build; a GREEN screen with flaky cases is followed and recorded; a NOT-RUN screen refuses verify with `QUEUE_SCREEN_NOT_RUN`, records `NOT_RUN` and is re-dispatched; a lagging or missing merge ref gets a scratch merge as `screen/-` and a real conflict stays `QUEUE_SCREEN_STALE_REF`; prune --stale settles only an all-merged-or-closed train and keeps one with an OPEN PR; train forget removes a train or a receipt and refuses a merging train; watch resumes from free-pending through the gate to the verdict and never dispatches twice; verify --post-merge waits, is red with the failed names, green when settled, resumable after a timeout; ready --dispatch-ci dispatches, waits, clears the matching lanes, refuses a red run and a fork head; --stack-tests catches a PR whose new test fails on T2 and passes after `--order` moves it first. Each was red before its change. +- `test/ship-queue-train.test.ts`: the stale-ref test now proves the scratch merge path; the planned-0-cases test expects the SCREEN: verdict lines. + +### Gate integrity +- No threshold, rubric, trial count, case kind or judge prompt changed. A screen's verdict is read as evals.yml bound it; nothing here changes what a case must pass. + ## [1.91.262.0] - 2026-10-27 **The weekly release accepts a commit a green train merged, and five lane frictions seen in operation are gone.** Trains merge faster than the main backstop settles, so most main commits ended with a neutral (`superseded`) `main-backstop` check run and the weekly cut returned `NO_GREEN_COMMIT` for a week of commits every one of which merged through a green train. The evidence already existed in `refs/notes/gstack-trains`; the gate now reads it. Alongside: `gstack-issue-links` no longer turns another repository's `(name #N, #M)` into this repository's Refs lines, `gstack-post` no longer asks for a confirmation on run ids written next to run words, `gstack-post pr-body` no longer wipes a PR body when a lane posts one section, and `gstack-ship-queue ready` finds the pregate verdict a lane recorded from a separate clone. diff --git a/VERSION b/VERSION index 89fd084b4..54b95e6c9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.262.0 +1.91.263.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 2b94af0ff..8fedef666 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.262.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.263.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-doctor-components.sh b/bin/gstack-doctor-components.sh index bc9fe7537..dca89bec9 100644 --- a/bin/gstack-doctor-components.sh +++ b/bin/gstack-doctor-components.sh @@ -286,6 +286,10 @@ gstack_result_anchor() { QUEUE_RECEIPT_MISSING) printf '%s' 'queue-receipt-missing' ;; QUEUE_RECEIPT_STALE) printf '%s' 'queue-receipt-stale' ;; QUEUE_SCREEN_STALE_REF) printf '%s' 'queue-screen-stale-ref' ;; + QUEUE_SCREEN_NOT_RUN) printf '%s' 'queue-screen-not-run' ;; + QUEUE_TRAIN_STACK_RED) printf '%s' 'queue-train-stack-red' ;; + QUEUE_POST_MERGE_RED) printf '%s' 'queue-post-merge-red' ;; + QUEUE_SCREEN_RED) printf '%s' 'queue-screen-red' ;; READY_SCREEN_CHOICE_REQUIRED) printf '%s' 'ready-screen-choice-required' ;; PREGATE_TIMING_ASSERTION) printf '%s' 'pregate-timing-assertion' ;; PREGATE_NO_POLICY) printf '%s' 'pregate-no-policy' ;; diff --git a/bin/gstack-ship-queue b/bin/gstack-ship-queue index a5f852d10..b130a7133 100755 --- a/bin/gstack-ship-queue +++ b/bin/gstack-ship-queue @@ -12,7 +12,9 @@ * gstack-ship-queue gate-ahead --after --pr [--predecessor-ref ] [--allow-repo-commands] * gstack-ship-queue gate-ahead --record --pr --attempt --exit --log * gstack-ship-queue gate-ahead --list --pr - * gstack-ship-queue train build|status|verdict|bisect|rebuild|prune ... (merge trains; train --help) + * gstack-ship-queue verify --pr --screen (train policy: the head's screen verdict from its run) + * gstack-ship-queue verify --post-merge [--timeout ] (after a merge: main's push workflows at that commit until they settle) + * gstack-ship-queue train build|status|verdict|bisect|rebuild|watch|prune ... (merge trains; train --help) * * Every subcommand takes --base (default origin/HEAD, else main) and * ends with GSTACK_RESULT: skill=ship-queue status=complete|refused run=. @@ -47,7 +49,10 @@ const HELP = `gstack-ship-queue — the shared merge queue's protocol steps (doc gate-ahead --record --pr --attempt --exit --log bind the gate result to that attempt's gated tree; removes its worktree gate-ahead --list --pr attempts for a PR, newest last (the current one) - train build|status|verdict|bisect|rebuild|prune ... + verify --pr --screen (train policy) the PR head's screen verdict read from its run (SCREEN_VERDICT) + verify --post-merge [--timeout ] wait for every check run at a merged commit to settle: POST_MERGE: green|red|pending + and one CHECK: line per workflow; resumable, default timeout 2700 s + train build|status|verdict|bisect|rebuild|watch|prune ... merge trains, when the base policy declares a train block (train --help; docs/ship-policy.md#run-a-merge-train); ready then takes --screen|--no-screen @@ -146,6 +151,7 @@ async function main(argv: string[]): Promise { if (!args[0] || args[0].startsWith('-')) { process.stdout.write(train.TRAIN_HELP + '\n'); return args.includes('--help') || args.includes('-h') ? EXIT.ok : EXIT.usage; } return train.trainMain(ctx, args); } + if (cmd === 'verify' && flag(args, '--post-merge') !== undefined) return (await import('../lib/ship-queue-train-cli')).postMergeMain(ctx, args); if (!pr || !/^\d+$/.test(pr)) return usage(`${cmd} needs --pr `); const gateExit = flag(args, '--gate-exit'); if (gateExit !== undefined && !/^-?\d+$/.test(gateExit)) return usage('--gate-exit needs an exit code'); @@ -168,7 +174,7 @@ async function main(argv: string[]): Promise { })); } if (cmd === 'verify') { - if (flag(args, '--train')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr); + if (flag(args, '--train') || args.includes('--screen')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr); const record = flag(args, '--record'); return finish('verify', verify(ctx, { pr, gated: flag(args, '--gated'), record: record ? path.resolve(record) : undefined, receipt: flag(args, '--receipt'), diff --git a/docs/ship-policy.md b/docs/ship-policy.md index 12a042f7e..789e8d65e 100644 --- a/docs/ship-policy.md +++ b/docs/ship-policy.md @@ -187,18 +187,25 @@ READY PR as conflicting each time main stamps; the train merges the CHANGELOG by section, so the lane never merges main for it. The conflict does mean GitHub has no merge ref for the PR, so no `pull_request` CI runs on the head and `ready` prints `ci: none at ` with a `ci-hint:` line. The lane then -dispatches the free workflows on the branch (`gh workflow run free-tests.yml ---ref && gh workflow run windows-free-tests.yml --ref `), -clears the requires-remote pregate lanes with those runs -(`gstack-pregate clear --remote --run-url `) and reruns `ready`, -which passes because the dispatched check runs attach to the commit; the -`ci-hint:` line then says those non-pull_request runs count. The `ci-hint:` line is -advisory text for the lane; nothing parses it, and a PR whose head is on a fork -gets a hint that leaves the dispatch to the coordinator. `ready --screen` on such a -PR refuses, because there is no merge ref to screen (`QUEUE_SCREEN_STALE_REF`, -` has no refs/pull//merge`); that refusal's "merge into the -branch" advice does not apply to a CHANGELOG-only conflict, so the lane READYs -with `--no-screen` after the skill's own paid cases ran focused. +runs `ready … --dispatch-ci free-tests.yml,windows-free-tests.yml [--timeout +]` (under a train policy): it dispatches each named workflow on the branch +(`gh workflow run --ref `, skipped when a run at this head already +concluded), waits for the runs (`CI_DISPATCH: --ref run +`), clears every requires-remote pregate lane whose workflow is one +of them with the run URL (`PREGATE_CLEARED: `; a lane of another +workflow stays uncleared and is named in the refusal), and then reads the head +as usual, which passes because the dispatched check runs attach to the commit; +the `ci-hint:` line then says those non-pull_request runs count. A red run +clears no lane and refuses (`QUEUE_NOT_READY`); a run still going at `--timeout` +(default 2700 s) is `QUEUE_TRAIN_PENDING`, and rerunning the command keeps +waiting; a fork head is never dispatched here (`QUEUE_TRAIN_UNTRUSTED`). By hand the same four steps are `gh workflow +run --ref `, wait, `gstack-pregate clear --remote --run-url +`, rerun `ready`. The `ci-hint:` line is advisory text for the lane; +nothing parses it. `ready --screen` on such a PR has no merge ref to screen, +so it builds the scratch merge described below (`SCREEN_MERGE:`, pushed as +`screen/-`) and screens that; only a real conflict with the base +(or a remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A CHANGELOG-only conflict is screenable, and a +lane whose skill's own paid cases ran focused may still READY with `--no-screen`. `ci:` counts the latest run per check (one app's check name), as GitHub's rollup does: a check that was re-run or superseded on the same commit (a @@ -393,7 +400,7 @@ explicit choice: ```bash gstack-ship-queue ready --pr 42 # prints SCREEN: lines and the recommendation, refuses gstack-ship-queue ready --pr 42 --no-screen # READY receipt; the PR rides unscreened -gstack-ship-queue ready --pr 42 --screen [--broad] # dispatches one paid screen on the PR's merge SHA (QUEUE_SCREEN_STALE_REF when GitHub's merge ref lags the head) +gstack-ship-queue ready --pr 42 --screen [--broad] # dispatches one paid screen on the PR's merge SHA (a scratch merge when GitHub's merge ref lags the head; QUEUE_SCREEN_STALE_REF only on a real conflict) ``` Without a choice it prints the `SCREEN:` selection (with @@ -410,10 +417,13 @@ the policy, so a repeat follows the same run and a broad screen never adopts a narrow one. The READY block gains `screen: …`, and the receipt is written locally and to the notes ref. `--dry-run` prints the dispatch without sending it. Before dispatching, `ready` fetches the merge ref and requires its second -parent to be the PR head: a missing or lagging merge ref (GitHub stops -recomputing it while the PR conflicts with the base, CHANGELOG-only conflicts -included) is `QUEUE_SCREEN_STALE_REF`, because a screen dispatched there plans -no cases and reports success. A finished screen that planned no cases (no +parent to be the PR head and its first parent the current base: a missing, +lagging or unfetchable merge ref (GitHub stops recomputing it while the PR +conflicts with the base, CHANGELOG-only conflicts included) is replaced by a +scratch merge of the head onto the base, pushed as `screen/-` +(the `SCREEN_MERGE:` line says which); a screen dispatched on a stale ref would +plan no cases and report success. Only a real conflict with the base (or a +remote whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. A finished screen that planned no cases (no `paid-plan` artifact, which a superseded run never uploads, or a plan manifest with nothing planned) is never followed as a screen: `ready --screen` dispatches a fresh one and `train build` prints `SCREEN: # run planned @@ -547,9 +557,81 @@ It runs the normal free tests, full gate and verification on main plus the repair. When verify confirms the repair merged, it prints `FREEZE: cleared` and `NEXT:` builds a train of the PRs still unmerged. +**5b. Coordinator: after the merge.** `gstack-ship-queue verify --post-merge + [--timeout ]` (the merged commit's full 40-character SHA) watches +every check run attached to it (the push workflows: free tests, the Windows +subset, the main backstop) and every commit status until none is still +running, prints one `CHECK: ` per check and +`POST_MERGE: green|red|pending|none (passed= failed= pending= skipped=)`. +Green needs at least one completed check and no failure; `neutral` and +`skipped` (the backstop superseding itself) count as skipped; a pending +commit status keeps the commit pending and a failed one makes it red. A +failure is `QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. It keeps no process +state, so rerunning it after a machine restart continues the watch. + **6. Clean up.** `gstack-ship-queue train prune [--train ] [--dry-run]` deletes the `train/` branches of merged, superseded and dropped trains -(`WOULD-DELETE:`, `DELETED:`, `GONE:`). +(`WOULD-DELETE:`, `DELETED:`, `GONE:`). `train prune --stale` also settles a +`merging` or `green` train whose every PR is MERGED or CLOSED on GitHub (its +PRs landed through another train or by hand): it logs each PR's state on a +`STALE:` line, marks the record `stale` and deletes its branch; a train with an +OPEN PR is kept (`KEEP: … still needs the train`). The same run deletes every +`screen/-*` scratch branch whose PR is MERGED or CLOSED and keeps one whose +PR is still OPEN. `train forget --train | --receipt +@ [--dry-run]` removes one record (a settled train with its attempts, +or one READY receipt) from local state and from `refs/notes/gstack-trains` +with an exact lease on the notes ref, printing `FORGET:` then `FORGOTTEN:`; a +train that is still building, running or merging (`built`, `pushed`, +`free-pending`, `free-green`, `dispatch-unknown`, `dispatch-absent`, +`gate-pending`, `merging`) is refused; a rejected lease removes nothing and +restores the local notes ref. + +**Following a train without a daemon.** `gstack-ship-queue train watch --train + [--timeout ]` is the one resumable follow command: it reads the +record, runs the step its state asks for (`train build --train` to push, run +the free tests and dispatch the gate; `train status` to settle an unconfirmed +dispatch), waits for the pending run by polling gh, reads the verdict and +prints the last step's `NEXT:`. It never dispatches anything the record would +not dispatch on its own, so rerunning it after an interruption continues from +the record (`WATCH:` lines say what it did; a timeout, default 7200 s, is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`; `--free-timeout `, +default 2700, bounds the wait for the free tests). + +**Screens are read from the run.** `ready --screen` and `verify --pr +--screen` read a dispatched screen's verdict from the run's own artifacts +(`gate-binding-a`, `report-verdict-a`), never from a lane's +prose, and record it in the READY receipt (`screen.verdict`): `SCREEN_VERDICT: +# run GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one +`FLAKY:` line per rule-16 flaky case. `PENDING` is a run still going, +`UNKNOWN` one the queue cannot read, and `INCOMPLETE` a finished run that bound +no verdict (its report job did not finish): `ready --screen` dispatches afresh +instead of following it, `verify --screen` refuses it (`QUEUE_PROOF_MISSING`) +and `train build` seats the pin unscreened, as for `NOT-RUN`. A RED screen refuses `ready` under +either choice, is never re-dispatched at the same merge SHA and is not seated +by `train build`, whose `NEXT:` names the build without that PR +(`QUEUE_SCREEN_RED`); a run that concluded success with no slice (its route +job found the head superseded or the plan selected nothing) is `NOT-RUN`: +`verify --screen` refuses with `QUEUE_SCREEN_NOT_RUN`, `ready --screen` +dispatches afresh instead of following it, and `train build` seats the pin +unscreened with a `SCREEN: … counted as not-run` line. When GitHub's +`refs/pull//merge` lags the PR head or is missing, `ready --screen` plans +the screen on a scratch merge of the head onto the base, pushed as +`screen/-` (`SCREEN_MERGE:` names it; `prune --stale` deletes it +once the PR merged or closed); only a real conflict with the base (or a remote +whose refs cannot be listed) is `QUEUE_SCREEN_STALE_REF`. + +**Stacked tests before the gate.** `train build --prs … --dry-run --stack-tests +[--stack-tests-command ""]` runs each PR's new test files (added relative +to the base, with a name ending in `.test`, `_test`, `.spec` or `_spec` and a +JS or TS extension, bun's own rule, so a helper or fixture under `test/` is not +one) on the integrated prefix that includes it, T, in a scratch worktree, and prints +`STACK_TESTS: # T new test file(s) pass|FAIL ()` or `no new +test files`. A failure is `QUEUE_TRAIN_STACK_RED` (nothing written, pushed or +dispatched): the tests pass on the PR head alone only because nothing before +it in the train had changed what they read, so fix them against that prefix or +reorder with `--order`. The default command is `bun --no-env-file +--config=/dev/null test` with the files appended. **Escape hatches.** Remove the `train` block on the base branch to return every lane to the per-PR queue at once. `--urgent` ships one PR alone, diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 289b5013a..acbbc22dc 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -2643,7 +2643,9 @@ or corrupt object) that `git ls-tree` cannot confirm is absent; it never falls through to the user-owned file. It also refuses when Bun loaded a `.env` file from the working directory (a bare `bun gstack-pregate`), since a branch could then pick the user policy: run `gstack-pregate` directly or as -`bun --no-env-file`. +`bun --no-env-file`. `gstack-ship-queue ready` under a train policy fetches +`origin/` first and refuses with this code, quoting git's error, when +that fetch fails. **Fix.** `git fetch origin --depth=1`, then rerun. To run against the branch's own file deliberately, pass `--policy-from head --allow-repo-commands`; @@ -4051,7 +4053,9 @@ time; `gstack-ship-policy validate` prints `MIN_READER:` for exactly that. coordinator could misread: the working tree has uncommitted tracked changes, HEAD is not the PR's pushed head (read through `gh pr view`), CI at that exact head is not green (failed, pending or unreadable), or the pregate result -is missing, failed or recorded on another tree (`PREGATE_STALE`). +is missing, failed or recorded on another tree (`PREGATE_STALE`). Under +`--dispatch-ci` it is also a dispatched workflow run that concluded red (a red +run clears no pregate lane) or a `gh workflow run` that failed. **Fix.** Commit and push, wait for CI on that head, rerun `gstack-pregate run --allow-repo-commands` when the tree changed, then rerun @@ -4065,8 +4069,13 @@ which line is not yet true. input does not match: the gated head is not a commit here, HEAD differs from the gated head beyond stamps (the gate never covered this tree), an earlier stamp with other inputs is unfinished, the stamp journal's head moved, or a -gate-ahead merge could not be recomputed with `git merge-tree`. Nothing was -written unless the `QUEUE_STAMP_REFUSED:` line says `committed=yes`. +gate-ahead merge could not be recomputed with `git merge-tree`. The train +commands use it too: `train forget` on a record that does not exist or on a +train still building, running or merging, `train watch` on an unknown record, +`verify --pr --screen` on a run with no verdict to read (an INCOMPLETE or +unreadable run), and `ready --screen` when the scratch merge branch cannot be +pushed. Nothing was written unless the `QUEUE_STAMP_REFUSED:` line says +`committed=yes`. **Fix.** Supply the named input (fetch the gated head, pass the current base sha), or regate the tree you mean to stamp. For an unfinished stamp, rerun the @@ -4141,7 +4150,8 @@ a `train` block with `min_gstack_version` on the base branch first **Meaning.** A pinned PR's head repository is not the base repository. The train branch and its dispatches run with repository secrets, so fork work -never rides a train. +never rides a train. `ready --dispatch-ci` refuses a fork head for the same +reason and dispatches nothing. **Fix.** Run the printed `train build --prs `; the fork PR stays on the trusted-branch path. @@ -4205,7 +4215,11 @@ one-PR train. ### `QUEUE_TRAIN_PENDING` **Meaning.** The train's run (free tests, the full gate or the bisect) has -not finished, so there is no verdict yet. +not finished, so there is no verdict yet. The same code covers a `train watch` +that reached its timeout, `verify --pr --screen` on a screen run still +going, `verify --post-merge` on checks that have not settled, and a +`ready --dispatch-ci` workflow run still going at its `--timeout`; each prints +the command to rerun as `NEXT:`. **Fix.** Rerun the printed `NEXT:` command later. Nothing is dispatched again while a run is pending. @@ -4343,10 +4357,14 @@ to fix this: the stamp comes only when the slot is called. ### `QUEUE_SCREEN_STALE_REF` -**Meaning.** `gstack-ship-queue ready --screen` dispatches the screen against -GitHub's `refs/pull//merge`, and that ref is missing or still merges an -older head than the PR's pushed head. GitHub stops recomputing the merge ref -while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp +**Meaning.** `ready --screen` refuses with this code when the PR conflicts with +the base beyond the CHANGELOG and the stamp files (the message lists the +conflicting paths) or when it cannot list the remote's refs. A missing or +lagging merge ref alone no longer refuses (see Fix). The rest of this +paragraph is why such a ref cannot be screened directly: `ready --screen` +dispatches the screen against GitHub's `refs/pull//merge`, and that ref can +be missing or still merge an older head than the PR's pushed head. GitHub +stops recomputing the merge ref while the PR conflicts with the base (a CHANGELOG-only conflict after a stamp counts), so a screen dispatched there checks out the old merge, sees itself superseded, plans 0 cases and reports success. GSTA-95 hit this on #3431: `ready` printed `SCREEN_DISPATCH:` as if a screen ran. `ready` now compares @@ -4362,16 +4380,113 @@ read (`… paid-plan artifact has expired`, `… manifest.json is unreadable`, a unreadable run or artifact list, an unreadable repository slug, or a READY whose run id is not a number) keeps the slot screened and prints `SCREEN: # ; counted as screened, unverified`; `ready --screen` -follows such a run. When `ready` cannot list the remote's refs or fetch the -merge ref at all, it refuses with the same code and a `could not list …` or -`could not fetch …` message whose NEXT line says to check the remote, not to -resolve a conflict. +follows such a run. When `ready` cannot list the remote's refs, it refuses with +the same code and a `could not list …` message whose NEXT line says to check +the remote, not to resolve a conflict; a merge ref that lists but cannot be +fetched is replaced by the scratch merge described under Fix. -**Fix.** Resolve the conflict with the base branch (or wait for GitHub to -recompute the merge ref), push, and run `gstack-ship-queue ready --pr ---screen` again. Under crew rule 14 a CHANGELOG-only conflict is left to the -train, so a lane that cannot get a fresh merge ref READYs `--no-screen` only -after its own focused paid cases passed (rule 6). +**Fix.** Since the scratch-merge screen landed, a lagging merge ref is no +longer a refusal: `ready --screen` builds the merge of the PR head onto +`origin/` itself with the train's merge rules (version files take the +base's side, the CHANGELOG merges by section), pushes it as +`screen/-` and dispatches the screen on that commit, printing +`SCREEN_MERGE: scratch merge …`. The code now means a real conflict: the +message lists the conflicting paths. Resolve those with the base branch, +push, and run `gstack-ship-queue ready --pr --screen` again. A +CHANGELOG-only or stamp-file-only conflict does not refuse, unless the PR's +CHANGELOG drops or renames a released heading (the message then names the +CHANGELOG). `train prune --stale` deletes the scratch branches of merged or +closed PRs. + + +### `QUEUE_SCREEN_RED` + +**Meaning.** The screen run dispatched for this PR head finished RED. The +verdict is read from the run itself, never from a lane's report: the +`gate-binding-a` artifact evals.yml's report job uploads (verdict, +reconcile exit, flaky cases) and the `report-verdict-a` artifact's +`collector-outcomes.json` (the panels that failed the lane and the rule-16 +flaky lines). A case the binding calls red already failed twice inside the +run (rule 16 reruns a first failure once), so the red is real. `ready` +refuses with this code when the head's recorded screen run is red, whichever +of `--screen` or `--no-screen` it is given, and never dispatches another +screen at the same merge SHA; `train build` refuses to seat the pin and its +`NEXT:` names the build without it; `verify --pr --screen` prints +`SCREEN_VERDICT: # run RED red= flaky=` and one +`FLAKY:` line per flaky case. The verdict is recorded in the READY receipt +(`screen.verdict`), locally and in `refs/notes/gstack-trains`. + +**Fix.** Read the red cases from the `SCREEN_VERDICT:` line, repair, push a +new head, run /ship and `gstack-ship-queue ready --pr --screen` again +(the new head gets its own screen). A GREEN screen with flaky cases is green: +the flaky cases are listed for the ledger and need no rerun. + + +### `QUEUE_TRAIN_STACK_RED` + +**Meaning.** `gstack-ship-queue train build --dry-run --stack-tests` ran a PR's +new test files (files the PR adds relative to the base whose name ends in +`.test`, `_test`, `.spec` or `_spec` plus a JS or TS extension, bun's own +rule; a helper or fixture under `test/` is not one) on the integrated prefix that includes it, +T (the base plus every slot up to and including that PR), in a scratch +worktree, and they failed. The `STACK_TESTS: # T new test file(s) +FAIL (): ` line names the files and the runner's +failing lines. Those tests pass on the PR head alone only because nothing +before it in the train had changed what they read (a PR stacked on another +PR's test files or fixtures hit this on 2026-10-26). Nothing was written, +pushed or dispatched; the ordinary dry run (without `--stack-tests`) does not +run them. + +**Fix.** Fix the PR's tests against that prefix, or reorder the train with +`--order` so the PR they depend on comes first, then rerun the dry run. Pass +`--stack-tests-command ""` when the repo's tests run under something +other than `bun test`; the files are appended to the command. + + +### `QUEUE_POST_MERGE_RED` + +**Meaning.** `gstack-ship-queue verify --post-merge ` (the full 40-character +SHA) watched the check runs attached to a commit the train merged onto the +base branch (the push workflows: free tests, the Windows subset, the main +backstop) and its commit statuses until none was still running, and at least +one check concluded `failure`, `timed_out`, `cancelled` or another non-success, +or a commit status failed (the message then says a commit status, not a check +run). A `neutral` or `skipped` check (the +backstop superseding itself when main moved on) is counted as skipped, never +red. The command keeps no process state: it polls gh until the checks settle +or `--timeout` (default 2700 s) passes, prints `POST_MERGE: +green|red|pending|none (passed= failed= pending= skipped=)` and one +`CHECK: ` line per check, and a timeout is +`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`, so rerunning it after +an interruption continues the watch. + +**Fix.** Read the red run named on its `CHECK:` line. Repair on a new PR or +roll back the merge; `gstack-ship-queue train status` names the freeze if the +backstop filed one. + + +### `QUEUE_SCREEN_NOT_RUN` + +**Meaning.** The screen run dispatched for this PR concluded `success` but +ran no slice. evals.yml's `route` job plans nothing when +`scripts/eval-readiness.ts wait` finds the PR head superseded (a push landed +between `ready --screen` and the plan) or when the plan selects no case; the +run then uploads no `paid-plan` artifact (or an empty manifest), every slice +job is skipped and GitHub shows a green check. On 2026-10-26, 18 of 41 PR +screens ended this way. The verdict is read from the run (`lib/ship-queue-screen.ts`), +so such a run is `NOT-RUN`: `verify --pr --screen` prints +`SCREEN_VERDICT: # run NOT-RUN ()` and refuses with this code, the +READY receipt records `screen.verdict.outcome: NOT_RUN`, `ready --screen` never +follows the run (its `SCREEN_DISPATCH:` line says `not following … a screen +that planned nothing is not a screen`) and `train build` seats the pin unscreened +with `SCREEN: # run planned no cases …; counted as not-run, the slot +rides unscreened`. + +**Fix.** `gstack-ship-queue ready --pr --screen` at the current head +dispatches a fresh screen. A head that moved since the dispatch needs a fresh +READY anyway (`QUEUE_TRAIN_MOVED`); a head that did not move and still plans +nothing has nothing diff-relevant to screen, and `--no-screen` with the +focused cases named in the READY block is the honest choice. ### `READY_SCREEN_CHOICE_REQUIRED` diff --git a/lib/result-codes-release.ts b/lib/result-codes-release.ts index d4551c814..5c25f026e 100644 --- a/lib/result-codes-release.ts +++ b/lib/result-codes-release.ts @@ -148,6 +148,26 @@ export const RELEASE_RESULT_CODES = { summary: 'GitHub’s refs/pull//merge is missing or still merges an older head than the PR’s: the PR conflicts with the base (a CHANGELOG-only conflict counts), so a screen dispatched on it would plan no cases and report success', fix: 'resolve the conflict with the base branch (or wait for GitHub to recompute the merge ref), push, then gstack-ship-queue ready --pr --screen again; a lane never READYs on a screen that planned nothing', }, + QUEUE_SCREEN_NOT_RUN: { + anchor: 'queue-screen-not-run', + summary: 'the PR’s dispatched screen run concluded success but ran no slice (the run planned no cases: its route job found the PR head superseded by a push, or the plan selected nothing), so the head is not screened and the run is counted as not-run, never as green', + fix: 'gstack-ship-queue ready --pr --screen dispatches a fresh screen at the current head (a not-run screen is never followed); train build seats the pin unscreened and says so on its SCREEN: line', + }, + QUEUE_TRAIN_STACK_RED: { + anchor: 'queue-train-stack-red', + summary: 'train build --dry-run --stack-tests ran a PR’s new test files on the integrated prefix that includes it (T: main plus every slot up to that PR) and they failed; the tests pass on the PR head alone only because nothing before it in the train had changed what they read', + fix: 'fix the PR’s tests against that prefix (read the STACK_TESTS: line for the files and the failing assertion), or reorder the train with --order so the PR they depend on comes first, then rerun the dry run; nothing was written, pushed or dispatched', + }, + QUEUE_POST_MERGE_RED: { + anchor: 'queue-post-merge-red', + summary: 'a push workflow on the merged commit (free tests, the Windows subset, the main backstop) concluded failure, timed out or was cancelled after the train merged it; a neutral or skipped check is not red', + fix: 'read the red run (gstack-ship-queue verify --post-merge prints one CHECK: line per workflow), repair on a new PR or roll back the merge; gstack-ship-queue train status names the freeze if the backstop filed one', + }, + QUEUE_SCREEN_RED: { + anchor: 'queue-screen-red', + summary: 'the PR head’s dispatched screen run finished RED (its gate binding says so; a case that failed twice is real), so the head is not READY and a train will not seat it', + fix: 'read the SCREEN_VERDICT line (gstack-ship-queue verify --pr --screen prints the red and flaky cases), repair, push a new head, run /ship and ready again; a red screen is never re-dispatched at the same merge SHA', + }, READY_SCREEN_CHOICE_REQUIRED: { anchor: 'ready-screen-choice-required', summary: 'the base policy turns trains on, so READY needs an explicit screen choice: --screen (pay for a diff-relevant screen now) or --no-screen', diff --git a/lib/ship-queue-dispatch-ci.ts b/lib/ship-queue-dispatch-ci.ts new file mode 100644 index 000000000..0c496ab8c --- /dev/null +++ b/lib/ship-queue-dispatch-ci.ts @@ -0,0 +1,80 @@ +/** + * `ready --dispatch-ci [--timeout ]`: a PR that + * conflicts with the base (a CHANGELOG-only conflict after every stamp + * counts) has no merge ref, so GitHub runs no pull_request CI on its head and + * `ready` prints `ci: none`. The lane used to dispatch the free workflows by + * hand, wait, clear the requires-remote pregate lanes with the run URLs and + * rerun ready. This does those four steps before `ready` reads the head: one + * `gh workflow run --ref ` per named workflow (skipped when a + * run at this head already concluded), a wait for each run, and one + * `PREGATE_CLEARED:` per requires-remote lane whose workflow is one of them + * and whose run concluded success. A red run clears no lane and refuses. A + * fork head is never dispatched here (QUEUE_TRAIN_UNTRUSTED). + */ +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import type { PregateFile } from './headless-artifacts'; +import { PREGATE_OUT_REL, clearObligation, readPregateFile, writePregateFile } from './pregate/index'; +import type { QueueContext } from './ship-queue'; +import type { QueueFailure } from './ship-queue-gate'; +import { prInfo, repoSlug, runView } from './ship-queue-train-github'; + +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface DispatchCiOptions { pr: string; workflows: string[]; pregate?: string; timeoutSec: number; pollMs: number } +export type DispatchCiResult = { lines: string[] } | { lines: string[]; failure: QueueFailure }; + +interface RunRow { databaseId: number; url: string; status: string; conclusion: string | null; headSha: string } + +function runsAt(ctx: QueueContext, workflow: string, branch: string, head: string): RunRow[] { + const r = ctx.gh(['run', 'list', '--workflow', workflow, '--branch', branch, '--event', 'workflow_dispatch', '--limit', '20', '--json', 'databaseId,url,status,conclusion,headSha']); + try { return r.ok ? (JSON.parse(r.out) as RunRow[]).filter(x => x.headSha === head) : []; } catch { return []; } +} + +export async function dispatchCi(ctx: QueueContext, o: DispatchCiOptions): Promise { + const info = prInfo(ctx, o.pr); + if ('error' in info) return { lines: [], failure: { code: 'QUEUE_NOT_READY', message: info.error } }; + if (!info.headRefName) return { lines: [], failure: { code: 'QUEUE_NOT_READY', message: `cannot read PR #${o.pr}'s branch through gh (no headRefName)` } }; + if (info.isCrossRepository) return { lines: [], failure: { code: 'QUEUE_TRAIN_UNTRUSTED', message: `#${o.pr}'s head is on a fork; its workflows are not dispatched here (the coordinator decides)` } }; + const head = info.headRefOid; + const branch = info.headRefName; + const slug = repoSlug(ctx); + const deadline = Date.now() + o.timeoutSec * 1000; + const lines: string[] = []; + const concluded = new Map(); + for (const wf of o.workflows) { + const atHead = runsAt(ctx, wf, branch, head); + const prior = atHead.find(r => r.status === 'completed'); + if (prior) { lines.push(`CI_DISPATCH: ${wf} not dispatched (run ${prior.databaseId} already concluded ${prior.conclusion ?? 'unknown'} at this head)`); concluded.set(wf, prior); continue; } + const live = atHead[0]; + if (!live) { + const d = ctx.gh(['workflow', 'run', wf, '--ref', branch]); + if (!d.ok) return { lines, failure: { code: 'QUEUE_NOT_READY', message: `gh workflow run ${wf} --ref ${branch} failed (${(d.err ?? d.out).trim().replace(/\s+/g, ' ').slice(0, 120) || 'no output'})` } }; + } + let run: RunRow | undefined = live; + for (;;) { + run = run && slug ? runView(ctx, slug, run.databaseId) ?? run : runsAt(ctx, wf, branch, head)[0]; + if (run?.status === 'completed') break; + if (Date.now() >= deadline) return { lines: [...lines, `CI_DISPATCH: ${wf} --ref ${branch} ${run ? `run ${run.databaseId} ${run.url} ${run.status}` : 'no run appeared'}`], failure: { code: 'QUEUE_TRAIN_PENDING', message: `${wf} ${run ? `run ${run.databaseId} is still ${run.status}` : 'has no workflow_dispatch run at this head yet'} at the --dispatch-ci timeout; rerun ready --dispatch-ci to continue waiting` } }; + await sleep(o.pollMs); + } + lines.push(`CI_DISPATCH: ${wf} --ref ${branch} run ${run.databaseId} ${run.url} ${run.conclusion ?? 'unknown'}`); + concluded.set(wf, run); + } + const red = [...concluded.entries()].filter(([, r]) => r.conclusion !== 'success'); + if (red.length) return { lines, failure: { code: 'QUEUE_NOT_READY', message: red.map(([wf, r]) => `${wf} run ${r.databaseId} concluded ${r.conclusion ?? 'unknown'} at ${head.slice(0, 12)}`).join('; ') + '; a red run clears no lane' } }; + const pregateFile = path.resolve(ctx.repoRoot, o.pregate ?? PREGATE_OUT_REL); + if (!fs.existsSync(pregateFile)) return { lines }; + const read = readPregateFile(pregateFile); + if (!read.file) return { lines }; + let file: PregateFile = read.file; + let changed = false; + for (const lane of (file.requires_remote ?? []).filter(l => !l.receipt && concluded.has(l.workflow))) { + const run = concluded.get(lane.workflow)!; + file = clearObligation(file, lane.lane, run.url, `gstack-ship-queue ready --dispatch-ci ${new Date().toISOString()}`); + lines.push(`PREGATE_CLEARED: ${lane.lane} ${run.url}`); + changed = true; + } + if (changed) writePregateFile(ctx.repoRoot, pregateFile, file); + return { lines }; +} diff --git a/lib/ship-queue-post-merge.ts b/lib/ship-queue-post-merge.ts new file mode 100644 index 000000000..05a5fc56a --- /dev/null +++ b/lib/ship-queue-post-merge.ts @@ -0,0 +1,45 @@ +/** + * `verify --post-merge [--timeout ]`: the coordinator's follow + * command after a train merged a PR onto the base branch. The push workflows + * run on the merged commit (free tests, the Windows subset, the main backstop + * that supersedes itself when main moves on), and a background `gh run + * watch` chain dies with the machine that started it. This command keeps no + * state: it reads every check run attached to the sha, waits (polling gh) + * until none is still running or the timeout passes, prints one `CHECK:` + * line per check, and a `POST_MERGE:` summary that is green only when at + * least one check completed and none failed; a neutral or skipped check is + * skipped, never red. Commit statuses count too: a pending status keeps the + * commit pending, a failed one makes it red (named as a commit status when no + * check run failed). Rerun it after an interruption and it continues. + */ +import { ciAtHead, type QueueContext } from './ship-queue'; +import { outFailed, outOk, outPending, type TrainOut } from './ship-queue-train'; + +const Q = 'gstack-ship-queue'; +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface PostMergeOptions { sha: string; timeoutSec: number; pollMs: number } + +export async function verifyPostMerge(ctx: QueueContext, o: PostMergeOptions): Promise { + const deadline = Date.now() + o.timeoutSec * 1000; + const at = o.sha.slice(0, 12); + const again = `${Q} verify --post-merge ${o.sha} --timeout ${o.timeoutSec}`; + for (;;) { + const ci = ciAtHead(ctx, o.sha); + const lines = [`POST_MERGE: ${ci.line.replace(/^(\S+) at ([0-9a-f]+)/, '$2 $1')}`, ...ci.checks.map(c => `CHECK: ${c.name} ${c.status === 'completed' ? c.conclusion ?? 'unknown' : c.status}`)]; + const settled = !ci.none && !/^(pending|unknown) /.test(ci.line) && !ci.checks.some(c => c.status !== 'completed'); + if (settled && ci.green) return outOk(lines, 'none', 'none'); + if (settled) { + const red = ci.checks.filter(c => c.status === 'completed' && !['success', 'skipped', 'neutral'].includes(c.conclusion ?? '')).map(c => c.name); + return outFailed({ code: 'QUEUE_POST_MERGE_RED', message: `main's push workflows at ${at} are red: ${red.join(', ') || 'a commit status (no check run failed)'}` }, lines, + `read the red run, repair on a new PR or roll back; ${Q} train status names the freeze if the backstop filed one`, 'none'); + } + if (Date.now() >= deadline) { + const pending = ci.checks.filter(c => c.status !== 'completed').map(c => c.name); + const message = ci.line.startsWith('unknown') ? `the check runs at ${at} are unreadable (${ci.line.replace(/^unknown at [0-9a-f]+ /, '')})` + : ci.none ? `no check run has attached to ${at} yet` : `main's push workflows at ${at} have not settled at the timeout: ${pending.join(', ') || 'a commit status'} still running`; + return outPending({ code: 'QUEUE_TRAIN_PENDING', message }, lines, again, 'none'); + } + await sleep(o.pollMs); + } +} diff --git a/lib/ship-queue-screen.ts b/lib/ship-queue-screen.ts new file mode 100644 index 000000000..febce6a21 --- /dev/null +++ b/lib/ship-queue-screen.ts @@ -0,0 +1,114 @@ +/** + * A dispatched screen's verdict, read from the run itself: + * never from a lane's prose. evals.yml's report job publishes + * `gate-binding-a` (scripts/gate-reuse.ts: verdict, reconcile exit, + * flaky cases) and `report-verdict-a` (collector-outcomes.json: the + * panels and the rule-16 flaky lines) on every executed screen; a superseded + * screen plans nothing and uploads neither (hollowScreen): it concluded + * success with no slice, so its outcome is `not-run`, never green. The outcome is + * recorded in the READY receipt (ScreenChoice.verdict) by `ready`, + * `verify --screen` and `train build`, so the controller reads one line. + */ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { REUSE_FILE, parseBinding, parseReuse } from '../scripts/gate-reuse'; +import type { QueueContext } from './ship-queue'; +import { downloadJson, hollowScreen, runView } from './ship-queue-train-github'; +import { nowIso, type ScreenChoice, type ScreenVerdictRecord } from './ship-queue-train-record'; + +export type ScreenOutcome = + | { kind: 'pending'; detail: string } + | { kind: 'not-run'; detail: string } + | { kind: 'unknown'; detail: string } + | { kind: 'incomplete'; detail: string; flaky: string[] } + | { kind: 'green'; detail: string; flaky: string[]; flakyLines: string[] } + | { kind: 'red'; detail: string; red: string[]; flaky: string[]; flakyLines: string[] }; + +const BINDING = 'gate-binding'; +const REPORT = 'report-verdict'; +/** The plan artifact; a confirmed verdict reuse (R2a/R2) leaves its record there and binds nothing new. */ +const PLAN = 'paid-plan'; + +/** The newest attempt's artifact of a name family (`-a`); null when the run uploaded none or the list is unreadable. */ +function newestArtifact(ctx: QueueContext, slug: string, run: number, name: string): string | null | 'unreadable' { + const artifacts: Array<{ name?: string; expired?: boolean }> = []; + for (let page = 1; page <= 20; page++) { + const r = ctx.gh(['api', `repos/${slug}/actions/runs/${run}/artifacts?per_page=100&page=${page}`]); + let list: { artifacts?: Array<{ name?: string; expired?: boolean }> } | null = null; + try { list = r.ok ? JSON.parse(r.out) : null; } catch {} + if (!Array.isArray(list?.artifacts)) return 'unreadable'; + artifacts.push(...list.artifacts); + if (list.artifacts.length < 100) break; + } + const hits = artifacts.map(a => ({ name: a.name ?? '', expired: !!a.expired, attempt: Number(new RegExp(`^${name}-a(\\d+)$`).exec(a.name ?? '')?.[1]) })) + .filter(a => Number.isSafeInteger(a.attempt)).sort((a, b) => b.attempt - a.attempt); + if (!hits.length) return null; + return hits[0]!.expired ? 'unreadable' : hits[0]!.name; +} + +interface Outcomes { panels?: Array<{ case?: string; status?: string; failsLane?: boolean; redClass?: string | null }>; flakyLines?: string[] } + +/** Read one screen run: status, plan, gate binding, panels. */ +export function screenVerdict(ctx: QueueContext, slug: string, run: number): ScreenOutcome { + const view = runView(ctx, slug, run); + if (!view) return { kind: 'unknown', detail: `run ${run} is unreadable` }; + if (view.status !== 'completed') return { kind: 'pending', detail: `run ${run} is ${view.status}` }; + const plan = hollowScreen(ctx, slug, run); + if (plan.kind === 'hollow') return { kind: 'not-run', detail: plan.reason }; + if (plan.kind === 'unknown') return { kind: 'unknown', detail: plan.reason }; + const bindingName = newestArtifact(ctx, slug, run, BINDING); + if (bindingName === 'unreadable') return { kind: 'unknown', detail: `run ${run}'s artifact list is unreadable or its ${BINDING} artifact has expired` }; + const scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-screen-verdict-')); + let binding: ReturnType = null; + let outcomes: Outcomes | null = null; + try { + if (!bindingName) { + const reuse = parseReuse(downloadJson(ctx, slug, run, PLAN, scratch, REUSE_FILE)); + if (reuse && view.conclusion === 'success') { + const flaky = [...(reuse.reused_flaky ?? [])].sort(); + return { kind: 'green', detail: `reused the GREEN verdict of run ${reuse.reused_run_id} (${reuse.kind}): ${reuse.why}${flaky.length ? `, flaky ${flaky.length}` : ''}`, flaky, flakyLines: [] }; + } + return { kind: 'incomplete', detail: `run ${run} concluded ${view.conclusion ?? 'unknown'} and bound no verdict (no ${BINDING} artifact: the report job did not finish)`, flaky: [] }; + } + binding = parseBinding(downloadJson(ctx, slug, run, bindingName, scratch, 'gate-binding.json')); + const reportName = newestArtifact(ctx, slug, run, REPORT); + outcomes = typeof reportName === 'string' ? downloadJson(ctx, slug, run, reportName, scratch, 'collector-outcomes.json') as Outcomes | null : null; + } finally { fs.rmSync(scratch, { recursive: true, force: true }); } + if (!binding) return { kind: 'unknown', detail: `run ${run}'s ${bindingName} artifact is unreadable` }; + const flaky = [...(binding.flaky_cases ?? [])].sort(); + const flakyLines = Array.isArray(outcomes?.flakyLines) ? outcomes!.flakyLines!.filter((l): l is string => typeof l === 'string') : []; + const red = (outcomes?.panels ?? []).filter(p => typeof p.case === 'string' && (p.failsLane === true || (p.status === 'FAIL' && p.redClass === 'VERDICT'))).map(p => p.case!).sort(); + const base = `verdict ${binding.verdict}, reconcile exit ${binding.reconcile_exit}, run concluded ${view.conclusion ?? 'unknown'}${flaky.length ? `, flaky ${flaky.length}` : ''}`; + if (binding.verdict === 'GREEN' && binding.reconcile_exit === 0 && view.conclusion === 'success') return { kind: 'green', detail: base, flaky, flakyLines }; + if (binding.verdict === 'RED' || binding.reconcile_exit !== 0 || red.length) return { kind: 'red', detail: base, red, flaky, flakyLines }; + return { kind: 'incomplete', detail: base, flaky }; +} + +/** The receipt field for an outcome that is settled (a pending or unknown run records nothing). */ +export function verdictRecord(o: ScreenOutcome, run: number): ScreenVerdictRecord | null { + if (o.kind === 'pending' || o.kind === 'unknown') return null; + const outcome = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind === 'not-run' ? 'NOT_RUN' : 'INCOMPLETE'; + return { outcome, run, red: o.kind === 'red' ? o.red : [], flaky: 'flaky' in o ? o.flaky : [], detail: o.detail, read_at: nowIso() }; +} + +/** `SCREEN_VERDICT:` and the per-case `FLAKY:` lines for one PR's screen. */ +export function verdictLines(pr: string, run: number, o: ScreenOutcome): string[] { + const word = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind.toUpperCase(); + const red = o.kind === 'red' && o.red.length ? ` red=${o.red.join(',')}` : ''; + const flaky = 'flaky' in o && o.flaky.length ? ` flaky=${o.flaky.join(',')}` : ''; + const lines = [`SCREEN_VERDICT: #${pr} run ${run} ${word}${red}${flaky} (${o.detail})`]; + if ('flakyLines' in o) for (const l of o.flakyLines) lines.push(`FLAKY: ${l}`); + return lines; +} + +/** Apply a read outcome to a receipt's screen choice; returns whether the receipt changed. */ +export function recordVerdict(screen: ScreenChoice, o: ScreenOutcome): boolean { + if (!screen.run) return false; + const rec = verdictRecord(o, screen.run); + if (!rec) return false; + const same = screen.verdict && screen.verdict.outcome === rec.outcome && screen.verdict.run === rec.run && screen.verdict.red.join() === rec.red.join() && screen.verdict.flaky.join() === rec.flaky.join(); + if (same) return false; + screen.verdict = rec; + return true; +} diff --git a/lib/ship-queue-stack-tests.ts b/lib/ship-queue-stack-tests.ts new file mode 100644 index 000000000..514ba01f6 --- /dev/null +++ b/lib/ship-queue-stack-tests.ts @@ -0,0 +1,63 @@ +/** + * `train build --dry-run --stack-tests`: before a train spends a gate, run + * each PR's new test files on the integrated prefix that includes it (T, + * the merge of main and every slot up to and including the PR), not on the + * PR head alone. A test that passes on its branch and fails once an earlier + * slot changed what it reads is a stack red, caught here for the price of a + * few test files instead of a gate. Each prefix is checked out in a scratch + * worktree (the repo's node_modules linked in when it has one), the files run + * with the test command (`bun --no-env-file --config=/dev/null test` by + * default; `--stack-tests-command ""` for another runner, the files + * appended), and the worktree is removed. A PR that adds no test file is + * reported as such and never fails. + */ +import * as fs from 'node:fs'; +import * as os from 'node:os'; +import * as path from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { bunIsolationArgs } from './bun-isolation'; +import { git } from './ship-queue-gate'; +import type { TrainPrefix, TrainSlot } from './ship-queue-train-record'; + +export const DEFAULT_STACK_TEST_COMMAND = ['bun', ...bunIsolationArgs(), 'test']; +/** Bun's own test-file rule: the name ends in `.test`, `_test`, `.spec` or `_spec` before its extension; a helper or fixture under test/ is not a test file and would make `bun test ` exit 1 for matching nothing. */ +const TEST_FILE = /[._](?:test|spec)\.[cm]?[jt]sx?$/; + +export interface StackTestResult { pr: string; prefix: number; files: string[]; ok: boolean; detail: string } + +/** Test files the PR adds relative to the base (added, not modified). */ +export function newTestFiles(repoRoot: string, baseSha: string, head: string): string[] { + const r = git(repoRoot, ['diff', '--name-only', '--diff-filter=A', `${baseSha}...${head}`]); + return r.ok ? r.out.split('\n').map(s => s.trim()).filter(f => f && TEST_FILE.test(f)) : []; +} + +export function stackTests(repoRoot: string, baseSha: string, slots: TrainSlot[], prefixes: TrainPrefix[], command: string[] = DEFAULT_STACK_TEST_COMMAND, timeoutMs = 600_000): StackTestResult[] { + const results: StackTestResult[] = []; + for (const [i, slot] of slots.entries()) { + const prefix = prefixes[i + 1]; + if (!prefix) break; + const files = newTestFiles(repoRoot, baseSha, slot.head); + if (!files.length) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: true, detail: 'no new test files' }); continue; } + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-stack-tests-')); + try { + const add = git(repoRoot, ['worktree', 'add', '--detach', '-q', dir, prefix.commit]); + if (!add.ok) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: false, detail: `cannot check out T${prefix.index}: ${add.err.slice(0, 160)}` }); continue; } + const modules = path.join(repoRoot, 'node_modules'); + if (fs.existsSync(modules) && !fs.existsSync(path.join(dir, 'node_modules'))) fs.symlinkSync(modules, path.join(dir, 'node_modules'), 'dir'); + const [bin, ...args] = command; + const run = spawnSync(bin!, [...args, ...files.map(f => `./${f}`)], { cwd: dir, encoding: 'utf8', timeout: timeoutMs, env: { ...process.env, CI: process.env.CI ?? '1' }, maxBuffer: 64 * 1024 * 1024 }); + const tail = `${run.stdout ?? ''}\n${run.stderr ?? ''}`.split('\n').filter(l => /^\(fail\)|^\s*\d+ (pass|fail)$|^error:/.test(l)).slice(-6).join('; '); + results.push({ pr: slot.pr, prefix: prefix.index, files, ok: run.status === 0, detail: run.status === 0 ? 'pass' : run.error ? `${run.error.message}` : tail || `exit ${run.status}` }); + } finally { + git(repoRoot, ['worktree', 'remove', '--force', dir]); + fs.rmSync(dir, { recursive: true, force: true }); + } + } + return results; +} + +/** One `STACK_TESTS:` line per slot. */ +export function stackTestLine(r: StackTestResult): string { + if (!r.files.length) return `STACK_TESTS: #${r.pr} T${r.prefix} no new test files`; + return `STACK_TESTS: #${r.pr} T${r.prefix} ${r.files.length} new test file(s) ${r.ok ? 'pass' : 'FAIL'} (${r.files.join(', ')})${r.ok ? '' : `: ${r.detail}`}`; +} diff --git a/lib/ship-queue-train-cli.ts b/lib/ship-queue-train-cli.ts index cc41f5988..01fb58477 100644 --- a/lib/ship-queue-train-cli.ts +++ b/lib/ship-queue-train-cli.ts @@ -10,12 +10,15 @@ import { EXIT, formatResult } from './headless-artifacts'; import { RESULT_CODES } from './result-codes'; import type { QueueContext } from './ship-queue'; import { buildTrain, pruneTrains, trainStatus, type TrainOut } from './ship-queue-train'; -import { trainReady } from './ship-queue-train-ready'; +import { trainReady, verifyScreen } from './ship-queue-train-ready'; import { trainBisect, trainRebuild, trainVerdict } from './ship-queue-train-verdict'; import { verifyTrain } from './ship-queue-train-verify'; +import { trainWatch } from './ship-queue-train-watch'; +import { verifyPostMerge } from './ship-queue-post-merge'; +import { trainForget } from './ship-queue-train-forget'; import { redactCause } from './ship-queue-train-record'; -export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] [--urgent | --repair] [--dry-run] [--free-timeout ] +export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] [--urgent | --repair] [--dry-run [--stack-tests [--stack-tests-command ""]]] [--free-timeout ] --order: explicit slot order, every PR in --prs once (default: screened first, then READY order) train build --train resume a train (push, free tests, dispatch) from its record @@ -27,12 +30,27 @@ export const TRAIN_HELP = ` train build --prs @[,...] [--order ,...] train bisect --train --spend-cap [--dry-run] red cases on every prefix T0..Tk, resumable and capped train rebuild --train [--infra] the one rebuild an all-green bisect earns (or the one INFRA re-dispatch) - train prune [--train ] [--dry-run] - delete train/ branches of merged, superseded or dropped trains - ready --pr --screen [--broad] | --no-screen [--dry-run] - (train policy) READY plus the screen choice, recorded as a receipt + train watch --train [--timeout ] + resumable follow: runs the step the record asks for (build, status), waits for the + pending run, reads the verdict, prints NEXT; rerun it after a restart (no daemon) + train forget --train | --receipt @ [--dry-run] + remove a settled train record (attempts, bisect) or a READY receipt from + refs/notes/gstack-trains under an exact lease, then delete the local files + train prune [--train ] [--dry-run] [--stale] + delete train/ branches of merged, superseded or dropped trains; --stale also + settles merging or green trains whose every PR is merged or closed on GitHub, and + deletes screen/-* branches of merged or closed PRs + ready --pr --screen [--broad] | --no-screen [--dry-run] [--dispatch-ci [--timeout ]] + (train policy) READY plus the screen choice, recorded as a receipt; a head whose + screen run is RED is refused (QUEUE_SCREEN_RED). --dispatch-ci runs the named + workflows on the branch first (a conflicting PR has no pull_request CI), waits for + them and clears the matching requires-remote pregate lanes with the run URLs verify --pr --train [--no-fetch] (train policy) the merge check against a green train record + verify --pr --screen (train policy) the PR head's screen verdict read from its run: SCREEN_VERDICT and + FLAKY lines, recorded in the READY receipt + verify --post-merge [--timeout ] after a merge: wait for every check run at the merged commit to settle; + POST_MERGE: green|red|pending plus one CHECK: line per workflow (resumable) Merge trains need a "train" block in the base revision's ship policy (docs/ship-policy.md#run-a-merge-train).`; function flag(args: string[], name: string): string | undefined { @@ -80,14 +98,30 @@ export async function trainMain(ctx: QueueContext, args: string[]): Promise,... orders a new train from --prs; a resumed train keeps its recorded order'); + const stackCommand = flag(rest, '--stack-tests-command'); + if ((rest.includes('--stack-tests') || stackCommand !== undefined) && !dryRun) return usage('--stack-tests runs each PR\'s new test files on its integrated prefix; it needs --dry-run'); + const stackWords = stackCommand?.split(/\s+/).filter(Boolean); + if (stackCommand !== undefined && !stackWords?.length) return usage('--stack-tests-command needs a command'); return guarded('train build', () => buildTrain(ctx, { pins: flag(rest, '--prs'), order: flag(rest, '--order'), resume: nonce, dryRun, urgent: rest.includes('--urgent'), repair: rest.includes('--repair'), freeTimeoutSec, - pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, + pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, stackTests: stackWords?.length ? stackWords : rest.includes('--stack-tests') ? true : undefined, })); } if (sub === 'status') return guarded('train status', () => trainStatus(ctx, { train: nonce })); - if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce })); + if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce, stale: rest.includes('--stale') })); + if (sub === 'forget') { + const receipt = flag(rest, '--receipt'); + if (!nonce && !receipt) return usage('train forget needs --train or --receipt @'); + return guarded('train forget', () => trainForget(ctx, { train: nonce, receipt, dryRun })); + } if (!nonce) return usage(`train ${sub ?? ''} needs --train `); + if (sub === 'watch') { + const timeoutSec = positive(flag(rest, '--timeout'), 7200); + if (Number.isNaN(timeoutSec)) return usage('--timeout needs seconds'); + const freeTimeoutSec = positive(flag(rest, '--free-timeout'), 2700); + if (Number.isNaN(freeTimeoutSec)) return usage('--free-timeout needs seconds'); + return guarded('train watch', () => trainWatch(ctx, { train: nonce, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, freeTimeoutSec })); + } if (sub === 'verdict') return guarded('train verdict', () => trainVerdict(ctx, { train: nonce })); if (sub === 'rebuild') return guarded('train rebuild', () => trainRebuild(ctx, { train: nonce, infra: rest.includes('--infra') })); if (sub === 'bisect') { @@ -103,9 +137,24 @@ export async function readyMain(ctx: QueueContext, args: string[], base: { pr: s const noScreen = args.includes('--no-screen'); if (screen && noScreen) return usage('pass one of --screen or --no-screen'); if (args.includes('--broad') && !screen) return usage('--broad selects the screen policy; it needs --screen'); - return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run') })); + const dispatchCi = flag(args, '--dispatch-ci'); + const workflows = dispatchCi === undefined ? [] : dispatchCi.split(',').map(s => s.trim()).filter(Boolean); + if (dispatchCi !== undefined && (!workflows.length || workflows.some(w => !/^[\w.-]+\.ya?ml$/.test(w)))) return usage('--dispatch-ci names workflow files under .github/workflows, comma-separated (free-tests.yml,windows-free-tests.yml)'); + const timeoutSec = positive(flag(args, '--timeout'), 2700); + if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds'); + return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run'), + dispatchCi: workflows.length ? { workflows, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 } : undefined })); +} + +export async function postMergeMain(ctx: QueueContext, args: string[]): Promise { + const sha = flag(args, '--post-merge'); + if (!sha || !/^[0-9a-f]{40}$/.test(sha)) return usage('verify --post-merge needs the merged commit\'s full 40-hex SHA'); + const timeoutSec = positive(flag(args, '--timeout'), 2700); + if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds'); + return guarded('verify', () => verifyPostMerge(ctx, { sha, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 })); } export async function verifyMain(ctx: QueueContext, args: string[], pr: string): Promise { + if (args.includes('--screen')) return guarded('verify', () => verifyScreen(ctx, { pr })); return guarded('verify', () => verifyTrain(ctx, { pr, train: flag(args, '--train')!, noFetch: args.includes('--no-fetch') })); } diff --git a/lib/ship-queue-train-forget.ts b/lib/ship-queue-train-forget.ts new file mode 100644 index 000000000..99895e6b9 --- /dev/null +++ b/lib/ship-queue-train-forget.ts @@ -0,0 +1,56 @@ +/** + * `train forget --train | --receipt @ [--dry-run]`: remove a + * train record (its attempts and bisect ledger) or a READY receipt from + * refs/notes/gstack-trains under an exact lease, then delete the local files + * (until now a stale record needed a hand-run + * `git notes remove` plus a file hunt on every machine that ran the train). + * A rejected lease removes nothing (the local notes ref is restored too, so a + * later write cannot carry the removal without a lease). Other machines that + * still hold the local files can push the record back with their next write; + * the output says so. + */ +import * as fs from 'node:fs'; +import type { QueueContext } from './ship-queue'; +import { outOk, outRefused, trainEnv, type TrainOut } from './ship-queue-train'; +import { attemptsOf, forgetNotes, loadBisect, loadReady, loadTrain, noteName, readyPath, rebuildFromNotes, trainDir, type AnyRecord, type TrainState } from './ship-queue-train-record'; + +const Q = 'gstack-ship-queue'; +/** States with work still in flight (a branch being built or pushed, free tests or a gate running, a merge under way): forget never settles them. */ +const LIVE_STATES: readonly TrainState[] = ['built', 'pushed', 'free-pending', 'free-green', 'dispatch-unknown', 'dispatch-absent', 'gate-pending', 'merging']; + +export function trainForget(ctx: QueueContext, o: { train?: string; receipt?: string; dryRun: boolean }): TrainOut { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + rebuildFromNotes(ctx.repoRoot, env.remote); + const records: AnyRecord[] = []; + const files: string[] = []; + let what = ''; + if (o.train) { + const t = loadTrain(o.train); + if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + if (LIVE_STATES.includes(t.state)) { + return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `train ${t.nonce} is ${t.state}; forget settles nothing that is still building, running or merging (train prune --stale for a train whose PRs merged elsewhere)` }, [], `${Q} train status --train ${t.nonce}`, trainDir(t.nonce)); + } + const bisect = loadBisect(t.nonce); + records.push(t, ...attemptsOf(t), ...(bisect ? [bisect] : [])); + files.push(trainDir(t.nonce)); + what = `train ${t.nonce} (${t.state}; ${t.attempts.length} attempt(s)${bisect ? ', a bisect ledger' : ''})`; + } else { + const m = /^#?(\d+)@([0-9a-f]{40,64})$/.exec(o.receipt ?? ''); + if (!m) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: '--receipt takes @' }, [], `${Q} train forget --receipt @`, 'none'); + const r = loadReady(env.repo.keys, m[1]!, m[2]!); + if (!r) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no READY receipt for #${m[1]} at ${m[2]!.slice(0, 12)} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + records.push(r); + files.push(readyPath(r.repo_key, r.pr, r.head)); + what = `READY receipt #${r.pr} at ${r.head.slice(0, 12)} (screen ${r.screen.choice}${r.screen.run ? `, run ${r.screen.run}` : ''})`; + } + const names = records.map(noteName); + const lines = [`FORGET: ${what}`, ...names.map(n => ` note ${n}`), ...files.map(f => ` local ${f}`)]; + if (o.dryRun) return outOk([...lines, 'DRY-RUN: nothing removed'], `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`}`, 'none'); + const f = forgetNotes(ctx.repoRoot, names, env.remote); + if (!f.pushed) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the notes ref was not updated (${f.detail}); nothing deleted locally` }, lines, `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`} (the lease is re-read each run)`, 'none'); + for (const file of files) fs.rmSync(file, { recursive: true, force: true }); + lines.push(`FORGOTTEN: ${f.removed.length} note(s) removed${f.missing.length ? `, ${f.missing.length} already absent` : ''}; ${f.detail}; ${files.length} local path(s) deleted`, + 'note: a machine that still holds these files locally can push the record back with its next write; run train forget there too, or clear its $GSTACK_STATE_ROOT/ship-queue copy'); + return outOk(lines, `${Q} train status`, 'none'); +} diff --git a/lib/ship-queue-train-github.ts b/lib/ship-queue-train-github.ts index 952f0a101..592a125b5 100644 --- a/lib/ship-queue-train-github.ts +++ b/lib/ship-queue-train-github.ts @@ -57,14 +57,21 @@ export function asRun(summary: RunSummary): { id: number; url: string } { return { id: summary.databaseId, url: summary.url }; } -/** One JSON artifact of a run (the named file, else the first .json file it holds); null when it is missing or unreadable. */ +/** + * One JSON artifact of a run: the named file wherever the artifact keeps it + * (a multi-path upload keeps its directories, so report-verdict-a1 holds + * paid-report/collector-outcomes.json), else the first .json file at its + * root; null when it is missing or unreadable. + */ export function downloadJson(ctx: QueueContext, slug: string, run: number, name: string, scratch: string, file?: string): unknown { const dir = path.join(scratch, `${run}-${name}`); fs.rmSync(dir, { recursive: true, force: true }); fs.mkdirSync(dir, { recursive: true }); try { if (!ctx.gh(['run', 'download', String(run), '--repo', slug, '-n', name, '-D', dir]).ok) return null; - const pick = file ?? fs.readdirSync(dir).find(f => f.endsWith('.json')); + const pick = file + ? (fs.readdirSync(dir, { recursive: true }) as string[]).filter(f => path.basename(f) === file).sort((a, b) => a.length - b.length)[0] + : fs.readdirSync(dir).find(f => f.endsWith('.json')); return pick && fs.existsSync(path.join(dir, pick)) ? JSON.parse(fs.readFileSync(path.join(dir, pick), 'utf8')) : null; } catch { return null; } finally { fs.rmSync(dir, { recursive: true, force: true }); } } diff --git a/lib/ship-queue-train-ready.ts b/lib/ship-queue-train-ready.ts index 66730e266..736b74809 100644 --- a/lib/ship-queue-train-ready.ts +++ b/lib/ship-queue-train-ready.ts @@ -2,13 +2,21 @@ * `ready --pr --screen | --no-screen` under a ship policy with a `train` * block (docs/ship-policy.md#run-a-merge-train; plan GSTA-49 T1). The * READY block is today's (lib/ship-queue.ts ready()); this adds the screen - * selection and Garry's default recommendation (screen when the screen + * selection and the default recommendation (screen when the screen * selects a skill's own cases), requires an explicit choice * (READY_SCREEN_CHOICE_REQUIRED), dispatches a chosen screen through * eval-bg's nonce with its purpose and policy in the key, and records the * choice as a READY receipt (local state and refs/notes/gstack-trains) that * `train build` requires. The screen's dollar price is left to the T0 cost * replay (lib/gate-cost.ts) and printed as pending until it lands. + * + * The screen's verdict comes from its run (lib/ship-queue-screen.ts): a head + * whose recorded screen is RED is refused (QUEUE_SCREEN_RED) under either + * choice, and a red run is never re-dispatched at its merge SHA. The screen + * is planned on GitHub's refs/pull//merge when that merges the head, else + * on a scratch merge of the head onto origin/ built with the train's + * merge rules and pushed as screen/- (#3465); only a real + * conflict is QUEUE_SCREEN_STALE_REF. */ import { randomBytes } from 'node:crypto'; import { DISPATCH_INPUTS, awaitNonceRun, dedupeKey, dispatchWorkflow, findDuplicateRun, listDispatchRuns, makeNonce, type WorkflowDispatch } from '../scripts/eval-bg'; @@ -16,13 +24,19 @@ import { EXIT } from './headless-artifacts'; import { ready, type QueueContext } from './ship-queue'; import { classifyTrainHead } from './version-gate-train'; import { git } from './ship-queue-gate'; -import { execFor, hollowScreen, repoSlug } from './ship-queue-train-github'; -import { loadReady, nowIso, persist, readyPath, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record'; -import { outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train'; +import { dispatchCi } from './ship-queue-dispatch-ci'; +import { recordVerdict, screenVerdict, verdictLines, type ScreenOutcome } from './ship-queue-screen'; +import { execFor, repoSlug } from './ship-queue-train-github'; +import { loadReady, nowIso, persist, readyPath, rebuildFromNotes, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record'; +import { fetchBase, mergeCommit, outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train'; const Q = 'gstack-ship-queue'; -export interface ReadyOptions { pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean } +export interface ReadyOptions { + pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean; + /** `--dispatch-ci`: run these workflows on the PR branch, wait, and clear the matching requires-remote pregate lanes before reading the head. */ + dispatchCi?: { workflows: string[]; timeoutSec: number; pollMs: number }; +} export interface ScreenSelection { lines: string[]; recommend: 'screen' | 'no-screen' | null } /** Skill directories a diff touches: top-level directories that hold a SKILL.md.tmpl at the head. */ @@ -70,8 +84,46 @@ async function screenSelection(env: TrainEnv, head: string, baseSha: string): Pr return { ...s, lines: [...s.lines, price] }; } +/** The scratch branch a screen's merge commit is pushed to when GitHub's merge ref lags the head. */ +export const screenBranch = (pr: string, head: string) => `screen/${pr}-${head.slice(0, 12)}`; + +/** + * The merge commit the screen tests: GitHub's refs/pull//merge when it + * merges the PR head, else a scratch merge of the head onto origin/ + * (the train's merge rules) pushed as screen/-, so evals.yml can + * check it out and read its parents. A conflict beyond the CHANGELOG and the + * stamp files is the only QUEUE_SCREEN_STALE_REF left. + */ +function screenMerge(env: TrainEnv, o: ReadyOptions, head: string, baseSha: string): { mergeSha: string; branch?: string; line: string } | TrainOut { + const { ctx } = env; + const ls = git(ctx.repoRoot, ['ls-remote', env.remote, `refs/pull/${o.pr}/merge`]); + if (!ls.ok) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `could not list ${env.remote}'s refs: ${(ls.err || ls.out).trim().replace(/\s+/g, ' ').slice(0, 200) || 'git ls-remote failed'}` }, [], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + const github = ls.out.split(/\s/)[0] ?? ''; + let lag = `${env.remote} has no refs/pull/${o.pr}/merge`; + if (/^[0-9a-f]{40,64}$/.test(github)) { + const fetched = git(ctx.repoRoot, ['fetch', '-q', env.remote, `refs/pull/${o.pr}/merge`]); + const parent = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^2`]).out : ''; + const onto = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^1`]).out : ''; + if (parent === head && onto === baseSha) return { mergeSha: github, line: `SCREEN_MERGE: GitHub's refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges the head ${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)}` }; + lag = !fetched.ok ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} could not be fetched` + : parent !== head ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges ${parent ? parent.slice(0, 12) : 'no PR head'}, not ${head.slice(0, 12)}` + : `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges onto ${onto ? onto.slice(0, 12) : 'no base'}, not ${ctx.base}@${baseSha.slice(0, 12)}`; + } + const m = mergeCommit(ctx.repoRoot, baseSha, head, env.policy.policy, `screen #${o.pr}: merge ${head.slice(0, 12)} onto ${ctx.base} ${baseSha.slice(0, 12)}`); + if ('conflicts' in m) { + return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `#${o.pr} conflicts with ${ctx.base}@${baseSha.slice(0, 12)} beyond the CHANGELOG and the stamp files (${m.conflicts.join(', ')}); GitHub's merge ref lags too (${lag})` }, [], + `resolve the conflict with ${ctx.base}, push, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + } + const branch = screenBranch(o.pr, head); + const line = `SCREEN_MERGE: scratch merge ${m.commit.slice(0, 12)} of #${o.pr}@${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)} (${lag})`; + if (o.dryRun) return { mergeSha: m.commit, branch, line: `${line}; would push ${branch} (dry run)` }; + const p = git(ctx.repoRoot, ['push', '-q', '-f', env.remote, `${m.commit}:refs/heads/${branch}`]); + if (!p.ok) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `could not push the screen's scratch merge as ${branch}: ${p.err.trim().replace(/\s+/g, ' ').slice(0, 200)}` }, [line], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + return { mergeSha: m.commit, branch, line: `${line}; pushed ${branch}` }; +} + /** Dispatch (or adopt) the PR's screen on the base branch: inputs name the PR, its merge SHA and the screen policy. */ -async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; line: string } | TrainOut> { +async function dispatchScreen(env: TrainEnv, o: ReadyOptions, slug: string | null, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; lines: string[] } | TrainOut> { const policy = o.broad ? 'broad' : 'narrow'; const scope = { purpose: 'screen' as const, policy: policy as 'narrow' | 'broad' }; const nonce = makeNonce('pr', mergeSha, baseSha, randomBytes(3).toString('hex'), scope); @@ -80,23 +132,45 @@ async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string, [DISPATCH_INPUTS.baseSha]: baseSha, [DISPATCH_INPUTS.screenPr]: o.pr, [DISPATCH_INPUTS.screenMergeSha]: mergeSha, [DISPATCH_INPUTS.screenPolicy]: policy, } }; const inputs = Object.entries(dispatch.inputs).map(([k, v]) => `${k}=${v}`).join(' '); - if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)` }; - const slug = repoSlug(env.ctx); + if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)`] }; if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, [], `${Q} ready --pr ${o.pr} --screen`, 'none'); const exec = execFor(env.ctx); const existing = findDuplicateRun(listDispatchRuns(dispatch.workflow, exec, slug) ?? [], dedupeKey('pr', mergeSha, baseSha, scope)); - const plan = existing ? hollowScreen(env.ctx, slug, existing.databaseId) : null; - const hollow = plan?.kind === 'hollow' ? plan.reason : null; - if (existing && !hollow) return { choice: { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)` }; + const prior: ScreenOutcome | null = existing ? screenVerdict(env.ctx, slug, existing.databaseId) : null; + if (existing && prior) { + const choice: ScreenChoice = { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha }; + recordVerdict(choice, prior); + const lines = verdictLines(o.pr, existing.databaseId, prior); + if (prior.kind === 'red') return outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at merge ${mergeSha.slice(0, 12)} (run ${existing.databaseId}) is RED on ${prior.red.join(', ') || 'its gate binding'}; a red screen is not re-dispatched at the same merge SHA` }, + lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none'); + if (prior.kind !== 'not-run' && prior.kind !== 'incomplete') { + return { choice, lines: [...lines, `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)`] }; + } + } + const notFollowing = existing && prior ? ` (not following ${prior.detail}: a screen that ${prior.kind === 'not-run' ? 'planned nothing' : 'bound no verdict'} is not a screen)` : ''; const found = dispatchWorkflow(dispatch, env.ctx.base, exec, slug) ? await awaitNonceRun(dispatch.workflow, nonce, exec, slug) : null; if (!found) return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `no ${dispatch.workflow} run named ${nonce} appeared after the screen dispatch` }, [], `${Q} ready --pr ${o.pr} --screen${o.broad ? ' --broad' : ''} (adopts the run by its key once it appears)`, 'none'); - return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${hollow ? ` (not following ${hollow}: a screen that planned nothing is not a screen)` : ''}` }; + return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${notFollowing}`] }; } +const screenLine = (c: ScreenChoice) => `screen: ${c.choice}${c.policy ? ` policy=${c.policy} nonce=${c.nonce} run=${c.run ?? '-'}${c.verdict ? ` verdict=${c.verdict.outcome}` : ''}` : ''}`; + export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise { + const fetchErr = fetchBase(ctx); const env = trainEnv(ctx); if ('code' in env) return outRefused(env, [], 'none', 'none'); + if (fetchErr) return outRefused({ code: 'POLICY_SOURCE_UNAVAILABLE', message: fetchErr }, [], 'none', 'none'); + const ciLines: string[] = []; + if (o.dispatchCi && !o.dryRun) { + const d = await dispatchCi(ctx, { pr: o.pr, pregate: o.pregate, ...o.dispatchCi }); + ciLines.push(...d.lines); + if ('failure' in d) { + const again = `${Q} ready --pr ${o.pr} ${o.screen === 'screen' ? '--screen' : '--no-screen'} --dispatch-ci ${o.dispatchCi.workflows.join(',')} --timeout ${o.dispatchCi.timeoutSec}`; + return (d.failure.code === 'QUEUE_TRAIN_PENDING' ? outPending : outRefused)(d.failure, ciLines, d.failure.code === 'QUEUE_TRAIN_PENDING' ? again : `read the run, repair, push, run /ship, then ${again}`, 'none'); + } + } const base = ready(ctx, o); + base.lines.unshift(...ciLines); if (!base.ok) { const out = outRefused(base.failure!, base.lines, `fix the refusal, then ${Q} ready --pr ${o.pr} --screen|--no-screen`, base.run); return ['QUEUE_VERIFY_MISMATCH', 'QUEUE_STAMP_NOT_STAMP_ONLY'].includes(base.failure!.code) ? { ...out, exit: EXIT.fail } : out; @@ -107,6 +181,18 @@ export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise l.startsWith('READY ')) + 1, 0, screenLine(choice)); const now = nowIso(); - const readyAt = loadReady(env.repo.keys, o.pr, head)?.ready_at ?? now; - const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: readyAt, screen: choice, updated_at: now }; + const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: prior?.ready_at ?? now, screen: choice, updated_at: now }; if (o.dryRun) return outOk([...lines, 'DRY-RUN: no READY receipt written, nothing dispatched'], `${Q} ready --pr ${o.pr} --${o.screen}${o.broad ? ' --broad' : ''}`, 'none'); const p = persist(ctx.repoRoot, [receipt], { remote: env.remote }); lines.push(`READY_RECEIPT: ${readyPath(env.repo.key, o.pr, head)} (${p.detail})`); return outOk(lines, `send the READY block; the coordinator builds the train: ${Q} train build --prs ${o.pr}@${head},...`, readyPath(env.repo.key, o.pr, head)); } + +/** + * `verify --pr --screen`: the screen verdict of the PR's pushed head from + * its run, recorded in the READY receipt; never a lane's prose. + */ +export function verifyScreen(ctx: QueueContext, o: { pr: string }): TrainOut { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + const view = ctx.gh(['pr', 'view', o.pr, '--json', 'headRefOid']); + let head = ''; + try { head = view.ok ? String(JSON.parse(view.out).headRefOid ?? '') : ''; } catch {} + if (!/^[0-9a-f]{40,64}$/.test(head)) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `cannot read PR #${o.pr}'s head through gh (${(view.err ?? view.out).trim().slice(0, 120) || 'no headRefOid'})` }, [], `${Q} verify --pr ${o.pr} --screen`, 'none'); + if (!loadReady(env.repo.keys, o.pr, head)) rebuildFromNotes(ctx.repoRoot, env.remote); + const receipt = loadReady(env.repo.keys, o.pr, head); + const run = receipt ? readyPath(receipt.repo_key, o.pr, head) : 'none'; + if (!receipt) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${o.pr} at ${head.slice(0, 12)} (local state or refs/notes/gstack-trains)` }, [], `the lane runs ${Q} ready --pr ${o.pr} --screen|--no-screen`, 'none'); + const lines = [`READY: #${o.pr} ${head} ready_at=${receipt.ready_at}`, screenLine(receipt.screen)]; + if (receipt.screen.choice !== 'screen' || !receipt.screen.run) return outOk([...lines, `SCREEN_VERDICT: #${o.pr} none (${receipt.screen.choice === 'screen' ? 'no run recorded' : 'READY --no-screen'})`], `${Q} train build --prs ${o.pr}@${head},...`, run); + const slug = repoSlug(ctx); + if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, lines, `${Q} verify --pr ${o.pr} --screen`, run); + const read = screenVerdict(ctx, slug, receipt.screen.run); + lines.push(...verdictLines(o.pr, receipt.screen.run, read)); + if (recordVerdict(receipt.screen, read)) { receipt.updated_at = nowIso(); lines.push(`READY_RECEIPT: ${run} (${persist(ctx.repoRoot, [receipt], { remote: env.remote }).detail}; verdict recorded)`); } + const again = `${Q} verify --pr ${o.pr} --screen`; + switch (read.kind) { + case 'green': return outOk(lines, `${Q} train build --prs ${o.pr}@${head},...`, run); + case 'pending': return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `the screen run ${receipt.screen.run} has not finished (${read.detail})` }, lines, again, run); + case 'red': return { ...outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at ${head.slice(0, 12)} (run ${receipt.screen.run}) is RED on ${read.red.join(', ') || 'its gate binding'}` }, lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, run), exit: EXIT.fail }; + case 'not-run': return outRefused({ code: 'QUEUE_SCREEN_NOT_RUN', message: `the screen of #${o.pr} (run ${receipt.screen.run}) concluded success with no slice: ${read.detail}; it is not a screen, and this head is not screened` }, lines, `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)`, run); + default: return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the screen of #${o.pr} (run ${receipt.screen.run}) has no verdict to read: ${read.detail}` }, lines, read.kind === 'incomplete' ? `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)` : again, run); + } +} diff --git a/lib/ship-queue-train-record.ts b/lib/ship-queue-train-record.ts index 16b4ac547..07e356791 100644 --- a/lib/ship-queue-train-record.ts +++ b/lib/ship-queue-train-record.ts @@ -20,11 +20,18 @@ const NOTES_REMOTE_REF = 'refs/notes/gstack-trains-origin'; export type TrainState = | 'built' | 'pushed' | 'free-pending' | 'free-red' | 'free-green' | 'dispatch-unknown' | 'dispatch-absent' | 'gate-pending' - | 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded'; -/** States after which a train never dispatches again; prune deletes their branches. */ -export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded']; + | 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded' | 'stale'; +/** States after which a train never dispatches again; prune deletes their branches (`stale`: every PR merged or closed outside the train, `train prune --stale`). */ +export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded', 'stale']; -export interface ScreenChoice { choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null } +/** What the screen run itself said (lib/ship-queue-screen.ts), recorded when it was read; absent while the run is pending or unread. */ +export interface ScreenVerdictRecord { outcome: 'GREEN' | 'RED' | 'NOT_RUN' | 'INCOMPLETE'; run: number; red: string[]; flaky: string[]; detail: string; read_at: string } +export interface ScreenChoice { + choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null; + /** The scratch branch `ready --screen` pushed when GitHub's merge ref lagged the head (docs/ship-policy.md#run-a-merge-train). */ + branch?: string; + verdict?: ScreenVerdictRecord; +} export interface TrainSlot { pr: string; head: string; ready_at: string; screen: ScreenChoice } export interface TrainPrefix { index: number; commit: string; tree: string; pr: string | null } export interface FreeRun { run: number | null; url: string | null; head_sha: string | null; conclusion: string | null; verdict: 'green' | 'red' | 'missing' | 'stale' | 'cancelled' | 'pending' } @@ -154,7 +161,7 @@ export function recordPath(r: AnyRecord): string { return readyPath(r.repo_key, r.pr, r.head); } -function noteName(r: AnyRecord): string { +export function noteName(r: AnyRecord): string { if (r.kind === 'train') return `gstack-trains:train:${r.nonce}`; if (r.kind === 'train-attempt') return `gstack-trains:attempt:${r.train}:${r.id}`; if (r.kind === 'train-bisect') return `gstack-trains:bisect:${r.train}`; @@ -227,6 +234,35 @@ function parseNote(text: string): AnyRecord | null { return null; } +/** + * Remove records from the notes ref (`train forget`): each named note is + * removed locally, then the ref is pushed under an exact lease on the remote + * tip this call fetched, so a concurrent writer's push is never overwritten + * (a rejected lease returns `pushed: false` and nothing is deleted locally). + * Names are note names (noteName); a name with no note is reported missing. + */ +export function forgetNotes(repoRoot: string, names: string[], remote = 'origin'): { removed: string[]; missing: string[]; failed: string[]; pushed: boolean; lease: string | null; detail: string } { + const f = fetchNotes(repoRoot, remote); + if (!f.ok) return { removed: [], missing: [], failed: [], pushed: false, lease: null, detail: f.detail }; + const lease = git(repoRoot, ['rev-parse', '--verify', '--quiet', NOTES_REMOTE_REF]).out || null; + const before = git(repoRoot, ['rev-parse', '--verify', '--quiet', NOTES_REF]).out || null; + const listed = git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'list']).out.split('\n').map(l => l.split(' ')[1]).filter(Boolean); + const removed: string[] = []; + const missing: string[] = []; + const failed: string[] = []; + const restore = () => { if (before) git(repoRoot, ['update-ref', NOTES_REF, before]); }; + for (const name of names) { + const blob = git(repoRoot, ['hash-object', '--stdin'], undefined, name).out; + if (!listed.includes(blob)) { missing.push(name); continue; } + if (git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'remove', blob], notesEnv()).ok) removed.push(name); else failed.push(name); + } + if (failed.length) { restore(); return { removed: [], missing, failed, pushed: false, lease, detail: `git notes remove failed for ${failed.join(', ')}; the local notes ref was restored` }; } + if (!removed.length) return { removed, missing, failed, pushed: false, lease, detail: 'no note to remove' }; + const p = git(repoRoot, ['push', '-q', lease ? `--force-with-lease=${NOTES_REF}:${lease}` : `--force-with-lease=${NOTES_REF}`, remote, `${NOTES_REF}:${NOTES_REF}`]); + if (!p.ok) restore(); + return { removed, missing, failed, pushed: p.ok, lease, detail: p.ok ? `notes pushed to ${remote} ${NOTES_REF} (lease ${lease?.slice(0, 12) ?? 'new ref'})` : `git push ${NOTES_REF} rejected: ${p.err.replace(/\s+/g, ' ').trim().slice(0, 160)}; the local notes ref was restored` }; +} + /** Rebuild local records from the notes ref: a note whose record is missing locally, or newer than the local copy, is written. */ export function rebuildFromNotes(repoRoot: string, remote = 'origin'): { restored: number; read: number; detail: string } { const f = fetchNotes(repoRoot, remote); diff --git a/lib/ship-queue-train-watch.ts b/lib/ship-queue-train-watch.ts new file mode 100644 index 000000000..160cb3dba --- /dev/null +++ b/lib/ship-queue-train-watch.ts @@ -0,0 +1,68 @@ +/** + * `train watch --train [--timeout ]`: the controller's one + * resumable follow command. Machines restart and kill + * background `gh run watch` chains, so watch keeps no process state: it + * reads the train record, runs the step the state asks for (train build + * --train to push, run the free tests and dispatch the gate; train status to + * settle an unconfirmed dispatch), waits for the pending run to finish by + * polling gh, then reads the verdict, and prints the last step's NEXT. Run it + * again after an interruption and it continues from the record. It never + * dispatches anything the record would not dispatch on its own. + */ +import type { QueueContext } from './ship-queue'; +import { buildTrain, findTrain, nextFor, outOk, outPending, outRefused, trainEnv, trainStatus, type TrainOut } from './ship-queue-train'; +import { repoSlug, runView } from './ship-queue-train-github'; +import { attemptsOf, trainFile, type TrainRecord } from './ship-queue-train-record'; +import { trainVerdict } from './ship-queue-train-verdict'; + +const Q = 'gstack-ship-queue'; +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +export interface WatchOptions { train: string; timeoutSec: number; pollMs: number; freeTimeoutSec: number } + +/** Wait for the train's current gate run to complete; null when it did, else why the wait stopped. */ +async function awaitGate(ctx: QueueContext, t: TrainRecord, deadline: number, pollMs: number): Promise { + const a = attemptsOf(t).filter(x => x.purpose !== 'bisect').at(-1); + if (!a?.run) return `no gate run is recorded for ${t.nonce}`; + const slug = repoSlug(ctx); + if (!slug) return 'gh repo view could not name the repository'; + for (;;) { + const view = runView(ctx, slug, a.run.id); + if (view?.status === 'completed') return null; + if (Date.now() >= deadline) return `run ${a.run.id} is still ${view?.status ?? 'unreadable'} at the watch timeout`; + await sleep(pollMs); + } +} + +export async function trainWatch(ctx: QueueContext, o: WatchOptions): Promise { + const env = trainEnv(ctx); + if ('code' in env) return outRefused(env, [], 'none', 'none'); + let t = findTrain(env, o.train); + if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none'); + const run = trainFile(t.nonce); + const deadline = Date.now() + o.timeoutSec * 1000; + const lines: string[] = [`WATCH: train ${t.nonce} state=${t.state} (resumable; rerun this command after an interruption)`]; + const again = `${Q} train watch --train ${t.nonce} --timeout ${o.timeoutSec}`; + const step = (label: string, out: TrainOut): TrainOut => ({ ...out, lines: [...lines, `WATCH: ${label}`, ...out.lines] }); + const reload = () => { t = findTrain(env, o.train) ?? t; }; + if (t.state === 'dispatch-unknown') { + const s = trainStatus(ctx, { train: t.nonce }); + lines.push(`WATCH: train status settled the unconfirmed dispatch`, ...s.lines.filter(l => l.startsWith('ATTEMPT:'))); + reload(); + } + if (['built', 'pushed', 'free-pending', 'free-green', 'dispatch-absent'].includes(t.state)) { + const remaining = Math.max(1, Math.floor((deadline - Date.now()) / 1000)); + const b = await buildTrain(ctx, { resume: t.nonce, dryRun: false, urgent: false, repair: false, freeTimeoutSec: Math.min(o.freeTimeoutSec, remaining), pollMs: o.pollMs }); + reload(); + if (b.status !== 'complete') return step(`train build --train ${t.nonce} stopped (${b.failure?.code ?? b.status})`, b.failure?.code === 'QUEUE_TRAIN_PENDING' ? { ...b, next: again } : b); + lines.push(`WATCH: train build --train ${t.nonce}: free tests green, gate dispatched`); + } + if (t.state === 'gate-pending') { + const why = await awaitGate(ctx, t, deadline, o.pollMs); + if (why) return outPending({ code: 'QUEUE_TRAIN_PENDING', message: why }, lines, again, run); + lines.push('WATCH: gate run completed; reading the verdict'); + return step(`train verdict --train ${t.nonce}`, trainVerdict(ctx, { train: t.nonce })); + } + if (t.state === 'dispatch-unknown') return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `the gate dispatch for ${t.nonce} is still unconfirmed` }, lines, again, run); + return outOk([...lines, `WATCH: nothing to wait for in state ${t.state}`], nextFor(t), run); +} diff --git a/lib/ship-queue-train.ts b/lib/ship-queue-train.ts index 698b78817..48c13cd80 100644 --- a/lib/ship-queue-train.ts +++ b/lib/ship-queue-train.ts @@ -23,7 +23,9 @@ import type { LoadedPolicy, ShipPolicy, TrainPolicy } from './ship-policy'; import { loadBasePolicy, type QueueContext } from './ship-queue'; import { bindReceipt } from './ship-receipt'; import { git, sha256, withIndex, type QueueFailure } from './ship-queue-gate'; -import { asRun, execFor, hollowScreen, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github'; +import { recordVerdict, screenVerdict, verdictLines } from './ship-queue-screen'; +import { stackTestLine, stackTests } from './ship-queue-stack-tests'; +import { asRun, execFor, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github'; import { TERMINAL_STATES, activeFreeze, attemptsOf, listTrains, loadBisect, loadReady, loadTrain, nowIso, persist, rebuildFromNotes, repoIdentity, trainFile, type AttemptPurpose, type AttemptRecord, type TrainRecord, type TrainSlot, @@ -51,7 +53,7 @@ export function trainEnv(ctx: QueueContext): TrainEnv | QueueFailure { export function fetchBase(ctx: QueueContext): string | null { const remote = ctx.remote ?? 'origin'; const f = git(ctx.repoRoot, ['fetch', '--no-tags', '-q', remote, `+refs/heads/${ctx.base}:refs/remotes/${remote}/${ctx.base}`]); - return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.slice(0, 160)}`; + return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.replace(/\s+/g, ' ').trim().slice(0, 160)}`; } /** Make the train's prefix commits local (another machine built them): fetch train/. */ @@ -195,17 +197,28 @@ export function mergeStep(repoRoot: string, pred: string, head: string, policy: const TRAIN_IDENTITY = { GIT_AUTHOR_NAME: 'gstack-ship-queue', GIT_AUTHOR_EMAIL: 'ship-queue@gstack.local', GIT_COMMITTER_NAME: 'gstack-ship-queue', GIT_COMMITTER_EMAIL: 'ship-queue@gstack.local', GIT_AUTHOR_DATE: '2000-01-01T00:00:00Z', GIT_COMMITTER_DATE: '2000-01-01T00:00:00Z' }; +/** + * One train merge as a commit with a fixed identity and date: parents are the + * predecessor (first, as GitHub's merge ref) and the PR head. Prefix commits + * and the scratch merge a screen is planned on (`ready --screen` when + * GitHub's refs/pull//merge lags the head) are both built here. + */ +export function mergeCommit(repoRoot: string, pred: string, head: string, policy: ShipPolicy, message: string): { commit: string; tree: string } | { conflicts: string[] } { + const m = mergeStep(repoRoot, pred, head, policy); + if ('conflicts' in m) return m; + const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', head, '-m', message], { ...process.env, ...TRAIN_IDENTITY }); + return c.ok ? { commit: c.out, tree: m.tree } : { conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`] }; +} + /** Build T1..Tk on T0 = base; a conflict stops at the PR that conflicts. */ export function buildChain(repoRoot: string, baseSha: string, slots: TrainSlot[], policy: ShipPolicy): { prefixes: TrainRecord['prefixes'] } | { stoppedAt: string; conflicts: string[]; merged: string[] } { const prefixes: TrainRecord['prefixes'] = [{ index: 0, commit: baseSha, tree: git(repoRoot, ['rev-parse', `${baseSha}^{tree}`]).out, pr: null }]; for (const [i, slot] of slots.entries()) { const pred = prefixes[i]!.commit; - const m = mergeStep(repoRoot, pred, slot.head, policy); + const m = mergeCommit(repoRoot, pred, slot.head, policy, `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`); if ('conflicts' in m) return { stoppedAt: slot.pr, conflicts: m.conflicts, merged: slots.slice(0, i).map(s => s.pr) }; - const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', slot.head, '-m', `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`], { ...process.env, ...TRAIN_IDENTITY }); - if (!c.ok) return { stoppedAt: slot.pr, conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`], merged: slots.slice(0, i).map(s => s.pr) }; - prefixes.push({ index: i + 1, commit: c.out, tree: m.tree, pr: slot.pr }); + prefixes.push({ index: i + 1, commit: m.commit, tree: m.tree, pr: slot.pr }); } return { prefixes }; } @@ -266,7 +279,11 @@ export function gateDispatch(env: TrainEnv, train: TrainRecord, nonce: string): return { workflow: env.train.gate_workflow, inputs: { [DISPATCH_INPUTS.evalsAll]: 'true', [DISPATCH_INPUTS.expectedSha]: train.head, [DISPATCH_INPUTS.nonce]: nonce } }; } -export interface BuildOptions { pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number } +export interface BuildOptions { + pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number; + /** `--dry-run --stack-tests`: run each PR's new test files on its integrated prefix (lib/ship-queue-stack-tests.ts); the command's words when `--stack-tests-command` names one. */ + stackTests?: string[] | true; +} /** `--order ,,...`: every pinned PR exactly once (a leading # allowed). */ export function parseOrder(spec: string | undefined, pins: Array<{ pr: string }>): string[] | undefined | { error: string } { @@ -279,7 +296,7 @@ export function parseOrder(spec: string | undefined, pins: Array<{ pr: string }> } /** The slots a new train would carry: trusted, unmoved, READY PRs, in slot order. */ -function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>): { slots: TrainSlot[]; lines: string[] } | TrainOut { +function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>, dryRun: boolean): { slots: TrainSlot[]; lines: string[] } | TrainOut { const { ctx } = env; const specs = pins.map(p => `+refs/pull/${p.pr}/head:refs/gstack/train/pr-${p.pr}`); const f = git(ctx.repoRoot, ['fetch', '--no-tags', '-q', env.remote, ...specs]); @@ -302,9 +319,16 @@ function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>): { if (!ready) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${p.pr} at ${info.headRefOid.slice(0, 12)} (local state or ${'refs/notes/gstack-trains'})` }, [], `the lane runs ${Q} ready --pr ${p.pr} --screen|--no-screen`, 'none'); let screen = ready.screen; if (screen.choice === 'screen' && screen.run && slug === undefined) slug = repoSlug(ctx); - const plan = screen.choice === 'screen' && screen.run ? (slug ? hollowScreen(ctx, slug, screen.run) : { kind: 'unknown' as const, reason: 'the repository slug is unreadable' }) : null; - if (plan?.kind === 'hollow') { lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; } - else if (plan?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as screened, unverified`); + const read = screen.choice === 'screen' && screen.run ? (slug ? screenVerdict(ctx, slug, screen.run) : { kind: 'unknown' as const, detail: 'the repository slug is unreadable' }) : null; + if (read && screen.run && recordVerdict(ready.screen, read) && !dryRun) { ready.updated_at = nowIso(); persist(ctx.repoRoot, [ready], { remote: env.remote }); } + if (read?.kind === 'red') { + return outRefused({ code: 'QUEUE_SCREEN_RED', message: `#${p.pr}'s screen run ${screen.run} is RED on ${read.red.join(', ') || 'its gate binding'} (${read.detail}); a red head is not seated` }, + [...lines, ...verdictLines(p.pr, screen.run!, read)], without ? `${Q} train build --prs ${without}` : 'none', 'none'); + } + if (read?.kind === 'not-run' || read?.kind === 'incomplete') { lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; } + else if (read?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as screened, unverified`); + else if (read?.kind === 'green') lines.push(`SCREEN: #${p.pr} run ${screen.run} GREEN${read.flaky.length ? ` (flaky: ${read.flaky.join(', ')})` : ''}`); + else if (read?.kind === 'pending') lines.push(`SCREEN: #${p.pr} run ${screen.run} still running; counted as screened`); slots.push({ pr: p.pr, head: info.headRefOid, ready_at: ready.ready_at, screen }); } return { slots, lines }; @@ -321,7 +345,7 @@ async function newTrain(env: TrainEnv, o: BuildOptions): Promise<{ t: TrainRecor frozenLines(freeze), `${Q} train build --repair --prs @`, 'none'); const explicit = parseOrder(o.order, pins); if (explicit && 'error' in explicit) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: explicit.error }, [], 'none', 'none'); - const admitted = admitSlots(env, pins); + const admitted = admitSlots(env, pins, o.dryRun); if ('exit' in admitted) return admitted; const order = orderSlots(admitted.slots, o.repair ? 1 : env.train.max_size, explicit); const lines = [ @@ -460,8 +484,18 @@ export async function buildTrain(ctx: QueueContext, o: BuildOptions): Promise !r.ok); + if (red.length) { + const again = `${Q} train build ${prsArgs(t.slots, t.explicit_order)} --dry-run --stack-tests`; + return outFailed({ code: 'QUEUE_TRAIN_STACK_RED', message: red.map(r => `#${r.pr}'s new tests fail on the integrated prefix T${r.prefix} (${r.files.join(', ')})`).join('; ') + '; they pass on the PR head alone only if nothing before it in the train changes what they read' }, + [...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `fix ${red.map(r => `#${r.pr}'s tests against T${r.prefix}`).join(' and ')}, or reorder with --order, then ${again}`, 'none'); + } + } + return outOk([...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `${Q} train build ${prsArgs(t.slots, t.explicit_order)}`, 'none'); } if (t.state === 'built' && !t.attempts.length && !loadTrain(t.nonce)) { effect(t, 'train.json written'); @@ -528,21 +562,64 @@ export function trainStatus(ctx: QueueContext, o: { train?: string }): TrainOut return outOk(lines, next, single ? trainFile(single.nonce) : 'none'); } -/** `train prune`: delete train/ on the remote for every finished (merged, superseded, dropped) train. */ -export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string }): TrainOut { +/** A PR's state on GitHub for the stale check: MERGED, CLOSED, OPEN, or unreadable. */ +function prState(ctx: QueueContext, pr: string): string { + const info = prInfo(ctx, pr); + return 'error' in info ? 'unreadable' : info.state.toUpperCase(); +} + +/** + * `--stale`: a merging or green train whose every PR is MERGED or CLOSED on + * GitHub (merged elsewhere, closed, or merged outside the train) never + * finishes on its own; it becomes `stale` (terminal) and is pruned, with + * each PR's state in the log. Any OPEN or unreadable PR keeps the train. + */ +function staleCheck(ctx: QueueContext, t: TrainRecord): { stale: true; detail: string } | { stale: false; detail: string } { + const unmerged = t.slots.filter(s => !t.merged.some(m => m.pr === s.pr)); + const states = unmerged.map(s => `#${s.pr} ${prState(ctx, s.pr)}`); + const open = states.filter(s => !/ (MERGED|CLOSED)$/.test(s)); + const detail = `${t.merged.map(m => `#${m.pr} merged by the train`).concat(states).join(', ')}`; + return open.length ? { stale: false, detail: `${open.join(', ')} still ${open.length === 1 ? 'needs' : 'need'} the train` } : { stale: true, detail }; +} + +/** `train prune`: delete train/ on the remote for every finished (merged, superseded, dropped, stale) train; `--stale` first settles merging and green trains whose PRs all merged or closed elsewhere, and the screen/-* branches of merged or closed PRs. */ +export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string; stale?: boolean }): TrainOut { const env = trainEnv(ctx); if ('code' in env) return outRefused(env, [], 'none', 'none'); rebuildFromNotes(ctx.repoRoot, env.remote); const lines: string[] = []; const rgit = (args: string[]) => git(ctx.repoRoot, args); + const remove = (branch: string, label: string, after?: () => void) => { + if (!remoteBranchSha(rgit, env.remote, branch)) { lines.push(`GONE: ${branch} (${label})`); return; } + if (o.dryRun) { lines.push(`WOULD-DELETE: ${branch} (${label})`); return; } + const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${branch}`]); + lines.push(d.ok ? `DELETED: ${branch} (${label})` : `FAILED: ${branch}: ${d.err.slice(0, 120)}`); + if (d.ok) after?.(); + }; for (const t of listTrains(env.repo.keys).filter(x => !o.train || x.nonce === o.train)) { + if (o.stale && (t.state === 'merging' || t.state === 'green')) { + const s = staleCheck(ctx, t); + if (s.stale) { + lines.push(`STALE: ${t.nonce} (${t.state}): ${s.detail}${o.dryRun ? '; would mark stale' : ''}`); + if (!o.dryRun) { t.state = 'stale'; effect(t, `marked stale by train prune --stale: ${s.detail}`); save(env, t); } + remove(t.branch, o.dryRun ? 'stale' : t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); }); + continue; + } + lines.push(`KEEP: ${t.branch} (${t.state}; ${s.detail})`); + continue; + } if (!TERMINAL_STATES.includes(t.state)) { lines.push(`KEEP: ${t.branch} (${t.state})`); continue; } - if (!remoteBranchSha(rgit, env.remote, t.branch)) { lines.push(`GONE: ${t.branch} (${t.state})`); continue; } - if (o.dryRun) { lines.push(`WOULD-DELETE: ${t.branch} (${t.state})`); continue; } - const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${t.branch}`]); - lines.push(d.ok ? `DELETED: ${t.branch} (${t.state})` : `FAILED: ${t.branch}: ${d.err.slice(0, 120)}`); - if (d.ok) { effect(t, `branch deleted: ${t.branch}`); save(env, t); } + remove(t.branch, t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); }); + } + if (o.stale && !o.train) { + const refs = git(ctx.repoRoot, ['ls-remote', env.remote, 'refs/heads/screen/*']); + for (const ref of refs.ok ? refs.out.split('\n') : []) { + const m = /refs\/heads\/(screen\/(\d+)-[0-9a-f]{12})$/.exec(ref.split(/\s/)[1] ?? ''); + if (!m) continue; + const state = prState(ctx, m[2]!); + if (state === 'MERGED' || state === 'CLOSED') remove(m[1]!, `#${m[2]} ${state}`); else lines.push(`KEEP: ${m[1]} (#${m[2]} ${state})`); + } } if (!lines.length) lines.push('PRUNE: no trains'); - return outOk(lines, o.dryRun ? `${Q} train prune` : 'none', 'none'); + return outOk(lines, o.dryRun ? `${Q} train prune${o.stale ? ' --stale' : ''}` : 'none', 'none'); } diff --git a/lib/ship-queue.ts b/lib/ship-queue.ts index e4b80b6b0..166a9a24b 100644 --- a/lib/ship-queue.ts +++ b/lib/ship-queue.ts @@ -146,20 +146,21 @@ export function latestRunPerName } { const at = sha.slice(0, 12); const runs = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/check-runs`, 'check_runs'); - if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false }; + if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false, checks: [] }; const statuses = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/status`, 'statuses'); - if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false }; + if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false, checks: [] }; + const checks = latestRunPerName(runs.items).map(c => ({ name: String(c.name ?? ''), status: String(c.status ?? ''), conclusion: c.conclusion === undefined || c.conclusion === null ? null : String(c.conclusion) })); const verdicts = [ - ...latestRunPerName(runs.items).map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'), + ...checks.map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'), ...statuses.items.map(s => s.state === 'success' ? 'passed' : s.state === 'pending' ? 'pending' : 'failed'), ]; const count = (v: string) => verdicts.filter(x => x === v).length; const [passed, failed, pending, skipped] = [count('passed'), count('failed'), count('pending'), count('skipped')]; const green = verdicts.length > 0 && pending === 0 && failed === 0; - return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0 }; + return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0, checks }; } /** diff --git a/package.json b/package.json index dee1c57e9..2ef96a72b 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.262", + "version": "1.91.263", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/lib/windows-curation.ts b/scripts/lib/windows-curation.ts index 5547efe53..772a4d663 100644 --- a/scripts/lib/windows-curation.ts +++ b/scripts/lib/windows-curation.ts @@ -295,6 +295,10 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }> file: 'test/ship-queue-train-cost.test.ts', reason: 'drives bin/gstack-ship-queue train verdict against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts; on windows-latest spawnSync(\'gh\') resolves the runner\'s gh.exe instead of the shim (windows-free-shard 2 on #3478 at d5e6a7d)', }, + { + file: 'test/ship-queue-train-ops.test.ts', + reason: 'drives bin/gstack-ship-queue (screen verdicts, scratch-merge screens, prune --stale, watch, forget, verify --post-merge, ready --dispatch-ci, --stack-tests) against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts', + }, { file: 'test/ship-policy-compat.test.ts', reason: 'spawns bin/gstack-ship-policy, bin/gstack-restamp and bin/gstack-tree-receipt (shebangs) through test/helpers/restamp-fixture.ts', diff --git a/test/helpers/ship-queue-train-fixture.ts b/test/helpers/ship-queue-train-fixture.ts index 87ac2286c..04d85ef65 100644 --- a/test/helpers/ship-queue-train-fixture.ts +++ b/test/helpers/ship-queue-train-fixture.ts @@ -31,13 +31,22 @@ export interface Rule { plan?: { planned: number } | { manifest: unknown } | null; /** Marks every artifact of runs dispatched under this rule expired in the artifacts listing. */ expired?: boolean; + /** A screen run's gate binding (gate-binding-a1/gate-binding.json, scripts/gate-reuse.ts) and its report's collector-outcomes.json (report-verdict-a1). */ + binding?: { verdict: 'GREEN' | 'RED'; reconcile_exit?: number; flaky?: string[] }; + outcomes?: { red?: string[]; flakyLines?: string[] }; + /** The run confirmed a verdict reuse (R2a/R2): paid-plan carries a valid gate-reuse.json naming this run and no binding is uploaded. */ + reuse?: { run: number; flaky?: string[] }; check?: string | null; + /** The check-run name a run dispatched under this rule attaches to its head (a push or workflow_dispatch run's job), with the run's status and conclusion. */ + ci?: string; bisect?: { prefixes: string[]; patterns: Record; costUsd?: number }; } export interface ShimData { repo: string; origin: string; nextId: number; prs: Record; runs: Array>; artifacts: Record>; checks: Record>; ci: Record; rules: Record; artifact: string; log: Array>; + /** Commit statuses (the legacy status API) at a sha, beside the check runs in `ci`. */ + statuses?: Record>; } export interface TrainFixture { repo: FixtureRepo; state: string; dataFile: string; env: Record; read: () => ShimData; write: (fn: (d: ShimData) => void) => void } @@ -163,6 +172,13 @@ function shimSource(): string { " if (rule.record && !inputs.prefix_shas) d.artifacts[id][d.artifact] = { schema: 1, kind: 'train-gate', nonce: rule.record.nonce || inputs.nonce, head_sha: head, tree: rev(head + '^{tree}'),", " verdict: rule.record.verdict || 'GREEN', census: rule.record.census, panels: rule.record.panels };", " if (rule.check && !inputs.prefix_shas) (d.checks[head] = d.checks[head] || []).push({ name: 'gstack/full-gate', status: 'completed', conclusion: rule.check });", + " if (rule.ci) (d.ci[head] = d.ci[head] || []).push({ name: rule.ci, status: run.status, conclusion: run.conclusion, id: id, completed_at: '2026-10-26T00:00:00Z' });", + " if (rule.binding) d.artifacts[id]['gate-binding-a1'] = { schema: 1, repository: d.repo, pr: Number(inputs.screen_pr) || 0, run_id: id, run_attempt: 1, head_sha: rev(head + '^2') || head, base_sha: rev(head + '^1') || head,", + " profile: 'pr', reconcile_exit: rule.binding.reconcile_exit === undefined ? (rule.binding.verdict === 'GREEN' ? 0 : 1) : rule.binding.reconcile_exit, verdict: rule.binding.verdict, manual_accepted: 0, selection_sha256: 'a'.repeat(64),", + " flaky_cases: rule.binding.flaky || [], purpose: 'screen', tested_sha: head, screen_policy: inputs.screen_policy || 'narrow' };", + " if (rule.reuse && d.artifacts[id]['paid-plan']) d.artifacts[id]['paid-plan'].__reuse = { schema: 1, kind: 'main-moved', pr: Number(inputs.screen_pr) || 0, reused_run_id: rule.reuse.run, reused_run_attempt: 1, gated_head: 'b'.repeat(40), current_head: 'c'.repeat(40),", + " gated_base: 'd'.repeat(40), base_sha: 'e'.repeat(40), selection_sha256: 'a'.repeat(64), why: 'same own diff on a moved main', gated_merge_base: 'f'.repeat(40), merge_base: '1'.repeat(40), own_diff_sha256: 'b'.repeat(64), reused_screen_policy: 'narrow', reused_flaky: rule.reuse.flaky || [] };", + " if (rule.outcomes) d.artifacts[id]['report-verdict-a1'] = { version: 2, panels: (rule.outcomes.red || []).map(c => ({ case: c, kind: 'judge', status: 'FAIL', failsLane: true, redClass: 'VERDICT' })), flakyLines: rule.outcomes.flakyLines || [] };", " if (rule.bisect && inputs.prefix_shas) for (const sha of inputs.prefix_shas.split(',')) {", " const at = rule.bisect.prefixes.indexOf(sha);", " d.artifacts[id][d.artifact + '-bisect-' + sha.slice(0, 12)] = { schema: 1, kind: 'train-bisect-prefix', nonce: inputs.nonce, prefix_sha: sha, checked_out_sha: sha,", @@ -178,8 +194,9 @@ function shimSource(): string { "if (a[0] === 'run' && a[1] === 'view') { const r = d.runs.find(x => String(x.databaseId) === a[2]); if (!r) fail('run not found'); out(r); }", "if (a[0] === 'run' && a[1] === 'download') {", " const art = (d.artifacts[a[2]] || {})[opt('-n')]; if (!art) fail('no artifact matches');", - " fs.mkdirSync(opt('-D'), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), opt('-n') === 'paid-plan' ? 'manifest.json' : 'record.json'), JSON.stringify(art));", - " if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);", + " const fileFor = n => n === 'paid-plan' ? 'manifest.json' : n.startsWith('gate-binding-') ? 'gate-binding.json' : n.startsWith('report-verdict-') ? 'paid-report/collector-outcomes.json' : 'record.json';", + " fs.mkdirSync(path.dirname(path.join(opt('-D'), fileFor(opt('-n')))), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), fileFor(opt('-n'))), JSON.stringify(art));", + " if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), art.__reuse ? JSON.stringify(art.__reuse) : '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);", "}", "if (a[0] === 'api') {", " const arts = /actions\\/runs\\/(\\d+)\\/artifacts(?:\\?name=([^&]+))?/.exec(a[1] || '');", @@ -187,7 +204,7 @@ function shimSource(): string { " const named = /commits\\/([0-9a-f]+)\\/check-runs\\?check_name=([^&]+)/.exec(a[1] || '');", " if (named) { const list = (d.checks[named[1]] || []).filter(c => c.name === decodeURIComponent(named[2])); out({ total_count: list.length, check_runs: list }); }", " const paged = /commits\\/([0-9a-f]+)\\/(check-runs|status)\\?.*page=(\\d+)/.exec(a[1] || '');", - " if (paged) { const list = paged[2] === 'check-runs' ? (d.ci[paged[1]] || []) : []; const key = paged[2] === 'check-runs' ? 'check_runs' : 'statuses';", + " if (paged) { const list = paged[2] === 'check-runs' ? (d.ci[paged[1]] || []) : ((d.statuses || {})[paged[1]] || []); const key = paged[2] === 'check-runs' ? 'check_runs' : 'statuses';", " out(paged[3] === '1' ? { total_count: list.length, state: 'success', [key]: list } : { total_count: list.length, [key]: [] }); }", "}", "fail('gh shim: unsupported ' + a.join(' '));", diff --git a/test/ship-queue-train-ops.test.ts b/test/ship-queue-train-ops.test.ts new file mode 100644 index 000000000..e2af535fe --- /dev/null +++ b/test/ship-queue-train-ops.test.ts @@ -0,0 +1,474 @@ +/** + * Operational gaps seen on live trains, through the real + * bin/gstack-ship-queue against the train fixture: the screen verdict is read + * from the run (ready refuses a red head, verify --screen prints it, train + * build refuses the pin), a lagging GitHub merge ref gets a scratch merge + * (#3465), `train prune --stale` settles trains whose PRs merged elsewhere, + * and `train watch` resumes from the record after an interruption. + */ +import { afterEach, describe, expect, test } from 'bun:test'; +import * as fs from 'node:fs'; +import * as path from 'node:path'; +import { EXIT } from '../lib/headless-artifacts'; +import { cleanup, runBin } from './helpers/restamp-fixture'; +import { + GREEN_GATE, TRAIN_POLICY, dispatches, line, nonceOf, openPr, queue, readyReceipt, repoKey, trainJson, trainFixture, type Rule, type TrainFixture, +} from './helpers/ship-queue-train-fixture'; + +const fixtures: TrainFixture[] = []; +const make = (policy: Record = TRAIN_POLICY) => { const f = trainFixture(policy); fixtures.push(f); return f; }; +afterEach(() => { while (fixtures.length) cleanup(fixtures.pop()!.repo); }); + +const next = (out: string) => line(out, 'NEXT:'); +const result = (out: string) => line(out, 'GSTACK_RESULT:'); + +function readyFixture() { + const fx = make(); + const head = openPr(fx, '1'); + fx.write(d => { d.ci[head] = [{ status: 'completed', conclusion: 'success' }]; }); + const tree = runBin(fx.repo, 'gstack-wtree', []).out.trim(); + const pregate = path.join(fx.repo.root, 'pregate.json'); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + return { fx, head, pregate }; +} +const receiptOf = (fx: TrainFixture, pr: string, head: string) => JSON.parse(fs.readFileSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `${pr}-${head}.json`), 'utf8')); + +const RED_SCREEN: Rule = { status: 'completed', conclusion: 'failure', plan: { planned: 3 }, binding: { verdict: 'RED', flaky: ['c-flaky'] }, outcomes: { red: ['judge-voice-tone'], flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } }; +const GREEN_SCREEN: Rule = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN', flaky: ['c-flaky'] }, outcomes: { flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } }; + +describe('the screen verdict comes from the run', () => { + // Value: protects=a RED screen run (runs 40269992077 and 40275443393 on 2026-10-26 were VERDICT RED while the lane reported green) can never be READY'd, followed or seated; + // fails_when=ready or train build trust the lane's choice without reading the run's gate binding; why_new=no test read a screen's verdict; seam=the gh shim's gate-binding and report-verdict artifacts; no_claim=evals.yml's real upload step + test('a RED screen refuses ready under either choice and is never re-dispatched; verify --screen prints the red and flaky cases; train build refuses the pin', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = RED_SCREEN; }); + const first = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([first.status, line(first.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, expect.stringMatching(/^SCREEN_DISPATCH: evals\.yml run 100 /)]); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(again.status).toBe(EXIT.refused); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)'); + expect(line(again.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'); + expect(again.err).toMatch(/the screen recorded for [0-9a-f]{12} \(run 100\) is RED on judge-voice-tone; this head is not READY under any screen choice; fix: .*\(QUEUE_SCREEN_RED\)/); + expect(next(again.out)).toBe('NEXT: repair, push a new head, run /ship, then gstack-ship-queue ready --pr 1 --screen'); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'RED', run: 100, red: ['judge-voice-tone'], flaky: ['c-flaky'] }); + const noScreen = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']); + expect([noScreen.status, noScreen.err]).toEqual([EXIT.refused, expect.stringContaining('(QUEUE_SCREEN_RED)')]); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(verify.status).toBe(EXIT.fail); + expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)'); + expect(verify.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=pr\.screen\.narrow-\S+ run=100 verdict=RED$/); + expect(result(verify.out)).toMatch(/status=refused run=.*1-/); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect(build.status).toBe(EXIT.refused); + expect(build.err).toMatch(/#1's screen run 100 is RED on judge-voice-tone .*a red head is not seated.*\(QUEUE_SCREEN_RED\)/); + expect(next(build.out)).toBe('NEXT: none'); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'trains'))).toBe(false); + }, 120_000); + + test('a GREEN screen with flaky cases is followed, recorded and printed by verify --screen and train build; a run that bound no verdict is re-dispatched', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(again.status).toBe(EXIT.ok); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 GREEN flaky=c-flaky (verdict GREEN, reconcile exit 0, run concluded success, flaky 1)'); + expect(line(again.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)'); + expect(again.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=\S+ run=100 verdict=GREEN$/); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'GREEN', run: 100, red: [], flaky: ['c-flaky'] }); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([verify.status, verify.err]).toEqual([EXIT.ok, '']); + expect(line(verify.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'); + expect(next(verify.out)).toBe(`NEXT: gstack-ship-queue train build --prs 1@${head},...`); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect([build.status, line(build.out, 'SCREEN:'), line(build.out, 'ORDER:')]).toEqual([EXIT.ok, 'SCREEN: #1 run 100 GREEN (flaky: c-flaky)', 'ORDER: 1:#1(screened)']); + fx.write(d => { delete d.artifacts['100']['gate-binding-a1']; }); + const unbound = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(unbound.status).toBe(EXIT.refused); + expect(unbound.err).toMatch(/has no verdict to read: run 100 concluded success and bound no verdict \(no gate-binding artifact.*\(QUEUE_PROOF_MISSING\)/); + expect(next(unbound.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)'); + const redispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(redispatch.status).toBe(EXIT.ok); + expect(line(redispatch.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 101 .*\(not following run 100 concluded success and bound no verdict .*: a screen that bound no verdict is not a screen\)$/); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, reuse: { run: 77, flaky: ['c-reused'] } }; }); + const reusedDispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--broad', '--screen']); + expect(reusedDispatch.status).toBe(EXIT.ok); + const reused = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([reused.status, line(reused.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #1 run 102 GREEN flaky=c-reused (reused the GREEN verdict of run 77 (main-moved): same own diff on a moved main, flaky 1)']); + const noRun = queue(fx, ['verify', '--pr', '2', '--screen']); + expect(noRun.err).toMatch(/cannot read PR #2's head through gh.*\(QUEUE_PROOF_MISSING\)/); + const unscreened = openPr(fx, '3'); + readyReceipt(fx, '3', unscreened); + const none = queue(fx, ['verify', '--pr', '3', '--screen']); + expect([none.status, line(none.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #3 none (READY --no-screen)']); + }, 120_000); + + // Value: protects=a screen that concluded success with no slice (18 of 41 PR screens on 2026-10-26, superseded by a push before the plan) is NOT-RUN at verdict, not a screen and not a stale ref; + // fails_when=verify --screen calls it GREEN, HOLLOW or QUEUE_SCREEN_STALE_REF, or ready follows it; why_new=the hollow outcome reused the stale-ref code; seam=the gh shim's run with no plan artifact; no_claim=why evals.yml planned nothing + // Value: protects=train build --dry-run (with or without --stack-tests) writes nothing: the screen verdict it reads is printed, not recorded in the READY receipt or pushed to refs/notes/gstack-trains, as its DRY-RUN line says; + // fails_when=admitSlots persists the read verdict under --dry-run (the notes ref moves and the receipt gains a verdict before any real build); why_new=the other dry-run tests only check the trains dir; seam=none; no_claim=a verdict that was already recorded by ready or verify + test('train build --dry-run prints the screen verdict it read without recording it in the receipt or pushing the notes ref', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + expect(receiptOf(fx, '1', head).screen.verdict).toBeUndefined(); + const notes = fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains'); + const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); + expect([build.status, line(build.out, 'SCREEN:'), line(build.out, 'DRY-RUN:')]).toEqual([EXIT.ok, 'SCREEN: #1 run 100 GREEN (flaky: c-flaky)', 'DRY-RUN: nothing written, pushed or dispatched']); + expect(receiptOf(fx, '1', head).screen.verdict).toBeUndefined(); + expect(fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains')).toBe(notes); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect([verify.status, receiptOf(fx, '1', head).screen.verdict?.outcome]).toEqual([EXIT.ok, 'GREEN']); + expect(fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains')).not.toBe(notes); + }, 120_000); + + test('a screen run that concluded success with no slice is NOT-RUN: verify --screen refuses with QUEUE_SCREEN_NOT_RUN, the receipt records NOT_RUN, and ready --screen dispatches afresh', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; }); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok); + const verify = queue(fx, ['verify', '--pr', '1', '--screen']); + expect(verify.status).toBe(EXIT.refused); + expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))'); + expect(verify.err).toMatch(/the screen of #1 \(run 100\) concluded success with no slice: run 100 planned no cases .*; it is not a screen, and this head is not screened; fix: .*\(QUEUE_SCREEN_NOT_RUN\)/); + expect(next(verify.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)'); + expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'NOT_RUN', run: 100 }); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.ok, 2]); + expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))'); + expect(line(again.out, 'SCREEN_DISPATCH:')).toMatch(/not following run 100 planned no cases .*: a screen that planned nothing is not a screen\)$/); + }, 120_000); +}); + +describe('#3465: the screen is planned on a scratch merge when GitHub\'s merge ref lags', () => { + // Value: protects=a CHANGELOG-only conflict (every main stamp leaves one) no longer blocks --screen: the screen tests the head merged onto current main by the train's rules; + // fails_when=ready still refuses on the lagging ref, pushes nothing, or the scratch merge's parents are not (base, head); why_new=the stale-ref test only asserted the refusal; seam=none; no_claim=evals.yml's checkout of the pushed branch + test('a lagging or missing merge ref gets a scratch merge pushed as screen/-; a real conflict is QUEUE_SCREEN_STALE_REF; prune --stale removes the branch once the PR merged', () => { + const { fx, head, pregate } = readyFixture(); + const { repo } = fx; + fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; }); + repo.git('checkout', '-q', '-B', 'mainsim', 'origin/main'); + const cl = fs.readFileSync(path.join(repo.work, 'CHANGELOG.md'), 'utf8'); + fs.writeFileSync(path.join(repo.work, 'CHANGELOG.md'), cl.replace('# Changelog\n', '# Changelog\n\n## [1.2.4.0] - 2026-10-18\n\n- another PR, stamped\n')); + fs.writeFileSync(path.join(repo.work, 'VERSION'), '1.2.4.0\n'); + repo.git('commit', '-qam', 'v1.2.4.0 stamp'); + repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main'); + repo.git('checkout', '-q', 'pr-1'); + const dry = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen', '--dry-run']); + expect([dry.status, dry.err]).toEqual([EXIT.ok, '']); + const main = repo.git('rev-parse', 'origin/main'); + expect(line(dry.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${head.slice(0, 12)} onto main@${main.slice(0, 12)} \\(refs/pull/1/merge [0-9a-f]{12} merges onto [0-9a-f]{12}, not main@${main.slice(0, 12)}\\); would push screen/1-${head.slice(0, 12)} \\(dry run\\)$`)); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); + const real = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([real.status, real.err]).toEqual([EXIT.ok, '']); + const branch = `screen/1-${head.slice(0, 12)}`; + expect(line(real.out, 'SCREEN_MERGE:')).toContain(`; pushed ${branch}`); + const scratch = repo.git('ls-remote', 'origin', `refs/heads/${branch}`).split(/\s/)[0]!; + expect(scratch).toMatch(/^[0-9a-f]{40}$/); + repo.git('fetch', '-q', 'origin', `refs/heads/${branch}`); + expect([repo.git('rev-parse', `${scratch}^1`), repo.git('rev-parse', `${scratch}^2`)]).toEqual([repo.git('rev-parse', 'origin/main'), head]); + expect(repo.git('show', `${scratch}:VERSION`)).toBe('1.2.4.0'); + expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- #1 change'); + expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- another PR, stamped'); + const d = dispatches(fx, 'evals.yml'); + expect(d.length).toBe(1); + expect((d[0]!.inputs as any)).toMatchObject({ screen_merge_sha: scratch, expected_sha: scratch, screen_pr: '1' }); + expect(receiptOf(fx, '1', head).screen).toMatchObject({ choice: 'screen', merge_sha: scratch, branch }); + repo.git('push', '-q', 'origin', ':refs/pull/1/merge'); + const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect([missing.status, line(missing.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, 'SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)']); + expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge)'); + const kept = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(kept.out, 'KEEP:')).toBe(`KEEP: ${branch} (#1 OPEN)`); + fx.write(data => { (data.prs['1'] as any).state = 'MERGED'; }); + const pruned = queue(fx, ['train', 'prune', '--stale']); + expect(line(pruned.out, 'DELETED:')).toBe(`DELETED: ${branch} (#1 MERGED)`); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); + fx.write(data => { (data.prs['1'] as any).state = 'OPEN'; }); + repo.git('checkout', '-q', '-B', 'mainsim2', 'origin/main'); + fs.writeFileSync(path.join(repo.work, 'pr-1.txt'), 'main added the same file\n'); + repo.git('add', 'pr-1.txt'); + repo.git('commit', '-qm', 'main adds pr-1.txt'); + repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main'); + repo.git('checkout', '-q', 'pr-1'); + const conflict = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); + expect(conflict.status).toBe(EXIT.refused); + expect(conflict.err).toMatch(/#1 conflicts with main@[0-9a-f]{12} beyond the CHANGELOG and the stamp files \(pr-1\.txt\); GitHub's merge ref lags too \(origin has no refs\/pull\/1\/merge\).*\(QUEUE_SCREEN_STALE_REF\)/); + expect(next(conflict.out)).toBe('NEXT: resolve the conflict with main, push, run /ship, then gstack-ship-queue ready --pr 1 --screen'); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + }, 120_000); +}); + +describe('train prune --stale and train watch', () => { + function greenTrain(fx: TrainFixture, gate: Rule = GREEN_GATE) { + const h1 = openPr(fx, '1'); + const h2 = openPr(fx, '2'); + readyReceipt(fx, '1', h1); + readyReceipt(fx, '2', h2); + fx.write(d => { d.rules['evals.yml'] = gate; }); + const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2}`, '--free-timeout', '1']); + return { h1, h2, r, nonce: nonceOf(r.out) }; + } + + // Value: protects=a merging or green train whose PRs all merged elsewhere (gate.train-23af94b3c5e9 and -6755ab829acf on 2026-10-26) can be pruned without hand-editing refs/notes/gstack-trains; + // fails_when=prune skips merging/green trains, prunes one with an OPEN PR, or prunes without --stale; why_new=prune only knew terminal states; seam=the gh shim's PR states; no_claim=GitHub's real PR state + test('--stale settles a merging or green train whose every PR is MERGED or CLOSED on GitHub, logging each PR; an OPEN PR keeps it; without --stale nothing changes', () => { + const fx = make(); + const { nonce } = greenTrain(fx); + expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok); + expect(trainJson(fx, nonce).state).toBe('green'); + const plain = queue(fx, ['train', 'prune']); + expect(line(plain.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green)`); + const open = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(open.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #1 OPEN, #2 OPEN still need the train)`); + fx.write(d => { (d.prs['1'] as any).state = 'MERGED'; }); + const half = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(half.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #2 OPEN still needs the train)`); + fx.write(d => { (d.prs['2'] as any).state = 'CLOSED'; }); + const dry = queue(fx, ['train', 'prune', '--stale', '--dry-run']); + expect(line(dry.out, 'STALE:')).toBe(`STALE: ${nonce} (green): #1 MERGED, #2 CLOSED; would mark stale`); + expect(line(dry.out, 'WOULD-DELETE:')).toBe(`WOULD-DELETE: train/${nonce} (stale)`); + expect(trainJson(fx, nonce).state).toBe('green'); + const pruned = queue(fx, ['train', 'prune', '--stale']); + expect([pruned.status, line(pruned.out, 'STALE:'), line(pruned.out, 'DELETED:')]).toEqual([EXIT.ok, `STALE: ${nonce} (green): #1 MERGED, #2 CLOSED`, `DELETED: train/${nonce} (stale)`]); + const t = trainJson(fx, nonce); + expect(t.state).toBe('stale'); + expect(t.effects.map((e: any) => e.what)).toContain('marked stale by train prune --stale: #1 MERGED, #2 CLOSED'); + expect(fx.repo.git('ls-remote', 'origin', `refs/heads/train/${nonce}`)).toBe(''); + const status = queue(fx, ['train', 'status']); + expect(status.out).toContain('TRAIN: none live'); + }, 120_000); + + // Value: protects=a stale train record or READY receipt leaves refs/notes/gstack-trains through one command under an exact lease, with the local files, instead of a hand-run git notes remove on every machine; + // fails_when=forget removes the wrong notes, deletes locally when the push failed, or forgets a merging train; why_new=no CLI removed notes (merge-train skill, known gaps); seam=none; no_claim=a concurrent writer's race (the lease is exercised only as a fast-forward) + test('train forget removes a settled train (attempts included) or a READY receipt from the notes ref and the state root, leaves other records alone, keeps everything (the local notes ref too) when the lease is rejected, and refuses a train still building, running or merging', () => { + const fx = make(); + const { nonce, h1 } = greenTrain(fx); + expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok); + const trainDir = path.join(fx.state, 'ship-queue', 'trains', nonce); + expect(fs.existsSync(path.join(trainDir, 'attempts', 'a1.json'))).toBe(true); + const dry = queue(fx, ['train', 'forget', '--train', nonce, '--dry-run']); + expect([dry.status, line(dry.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: train ${nonce} (green; 1 attempt(s))`]); + expect(dry.out).toContain(` note gstack-trains:train:${nonce}`); + expect(dry.out).toContain(` note gstack-trains:attempt:${nonce}:a1`); + expect(fs.existsSync(trainDir)).toBe(true); + const notesBefore = fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains').split(/\s/)[0]!; + const forgot = queue(fx, ['train', 'forget', '--train', nonce]); + expect([forgot.status, forgot.err]).toEqual([EXIT.ok, '']); + expect(line(forgot.out, 'FORGOTTEN:')).toMatch(new RegExp(`^FORGOTTEN: 2 note\\(s\\) removed; notes pushed to origin refs/notes/gstack-trains \\(lease ${notesBefore.slice(0, 12)}\\); 1 local path\\(s\\) deleted$`)); + expect(fs.existsSync(trainDir)).toBe(false); + const clean = path.join(fx.repo.root, 'clean-state'); + const status = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: clean }); + expect(status.out).toContain('TRAIN: none live'); + expect(fs.existsSync(path.join(clean, 'ship-queue', 'trains', nonce))).toBe(false); + expect(fs.existsSync(path.join(clean, 'ship-queue', 'ready', repoKey(fx), `1-${h1}.json`))).toBe(false); + const again = queue(fx, ['train', 'forget', '--train', nonce]); + expect(again.err).toMatch(new RegExp(`no train record ${nonce} here or in refs/notes/gstack-trains.*\\(QUEUE_PROOF_MISSING\\)`)); + const h3 = openPr(fx, '3'); + fx.write(d => { d.ci[h3] = [{ status: 'completed', conclusion: 'success' }]; }); + const pregate = path.join(fx.repo.root, 'pregate3.json'); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h3, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + expect(queue(fx, ['ready', '--pr', '3', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok); + const h4 = openPr(fx, '4'); + fx.write(d => { d.ci[h4] = [{ status: 'completed', conclusion: 'success' }]; }); + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h4, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); + expect(queue(fx, ['ready', '--pr', '4', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok); + const receipt = queue(fx, ['train', 'forget', '--receipt', `3@${h3}`]); + expect([receipt.status, line(receipt.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: READY receipt #3 at ${h3.slice(0, 12)} (screen no-screen)`]); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false); + const rebuilt = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: path.join(fx.repo.root, 'clean-state-2') }); + expect(rebuilt.status).toBe(EXIT.ok); + expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `4-${h4}.json`))).toBe(true); + expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false); + expect(queue(fx, ['train', 'forget', '--receipt', '3@abc']).err).toMatch(/--receipt takes @.*\(QUEUE_PROOF_MISSING\)/); + expect(queue(fx, ['train', 'forget']).status).toBe(EXIT.usage); + const { nonce: second } = (() => { readyReceipt(fx, '3', h3); readyReceipt(fx, '4', h4); const r = queue(fx, ['train', 'build', '--prs', `3@${h3},4@${h4}`, '--free-timeout', '1']); return { nonce: nonceOf(r.out) }; })(); + expect(queue(fx, ['train', 'verdict', '--train', second]).status).toBe(EXIT.ok); + const hook = path.join(fx.repo.origin, 'hooks', 'pre-receive'); + fs.mkdirSync(path.dirname(hook), { recursive: true }); + fs.writeFileSync(hook, '#!/bin/sh\nwhile read o n r; do case "$r" in refs/notes/*) echo "notes frozen" >&2; exit 1;; esac; done\nexit 0\n', { mode: 0o755 }); + const localNotes = fx.repo.git('rev-parse', 'refs/notes/gstack-trains'); + const rejected = queue(fx, ['train', 'forget', '--train', second]); + expect(rejected.status).toBe(EXIT.refused); + expect(rejected.err).toMatch(/the notes ref was not updated \(git push refs\/notes\/gstack-trains rejected: .*notes frozen.*; the local notes ref was restored\); nothing deleted locally.*\(QUEUE_PROOF_MISSING\)/); + expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'))).toBe(true); + expect(fx.repo.git('rev-parse', 'refs/notes/gstack-trains')).toBe(localNotes); + expect(fx.repo.git('notes', '--ref=refs/notes/gstack-trains', 'list').split('\n').length).toBeGreaterThan(0); + fs.rmSync(hook); + const trainFile = path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'); + for (const state of ['dispatch-unknown', 'merging']) { + const t = JSON.parse(fs.readFileSync(trainFile, 'utf8')); + t.state = state; + fs.writeFileSync(trainFile, JSON.stringify(t)); + const live = queue(fx, ['train', 'forget', '--train', second]); + expect(live.status).toBe(EXIT.refused); + expect(live.err).toMatch(new RegExp(`is ${state}; forget settles nothing that is still building, running or merging .*\\(QUEUE_PROOF_MISSING\\)`)); + expect(fs.existsSync(trainFile)).toBe(true); + } + }, 120_000); + + // Value: protects=a restarted controller re-arms with one command: watch continues from the record (free tests, gate dispatch, the pending run, the verdict) and prints NEXT; + // fails_when=watch dispatches twice, blocks past --timeout, silently replaces a malformed --free-timeout with the default, or does not reach the verdict after a restart; why_new=no resumable follow existed (hand-run gh run watch chains died with the machine); seam=GSTACK_SHIP_QUEUE_POLL_MS; no_claim=real GitHub run timing + test('watch resumes from free-pending through the gate dispatch to the verdict, times out with its own NEXT, and never dispatches twice', () => { + const fx = make(); + fx.write(d => { d.rules['free-tests.yml'] = { status: 'in_progress', conclusion: null }; }); + const { nonce, r } = greenTrain(fx, { ...GREEN_GATE, status: 'in_progress', conclusion: null }); + expect(r.status).toBe(EXIT.refused); + expect(trainJson(fx, nonce).state).toBe('free-pending'); + const waiting = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect([waiting.status, result(waiting.out)]).toEqual([EXIT.refused, expect.stringContaining('status=incomplete')]); + expect(waiting.out).toContain(`WATCH: train ${nonce} state=free-pending (resumable; rerun this command after an interruption)`); + expect(next(waiting.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`); + expect(dispatches(fx, 'evals.yml')).toEqual([]); + fx.write(d => { for (const run of d.runs) if (run.workflow === 'free-tests.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); }); + const gate = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect(gate.status).toBe(EXIT.refused); + expect(gate.out).toContain(`WATCH: train build --train ${nonce}: free tests green, gate dispatched`); + expect(gate.err).toMatch(/is still in_progress at the watch timeout.*\(QUEUE_TRAIN_PENDING\)/); + expect(next(gate.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`); + expect(dispatches(fx, 'evals.yml').length).toBe(1); + expect(trainJson(fx, nonce).state).toBe('gate-pending'); + const again = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']); + expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.refused, 1]); + fx.write(d => { for (const run of d.runs) if (run.workflow === 'evals.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); }); + const done = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '5']); + expect([done.status, done.err]).toEqual([EXIT.ok, '']); + expect(done.out).toContain('WATCH: gate run completed; reading the verdict'); + expect(line(done.out, 'VERDICT:')).toMatch(new RegExp(`^VERDICT: GREEN train ${nonce} run 10[0-9] `)); + expect(next(done.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`); + const idle = queue(fx, ['train', 'watch', '--train', nonce]); + expect([idle.status, line(idle.out, 'WATCH: nothing')]).toEqual([EXIT.ok, 'WATCH: nothing to wait for in state green']); + expect(next(idle.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`); + expect(queue(fx, ['train', 'watch']).status).toBe(EXIT.usage); + expect(queue(fx, ['train', 'watch', '--train', nonce, '--free-timeout', 'abc']).status).toBe(EXIT.usage); + expect(queue(fx, ['train', 'watch', '--train', 'no-such-train']).err).toMatch(/no train record no-such-train.*\(QUEUE_PROOF_MISSING\)/); + }, 120_000); +}); + +describe('verify --post-merge', () => { + // Value: protects=after a train merges, the coordinator watches main's push workflows at the merged sha until they settle instead of a background gh chain a restart kills (GBrain practice 8); + // fails_when=a pending check is called settled, a red check is called green, a pending or failed commit status (the legacy status API, which ciAtHead counts) is reported as red with no name or as green, or the command needs process state; why_new=no post-merge watch existed; seam=the gh shim's check runs and commit statuses at a sha; no_claim=which workflows main runs + test('waits for every check run at the sha to complete, prints each, is green only when none failed, red with the failed names, pending or red on a commit status, and resumable after a timeout', () => { + const fx = make(); + const sha = fx.repo.git('rev-parse', 'HEAD'); + fx.write(d => { d.ci[sha] = [{ name: 'free-tests', status: 'completed', conclusion: 'success' }, { name: 'windows-free-tests', status: 'in_progress', conclusion: null }, { name: 'main-backstop', status: 'completed', conclusion: 'neutral' }]; }); + const pending = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect(pending.status).toBe(EXIT.refused); + expect(line(pending.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} pending (passed=1 failed=0 pending=1 skipped=1)`); + expect(pending.out).toContain('CHECK: windows-free-tests in_progress'); + expect(pending.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} have not settled .*windows-free-tests.*\\(QUEUE_TRAIN_PENDING\\)`)); + expect(next(pending.out)).toBe(`NEXT: gstack-ship-queue verify --post-merge ${sha} --timeout 1`); + fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'failure' }; }); + const red = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect(red.status).toBe(EXIT.fail); + expect(line(red.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} red (passed=1 failed=1 pending=0 skipped=1)`); + expect(red.out).toContain('CHECK: windows-free-tests failure'); + expect(red.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} are red: windows-free-tests.*\\(QUEUE_POST_MERGE_RED\\)`)); + expect(next(red.out)).toBe('NEXT: read the red run, repair on a new PR or roll back; gstack-ship-queue train status names the freeze if the backstop filed one'); + fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'success' }; }); + const green = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([green.status, green.err]).toEqual([EXIT.ok, '']); + expect(line(green.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} green (passed=2 failed=0 pending=0 skipped=1)`); + expect(green.out.split('\n').filter(l => l.startsWith('CHECK: ')).sort()).toEqual(['CHECK: free-tests success', 'CHECK: main-backstop neutral', 'CHECK: windows-free-tests success']); + expect(next(green.out)).toBe('NEXT: none'); + fx.write(d => { d.statuses = { [sha]: [{ state: 'pending', context: 'ci/vendor' }] }; }); + const statusPending = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([statusPending.status, line(statusPending.out, 'POST_MERGE:')]).toEqual([EXIT.refused, `POST_MERGE: ${sha.slice(0, 12)} pending (passed=2 failed=0 pending=1 skipped=1)`]); + expect(statusPending.err).toMatch(/have not settled at the timeout: a commit status still running.*\(QUEUE_TRAIN_PENDING\)/); + fx.write(d => { d.statuses = { [sha]: [{ state: 'failure', context: 'ci/vendor' }] }; }); + const statusRed = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([statusRed.status, line(statusRed.out, 'POST_MERGE:')]).toEqual([EXIT.fail, `POST_MERGE: ${sha.slice(0, 12)} red (passed=2 failed=1 pending=0 skipped=1)`]); + expect(statusRed.err).toMatch(/are red: a commit status \(no check run failed\).*\(QUEUE_POST_MERGE_RED\)/); + fx.write(d => { d.statuses = {}; d.ci[sha] = []; }); + const none = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']); + expect([none.status, line(none.out, 'POST_MERGE:')]).toEqual([EXIT.refused, `POST_MERGE: ${sha.slice(0, 12)} none (passed=0 failed=0 pending=0 skipped=0)`]); + expect(none.err).toMatch(new RegExp(`no check run has attached to ${sha.slice(0, 12)} yet.*\\(QUEUE_TRAIN_PENDING\\)`)); + expect(queue(fx, ['verify', '--post-merge', 'not-a-sha']).status).toBe(EXIT.usage); + }, 120_000); +}); + +describe('ready --dispatch-ci', () => { + // Value: protects=a PR that conflicts with the base (a CHANGELOG-only conflict after every stamp) gets its free workflows dispatched on the branch, waited for, and its requires-remote pregate lanes cleared with the run URLs in one command, replacing the four-step hand procedure; + // fails_when=the dispatch happens on a fork head, a red run clears a lane, a lane whose workflow was not dispatched is cleared, ready still says ci: none, or the CI_DISPATCH lines push the screen: line in front of READY (identifiers first); why_new=the ci-hint was advisory prose; seam=the gh shim's workflow run, run list and check runs; no_claim=GitHub's real dispatch latency + test('dispatches the named workflows on a conflicting PR, waits for their runs, clears the matching pregate lanes with the run URLs and then READYs; a fork head and a red run are refused', () => { + const { fx, head, pregate } = readyFixture(); + fx.write(d => { + (d.prs['1'] as any).mergeable = 'CONFLICTING'; + d.ci[head] = []; + d.rules['free-tests.yml'] = { ci: 'free-tests', headSha: head }; + d.rules['windows-free-tests.yml'] = { ci: 'windows-free-tests', headSha: head }; + }); + const tree = JSON.parse(fs.readFileSync(pregate, 'utf8')).tree; + fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'requires-remote', checks: [{ id: 'unit', status: 'pass' }], requires_remote: [ + { lane: 'free-tests.yml/free-tests', platform: 'linux', workflow: 'free-tests.yml', job: 'free-tests', tests: ['test/a.test.ts'], receipt: null }, + { lane: 'windows-free-tests.yml/windows-free-tests', platform: 'windows', workflow: 'windows-free-tests.yml', job: 'windows-free-tests', tests: ['test/a.test.ts'], receipt: null }, + { lane: 'macos-free-tests.yml/macos', platform: 'macos', workflow: 'macos-free-tests.yml', job: 'macos', tests: ['test/b.test.ts'], receipt: null }] })); + const plain = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']); + expect([plain.status, line(plain.out, 'ci:')]).toEqual([EXIT.refused, `ci: none at ${head.slice(0, 12)} (passed=0 failed=0 pending=0 skipped=0)`]); + const r = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml,windows-free-tests.yml', '--timeout', '5']); + expect(r.out.split('\n').filter(l => l.startsWith('CI_DISPATCH:'))).toEqual([ + 'CI_DISPATCH: free-tests.yml --ref pr-1 run 100 https://example.invalid/runs/100 success', + 'CI_DISPATCH: windows-free-tests.yml --ref pr-1 run 101 https://example.invalid/runs/101 success']); + expect(r.out.split('\n').filter(l => l.startsWith('PREGATE_CLEARED:'))).toEqual([ + 'PREGATE_CLEARED: free-tests.yml/free-tests https://example.invalid/runs/100', + 'PREGATE_CLEARED: windows-free-tests.yml/windows-free-tests https://example.invalid/runs/101']); + expect(line(r.out, 'ci:')).toBe(`ci: green at ${head.slice(0, 12)} (passed=2 failed=0 pending=0 skipped=0)`); + expect(line(r.out, 'PREGATE:')).toMatch(/^PREGATE: requires-remote .*/); + expect(r.err).toMatch(/1 requires-remote obligation\(s\) without a receipt: macos-free-tests\.yml\/macos.*\(QUEUE_NOT_READY\)/); + const saved = JSON.parse(fs.readFileSync(pregate, 'utf8')); + expect(saved.requires_remote.map((l: any) => l.receipt)).toEqual(['https://example.invalid/runs/100', 'https://example.invalid/runs/101', null]); + expect(dispatches(fx).map(l => l.workflow)).toEqual(['free-tests.yml', 'windows-free-tests.yml']); + const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect(dispatches(fx).length).toBe(2); + expect(line(again.out, 'CI_DISPATCH:')).toBe('CI_DISPATCH: free-tests.yml not dispatched (run 100 already concluded success at this head)'); + const cleared = JSON.parse(fs.readFileSync(pregate, 'utf8')); + cleared.requires_remote[2].receipt = 'https://example.invalid/runs/7'; + fs.writeFileSync(pregate, JSON.stringify(cleared)); + const ok = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect([ok.status, ok.err]).toEqual([EXIT.ok, '']); + const okLines = ok.out.split('\n'); + expect(okLines[0]).toBe('CI_DISPATCH: free-tests.yml not dispatched (run 100 already concluded success at this head)'); + expect(okLines[okLines.findIndex(l => l.startsWith('READY #1 ')) + 1]).toBe('screen: no-screen'); + fx.write(d => { d.rules['free-tests.yml'] = { ci: 'free-tests', conclusion: 'failure', headSha: head }; d.ci[head] = []; d.runs.length = 0; }); + const red = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']); + expect([red.status, line(red.out, 'CI_DISPATCH:')]).toEqual([EXIT.refused, 'CI_DISPATCH: free-tests.yml --ref pr-1 run 102 https://example.invalid/runs/102 failure']); + expect(red.err).toMatch(new RegExp(`free-tests\\.yml run 102 concluded failure at ${head.slice(0, 12)}; a red run clears no lane.*\\(QUEUE_NOT_READY\\)`)); + expect(JSON.parse(fs.readFileSync(pregate, 'utf8')).requires_remote[0].receipt).toBe('https://example.invalid/runs/100'); + fx.write(d => { (d.prs['1'] as any).isCrossRepository = true; }); + const fork = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml']); + expect([fork.status, fork.err]).toEqual([EXIT.refused, expect.stringMatching(/#1's head is on a fork.*not dispatched.*\(QUEUE_TRAIN_UNTRUSTED\)/)]); + expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', '../evil.yml']).status).toBe(EXIT.usage); + }, 120_000); +}); + +describe('train build --dry-run --stack-tests', () => { + // Value: protects=a PR whose new tests pass on its own branch but fail once stacked behind another PR's change is caught before the train spends a gate (#3460 on #3451, 2026-10-26); + // fails_when=the new test files run against the PR head instead of the integrated prefix, a failing prefix still dry-runs green, or a PR without new tests (a helper under test/ included: bun exits 1 when a positional filter matches no test file) is misreported; why_new=no pre-dispatch stack check existed; seam=bun test in a scratch worktree; no_claim=the repo's full suite on the prefix + test('runs each PR\'s new test files on its integrated prefix T: a pass and a helper-only PR are reported, a failure refuses the dry run and names the prefix, and the ordinary dry run skips it', () => { + const fx = make(); + const h1 = openPr(fx, '1', { file: 'test/pr-1_test.ts', text: "import { expect, test } from 'bun:test';\ntest('pr-1', () => { expect(1).toBe(1); });\n" }); + const h2 = openPr(fx, '2', { file: 'test/pr-2.test.ts', text: "import { existsSync } from 'node:fs';\nimport { expect, test } from 'bun:test';\ntest('pr-2 wants no pr-1 test beside it', () => { expect(existsSync('test/pr-1_test.ts')).toBe(false); });\n" }); + const h3 = openPr(fx, '3', { file: 'test/helpers/pr-3-helper.ts', text: 'export const three = 3;\n' }); + readyReceipt(fx, '1', h1); + readyReceipt(fx, '2', h2); + readyReceipt(fx, '3', h3); + const plain = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run']); + expect([plain.status, plain.out.includes('STACK_TESTS:')]).toEqual([EXIT.ok, false]); + const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run', '--stack-tests']); + expect(r.status).toBe(EXIT.fail); + expect(r.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([ + 'STACK_TESTS: #1 T1 1 new test file(s) pass (test/pr-1_test.ts)', + expect.stringMatching(/^STACK_TESTS: #2 T2 1 new test file\(s\) FAIL \(test\/pr-2\.test\.ts\): .*\(fail\) pr-2 wants no pr-1 test beside it.*1 fail$/), + 'STACK_TESTS: #3 T3 no new test files']); + expect(r.err).toMatch(/#2's new tests fail on the integrated prefix T2 \(test\/pr-2\.test\.ts\); they pass on the PR head alone only if nothing before it in the train changes what they read.*\(QUEUE_TRAIN_STACK_RED\)/); + expect(next(r.out)).toBe(`NEXT: fix #2's tests against T2, or reorder with --order, then gstack-ship-queue train build --prs 1@${h1},2@${h2},3@${h3} --dry-run --stack-tests`); + expect(line(r.out, 'DRY-RUN:')).toBe('DRY-RUN: nothing written, pushed or dispatched'); + const reordered = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--order', '2,1,3', '--dry-run', '--stack-tests']); + expect(reordered.status).toBe(EXIT.ok); + expect(reordered.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([ + 'STACK_TESTS: #2 T1 1 new test file(s) pass (test/pr-2.test.ts)', + 'STACK_TESTS: #1 T2 1 new test file(s) pass (test/pr-1_test.ts)', + 'STACK_TESTS: #3 T3 no new test files']); + expect(fx.repo.git('worktree', 'list', '--porcelain')).not.toContain('gstack-stack-tests-'); + expect(queue(fx, ['train', 'build', '--prs', `1@${h1}`, '--stack-tests']).status).toBe(EXIT.usage); + }, 180_000); +}); diff --git a/test/ship-queue-train.test.ts b/test/ship-queue-train.test.ts index 97ae85475..318a25657 100644 --- a/test/ship-queue-train.test.ts +++ b/test/ship-queue-train.test.ts @@ -196,9 +196,9 @@ describe('ready under a train policy: the screen choice', () => { expect(queue(fx, ['ready', '--pr', '1', '--broad']).status).toBe(EXIT.usage); }, 120_000); - // Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success); - // fails_when=the merge ref's second parent is not compared with the PR head, or a missing merge ref keeps the old QUEUE_NOT_READY code; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in) - test('--screen refuses QUEUE_SCREEN_STALE_REF when refs/pull//merge lags the PR head or is missing, dispatching nothing; a recomputed merge ref admits it', () => { + // Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success): a lagging, missing or unfetchable ref is replaced by a scratch merge of the head onto current main (#3465), and an unreachable remote is still refused; + // fails_when=the merge ref's parents are not compared with the PR head and current main, or the scratch merge is skipped; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in) + test('--screen plans the screen on a scratch merge when refs/pull//merge lags the PR head, is missing or cannot be fetched; an unreachable remote is POLICY_SOURCE_UNAVAILABLE; a recomputed merge ref is used as is', () => { const { fx, head, pregate } = readyFixture(); const { repo } = fx; fs.appendFileSync(path.join(repo.work, 'pr-1.txt'), 'pushed after the merge ref was computed\n'); @@ -208,41 +208,41 @@ describe('ready under a train policy: the screen choice', () => { fx.write(d => { (d.prs['1'] as any).headRefOid = moved; (d.prs['1'] as any).body = shippedBody('1', moved); d.ci[moved] = [{ status: 'completed', conclusion: 'success' }]; }); const pregate2 = path.join(repo.root, 'pregate2.json'); fs.writeFileSync(pregate2, JSON.stringify({ schema_version: 1, tree: runBin(repo, 'gstack-wtree', []).out.trim(), head: moved, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] })); - const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(stale.status).toBe(EXIT.refused); - expect(stale.err).toMatch(new RegExp(`refs/pull/1/merge \\([0-9a-f]{12}\\) merges ${head.slice(0, 12)}, not the PR head ${moved.slice(0, 12)}.*\\(QUEUE_SCREEN_STALE_REF\\)`)); - expect(line(stale.out, 'NEXT:')).toBe('NEXT: resolve the conflict with main (or wait for GitHub to recompute refs/pull/1/merge), push, then gstack-ship-queue ready --pr 1 --screen; a screen dispatched on a stale merge ref plans no cases'); + const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([stale.status, stale.err]).toEqual([EXIT.ok, '']); + expect(line(stale.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${moved.slice(0, 12)} onto main@[0-9a-f]{12} \\(refs/pull/1/merge [0-9a-f]{12} merges ${head.slice(0, 12)}, not ${moved.slice(0, 12)}\\); would push screen/1-${moved.slice(0, 12)} \\(dry run\\)$`)); repo.git('push', '-q', 'origin', ':refs/pull/1/merge'); - const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(missing.status).toBe(EXIT.refused); - expect(missing.err).toMatch(/has no refs\/pull\/1\/merge.*\(QUEUE_SCREEN_STALE_REF\)/); + const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([missing.status, missing.err]).toEqual([EXIT.ok, '']); + expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge); would push'); const tree = repo.git('rev-parse', `${moved}^{tree}`); const main = repo.git('rev-parse', 'origin/main'); const badMerge = spawnSync('git', ['-C', repo.origin, 'hash-object', '-t', 'commit', '-w', '--stdin', '--literally'], { input: `tree ${tree}\nparent ${main}\nparent ${moved}\nauthor A 1700000000 +0000\ncommitter B 1700000000 +0000\n\nmerge without a committer email\n`, encoding: 'utf8', timeout: 30_000 }).stdout.trim(); spawnSync('git', ['-C', repo.origin, 'update-ref', 'refs/pull/1/merge', badMerge], { timeout: 30_000 }); repo.git('config', 'fetch.fsckObjects', 'true'); - const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); - expect(unfetchable.status).toBe(EXIT.refused); - expect(unfetchable.err).toMatch(/could not fetch refs\/pull\/1\/merge from origin: .*\(QUEUE_SCREEN_STALE_REF\)/); - expect(line(unfetchable.out, 'NEXT:')).toBe('NEXT: check the remote, then gstack-ship-queue ready --pr 1 --screen'); + const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); + expect([unfetchable.status, unfetchable.err]).toEqual([EXIT.ok, '']); + expect(line(unfetchable.out, 'SCREEN_MERGE:')).toMatch(/\(refs\/pull\/1\/merge [0-9a-f]{12} could not be fetched\); would push/); repo.git('config', '--unset', 'fetch.fsckObjects'); const originUrl = repo.git('remote', 'get-url', 'origin'); repo.git('remote', 'set-url', 'origin', path.join(repo.root, 'no-such-origin.git')); const unlisted = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']); repo.git('remote', 'set-url', 'origin', originUrl); expect(unlisted.status).toBe(EXIT.refused); - expect(unlisted.err).toMatch(/could not list origin's refs: .*\(QUEUE_SCREEN_STALE_REF\)/); + expect(unlisted.err).toMatch(/git fetch origin main failed: .*\(POLICY_SOURCE_UNAVAILABLE\)/); expect(unlisted.err.split('\n').filter(Boolean).length).toBe(1); expect(dispatches(fx)).toEqual([]); expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready'))).toBe(false); repo.git('push', '-q', '-f', 'origin', `${mergeRefCommit(fx, moved)}:refs/pull/1/merge`); const fresh = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']); expect([fresh.status, fresh.err]).toEqual([EXIT.ok, '']); + expect(line(fresh.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: GitHub's refs/pull/1/merge [0-9a-f]{12} merges the head ${moved.slice(0, 12)} onto main@[0-9a-f]{12}$`)); expect(line(fresh.out, 'SCREEN_DISPATCH:')).toContain('(dry run: not dispatched)'); + expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe(''); }, 120_000); // Value: protects=a finished screen that planned 0 cases never counts as a screen: ready dispatches a fresh one instead of adopting it and train build seats the slot unscreened with a SCREEN: line; - // fails_when=the census is not read (a hollow run is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in) + // fails_when=the census is not read (a not-run screen is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in) test('a screen run that planned 0 cases is not followed by ready and is counted as not-run by train build', () => { const { fx, head, pregate } = readyFixture(); fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; }); @@ -260,7 +260,7 @@ describe('ready under a train policy: the screen choice', () => { fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 0 } }; }); const emptyPlan = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); expect(line(emptyPlan.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 102 .*not following run 101 planned no cases/); - fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 } }; }); + fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN' } }; }); const live = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']); expect(live.status).toBe(EXIT.ok); expect(line(live.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 103 .*not following run 102 planned no cases \(an empty plan manifest; concluded success\)/); @@ -269,10 +269,11 @@ describe('ready under a train policy: the screen choice', () => { expect(line(followed.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 103 (same merge SHA, base and policy; no duplicate)'); expect(dispatches(fx, 'evals.yml').length).toBe(4); const liveBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); - expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']); - fx.write(d => { d.rules['evals.yml'] = { status: 'in_progress', conclusion: null }; }); + expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 GREEN']); + fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'in_progress', conclusion: null }); }); const runningBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); - expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']); + expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 still running; counted as screened']); + fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'completed', conclusion: 'success' }); }); fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success' }; d.artifacts['103']['paid-plan'] = { reuse: 'confirmed' }; }); const unreadable = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']); expect([unreadable.status, line(unreadable.out, 'ORDER:'), line(unreadable.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', "SCREEN: #1 run 103's paid-plan manifest.json is unreadable; counted as screened, unverified"]);