mirror of
https://github.com/garrytan/gstack.git
synced 2026-10-31 22:14:25 +01:00
feat(ship-queue): screens read from the run, train watch/forget/prune --stale, verify --post-merge, ready --dispatch-ci, --stack-tests
Operating merge trains on 2026-10-26 showed the gaps this closes. A dispatched screen's verdict is read from the run's gate-binding and report-verdict artifacts (lib/ship-queue-screen.ts) and recorded in the READY receipt: a RED screen refuses ready under either choice and is never re-dispatched at the same merge SHA (QUEUE_SCREEN_RED); a run that concluded success with no slice is NOT-RUN (QUEUE_SCREEN_NOT_RUN), never followed, and counted as unscreened by train build. When refs/pull/<n>/merge lags or is missing the screen is planned on a scratch merge pushed as screen/<pr>-<head12> (#3465); QUEUE_SCREEN_STALE_REF means a real conflict only. train watch --train is the resumable follow command; train prune --stale settles a merging/green train whose PRs all merged or closed elsewhere; train forget removes one record with an exact lease on refs/notes/gstack-trains; verify --post-merge <sha> watches main's push workflows until they settle; ready --dispatch-ci <wf,…> dispatches the free workflows on a conflicting PR, waits, and clears the matching requires-remote pregate lanes with the run URLs; train build --dry-run --stack-tests runs each PR's new test files on its integrated prefix before anything is dispatched (QUEUE_TRAIN_STACK_RED). Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
garrytanandcapy-ai[bot] committed
1 parent
ab9a854be2
commit
c75ec3ad5a
21 files changed
+1450
-108
No files matched your search
@@ -1,5 +1,30 @@
|
||||
# Changelog
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
**Merge trains read their screens from the run, follow themselves without a daemon, and refuse what would have cost a gate.** Operating trains on 2026-10-26 showed the gaps: two screens were VERDICT RED while their lanes reported green and the heads were READY'd anyway; 18 of 41 PR screens concluded success with no slice; a PR whose merge ref lagged main could not be screened at all (#3465); two trains whose PRs merged elsewhere sat in `merging`/`green` forever; a background `gh run watch` chain died with every machine restart; a PR's new tests passed on its branch and failed once stacked behind another PR; and the four-step hand procedure for a conflicting PR's CI was typed by hand each time.
|
||||
|
||||
### Added
|
||||
- **`lib/ship-queue-screen.ts`**: a dispatched screen's verdict is read from the run's own artifacts (`gate-binding-a<attempt>`: verdict, reconcile exit, flaky cases; `report-verdict-a<attempt>/paid-report/collector-outcomes.json`: the panels that failed the lane and the rule-16 flaky lines; a confirmed verdict reuse in `paid-plan/gate-reuse.json`), never from a lane's prose. `ready --screen`, `verify --pr <n> --screen` and `train build` print `SCREEN_VERDICT: #<n> run <id> GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one `FLAKY:` line per flaky case, and record the verdict in the READY receipt (`screen.verdict`, locally and in `refs/notes/gstack-trains`).
|
||||
- **`QUEUE_SCREEN_RED`**: a RED screen refuses `ready` under `--screen` and `--no-screen` alike, is never re-dispatched at the same merge SHA, and `train build` refuses to seat the pin. **`QUEUE_SCREEN_NOT_RUN`**: a screen run that concluded success with no slice (its route job found the head superseded by a push, or the plan selected nothing) is `NOT-RUN`: `verify --screen` refuses, the receipt records `NOT_RUN`, `ready --screen` dispatches afresh instead of following it, and `train build` seats the pin unscreened with a `SCREEN: … counted as not-run` line.
|
||||
- **Scratch-merge screens (#3465)**: when GitHub's `refs/pull/<n>/merge` lags the PR head, is missing or cannot be fetched, `ready --screen` plans the screen on a scratch merge of the head onto the base, pushed as `screen/<pr>-<head12>` (`SCREEN_MERGE:` names it); only a real conflict with the base is `QUEUE_SCREEN_STALE_REF`, whose text now says so, and an unreachable remote is `POLICY_SOURCE_UNAVAILABLE`.
|
||||
- **`train watch --train <nonce> [--timeout <s>]`** (`lib/ship-queue-train-watch.ts`): the one resumable follow command. It reads the record, runs the step its state asks for (`train build --train` to push, run the free tests and dispatch the gate; `train status` to settle an unconfirmed dispatch), waits for the pending run by polling gh, reads the verdict and prints the last step's `NEXT:`. No process state: rerun it after an interruption and it continues from the record; it never dispatches anything the record would not dispatch on its own.
|
||||
- **`train prune --stale`**: a `merging` or `green` train whose every PR is MERGED or CLOSED on GitHub is marked `stale` (a new terminal state), its branch and the PRs' `screen/<pr>-*` branches deleted, each PR's state logged on a `STALE:` line; a train with an OPEN PR is kept and says why. **`train forget --train <nonce> | --receipt <pr>@<sha> [--dry-run]`** (`lib/ship-queue-train-forget.ts`) removes one settled train (attempts included) or one READY receipt from local state and from the notes ref with an exact `--force-with-lease` on `refs/notes/gstack-trains`; a merging, gate-pending or free-pending train is refused.
|
||||
- **`verify --post-merge <sha> [--timeout <s>]`** (`lib/ship-queue-post-merge.ts`): after a merge, waits for every check run attached to the merged commit to settle, prints one `CHECK: <name> <conclusion>` per check and `POST_MERGE: <sha12> green|red|pending|none (passed= failed= pending= skipped=)`; green needs a completed check and no failure, `neutral`/`skipped` count as skipped, a failure is **`QUEUE_POST_MERGE_RED`**, a timeout is `QUEUE_TRAIN_PENDING` with the same command as `NEXT:`.
|
||||
- **`ready … --dispatch-ci <wf.yml[,…]> [--timeout <s>]`** (`lib/ship-queue-dispatch-ci.ts`): for a PR that conflicts with the base and so has no pull_request CI, dispatches each named workflow on the branch (skipped when a run at this head already concluded), waits for the runs (`CI_DISPATCH:` lines), clears every requires-remote pregate lane whose workflow is one of them with the run URL (`PREGATE_CLEARED:`), then reads the head as usual. A red run clears no lane and refuses; a fork head is `QUEUE_TRAIN_UNTRUSTED`. This replaces the four-step hand procedure in docs/ship-policy.md.
|
||||
- **`train build --dry-run --stack-tests [--stack-tests-command "<cmd>"]`** (`lib/ship-queue-stack-tests.ts`): runs each PR's new test files on the integrated prefix that includes it (T<k>) in a scratch worktree and prints `STACK_TESTS: #<pr> T<k> <n> new test file(s) pass|FAIL (<files>)` or `no new test files`; a failure is **`QUEUE_TRAIN_STACK_RED`** with nothing written, pushed or dispatched.
|
||||
|
||||
### Changed
|
||||
- **`lib/ship-queue-train-github.ts`**: `downloadJson` finds a named file anywhere in a multi-path artifact (`report-verdict-a1` keeps its `paid-report/` directory). **`lib/ship-queue.ts`**: `ciAtHead` is exported and returns the per-check list it already read. **`scripts/lib/windows-curation.ts`**: `test/ship-queue-train-ops.test.ts` is excluded from the Windows subset like the other gh-shim drivers.
|
||||
- docs/ship-policy.md's merge-train section documents every command above; docs/troubleshooting.md gains `QUEUE_SCREEN_RED`, `QUEUE_SCREEN_NOT_RUN`, `QUEUE_TRAIN_STACK_RED` and `QUEUE_POST_MERGE_RED`, and `QUEUE_SCREEN_STALE_REF` now means a real conflict only.
|
||||
|
||||
### Tests
|
||||
- `test/ship-queue-train-ops.test.ts` (new; the gh shim in `test/helpers/ship-queue-train-fixture.ts` gained `binding`, `outcomes`, `reuse`, `plan: null` and `ci` rules with the real artifact layout): a RED screen refuses ready under either choice, is printed by verify --screen with its red and flaky cases and refuses the pin in train build; a GREEN screen with flaky cases is followed and recorded; a NOT-RUN screen refuses verify with `QUEUE_SCREEN_NOT_RUN`, records `NOT_RUN` and is re-dispatched; a lagging or missing merge ref gets a scratch merge as `screen/<pr>-<head12>` and a real conflict stays `QUEUE_SCREEN_STALE_REF`; prune --stale settles only an all-merged-or-closed train and keeps one with an OPEN PR; train forget removes a train or a receipt and refuses a merging train; watch resumes from free-pending through the gate to the verdict and never dispatches twice; verify --post-merge waits, is red with the failed names, green when settled, resumable after a timeout; ready --dispatch-ci dispatches, waits, clears the matching lanes, refuses a red run and a fork head; --stack-tests catches a PR whose new test fails on T2 and passes after `--order` moves it first. Each was red before its change.
|
||||
- `test/ship-queue-train.test.ts`: the stale-ref test now proves the scratch merge path; the planned-0-cases test expects the SCREEN: verdict lines.
|
||||
|
||||
### Gate integrity
|
||||
- No threshold, rubric, trial count, case kind or judge prompt changed. A screen's verdict is read as evals.yml bound it; nothing here changes what a case must pass.
|
||||
|
||||
## [1.91.246.0] - 2026-10-26
|
||||
|
||||
**The README is a first-ten-minutes page again: what gstack is, how to install it, the first three skills to try, the skills tables, platform notes, and links to docs/ for everything else.** It was 862 lines and 90 KB, with CSO producer sandboxing, Codex model-selection order and the uninstall script in the path of someone deciding whether to try it. It is now about 300 lines and 33 KB. Nothing was deleted: every removed section moved to a docs/ page, verbatim except the "Proactive skill suggestions" paragraph in docs/why-gstack.md, which now describes the core-set default and `gstack-config set proactive core|all|off` landing in #3432, and every anchor other files link to (`#install--about-30-seconds`, `#gbrain--persistent-knowledge-for-your-coding-agent`) still resolves.
|
||||
|
||||
@@ -12,7 +12,9 @@
|
||||
* gstack-ship-queue gate-ahead --after <pr> --pr <n> [--predecessor-ref <ref>] [--allow-repo-commands]
|
||||
* gstack-ship-queue gate-ahead --record --pr <n> --attempt <id> --exit <rc> --log <file>
|
||||
* gstack-ship-queue gate-ahead --list --pr <n>
|
||||
* gstack-ship-queue train build|status|verdict|bisect|rebuild|prune ... (merge trains; train --help)
|
||||
* gstack-ship-queue verify --pr <n> --screen (train policy: the head's screen verdict from its run)
|
||||
* gstack-ship-queue verify --post-merge <sha> [--timeout <s>] (after a merge: main's push workflows at that commit until they settle)
|
||||
* gstack-ship-queue train build|status|verdict|bisect|rebuild|watch|prune ... (merge trains; train --help)
|
||||
*
|
||||
* Every subcommand takes --base <branch> (default origin/HEAD, else main) and
|
||||
* ends with GSTACK_RESULT: skill=ship-queue status=complete|refused run=<path>.
|
||||
@@ -47,7 +49,10 @@ const HELP = `gstack-ship-queue — the shared merge queue's protocol steps (doc
|
||||
gate-ahead --record --pr <n> --attempt <id> --exit <rc> --log <file>
|
||||
bind the gate result to that attempt's gated tree; removes its worktree
|
||||
gate-ahead --list --pr <n> attempts for a PR, newest last (the current one)
|
||||
train build|status|verdict|bisect|rebuild|prune ...
|
||||
verify --pr <n> --screen (train policy) the PR head's screen verdict read from its run (SCREEN_VERDICT)
|
||||
verify --post-merge <sha> [--timeout <s>] wait for every check run at a merged commit to settle: POST_MERGE: green|red|pending
|
||||
and one CHECK: line per workflow; resumable, default timeout 2700 s
|
||||
train build|status|verdict|bisect|rebuild|watch|prune ...
|
||||
merge trains, when the base policy declares a train block (train --help;
|
||||
docs/ship-policy.md#run-a-merge-train); ready then takes --screen|--no-screen
|
||||
|
||||
@@ -146,6 +151,7 @@ async function main(argv: string[]): Promise<number> {
|
||||
if (!args[0] || args[0].startsWith('-')) { process.stdout.write(train.TRAIN_HELP + '\n'); return args.includes('--help') || args.includes('-h') ? EXIT.ok : EXIT.usage; }
|
||||
return train.trainMain(ctx, args);
|
||||
}
|
||||
if (cmd === 'verify' && flag(args, '--post-merge') !== undefined) return (await import('../lib/ship-queue-train-cli')).postMergeMain(ctx, args);
|
||||
if (!pr || !/^\d+$/.test(pr)) return usage(`${cmd} needs --pr <number>`);
|
||||
const gateExit = flag(args, '--gate-exit');
|
||||
if (gateExit !== undefined && !/^-?\d+$/.test(gateExit)) return usage('--gate-exit needs an exit code');
|
||||
@@ -168,7 +174,7 @@ async function main(argv: string[]): Promise<number> {
|
||||
}));
|
||||
}
|
||||
if (cmd === 'verify') {
|
||||
if (flag(args, '--train')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr);
|
||||
if (flag(args, '--train') || args.includes('--screen')) return (await import('../lib/ship-queue-train-cli')).verifyMain(ctx, args, pr);
|
||||
const record = flag(args, '--record');
|
||||
return finish('verify', verify(ctx, {
|
||||
pr, gated: flag(args, '--gated'), record: record ? path.resolve(record) : undefined, receipt: flag(args, '--receipt'),
|
||||
|
||||
+74
-9
@@ -186,14 +186,20 @@ READY PR as conflicting each time main stamps; the train merges the CHANGELOG
|
||||
by section, so the lane never merges main for it. The conflict does mean GitHub
|
||||
has no merge ref for the PR, so no `pull_request` CI runs on the head and
|
||||
`ready` prints `ci: none at <head12>` with a `ci-hint:` line. The lane then
|
||||
dispatches the free workflows on the branch (`gh workflow run free-tests.yml
|
||||
--ref <branch> && gh workflow run windows-free-tests.yml --ref <branch>`),
|
||||
clears the requires-remote pregate lanes with those runs
|
||||
(`gstack-pregate clear --remote <lane> --run-url <run>`) and reruns `ready`,
|
||||
which passes because the dispatched check runs attach to the commit; the
|
||||
`ci-hint:` line then says those non-pull_request runs count. The `ci-hint:` line is
|
||||
advisory text for the lane; nothing parses it, and a PR whose head is on a fork
|
||||
gets a hint that leaves the dispatch to the coordinator. `ready --screen` on such a
|
||||
runs `ready … --dispatch-ci free-tests.yml,windows-free-tests.yml [--timeout
|
||||
<s>]` (under a train policy): it dispatches each named workflow on the branch
|
||||
(`gh workflow run <wf> --ref <branch>`, skipped when a run at this head already
|
||||
concluded), waits for the runs (`CI_DISPATCH: <wf> --ref <branch> run <id> <url>
|
||||
<conclusion>`), clears every requires-remote pregate lane whose workflow is one
|
||||
of them with the run URL (`PREGATE_CLEARED: <lane> <url>`; a lane of another
|
||||
workflow stays uncleared and is named in the refusal), and then reads the head
|
||||
as usual, which passes because the dispatched check runs attach to the commit;
|
||||
the `ci-hint:` line then says those non-pull_request runs count. A red run
|
||||
clears no lane and refuses (`QUEUE_NOT_READY`); a fork head is never dispatched
|
||||
here (`QUEUE_TRAIN_UNTRUSTED`). By hand the same four steps are `gh workflow
|
||||
run <wf> --ref <branch>`, wait, `gstack-pregate clear --remote <lane> --run-url
|
||||
<run>`, rerun `ready`. The `ci-hint:` line is advisory text for the lane;
|
||||
nothing parses it. `ready --screen` on such a
|
||||
PR refuses, because there is no merge ref to screen (`QUEUE_SCREEN_STALE_REF`,
|
||||
`<remote> has no refs/pull/<n>/merge`); that refusal's "merge <base> into the
|
||||
branch" advice does not apply to a CHANGELOG-only conflict, so the lane READYs
|
||||
@@ -544,9 +550,68 @@ It runs the normal free tests, full gate and verification on main plus the
|
||||
repair. When verify confirms the repair merged, it prints `FREEZE: cleared`
|
||||
and `NEXT:` builds a train of the PRs still unmerged.
|
||||
|
||||
**5b. Coordinator: after the merge.** `gstack-ship-queue verify --post-merge
|
||||
<sha> [--timeout <s>]` watches every check run attached to the merged commit
|
||||
(the push workflows: free tests, the Windows subset, the main backstop) until
|
||||
none is still running, prints one `CHECK: <name> <conclusion>` per check and
|
||||
`POST_MERGE: <sha12> green|red|pending|none (passed= failed= pending= skipped=)`.
|
||||
Green needs at least one completed check and no failure; `neutral` and
|
||||
`skipped` (the backstop superseding itself) count as skipped. A failure is
|
||||
`QUEUE_POST_MERGE_RED` naming the checks; a timeout (default 2700 s) is
|
||||
`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`. It keeps no process
|
||||
state, so rerunning it after a machine restart continues the watch.
|
||||
|
||||
**6. Clean up.** `gstack-ship-queue train prune [--train <nonce>] [--dry-run]`
|
||||
deletes the `train/<nonce>` branches of merged, superseded and dropped trains
|
||||
(`WOULD-DELETE:`, `DELETED:`, `GONE:`).
|
||||
(`WOULD-DELETE:`, `DELETED:`, `GONE:`). `train prune --stale` also settles a
|
||||
`merging` or `green` train whose every PR is MERGED or CLOSED on GitHub (its
|
||||
PRs landed through another train or by hand): it logs each PR's state on a
|
||||
`STALE:` line, marks the record `stale`, deletes its branch and any
|
||||
`screen/<pr>-*` scratch branches of those PRs; a train with an OPEN PR is kept
|
||||
(`KEEP: … still needs the train`). `train forget --train <nonce> | --receipt
|
||||
<pr>@<sha> [--dry-run]` removes one record (a settled train with its attempts,
|
||||
or one READY receipt) from local state and from `refs/notes/gstack-trains`
|
||||
with an exact lease on the notes ref, printing `FORGET:` then `FORGOTTEN:`; a
|
||||
merging, gate-pending or free-pending train is refused.
|
||||
|
||||
**Following a train without a daemon.** `gstack-ship-queue train watch --train
|
||||
<nonce> [--timeout <s>]` is the one resumable follow command: it reads the
|
||||
record, runs the step its state asks for (`train build --train` to push, run
|
||||
the free tests and dispatch the gate; `train status` to settle an unconfirmed
|
||||
dispatch), waits for the pending run by polling gh, reads the verdict and
|
||||
prints the last step's `NEXT:`. It never dispatches anything the record would
|
||||
not dispatch on its own, so rerunning it after an interruption continues from
|
||||
the record (`WATCH:` lines say what it did; a timeout, default 7200 s, is
|
||||
`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`).
|
||||
|
||||
**Screens are read from the run.** `ready --screen` and `verify --pr <n>
|
||||
--screen` read a dispatched screen's verdict from the run's own artifacts
|
||||
(`gate-binding-a<attempt>`, `report-verdict-a<attempt>`), never from a lane's
|
||||
prose, and record it in the READY receipt (`screen.verdict`): `SCREEN_VERDICT:
|
||||
#<n> run <id> GREEN|RED|NOT-RUN|INCOMPLETE [red=…] [flaky=…]` plus one
|
||||
`FLAKY:` line per rule-16 flaky case. A RED screen refuses `ready` under
|
||||
either choice and is never re-dispatched at the same merge SHA
|
||||
(`QUEUE_SCREEN_RED`); a run that concluded success with no slice (its route
|
||||
job found the head superseded or the plan selected nothing) is `NOT-RUN`:
|
||||
`verify --screen` refuses with `QUEUE_SCREEN_NOT_RUN`, `ready --screen`
|
||||
dispatches afresh instead of following it, and `train build` seats the pin
|
||||
unscreened with a `SCREEN: … counted as not-run` line. When GitHub's
|
||||
`refs/pull/<n>/merge` lags the PR head or is missing, `ready --screen` plans
|
||||
the screen on a scratch merge of the head onto the base, pushed as
|
||||
`screen/<pr>-<head12>` (`SCREEN_MERGE:` names it; `prune --stale` deletes it
|
||||
once the PR merged); only a real conflict with the base is
|
||||
`QUEUE_SCREEN_STALE_REF`.
|
||||
|
||||
**Stacked tests before the gate.** `train build --prs … --dry-run --stack-tests
|
||||
[--stack-tests-command "<cmd>"]` runs each PR's new test files (added relative
|
||||
to the base, under a test directory or named `*.test.*`/`*.spec.*`) on the
|
||||
integrated prefix that includes it, T<k>, in a scratch worktree, and prints
|
||||
`STACK_TESTS: #<pr> T<k> <n> new test file(s) pass|FAIL (<files>)` or `no new
|
||||
test files`. A failure is `QUEUE_TRAIN_STACK_RED` (nothing written, pushed or
|
||||
dispatched): the tests pass on the PR head alone only because nothing before
|
||||
it in the train had changed what they read, so fix them against that prefix or
|
||||
reorder with `--order`. The default command is `bun --no-env-file
|
||||
--config=/dev/null test` with the files appended.
|
||||
|
||||
**Escape hatches.** Remove the `train` block on the base branch to return
|
||||
every lane to the per-PR queue at once. `--urgent` ships one PR alone,
|
||||
|
||||
+97
-5
@@ -4134,11 +4134,103 @@ merge ref at all, it refuses with the same code and a `could not list …` or
|
||||
`could not fetch …` message whose NEXT line says to check the remote, not to
|
||||
resolve a conflict.
|
||||
|
||||
**Fix.** Resolve the conflict with the base branch (or wait for GitHub to
|
||||
recompute the merge ref), push, and run `gstack-ship-queue ready --pr <n>
|
||||
--screen` again. Under crew rule 14 a CHANGELOG-only conflict is left to the
|
||||
train, so a lane that cannot get a fresh merge ref READYs `--no-screen` only
|
||||
after its own focused paid cases passed (rule 6).
|
||||
**Fix.** Since the scratch-merge screen landed, a lagging merge ref is no
|
||||
longer a refusal: `ready --screen` builds the merge of the PR head onto
|
||||
`origin/<base>` itself with the train's merge rules (version files take the
|
||||
base's side, the CHANGELOG merges by section), pushes it as
|
||||
`screen/<pr>-<head12>` and dispatches the screen on that commit, printing
|
||||
`SCREEN_MERGE: scratch merge …`. The code now means a real conflict: the
|
||||
message lists the conflicting paths. Resolve those with the base branch,
|
||||
push, and run `gstack-ship-queue ready --pr <n> --screen` again. A
|
||||
CHANGELOG-only or stamp-file-only conflict never refuses. `train prune
|
||||
--stale` deletes the scratch branches of merged or closed PRs.
|
||||
|
||||
<a id="queue-screen-red"></a>
|
||||
### `QUEUE_SCREEN_RED`
|
||||
|
||||
**Meaning.** The screen run dispatched for this PR head finished RED. The
|
||||
verdict is read from the run itself, never from a lane's report: the
|
||||
`gate-binding-a<attempt>` artifact evals.yml's report job uploads (verdict,
|
||||
reconcile exit, flaky cases) and the `report-verdict-a<attempt>` artifact's
|
||||
`collector-outcomes.json` (the panels that failed the lane and the rule-16
|
||||
flaky lines). A case the binding calls red already failed twice inside the
|
||||
run (rule 16 reruns a first failure once), so the red is real. `ready`
|
||||
refuses with this code when the head's recorded screen run is red, whichever
|
||||
of `--screen` or `--no-screen` it is given, and never dispatches another
|
||||
screen at the same merge SHA; `train build` refuses to seat the pin and its
|
||||
`NEXT:` names the build without it; `verify --pr <n> --screen` prints
|
||||
`SCREEN_VERDICT: #<n> run <id> RED red=<cases> flaky=<cases>` and one
|
||||
`FLAKY:` line per flaky case. The verdict is recorded in the READY receipt
|
||||
(`screen.verdict`), locally and in `refs/notes/gstack-trains`.
|
||||
|
||||
**Fix.** Read the red cases from the `SCREEN_VERDICT:` line, repair, push a
|
||||
new head, run /ship and `gstack-ship-queue ready --pr <n> --screen` again
|
||||
(the new head gets its own screen). A GREEN screen with flaky cases is green:
|
||||
the flaky cases are listed for the ledger and need no rerun.
|
||||
|
||||
<a id="queue-train-stack-red"></a>
|
||||
### `QUEUE_TRAIN_STACK_RED`
|
||||
|
||||
**Meaning.** `gstack-ship-queue train build --dry-run --stack-tests` ran a PR's
|
||||
new test files (files the PR adds relative to the base under a test directory
|
||||
or named `*.test.*` / `*.spec.*`) on the integrated prefix that includes it,
|
||||
T<k> (the base plus every slot up to and including that PR), in a scratch
|
||||
worktree, and they failed. The `STACK_TESTS: #<pr> T<k> <n> new test file(s)
|
||||
FAIL (<files>): <failing assertion>` line names the files and the runner's
|
||||
failing lines. Those tests pass on the PR head alone only because nothing
|
||||
before it in the train had changed what they read (a PR stacked on another
|
||||
PR's test files or fixtures hit this on 2026-10-26). Nothing was written,
|
||||
pushed or dispatched; the ordinary dry run (without `--stack-tests`) does not
|
||||
run them.
|
||||
|
||||
**Fix.** Fix the PR's tests against that prefix, or reorder the train with
|
||||
`--order` so the PR they depend on comes first, then rerun the dry run. Pass
|
||||
`--stack-tests-command "<cmd>"` when the repo's tests run under something
|
||||
other than `bun test`; the files are appended to the command.
|
||||
|
||||
<a id="queue-post-merge-red"></a>
|
||||
### `QUEUE_POST_MERGE_RED`
|
||||
|
||||
**Meaning.** `gstack-ship-queue verify --post-merge <sha>` watched the check
|
||||
runs attached to a commit the train merged onto the base branch (the push
|
||||
workflows: free tests, the Windows subset, the main backstop) until none was
|
||||
still running, and at least one concluded `failure`, `timed_out`,
|
||||
`cancelled` or another non-success. A `neutral` or `skipped` check (the
|
||||
backstop superseding itself when main moved on) is counted as skipped, never
|
||||
red. The command keeps no process state: it polls gh until the checks settle
|
||||
or `--timeout` (default 2700 s) passes, prints `POST_MERGE: <sha12>
|
||||
green|red|pending|none (passed= failed= pending= skipped=)` and one
|
||||
`CHECK: <name> <conclusion|status>` line per check, and a timeout is
|
||||
`QUEUE_TRAIN_PENDING` with the same command as `NEXT:`, so rerunning it after
|
||||
an interruption continues the watch.
|
||||
|
||||
**Fix.** Read the red run named on its `CHECK:` line. Repair on a new PR or
|
||||
roll back the merge; `gstack-ship-queue train status` names the freeze if the
|
||||
backstop filed one.
|
||||
|
||||
<a id="queue-screen-not-run"></a>
|
||||
### `QUEUE_SCREEN_NOT_RUN`
|
||||
|
||||
**Meaning.** The screen run dispatched for this PR concluded `success` but
|
||||
ran no slice. evals.yml's `route` job plans nothing when
|
||||
`scripts/eval-readiness.ts wait` finds the PR head superseded (a push landed
|
||||
between `ready --screen` and the plan) or when the plan selects no case; the
|
||||
run then uploads no `paid-plan` artifact (or an empty manifest), every slice
|
||||
job is skipped and GitHub shows a green check. On 2026-10-26, 18 of 41 PR
|
||||
screens ended this way. The verdict is read from the run (`lib/ship-queue-screen.ts`),
|
||||
so such a run is `NOT-RUN`: `verify --pr <n> --screen` prints
|
||||
`SCREEN_VERDICT: #<n> run <id> NOT-RUN (<why>)` and refuses with this code, the
|
||||
READY receipt records `screen.verdict.outcome: NOT_RUN`, `ready --screen` never
|
||||
follows the run (its `SCREEN_DISPATCH:` line says `not following … a screen
|
||||
that planned nothing is not a screen`) and `train build` seats the pin unscreened
|
||||
with `SCREEN: #<n> run <id> planned no cases …; counted as not-run, the slot
|
||||
rides unscreened`.
|
||||
|
||||
**Fix.** `gstack-ship-queue ready --pr <n> --screen` at the current head
|
||||
dispatches a fresh screen. A head that moved since the dispatch needs a fresh
|
||||
READY anyway (`QUEUE_TRAIN_MOVED`); a head that did not move and still plans
|
||||
nothing has nothing diff-relevant to screen, and `--no-screen` with the
|
||||
focused cases named in the READY block is the honest choice.
|
||||
|
||||
<a id="ready-screen-choice-required"></a>
|
||||
### `READY_SCREEN_CHOICE_REQUIRED`
|
||||
|
||||
@@ -148,6 +148,26 @@ export const RELEASE_RESULT_CODES = {
|
||||
summary: 'GitHub’s refs/pull/<n>/merge is missing or still merges an older head than the PR’s: the PR conflicts with the base (a CHANGELOG-only conflict counts), so a screen dispatched on it would plan no cases and report success',
|
||||
fix: 'resolve the conflict with the base branch (or wait for GitHub to recompute the merge ref), push, then gstack-ship-queue ready --pr <n> --screen again; a lane never READYs on a screen that planned nothing',
|
||||
},
|
||||
QUEUE_SCREEN_NOT_RUN: {
|
||||
anchor: 'queue-screen-not-run',
|
||||
summary: 'the PR’s dispatched screen run concluded success but ran no slice (the run planned no cases: its route job found the PR head superseded by a push, or the plan selected nothing), so the head is not screened and the run is counted as not-run, never as green',
|
||||
fix: 'gstack-ship-queue ready --pr <n> --screen dispatches a fresh screen at the current head (a not-run screen is never followed); train build seats the pin unscreened and says so on its SCREEN: line',
|
||||
},
|
||||
QUEUE_TRAIN_STACK_RED: {
|
||||
anchor: 'queue-train-stack-red',
|
||||
summary: 'train build --dry-run --stack-tests ran a PR’s new test files on the integrated prefix that includes it (T<k>: main plus every slot up to that PR) and they failed; the tests pass on the PR head alone only because nothing before it in the train had changed what they read',
|
||||
fix: 'fix the PR’s tests against that prefix (read the STACK_TESTS: line for the files and the failing assertion), or reorder the train with --order so the PR they depend on comes first, then rerun the dry run; nothing was written, pushed or dispatched',
|
||||
},
|
||||
QUEUE_POST_MERGE_RED: {
|
||||
anchor: 'queue-post-merge-red',
|
||||
summary: 'a push workflow on the merged commit (free tests, the Windows subset, the main backstop) concluded failure, timed out or was cancelled after the train merged it; a neutral or skipped check is not red',
|
||||
fix: 'read the red run (gstack-ship-queue verify --post-merge <sha> prints one CHECK: line per workflow), repair on a new PR or roll back the merge; gstack-ship-queue train status names the freeze if the backstop filed one',
|
||||
},
|
||||
QUEUE_SCREEN_RED: {
|
||||
anchor: 'queue-screen-red',
|
||||
summary: 'the PR head’s dispatched screen run finished RED (its gate binding says so; a case that failed twice is real), so the head is not READY and a train will not seat it',
|
||||
fix: 'read the SCREEN_VERDICT line (gstack-ship-queue verify --pr <n> --screen prints the red and flaky cases), repair, push a new head, run /ship and ready again; a red screen is never re-dispatched at the same merge SHA',
|
||||
},
|
||||
READY_SCREEN_CHOICE_REQUIRED: {
|
||||
anchor: 'ready-screen-choice-required',
|
||||
summary: 'the base policy turns trains on, so READY needs an explicit screen choice: --screen (pay for a diff-relevant screen now) or --no-screen',
|
||||
|
||||
@@ -0,0 +1,81 @@
|
||||
/**
|
||||
* `ready --dispatch-ci <workflow.yml[,…]> [--timeout <s>]`: a PR that
|
||||
* conflicts with the base (a CHANGELOG-only conflict after every stamp
|
||||
* counts) has no merge ref, so GitHub runs no pull_request CI on its head and
|
||||
* `ready` prints `ci: none`. The lane used to dispatch the free workflows by
|
||||
* hand, wait, clear the requires-remote pregate lanes with the run URLs and
|
||||
* rerun ready. This does those four steps before `ready` reads the head: one
|
||||
* `gh workflow run <wf> --ref <branch>` per named workflow (skipped when a
|
||||
* run at this head already concluded), a wait for each run, and one
|
||||
* `PREGATE_CLEARED:` per requires-remote lane whose workflow is one of them
|
||||
* and whose run concluded success. A red run clears no lane and refuses. A
|
||||
* fork head is never dispatched here (QUEUE_TRAIN_UNTRUSTED).
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import type { PregateFile } from './headless-artifacts';
|
||||
import { PREGATE_OUT_REL, clearObligation, readPregateFile, writePregateFile } from './pregate/index';
|
||||
import type { QueueContext } from './ship-queue';
|
||||
import type { QueueFailure } from './ship-queue-gate';
|
||||
import { execFor, repoSlug, runView } from './ship-queue-train-github';
|
||||
|
||||
const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
export interface DispatchCiOptions { pr: string; workflows: string[]; pregate?: string; timeoutSec: number; pollMs: number }
|
||||
export type DispatchCiResult = { lines: string[] } | { lines: string[]; failure: QueueFailure };
|
||||
|
||||
interface RunRow { databaseId: number; url: string; status: string; conclusion: string | null; headSha: string }
|
||||
|
||||
function runsAt(ctx: QueueContext, workflow: string, branch: string, head: string): RunRow[] {
|
||||
const r = ctx.gh(['run', 'list', '--workflow', workflow, '--branch', branch, '--event', 'workflow_dispatch', '--limit', '20', '--json', 'databaseId,url,status,conclusion,headSha']);
|
||||
try { return r.ok ? (JSON.parse(r.out) as RunRow[]).filter(x => x.headSha === head) : []; } catch { return []; }
|
||||
}
|
||||
|
||||
export async function dispatchCi(ctx: QueueContext, o: DispatchCiOptions): Promise<DispatchCiResult> {
|
||||
const view = ctx.gh(['pr', 'view', o.pr, '--json', 'headRefOid,headRefName,isCrossRepository']);
|
||||
let info: { headRefOid?: string; headRefName?: string; isCrossRepository?: boolean } = {};
|
||||
try { info = view.ok ? JSON.parse(view.out) : {}; } catch {}
|
||||
if (!info.headRefOid || !info.headRefName) return { lines: [], failure: { code: 'QUEUE_NOT_READY', message: `cannot read PR #${o.pr}'s head and branch through gh (${(view.err ?? view.out).trim().slice(0, 120)})` } };
|
||||
if (info.isCrossRepository) return { lines: [], failure: { code: 'QUEUE_TRAIN_UNTRUSTED', message: `#${o.pr}'s head is on a fork; its workflows are not dispatched here (the coordinator decides)` } };
|
||||
const head = info.headRefOid;
|
||||
const branch = info.headRefName;
|
||||
const slug = repoSlug(ctx);
|
||||
const exec = execFor(ctx);
|
||||
const deadline = Date.now() + o.timeoutSec * 1000;
|
||||
const lines: string[] = [];
|
||||
const concluded = new Map<string, RunRow>();
|
||||
for (const wf of o.workflows) {
|
||||
const prior = runsAt(ctx, wf, branch, head).find(r => r.status === 'completed');
|
||||
if (prior) { lines.push(`CI_DISPATCH: ${wf} not dispatched (run ${prior.databaseId} already concluded ${prior.conclusion ?? 'unknown'} at this head)`); concluded.set(wf, prior); continue; }
|
||||
const live = runsAt(ctx, wf, branch, head)[0];
|
||||
if (!live) {
|
||||
const d = exec('gh', ['workflow', 'run', wf, '--ref', branch]);
|
||||
if (d.status !== 0) return { lines, failure: { code: 'QUEUE_NOT_READY', message: `gh workflow run ${wf} --ref ${branch} failed (${d.stdout.trim().slice(0, 120) || 'no output'})` } };
|
||||
}
|
||||
let run: RunRow | undefined = live;
|
||||
for (;;) {
|
||||
run = run && slug ? runView(ctx, slug, run.databaseId) ?? run : runsAt(ctx, wf, branch, head)[0];
|
||||
if (run?.status === 'completed') break;
|
||||
if (Date.now() >= deadline) return { lines: [...lines, `CI_DISPATCH: ${wf} --ref ${branch} ${run ? `run ${run.databaseId} ${run.url} ${run.status}` : 'no run appeared'}`], failure: { code: 'QUEUE_TRAIN_PENDING', message: `${wf} ${run ? `run ${run.databaseId} is still ${run.status}` : 'has no workflow_dispatch run at this head yet'} at the --dispatch-ci timeout; rerun ready --dispatch-ci to continue waiting` } };
|
||||
await sleep(o.pollMs);
|
||||
}
|
||||
lines.push(`CI_DISPATCH: ${wf} --ref ${branch} run ${run.databaseId} ${run.url} ${run.conclusion ?? 'unknown'}`);
|
||||
concluded.set(wf, run);
|
||||
}
|
||||
const red = [...concluded.entries()].filter(([, r]) => r.conclusion !== 'success');
|
||||
if (red.length) return { lines, failure: { code: 'QUEUE_NOT_READY', message: red.map(([wf, r]) => `${wf} run ${r.databaseId} concluded ${r.conclusion ?? 'unknown'} at ${head.slice(0, 12)}`).join('; ') + '; a red run clears no lane' } };
|
||||
const pregateFile = path.resolve(ctx.repoRoot, o.pregate ?? PREGATE_OUT_REL);
|
||||
if (!fs.existsSync(pregateFile)) return { lines };
|
||||
const read = readPregateFile(pregateFile);
|
||||
if (!read.file) return { lines };
|
||||
let file: PregateFile = read.file;
|
||||
let changed = false;
|
||||
for (const lane of (file.requires_remote ?? []).filter(l => !l.receipt && concluded.has(l.workflow))) {
|
||||
const run = concluded.get(lane.workflow)!;
|
||||
file = clearObligation(file, lane.lane, run.url, `gstack-ship-queue ready --dispatch-ci ${new Date().toISOString()}`);
|
||||
lines.push(`PREGATE_CLEARED: ${lane.lane} ${run.url}`);
|
||||
changed = true;
|
||||
}
|
||||
if (changed) writePregateFile(ctx.repoRoot, pregateFile, file);
|
||||
return { lines };
|
||||
}
|
||||
@@ -0,0 +1,43 @@
|
||||
/**
|
||||
* `verify --post-merge <sha> [--timeout <s>]`: the coordinator's follow
|
||||
* command after a train merged a PR onto the base branch. The push workflows
|
||||
* run on the merged commit (free tests, the Windows subset, the main backstop
|
||||
* that supersedes itself when main moves on), and a background `gh run
|
||||
* watch` chain dies with the machine that started it. This command keeps no
|
||||
* state: it reads every check run attached to the sha, waits (polling gh)
|
||||
* until none is still running or the timeout passes, prints one `CHECK:`
|
||||
* line per check, and a `POST_MERGE:` summary that is green only when at
|
||||
* least one check completed and none failed; a neutral or skipped check is
|
||||
* skipped, never red. Rerun it after an interruption and it continues.
|
||||
*/
|
||||
import { ciAtHead, type QueueContext } from './ship-queue';
|
||||
import { outFailed, outOk, outPending, type TrainOut } from './ship-queue-train';
|
||||
|
||||
const Q = 'gstack-ship-queue';
|
||||
const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
export interface PostMergeOptions { sha: string; timeoutSec: number; pollMs: number }
|
||||
|
||||
export async function verifyPostMerge(ctx: QueueContext, o: PostMergeOptions): Promise<TrainOut> {
|
||||
const deadline = Date.now() + o.timeoutSec * 1000;
|
||||
const at = o.sha.slice(0, 12);
|
||||
const again = `${Q} verify --post-merge ${o.sha} --timeout ${o.timeoutSec}`;
|
||||
for (;;) {
|
||||
const ci = ciAtHead(ctx, o.sha);
|
||||
const lines = [`POST_MERGE: ${ci.line.replace(/^(\S+) at ([0-9a-f]+)/, '$2 $1')}`, ...ci.checks.map(c => `CHECK: ${c.name} ${c.status === 'completed' ? c.conclusion ?? 'unknown' : c.status}`)];
|
||||
const settled = !ci.checks.some(c => c.status !== 'completed') && !ci.line.startsWith('unknown');
|
||||
if (settled && ci.green) return outOk(lines, 'none', 'none');
|
||||
if (settled && ci.checks.length) {
|
||||
const red = ci.checks.filter(c => c.status === 'completed' && !['success', 'skipped', 'neutral'].includes(c.conclusion ?? '')).map(c => c.name);
|
||||
return outFailed({ code: 'QUEUE_POST_MERGE_RED', message: `main's push workflows at ${at} are red: ${red.join(', ')}` }, lines,
|
||||
`read the red run, repair on a new PR or roll back; ${Q} train status names the freeze if the backstop filed one`, 'none');
|
||||
}
|
||||
if (Date.now() >= deadline) {
|
||||
const pending = ci.checks.filter(c => c.status !== 'completed').map(c => c.name);
|
||||
const message = ci.line.startsWith('unknown') ? `the check runs at ${at} are unreadable (${ci.line.replace(/^unknown at [0-9a-f]+ /, '')})`
|
||||
: ci.checks.length ? `main's push workflows at ${at} have not settled at the timeout: ${pending.join(', ')} still running` : `no check run has attached to ${at} yet`;
|
||||
return outPending({ code: 'QUEUE_TRAIN_PENDING', message }, lines, again, 'none');
|
||||
}
|
||||
await sleep(o.pollMs);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
/**
|
||||
* A dispatched screen's verdict, read from the run itself:
|
||||
* never from a lane's prose. evals.yml's report job publishes
|
||||
* `gate-binding-a<attempt>` (scripts/gate-reuse.ts: verdict, reconcile exit,
|
||||
* flaky cases) and `report-verdict-a<attempt>` (collector-outcomes.json: the
|
||||
* panels and the rule-16 flaky lines) on every executed screen; a superseded
|
||||
* screen plans nothing and uploads neither (hollowScreen): it concluded
|
||||
* success with no slice, so its outcome is `not-run`, never green. The outcome is
|
||||
* recorded in the READY receipt (ScreenChoice.verdict) by `ready`,
|
||||
* `verify --screen` and `train build`, so the controller reads one line.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { REUSE_FILE, parseBinding, parseReuse } from '../scripts/gate-reuse';
|
||||
import type { QueueContext } from './ship-queue';
|
||||
import { downloadJson, hollowScreen, runView } from './ship-queue-train-github';
|
||||
import { nowIso, type ScreenChoice, type ScreenVerdictRecord } from './ship-queue-train-record';
|
||||
|
||||
export type ScreenOutcome =
|
||||
| { kind: 'pending'; detail: string }
|
||||
| { kind: 'not-run'; detail: string }
|
||||
| { kind: 'unknown'; detail: string }
|
||||
| { kind: 'incomplete'; detail: string; flaky: string[] }
|
||||
| { kind: 'green'; detail: string; flaky: string[]; flakyLines: string[] }
|
||||
| { kind: 'red'; detail: string; red: string[]; flaky: string[]; flakyLines: string[] };
|
||||
|
||||
const BINDING = 'gate-binding';
|
||||
const REPORT = 'report-verdict';
|
||||
/** The plan artifact; a confirmed verdict reuse (R2a/R2) leaves its record there and binds nothing new. */
|
||||
const PLAN = 'paid-plan';
|
||||
|
||||
/** The newest attempt's artifact of a name family (`<name>-a<attempt>`); null when the run uploaded none or the list is unreadable. */
|
||||
function newestArtifact(ctx: QueueContext, slug: string, run: number, name: string): string | null | 'unreadable' {
|
||||
const r = ctx.gh(['api', `repos/${slug}/actions/runs/${run}/artifacts?per_page=100`]);
|
||||
let list: { artifacts?: Array<{ name?: string; expired?: boolean }> } | null = null;
|
||||
try { list = r.ok ? JSON.parse(r.out) : null; } catch {}
|
||||
if (!Array.isArray(list?.artifacts)) return 'unreadable';
|
||||
const hits = list.artifacts.map(a => ({ name: a.name ?? '', expired: !!a.expired, attempt: Number(new RegExp(`^${name}-a(\\d+)$`).exec(a.name ?? '')?.[1]) }))
|
||||
.filter(a => Number.isSafeInteger(a.attempt)).sort((a, b) => b.attempt - a.attempt);
|
||||
if (!hits.length) return null;
|
||||
return hits[0]!.expired ? 'unreadable' : hits[0]!.name;
|
||||
}
|
||||
|
||||
interface Outcomes { panels?: Array<{ case?: string; status?: string; failsLane?: boolean; redClass?: string | null }>; flakyLines?: string[] }
|
||||
|
||||
/** Read one screen run: status, plan, gate binding, panels. */
|
||||
export function screenVerdict(ctx: QueueContext, slug: string, run: number): ScreenOutcome {
|
||||
const view = runView(ctx, slug, run);
|
||||
if (!view) return { kind: 'unknown', detail: `run ${run} is unreadable` };
|
||||
if (view.status !== 'completed') return { kind: 'pending', detail: `run ${run} is ${view.status}` };
|
||||
const plan = hollowScreen(ctx, slug, run);
|
||||
if (plan.kind === 'hollow') return { kind: 'not-run', detail: plan.reason };
|
||||
if (plan.kind === 'unknown') return { kind: 'unknown', detail: plan.reason };
|
||||
const bindingName = newestArtifact(ctx, slug, run, BINDING);
|
||||
if (bindingName === 'unreadable') return { kind: 'unknown', detail: `run ${run}'s artifact list is unreadable or its ${BINDING} artifact has expired` };
|
||||
const scratch = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-screen-verdict-'));
|
||||
let binding: ReturnType<typeof parseBinding> = null;
|
||||
let outcomes: Outcomes | null = null;
|
||||
try {
|
||||
if (!bindingName) {
|
||||
const reuse = parseReuse(downloadJson(ctx, slug, run, PLAN, scratch, REUSE_FILE));
|
||||
if (reuse && view.conclusion === 'success') {
|
||||
const flaky = [...(reuse.reused_flaky ?? [])].sort();
|
||||
return { kind: 'green', detail: `reused the GREEN verdict of run ${reuse.reused_run_id} (${reuse.kind}): ${reuse.why}${flaky.length ? `, flaky ${flaky.length}` : ''}`, flaky, flakyLines: [] };
|
||||
}
|
||||
return { kind: 'incomplete', detail: `run ${run} concluded ${view.conclusion ?? 'unknown'} and bound no verdict (no ${BINDING} artifact: the report job did not finish)`, flaky: [] };
|
||||
}
|
||||
binding = parseBinding(downloadJson(ctx, slug, run, bindingName, scratch, 'gate-binding.json'));
|
||||
const reportName = newestArtifact(ctx, slug, run, REPORT);
|
||||
outcomes = typeof reportName === 'string' ? downloadJson(ctx, slug, run, reportName, scratch, 'collector-outcomes.json') as Outcomes | null : null;
|
||||
} finally { fs.rmSync(scratch, { recursive: true, force: true }); }
|
||||
if (!binding) return { kind: 'unknown', detail: `run ${run}'s ${bindingName} artifact is unreadable` };
|
||||
const flaky = [...(binding.flaky_cases ?? [])].sort();
|
||||
const flakyLines = Array.isArray(outcomes?.flakyLines) ? outcomes!.flakyLines!.filter((l): l is string => typeof l === 'string') : [];
|
||||
const red = (outcomes?.panels ?? []).filter(p => typeof p.case === 'string' && (p.failsLane === true || (p.status === 'FAIL' && p.redClass === 'VERDICT'))).map(p => p.case!).sort();
|
||||
const base = `verdict ${binding.verdict}, reconcile exit ${binding.reconcile_exit}, run concluded ${view.conclusion ?? 'unknown'}${flaky.length ? `, flaky ${flaky.length}` : ''}`;
|
||||
if (binding.verdict === 'GREEN' && binding.reconcile_exit === 0 && view.conclusion === 'success') return { kind: 'green', detail: base, flaky, flakyLines };
|
||||
if (binding.verdict === 'RED' || binding.reconcile_exit !== 0 || red.length) return { kind: 'red', detail: base, red, flaky, flakyLines };
|
||||
return { kind: 'incomplete', detail: base, flaky };
|
||||
}
|
||||
|
||||
/** The receipt field for an outcome that is settled (a pending or unknown run records nothing). */
|
||||
export function verdictRecord(o: ScreenOutcome, run: number): ScreenVerdictRecord | null {
|
||||
if (o.kind === 'pending' || o.kind === 'unknown') return null;
|
||||
const outcome = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind === 'not-run' ? 'NOT_RUN' : 'INCOMPLETE';
|
||||
return { outcome, run, red: o.kind === 'red' ? o.red : [], flaky: 'flaky' in o ? o.flaky : [], detail: o.detail, read_at: nowIso() };
|
||||
}
|
||||
|
||||
/** `SCREEN_VERDICT:` and the per-case `FLAKY:` lines for one PR's screen. */
|
||||
export function verdictLines(pr: string, run: number, o: ScreenOutcome): string[] {
|
||||
const word = o.kind === 'green' ? 'GREEN' : o.kind === 'red' ? 'RED' : o.kind.toUpperCase();
|
||||
const red = o.kind === 'red' && o.red.length ? ` red=${o.red.join(',')}` : '';
|
||||
const flaky = 'flaky' in o && o.flaky.length ? ` flaky=${o.flaky.join(',')}` : '';
|
||||
const lines = [`SCREEN_VERDICT: #${pr} run ${run} ${word}${red}${flaky} (${o.detail})`];
|
||||
if ('flakyLines' in o) for (const l of o.flakyLines) lines.push(`FLAKY: ${l}`);
|
||||
return lines;
|
||||
}
|
||||
|
||||
/** Apply a read outcome to a receipt's screen choice; returns whether the receipt changed. */
|
||||
export function recordVerdict(screen: ScreenChoice, o: ScreenOutcome): boolean {
|
||||
if (!screen.run) return false;
|
||||
const rec = verdictRecord(o, screen.run);
|
||||
if (!rec) return false;
|
||||
const same = screen.verdict && screen.verdict.outcome === rec.outcome && screen.verdict.run === rec.run && screen.verdict.red.join() === rec.red.join() && screen.verdict.flaky.join() === rec.flaky.join();
|
||||
if (same) return false;
|
||||
screen.verdict = rec;
|
||||
return true;
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
/**
|
||||
* `train build --dry-run --stack-tests`: before a train spends a gate, run
|
||||
* each PR's new test files on the integrated prefix that includes it (T<k>,
|
||||
* the merge of main and every slot up to and including the PR), not on the
|
||||
* PR head alone. A test that passes on its branch and fails once an earlier
|
||||
* slot changed what it reads is a stack red, caught here for the price of a
|
||||
* few test files instead of a gate. Each prefix is checked out in a scratch
|
||||
* worktree (the repo's node_modules linked in when it has one), the files run
|
||||
* with the test command (`bun --no-env-file --config=/dev/null test` by
|
||||
* default; `--stack-tests-command "<cmd>"` for another runner, the files
|
||||
* appended), and the worktree is removed. A PR that adds no test file is
|
||||
* reported as such and never fails.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import * as os from 'node:os';
|
||||
import * as path from 'node:path';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { bunIsolationArgs } from './bun-isolation';
|
||||
import { git } from './ship-queue-gate';
|
||||
import type { TrainPrefix, TrainSlot } from './ship-queue-train-record';
|
||||
|
||||
export const DEFAULT_STACK_TEST_COMMAND = ['bun', ...bunIsolationArgs(), 'test'];
|
||||
const TEST_FILE = /(?:^|\/)(?:tests?|__tests__|spec)\/.*\.[cm]?[jt]sx?$|\.(?:test|spec)\.[cm]?[jt]sx?$/;
|
||||
|
||||
export interface StackTestResult { pr: string; prefix: number; files: string[]; ok: boolean; detail: string }
|
||||
|
||||
/** Test files the PR adds relative to the base (added, not modified). */
|
||||
export function newTestFiles(repoRoot: string, baseSha: string, head: string): string[] {
|
||||
const r = git(repoRoot, ['diff', '--name-only', '--diff-filter=A', `${baseSha}...${head}`]);
|
||||
return r.ok ? r.out.split('\n').map(s => s.trim()).filter(f => f && TEST_FILE.test(f)) : [];
|
||||
}
|
||||
|
||||
export function stackTests(repoRoot: string, baseSha: string, slots: TrainSlot[], prefixes: TrainPrefix[], command: string[] = DEFAULT_STACK_TEST_COMMAND, timeoutMs = 600_000): StackTestResult[] {
|
||||
const results: StackTestResult[] = [];
|
||||
for (const [i, slot] of slots.entries()) {
|
||||
const prefix = prefixes[i + 1];
|
||||
if (!prefix) break;
|
||||
const files = newTestFiles(repoRoot, baseSha, slot.head);
|
||||
if (!files.length) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: true, detail: 'no new test files' }); continue; }
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gstack-stack-tests-'));
|
||||
try {
|
||||
const add = git(repoRoot, ['worktree', 'add', '--detach', '-q', dir, prefix.commit]);
|
||||
if (!add.ok) { results.push({ pr: slot.pr, prefix: prefix.index, files, ok: false, detail: `cannot check out T${prefix.index}: ${add.err.slice(0, 160)}` }); continue; }
|
||||
const modules = path.join(repoRoot, 'node_modules');
|
||||
if (fs.existsSync(modules) && !fs.existsSync(path.join(dir, 'node_modules'))) fs.symlinkSync(modules, path.join(dir, 'node_modules'), 'dir');
|
||||
const [bin, ...args] = command;
|
||||
const run = spawnSync(bin!, [...args, ...files], { cwd: dir, encoding: 'utf8', timeout: timeoutMs, env: { ...process.env, CI: process.env.CI ?? '1' }, maxBuffer: 64 * 1024 * 1024 });
|
||||
const tail = `${run.stdout ?? ''}\n${run.stderr ?? ''}`.split('\n').filter(l => /^\(fail\)|^\s*\d+ (pass|fail)$|^error:/.test(l)).slice(-6).join('; ');
|
||||
results.push({ pr: slot.pr, prefix: prefix.index, files, ok: run.status === 0, detail: run.status === 0 ? 'pass' : run.error ? `${run.error.message}` : tail || `exit ${run.status}` });
|
||||
} finally {
|
||||
git(repoRoot, ['worktree', 'remove', '--force', dir]);
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
/** One `STACK_TESTS:` line per slot. */
|
||||
export function stackTestLine(r: StackTestResult): string {
|
||||
if (!r.files.length) return `STACK_TESTS: #${r.pr} T${r.prefix} no new test files`;
|
||||
return `STACK_TESTS: #${r.pr} T${r.prefix} ${r.files.length} new test file(s) ${r.ok ? 'pass' : 'FAIL'} (${r.files.join(', ')})${r.ok || r.detail === 'pass' ? '' : `: ${r.detail}`}`;
|
||||
}
|
||||
@@ -10,12 +10,15 @@ import { EXIT, formatResult } from './headless-artifacts';
|
||||
import { RESULT_CODES } from './result-codes';
|
||||
import type { QueueContext } from './ship-queue';
|
||||
import { buildTrain, pruneTrains, trainStatus, type TrainOut } from './ship-queue-train';
|
||||
import { trainReady } from './ship-queue-train-ready';
|
||||
import { trainReady, verifyScreen } from './ship-queue-train-ready';
|
||||
import { trainBisect, trainRebuild, trainVerdict } from './ship-queue-train-verdict';
|
||||
import { verifyTrain } from './ship-queue-train-verify';
|
||||
import { trainWatch } from './ship-queue-train-watch';
|
||||
import { verifyPostMerge } from './ship-queue-post-merge';
|
||||
import { trainForget } from './ship-queue-train-forget';
|
||||
import { redactCause } from './ship-queue-train-record';
|
||||
|
||||
export const TRAIN_HELP = ` train build --prs <n>@<sha>[,...] [--order <n>,...] [--urgent | --repair] [--dry-run] [--free-timeout <s>]
|
||||
export const TRAIN_HELP = ` train build --prs <n>@<sha>[,...] [--order <n>,...] [--urgent | --repair] [--dry-run [--stack-tests [--stack-tests-command "<cmd>"]]] [--free-timeout <s>]
|
||||
--order: explicit slot order, every PR in --prs once (default: screened
|
||||
first, then READY order)
|
||||
train build --train <nonce> resume a train (push, free tests, dispatch) from its record
|
||||
@@ -27,12 +30,27 @@ export const TRAIN_HELP = ` train build --prs <n>@<sha>[,...] [--order <n>,...]
|
||||
train bisect --train <nonce> --spend-cap <usd> [--dry-run]
|
||||
red cases on every prefix T0..Tk, resumable and capped
|
||||
train rebuild --train <nonce> [--infra] the one rebuild an all-green bisect earns (or the one INFRA re-dispatch)
|
||||
train prune [--train <nonce>] [--dry-run]
|
||||
delete train/<nonce> branches of merged, superseded or dropped trains
|
||||
ready --pr <n> --screen [--broad] | --no-screen [--dry-run]
|
||||
(train policy) READY plus the screen choice, recorded as a receipt
|
||||
train watch --train <nonce> [--timeout <s>]
|
||||
resumable follow: runs the step the record asks for (build, status), waits for the
|
||||
pending run, reads the verdict, prints NEXT; rerun it after a restart (no daemon)
|
||||
train forget --train <nonce> | --receipt <pr>@<sha> [--dry-run]
|
||||
remove a settled train record (attempts, bisect) or a READY receipt from
|
||||
refs/notes/gstack-trains under an exact lease, then delete the local files
|
||||
train prune [--train <nonce>] [--dry-run] [--stale]
|
||||
delete train/<nonce> branches of merged, superseded or dropped trains; --stale also
|
||||
settles merging or green trains whose every PR is merged or closed on GitHub, and
|
||||
deletes screen/<pr>-* branches of merged or closed PRs
|
||||
ready --pr <n> --screen [--broad] | --no-screen [--dry-run] [--dispatch-ci <wf.yml[,…]> [--timeout <s>]]
|
||||
(train policy) READY plus the screen choice, recorded as a receipt; a head whose
|
||||
screen run is RED is refused (QUEUE_SCREEN_RED). --dispatch-ci runs the named
|
||||
workflows on the branch first (a conflicting PR has no pull_request CI), waits for
|
||||
them and clears the matching requires-remote pregate lanes with the run URLs
|
||||
verify --pr <n> --train <nonce> [--no-fetch]
|
||||
(train policy) the merge check against a green train record
|
||||
verify --pr <n> --screen (train policy) the PR head's screen verdict read from its run: SCREEN_VERDICT and
|
||||
FLAKY lines, recorded in the READY receipt
|
||||
verify --post-merge <sha> [--timeout <s>] after a merge: wait for every check run at the merged commit to settle;
|
||||
POST_MERGE: green|red|pending plus one CHECK: line per workflow (resumable)
|
||||
Merge trains need a "train" block in the base revision's ship policy (docs/ship-policy.md#run-a-merge-train).`;
|
||||
|
||||
function flag(args: string[], name: string): string | undefined {
|
||||
@@ -80,14 +98,28 @@ export async function trainMain(ctx: QueueContext, args: string[]): Promise<numb
|
||||
if (Number.isNaN(freeTimeoutSec)) return usage('--free-timeout needs seconds');
|
||||
if (rest.includes('--urgent') && rest.includes('--repair')) return usage('--urgent and --repair are separate one-PR trains; pass one');
|
||||
if (rest.includes('--order') && (nonce || !flag(rest, '--order'))) return usage('--order <n>,... orders a new train from --prs; a resumed train keeps its recorded order');
|
||||
const stackCommand = flag(rest, '--stack-tests-command');
|
||||
if ((rest.includes('--stack-tests') || stackCommand !== undefined) && !dryRun) return usage('--stack-tests runs each PR\'s new test files on its integrated prefix; it needs --dry-run');
|
||||
const stackWords = stackCommand?.split(/\s+/).filter(Boolean);
|
||||
if (stackCommand !== undefined && !stackWords?.length) return usage('--stack-tests-command needs a command');
|
||||
return guarded('train build', () => buildTrain(ctx, {
|
||||
pins: flag(rest, '--prs'), order: flag(rest, '--order'), resume: nonce, dryRun, urgent: rest.includes('--urgent'), repair: rest.includes('--repair'), freeTimeoutSec,
|
||||
pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000,
|
||||
pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, stackTests: stackWords?.length ? stackWords : rest.includes('--stack-tests') ? true : undefined,
|
||||
}));
|
||||
}
|
||||
if (sub === 'status') return guarded('train status', () => trainStatus(ctx, { train: nonce }));
|
||||
if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce }));
|
||||
if (sub === 'prune') return guarded('train prune', () => pruneTrains(ctx, { dryRun, train: nonce, stale: rest.includes('--stale') }));
|
||||
if (sub === 'forget') {
|
||||
const receipt = flag(rest, '--receipt');
|
||||
if (!nonce && !receipt) return usage('train forget needs --train <nonce> or --receipt <pr>@<sha>');
|
||||
return guarded('train forget', () => trainForget(ctx, { train: nonce, receipt, dryRun }));
|
||||
}
|
||||
if (!nonce) return usage(`train ${sub ?? '<subcommand>'} needs --train <nonce>`);
|
||||
if (sub === 'watch') {
|
||||
const timeoutSec = positive(flag(rest, '--timeout'), 7200);
|
||||
if (Number.isNaN(timeoutSec)) return usage('--timeout needs seconds');
|
||||
return guarded('train watch', () => trainWatch(ctx, { train: nonce, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000, freeTimeoutSec: positive(flag(rest, '--free-timeout'), 2700) || 2700 }));
|
||||
}
|
||||
if (sub === 'verdict') return guarded('train verdict', () => trainVerdict(ctx, { train: nonce }));
|
||||
if (sub === 'rebuild') return guarded('train rebuild', () => trainRebuild(ctx, { train: nonce, infra: rest.includes('--infra') }));
|
||||
if (sub === 'bisect') {
|
||||
@@ -103,9 +135,24 @@ export async function readyMain(ctx: QueueContext, args: string[], base: { pr: s
|
||||
const noScreen = args.includes('--no-screen');
|
||||
if (screen && noScreen) return usage('pass one of --screen or --no-screen');
|
||||
if (args.includes('--broad') && !screen) return usage('--broad selects the screen policy; it needs --screen');
|
||||
return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run') }));
|
||||
const dispatchCi = flag(args, '--dispatch-ci');
|
||||
const workflows = dispatchCi === undefined ? [] : dispatchCi.split(',').map(s => s.trim()).filter(Boolean);
|
||||
if (dispatchCi !== undefined && (!workflows.length || workflows.some(w => !/^[\w.-]+\.ya?ml$/.test(w)))) return usage('--dispatch-ci names workflow files under .github/workflows, comma-separated (free-tests.yml,windows-free-tests.yml)');
|
||||
const timeoutSec = positive(flag(args, '--timeout'), 2700);
|
||||
if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds');
|
||||
return guarded('ready', () => trainReady(ctx, { ...base, screen: screen ? 'screen' : noScreen ? 'no-screen' : undefined, broad: args.includes('--broad'), dryRun: args.includes('--dry-run'),
|
||||
dispatchCi: workflows.length ? { workflows, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 } : undefined }));
|
||||
}
|
||||
|
||||
export async function postMergeMain(ctx: QueueContext, args: string[]): Promise<number> {
|
||||
const sha = flag(args, '--post-merge');
|
||||
if (!sha || !/^[0-9a-f]{40}$/.test(sha)) return usage('verify --post-merge needs the merged commit\'s full 40-hex SHA');
|
||||
const timeoutSec = positive(flag(args, '--timeout'), 2700);
|
||||
if (!Number.isFinite(timeoutSec)) return usage('--timeout needs a positive number of seconds');
|
||||
return guarded('verify', () => verifyPostMerge(ctx, { sha, timeoutSec, pollMs: Number(process.env.GSTACK_SHIP_QUEUE_POLL_MS) || 30_000 }));
|
||||
}
|
||||
|
||||
export async function verifyMain(ctx: QueueContext, args: string[], pr: string): Promise<number> {
|
||||
if (args.includes('--screen')) return guarded('verify', () => verifyScreen(ctx, { pr }));
|
||||
return guarded('verify', () => verifyTrain(ctx, { pr, train: flag(args, '--train')!, noFetch: args.includes('--no-fetch') }));
|
||||
}
|
||||
@@ -0,0 +1,52 @@
|
||||
/**
|
||||
* `train forget --train <nonce> | --receipt <pr>@<head> [--dry-run]`: remove a
|
||||
* train record (its attempts and bisect ledger) or a READY receipt from
|
||||
* refs/notes/gstack-trains under an exact lease, then delete the local files
|
||||
* (until now a stale record needed a hand-run
|
||||
* `git notes remove` plus a file hunt on every machine that ran the train).
|
||||
* A rejected lease removes nothing. Other machines that still hold the local
|
||||
* files can push the record back with their next write; the output says so.
|
||||
*/
|
||||
import * as fs from 'node:fs';
|
||||
import type { QueueContext } from './ship-queue';
|
||||
import { outOk, outRefused, trainEnv, type TrainOut } from './ship-queue-train';
|
||||
import { attemptsOf, forgetNotes, loadBisect, loadReady, loadTrain, noteName, readyPath, rebuildFromNotes, trainDir, type AnyRecord } from './ship-queue-train-record';
|
||||
|
||||
const Q = 'gstack-ship-queue';
|
||||
|
||||
export function trainForget(ctx: QueueContext, o: { train?: string; receipt?: string; dryRun: boolean }): TrainOut {
|
||||
const env = trainEnv(ctx);
|
||||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||||
rebuildFromNotes(ctx.repoRoot, env.remote);
|
||||
const records: AnyRecord[] = [];
|
||||
const files: string[] = [];
|
||||
let what = '';
|
||||
if (o.train) {
|
||||
const t = loadTrain(o.train);
|
||||
if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||||
if (t.state === 'merging' || t.state === 'gate-pending' || t.state === 'free-pending') {
|
||||
return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `train ${t.nonce} is ${t.state}; forget settles nothing that is still running or merging (train prune --stale for a train whose PRs merged elsewhere)` }, [], `${Q} train status --train ${t.nonce}`, trainDir(t.nonce));
|
||||
}
|
||||
const bisect = loadBisect(t.nonce);
|
||||
records.push(t, ...attemptsOf(t), ...(bisect ? [bisect] : []));
|
||||
files.push(trainDir(t.nonce));
|
||||
what = `train ${t.nonce} (${t.state}; ${t.attempts.length} attempt(s)${bisect ? ', a bisect ledger' : ''})`;
|
||||
} else {
|
||||
const m = /^#?(\d+)@([0-9a-f]{40,64})$/.exec(o.receipt ?? '');
|
||||
if (!m) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: '--receipt takes <pr>@<full head sha>' }, [], `${Q} train forget --receipt <pr>@<sha>`, 'none');
|
||||
const r = loadReady(env.repo.keys, m[1]!, m[2]!);
|
||||
if (!r) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no READY receipt for #${m[1]} at ${m[2]!.slice(0, 12)} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||||
records.push(r);
|
||||
files.push(readyPath(r.repo_key, r.pr, r.head));
|
||||
what = `READY receipt #${r.pr} at ${r.head.slice(0, 12)} (screen ${r.screen.choice}${r.screen.run ? `, run ${r.screen.run}` : ''})`;
|
||||
}
|
||||
const names = records.map(noteName);
|
||||
const lines = [`FORGET: ${what}`, ...names.map(n => ` note ${n}`), ...files.map(f => ` local ${f}`)];
|
||||
if (o.dryRun) return outOk([...lines, 'DRY-RUN: nothing removed'], `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`}`, 'none');
|
||||
const f = forgetNotes(ctx.repoRoot, names, env.remote);
|
||||
if (!f.pushed) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the notes ref was not updated (${f.detail}); nothing deleted locally` }, lines, `${Q} train forget ${o.train ? `--train ${o.train}` : `--receipt ${o.receipt}`} (the lease is re-read each run)`, 'none');
|
||||
for (const file of files) fs.rmSync(file, { recursive: true, force: true });
|
||||
lines.push(`FORGOTTEN: ${f.removed.length} note(s) removed${f.missing.length ? `, ${f.missing.length} already absent` : ''}; ${f.detail}; ${files.length} local path(s) deleted`,
|
||||
'note: a machine that still holds these files locally can push the record back with its next write; run train forget there too, or clear its $GSTACK_STATE_ROOT/ship-queue copy');
|
||||
return outOk(lines, `${Q} train status`, 'none');
|
||||
}
|
||||
@@ -57,14 +57,21 @@ export function asRun(summary: RunSummary): { id: number; url: string } {
|
||||
return { id: summary.databaseId, url: summary.url };
|
||||
}
|
||||
|
||||
/** One JSON artifact of a run (the named file, else the first .json file it holds); null when it is missing or unreadable. */
|
||||
/**
|
||||
* One JSON artifact of a run: the named file wherever the artifact keeps it
|
||||
* (a multi-path upload keeps its directories, so report-verdict-a1 holds
|
||||
* paid-report/collector-outcomes.json), else the first .json file at its
|
||||
* root; null when it is missing or unreadable.
|
||||
*/
|
||||
export function downloadJson(ctx: QueueContext, slug: string, run: number, name: string, scratch: string, file?: string): unknown {
|
||||
const dir = path.join(scratch, `${run}-${name}`);
|
||||
fs.rmSync(dir, { recursive: true, force: true });
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
try {
|
||||
if (!ctx.gh(['run', 'download', String(run), '--repo', slug, '-n', name, '-D', dir]).ok) return null;
|
||||
const pick = file ?? fs.readdirSync(dir).find(f => f.endsWith('.json'));
|
||||
const pick = file
|
||||
? (fs.readdirSync(dir, { recursive: true }) as string[]).filter(f => path.basename(f) === file).sort((a, b) => a.length - b.length)[0]
|
||||
: fs.readdirSync(dir).find(f => f.endsWith('.json'));
|
||||
return pick && fs.existsSync(path.join(dir, pick)) ? JSON.parse(fs.readFileSync(path.join(dir, pick), 'utf8')) : null;
|
||||
} catch { return null; } finally { fs.rmSync(dir, { recursive: true, force: true }); }
|
||||
}
|
||||
|
||||
+138
-27
@@ -2,13 +2,21 @@
|
||||
* `ready --pr <n> --screen | --no-screen` under a ship policy with a `train`
|
||||
* block (docs/ship-policy.md#run-a-merge-train; plan GSTA-49 T1). The
|
||||
* READY block is today's (lib/ship-queue.ts ready()); this adds the screen
|
||||
* selection and Garry's default recommendation (screen when the screen
|
||||
* selection and the default recommendation (screen when the screen
|
||||
* selects a skill's own cases), requires an explicit choice
|
||||
* (READY_SCREEN_CHOICE_REQUIRED), dispatches a chosen screen through
|
||||
* eval-bg's nonce with its purpose and policy in the key, and records the
|
||||
* choice as a READY receipt (local state and refs/notes/gstack-trains) that
|
||||
* `train build` requires. The screen's dollar price is left to the T0 cost
|
||||
* replay (lib/gate-cost.ts) and printed as pending until it lands.
|
||||
*
|
||||
* The screen's verdict comes from its run (lib/ship-queue-screen.ts): a head
|
||||
* whose recorded screen is RED is refused (QUEUE_SCREEN_RED) under either
|
||||
* choice, and a red run is never re-dispatched at its merge SHA. The screen
|
||||
* is planned on GitHub's refs/pull/<n>/merge when that merges the head, else
|
||||
* on a scratch merge of the head onto origin/<base> built with the train's
|
||||
* merge rules and pushed as screen/<pr>-<head12> (#3465); only a real
|
||||
* conflict is QUEUE_SCREEN_STALE_REF.
|
||||
*/
|
||||
import { randomBytes } from 'node:crypto';
|
||||
import { DISPATCH_INPUTS, awaitNonceRun, dedupeKey, dispatchWorkflow, findDuplicateRun, listDispatchRuns, makeNonce, type WorkflowDispatch } from '../scripts/eval-bg';
|
||||
@@ -16,13 +24,19 @@ import { EXIT } from './headless-artifacts';
|
||||
import { ready, type QueueContext } from './ship-queue';
|
||||
import { classifyTrainHead } from './version-gate-train';
|
||||
import { git } from './ship-queue-gate';
|
||||
import { execFor, hollowScreen, repoSlug } from './ship-queue-train-github';
|
||||
import { loadReady, nowIso, persist, readyPath, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record';
|
||||
import { outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train';
|
||||
import { dispatchCi } from './ship-queue-dispatch-ci';
|
||||
import { recordVerdict, screenVerdict, verdictLines, type ScreenOutcome } from './ship-queue-screen';
|
||||
import { execFor, repoSlug } from './ship-queue-train-github';
|
||||
import { loadReady, nowIso, persist, readyPath, rebuildFromNotes, type ReadyReceipt, type ScreenChoice } from './ship-queue-train-record';
|
||||
import { fetchBase, mergeCommit, outOk, outPending, outRefused, trainEnv, type TrainEnv, type TrainOut } from './ship-queue-train';
|
||||
|
||||
const Q = 'gstack-ship-queue';
|
||||
|
||||
export interface ReadyOptions { pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean }
|
||||
export interface ReadyOptions {
|
||||
pr: string; pregate?: string; gateExit?: number; gateLog?: string; screen?: 'screen' | 'no-screen'; broad: boolean; dryRun: boolean;
|
||||
/** `--dispatch-ci`: run these workflows on the PR branch, wait, and clear the matching requires-remote pregate lanes before reading the head. */
|
||||
dispatchCi?: { workflows: string[]; timeoutSec: number; pollMs: number };
|
||||
}
|
||||
export interface ScreenSelection { lines: string[]; recommend: 'screen' | 'no-screen' | null }
|
||||
|
||||
/** Skill directories a diff touches: top-level directories that hold a SKILL.md.tmpl at the head. */
|
||||
@@ -70,8 +84,46 @@ async function screenSelection(env: TrainEnv, head: string, baseSha: string): Pr
|
||||
return { ...s, lines: [...s.lines, price] };
|
||||
}
|
||||
|
||||
/** The scratch branch a screen's merge commit is pushed to when GitHub's merge ref lags the head. */
|
||||
export const screenBranch = (pr: string, head: string) => `screen/${pr}-${head.slice(0, 12)}`;
|
||||
|
||||
/**
|
||||
* The merge commit the screen tests: GitHub's refs/pull/<n>/merge when it
|
||||
* merges the PR head, else a scratch merge of the head onto origin/<base>
|
||||
* (the train's merge rules) pushed as screen/<pr>-<head12>, so evals.yml can
|
||||
* check it out and read its parents. A conflict beyond the CHANGELOG and the
|
||||
* stamp files is the only QUEUE_SCREEN_STALE_REF left.
|
||||
*/
|
||||
function screenMerge(env: TrainEnv, o: ReadyOptions, head: string, baseSha: string): { mergeSha: string; branch?: string; line: string } | TrainOut {
|
||||
const { ctx } = env;
|
||||
const ls = git(ctx.repoRoot, ['ls-remote', env.remote, `refs/pull/${o.pr}/merge`]);
|
||||
if (!ls.ok) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `could not list ${env.remote}'s refs: ${(ls.err || ls.out).trim().replace(/\s+/g, ' ').slice(0, 200) || 'git ls-remote failed'}` }, [], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
const github = ls.out.split(/\s/)[0] ?? '';
|
||||
let lag = `${env.remote} has no refs/pull/${o.pr}/merge`;
|
||||
if (/^[0-9a-f]{40,64}$/.test(github)) {
|
||||
const fetched = git(ctx.repoRoot, ['fetch', '-q', env.remote, `refs/pull/${o.pr}/merge`]);
|
||||
const parent = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^2`]).out : '';
|
||||
const onto = fetched.ok ? git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${github}^1`]).out : '';
|
||||
if (parent === head && onto === baseSha) return { mergeSha: github, line: `SCREEN_MERGE: GitHub's refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges the head ${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)}` };
|
||||
lag = !fetched.ok ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} could not be fetched`
|
||||
: parent !== head ? `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges ${parent ? parent.slice(0, 12) : 'no PR head'}, not ${head.slice(0, 12)}`
|
||||
: `refs/pull/${o.pr}/merge ${github.slice(0, 12)} merges onto ${onto ? onto.slice(0, 12) : 'no base'}, not ${ctx.base}@${baseSha.slice(0, 12)}`;
|
||||
}
|
||||
const m = mergeCommit(ctx.repoRoot, baseSha, head, env.policy.policy, `screen #${o.pr}: merge ${head.slice(0, 12)} onto ${ctx.base} ${baseSha.slice(0, 12)}`);
|
||||
if ('conflicts' in m) {
|
||||
return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `#${o.pr} conflicts with ${ctx.base}@${baseSha.slice(0, 12)} beyond the CHANGELOG and the stamp files (${m.conflicts.join(', ')}); GitHub's merge ref lags too (${lag})` }, [],
|
||||
`resolve the conflict with ${ctx.base}, push, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
}
|
||||
const branch = screenBranch(o.pr, head);
|
||||
const line = `SCREEN_MERGE: scratch merge ${m.commit.slice(0, 12)} of #${o.pr}@${head.slice(0, 12)} onto ${ctx.base}@${baseSha.slice(0, 12)} (${lag})`;
|
||||
if (o.dryRun) return { mergeSha: m.commit, branch, line: `${line}; would push ${branch} (dry run)` };
|
||||
const p = git(ctx.repoRoot, ['push', '-q', '-f', env.remote, `${m.commit}:refs/heads/${branch}`]);
|
||||
if (!p.ok) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `could not push the screen's scratch merge as ${branch}: ${p.err.trim().replace(/\s+/g, ' ').slice(0, 200)}` }, [line], `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
return { mergeSha: m.commit, branch, line: `${line}; pushed ${branch}` };
|
||||
}
|
||||
|
||||
/** Dispatch (or adopt) the PR's screen on the base branch: inputs name the PR, its merge SHA and the screen policy. */
|
||||
async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; line: string } | TrainOut> {
|
||||
async function dispatchScreen(env: TrainEnv, o: ReadyOptions, slug: string | null, mergeSha: string, baseSha: string): Promise<{ choice: ScreenChoice; lines: string[] } | TrainOut> {
|
||||
const policy = o.broad ? 'broad' : 'narrow';
|
||||
const scope = { purpose: 'screen' as const, policy: policy as 'narrow' | 'broad' };
|
||||
const nonce = makeNonce('pr', mergeSha, baseSha, randomBytes(3).toString('hex'), scope);
|
||||
@@ -80,23 +132,45 @@ async function dispatchScreen(env: TrainEnv, o: ReadyOptions, mergeSha: string,
|
||||
[DISPATCH_INPUTS.baseSha]: baseSha, [DISPATCH_INPUTS.screenPr]: o.pr, [DISPATCH_INPUTS.screenMergeSha]: mergeSha, [DISPATCH_INPUTS.screenPolicy]: policy,
|
||||
} };
|
||||
const inputs = Object.entries(dispatch.inputs).map(([k, v]) => `${k}=${v}`).join(' ');
|
||||
if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)` };
|
||||
const slug = repoSlug(env.ctx);
|
||||
if (o.dryRun) return { choice: { choice: 'screen', policy, nonce, run: null, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} --ref ${env.ctx.base} ${inputs} (dry run: not dispatched)`] };
|
||||
if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, [], `${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
const exec = execFor(env.ctx);
|
||||
const existing = findDuplicateRun(listDispatchRuns(dispatch.workflow, exec, slug) ?? [], dedupeKey('pr', mergeSha, baseSha, scope));
|
||||
const plan = existing ? hollowScreen(env.ctx, slug, existing.databaseId) : null;
|
||||
const hollow = plan?.kind === 'hollow' ? plan.reason : null;
|
||||
if (existing && !hollow) return { choice: { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)` };
|
||||
const prior: ScreenOutcome | null = existing ? screenVerdict(env.ctx, slug, existing.databaseId) : null;
|
||||
if (existing && prior) {
|
||||
const choice: ScreenChoice = { choice: 'screen', policy, nonce: /eval-bg (\S+?)\]/.exec(existing.displayTitle)?.[1] ?? nonce, run: existing.databaseId, merge_sha: mergeSha };
|
||||
recordVerdict(choice, prior);
|
||||
const lines = verdictLines(o.pr, existing.databaseId, prior);
|
||||
if (prior.kind === 'red') return outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at merge ${mergeSha.slice(0, 12)} (run ${existing.databaseId}) is RED on ${prior.red.join(', ') || 'its gate binding'}; a red screen is not re-dispatched at the same merge SHA` },
|
||||
lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
if (prior.kind !== 'not-run' && prior.kind !== 'incomplete') {
|
||||
return { choice, lines: [...lines, `SCREEN_DISPATCH: following run ${existing.databaseId} (same merge SHA, base and policy; no duplicate)`] };
|
||||
}
|
||||
}
|
||||
const notFollowing = existing && prior ? ` (not following ${prior.detail}: a screen that ${prior.kind === 'not-run' ? 'planned nothing' : 'bound no verdict'} is not a screen)` : '';
|
||||
const found = dispatchWorkflow(dispatch, env.ctx.base, exec, slug) ? await awaitNonceRun(dispatch.workflow, nonce, exec, slug) : null;
|
||||
if (!found) return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `no ${dispatch.workflow} run named ${nonce} appeared after the screen dispatch` }, [], `${Q} ready --pr ${o.pr} --screen${o.broad ? ' --broad' : ''} (adopts the run by its key once it appears)`, 'none');
|
||||
return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, line: `SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${hollow ? ` (not following ${hollow}: a screen that planned nothing is not a screen)` : ''}` };
|
||||
return { choice: { choice: 'screen', policy, nonce, run: found.databaseId, merge_sha: mergeSha }, lines: [`SCREEN_DISPATCH: ${dispatch.workflow} run ${found.databaseId} ${found.url} nonce ${nonce}${notFollowing}`] };
|
||||
}
|
||||
|
||||
const screenLine = (c: ScreenChoice) => `screen: ${c.choice}${c.policy ? ` policy=${c.policy} nonce=${c.nonce} run=${c.run ?? '-'}${c.verdict ? ` verdict=${c.verdict.outcome}` : ''}` : ''}`;
|
||||
|
||||
export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise<TrainOut> {
|
||||
const fetchErr = fetchBase(ctx);
|
||||
const env = trainEnv(ctx);
|
||||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||||
if (fetchErr) return outRefused({ code: 'POLICY_SOURCE_UNAVAILABLE', message: fetchErr }, [], 'none', 'none');
|
||||
const ciLines: string[] = [];
|
||||
if (o.dispatchCi && !o.dryRun) {
|
||||
const d = await dispatchCi(ctx, { pr: o.pr, pregate: o.pregate, ...o.dispatchCi });
|
||||
ciLines.push(...d.lines);
|
||||
if ('failure' in d) {
|
||||
const again = `${Q} ready --pr ${o.pr} ${o.screen === 'screen' ? '--screen' : '--no-screen'} --dispatch-ci ${o.dispatchCi.workflows.join(',')} --timeout ${o.dispatchCi.timeoutSec}`;
|
||||
return (d.failure.code === 'QUEUE_TRAIN_PENDING' ? outPending : outRefused)(d.failure, ciLines, d.failure.code === 'QUEUE_TRAIN_PENDING' ? again : `read the run, repair, push, run /ship, then ${again}`, 'none');
|
||||
}
|
||||
}
|
||||
const base = ready(ctx, o);
|
||||
base.lines.unshift(...ciLines);
|
||||
if (!base.ok) {
|
||||
const out = outRefused(base.failure!, base.lines, `fix the refusal, then ${Q} ready --pr ${o.pr} --screen|--no-screen`, base.run);
|
||||
return ['QUEUE_VERIFY_MISMATCH', 'QUEUE_STAMP_NOT_STAMP_ONLY'].includes(base.failure!.code) ? { ...out, exit: EXIT.fail } : out;
|
||||
@@ -107,6 +181,18 @@ export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise<Tr
|
||||
if (kind.kind === 'invalid') return outRefused({ code: 'QUEUE_NOT_READY', message: `the Version Gate refuses ${head.slice(0, 12)}: ${kind.reason}` }, base.lines, `fix the head, push, then ${Q} ready --pr ${o.pr} --screen|--no-screen`, 'none');
|
||||
const sel = await screenSelection(env, head, baseSha);
|
||||
const lines = [...base.lines, ...sel.lines];
|
||||
if (!loadReady(env.repo.keys, o.pr, head)) rebuildFromNotes(ctx.repoRoot, env.remote);
|
||||
const prior = loadReady(env.repo.keys, o.pr, head);
|
||||
const slug = repoSlug(ctx);
|
||||
if (prior?.screen.run && slug) {
|
||||
const read = screenVerdict(ctx, slug, prior.screen.run);
|
||||
lines.push(...verdictLines(o.pr, prior.screen.run, read));
|
||||
if (recordVerdict(prior.screen, read) && !o.dryRun) { prior.updated_at = nowIso(); persist(ctx.repoRoot, [prior], { remote: env.remote }); }
|
||||
if (read.kind === 'red') {
|
||||
return outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen recorded for ${head.slice(0, 12)} (run ${prior.screen.run}) is RED on ${read.red.join(', ') || 'its gate binding'}; this head is not READY under any screen choice` },
|
||||
lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, readyPath(env.repo.key, o.pr, head));
|
||||
}
|
||||
}
|
||||
const rec = sel.recommend ?? 'no-screen';
|
||||
if (!o.screen) {
|
||||
return outRefused({ code: 'READY_SCREEN_CHOICE_REQUIRED', message: `the base policy runs merge trains; choose --screen or --no-screen (recommended: --${rec})` }, lines,
|
||||
@@ -114,26 +200,51 @@ export async function trainReady(ctx: QueueContext, o: ReadyOptions): Promise<Tr
|
||||
}
|
||||
let choice: ScreenChoice = { choice: 'no-screen', policy: null, nonce: null, run: null, merge_sha: null };
|
||||
if (o.screen === 'screen') {
|
||||
const ls = git(ctx.repoRoot, ['ls-remote', env.remote, `refs/pull/${o.pr}/merge`]);
|
||||
if (!ls.ok) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `could not list ${env.remote}'s refs: ${(ls.err || ls.out).trim().replace(/\s+/g, ' ').slice(0, 200) || 'git ls-remote failed'}` }, lines, `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
const mergeSha = ls.out.split(/\s/)[0] ?? '';
|
||||
const again = `resolve the conflict with ${ctx.base} (or wait for GitHub to recompute refs/pull/${o.pr}/merge), push, then ${Q} ready --pr ${o.pr} --screen; a screen dispatched on a stale merge ref plans no cases`;
|
||||
if (!/^[0-9a-f]{40,64}$/.test(mergeSha)) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `${env.remote} has no refs/pull/${o.pr}/merge (the PR conflicts with ${ctx.base}, or GitHub has not computed it yet)` }, lines, again, 'none');
|
||||
const fetched = git(ctx.repoRoot, ['fetch', '-q', env.remote, `refs/pull/${o.pr}/merge`]);
|
||||
if (!fetched.ok) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `could not fetch refs/pull/${o.pr}/merge from ${env.remote}: ${(fetched.err || fetched.out).trim().replace(/\s+/g, ' ').slice(0, 200) || 'git fetch failed'}` }, lines, `check the remote, then ${Q} ready --pr ${o.pr} --screen`, 'none');
|
||||
const mergeParent = git(ctx.repoRoot, ['rev-parse', '--verify', '--quiet', `${mergeSha}^2`]).out;
|
||||
if (mergeParent !== head) return outRefused({ code: 'QUEUE_SCREEN_STALE_REF', message: `${env.remote}'s refs/pull/${o.pr}/merge (${mergeSha.slice(0, 12)}) merges ${mergeParent ? mergeParent.slice(0, 12) : 'no PR head'}, not the PR head ${head.slice(0, 12)}: GitHub has not recomputed it, which is what a conflicting PR looks like (CHANGELOG-only conflicts included)` }, lines, again, 'none');
|
||||
const d = await dispatchScreen(env, o, mergeSha, baseSha);
|
||||
const merge = screenMerge(env, o, head, baseSha);
|
||||
if ('exit' in merge) return { ...merge, lines: [...lines, ...merge.lines] };
|
||||
lines.push(merge.line);
|
||||
const d = await dispatchScreen(env, o, slug, merge.mergeSha, baseSha);
|
||||
if ('exit' in d) return { ...d, lines: [...lines, ...d.lines] };
|
||||
choice = d.choice;
|
||||
lines.push(d.line);
|
||||
choice = merge.branch ? { ...d.choice, branch: merge.branch } : d.choice;
|
||||
lines.push(...d.lines);
|
||||
}
|
||||
lines.splice(1, 0, `screen: ${choice.choice}${choice.policy ? ` policy=${choice.policy} nonce=${choice.nonce} run=${choice.run ?? '-'}` : ''}`);
|
||||
lines.splice(1, 0, screenLine(choice));
|
||||
const now = nowIso();
|
||||
const readyAt = loadReady(env.repo.keys, o.pr, head)?.ready_at ?? now;
|
||||
const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: readyAt, screen: choice, updated_at: now };
|
||||
const receipt: ReadyReceipt = { schema: 1, kind: 'ready', repo_key: env.repo.key, pr: o.pr, head, base_sha: baseSha, ready_at: prior?.ready_at ?? now, screen: choice, updated_at: now };
|
||||
if (o.dryRun) return outOk([...lines, 'DRY-RUN: no READY receipt written, nothing dispatched'], `${Q} ready --pr ${o.pr} --${o.screen}${o.broad ? ' --broad' : ''}`, 'none');
|
||||
const p = persist(ctx.repoRoot, [receipt], { remote: env.remote });
|
||||
lines.push(`READY_RECEIPT: ${readyPath(env.repo.key, o.pr, head)} (${p.detail})`);
|
||||
return outOk(lines, `send the READY block; the coordinator builds the train: ${Q} train build --prs ${o.pr}@${head},...`, readyPath(env.repo.key, o.pr, head));
|
||||
}
|
||||
|
||||
/**
|
||||
* `verify --pr <n> --screen`: the screen verdict of the PR's pushed head from
|
||||
* its run, recorded in the READY receipt; never a lane's prose.
|
||||
*/
|
||||
export function verifyScreen(ctx: QueueContext, o: { pr: string }): TrainOut {
|
||||
const env = trainEnv(ctx);
|
||||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||||
const view = ctx.gh(['pr', 'view', o.pr, '--json', 'headRefOid']);
|
||||
let head = '';
|
||||
try { head = view.ok ? String(JSON.parse(view.out).headRefOid ?? '') : ''; } catch {}
|
||||
if (!/^[0-9a-f]{40,64}$/.test(head)) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `cannot read PR #${o.pr}'s head through gh (${(view.err ?? view.out).trim().slice(0, 120) || 'no headRefOid'})` }, [], `${Q} verify --pr ${o.pr} --screen`, 'none');
|
||||
if (!loadReady(env.repo.keys, o.pr, head)) rebuildFromNotes(ctx.repoRoot, env.remote);
|
||||
const receipt = loadReady(env.repo.keys, o.pr, head);
|
||||
const run = receipt ? readyPath(receipt.repo_key, o.pr, head) : 'none';
|
||||
if (!receipt) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${o.pr} at ${head.slice(0, 12)} (local state or refs/notes/gstack-trains)` }, [], `the lane runs ${Q} ready --pr ${o.pr} --screen|--no-screen`, 'none');
|
||||
const lines = [`READY: #${o.pr} ${head} ready_at=${receipt.ready_at}`, screenLine(receipt.screen)];
|
||||
if (receipt.screen.choice !== 'screen' || !receipt.screen.run) return outOk([...lines, `SCREEN_VERDICT: #${o.pr} none (${receipt.screen.choice === 'screen' ? 'no run recorded' : 'READY --no-screen'})`], `${Q} train build --prs ${o.pr}@${head},...`, run);
|
||||
const slug = repoSlug(ctx);
|
||||
if (!slug) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: 'gh repo view could not name the repository' }, lines, `${Q} verify --pr ${o.pr} --screen`, run);
|
||||
const read = screenVerdict(ctx, slug, receipt.screen.run);
|
||||
lines.push(...verdictLines(o.pr, receipt.screen.run, read));
|
||||
if (recordVerdict(receipt.screen, read)) { receipt.updated_at = nowIso(); lines.push(`READY_RECEIPT: ${run} (${persist(ctx.repoRoot, [receipt], { remote: env.remote }).detail}; verdict recorded)`); }
|
||||
const again = `${Q} verify --pr ${o.pr} --screen`;
|
||||
switch (read.kind) {
|
||||
case 'green': return outOk(lines, `${Q} train build --prs ${o.pr}@${head},...`, run);
|
||||
case 'pending': return outPending({ code: 'QUEUE_TRAIN_PENDING', message: `the screen run ${receipt.screen.run} has not finished (${read.detail})` }, lines, again, run);
|
||||
case 'red': return { ...outRefused({ code: 'QUEUE_SCREEN_RED', message: `the screen of #${o.pr} at ${head.slice(0, 12)} (run ${receipt.screen.run}) is RED on ${read.red.join(', ') || 'its gate binding'}` }, lines, `repair, push a new head, run /ship, then ${Q} ready --pr ${o.pr} --screen`, run), exit: EXIT.fail };
|
||||
case 'not-run': return outRefused({ code: 'QUEUE_SCREEN_NOT_RUN', message: `the screen of #${o.pr} (run ${receipt.screen.run}) concluded success with no slice: ${read.detail}; it is not a screen, and this head is not screened` }, lines, `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)`, run);
|
||||
default: return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `the screen of #${o.pr} (run ${receipt.screen.run}) has no verdict to read: ${read.detail}` }, lines, read.kind === 'incomplete' ? `${Q} ready --pr ${o.pr} --screen (dispatches a fresh screen)` : again, run);
|
||||
}
|
||||
}
|
||||
@@ -20,11 +20,18 @@ const NOTES_REMOTE_REF = 'refs/notes/gstack-trains-origin';
|
||||
|
||||
export type TrainState =
|
||||
| 'built' | 'pushed' | 'free-pending' | 'free-red' | 'free-green' | 'dispatch-unknown' | 'dispatch-absent' | 'gate-pending'
|
||||
| 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded';
|
||||
/** States after which a train never dispatches again; prune deletes their branches. */
|
||||
export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded'];
|
||||
| 'green' | 'red' | 'incomplete' | 'attributed' | 'red-on-main' | 'unattributed' | 'merging' | 'merged' | 'superseded' | 'stale';
|
||||
/** States after which a train never dispatches again; prune deletes their branches (`stale`: every PR merged or closed outside the train, `train prune --stale`). */
|
||||
export const TERMINAL_STATES: readonly TrainState[] = ['free-red', 'attributed', 'red-on-main', 'merged', 'superseded', 'stale'];
|
||||
|
||||
export interface ScreenChoice { choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null }
|
||||
/** What the screen run itself said (lib/ship-queue-screen.ts), recorded when it was read; absent while the run is pending or unread. */
|
||||
export interface ScreenVerdictRecord { outcome: 'GREEN' | 'RED' | 'NOT_RUN' | 'INCOMPLETE'; run: number; red: string[]; flaky: string[]; detail: string; read_at: string }
|
||||
export interface ScreenChoice {
|
||||
choice: 'screen' | 'no-screen'; policy: 'narrow' | 'broad' | null; nonce: string | null; run: number | null; merge_sha: string | null;
|
||||
/** The scratch branch `ready --screen` pushed when GitHub's merge ref lagged the head (docs/ship-policy.md#run-a-merge-train). */
|
||||
branch?: string;
|
||||
verdict?: ScreenVerdictRecord;
|
||||
}
|
||||
export interface TrainSlot { pr: string; head: string; ready_at: string; screen: ScreenChoice }
|
||||
export interface TrainPrefix { index: number; commit: string; tree: string; pr: string | null }
|
||||
export interface FreeRun { run: number | null; url: string | null; head_sha: string | null; conclusion: string | null; verdict: 'green' | 'red' | 'missing' | 'stale' | 'cancelled' | 'pending' }
|
||||
@@ -154,7 +161,7 @@ export function recordPath(r: AnyRecord): string {
|
||||
return readyPath(r.repo_key, r.pr, r.head);
|
||||
}
|
||||
|
||||
function noteName(r: AnyRecord): string {
|
||||
export function noteName(r: AnyRecord): string {
|
||||
if (r.kind === 'train') return `gstack-trains:train:${r.nonce}`;
|
||||
if (r.kind === 'train-attempt') return `gstack-trains:attempt:${r.train}:${r.id}`;
|
||||
if (r.kind === 'train-bisect') return `gstack-trains:bisect:${r.train}`;
|
||||
@@ -227,6 +234,30 @@ function parseNote(text: string): AnyRecord | null {
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove records from the notes ref (`train forget`): each named note is
|
||||
* removed locally, then the ref is pushed under an exact lease on the remote
|
||||
* tip this call fetched, so a concurrent writer's push is never overwritten
|
||||
* (a rejected lease returns `pushed: false` and nothing is deleted locally).
|
||||
* Names are note names (noteName); a name with no note is reported missing.
|
||||
*/
|
||||
export function forgetNotes(repoRoot: string, names: string[], remote = 'origin'): { removed: string[]; missing: string[]; pushed: boolean; lease: string | null; detail: string } {
|
||||
const f = fetchNotes(repoRoot, remote);
|
||||
if (!f.ok) return { removed: [], missing: [], pushed: false, lease: null, detail: f.detail };
|
||||
const lease = git(repoRoot, ['rev-parse', '--verify', '--quiet', NOTES_REMOTE_REF]).out || null;
|
||||
const listed = git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'list']).out.split('\n').map(l => l.split(' ')[1]).filter(Boolean);
|
||||
const removed: string[] = [];
|
||||
const missing: string[] = [];
|
||||
for (const name of names) {
|
||||
const blob = git(repoRoot, ['hash-object', '--stdin'], undefined, name).out;
|
||||
if (!listed.includes(blob)) { missing.push(name); continue; }
|
||||
if (git(repoRoot, ['notes', `--ref=${NOTES_REF}`, 'remove', blob], notesEnv()).ok) removed.push(name); else missing.push(name);
|
||||
}
|
||||
if (!removed.length) return { removed, missing, pushed: false, lease, detail: 'no note to remove' };
|
||||
const p = git(repoRoot, ['push', '-q', lease ? `--force-with-lease=${NOTES_REF}:${lease}` : `--force-with-lease=${NOTES_REF}`, remote, `${NOTES_REF}:${NOTES_REF}`]);
|
||||
return { removed, missing, pushed: p.ok, lease, detail: p.ok ? `notes pushed to ${remote} ${NOTES_REF} (lease ${lease?.slice(0, 12) ?? 'new ref'})` : `git push ${NOTES_REF} rejected: ${p.err.replace(/\s+/g, ' ').trim().slice(0, 160)}` };
|
||||
}
|
||||
|
||||
/** Rebuild local records from the notes ref: a note whose record is missing locally, or newer than the local copy, is written. */
|
||||
export function rebuildFromNotes(repoRoot: string, remote = 'origin'): { restored: number; read: number; detail: string } {
|
||||
const f = fetchNotes(repoRoot, remote);
|
||||
|
||||
@@ -0,0 +1,68 @@
|
||||
/**
|
||||
* `train watch --train <nonce> [--timeout <s>]`: the controller's one
|
||||
* resumable follow command. Machines restart and kill
|
||||
* background `gh run watch` chains, so watch keeps no process state: it
|
||||
* reads the train record, runs the step the state asks for (train build
|
||||
* --train to push, run the free tests and dispatch the gate; train status to
|
||||
* settle an unconfirmed dispatch), waits for the pending run to finish by
|
||||
* polling gh, then reads the verdict, and prints the last step's NEXT. Run it
|
||||
* again after an interruption and it continues from the record. It never
|
||||
* dispatches anything the record would not dispatch on its own.
|
||||
*/
|
||||
import type { QueueContext } from './ship-queue';
|
||||
import { buildTrain, findTrain, nextFor, outOk, outPending, outRefused, trainEnv, trainStatus, type TrainOut } from './ship-queue-train';
|
||||
import { repoSlug, runView } from './ship-queue-train-github';
|
||||
import { attemptsOf, trainFile, type TrainRecord } from './ship-queue-train-record';
|
||||
import { trainVerdict } from './ship-queue-train-verdict';
|
||||
|
||||
const Q = 'gstack-ship-queue';
|
||||
const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms));
|
||||
|
||||
export interface WatchOptions { train: string; timeoutSec: number; pollMs: number; freeTimeoutSec: number }
|
||||
|
||||
/** Wait for the train's current gate run to complete; null when it did, else why the wait stopped. */
|
||||
async function awaitGate(ctx: QueueContext, t: TrainRecord, deadline: number, pollMs: number): Promise<string | null> {
|
||||
const a = attemptsOf(t).filter(x => x.purpose !== 'bisect').at(-1);
|
||||
if (!a?.run) return `no gate run is recorded for ${t.nonce}`;
|
||||
const slug = repoSlug(ctx);
|
||||
if (!slug) return 'gh repo view could not name the repository';
|
||||
for (;;) {
|
||||
const view = runView(ctx, slug, a.run.id);
|
||||
if (view?.status === 'completed') return null;
|
||||
if (Date.now() >= deadline) return `run ${a.run.id} is still ${view?.status ?? 'unreadable'} at the watch timeout`;
|
||||
await sleep(pollMs);
|
||||
}
|
||||
}
|
||||
|
||||
export async function trainWatch(ctx: QueueContext, o: WatchOptions): Promise<TrainOut> {
|
||||
const env = trainEnv(ctx);
|
||||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||||
let t = findTrain(env, o.train);
|
||||
if (!t) return outRefused({ code: 'QUEUE_PROOF_MISSING', message: `no train record ${o.train} here or in refs/notes/gstack-trains` }, [], `${Q} train status`, 'none');
|
||||
const run = trainFile(t.nonce);
|
||||
const deadline = Date.now() + o.timeoutSec * 1000;
|
||||
const lines: string[] = [`WATCH: train ${t.nonce} state=${t.state} (resumable; rerun this command after an interruption)`];
|
||||
const again = `${Q} train watch --train ${t.nonce} --timeout ${o.timeoutSec}`;
|
||||
const step = (label: string, out: TrainOut): TrainOut => ({ ...out, lines: [...lines, `WATCH: ${label}`, ...out.lines] });
|
||||
const reload = () => { t = findTrain(env, o.train) ?? t; };
|
||||
if (t.state === 'dispatch-unknown') {
|
||||
const s = trainStatus(ctx, { train: t.nonce });
|
||||
lines.push(`WATCH: train status settled the unconfirmed dispatch`, ...s.lines.filter(l => l.startsWith('ATTEMPT:')));
|
||||
reload();
|
||||
}
|
||||
if (['built', 'pushed', 'free-pending', 'free-green', 'dispatch-absent'].includes(t.state)) {
|
||||
const remaining = Math.max(1, Math.floor((deadline - Date.now()) / 1000));
|
||||
const b = await buildTrain(ctx, { resume: t.nonce, dryRun: false, urgent: false, repair: false, freeTimeoutSec: Math.min(o.freeTimeoutSec, remaining), pollMs: o.pollMs });
|
||||
reload();
|
||||
if (b.status !== 'complete') return step(`train build --train ${t.nonce} stopped (${b.failure?.code ?? b.status})`, b.failure?.code === 'QUEUE_TRAIN_PENDING' ? { ...b, next: again } : b);
|
||||
lines.push(`WATCH: train build --train ${t.nonce}: free tests green, gate dispatched`);
|
||||
}
|
||||
if (t.state === 'gate-pending') {
|
||||
const why = await awaitGate(ctx, t, deadline, o.pollMs);
|
||||
if (why) return outPending({ code: 'QUEUE_TRAIN_PENDING', message: why }, lines, again, run);
|
||||
lines.push('WATCH: gate run completed; reading the verdict');
|
||||
return step(`train verdict --train ${t.nonce}`, trainVerdict(ctx, { train: t.nonce }));
|
||||
}
|
||||
if (t.state === 'dispatch-unknown') return outPending({ code: 'QUEUE_TRAIN_DISPATCH_UNKNOWN', message: `the gate dispatch for ${t.nonce} is still unconfirmed` }, lines, again, run);
|
||||
return outOk([...lines, `WATCH: nothing to wait for in state ${t.state}`], nextFor(t), run);
|
||||
}
|
||||
+97
-20
@@ -23,7 +23,9 @@ import type { LoadedPolicy, ShipPolicy, TrainPolicy } from './ship-policy';
|
||||
import { loadBasePolicy, type QueueContext } from './ship-queue';
|
||||
import { bindReceipt } from './ship-receipt';
|
||||
import { git, sha256, withIndex, type QueueFailure } from './ship-queue-gate';
|
||||
import { asRun, execFor, hollowScreen, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github';
|
||||
import { recordVerdict, screenVerdict, verdictLines } from './ship-queue-screen';
|
||||
import { stackTestLine, stackTests } from './ship-queue-stack-tests';
|
||||
import { asRun, execFor, newestBranchRun, prInfo, remoteBranchSha, repoSlug, runView, checkRun } from './ship-queue-train-github';
|
||||
import {
|
||||
TERMINAL_STATES, activeFreeze, attemptsOf, listTrains, loadBisect, loadReady, loadTrain, nowIso, persist, rebuildFromNotes, repoIdentity, trainFile,
|
||||
type AttemptPurpose, type AttemptRecord, type TrainRecord, type TrainSlot,
|
||||
@@ -51,7 +53,7 @@ export function trainEnv(ctx: QueueContext): TrainEnv | QueueFailure {
|
||||
export function fetchBase(ctx: QueueContext): string | null {
|
||||
const remote = ctx.remote ?? 'origin';
|
||||
const f = git(ctx.repoRoot, ['fetch', '--no-tags', '-q', remote, `+refs/heads/${ctx.base}:refs/remotes/${remote}/${ctx.base}`]);
|
||||
return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.slice(0, 160)}`;
|
||||
return f.ok ? null : `git fetch ${remote} ${ctx.base} failed: ${f.err.replace(/\s+/g, ' ').trim().slice(0, 160)}`;
|
||||
}
|
||||
|
||||
/** Make the train's prefix commits local (another machine built them): fetch train/<nonce>. */
|
||||
@@ -195,17 +197,28 @@ export function mergeStep(repoRoot: string, pred: string, head: string, policy:
|
||||
const TRAIN_IDENTITY = { GIT_AUTHOR_NAME: 'gstack-ship-queue', GIT_AUTHOR_EMAIL: 'ship-queue@gstack.local', GIT_COMMITTER_NAME: 'gstack-ship-queue',
|
||||
GIT_COMMITTER_EMAIL: 'ship-queue@gstack.local', GIT_AUTHOR_DATE: '2000-01-01T00:00:00Z', GIT_COMMITTER_DATE: '2000-01-01T00:00:00Z' };
|
||||
|
||||
/**
|
||||
* One train merge as a commit with a fixed identity and date: parents are the
|
||||
* predecessor (first, as GitHub's merge ref) and the PR head. Prefix commits
|
||||
* and the scratch merge a screen is planned on (`ready --screen` when
|
||||
* GitHub's refs/pull/<n>/merge lags the head) are both built here.
|
||||
*/
|
||||
export function mergeCommit(repoRoot: string, pred: string, head: string, policy: ShipPolicy, message: string): { commit: string; tree: string } | { conflicts: string[] } {
|
||||
const m = mergeStep(repoRoot, pred, head, policy);
|
||||
if ('conflicts' in m) return m;
|
||||
const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', head, '-m', message], { ...process.env, ...TRAIN_IDENTITY });
|
||||
return c.ok ? { commit: c.out, tree: m.tree } : { conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`] };
|
||||
}
|
||||
|
||||
/** Build T1..Tk on T0 = base; a conflict stops at the PR that conflicts. */
|
||||
export function buildChain(repoRoot: string, baseSha: string, slots: TrainSlot[], policy: ShipPolicy):
|
||||
{ prefixes: TrainRecord['prefixes'] } | { stoppedAt: string; conflicts: string[]; merged: string[] } {
|
||||
const prefixes: TrainRecord['prefixes'] = [{ index: 0, commit: baseSha, tree: git(repoRoot, ['rev-parse', `${baseSha}^{tree}`]).out, pr: null }];
|
||||
for (const [i, slot] of slots.entries()) {
|
||||
const pred = prefixes[i]!.commit;
|
||||
const m = mergeStep(repoRoot, pred, slot.head, policy);
|
||||
const m = mergeCommit(repoRoot, pred, slot.head, policy, `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`);
|
||||
if ('conflicts' in m) return { stoppedAt: slot.pr, conflicts: m.conflicts, merged: slots.slice(0, i).map(s => s.pr) };
|
||||
const c = git(repoRoot, ['commit-tree', m.tree, '-p', pred, '-p', slot.head, '-m', `train T${i + 1}: merge #${slot.pr} at ${slot.head.slice(0, 12)}`], { ...process.env, ...TRAIN_IDENTITY });
|
||||
if (!c.ok) return { stoppedAt: slot.pr, conflicts: [`(commit-tree failed: ${c.err.slice(0, 120)})`], merged: slots.slice(0, i).map(s => s.pr) };
|
||||
prefixes.push({ index: i + 1, commit: c.out, tree: m.tree, pr: slot.pr });
|
||||
prefixes.push({ index: i + 1, commit: m.commit, tree: m.tree, pr: slot.pr });
|
||||
}
|
||||
return { prefixes };
|
||||
}
|
||||
@@ -266,7 +279,11 @@ export function gateDispatch(env: TrainEnv, train: TrainRecord, nonce: string):
|
||||
return { workflow: env.train.gate_workflow, inputs: { [DISPATCH_INPUTS.evalsAll]: 'true', [DISPATCH_INPUTS.expectedSha]: train.head, [DISPATCH_INPUTS.nonce]: nonce } };
|
||||
}
|
||||
|
||||
export interface BuildOptions { pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number }
|
||||
export interface BuildOptions {
|
||||
pins?: string; order?: string; resume?: string; dryRun: boolean; urgent: boolean; repair: boolean; freeTimeoutSec: number; pollMs: number;
|
||||
/** `--dry-run --stack-tests`: run each PR's new test files on its integrated prefix (lib/ship-queue-stack-tests.ts); the command's words when `--stack-tests-command` names one. */
|
||||
stackTests?: string[] | true;
|
||||
}
|
||||
|
||||
/** `--order <pr>,<pr>,...`: every pinned PR exactly once (a leading # allowed). */
|
||||
export function parseOrder(spec: string | undefined, pins: Array<{ pr: string }>): string[] | undefined | { error: string } {
|
||||
@@ -302,9 +319,16 @@ function admitSlots(env: TrainEnv, pins: Array<{ pr: string; sha: string }>): {
|
||||
if (!ready) return outRefused({ code: 'QUEUE_NOT_READY', message: `no READY receipt for #${p.pr} at ${info.headRefOid.slice(0, 12)} (local state or ${'refs/notes/gstack-trains'})` }, [], `the lane runs ${Q} ready --pr ${p.pr} --screen|--no-screen`, 'none');
|
||||
let screen = ready.screen;
|
||||
if (screen.choice === 'screen' && screen.run && slug === undefined) slug = repoSlug(ctx);
|
||||
const plan = screen.choice === 'screen' && screen.run ? (slug ? hollowScreen(ctx, slug, screen.run) : { kind: 'unknown' as const, reason: 'the repository slug is unreadable' }) : null;
|
||||
if (plan?.kind === 'hollow') { lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; }
|
||||
else if (plan?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${plan.reason}; counted as screened, unverified`);
|
||||
const read = screen.choice === 'screen' && screen.run ? (slug ? screenVerdict(ctx, slug, screen.run) : { kind: 'unknown' as const, detail: 'the repository slug is unreadable' }) : null;
|
||||
if (read && screen.run && recordVerdict(ready.screen, read)) { ready.updated_at = nowIso(); persist(ctx.repoRoot, [ready], { remote: env.remote }); }
|
||||
if (read?.kind === 'red') {
|
||||
return outRefused({ code: 'QUEUE_SCREEN_RED', message: `#${p.pr}'s screen run ${screen.run} is RED on ${read.red.join(', ') || 'its gate binding'} (${read.detail}); a red head is not seated` },
|
||||
[...lines, ...verdictLines(p.pr, screen.run!, read)], without ? `${Q} train build --prs ${without}` : 'none', 'none');
|
||||
}
|
||||
if (read?.kind === 'not-run' || read?.kind === 'incomplete') { lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as not-run, the slot rides unscreened`); screen = { ...screen, choice: 'no-screen' }; }
|
||||
else if (read?.kind === 'unknown') lines.push(`SCREEN: #${p.pr} ${read.detail}; counted as screened, unverified`);
|
||||
else if (read?.kind === 'green') lines.push(`SCREEN: #${p.pr} run ${screen.run} GREEN${read.flaky.length ? ` (flaky: ${read.flaky.join(', ')})` : ''}`);
|
||||
else if (read?.kind === 'pending') lines.push(`SCREEN: #${p.pr} run ${screen.run} still running; counted as screened`);
|
||||
slots.push({ pr: p.pr, head: info.headRefOid, ready_at: ready.ready_at, screen });
|
||||
}
|
||||
return { slots, lines };
|
||||
@@ -460,8 +484,18 @@ export async function buildTrain(ctx: QueueContext, o: BuildOptions): Promise<Tr
|
||||
}
|
||||
if (!('kind' in t)) return t;
|
||||
if (o.dryRun) {
|
||||
return outOk([...lines, ...describe(t), `ESTIMATE: one ${env.train.gate_workflow} full gate (evals_all) on T${t.prefixes.length - 1} plus ${env.train.free_workflow}; dollar figure pending the T0 cost replay (lib/gate-cost.ts)`,
|
||||
'DRY-RUN: nothing written, pushed or dispatched'], `${Q} train build ${prsArgs(t.slots, t.explicit_order)}`, 'none');
|
||||
const dry = [...lines, ...describe(t), `ESTIMATE: one ${env.train.gate_workflow} full gate (evals_all) on T${t.prefixes.length - 1} plus ${env.train.free_workflow}; dollar figure pending the T0 cost replay (lib/gate-cost.ts)`];
|
||||
if (o.stackTests) {
|
||||
const results = stackTests(env.ctx.repoRoot, t.prefixes[0]!.commit, t.slots, t.prefixes, o.stackTests === true ? undefined : o.stackTests);
|
||||
dry.push(...results.map(stackTestLine));
|
||||
const red = results.filter(r => !r.ok);
|
||||
if (red.length) {
|
||||
const again = `${Q} train build ${prsArgs(t.slots, t.explicit_order)} --dry-run --stack-tests`;
|
||||
return outFailed({ code: 'QUEUE_TRAIN_STACK_RED', message: red.map(r => `#${r.pr}'s new tests fail on the integrated prefix T${r.prefix} (${r.files.join(', ')})`).join('; ') + '; they pass on the PR head alone only if nothing before it in the train changes what they read' },
|
||||
[...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `fix ${red.map(r => `#${r.pr}'s tests against T${r.prefix}`).join(' and ')}, or reorder with --order, then ${again}`, 'none');
|
||||
}
|
||||
}
|
||||
return outOk([...dry, 'DRY-RUN: nothing written, pushed or dispatched'], `${Q} train build ${prsArgs(t.slots, t.explicit_order)}`, 'none');
|
||||
}
|
||||
if (t.state === 'built' && !t.attempts.length && !loadTrain(t.nonce)) {
|
||||
effect(t, 'train.json written');
|
||||
@@ -528,21 +562,64 @@ export function trainStatus(ctx: QueueContext, o: { train?: string }): TrainOut
|
||||
return outOk(lines, next, single ? trainFile(single.nonce) : 'none');
|
||||
}
|
||||
|
||||
/** `train prune`: delete train/<nonce> on the remote for every finished (merged, superseded, dropped) train. */
|
||||
export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string }): TrainOut {
|
||||
/** A PR's state on GitHub for the stale check: MERGED, CLOSED, OPEN, or unreadable. */
|
||||
function prState(ctx: QueueContext, pr: string): string {
|
||||
const info = prInfo(ctx, pr);
|
||||
return 'error' in info ? 'unreadable' : info.state.toUpperCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* `--stale`: a merging or green train whose every PR is MERGED or CLOSED on
|
||||
* GitHub (merged elsewhere, closed, or merged outside the train) never
|
||||
* finishes on its own; it becomes `stale` (terminal) and is pruned, with
|
||||
* each PR's state in the log. Any OPEN or unreadable PR keeps the train.
|
||||
*/
|
||||
function staleCheck(ctx: QueueContext, t: TrainRecord): { stale: true; detail: string } | { stale: false; detail: string } {
|
||||
const unmerged = t.slots.filter(s => !t.merged.some(m => m.pr === s.pr));
|
||||
const states = unmerged.map(s => `#${s.pr} ${prState(ctx, s.pr)}`);
|
||||
const open = states.filter(s => !/ (MERGED|CLOSED)$/.test(s));
|
||||
const detail = `${t.merged.map(m => `#${m.pr} merged by the train`).concat(states).join(', ')}`;
|
||||
return open.length ? { stale: false, detail: `${open.join(', ')} still ${open.length === 1 ? 'needs' : 'need'} the train` } : { stale: true, detail };
|
||||
}
|
||||
|
||||
/** `train prune`: delete train/<nonce> on the remote for every finished (merged, superseded, dropped, stale) train; `--stale` first settles merging and green trains whose PRs all merged or closed elsewhere, and the screen/<pr>-* branches of merged or closed PRs. */
|
||||
export function pruneTrains(ctx: QueueContext, o: { dryRun: boolean; train?: string; stale?: boolean }): TrainOut {
|
||||
const env = trainEnv(ctx);
|
||||
if ('code' in env) return outRefused(env, [], 'none', 'none');
|
||||
rebuildFromNotes(ctx.repoRoot, env.remote);
|
||||
const lines: string[] = [];
|
||||
const rgit = (args: string[]) => git(ctx.repoRoot, args);
|
||||
const remove = (branch: string, label: string, after?: () => void) => {
|
||||
if (!remoteBranchSha(rgit, env.remote, branch)) { lines.push(`GONE: ${branch} (${label})`); return; }
|
||||
if (o.dryRun) { lines.push(`WOULD-DELETE: ${branch} (${label})`); return; }
|
||||
const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${branch}`]);
|
||||
lines.push(d.ok ? `DELETED: ${branch} (${label})` : `FAILED: ${branch}: ${d.err.slice(0, 120)}`);
|
||||
if (d.ok) after?.();
|
||||
};
|
||||
for (const t of listTrains(env.repo.keys).filter(x => !o.train || x.nonce === o.train)) {
|
||||
if (o.stale && (t.state === 'merging' || t.state === 'green')) {
|
||||
const s = staleCheck(ctx, t);
|
||||
if (s.stale) {
|
||||
lines.push(`STALE: ${t.nonce} (${t.state}): ${s.detail}${o.dryRun ? '; would mark stale' : ''}`);
|
||||
if (!o.dryRun) { t.state = 'stale'; effect(t, `marked stale by train prune --stale: ${s.detail}`); save(env, t); }
|
||||
remove(t.branch, o.dryRun ? 'stale' : t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); });
|
||||
continue;
|
||||
}
|
||||
lines.push(`KEEP: ${t.branch} (${t.state}; ${s.detail})`);
|
||||
continue;
|
||||
}
|
||||
if (!TERMINAL_STATES.includes(t.state)) { lines.push(`KEEP: ${t.branch} (${t.state})`); continue; }
|
||||
if (!remoteBranchSha(rgit, env.remote, t.branch)) { lines.push(`GONE: ${t.branch} (${t.state})`); continue; }
|
||||
if (o.dryRun) { lines.push(`WOULD-DELETE: ${t.branch} (${t.state})`); continue; }
|
||||
const d = git(ctx.repoRoot, ['push', '-q', env.remote, '--delete', `refs/heads/${t.branch}`]);
|
||||
lines.push(d.ok ? `DELETED: ${t.branch} (${t.state})` : `FAILED: ${t.branch}: ${d.err.slice(0, 120)}`);
|
||||
if (d.ok) { effect(t, `branch deleted: ${t.branch}`); save(env, t); }
|
||||
remove(t.branch, t.state, () => { effect(t, `branch deleted: ${t.branch}`); save(env, t); });
|
||||
}
|
||||
if (o.stale && !o.train) {
|
||||
const refs = git(ctx.repoRoot, ['ls-remote', env.remote, 'refs/heads/screen/*']);
|
||||
for (const ref of refs.ok ? refs.out.split('\n') : []) {
|
||||
const m = /refs\/heads\/(screen\/(\d+)-[0-9a-f]{12})$/.exec(ref.split(/\s/)[1] ?? '');
|
||||
if (!m) continue;
|
||||
const state = prState(ctx, m[2]!);
|
||||
if (state === 'MERGED' || state === 'CLOSED') remove(m[1]!, `#${m[2]} ${state}`); else lines.push(`KEEP: ${m[1]} (#${m[2]} ${state})`);
|
||||
}
|
||||
}
|
||||
if (!lines.length) lines.push('PRUNE: no trains');
|
||||
return outOk(lines, o.dryRun ? `${Q} train prune` : 'none', 'none');
|
||||
return outOk(lines, o.dryRun ? `${Q} train prune${o.stale ? ' --stale' : ''}` : 'none', 'none');
|
||||
}
|
||||
+6
-5
@@ -138,20 +138,21 @@ export function latestRunPerName<T extends { name?: unknown; id?: unknown; statu
|
||||
}
|
||||
|
||||
/** CI at one exact SHA: the latest check run per name (as GitHub's rollup counts) and every commit-status context; unreadable or incomplete is unknown, never green. */
|
||||
function ciAtHead(ctx: QueueContext, sha: string): { line: string; green: boolean; none: boolean } {
|
||||
export function ciAtHead(ctx: QueueContext, sha: string): { line: string; green: boolean; none: boolean; checks: Array<{ name: string; status: string; conclusion: string | null }> } {
|
||||
const at = sha.slice(0, 12);
|
||||
const runs = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/check-runs`, 'check_runs');
|
||||
if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false };
|
||||
if ('error' in runs) return { line: `unknown at ${at} (${runs.error})`, green: false, none: false, checks: [] };
|
||||
const statuses = readAllPages(ctx, `repos/{owner}/{repo}/commits/${sha}/status`, 'statuses');
|
||||
if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false };
|
||||
if ('error' in statuses) return { line: `unknown at ${at} (${statuses.error})`, green: false, none: false, checks: [] };
|
||||
const checks = latestRunPerName(runs.items).map(c => ({ name: String(c.name ?? ''), status: String(c.status ?? ''), conclusion: c.conclusion === undefined || c.conclusion === null ? null : String(c.conclusion) }));
|
||||
const verdicts = [
|
||||
...latestRunPerName(runs.items).map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'),
|
||||
...checks.map(c => c.status !== 'completed' ? 'pending' : c.conclusion === 'success' ? 'passed' : c.conclusion === 'skipped' || c.conclusion === 'neutral' ? 'skipped' : 'failed'),
|
||||
...statuses.items.map(s => s.state === 'success' ? 'passed' : s.state === 'pending' ? 'pending' : 'failed'),
|
||||
];
|
||||
const count = (v: string) => verdicts.filter(x => x === v).length;
|
||||
const [passed, failed, pending, skipped] = [count('passed'), count('failed'), count('pending'), count('skipped')];
|
||||
const green = verdicts.length > 0 && pending === 0 && failed === 0;
|
||||
return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0 };
|
||||
return { line: `${green ? 'green' : pending ? 'pending' : verdicts.length ? 'red' : 'none'} at ${at} (passed=${passed} failed=${failed} pending=${pending} skipped=${skipped})`, green, none: verdicts.length === 0, checks };
|
||||
}
|
||||
|
||||
/**
|
||||
|
||||
@@ -287,6 +287,10 @@ export const KNOWN_WINDOWS_INCOMPATIBLE: Array<{ file: string; reason: string }>
|
||||
file: 'test/ship-queue-train.test.ts',
|
||||
reason: 'drives bin/gstack-ship-queue against the stateful #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts; on windows-latest spawnSync(\'gh\') resolves the runner\'s gh.exe instead of the shim (windows-free-shard run 39152702056)',
|
||||
},
|
||||
{
|
||||
file: 'test/ship-queue-train-ops.test.ts',
|
||||
reason: 'drives bin/gstack-ship-queue (screen verdicts, scratch-merge screens, prune --stale, watch, forget, verify --post-merge, ready --dispatch-ci, --stack-tests) against the same #!/bin/sh gh shim in test/helpers/ship-queue-train-fixture.ts',
|
||||
},
|
||||
{
|
||||
file: 'test/ship-policy-compat.test.ts',
|
||||
reason: 'spawns bin/gstack-ship-policy, bin/gstack-restamp and bin/gstack-tree-receipt (shebangs) through test/helpers/restamp-fixture.ts',
|
||||
|
||||
@@ -31,7 +31,14 @@ export interface Rule {
|
||||
plan?: { planned: number } | { manifest: unknown } | null;
|
||||
/** Marks every artifact of runs dispatched under this rule expired in the artifacts listing. */
|
||||
expired?: boolean;
|
||||
/** A screen run's gate binding (gate-binding-a1/gate-binding.json, scripts/gate-reuse.ts) and its report's collector-outcomes.json (report-verdict-a1). */
|
||||
binding?: { verdict: 'GREEN' | 'RED'; reconcile_exit?: number; flaky?: string[] };
|
||||
outcomes?: { red?: string[]; flakyLines?: string[] };
|
||||
/** The run confirmed a verdict reuse (R2a/R2): paid-plan carries a valid gate-reuse.json naming this run and no binding is uploaded. */
|
||||
reuse?: { run: number; flaky?: string[] };
|
||||
check?: string | null;
|
||||
/** The check-run name a run dispatched under this rule attaches to its head (a push or workflow_dispatch run's job), with the run's status and conclusion. */
|
||||
ci?: string;
|
||||
bisect?: { prefixes: string[]; patterns: Record<string, string>; costUsd?: number };
|
||||
}
|
||||
export interface ShimData {
|
||||
@@ -163,6 +170,13 @@ function shimSource(): string {
|
||||
" if (rule.record && !inputs.prefix_shas) d.artifacts[id][d.artifact] = { schema: 1, kind: 'train-gate', nonce: rule.record.nonce || inputs.nonce, head_sha: head, tree: rev(head + '^{tree}'),",
|
||||
" verdict: rule.record.verdict || 'GREEN', census: rule.record.census, panels: rule.record.panels };",
|
||||
" if (rule.check && !inputs.prefix_shas) (d.checks[head] = d.checks[head] || []).push({ name: 'gstack/full-gate', status: 'completed', conclusion: rule.check });",
|
||||
" if (rule.ci) (d.ci[head] = d.ci[head] || []).push({ name: rule.ci, status: run.status, conclusion: run.conclusion, id: id, completed_at: '2026-10-26T00:00:00Z' });",
|
||||
" if (rule.binding) d.artifacts[id]['gate-binding-a1'] = { schema: 1, repository: d.repo, pr: Number(inputs.screen_pr) || 0, run_id: id, run_attempt: 1, head_sha: rev(head + '^2') || head, base_sha: rev(head + '^1') || head,",
|
||||
" profile: 'pr', reconcile_exit: rule.binding.reconcile_exit === undefined ? (rule.binding.verdict === 'GREEN' ? 0 : 1) : rule.binding.reconcile_exit, verdict: rule.binding.verdict, manual_accepted: 0, selection_sha256: 'a'.repeat(64),",
|
||||
" flaky_cases: rule.binding.flaky || [], purpose: 'screen', tested_sha: head, screen_policy: inputs.screen_policy || 'narrow' };",
|
||||
" if (rule.reuse && d.artifacts[id]['paid-plan']) d.artifacts[id]['paid-plan'].__reuse = { schema: 1, kind: 'main-moved', pr: Number(inputs.screen_pr) || 0, reused_run_id: rule.reuse.run, reused_run_attempt: 1, gated_head: 'b'.repeat(40), current_head: 'c'.repeat(40),",
|
||||
" gated_base: 'd'.repeat(40), base_sha: 'e'.repeat(40), selection_sha256: 'a'.repeat(64), why: 'same own diff on a moved main', gated_merge_base: 'f'.repeat(40), merge_base: '1'.repeat(40), own_diff_sha256: 'b'.repeat(64), reused_screen_policy: 'narrow', reused_flaky: rule.reuse.flaky || [] };",
|
||||
" if (rule.outcomes) d.artifacts[id]['report-verdict-a1'] = { version: 2, panels: (rule.outcomes.red || []).map(c => ({ case: c, kind: 'judge', status: 'FAIL', failsLane: true, redClass: 'VERDICT' })), flakyLines: rule.outcomes.flakyLines || [] };",
|
||||
" if (rule.bisect && inputs.prefix_shas) for (const sha of inputs.prefix_shas.split(',')) {",
|
||||
" const at = rule.bisect.prefixes.indexOf(sha);",
|
||||
" d.artifacts[id][d.artifact + '-bisect-' + sha.slice(0, 12)] = { schema: 1, kind: 'train-bisect-prefix', nonce: inputs.nonce, prefix_sha: sha, checked_out_sha: sha,",
|
||||
@@ -178,8 +192,9 @@ function shimSource(): string {
|
||||
"if (a[0] === 'run' && a[1] === 'view') { const r = d.runs.find(x => String(x.databaseId) === a[2]); if (!r) fail('run not found'); out(r); }",
|
||||
"if (a[0] === 'run' && a[1] === 'download') {",
|
||||
" const art = (d.artifacts[a[2]] || {})[opt('-n')]; if (!art) fail('no artifact matches');",
|
||||
" fs.mkdirSync(opt('-D'), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), opt('-n') === 'paid-plan' ? 'manifest.json' : 'record.json'), JSON.stringify(art));",
|
||||
" if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);",
|
||||
" const fileFor = n => n === 'paid-plan' ? 'manifest.json' : n.startsWith('gate-binding-') ? 'gate-binding.json' : n.startsWith('report-verdict-') ? 'paid-report/collector-outcomes.json' : 'record.json';",
|
||||
" fs.mkdirSync(path.dirname(path.join(opt('-D'), fileFor(opt('-n')))), { recursive: true }); fs.writeFileSync(path.join(opt('-D'), fileFor(opt('-n'))), JSON.stringify(art));",
|
||||
" if (opt('-n') === 'paid-plan') fs.writeFileSync(path.join(opt('-D'), 'gate-reuse.json'), art.__reuse ? JSON.stringify(art.__reuse) : '{\"reuse\":\"confirmed\"}'); save(); process.exit(0);",
|
||||
"}",
|
||||
"if (a[0] === 'api') {",
|
||||
" const arts = /actions\\/runs\\/(\\d+)\\/artifacts(?:\\?name=([^&]+))?/.exec(a[1] || '');",
|
||||
|
||||
@@ -0,0 +1,425 @@
|
||||
/**
|
||||
* Operational gaps seen on live trains, through the real
|
||||
* bin/gstack-ship-queue against the train fixture: the screen verdict is read
|
||||
* from the run (ready refuses a red head, verify --screen prints it, train
|
||||
* build refuses the pin), a lagging GitHub merge ref gets a scratch merge
|
||||
* (#3465), `train prune --stale` settles trains whose PRs merged elsewhere,
|
||||
* and `train watch` resumes from the record after an interruption.
|
||||
*/
|
||||
import { afterEach, describe, expect, test } from 'bun:test';
|
||||
import * as fs from 'node:fs';
|
||||
import * as path from 'node:path';
|
||||
import { EXIT } from '../lib/headless-artifacts';
|
||||
import { cleanup, runBin } from './helpers/restamp-fixture';
|
||||
import {
|
||||
GREEN_GATE, TRAIN_POLICY, dispatches, line, nonceOf, openPr, queue, readyReceipt, repoKey, trainJson, trainFixture, type Rule, type TrainFixture,
|
||||
} from './helpers/ship-queue-train-fixture';
|
||||
|
||||
const fixtures: TrainFixture[] = [];
|
||||
const make = (policy: Record<string, unknown> = TRAIN_POLICY) => { const f = trainFixture(policy); fixtures.push(f); return f; };
|
||||
afterEach(() => { while (fixtures.length) cleanup(fixtures.pop()!.repo); });
|
||||
|
||||
const next = (out: string) => line(out, 'NEXT:');
|
||||
const result = (out: string) => line(out, 'GSTACK_RESULT:');
|
||||
|
||||
function readyFixture() {
|
||||
const fx = make();
|
||||
const head = openPr(fx, '1');
|
||||
fx.write(d => { d.ci[head] = [{ status: 'completed', conclusion: 'success' }]; });
|
||||
const tree = runBin(fx.repo, 'gstack-wtree', []).out.trim();
|
||||
const pregate = path.join(fx.repo.root, 'pregate.json');
|
||||
fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] }));
|
||||
return { fx, head, pregate };
|
||||
}
|
||||
const receiptOf = (fx: TrainFixture, pr: string, head: string) => JSON.parse(fs.readFileSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `${pr}-${head}.json`), 'utf8'));
|
||||
|
||||
const RED_SCREEN: Rule = { status: 'completed', conclusion: 'failure', plan: { planned: 3 }, binding: { verdict: 'RED', flaky: ['c-flaky'] }, outcomes: { red: ['judge-voice-tone'], flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } };
|
||||
const GREEN_SCREEN: Rule = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN', flaky: ['c-flaky'] }, outcomes: { flakyLines: ['c-flaky rule FLAKY: failed in slice 2, passed on its one rerun'] } };
|
||||
|
||||
describe('the screen verdict comes from the run', () => {
|
||||
// Value: protects=a RED screen run (runs 40269992077 and 40275443393 on 2026-10-26 were VERDICT RED while the lane reported green) can never be READY'd, followed or seated;
|
||||
// fails_when=ready or train build trust the lane's choice without reading the run's gate binding; why_new=no test read a screen's verdict; seam=the gh shim's gate-binding and report-verdict artifacts; no_claim=evals.yml's real upload step
|
||||
test('a RED screen refuses ready under either choice and is never re-dispatched; verify --screen prints the red and flaky cases; train build refuses the pin', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
fx.write(d => { d.rules['evals.yml'] = RED_SCREEN; });
|
||||
const first = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect([first.status, line(first.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, expect.stringMatching(/^SCREEN_DISPATCH: evals\.yml run 100 /)]);
|
||||
const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(again.status).toBe(EXIT.refused);
|
||||
expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)');
|
||||
expect(line(again.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun');
|
||||
expect(again.err).toMatch(/the screen recorded for [0-9a-f]{12} \(run 100\) is RED on judge-voice-tone; this head is not READY under any screen choice; fix: .*\(QUEUE_SCREEN_RED\)/);
|
||||
expect(next(again.out)).toBe('NEXT: repair, push a new head, run /ship, then gstack-ship-queue ready --pr 1 --screen');
|
||||
expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'RED', run: 100, red: ['judge-voice-tone'], flaky: ['c-flaky'] });
|
||||
const noScreen = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']);
|
||||
expect([noScreen.status, noScreen.err]).toEqual([EXIT.refused, expect.stringContaining('(QUEUE_SCREEN_RED)')]);
|
||||
expect(dispatches(fx, 'evals.yml').length).toBe(1);
|
||||
const verify = queue(fx, ['verify', '--pr', '1', '--screen']);
|
||||
expect(verify.status).toBe(EXIT.fail);
|
||||
expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 RED red=judge-voice-tone flaky=c-flaky (verdict RED, reconcile exit 1, run concluded failure, flaky 1)');
|
||||
expect(verify.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=pr\.screen\.narrow-\S+ run=100 verdict=RED$/);
|
||||
expect(result(verify.out)).toMatch(/status=refused run=.*1-/);
|
||||
const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']);
|
||||
expect(build.status).toBe(EXIT.refused);
|
||||
expect(build.err).toMatch(/#1's screen run 100 is RED on judge-voice-tone .*a red head is not seated.*\(QUEUE_SCREEN_RED\)/);
|
||||
expect(next(build.out)).toBe('NEXT: none');
|
||||
expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'trains'))).toBe(false);
|
||||
}, 120_000);
|
||||
|
||||
test('a GREEN screen with flaky cases is followed, recorded and printed by verify --screen and train build; a run that bound no verdict is re-dispatched', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; });
|
||||
expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok);
|
||||
const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(again.status).toBe(EXIT.ok);
|
||||
expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 GREEN flaky=c-flaky (verdict GREEN, reconcile exit 0, run concluded success, flaky 1)');
|
||||
expect(line(again.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)');
|
||||
expect(again.out.split('\n')[1]).toMatch(/^screen: screen policy=narrow nonce=\S+ run=100 verdict=GREEN$/);
|
||||
expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'GREEN', run: 100, red: [], flaky: ['c-flaky'] });
|
||||
const verify = queue(fx, ['verify', '--pr', '1', '--screen']);
|
||||
expect([verify.status, verify.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(verify.out, 'FLAKY:')).toBe('FLAKY: c-flaky rule FLAKY: failed in slice 2, passed on its one rerun');
|
||||
expect(next(verify.out)).toBe(`NEXT: gstack-ship-queue train build --prs 1@${head},...`);
|
||||
const build = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']);
|
||||
expect([build.status, line(build.out, 'SCREEN:'), line(build.out, 'ORDER:')]).toEqual([EXIT.ok, 'SCREEN: #1 run 100 GREEN (flaky: c-flaky)', 'ORDER: 1:#1(screened)']);
|
||||
fx.write(d => { delete d.artifacts['100']['gate-binding-a1']; });
|
||||
const unbound = queue(fx, ['verify', '--pr', '1', '--screen']);
|
||||
expect(unbound.status).toBe(EXIT.refused);
|
||||
expect(unbound.err).toMatch(/has no verdict to read: run 100 concluded success and bound no verdict \(no gate-binding artifact.*\(QUEUE_PROOF_MISSING\)/);
|
||||
expect(next(unbound.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)');
|
||||
const redispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(redispatch.status).toBe(EXIT.ok);
|
||||
expect(line(redispatch.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 101 .*\(not following run 100 concluded success and bound no verdict .*: a screen that bound no verdict is not a screen\)$/);
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, reuse: { run: 77, flaky: ['c-reused'] } }; });
|
||||
const reusedDispatch = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--broad', '--screen']);
|
||||
expect(reusedDispatch.status).toBe(EXIT.ok);
|
||||
const reused = queue(fx, ['verify', '--pr', '1', '--screen']);
|
||||
expect([reused.status, line(reused.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #1 run 102 GREEN flaky=c-reused (reused the GREEN verdict of run 77 (main-moved): same own diff on a moved main, flaky 1)']);
|
||||
const noRun = queue(fx, ['verify', '--pr', '2', '--screen']);
|
||||
expect(noRun.err).toMatch(/cannot read PR #2's head through gh.*\(QUEUE_PROOF_MISSING\)/);
|
||||
const unscreened = openPr(fx, '3');
|
||||
readyReceipt(fx, '3', unscreened);
|
||||
const none = queue(fx, ['verify', '--pr', '3', '--screen']);
|
||||
expect([none.status, line(none.out, 'SCREEN_VERDICT:')]).toEqual([EXIT.ok, 'SCREEN_VERDICT: #3 none (READY --no-screen)']);
|
||||
}, 120_000);
|
||||
|
||||
// Value: protects=a screen that concluded success with no slice (18 of 41 PR screens on 2026-10-26, superseded by a push before the plan) is NOT-RUN at verdict, not a screen and not a stale ref;
|
||||
// fails_when=verify --screen calls it GREEN, HOLLOW or QUEUE_SCREEN_STALE_REF, or ready follows it; why_new=the hollow outcome reused the stale-ref code; seam=the gh shim's run with no plan artifact; no_claim=why evals.yml planned nothing
|
||||
test('a screen run that concluded success with no slice is NOT-RUN: verify --screen refuses with QUEUE_SCREEN_NOT_RUN, the receipt records NOT_RUN, and ready --screen dispatches afresh', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; });
|
||||
expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']).status).toBe(EXIT.ok);
|
||||
const verify = queue(fx, ['verify', '--pr', '1', '--screen']);
|
||||
expect(verify.status).toBe(EXIT.refused);
|
||||
expect(line(verify.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))');
|
||||
expect(verify.err).toMatch(/the screen of #1 \(run 100\) concluded success with no slice: run 100 planned no cases .*; it is not a screen, and this head is not screened; fix: .*\(QUEUE_SCREEN_NOT_RUN\)/);
|
||||
expect(next(verify.out)).toBe('NEXT: gstack-ship-queue ready --pr 1 --screen (dispatches a fresh screen)');
|
||||
expect(receiptOf(fx, '1', head).screen.verdict).toMatchObject({ outcome: 'NOT_RUN', run: 100 });
|
||||
const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.ok, 2]);
|
||||
expect(line(again.out, 'SCREEN_VERDICT:')).toBe('SCREEN_VERDICT: #1 run 100 NOT-RUN (run 100 planned no cases (no paid-plan artifact; concluded success))');
|
||||
expect(line(again.out, 'SCREEN_DISPATCH:')).toMatch(/not following run 100 planned no cases .*: a screen that planned nothing is not a screen\)$/);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('#3465: the screen is planned on a scratch merge when GitHub\'s merge ref lags', () => {
|
||||
// Value: protects=a CHANGELOG-only conflict (every main stamp leaves one) no longer blocks --screen: the screen tests the head merged onto current main by the train's rules;
|
||||
// fails_when=ready still refuses on the lagging ref, pushes nothing, or the scratch merge's parents are not (base, head); why_new=the stale-ref test only asserted the refusal; seam=none; no_claim=evals.yml's checkout of the pushed branch
|
||||
test('a lagging or missing merge ref gets a scratch merge pushed as screen/<pr>-<head12>; a real conflict is QUEUE_SCREEN_STALE_REF; prune --stale removes the branch once the PR merged', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
const { repo } = fx;
|
||||
fx.write(d => { d.rules['evals.yml'] = GREEN_SCREEN; });
|
||||
repo.git('checkout', '-q', '-B', 'mainsim', 'origin/main');
|
||||
const cl = fs.readFileSync(path.join(repo.work, 'CHANGELOG.md'), 'utf8');
|
||||
fs.writeFileSync(path.join(repo.work, 'CHANGELOG.md'), cl.replace('# Changelog\n', '# Changelog\n\n## [1.2.4.0] - 2026-10-18\n\n- another PR, stamped\n'));
|
||||
fs.writeFileSync(path.join(repo.work, 'VERSION'), '1.2.4.0\n');
|
||||
repo.git('commit', '-qam', 'v1.2.4.0 stamp');
|
||||
repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main');
|
||||
repo.git('checkout', '-q', 'pr-1');
|
||||
const dry = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen', '--dry-run']);
|
||||
expect([dry.status, dry.err]).toEqual([EXIT.ok, '']);
|
||||
const main = repo.git('rev-parse', 'origin/main');
|
||||
expect(line(dry.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${head.slice(0, 12)} onto main@${main.slice(0, 12)} \\(refs/pull/1/merge [0-9a-f]{12} merges onto [0-9a-f]{12}, not main@${main.slice(0, 12)}\\); would push screen/1-${head.slice(0, 12)} \\(dry run\\)$`));
|
||||
expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe('');
|
||||
const real = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect([real.status, real.err]).toEqual([EXIT.ok, '']);
|
||||
const branch = `screen/1-${head.slice(0, 12)}`;
|
||||
expect(line(real.out, 'SCREEN_MERGE:')).toContain(`; pushed ${branch}`);
|
||||
const scratch = repo.git('ls-remote', 'origin', `refs/heads/${branch}`).split(/\s/)[0]!;
|
||||
expect(scratch).toMatch(/^[0-9a-f]{40}$/);
|
||||
repo.git('fetch', '-q', 'origin', `refs/heads/${branch}`);
|
||||
expect([repo.git('rev-parse', `${scratch}^1`), repo.git('rev-parse', `${scratch}^2`)]).toEqual([repo.git('rev-parse', 'origin/main'), head]);
|
||||
expect(repo.git('show', `${scratch}:VERSION`)).toBe('1.2.4.0');
|
||||
expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- #1 change');
|
||||
expect(repo.git('show', `${scratch}:CHANGELOG.md`)).toContain('- another PR, stamped');
|
||||
const d = dispatches(fx, 'evals.yml');
|
||||
expect(d.length).toBe(1);
|
||||
expect((d[0]!.inputs as any)).toMatchObject({ screen_merge_sha: scratch, expected_sha: scratch, screen_pr: '1' });
|
||||
expect(receiptOf(fx, '1', head).screen).toMatchObject({ choice: 'screen', merge_sha: scratch, branch });
|
||||
repo.git('push', '-q', 'origin', ':refs/pull/1/merge');
|
||||
const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect([missing.status, line(missing.out, 'SCREEN_DISPATCH:')]).toEqual([EXIT.ok, 'SCREEN_DISPATCH: following run 100 (same merge SHA, base and policy; no duplicate)']);
|
||||
expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge)');
|
||||
const kept = queue(fx, ['train', 'prune', '--stale', '--dry-run']);
|
||||
expect(line(kept.out, 'KEEP:')).toBe(`KEEP: ${branch} (#1 OPEN)`);
|
||||
fx.write(data => { (data.prs['1'] as any).state = 'MERGED'; });
|
||||
const pruned = queue(fx, ['train', 'prune', '--stale']);
|
||||
expect(line(pruned.out, 'DELETED:')).toBe(`DELETED: ${branch} (#1 MERGED)`);
|
||||
expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe('');
|
||||
fx.write(data => { (data.prs['1'] as any).state = 'OPEN'; });
|
||||
repo.git('checkout', '-q', '-B', 'mainsim2', 'origin/main');
|
||||
fs.writeFileSync(path.join(repo.work, 'pr-1.txt'), 'main added the same file\n');
|
||||
repo.git('add', 'pr-1.txt');
|
||||
repo.git('commit', '-qm', 'main adds pr-1.txt');
|
||||
repo.git('push', '-q', 'origin', 'HEAD:refs/heads/main');
|
||||
repo.git('checkout', '-q', 'pr-1');
|
||||
const conflict = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(conflict.status).toBe(EXIT.refused);
|
||||
expect(conflict.err).toMatch(/#1 conflicts with main@[0-9a-f]{12} beyond the CHANGELOG and the stamp files \(pr-1\.txt\); GitHub's merge ref lags too \(origin has no refs\/pull\/1\/merge\).*\(QUEUE_SCREEN_STALE_REF\)/);
|
||||
expect(next(conflict.out)).toBe('NEXT: resolve the conflict with main, push, run /ship, then gstack-ship-queue ready --pr 1 --screen');
|
||||
expect(dispatches(fx, 'evals.yml').length).toBe(1);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('train prune --stale and train watch', () => {
|
||||
function greenTrain(fx: TrainFixture, gate: Rule = GREEN_GATE) {
|
||||
const h1 = openPr(fx, '1');
|
||||
const h2 = openPr(fx, '2');
|
||||
readyReceipt(fx, '1', h1);
|
||||
readyReceipt(fx, '2', h2);
|
||||
fx.write(d => { d.rules['evals.yml'] = gate; });
|
||||
const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2}`, '--free-timeout', '1']);
|
||||
return { h1, h2, r, nonce: nonceOf(r.out) };
|
||||
}
|
||||
|
||||
// Value: protects=a merging or green train whose PRs all merged elsewhere (gate.train-23af94b3c5e9 and -6755ab829acf on 2026-10-26) can be pruned without hand-editing refs/notes/gstack-trains;
|
||||
// fails_when=prune skips merging/green trains, prunes one with an OPEN PR, or prunes without --stale; why_new=prune only knew terminal states; seam=the gh shim's PR states; no_claim=GitHub's real PR state
|
||||
test('--stale settles a merging or green train whose every PR is MERGED or CLOSED on GitHub, logging each PR; an OPEN PR keeps it; without --stale nothing changes', () => {
|
||||
const fx = make();
|
||||
const { nonce } = greenTrain(fx);
|
||||
expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok);
|
||||
expect(trainJson(fx, nonce).state).toBe('green');
|
||||
const plain = queue(fx, ['train', 'prune']);
|
||||
expect(line(plain.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green)`);
|
||||
const open = queue(fx, ['train', 'prune', '--stale', '--dry-run']);
|
||||
expect(line(open.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #1 OPEN, #2 OPEN still need the train)`);
|
||||
fx.write(d => { (d.prs['1'] as any).state = 'MERGED'; });
|
||||
const half = queue(fx, ['train', 'prune', '--stale', '--dry-run']);
|
||||
expect(line(half.out, 'KEEP:')).toBe(`KEEP: train/${nonce} (green; #2 OPEN still needs the train)`);
|
||||
fx.write(d => { (d.prs['2'] as any).state = 'CLOSED'; });
|
||||
const dry = queue(fx, ['train', 'prune', '--stale', '--dry-run']);
|
||||
expect(line(dry.out, 'STALE:')).toBe(`STALE: ${nonce} (green): #1 MERGED, #2 CLOSED; would mark stale`);
|
||||
expect(line(dry.out, 'WOULD-DELETE:')).toBe(`WOULD-DELETE: train/${nonce} (stale)`);
|
||||
expect(trainJson(fx, nonce).state).toBe('green');
|
||||
const pruned = queue(fx, ['train', 'prune', '--stale']);
|
||||
expect([pruned.status, line(pruned.out, 'STALE:'), line(pruned.out, 'DELETED:')]).toEqual([EXIT.ok, `STALE: ${nonce} (green): #1 MERGED, #2 CLOSED`, `DELETED: train/${nonce} (stale)`]);
|
||||
const t = trainJson(fx, nonce);
|
||||
expect(t.state).toBe('stale');
|
||||
expect(t.effects.map((e: any) => e.what)).toContain('marked stale by train prune --stale: #1 MERGED, #2 CLOSED');
|
||||
expect(fx.repo.git('ls-remote', 'origin', `refs/heads/train/${nonce}`)).toBe('');
|
||||
const status = queue(fx, ['train', 'status']);
|
||||
expect(status.out).toContain('TRAIN: none live');
|
||||
}, 120_000);
|
||||
|
||||
// Value: protects=a stale train record or READY receipt leaves refs/notes/gstack-trains through one command under an exact lease, with the local files, instead of a hand-run git notes remove on every machine;
|
||||
// fails_when=forget removes the wrong notes, deletes locally when the push failed, or forgets a merging train; why_new=no CLI removed notes (merge-train skill, known gaps); seam=none; no_claim=a concurrent writer's race (the lease is exercised only as a fast-forward)
|
||||
test('train forget removes a settled train (attempts included) or a READY receipt from the notes ref and the state root, leaves other records alone, and refuses a merging train', () => {
|
||||
const fx = make();
|
||||
const { nonce, h1 } = greenTrain(fx);
|
||||
expect(queue(fx, ['train', 'verdict', '--train', nonce]).status).toBe(EXIT.ok);
|
||||
const trainDir = path.join(fx.state, 'ship-queue', 'trains', nonce);
|
||||
expect(fs.existsSync(path.join(trainDir, 'attempts', 'a1.json'))).toBe(true);
|
||||
const dry = queue(fx, ['train', 'forget', '--train', nonce, '--dry-run']);
|
||||
expect([dry.status, line(dry.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: train ${nonce} (green; 1 attempt(s))`]);
|
||||
expect(dry.out).toContain(` note gstack-trains:train:${nonce}`);
|
||||
expect(dry.out).toContain(` note gstack-trains:attempt:${nonce}:a1`);
|
||||
expect(fs.existsSync(trainDir)).toBe(true);
|
||||
const notesBefore = fx.repo.git('ls-remote', 'origin', 'refs/notes/gstack-trains').split(/\s/)[0]!;
|
||||
const forgot = queue(fx, ['train', 'forget', '--train', nonce]);
|
||||
expect([forgot.status, forgot.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(forgot.out, 'FORGOTTEN:')).toMatch(new RegExp(`^FORGOTTEN: 2 note\\(s\\) removed; notes pushed to origin refs/notes/gstack-trains \\(lease ${notesBefore.slice(0, 12)}\\); 1 local path\\(s\\) deleted$`));
|
||||
expect(fs.existsSync(trainDir)).toBe(false);
|
||||
const clean = path.join(fx.repo.root, 'clean-state');
|
||||
const status = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: clean });
|
||||
expect(status.out).toContain('TRAIN: none live');
|
||||
expect(fs.existsSync(path.join(clean, 'ship-queue', 'trains', nonce))).toBe(false);
|
||||
expect(fs.existsSync(path.join(clean, 'ship-queue', 'ready', repoKey(fx), `1-${h1}.json`))).toBe(false);
|
||||
const again = queue(fx, ['train', 'forget', '--train', nonce]);
|
||||
expect(again.err).toMatch(new RegExp(`no train record ${nonce} here or in refs/notes/gstack-trains.*\\(QUEUE_PROOF_MISSING\\)`));
|
||||
const h3 = openPr(fx, '3');
|
||||
fx.write(d => { d.ci[h3] = [{ status: 'completed', conclusion: 'success' }]; });
|
||||
const pregate = path.join(fx.repo.root, 'pregate3.json');
|
||||
fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h3, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] }));
|
||||
expect(queue(fx, ['ready', '--pr', '3', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok);
|
||||
const h4 = openPr(fx, '4');
|
||||
fx.write(d => { d.ci[h4] = [{ status: 'completed', conclusion: 'success' }]; });
|
||||
fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree: runBin(fx.repo, 'gstack-wtree', []).out.trim(), head: h4, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] }));
|
||||
expect(queue(fx, ['ready', '--pr', '4', '--pregate', pregate, '--no-screen']).status).toBe(EXIT.ok);
|
||||
const receipt = queue(fx, ['train', 'forget', '--receipt', `3@${h3}`]);
|
||||
expect([receipt.status, line(receipt.out, 'FORGET:')]).toEqual([EXIT.ok, `FORGET: READY receipt #3 at ${h3.slice(0, 12)} (screen no-screen)`]);
|
||||
expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false);
|
||||
const rebuilt = queue(fx, ['train', 'status'], { GSTACK_STATE_ROOT: path.join(fx.repo.root, 'clean-state-2') });
|
||||
expect(rebuilt.status).toBe(EXIT.ok);
|
||||
expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `4-${h4}.json`))).toBe(true);
|
||||
expect(fs.existsSync(path.join(fx.repo.root, 'clean-state-2', 'ship-queue', 'ready', repoKey(fx), `3-${h3}.json`))).toBe(false);
|
||||
expect(queue(fx, ['train', 'forget', '--receipt', '3@abc']).err).toMatch(/--receipt takes <pr>@<full head sha>.*\(QUEUE_PROOF_MISSING\)/);
|
||||
expect(queue(fx, ['train', 'forget']).status).toBe(EXIT.usage);
|
||||
const { nonce: second } = (() => { readyReceipt(fx, '3', h3); readyReceipt(fx, '4', h4); const r = queue(fx, ['train', 'build', '--prs', `3@${h3},4@${h4}`, '--free-timeout', '1']); return { nonce: nonceOf(r.out) }; })();
|
||||
expect(queue(fx, ['train', 'verdict', '--train', second]).status).toBe(EXIT.ok);
|
||||
const t = JSON.parse(fs.readFileSync(path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'), 'utf8'));
|
||||
t.state = 'merging';
|
||||
fs.writeFileSync(path.join(fx.state, 'ship-queue', 'trains', second, 'train.json'), JSON.stringify(t));
|
||||
const merging = queue(fx, ['train', 'forget', '--train', second]);
|
||||
expect(merging.status).toBe(EXIT.refused);
|
||||
expect(merging.err).toMatch(/is merging; forget settles nothing that is still running or merging .*\(QUEUE_PROOF_MISSING\)/);
|
||||
}, 120_000);
|
||||
|
||||
// Value: protects=a restarted controller re-arms with one command: watch continues from the record (free tests, gate dispatch, the pending run, the verdict) and prints NEXT;
|
||||
// fails_when=watch dispatches twice, blocks past --timeout, or does not reach the verdict after a restart; why_new=no resumable follow existed (hand-run gh run watch chains died with the machine); seam=GSTACK_SHIP_QUEUE_POLL_MS; no_claim=real GitHub run timing
|
||||
test('watch resumes from free-pending through the gate dispatch to the verdict, times out with its own NEXT, and never dispatches twice', () => {
|
||||
const fx = make();
|
||||
fx.write(d => { d.rules['free-tests.yml'] = { status: 'in_progress', conclusion: null }; });
|
||||
const { nonce, r } = greenTrain(fx, { ...GREEN_GATE, status: 'in_progress', conclusion: null });
|
||||
expect(r.status).toBe(EXIT.refused);
|
||||
expect(trainJson(fx, nonce).state).toBe('free-pending');
|
||||
const waiting = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']);
|
||||
expect([waiting.status, result(waiting.out)]).toEqual([EXIT.refused, expect.stringContaining('status=incomplete')]);
|
||||
expect(waiting.out).toContain(`WATCH: train ${nonce} state=free-pending (resumable; rerun this command after an interruption)`);
|
||||
expect(next(waiting.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`);
|
||||
expect(dispatches(fx, 'evals.yml')).toEqual([]);
|
||||
fx.write(d => { for (const run of d.runs) if (run.workflow === 'free-tests.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); });
|
||||
const gate = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']);
|
||||
expect(gate.status).toBe(EXIT.refused);
|
||||
expect(gate.out).toContain(`WATCH: train build --train ${nonce}: free tests green, gate dispatched`);
|
||||
expect(gate.err).toMatch(/is still in_progress at the watch timeout.*\(QUEUE_TRAIN_PENDING\)/);
|
||||
expect(next(gate.out)).toBe(`NEXT: gstack-ship-queue train watch --train ${nonce} --timeout 1`);
|
||||
expect(dispatches(fx, 'evals.yml').length).toBe(1);
|
||||
expect(trainJson(fx, nonce).state).toBe('gate-pending');
|
||||
const again = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '1']);
|
||||
expect([again.status, dispatches(fx, 'evals.yml').length]).toEqual([EXIT.refused, 1]);
|
||||
fx.write(d => { for (const run of d.runs) if (run.workflow === 'evals.yml') Object.assign(run, { status: 'completed', conclusion: 'success' }); });
|
||||
const done = queue(fx, ['train', 'watch', '--train', nonce, '--timeout', '5']);
|
||||
expect([done.status, done.err]).toEqual([EXIT.ok, '']);
|
||||
expect(done.out).toContain('WATCH: gate run completed; reading the verdict');
|
||||
expect(line(done.out, 'VERDICT:')).toMatch(new RegExp(`^VERDICT: GREEN train ${nonce} run 10[0-9] `));
|
||||
expect(next(done.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`);
|
||||
const idle = queue(fx, ['train', 'watch', '--train', nonce]);
|
||||
expect([idle.status, line(idle.out, 'WATCH: nothing')]).toEqual([EXIT.ok, 'WATCH: nothing to wait for in state green']);
|
||||
expect(next(idle.out)).toBe(`NEXT: gstack-ship-queue verify --pr 1 --train ${nonce}`);
|
||||
expect(queue(fx, ['train', 'watch']).status).toBe(EXIT.usage);
|
||||
expect(queue(fx, ['train', 'watch', '--train', 'no-such-train']).err).toMatch(/no train record no-such-train.*\(QUEUE_PROOF_MISSING\)/);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('verify --post-merge', () => {
|
||||
// Value: protects=after a train merges, the coordinator watches main's push workflows at the merged sha until they settle instead of a background gh chain a restart kills (GBrain practice 8);
|
||||
// fails_when=a pending check is called settled, a red check is called green, or the command needs process state; why_new=no post-merge watch existed; seam=the gh shim's check runs at a sha; no_claim=which workflows main runs
|
||||
test('waits for every check run at the sha to complete, prints each, is green only when none failed, red with the failed names, and resumable after a timeout', () => {
|
||||
const fx = make();
|
||||
const sha = fx.repo.git('rev-parse', 'HEAD');
|
||||
fx.write(d => { d.ci[sha] = [{ name: 'free-tests', status: 'completed', conclusion: 'success' }, { name: 'windows-free-tests', status: 'in_progress', conclusion: null }, { name: 'main-backstop', status: 'completed', conclusion: 'neutral' }]; });
|
||||
const pending = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']);
|
||||
expect(pending.status).toBe(EXIT.refused);
|
||||
expect(line(pending.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} pending (passed=1 failed=0 pending=1 skipped=1)`);
|
||||
expect(pending.out).toContain('CHECK: windows-free-tests in_progress');
|
||||
expect(pending.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} have not settled .*windows-free-tests.*\\(QUEUE_TRAIN_PENDING\\)`));
|
||||
expect(next(pending.out)).toBe(`NEXT: gstack-ship-queue verify --post-merge ${sha} --timeout 1`);
|
||||
fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'failure' }; });
|
||||
const red = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']);
|
||||
expect(red.status).toBe(EXIT.fail);
|
||||
expect(line(red.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} red (passed=1 failed=1 pending=0 skipped=1)`);
|
||||
expect(red.out).toContain('CHECK: windows-free-tests failure');
|
||||
expect(red.err).toMatch(new RegExp(`main's push workflows at ${sha.slice(0, 12)} are red: windows-free-tests.*\\(QUEUE_POST_MERGE_RED\\)`));
|
||||
expect(next(red.out)).toBe('NEXT: read the red run, repair on a new PR or roll back; gstack-ship-queue train status names the freeze if the backstop filed one');
|
||||
fx.write(d => { d.ci[sha]![1] = { name: 'windows-free-tests', status: 'completed', conclusion: 'success' }; });
|
||||
const green = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']);
|
||||
expect([green.status, green.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(green.out, 'POST_MERGE:')).toBe(`POST_MERGE: ${sha.slice(0, 12)} green (passed=2 failed=0 pending=0 skipped=1)`);
|
||||
expect(green.out.split('\n').filter(l => l.startsWith('CHECK: ')).sort()).toEqual(['CHECK: free-tests success', 'CHECK: main-backstop neutral', 'CHECK: windows-free-tests success']);
|
||||
expect(next(green.out)).toBe('NEXT: none');
|
||||
fx.write(d => { d.ci[sha] = []; });
|
||||
const none = queue(fx, ['verify', '--post-merge', sha, '--timeout', '1']);
|
||||
expect([none.status, line(none.out, 'POST_MERGE:')]).toEqual([EXIT.refused, `POST_MERGE: ${sha.slice(0, 12)} none (passed=0 failed=0 pending=0 skipped=0)`]);
|
||||
expect(none.err).toMatch(new RegExp(`no check run has attached to ${sha.slice(0, 12)} yet.*\\(QUEUE_TRAIN_PENDING\\)`));
|
||||
expect(queue(fx, ['verify', '--post-merge', 'not-a-sha']).status).toBe(EXIT.usage);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('ready --dispatch-ci', () => {
|
||||
// Value: protects=a PR that conflicts with the base (a CHANGELOG-only conflict after every stamp) gets its free workflows dispatched on the branch, waited for, and its requires-remote pregate lanes cleared with the run URLs in one command, replacing the four-step hand procedure;
|
||||
// fails_when=the dispatch happens on a fork head, a red run clears a lane, a lane whose workflow was not dispatched is cleared, or ready still says ci: none; why_new=the ci-hint was advisory prose; seam=the gh shim's workflow run, run list and check runs; no_claim=GitHub's real dispatch latency
|
||||
test('dispatches the named workflows on a conflicting PR, waits for their runs, clears the matching pregate lanes with the run URLs and then READYs; a fork head and a red run are refused', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
fx.write(d => {
|
||||
(d.prs['1'] as any).mergeable = 'CONFLICTING';
|
||||
d.ci[head] = [];
|
||||
d.rules['free-tests.yml'] = { ci: 'free-tests', headSha: head };
|
||||
d.rules['windows-free-tests.yml'] = { ci: 'windows-free-tests', headSha: head };
|
||||
});
|
||||
const tree = JSON.parse(fs.readFileSync(pregate, 'utf8')).tree;
|
||||
fs.writeFileSync(pregate, JSON.stringify({ schema_version: 1, tree, head, verdict: 'requires-remote', checks: [{ id: 'unit', status: 'pass' }], requires_remote: [
|
||||
{ lane: 'free-tests.yml/free-tests', platform: 'linux', workflow: 'free-tests.yml', job: 'free-tests', tests: ['test/a.test.ts'], receipt: null },
|
||||
{ lane: 'windows-free-tests.yml/windows-free-tests', platform: 'windows', workflow: 'windows-free-tests.yml', job: 'windows-free-tests', tests: ['test/a.test.ts'], receipt: null },
|
||||
{ lane: 'macos-free-tests.yml/macos', platform: 'macos', workflow: 'macos-free-tests.yml', job: 'macos', tests: ['test/b.test.ts'], receipt: null }] }));
|
||||
const plain = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen']);
|
||||
expect([plain.status, line(plain.out, 'ci:')]).toEqual([EXIT.refused, `ci: none at ${head.slice(0, 12)} (passed=0 failed=0 pending=0 skipped=0)`]);
|
||||
const r = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml,windows-free-tests.yml', '--timeout', '5']);
|
||||
expect(r.out.split('\n').filter(l => l.startsWith('CI_DISPATCH:'))).toEqual([
|
||||
'CI_DISPATCH: free-tests.yml --ref pr-1 run 100 https://example.invalid/runs/100 success',
|
||||
'CI_DISPATCH: windows-free-tests.yml --ref pr-1 run 101 https://example.invalid/runs/101 success']);
|
||||
expect(r.out.split('\n').filter(l => l.startsWith('PREGATE_CLEARED:'))).toEqual([
|
||||
'PREGATE_CLEARED: free-tests.yml/free-tests https://example.invalid/runs/100',
|
||||
'PREGATE_CLEARED: windows-free-tests.yml/windows-free-tests https://example.invalid/runs/101']);
|
||||
expect(line(r.out, 'ci:')).toBe(`ci: green at ${head.slice(0, 12)} (passed=2 failed=0 pending=0 skipped=0)`);
|
||||
expect(line(r.out, 'PREGATE:')).toMatch(/^PREGATE: requires-remote .*/);
|
||||
expect(r.err).toMatch(/1 requires-remote obligation\(s\) without a receipt: macos-free-tests\.yml\/macos.*\(QUEUE_NOT_READY\)/);
|
||||
const saved = JSON.parse(fs.readFileSync(pregate, 'utf8'));
|
||||
expect(saved.requires_remote.map((l: any) => l.receipt)).toEqual(['https://example.invalid/runs/100', 'https://example.invalid/runs/101', null]);
|
||||
expect(dispatches(fx).map(l => l.workflow)).toEqual(['free-tests.yml', 'windows-free-tests.yml']);
|
||||
const again = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']);
|
||||
expect(dispatches(fx).length).toBe(2);
|
||||
expect(line(again.out, 'CI_DISPATCH:')).toBe('CI_DISPATCH: free-tests.yml not dispatched (run 100 already concluded success at this head)');
|
||||
fx.write(d => { d.rules['free-tests.yml'] = { ci: 'free-tests', conclusion: 'failure', headSha: head }; d.ci[head] = []; d.runs.length = 0; });
|
||||
const red = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml', '--timeout', '5']);
|
||||
expect([red.status, line(red.out, 'CI_DISPATCH:')]).toEqual([EXIT.refused, 'CI_DISPATCH: free-tests.yml --ref pr-1 run 102 https://example.invalid/runs/102 failure']);
|
||||
expect(red.err).toMatch(new RegExp(`free-tests\\.yml run 102 concluded failure at ${head.slice(0, 12)}; a red run clears no lane.*\\(QUEUE_NOT_READY\\)`));
|
||||
expect(JSON.parse(fs.readFileSync(pregate, 'utf8')).requires_remote[0].receipt).toBe('https://example.invalid/runs/100');
|
||||
fx.write(d => { (d.prs['1'] as any).isCrossRepository = true; });
|
||||
const fork = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', 'free-tests.yml']);
|
||||
expect([fork.status, fork.err]).toEqual([EXIT.refused, expect.stringMatching(/#1's head is on a fork.*not dispatched.*\(QUEUE_TRAIN_UNTRUSTED\)/)]);
|
||||
expect(queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--no-screen', '--dispatch-ci', '../evil.yml']).status).toBe(EXIT.usage);
|
||||
}, 120_000);
|
||||
});
|
||||
|
||||
describe('train build --dry-run --stack-tests', () => {
|
||||
// Value: protects=a PR whose new tests pass on its own branch but fail once stacked behind another PR's change is caught before the train spends a gate (#3460 on #3451, 2026-10-26);
|
||||
// fails_when=the new test files run against the PR head instead of the integrated prefix, a failing prefix still dry-runs green, or a PR without new tests is misreported; why_new=no pre-dispatch stack check existed; seam=bun test in a scratch worktree; no_claim=the repo's full suite on the prefix
|
||||
test('runs each PR\'s new test files on its integrated prefix T<k>: a pass and a no-tests PR are reported, a failure refuses the dry run and names the prefix, and the ordinary dry run skips it', () => {
|
||||
const fx = make();
|
||||
const h1 = openPr(fx, '1', { file: 'test/pr-1.test.ts', text: "import { expect, test } from 'bun:test';\ntest('pr-1', () => { expect(1).toBe(1); });\n" });
|
||||
const h2 = openPr(fx, '2', { file: 'test/pr-2.test.ts', text: "import { existsSync } from 'node:fs';\nimport { expect, test } from 'bun:test';\ntest('pr-2 wants no pr-1 test beside it', () => { expect(existsSync('test/pr-1.test.ts')).toBe(false); });\n" });
|
||||
const h3 = openPr(fx, '3');
|
||||
readyReceipt(fx, '1', h1);
|
||||
readyReceipt(fx, '2', h2);
|
||||
readyReceipt(fx, '3', h3);
|
||||
const plain = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run']);
|
||||
expect([plain.status, plain.out.includes('STACK_TESTS:')]).toEqual([EXIT.ok, false]);
|
||||
const r = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--dry-run', '--stack-tests']);
|
||||
expect(r.status).toBe(EXIT.fail);
|
||||
expect(r.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([
|
||||
'STACK_TESTS: #1 T1 1 new test file(s) pass (test/pr-1.test.ts)',
|
||||
expect.stringMatching(/^STACK_TESTS: #2 T2 1 new test file\(s\) FAIL \(test\/pr-2\.test\.ts\): .*\(fail\) pr-2 wants no pr-1 test beside it.*1 fail$/),
|
||||
'STACK_TESTS: #3 T3 no new test files']);
|
||||
expect(r.err).toMatch(/#2's new tests fail on the integrated prefix T2 \(test\/pr-2\.test\.ts\); they pass on the PR head alone only if nothing before it in the train changes what they read.*\(QUEUE_TRAIN_STACK_RED\)/);
|
||||
expect(next(r.out)).toBe(`NEXT: fix #2's tests against T2, or reorder with --order, then gstack-ship-queue train build --prs 1@${h1},2@${h2},3@${h3} --dry-run --stack-tests`);
|
||||
expect(line(r.out, 'DRY-RUN:')).toBe('DRY-RUN: nothing written, pushed or dispatched');
|
||||
const reordered = queue(fx, ['train', 'build', '--prs', `1@${h1},2@${h2},3@${h3}`, '--order', '2,1,3', '--dry-run', '--stack-tests']);
|
||||
expect(reordered.status).toBe(EXIT.ok);
|
||||
expect(reordered.out.split('\n').filter(l => l.startsWith('STACK_TESTS:'))).toEqual([
|
||||
'STACK_TESTS: #2 T1 1 new test file(s) pass (test/pr-2.test.ts)',
|
||||
'STACK_TESTS: #1 T2 1 new test file(s) pass (test/pr-1.test.ts)',
|
||||
'STACK_TESTS: #3 T3 no new test files']);
|
||||
expect(fs.readdirSync(path.join(fx.state, 'ship-queue')).filter(n => n.startsWith('stack-tests'))).toEqual([]);
|
||||
expect(queue(fx, ['train', 'build', '--prs', `1@${h1}`, '--stack-tests']).status).toBe(EXIT.usage);
|
||||
}, 180_000);
|
||||
});
|
||||
@@ -196,9 +196,9 @@ describe('ready under a train policy: the screen choice', () => {
|
||||
expect(queue(fx, ['ready', '--pr', '1', '--broad']).status).toBe(EXIT.usage);
|
||||
}, 120_000);
|
||||
|
||||
// Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success);
|
||||
// fails_when=the merge ref's second parent is not compared with the PR head, or a missing merge ref keeps the old QUEUE_NOT_READY code; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in)
|
||||
test('--screen refuses QUEUE_SCREEN_STALE_REF when refs/pull/<n>/merge lags the PR head or is missing, dispatching nothing; a recomputed merge ref admits it', () => {
|
||||
// Value: protects=--screen never dispatches against a merge ref GitHub has not recomputed for the PR head (GSTA-95 on #3431: a stale ref made evals.yml plan 0 cases and report success): a lagging, missing or unfetchable ref is replaced by a scratch merge of the head onto current main (#3465), and an unreachable remote is still refused;
|
||||
// fails_when=the merge ref's parents are not compared with the PR head and current main, or the scratch merge is skipped; why_new=the --screen test above only sees a fresh merge ref; seam=none; no_claim=GitHub's real recompute timing (a bare origin stands in)
|
||||
test('--screen plans the screen on a scratch merge when refs/pull/<n>/merge lags the PR head, is missing or cannot be fetched; an unreachable remote is POLICY_SOURCE_UNAVAILABLE; a recomputed merge ref is used as is', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
const { repo } = fx;
|
||||
fs.appendFileSync(path.join(repo.work, 'pr-1.txt'), 'pushed after the merge ref was computed\n');
|
||||
@@ -208,41 +208,41 @@ describe('ready under a train policy: the screen choice', () => {
|
||||
fx.write(d => { (d.prs['1'] as any).headRefOid = moved; (d.prs['1'] as any).body = shippedBody('1', moved); d.ci[moved] = [{ status: 'completed', conclusion: 'success' }]; });
|
||||
const pregate2 = path.join(repo.root, 'pregate2.json');
|
||||
fs.writeFileSync(pregate2, JSON.stringify({ schema_version: 1, tree: runBin(repo, 'gstack-wtree', []).out.trim(), head: moved, verdict: 'pass', checks: [{ id: 'unit', status: 'pass' }] }));
|
||||
const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']);
|
||||
expect(stale.status).toBe(EXIT.refused);
|
||||
expect(stale.err).toMatch(new RegExp(`refs/pull/1/merge \\([0-9a-f]{12}\\) merges ${head.slice(0, 12)}, not the PR head ${moved.slice(0, 12)}.*\\(QUEUE_SCREEN_STALE_REF\\)`));
|
||||
expect(line(stale.out, 'NEXT:')).toBe('NEXT: resolve the conflict with main (or wait for GitHub to recompute refs/pull/1/merge), push, then gstack-ship-queue ready --pr 1 --screen; a screen dispatched on a stale merge ref plans no cases');
|
||||
const stale = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']);
|
||||
expect([stale.status, stale.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(stale.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: scratch merge [0-9a-f]{12} of #1@${moved.slice(0, 12)} onto main@[0-9a-f]{12} \\(refs/pull/1/merge [0-9a-f]{12} merges ${head.slice(0, 12)}, not ${moved.slice(0, 12)}\\); would push screen/1-${moved.slice(0, 12)} \\(dry run\\)$`));
|
||||
repo.git('push', '-q', 'origin', ':refs/pull/1/merge');
|
||||
const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']);
|
||||
expect(missing.status).toBe(EXIT.refused);
|
||||
expect(missing.err).toMatch(/has no refs\/pull\/1\/merge.*\(QUEUE_SCREEN_STALE_REF\)/);
|
||||
const missing = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']);
|
||||
expect([missing.status, missing.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(missing.out, 'SCREEN_MERGE:')).toContain('(origin has no refs/pull/1/merge); would push');
|
||||
const tree = repo.git('rev-parse', `${moved}^{tree}`);
|
||||
const main = repo.git('rev-parse', 'origin/main');
|
||||
const badMerge = spawnSync('git', ['-C', repo.origin, 'hash-object', '-t', 'commit', '-w', '--stdin', '--literally'], { input: `tree ${tree}\nparent ${main}\nparent ${moved}\nauthor A <a@example.com> 1700000000 +0000\ncommitter B 1700000000 +0000\n\nmerge without a committer email\n`, encoding: 'utf8', timeout: 30_000 }).stdout.trim();
|
||||
spawnSync('git', ['-C', repo.origin, 'update-ref', 'refs/pull/1/merge', badMerge], { timeout: 30_000 });
|
||||
repo.git('config', 'fetch.fsckObjects', 'true');
|
||||
const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']);
|
||||
expect(unfetchable.status).toBe(EXIT.refused);
|
||||
expect(unfetchable.err).toMatch(/could not fetch refs\/pull\/1\/merge from origin: .*\(QUEUE_SCREEN_STALE_REF\)/);
|
||||
expect(line(unfetchable.out, 'NEXT:')).toBe('NEXT: check the remote, then gstack-ship-queue ready --pr 1 --screen');
|
||||
const unfetchable = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']);
|
||||
expect([unfetchable.status, unfetchable.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(unfetchable.out, 'SCREEN_MERGE:')).toMatch(/\(refs\/pull\/1\/merge [0-9a-f]{12} could not be fetched\); would push/);
|
||||
repo.git('config', '--unset', 'fetch.fsckObjects');
|
||||
const originUrl = repo.git('remote', 'get-url', 'origin');
|
||||
repo.git('remote', 'set-url', 'origin', path.join(repo.root, 'no-such-origin.git'));
|
||||
const unlisted = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen']);
|
||||
repo.git('remote', 'set-url', 'origin', originUrl);
|
||||
expect(unlisted.status).toBe(EXIT.refused);
|
||||
expect(unlisted.err).toMatch(/could not list origin's refs: .*\(QUEUE_SCREEN_STALE_REF\)/);
|
||||
expect(unlisted.err.split('\n').filter(Boolean).length).toBe(1);
|
||||
expect(unlisted.err).toMatch(/git fetch origin main failed: .*\(POLICY_SOURCE_UNAVAILABLE\)/);
|
||||
expect(unlisted.err.split('\n').filter(Boolean).length).toBeGreaterThan(0);
|
||||
expect(dispatches(fx)).toEqual([]);
|
||||
expect(fs.existsSync(path.join(fx.state, 'ship-queue', 'ready'))).toBe(false);
|
||||
repo.git('push', '-q', '-f', 'origin', `${mergeRefCommit(fx, moved)}:refs/pull/1/merge`);
|
||||
const fresh = queue(fx, ['ready', '--pr', '1', '--pregate', pregate2, '--screen', '--dry-run']);
|
||||
expect([fresh.status, fresh.err]).toEqual([EXIT.ok, '']);
|
||||
expect(line(fresh.out, 'SCREEN_MERGE:')).toMatch(new RegExp(`^SCREEN_MERGE: GitHub's refs/pull/1/merge [0-9a-f]{12} merges the head ${moved.slice(0, 12)} onto main@[0-9a-f]{12}$`));
|
||||
expect(line(fresh.out, 'SCREEN_DISPATCH:')).toContain('(dry run: not dispatched)');
|
||||
expect(repo.git('ls-remote', 'origin', 'refs/heads/screen/*')).toBe('');
|
||||
}, 120_000);
|
||||
|
||||
// Value: protects=a finished screen that planned 0 cases never counts as a screen: ready dispatches a fresh one instead of adopting it and train build seats the slot unscreened with a SCREEN: line;
|
||||
// fails_when=the census is not read (a hollow run is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in)
|
||||
// fails_when=the census is not read (a not-run screen is followed, or ORDER says screened); why_new=no test reads a screen run's record; seam=none; no_claim=evals.yml's real artifact (the shim's record stands in)
|
||||
test('a screen run that planned 0 cases is not followed by ready and is counted as not-run by train build', () => {
|
||||
const { fx, head, pregate } = readyFixture();
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: null }; });
|
||||
@@ -260,7 +260,7 @@ describe('ready under a train policy: the screen choice', () => {
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 0 } }; });
|
||||
const emptyPlan = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(line(emptyPlan.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 102 .*not following run 101 planned no cases/);
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 } }; });
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success', plan: { planned: 3 }, binding: { verdict: 'GREEN' } }; });
|
||||
const live = queue(fx, ['ready', '--pr', '1', '--pregate', pregate, '--screen']);
|
||||
expect(live.status).toBe(EXIT.ok);
|
||||
expect(line(live.out, 'SCREEN_DISPATCH:')).toMatch(/^SCREEN_DISPATCH: evals\.yml run 103 .*not following run 102 planned no cases \(an empty plan manifest; concluded success\)/);
|
||||
@@ -269,10 +269,11 @@ describe('ready under a train policy: the screen choice', () => {
|
||||
expect(line(followed.out, 'SCREEN_DISPATCH:')).toBe('SCREEN_DISPATCH: following run 103 (same merge SHA, base and policy; no duplicate)');
|
||||
expect(dispatches(fx, 'evals.yml').length).toBe(4);
|
||||
const liveBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']);
|
||||
expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']);
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'in_progress', conclusion: null }; });
|
||||
expect([liveBuild.status, line(liveBuild.out, 'ORDER:'), line(liveBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 GREEN']);
|
||||
fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'in_progress', conclusion: null }); });
|
||||
const runningBuild = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']);
|
||||
expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', '']);
|
||||
expect([runningBuild.status, line(runningBuild.out, 'ORDER:'), line(runningBuild.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', 'SCREEN: #1 run 103 still running; counted as screened']);
|
||||
fx.write(d => { for (const run of d.runs) if ((run as any).databaseId === 103) Object.assign(run, { status: 'completed', conclusion: 'success' }); });
|
||||
fx.write(d => { d.rules['evals.yml'] = { status: 'completed', conclusion: 'success' }; d.artifacts['103']['paid-plan'] = { reuse: 'confirmed' }; });
|
||||
const unreadable = queue(fx, ['train', 'build', '--prs', `1@${head}`, '--dry-run']);
|
||||
expect([unreadable.status, line(unreadable.out, 'ORDER:'), line(unreadable.out, 'SCREEN:')]).toEqual([EXIT.ok, 'ORDER: 1:#1(screened)', "SCREEN: #1 run 103's paid-plan manifest.json is unreadable; counted as screened, unverified"]);
|
||||
|
||||
Reference in new issue
Block a user