From 7d4e71bd940a9611ffc7020a4128403952c77af9 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 31 Oct 2026 09:50:03 -0700 Subject: [PATCH 1/6] v1.91.375.0 captain: stop a finished /ship subagent at READY once its parent subscribes to the PR (#3597) * captain: stop a finished /ship subagent at its READY after subscribing to its PR Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: CHANGELOG for stopping finished /ship subagents (unreleased) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * captain: /ship subagent stop review fixes (subscribe at PR open, parent owns post-READY work, unconditional one-line wake, tight budget) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 9 +++++++++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- captain/templates/brief.md | 1 + capy/skill/gstack/SKILL.md | 1 + docs/captain.md | 7 +++++++ package.json | 2 +- test/fixtures/context-budget.json | 2 +- 8 files changed, 22 insertions(+), 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 220029eb1..7d56aa2a2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## [1.91.375.0] - 2026-10-31 + +**A /ship subagent now stops once its READY is sent, so CI and review callbacks on a frozen head stop waking it.** Finished /ship subagents kept getting woken by checks and reviews on a PR they had already READY'd. Each wake cost tokens, and some acted on a head that was supposed to be frozen. The parent now subscribes to the PR first, so red checks and human review still reach an owner, and then stops the subagent. + +### Changed +- **`capy/skill/gstack/SKILL.md`** (Plan, approve, ship): the parent subscribes to the PR when the /ship subagent opens it and keeps that subscription until the PR merges or closes. It stops the subagent at READY and owns the PR from then on. A stopped /ship subagent woken by a check, review or merge notice reports one line (what woke it, its pin, the PR head) and ends the turn, while a message from its parent is an instruction it carries out. +- **`docs/captain.md`** (the lane's /ship sequence): the host-neutral rule (the parent owns red checks, review, the slot stamp, re-READY and receipt carry after READY), with a pointer to the Capy mechanics. **`captain/templates/brief.md`**: one pointer line. +- **`test/fixtures/context-budget.json`**: the Capy skill's eager ceiling moves from 5136 to 5312 for the 176 tokens it grew (5136 to 5312, measured with `lib/context-bill`), set at the captured value as main's last raise was, rather than a fresh 10% headroom. + ## [1.91.374.0] - 2026-10-31 **/ship pauses on a red base instead of failing.** When main is red for a reason outside a PR, /ship's Step 5 used to fail or ask the fix/TODO/blame/skip question, and unattended lanes either burned runs or held by hand. Truther's two main-red fixes (#815 and the cover-contract branch) each failed only on the other's broken cases, so neither could pass /ship alone until a Captain folded them into #818 by hand (GSTA-157, from the 2026-10-29 speed feedback). diff --git a/VERSION b/VERSION index e849722ad..d0cfa08b9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.374.0 +1.91.375.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 2817a10b0..4bf39473a 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.374.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.375.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/captain/templates/brief.md b/captain/templates/brief.md index efb283815..6af25e7cd 100644 --- a/captain/templates/brief.md +++ b/captain/templates/brief.md @@ -34,6 +34,7 @@ Spend cap: 3. Ship. CHANGELOG entry under [Unreleased]; GUARD_CHECKS, pregate and /ship at the final head; one unstamped READY block (captain/templates/READY.md) to me; the PR body carries gstack-approvals render. Never merge; stamp only on my slot call. + The lane subscribes to the PR, then stops its /ship subagent at READY (docs/captain.md, the lane's /ship sequence). Done when: the READY block is sent at a frozen head, reports/.md is written, and its Approvals table carries the plan approval row (the owner's words, the date) or the diff --git a/capy/skill/gstack/SKILL.md b/capy/skill/gstack/SKILL.md index 6bbb4e0c5..b513fdbdf 100644 --- a/capy/skill/gstack/SKILL.md +++ b/capy/skill/gstack/SKILL.md @@ -55,6 +55,7 @@ Every Capy thread that uses gstack follows these for new work, under a Captain o - **Plan first.** Before building new functionality, run `/autoplan` on it as the first act after the install check, sized to the work (scope flags and `--max-rounds`; a one-file fix gets `--no-ui --no-dx --max-rounds 1` with `--estimate-usd` and a small `--spend-cap`, never no run). - **ELI5, then the word.** Send the plan and gate to the person, or the Captain, in plain language: `gstack-owner-brief render --captain --out --captain-name ""`, its choices as lettered lines in the ask format (`docs/captain.md`) and "do it" taking every recommendation at full scope. Build only after their word is answered into the run and it reports `APPROVED`; then `gstack-approvals import --autoplan --by ""`. - **/ship before ready.** Run /ship at the final pushed head before a PR is called ready; `SHIP_RECEIPT: pr= head=` must name that head, and a later push needs a new /ship. The review inside /ship, like the autoplan, is sized to the change. +- **Stop a finished /ship subagent.** When a /ship subagent opens its PR, the parent subscribes to that PR with Capy's subscribe tool and keeps that one subscription until the PR merges or closes, re-subscribing if the watch expires, then cancels it. Once the subscription is in place and READY arrives, the parent stops the subagent and owns the PR from then on, slot stamp, re-READY and receipt carry included. A stopped /ship subagent woken by a check, review or merge notice, not by its parent, reports one line naming what woke it, its pin and the PR head (`gh pr view --json headRefOid --jq .headRefOid`) and ends the turn without acting, whether or not the head moved. A message from its parent is an instruction it carries out. - **No merge without the owner's word.** A thread never merges on its own; under a Captain the controller merges on a recorded standing approval, and a thread with no Captain asks the person. Exempt from the plan step: read-only work, the daily flaky pass's repairs, slot stamps, CHANGELOG-only resolves, cap re-pins under a written grant, S1 emergencies (a broken install or upgrade, a red main, a security bug) and fixes from an already-diagnosed, audited problem list. Every other fix gets a one-round, eng-only plan. diff --git a/docs/captain.md b/docs/captain.md index f804bbf7c..65cee4050 100644 --- a/docs/captain.md +++ b/docs/captain.md @@ -254,6 +254,13 @@ a push included. The sequence that works unattended on Capy: 10. After the slot call, the stamp commit keeps the pin's receipt; re-READY with the same receipt, no second /ship. +When /ship runs in a subagent, the parent subscribes to the pull request's check and +review notices as soon as the subagent opens it, and stops the /ship subagent only once +that subscription is in place and its READY arrives. From then on the parent owns the pull +request: red checks, human review, the slot stamp and re-READY of step 10, and any receipt +carry. It does that work itself or sends it to the subagent as an explicit instruction. On +Capy the mechanics are in [the gstack skill](../capy/skill/gstack/SKILL.md#plan-approve-ship). + When the base moves under a reviewed branch, merge it in; never rebase. A merge that leaves the product diff unchanged carries the receipt with `gstack-ship-receipt carry` (pregate at HEAD, no review rerun), and a merge whose resolution changed product code diff --git a/package.json b/package.json index 36f46b3a6..81a67521f 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.374", + "version": "1.91.375", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/test/fixtures/context-budget.json b/test/fixtures/context-budget.json index 9f26e133f..ba51a7dba 100644 --- a/test/fixtures/context-budget.json +++ b/test/fixtures/context-budget.json @@ -9,7 +9,7 @@ "browser-skills/hackernews-frontpage": 371, "canary": 7738, "captain": 6551, - "capy/skill/gstack": 5136, + "capy/skill/gstack": 5312, "careful": 1239, "codex": 9890, "context-restore": 6287, From 5ba425af4b9a6734317d43b6ea9b4077d08ff1af Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 31 Oct 2026 10:05:40 -0700 Subject: [PATCH 2/6] v1.91.376.0 feat(ubicloud): reattach to a live verify VM and checkpoint tiers (GSTA-158 P3) (#3615) * feat(ubicloud): reattach to a live verify VM and checkpoint tiers (GSTA-158 P3) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ubicloud): replay on reattach, tear down TTL-less VMs on HUP, trim skill prose (GSTA-158 P3) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 16 + VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- docs/TESTING_INTERNALS.md | 16 + package.json | 2 +- scripts/ubicloud/pregate-remote.sh | 41 +- scripts/ubicloud/ubi-runner.sh | 668 +++++++++++++++++++++++++-- test/ubicloud-pregate-remote.test.ts | 51 +- test/ubicloud-runner.test.ts | 422 +++++++++++++++-- ubicloud/SKILL.md | 20 +- ubicloud/SKILL.md.tmpl | 20 +- 11 files changed, 1168 insertions(+), 92 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7d56aa2a2..67649e356 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,21 @@ # Changelog +## [1.91.376.0] - 2026-10-31 + +bump: minor +--- +**A remote verify now survives a dead local runner, and a long verify can resume from the tiers that already passed (GSTA-158 P3).** Before, a Capy machine that went to sleep or a runner that got killed lost the whole remote run, and the paid VM ran on until its TTL. Now `ubi-runner run` and `verify` run each phase detached on the VM, and running the same command again reattaches to the live VM. A verify can be split into tiers, and a rerun skips the tiers that passed. + +### Added +- **Reattach by run key.** The key covers the checkout's content, the command, setup, env, a hash of each `--pass` value, size, image, origin/HEAD and the runner itself. A rerun prints `ubi-runner: reattached to created_at= key=`, replays what the phase already printed and follows it from there. A finished setup is not rerun. `UBI_ATTACH_BUSY` means another live runner owns the VM. Exit 125 `UBI_ATTACH_LOST` means the remote session died without an exit status. +- **Reattach only while the TTL lasts.** A reattach needs the VM's remaining TTL to cover the run still to do (each unfinished phase's `UBI_RUN_TIMEOUT` bound) plus `UBI_REATTACH_MARGIN` (900 s). Otherwise the runner prints `UBI_REATTACH_TTL_SHORT` with the numbers, destroys the old VM and starts a fresh one. Tier checkpoints carry over to it. +- **Verify tiers.** `.gstack/ubicloud.json` can list `tiers: [{name, command}]` in place of `command`. Each tier writes a start, pass or fail checkpoint under `UBI_CHECKPOINT_DIR`. A rerun prints `UBI_VERIFY_RESUME:` per tier and reuses passes younger than `UBI_CHECKPOINT_MAX_AGE` whose logs still hash. It reruns a recorded fail only with `--rerun-failed`. The last line is `UBI_VERIFY: …`. +- **`attach NAME`, `run --probe` and `--key-input`.** + +### Changed +- **HUP once a phase is live detaches** (`UBI_DETACHED`, exit 129, `UBICLOUD: left=1 … detached=1`): the VM keeps running under its TTL. INT and TERM still tear it down. +- **`gstack-pregate --stage tests --remote` reattaches** to the VM a killed run left. It bills `spent=` from the original create time, and a cap that is already spent refuses with `PREGATE_REMOTE_CAP_SPENT`. The `PREGATE_REMOTE:` receipt is unchanged. + ## [1.91.375.0] - 2026-10-31 **A /ship subagent now stops once its READY is sent, so CI and review callbacks on a frozen head stop waking it.** Finished /ship subagents kept getting woken by checks and reviews on a PR they had already READY'd. Each wake cost tokens, and some acted on a head that was supposed to be frozen. The parent now subscribes to the PR first, so red checks and human review still reach an owner, and then stops the subagent. diff --git a/VERSION b/VERSION index d0cfa08b9..4f0a874d2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.375.0 +1.91.376.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index 4bf39473a..b6a329420 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.375.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.376.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 2ed948c98..278790990 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1034,6 +1034,22 @@ never forwards `UBICLOUD_*`. All of this is pinned offline in `test/ubicloud-runner.test.ts` against a fake API and a fake `ssh`, including a fixture whose fake API creates, so `run` and `verify` execute end to end. +`run` and `verify` start their setup, command and tier phases detached on the +VM (output, leader pid and exit status in `~/.ubirun-.*`) and key each +run locally before any VM exists (content tree, command, setup content, env, +pass-value hashes, size, location, image, storage, origin/HEAD, the runner's +own sha256). A rerun of the same command after the runner died reattaches to +the live VM and continues from the first phase without an exit file +(`UBI_ATTACH_BUSY` while another runner owns it, exit 125 `UBI_ATTACH_LOST` +when the session left no status); HUP detaches (`UBI_DETACHED`, exit 129), INT +and TERM still tear down. A `.gstack/ubicloud.json` with `tiers` records +start/pass/fail per tier under `UBI_CHECKPOINT_DIR//`, so a resume on +a fresh VM reruns setup and only the tiers without a pass; `pregate-remote.sh` +adds its tree and planned-set digest to the key, so a rerun of +`gstack-pregate --stage tests --remote` reattaches and bills from the original +`created_at`. The fake VM in `test/ubicloud-runner.test.ts` runs these phases +for real in a directory. + Under 8 cores (`gstack-doctor`'s `cores` row says so), run the full unit and E2E tiers on a remote gate: `gstack-config set test_backend ubicloud` records the choice and the row names the runner. Resume checklist after a wake or a diff --git a/package.json b/package.json index 81a67521f..54a42668a 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "gstack", - "version": "1.91.375", + "version": "1.91.376", "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.", "license": "MIT", "type": "module", diff --git a/scripts/ubicloud/pregate-remote.sh b/scripts/ubicloud/pregate-remote.sh index 68e5b5692..c7eb3f90f 100755 --- a/scripts/ubicloud/pregate-remote.sh +++ b/scripts/ubicloud/pregate-remote.sh @@ -27,10 +27,18 @@ # teardown) times the size's price. Without a key it runs the same files # locally in parallel shards and reports spent=0. # +# Reattach: the tree and the planned set's digest go into ubi-runner's run key +# (--key-input), so a rerun after the local runner died (a machine stop, a +# SIGKILL) reattaches to the VM still running this gate instead of paying for a +# second one. Its minutes since created_at count against the cap: the wall +# budget shrinks by them (PREGATE_REMOTE_CAP_SPENT when none is left), and +# spent= bills from the original create. +# # Exit codes: 0 pass (or a paid-only selection, tests=0); 1 test failures (or # a rejected file list); 2 missing input; 3 tree or plan mismatch (or the tree # moved during the run); 4 infrastructure failure (create, setup, SSH, stop -# deadline); 5 the spend cap or pregate's deadline refuses the run. Every refusal line starts with a +# deadline, another gate on this machine owning the VM to reattach to); 5 the +# spend cap or pregate's deadline refuses the run. Every refusal line starts with a # PREGATE_REMOTE_* reason code (docs/pregate.md#remote-gate). Nothing is # uploaded or spent before the local tree check. set -euo pipefail @@ -147,8 +155,6 @@ if [ -n "$outer_s" ]; then fi if [ "$outer_stop" -lt "$stop_after" ]; then stop_after=$outer_stop stop_why="$outer_from ${outer_s}s minus 120 s teardown reserve and 15 s margin"; fi fi -say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)" - digest=$(tr -d '\r' <"$files" | grep . | LC_ALL=C sort | sha256sum | cut -d' ' -f1) prelude="$logs/vm-setup.sh" { @@ -183,6 +189,28 @@ args=(run -s "$size" --src "$root" --setup "$prelude" --pull "/tmp/gstack-free-test-*:$logs" --pull "work/$(basename "$root")/.context/free-test-logs:$logs") [ -z "${GSTACK_FREE_JOBS:-}" ] || args+=(--env "GSTACK_FREE_JOBS=$GSTACK_FREE_JOBS") +args+=(--key-input "pregate-remote tree=$tree plan=$digest") + +# A rerun of this gate for the same tree and planned set has the same +# ubi-runner run key, so it reattaches to the VM a killed run left behind +# (`ubi-runner: reattached to created_at=`). The minutes that VM +# has billed since created_at come off the cap's wall budget first. +prior_vm="" +attach=$(bash "$here/ubi-runner.sh" "${args[@]}" --probe -- "bash ~/.pregate-run.sh" 2>/dev/null \ + | sed -n 's/^UBI_ATTACH_INFO: vm=\([^ ]*\) created_at=\([0-9]*\) key=[0-9a-f]* busy=\([01]\)$/\1 \2 \3/p' | tail -1) || attach="" +if [ -n "$attach" ]; then + read -r prior_vm prior_created prior_busy <<<"$attach" + [ "$prior_busy" = 0 ] || stop 4 PREGATE_REMOTE_ATTACH_BUSY "another gate on this machine still runs VM $prior_vm for this tree and planned set" "wait for it; this run created nothing and spent nothing" + prior_min=$(( ($(date +%s) - prior_created + 59) / 60 )) + left_s=$(( (wall_min - 3 - prior_min) * 60 )) + if [ "$left_s" -le 0 ]; then + stop 5 PREGATE_REMOTE_CAP_SPENT "VM $prior_vm, created ${prior_min} min ago for this tree and planned set, has already used the cap's $wall_min min wall (\$$(usd "$prior_min") of \$$cap)" \ + "nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down $prior_vm" "or rerun with a higher --spend-cap to let it finish: $rerun" + fi + if [ "$left_s" -lt "$stop_after" ]; then stop_after=$left_s stop_why="the cap's $wall_min min wall minus 3 min for teardown and $prior_min min already billed since $prior_vm was created"; fi + say "reattaching to VM $prior_vm, created ${prior_min} min ago by an earlier run of this gate for the same tree and planned set" +fi +say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)" # The VM run gets its own session (python3, which ubi-runner already needs, for # setsid on Linux and macOS): pregate's --timeout kill signals this @@ -220,7 +248,7 @@ while :; do rm -f "$log.pid" "$log.timedout" "$log.left" t0=$(date +%s) vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))" - say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name" + [ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name" UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \ bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" & vmrun=$! @@ -236,6 +264,9 @@ while :; do tailer="" t_end=$(date +%s) t_create=$(at 'ubi-runner: creating ') + # A reattached VM is billed from its original create. + reattached=$(sed -n 's/^[0-9]* ubi-runner: reattached to [^ ]* created_at=\([0-9][0-9]*\) .*/\1/p' "$log" | tail -1) + [ -z "$reattached" ] || t_create=$reattached if [ -n "$t_create" ] && ! grep -q 'ubi-runner: create failed' "$log"; then minutes=$(( (t_end - t_create + 59) / 60 )); billed=$(( billed + (minutes > 0 ? minutes : 1) )); fi if [ "$rc" -ne 0 ] && [ "$attempt" -eq 1 ] && grep -q 'vCPU quota exhausted' "$log"; then say "vCPU quota refused the VM; one retry in ${PREGATE_REMOTE_QUOTA_WAIT_S:-60} s" @@ -279,7 +310,7 @@ if [ -n "$result" ] && [ "$rc" -ne 124 ]; then echo "PREGATE_REMOTE: tree=$vm_tree tests=$(field tests "$result") failed=$(field failed "$result") spent=$spent" exit $(( rc == 0 ? 0 : 1 )) fi -vm=$(sed -n 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' "$log" | tail -1) +vm=$(sed -n -e 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' -e 's/^[0-9]* ubi-runner: reattached to \([^ ]*\) .*/\1/p' "$log" | tail -1) down=$(grep -E 'ubi-runner: (destroyed|gone:|WARNING|FAILED to destroy)' "$log" | tail -1 | cut -d' ' -f2- || true) cause=$(grep -E 'ubi-runner: ' "$log" | grep -vE 'ubi-runner: (destroyed|gone:|destroy requested|waiting for an in-flight)' | tail -1 | cut -d' ' -f2- || true) [ "$rc" -ne 124 ] || cause="the VM was told to stop after ${stop_after}s ($stop_why) and torn down" diff --git a/scripts/ubicloud/ubi-runner.sh b/scripts/ubicloud/ubi-runner.sh index 5d15ee8d2..441eaec8d 100755 --- a/scripts/ubicloud/ubi-runner.sh +++ b/scripts/ubicloud/ubi-runner.sh @@ -33,8 +33,9 @@ # Teardown: `up` records each VM's state (location, create status) before it # sends the create request, and `down` destroys and polls until the VM is # confirmed gone or provably never existed, waiting out a create that may still -# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, HUP, QUIT) runs -# that `down`; further signals are ignored until it finishes. A create refused +# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, QUIT, and HUP +# until a remote phase is live) runs that `down`; further signals are ignored +# until it finishes. A create refused # for vCPU quota prints the project's usage per owner. # # The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive @@ -69,6 +70,33 @@ # UBI_RUN_TIMEOUT seconds (0, unlimited), exiting 124 with the last 40 lines, # so a hung verify fails fast instead of holding the VM until its TTL. # +# Detached phases and reattach: `run` and `verify` start their setup, command +# and tier phases on the VM detached from the ssh connection (setsid; stdout to +# ~/.ubirun-.out, stderr to .err, the session leader's pid in .pid, and +# the exit status in .exit, written by rename once the streams are closed) and +# stream both files back. Before creating a VM they compute a run key locally +# (the checkout's content tree, the command, the setup script's content, every +# --env entry, each --pass name with a sha256 of its value, size, location, +# image, storage, origin/HEAD, tiers and this script's own sha256) and look for +# a run record (//run) with the same key whose VM still exists: +# its runner alive (lease pid and start time) refuses with UBI_ATTACH_BUSY, +# otherwise the run reattaches, printing `reattached to created_at= +# key=`, and continues from the first phase without an exit file. After +# unpack the VM recomputes the tree and refuses a mismatch (UBI_VM_TREE_MISMATCH). +# A leader gone with no exit file exits 125 (UBI_ATTACH_LOST). Signals: INT +# and TERM tear the VM down as before; HUP once a remote phase is live detaches +# (UBI_DETACHED, exit 129) and leaves the VM under its TTL; SIGKILL or a machine +# stop leaves the record and the VM, and the next run of the same command +# reattaches. `attach NAME` resumes a recorded run explicitly. Reattach never +# extends a TTL, and the reaper still applies. +# +# Tier checkpoints: a .gstack/ubicloud.json with `tiers` runs each tier as its +# own detached phase after setup, and records start/pass/fail per tier under +# UBI_CHECKPOINT_DIR// (write-once JSON plus the tier's log, published +# under a lock with a monotonic generation). A rerun reuses a pass younger than +# UBI_CHECKPOINT_MAX_AGE whose log re-hashes, so a resume on a fresh VM reruns +# only setup and the tiers without a pass. +# # `verify` is the one entry point for a repository's full remote verify: it # reads .gstack/ubicloud.json (setup, command, size, location, env, pass, pull) # and runs it through `run`, so lanes share one committed definition instead of @@ -100,6 +128,11 @@ PREFIX="ubirun" if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD" elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv" else VM_RECORD="$STATE_ROOT/vms.tsv"; fi +if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR" +elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints" +else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi +CHECKPOINT_MAX_AGE="${UBI_CHECKPOINT_MAX_AGE:-86400}" +REATTACH_MARGIN="${UBI_REATTACH_MARGIN:-900}" die() { echo "ubi-runner: $*" >&2; exit 1; } log() { echo "ubi-runner: $*" >&2; } @@ -109,7 +142,7 @@ for bin in curl python3 ssh ssh-keygen tar; do done UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}" [[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)" -for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT; do +for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT CHECKPOINT_MAX_AGE REATTACH_MARGIN; do [[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)" done [ -z "$QUOTA_VCPUS" ] || [[ "$QUOTA_VCPUS" =~ ^[0-9]+$ ]] || die "UBI_QUOTA_VCPUS must be a whole number of vCPUs" @@ -296,6 +329,410 @@ watched() { return "$rc" } +proc_start() { ps -o lstart= -p "$1" 2>/dev/null | tr -s ' ' || true; } +fsize() { if [ -f "$1" ]; then wc -c <"$1" | tr -d ' '; else echo 0; fi; } +sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | cut -d' ' -f1; else shasum -a 256 | cut -d' ' -f1; fi; } + +# TREE_SH: print the content tree of the current directory, the same way here +# and on the VM: at a checkout's root, `git add -A` into a copy of its index +# (tracked plus untracked files that are not ignored, the files pack uploads); +# elsewhere, the same into a scratch repository over the directory. +TREE_SH='idx=$(mktemp -d "${TMPDIR:-/tmp}/ubirun-tree.XXXXXX") || exit 1 +if [ "$(git rev-parse --show-toplevel 2>/dev/null)" = "$(pwd -P)" ]; then + gi=$(git rev-parse --git-path index) + if [ "$(git config --bool core.splitIndex)" != true ] && [ -f "$gi" ]; then cp "$gi" "$idx/index"; fi +else + git init -q --bare "$idx/git" && export GIT_DIR="$idx/git" GIT_WORK_TREE=. +fi +GIT_INDEX_FILE="$idx/index" git add -A >/dev/null 2>&1 && GIT_INDEX_FILE="$idx/index" git write-tree +rc=$?; rm -rf "$idx"; exit $rc' + +# vm_tree NAME QDEST: the content tree of QDEST on the VM. +vm_tree() { + remote "$1" "command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git; } >/dev/null 2>&1; cd $2 && bash -c $(sq "$TREE_SH")" /dev/null || echo none; } + +# phase_status NAME PHASE: "exit " once the phase wrote its exit file, +# "alive" while its session leader runs, "lost" when the leader is gone with no +# exit file, "none" when it never started. +phase_status() { + local p="~/.ubirun-$2" + remote "$1" "if [ -f $p.exit ]; then echo exit \$(cat $p.exit); elif [ -s $p.pid ] && kill -0 \$(cat $p.pid) 2>/dev/null; then echo alive; elif [ -s $p.pid ]; then echo lost; else echo none; fi" $p.pid.tmp && mv $p.pid.tmp $p.pid; ( $3 ) >$p.out 2>$p.err $p.exit.tmp && mv $p.exit.tmp $p.exit") + remote "$1" "rm -f $p.exit $p.pid; : >$p.out; : >$p.err; if command -v setsid >/dev/null 2>&1; then setsid -f bash -c $inner /dev/null 2>&1; else nohup bash -c $inner /dev/null 2>&1 & fi" /dev/null +} + +# stream_cmd PHASE OUT_OFFSET ERR_OFFSET: follow both output files from the +# given byte offsets until the session leader exits. +stream_cmd() { + local p="~/.ubirun-$1" + printf '%s' "i=0; while [ ! -s $p.pid ] && [ ! -f $p.exit ] && [ \$i -lt 600 ]; do sleep 0.1; i=\$((i + 1)); done; pid=\$(cat $p.pid 2>/dev/null); f=; [ -z \"\$pid\" ] || f=\"-f --pid=\$pid\"; tail -c +$(( $3 + 1 )) \$f $p.err >&2 & tail -c +$(( $2 + 1 )) \$f $p.out; wait" +} + +PHASE_LIVE="" STREAM_PID="" RUN_KEY="" TREE="" + +# on_hup: once a remote phase is live, HUP detaches: the VM keeps running under +# its TTL and the next run of the same command reattaches. Before that, HUP +# tears down like INT and TERM. +on_hup() { + [ -n "$PHASE_LIVE" ] || exit 130 + [ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null || true + trap - EXIT + rm -f "$(state_dir "$RUN_VM")/lease" + leftover "left=1 owner=$OWNER vm=$RUN_VM detached=1" + echo "UBI_DETACHED: vm=$RUN_VM key=${RUN_KEY:0:12} reattach: rerun the same command" >&2 + exit 129 +} + +# detached NAME PHASE INNER: run INNER as a detached phase (or reattach to it), +# streaming its output to this process and to the VM's state dir +# (.out, .err and the combined .log) under the same +# heartbeat and watchdog as watched. RUN_TIMEOUT counts from the phase's +# recorded start, not from a reattach. Returns the phase's exit status, 124 +# after a watchdog kill, 125 when the session is gone without an exit status. +detached() { + local name=$1 phase=$2 inner=$3 dir st rc=0 tries=0 start now idle elapsed last_beat why old_traps spid sst oo oe + load "$name" + dir=$(state_dir "$name") + local lout="$dir/$phase.out" lerr="$dir/$phase.err" out="$dir/$phase.log" p="~/.ubirun-$phase" + st=none + if [ -f "$dir/phase.$phase.start" ]; then + st=$(phase_status "$name" "$phase") || st="" + fi + case $st in + none) + : >"$lout"; : >"$lerr"; : >"$out" + date +%s >"$dir/phase.$phase.start" + phase_start "$name" "$phase" "$inner" || { log "could not start $phase on $name"; return 255; } ;; + alive) log "$phase is still running on $name; following it" ;; + "exit "*) ;; + lost) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; return 125 ;; + *) log "could not read the state of $phase on $name"; return 255 ;; + esac + PHASE_LIVE=1 + start=$(cat "$dir/phase.$phase.start") + # A runner killed on this machine leaves its stream client behind; it would + # append to the same local copies. + if [ -f "$dir/stream" ] && read -r spid sst <"$dir/stream" && [ -n "$spid" ] && [ "$(proc_start "$spid")" = "$sst" ]; then + kill -TERM -- "-$spid" 2>/dev/null || true + fi + oo=$(fsize "$lout") oe=$(fsize "$lerr") + if [ "$st" != none ]; then head -c "$oo" "$lout"; head -c "$oe" "$lerr" >&2; fi + old_traps=$(trap -p INT TERM) + trap '[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null; exit 130' INT TERM + last_beat=$(date +%s) + while :; do + touch "$out" + # shellcheck disable=SC2046 + ( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \ + ssh $(ssh_opts) "ubi@$IP" "umask 022; $(stream_cmd "$phase" "$oo" "$oe")" ) \ + >(tee -a "$lout" | tee -a "$out") 2> >(tee -a "$lerr" | tee -a "$out" >&2) & + STREAM_PID=$! + echo "$STREAM_PID $(proc_start "$STREAM_PID")" >"$dir/stream" + why="" + while :; do + sleep 1 + kill -0 "$STREAM_PID" 2>/dev/null || break + now=$(date +%s) + elapsed=$(( now - start )) + idle=$(( now - $(mtime "$out") )) + if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then + why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)" + elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then + why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)" + fi + [ -z "$why" ] || break + if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then + log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago" + last_beat=$now + fi + done + if [ -n "$why" ]; then + log "watchdog: $phase $why; killing it. Last output:" + tail -n 40 "$out" >&2 + kill -TERM -- "-$STREAM_PID" 2>/dev/null || kill "$STREAM_PID" 2>/dev/null || true + wait "$STREAM_PID" 2>/dev/null || true + STREAM_PID="" + # A phase that finished in the instant before the kill keeps its exit 0. + if [ "$(phase_status "$name" "$phase" 2>/dev/null)" = "exit 0" ]; then + log "watchdog: $phase finished before it could be killed; its exit 0 stands" + rc=0 + else + remote_cancel "$name" "$p.pid" + rc=124 + fi + break + fi + wait "$STREAM_PID" 2>/dev/null || true + STREAM_PID="" + st=$(phase_status "$name" "$phase") || st="" + case $st in + "exit "*) rc=${st#exit }; break ;; + lost|none) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; rc=125; break ;; + esac + tries=$(( tries + 1 )) + if [ "$tries" -ge 5 ]; then log "lost the connection to $phase on $name 5 times; giving up (the VM keeps it until teardown)"; rc=255; break; fi + log "the stream of $phase on $name dropped; following it again" + sleep "$POLL" + oo=$(fsize "$lout") oe=$(fsize "$lerr") + done + rm -f "$dir/stream" + trap - INT TERM; eval "$old_traps" + return "$rc" +} + +# write_record DIR: the run record a later run of the same command reattaches +# by (key, tree, and this invocation for `attach NAME`). Remote files of each +# phase: ~/.ubirun-.{out,err,exit,pid} (setup, command, tier-). +qwords() { local x; for x in "$@"; do printf '%q ' "$x"; done; } +write_record() { + { + printf 'RUN_KEY=%q\nTREE=%q\nPULL_ROOT=%q\nRESUME=%q\nRERUN_FAILED=%q\n' "$RUN_KEY" "$TREE" "$PULL_ROOT" "$RESUME" "$RERUN_FAILED" + printf 'RUN_ARGS=(%s)\n' "$(qwords "${RUN_ARGS[@]}")" + printf 'TIER_NAMES=(%s)\n' "$(qwords ${TIER_NAMES[@]+"${TIER_NAMES[@]}"})" + printf 'TIER_CMDS=(%s)\n' "$(qwords ${TIER_CMDS[@]+"${TIER_CMDS[@]}"})" + echo "# remote files: ~/.ubirun-.{out,err,exit,pid}; phase starts: phase..start; lease: lease (pid start time)" + } >"$1/run.$$" && mv "$1/run.$$" "$1/run" +} + +lease_take() { echo "$$ $(proc_start $$)" >"$1/lease.$$" && mv "$1/lease.$$" "$1/lease"; } +lease_busy() { + local pid st + [ -f "$1/lease" ] || return 1 + read -r pid st <"$1/lease" || true + [ -n "$pid" ] && [ "$pid" != "$$" ] && kill -0 "$pid" 2>/dev/null && [ "$(proc_start "$pid")" = "$st" ] +} + +# vm_alive NAME: the recorded VM still answers show. One confirmed gone is +# cleaned up (state dir and scratch record), so its record is never tried again. +vm_alive() { + local n=$1 out code + ( load "$n"; [ "$CREATED" = yes ] && [ -n "$IP" ] ) 2>/dev/null || return 1 + out=$(mktemp) + code=$(load "$n"; api "$out" vm "$LOCATION/$n" show) + rm -f "$out" + case $code in + 200) return 0 ;; + 404) log "run record for $n: the VM is gone; starting a new one"; cmd_down "$n" >/dev/null 2>&1 || true; return 1 ;; + *) return 1 ;; + esac +} + +# ttl_short NAME SETUP: true when NAME's TTL (its created_at plus the name's +# -t, minus now) does not cover the run still to do plus +# REATTACH_MARGIN, with the numbers in TTL_WHY. The run still to do is the +# UBI_RUN_TIMEOUT bound of each phase that has not finished (setup when SETUP is +# set, then the command or every tier the plan does not reuse), counted from a +# started phase's recorded start; it is 0 when UBI_RUN_TIMEOUT is 0. +TTL_WHY="" +ttl_short() { + local name=$1 dir created left now need=0 p i phases=() + [[ "$name" =~ -t([0-9]{1,3})$ ]] || return 1 + dir=$(state_dir "$name") + created=$(load "$name"; echo "$CREATE_AT") + now=$(date +%s) + left=$(( created + 10#${BASH_REMATCH[1]} * 3600 - now )) + [ -z "$2" ] || phases+=(setup) + if [ ${#TIER_NAMES[@]} -gt 0 ]; then + for i in "${!TIER_NAMES[@]}"; do [ "${TIER_ACT[i]}" = reuse ] || phases+=("tier-${TIER_NAMES[i]}"); done + else + phases+=(command) + fi + if [ "$RUN_TIMEOUT" -gt 0 ]; then + for p in "${phases[@]}"; do + if [ -f "$dir/phase.$p.start" ]; then i=$(( RUN_TIMEOUT - (now - $(cat "$dir/phase.$p.start")) )); [ "$i" -gt 0 ] || i=0 + else i=$RUN_TIMEOUT; fi + need=$(( need + i )) + done + fi + TTL_WHY="ttl_left=${left}s needs=$(( need + REATTACH_MARGIN ))s (run still to do ${need}s + UBI_REATTACH_MARGIN=${REATTACH_MARGIN}s)" + [ "$left" -lt $(( need + REATTACH_MARGIN )) ] +} + +# find_record: the VM of a run record with this RUN_KEY whose VM still exists. +find_record() { + local f n + for f in "$STATE_ROOT"/*/run; do + [ -f "$f" ] || continue + grep -qx "RUN_KEY=$RUN_KEY" "$f" || continue + n=$(basename "$(dirname "$f")") + if vm_alive "$n"; then echo "$n"; return 0; fi + done +} + +# Tier checkpoints: write-once records ...json beside a copy +# of the tier's log, under CHECKPOINT_DIR//, published under a lock +# with a monotonic generation; the newest generation per tier decides. +CKPT_PY=' +import fcntl, hashlib, json, os, shutil, sys, time +op, d = sys.argv[1], sys.argv[2] +def records(): + out = [] + for f in (os.listdir(d) if os.path.isdir(d) else []): + if not f.endswith(".json") or f.startswith("."): + continue + try: + r = json.load(open(os.path.join(d, f))) + except (OSError, ValueError): + continue + if isinstance(r, dict) and isinstance(r.get("generation"), int): + r["_file"] = f + out.append(r) + return out +def newest(rs, tier): + rs = [r for r in rs if r.get("tier") == tier] + return max(rs, key=lambda r: r["generation"]) if rs else None +def sha(p): + h = hashlib.sha256() + with open(p, "rb") as fh: + for b in iter(lambda: fh.read(65536), b""): + h.update(b) + return h.hexdigest() +if op == "decide": + now, maxage, resume, rerun = int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", sys.argv[6] == "1" + rs = records() + for tier in sys.argv[7:]: + r = newest(rs, tier) + f = r["_file"] if r else "-" + age = now - int(r.get("finished_at") or r.get("started_at") or now) if r else 0 + dur = int(r.get("duration_s") or 0) if r else 0 + code = str(r.get("exit", "-")) if r else "-" + log = os.path.join(d, str(r.get("log"))) if r and r.get("log") else "" + if not resume: + act, why = "run", "--no-resume" + elif r is None: + act, why = "run", "no record" + elif r.get("status") == "start": + act, why = "run", "interrupted (only a start record, %s)" % f + elif r.get("status") == "pass": + if age >= maxage: + act, why = "run", "pass %s is %ds old (UBI_CHECKPOINT_MAX_AGE=%d)" % (f, age, maxage) + elif not log or not os.path.isfile(log) or sha(log) != r.get("log_sha256"): + act, why = "run", "pass %s: its log does not re-hash" % f + else: + act, why = "reuse", "-" + elif r.get("status") == "fail": + act, why = ("rerun", "prior fail %s (exit %s; --rerun-failed)" % (f, code)) if rerun else ("blocked", "-") + else: + act, why = "run", "unreadable record %s" % f + print("\t".join([tier, act, f, str(age), str(dur), code, why])) +elif op == "publish": + tier, status, vm, started, finished, code, log, startgen, key, tree, pid = sys.argv[3:14] + os.makedirs(d, exist_ok=True) + with open(os.path.join(d, ".lock"), "a") as lk: + fcntl.flock(lk, fcntl.LOCK_EX) + rs = records() + gen = max([r["generation"] for r in rs] + [0]) + 1 + if status != "start": + r = newest(rs, tier) + if r is None or str(r["generation"]) != startgen: + print("stale writer: its start record is generation %s, the newest for %s is %s" % (startgen, tier, r["generation"] if r else "none"), file=sys.stderr) + sys.exit(2) + rec = {"tier": tier, "status": status, "generation": gen, "vm": vm, "key": key, "tree": tree, "started_at": int(started)} + if status != "start": + rec.update(finished_at=int(finished), duration_s=int(finished) - int(started), exit=int(code)) + epoch = int(time.time()) + while os.path.exists(os.path.join(d, "%s.%d.%s.json" % (tier, epoch, pid))) or os.path.exists(os.path.join(d, "%s.%d.%s.log" % (tier, epoch, pid))): + epoch += 1 + stem = "%s.%d.%s" % (tier, epoch, pid) + if status != "start": + shutil.copyfile(log, os.path.join(d, stem + ".log")) + rec["log"], rec["log_sha256"] = stem + ".log", sha(os.path.join(d, stem + ".log")) + tmp = os.path.join(d, ".%s.tmp" % stem) + with open(tmp, "w") as fh: + json.dump(rec, fh, sort_keys=True) + fh.write("\n") + os.link(tmp, os.path.join(d, stem + ".json")) + os.unlink(tmp) + print(gen) +' + +# ckpt_publish TIER STATUS VM STARTED FINISHED EXIT LOG STARTGEN: print the new +# record's generation; fails (nothing published) for a stale writer. +ckpt_publish() { python3 -c "$CKPT_PY" publish "$CHECKPOINT_DIR/$RUN_KEY" "$@" "$RUN_KEY" "$TREE" "$$"; } + +TIER_ACT=() TIER_TO_RUN=0 TIER_PASSED=0 TIER_REUSED="" TIER_FAILED="" TIER_PRIOR="" TIER_SAVED=0 TIER_BLOCKED="" TIER_BLOCKED_RC=1 + +# tier_plan: decide each tier from its newest checkpoint and print one +# UBI_VERIFY_RESUME line per tier before anything runs. +tier_plan() { + local plan t act f age dur code why + plan=$(python3 -c "$CKPT_PY" decide "$CHECKPOINT_DIR/$RUN_KEY" "$(date +%s)" "$CHECKPOINT_MAX_AGE" "$RESUME" "$RERUN_FAILED" "${TIER_NAMES[@]}") \ + || die "verify: could not read the tier checkpoints in $CHECKPOINT_DIR/$RUN_KEY" + while IFS=$'\t' read -r t act f age dur code why; do + case $act in + reuse) + echo "UBI_VERIFY_RESUME: tier=$t reuse $f (${age}s)" >&2 + TIER_REUSED=${TIER_REUSED:+$TIER_REUSED,}$t TIER_SAVED=$(( TIER_SAVED + dur )) TIER_PASSED=$(( TIER_PASSED + 1 )) ;; + blocked) + echo "UBI_VERIFY_RESUME: tier=$t fail $f (${age}s, exit $code): not rerun without --rerun-failed" >&2 + if [ -z "$TIER_BLOCKED" ]; then TIER_BLOCKED=$t; if [[ "$code" =~ ^[0-9]+$ ]]; then TIER_BLOCKED_RC=$code; fi; fi ;; + rerun) + echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2 + TIER_PRIOR=${TIER_PRIOR:+$TIER_PRIOR,}$t:$code TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;; + *) + echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2 + TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;; + esac + TIER_ACT+=("$act") + done <<<"$plan" +} + +verify_line() { + echo "UBI_VERIFY: tree=${TREE:0:12} key=${RUN_KEY:0:12} tiers=${#TIER_NAMES[@]} passed=$TIER_PASSED reused=${TIER_REUSED:-none} failed=${TIER_FAILED:-none} prior_failures=${TIER_PRIOR:-none} saved_s=$TIER_SAVED" +} + +# run_tiers NAME QDEST: run every tier the plan did not reuse, in order, each +# as its own detached phase; stop at the first failure and return its status. +# A pass is recorded only when the VM's tree after the tier equals the tree +# before it. +run_tiers() { + local name=$1 qdest=$2 dir i t phase before after trc started finished gen + dir=$(state_dir "$name") + for i in "${!TIER_NAMES[@]}"; do + [ "${TIER_ACT[i]}" != reuse ] || continue + t=${TIER_NAMES[i]} phase="tier-${TIER_NAMES[i]}" + if [ ! -f "$dir/phase.$phase.start" ]; then + before=$(vm_tree "$name" "$qdest") || before=unavailable + echo "$before" >"$dir/phase.$phase.tree" + if gen=$(ckpt_publish "$t" start "$name" "$(date +%s)" 0 0 "" 0); then echo "$gen" >"$dir/phase.$phase.gen" + else echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t start record (the tier still runs)" >&2; fi + fi + log "running tier $t on $name: ${TIER_CMDS[i]}" + trc=0 + detached "$name" "$phase" "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "${TIER_CMDS[i]}")" || trc=$? + started=$(cat "$dir/phase.$phase.start" 2>/dev/null || date +%s) finished=$(date +%s) + log "tier $t exited $trc after $(( finished - started ))s" + case $trc in 125|255) TIER_FAILED=$t; return "$trc" ;; esac + gen=$(cat "$dir/phase.$phase.gen" 2>/dev/null || echo 0) + if [ "$trc" -ne 0 ]; then + ckpt_publish "$t" fail "$name" "$started" "$finished" "$trc" "$dir/$phase.log" "$gen" >/dev/null \ + || echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t fail record (the result stands)" >&2 + TIER_FAILED=$t + return "$trc" + fi + TIER_PASSED=$(( TIER_PASSED + 1 )) + after=$(vm_tree "$name" "$qdest") || after=unavailable + before=$(cat "$dir/phase.$phase.tree" 2>/dev/null || echo unavailable) + if [ "$after" != "$before" ] || [ "$after" = unavailable ]; then + echo "UBI_VERIFY_TREE_MOVED: tier=$t before=${before:0:12} after=${after:0:12}: a tier must leave the source tree as it found it; no pass record" >&2 + continue + fi + ckpt_publish "$t" pass "$name" "$started" "$finished" 0 "$dir/$phase.log" "$gen" >/dev/null \ + || echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t pass record (the result stands)" >&2 + done +} + # vm_rows: "location name owner expires created" for every VM in the project; # owner is "-" for untagged ubirun--* names and "." for VMs this runner # did not name; expires is the epoch the name's -t TTL ends at, or "-"; @@ -770,9 +1207,11 @@ run_teardown() { [ "$rc" -ne 0 ] || exit 1 } +TIER_NAMES=() TIER_CMDS=() RESUME=1 RERUN_FAILED=0 FORCE_ATTACH="" RUN_ARGS=() cmd_run() { require_owner - local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() + local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() passes=() key_inputs=() probe=0 + RUN_ARGS=("$@") while [ $# -gt 0 ]; do case $1 in -S|--storage) check_storage "$2"; up_args+=("$1" "$2"); shift 2 ;; @@ -781,60 +1220,161 @@ cmd_run() { --dest) dest=$2; shift 2 ;; --setup) setup=$2; shift 2 ;; --env) envs+=("$2"); shift 2 ;; - --pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; envs+=("$2=${!2}"); shift 2 ;; + --pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; passes+=("$2"); shift 2 ;; --pull) pulls+=("$2"); shift 2 ;; --keep) keep=1; shift ;; + --key-input) key_inputs+=("$2"); shift 2 ;; + --probe) probe=1; shift ;; + --resume) RESUME=1; shift ;; + --no-resume) RESUME=0; shift ;; + --rerun-failed) RERUN_FAILED=1; shift ;; --) shift; break ;; *) die "run: unknown option $1" ;; esac done - [ $# -gt 0 ] || die "run: missing command after --" + [ $# -gt 0 ] || [ ${#TIER_NAMES[@]} -gt 0 ] || die "run: missing command after --" [ -z "$setup" ] || [ -f "$setup" ] || die "setup script not found: $setup" + command -v git >/dev/null || die "git is required: the run key fingerprints the checkout with it" src=$(cd "$src" && pwd) [ -n "$dest" ] || dest="work/$(basename "$src")" - local i name="" + local i name="" size=$DEFAULT_SIZE location=$DEFAULT_LOCATION image=ubuntu-noble storage="" e n for (( i = 0; i < ${#up_args[@]}; i += 2 )); do - case ${up_args[i]} in -n|--name) name=${up_args[i+1]} ;; esac + case ${up_args[i]} in + -n|--name) name=${up_args[i+1]} ;; + -s|--size) size=${up_args[i+1]} ;; + -l|--location) location=${up_args[i+1]} ;; + -b|--image) image=${up_args[i+1]} ;; + -S|--storage) storage=${up_args[i+1]} ;; + esac done + # The run key: everything that decides what the VM runs, computed before any + # VM exists. Pass values enter only as their sha256. + if [ -z "$FORCE_ATTACH" ]; then + TREE=$(cd "$src" && bash -c "$TREE_SH") || die "could not fingerprint $src (git add -A into a temporary index failed)" + RUN_KEY=$({ + echo "ubi-runner run key v1" + echo "tree $TREE" + printf 'command %q\n' "$*" + if [ -n "$setup" ]; then echo "setup $(sha256 <"$setup")"; else echo "setup none"; fi + for e in ${envs[@]+"${envs[@]}"}; do printf 'env %q\n' "$e"; done + for n in ${passes[@]+"${passes[@]}"}; do printf 'pass %s %s\n' "$n" "$(printf %s "${!n}" | sha256)"; done + printf 'vm %q %q %q %q\n' "$size" "$location" "$image" "$storage" + echo "base $(base_sha "$src")" + echo "runner $(sha256 <"${BASH_SOURCE[0]}")" + for e in ${key_inputs[@]+"${key_inputs[@]}"}; do printf 'input %q\n' "$e"; done + for i in "${!TIER_NAMES[@]}"; do printf 'tier %q %q\n' "${TIER_NAMES[i]}" "${TIER_CMDS[i]}"; done + } | sha256) + fi + [ ${#TIER_NAMES[@]} -eq 0 ] || tier_plan + + local attach="" + if [ -n "$FORCE_ATTACH" ]; then + vm_alive "$FORCE_ATTACH" || die "attach: $FORCE_ATTACH no longer exists; rerun the original command to start fresh" + attach=$FORCE_ATTACH + else + attach=$(find_record) + fi + if [ "$probe" = 1 ]; then + if [ -z "$attach" ]; then echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12}" + else + local busy=0 + ! lease_busy "$(state_dir "$attach")" || busy=1 + if [ "$busy" = 0 ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then + echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12} ttl_short vm=$attach $TTL_WHY" + else + echo "UBI_ATTACH_INFO: vm=$attach created_at=$(load "$attach"; echo "$CREATE_AT") key=${RUN_KEY:0:12} busy=$busy" + fi + fi + return 0 + fi + if [ ${#TIER_NAMES[@]} -gt 0 ] && { [ -n "$TIER_BLOCKED" ] || [ "$TIER_TO_RUN" -eq 0 ]; } && [ -n "$attach" ] && ! lease_busy "$(state_dir "$attach")"; then + log "verify: no tier needs $attach, which a dead runner left behind; destroying it" + cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach" + fi + if [ ${#TIER_NAMES[@]} -gt 0 ] && [ -n "$TIER_BLOCKED" ]; then + TIER_FAILED=$TIER_BLOCKED + verify_line + return "$TIER_BLOCKED_RC" + fi + if [ ${#TIER_NAMES[@]} -gt 0 ] && [ "$TIER_TO_RUN" -eq 0 ]; then + log "verify: every tier passed under this key; no VM needed" + verify_line + return 0 + fi + if [ -n "$attach" ] && lease_busy "$(state_dir "$attach")"; then + die "UBI_ATTACH_BUSY: vm=$attach key=${RUN_KEY:0:12}: runner pid $(cut -d' ' -f1 <"$(state_dir "$attach")/lease") on this machine still owns it; wait for it or stop it, then rerun" + fi + if [ -n "$attach" ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then + log "UBI_REATTACH_TTL_SHORT: vm=$attach key=${RUN_KEY:0:12} $TTL_WHY: starting a fresh VM (tier checkpoints carry over) and destroying $attach" + cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach" + attach="" + fi + [ -z "$attach" ] || name=$attach [ -n "$name" ] || { name=$(new_name); up_args+=(-n "$name"); } RUN_VM=$name # Armed before the create request: an interrupted `up` still destroys its VM. # A second signal must not cut teardown short, so the EXIT handler ignores them. trap 'run_teardown "$?"' EXIT - trap 'exit 130' INT TERM HUP QUIT - cmd_up "${up_args[@]}" >/dev/null - if [[ "$name" =~ -t([0-9]+)$ ]]; then - log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name" - elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then - log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name" + trap 'exit 130' INT TERM QUIT + trap on_hup HUP + if [ -n "$attach" ]; then + load "$name" + log "reattached to $name created_at=$CREATE_AT key=${RUN_KEY:0:12}" else - log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name" + cmd_up "${up_args[@]}" >/dev/null + trap 'exit 130' INT TERM QUIT + trap on_hup HUP + if [[ "$name" =~ -t([0-9]+)$ ]]; then + log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name" + elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then + log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name" + else + log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name" + fi fi + local dir + dir=$(state_dir "$name") + write_record "$dir" + lease_take "$dir" if [ "$keep" = 1 ]; then - trap - EXIT + # A kept VM's finished run is not reattached by the next run; a detach (HUP) keeps the record. + trap 'rm -f "$(state_dir "$RUN_VM")/run"' EXIT leftover "left=1 owner=$OWNER vm=$name kept=1" log "--keep: leaving $name running; destroy with: $0 down $name" fi - local e - for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done \ - | remote "$name" 'umask 077 && cat > ~/.ubirun-env' - - cmd_sync "$name" "$src" "$dest" - local qdest rc=0 + local qdest rc=0 vmt qdest=$(printf %q "$dest") + if [ ! -f "$dir/synced" ]; then + { for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done + for n in ${passes[@]+"${passes[@]}"}; do printf 'export %s=%q\n' "$n" "${!n}"; done; } \ + | remote "$name" 'umask 077 && cat > ~/.ubirun-env' + cmd_sync "$name" "$src" "$dest" + # A subdirectory of a checkout uploads the whole .git with only its own + # files, so its tree is compared VM to VM (tiers) but not to this side. + if [ -z "$(git -C "$src" rev-parse --show-prefix 2>/dev/null)" ]; then + vmt=$(vm_tree "$name" "$qdest") || vmt=unavailable + if [ "$vmt" != "$TREE" ]; then + log "UBI_VM_TREE_MISMATCH: vm=$name tree=${vmt:0:12} expected=${TREE:0:12}: the unpacked checkout is not the one this run was keyed on" + exit 3 + fi + fi + [ -z "$setup" ] || remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup" + : >"$dir/synced" + fi if [ -n "$setup" ]; then - remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup" log "running setup $(basename "$setup")" - watched "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; } + detached "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; } fi - local start - start=$(date +%s) - log "running on $name: $*" - watched "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$? - log "command exited $rc after $(( $(date +%s) - start ))s" + if [ ${#TIER_NAMES[@]} -gt 0 ]; then + run_tiers "$name" "$qdest" || rc=$? + else + log "running on $name: $*" + detached "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$? + log "command exited $rc after $(( $(date +%s) - $(cat "$dir/phase.command.start" 2>/dev/null || date +%s) ))s" + fi local p for p in ${pulls[@]+"${pulls[@]}"}; do @@ -845,9 +1385,23 @@ cmd_run() { fi cmd_pull "$name" "${p%%:*}" "${p#*:}" || log "pull failed: $p" done + [ ${#TIER_NAMES[@]} -eq 0 ] || verify_line return "$rc" } +# attach NAME: resume the run recorded for NAME on this machine (the same +# invocation, key and tiers), as a rerun of the same command would. +cmd_attach() { + local name=${1:-} + [ -n "$name" ] || die "attach: missing NAME" + valid_name "$name" || die "invalid VM name '$name'" + [ -f "$(state_dir "$name")/run" ] || die "attach: no run record for $name on this machine ($(state_dir "$name")/run); a machine replacement loses it" + # shellcheck disable=SC1091 + . "$(state_dir "$name")/run" + FORCE_ATTACH=$name + cmd_run "${RUN_ARGS[@]}" +} + # verify [run opts] [--trust-config] [-- COMMAND]: run the repository's # committed remote verify. .gstack/ubicloud.json under --src (default .) # supplies setup, command, size, location, storage, image, env, pass and pull; @@ -860,7 +1414,7 @@ cmd_run() { # local variable it lists under pass is forwarded only when this command line # also names it (--pass NAME) or grants every listed name (--trust-config). cmd_verify() { - local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() + local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() cfg_tier_names=() cfg_tier_cmds=() for (( i = 1; i <= $#; i++ )); do [ "${!i}" = -- ] && break if [ "${!i}" = --src ]; then j=$(( i + 1 )); src=${!j}; fi @@ -869,7 +1423,7 @@ cmd_verify() { [ -f "$cfg" ] || die "verify: no $cfg; commit one ({\"setup\": \"scripts/ubicloud/bootstrap.sh\", \"command\": \"bash scripts/ubicloud/verify.sh\"}) so every lane runs the same verify instead of its own wrapper" err=$(mktemp) parsed=$(python3 -c ' -import json, os, shlex, sys +import json, os, re, shlex, sys cfg, src = sys.argv[1], sys.argv[2] try: c = json.load(open(cfg)) @@ -895,7 +1449,25 @@ for item in c.get("pull", []): if not sep or not remote: print("pull entry %r must be REMOTE_GLOB:LOCAL_DIR" % (item,), file=sys.stderr); sys.exit(2) args += ["--pull", q(remote + ":" + inside(local, "pull destination"))] +tiers = c.get("tiers") +names = [] +if tiers is not None: + if "command" in c: + print("UBI_CONFIG_TIERS_AND_COMMAND: give either command or tiers, not both", file=sys.stderr); sys.exit(2) + if not isinstance(tiers, list) or not tiers: + print("UBI_CONFIG_TIER_NAME: tiers must be a non-empty list of {\"name\", \"command\"}", file=sys.stderr); sys.exit(2) + for t in tiers: + name = t.get("name") if isinstance(t, dict) else None + if not isinstance(name, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,31}", name): + print("UBI_CONFIG_TIER_NAME: tier name %r must match ^[a-z0-9][a-z0-9_-]{0,31}$" % (name,), file=sys.stderr); sys.exit(2) + if name in names: + print("UBI_CONFIG_TIER_NAME: tier name %r appears twice" % name, file=sys.stderr); sys.exit(2) + if not isinstance(t.get("command"), str) or not t["command"].strip(): + print("UBI_CONFIG_TIER_NAME: tier %r needs a command" % name, file=sys.stderr); sys.exit(2) + names.append(name) setup = c.get("setup", "") +print("cfg_tier_names=(" + " ".join(q(n) for n in names) + ")") +print("cfg_tier_cmds=(" + " ".join(q(t["command"]) for t in (tiers or [])) + ")") print("cfg_args=(" + " ".join(args) + ")") print("cfg_pass=(" + " ".join(q(x) for x in c.get("pass", [])) + ")") print("cfg_setup=" + q(inside(setup, "setup") if setup else "")) @@ -912,7 +1484,7 @@ print("cfg_cmd=" + q(c.get("command", ""))) case $1 in --) shift; user_cmd=("$@"); break ;; --trust-config) trust=1; shift ;; - --keep) user_pre+=("$1"); shift ;; + --keep|--resume|--no-resume|--rerun-failed|--probe) user_pre+=("$1"); shift ;; --pass|-S|--storage|-s|--size|-l|--location|-n|--name|-b|--image|--src|--dest|--setup|--env|--pull) [ $# -ge 2 ] || die "verify: $1 needs a value" [ "$1" != --pass ] || granted+=("$2") @@ -927,11 +1499,16 @@ print("cfg_cmd=" + q(c.get("command", ""))) missing+=("$n") done [ ${#missing[@]} -eq 0 ] || die "verify: $cfg asks to forward local ${missing[*]}; grant each on this command line with --pass NAME, or every name it lists with --trust-config" - [ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --" - [ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd") + # A trailing -- COMMAND replaces the configured tiers as it replaces command. + if [ ${#user_cmd[@]} -eq 0 ] && [ ${#cfg_tier_names[@]} -gt 0 ]; then + TIER_NAMES=("${cfg_tier_names[@]}") TIER_CMDS=("${cfg_tier_cmds[@]}") + else + [ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --" + [ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd") + fi log "verify: $cfg" PULL_ROOT=$(cd "$src" && pwd) - cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- "${user_cmd[@]}" + cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- ${user_cmd[@]+"${user_cmd[@]}"} } usage() { @@ -939,13 +1516,17 @@ usage() { usage: ubi-runner.sh [args] run [up opts] [--src DIR] [--dest PATH] [--setup FILE] [--env K=V]... - [--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] -- COMMAND - up + sync + setup + command + pull + destroy; exits with COMMAND's status - verify [run opts] [--trust-config] [-- COMMAND] + [--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] [--key-input S]... [--probe] -- COMMAND + up + sync + setup + command + pull + destroy; exits with COMMAND's status. + A rerun of the same command (same run key) reattaches to its live VM; + --probe prints UBI_ATTACH_INFO (and the tier plan) and exits + verify [run opts] [--trust-config] [--resume|--no-resume] [--rerun-failed] [-- COMMAND] run the repository's committed verify (.gstack/ubicloud.json under --src); its setup script and every pull destination stay inside --src (checked again at pull time), and its pass names are forwarded only when granted - here (--pass NAME, or all: --trust-config) + here (--pass NAME, or all: --trust-config). With tiers, a pass checkpoint + under the same run key is reused (resume is the default) + attach NAME resume the run recorded for NAME on this machine, as a rerun would up [-s SIZE] [-l LOCATION] [-n NAME] [-S STORAGE_GIB] [-b IMAGE] create a VM and wait for SSH; prints its name (STORAGE_GIB: one of $ALLOWED_STORAGE) @@ -965,6 +1546,11 @@ usage: ubi-runner.sh [args] owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list) +signals: INT and TERM tear the VM down. HUP, once a remote phase is live, detaches +(UBI_DETACHED, exit 129; the VM runs on under its TTL). SIGKILL or a machine stop leaves +the run record and the VM; rerun the same command to reattach (UBI_ATTACH_BUSY while +another runner owns it; exit 125 UBI_ATTACH_LOST when the remote session left no status). + VMs are named $PREFIX----t; up, run and verify need UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left= line. defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION) @@ -976,7 +1562,8 @@ env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing), UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s), UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file), - UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640") + UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"), + UBI_CHECKPOINT_DIR (tier checkpoints; $CHECKPOINT_DIR), UBI_CHECKPOINT_MAX_AGE (${CHECKPOINT_MAX_AGE}s a pass is reused) EOF } @@ -989,6 +1576,7 @@ esac case $cmd in run) cmd_run "$@" ;; verify) cmd_verify "$@" ;; + attach) cmd_attach "$@" ;; up) trap 'trap "" INT TERM HUP QUIT; [ -z "$UP_NAME" ] || [ -n "$UP_READY" ] || cmd_down "$UP_NAME" || log "WARNING: failed to destroy $UP_NAME; run: $0 down $UP_NAME"' EXIT trap 'exit 130' INT TERM HUP QUIT cmd_up "$@" ;; diff --git a/test/ubicloud-pregate-remote.test.ts b/test/ubicloud-pregate-remote.test.ts index 852170405..2263b617a 100644 --- a/test/ubicloud-pregate-remote.test.ts +++ b/test/ubicloud-pregate-remote.test.ts @@ -37,12 +37,22 @@ const UBI_RUNNER_STUB = `#!/usr/bin/env bash # VM directory, runs setup and the command there, and prints ubi-runner's log # lines. STUB_FAIL picks a failure: create, quota, quota-once, ssh. STUB_HANG # sleeps before the command; TERM tears down as ubi-runner's trap does. -# Every teardown is appended to $STUB_CALLS.down. +# Every teardown is appended to $STUB_CALLS.down. STUB_REATTACH=: +# plays a live run record for this key: --probe reports it (busy per +# STUB_REATTACH_BUSY) and run prints ubi-runner's reattached line instead of +# creating a VM. set -uo pipefail [ "$1" != owner ] || { echo stub; exit 0; } left() { [ "\${STUB_LEFT:-0}" = none ] || echo "UBICLOUD: left=\${STUB_LEFT:-0} owner=stub vm=$name others=0" | tee -a "\${UBI_LEFT_FILE:-/dev/null}" >&2; } [ "$1" = run ] || exit 2 shift +case " $* " in + *" --probe "*) + printf '%s\\n' "$*" >>"$STUB_CALLS.probe" + if [ -n "\${STUB_REATTACH:-}" ]; then echo "UBI_ATTACH_INFO: vm=\${STUB_REATTACH%%:*} created_at=\${STUB_REATTACH#*:} key=0123456789ab busy=\${STUB_REATTACH_BUSY:-0}" + else echo "UBI_ATTACH_INFO: none key=0123456789ab"; fi + exit 0 ;; +esac printf '%s\\n' "$*" >>"$STUB_CALLS" src=. setup="" size="" name=unnamed envs=() while [ $# -gt 0 ]; do @@ -52,16 +62,20 @@ while [ $# -gt 0 ]; do --src) src=$2; shift 2 ;; --setup) setup=$2; shift 2 ;; --env) envs+=("$2"); shift 2 ;; - --pull) shift 2 ;; + --pull|--key-input) shift 2 ;; --) shift; break ;; *) echo "stub: unknown option $1" >&2; exit 2 ;; esac done cmd=$* log() { echo "ubi-runner: $*" >&2; } -log "creating eu-central-h1/$name ($size)" calls=$(wc -l <"$STUB_CALLS") fail=\${STUB_FAIL:-} +if [ -n "\${STUB_REATTACH:-}" ]; then + name=\${STUB_REATTACH%%:*} + log "reattached to $name created_at=\${STUB_REATTACH#*:} key=0123456789ab" +else +log "creating eu-central-h1/$name ($size)" if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; then log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64" log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)" @@ -77,6 +91,7 @@ if [ "$fail" = create ]; then exit 1 fi log "ready: $name (192.0.2.1)" +fi trap 'log "destroyed eu-central-h1/$name"; left; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM vm=$STUB_VM_HOME dest="$vm/work/$(basename "$src")" @@ -282,6 +297,36 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => { expect(twice.out).toContain('create failed: vCPU quota exhausted'); }, 90_000); + test('a rerun for the same tree and planned set reattaches: the receipt still parses, spent= bills from the original created_at, and a used-up wall budget refuses', () => { + const created = Math.floor(Date.now() / 1000) - 5 * 60 - 10; + const f = fixture(); + const r = gate(f, ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}` }); + expect(r.exit, r.out).toBe(0); + expect(r.out).toContain('reattaching to VM ubirun-stub-old, created 6 min ago by an earlier run of this gate for the same tree and planned set'); + expect(r.out).toContain("the VM is told to stop after 2160s (the cap's 45 min wall minus 3 min for teardown and 6 min already billed since ubirun-stub-old was created)"); + expect(r.out).not.toContain('if this run is killed before its teardown line'); + expect(r.out.trim().split('\n').at(-1)).toMatch(/^PREGATE_REMOTE: /); + expect(r.report).toMatchObject({ tree: f.tree, tests: 1, failed: 0 }); + // Billed from created_at (5 min 10 s before the gate started), not from the reattach: 6 min at $0.00831. + expect(r.report!.spent).toBe('0.05'); + expect(remoteReportProblem(r.report!, f.tree)).toBeNull(); + const plan = new RegExp(`--key-input pregate-remote tree=${f.tree} plan=[0-9a-f]{64}`); + expect(r.calls[0]).toMatch(plan); + expect(fs.readFileSync(f.calls + '.probe', 'utf8')).toMatch(plan); + + const spent = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created - 40 * 60}` }); + expect(spent.exit, spent.out).toBe(5); + expect(spent.out).toContain("PREGATE_REMOTE_CAP_SPENT: VM ubirun-stub-old, created 46 min ago for this tree and planned set, has already used the cap's 45 min wall ($0.39 of $2)"); + expect(spent.out).toContain('nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down ubirun-stub-old'); + expect(spent.report).toBeNull(); + expect(spent.calls).toEqual([]); + + const busy = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}`, STUB_REATTACH_BUSY: '1' }); + expect(busy.exit, busy.out).toBe(4); + expect(busy.out).toContain('PREGATE_REMOTE_ATTACH_BUSY: another gate on this machine still runs VM ubirun-stub-old'); + expect(busy.calls).toEqual([]); + }, 90_000); + test('the VM refuses a tree that differs after unpack, a planned set that differs from the list, and a tree that moved during the run', () => { const edited = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_EDIT: 'test/alpha.test.ts' }); expect(edited.exit, edited.out).toBe(3); diff --git a/test/ubicloud-runner.test.ts b/test/ubicloud-runner.test.ts index aa19c5372..fd6580884 100644 --- a/test/ubicloud-runner.test.ts +++ b/test/ubicloud-runner.test.ts @@ -389,18 +389,23 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( /** * Fake Ubicloud API that can create: create adds the name to the listed file * (or answers the quota refusal in refuse with HTTP 400), show is running at - * 192.0.2.1 while listed, destroy unlists. The fake ssh records every remote - * command in ssh.log and plays the role the command asks for: it stores - * ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, drains tar - * uploads, runs the setup phase per setup-mode (ok or hang) and echoes the - * command phase. No network, no VM. + * 192.0.2.1 while listed, destroy unlists. The fake VM is the directory vm/: + * the fake ssh records every remote command in ssh.log, keeps copies of + * ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, serves pulls + * from remote-fs/, and runs every other command there for real (HOME=vm/, + * vmbin/ first on PATH), so uploads unpack, detached phases write their pid, + * output and exit files, tail --pid follows them and pkill ends them. The stub + * make in vmbin/ echoes `ran: make ARGS`, or runs command-mode or + * make-.sh from the fixture root. No network, no VM. */ function fakeVm(opts: { listed?: string; refuse?: string } = {}) { const root = mkdtempSync(join(tmpdir(), 'ubi-run-')); const bin = join(root, 'bin'), state = join(root, 'state'), log = join(root, 'calls.log'), sshLog = join(root, 'ssh.log'), record = join(root, 'vms.tsv'); + const vm = join(root, 'vm'), vmbin = join(root, 'vmbin'), srcDir = join(root, 'src'), checkpoints = join(root, 'checkpoints'); + mkdirSync(vm); mkdirSync(vmbin); mkdirSync(srcDir); + writeFileSync(join(srcDir, 'README'), 'fixture checkout\n'); mkdirSync(bin); mkdirSync(state, { recursive: true }); writeFileSync(join(root, 'listed'), opts.listed ?? ''); - writeFileSync(join(root, 'setup-mode'), 'ok'); writeFileSync(join(bin, 'curl'), '#!/usr/bin/env bash\n' + 'out=""; while [ $# -gt 0 ]; do [ "$1" = -o ] && out=$2; shift; done\n' + 'body=$(cat); echo "$body" >> ' + JSON.stringify(log) + '\n' + @@ -415,26 +420,33 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( ' *) : > "$out"; printf 200 ;;\n' + 'esac\n'); writeFileSync(join(bin, 'ssh'), '#!/usr/bin/env bash\n' + - 'root=' + JSON.stringify(root) + '\n' + + 'root=' + JSON.stringify(root) + '; vm="$root/vm"\n' + 'cmd=${*: -1}\n' + 'printf \'%s\\n\' "$cmd" >> "$root/ssh.log"\n' + + 'export HOME="$vm" PATH="$root/vmbin:$PATH"\n' + + 'cd "$vm" || exit 255\n' + 'case "$cmd" in\n' + - ' *pkill*) echo "$cmd" >> "$root/cancel.log" ;;\n' + + ' *pkill*) echo "$cmd" >> "$root/cancel.log"; exec bash -c "$cmd" ;;\n' + ' *captured*) cat > "$root/captured" ;;\n' + - ' *"cat > ~/.ubirun-env"*) cat > "$root/ubirun-env" ;;\n' + - ' *"cat > ~/.ubirun-setup.sh"*) cat > "$root/ubirun-setup.sh" ;;\n' + - ' *"tar -xzf -"*) cat > /dev/null ;;\n' + + ' *"cat > ~/.ubirun-env"*) tee "$root/ubirun-env" > "$vm/.ubirun-env" ;;\n' + + ' *"cat > ~/.ubirun-setup.sh"*) tee "$root/ubirun-setup.sh" > "$vm/.ubirun-setup.sh" ;;\n' + ' *"tar -czf -"*) base=${cmd##* }; tar -czf - -C "$root/remote-fs" "$base" ;;\n' + - ' *"bash -l ~/.ubirun-setup.sh"*) echo "setup start"; if [ "$(cat "$root/setup-mode")" = hang ]; then sleep 20; fi; echo "setup done" ;;\n' + - ' *"bash -lc "*) if [ -f "$root/command-mode" ]; then bash -c "$(cat "$root/command-mode")"; else echo "ran: $cmd"; fi ;;\n' + - ' *) : ;;\n' + + ' *"ls -d -- "*) : ;;\n' + + ' *) exec bash -c "$cmd" ;;\n' + 'esac\n'); - chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755); - const env: Record = { ...(process.env as Record), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0' }; + writeFileSync(join(vmbin, 'make'), '#!/usr/bin/env bash\n' + + 'root=' + JSON.stringify(root) + '\n' + + 'if [ -f "$root/command-mode" ]; then exec bash -c "$(cat "$root/command-mode")"; fi\n' + + 'echo "ran: make $*"\n' + + '[ ! -f "$root/make-$1.sh" ] || exec bash "$root/make-$1.sh"\n'); + chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755); chmodSync(join(vmbin, 'make'), 0o755); + const env: Record = { ...(process.env as Record), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0', UBI_CHECKPOINT_DIR: checkpoints }; const run = (args: string[], extra: Record = {}, stdin?: string) => Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: stdin === undefined ? undefined : Buffer.from(stdin), timeout: 90_000 }); + const spawn = (args: string[], extra: Record = {}) => + Bun.spawn(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: 'ignore', stdout: 'pipe', stderr: 'pipe', timeout: 120_000 }); return { - root, state, record, run, + root, state, record, run, spawn, vm, src: srcDir, checkpoints, calls: () => (existsSync(log) ? readFileSync(log, 'utf8') : ''), ssh: () => (existsSync(sshLog) ? readFileSync(sshLog, 'utf8') : ''), listed: () => readFileSync(join(root, 'listed'), 'utf8'), @@ -455,17 +467,18 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( test('run: a hung setup is killed by the watchdog, exits 124, never reaches the command, and the VM is destroyed', () => { const f = fakeVm(); try { - writeFileSync(join(f.root, 'setup-mode'), 'hang'); - const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo hi\n'); - const r = run(f, ['run', '--src', f.root, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' }); + const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo "setup start"; sleep 20\n'); + const r = run(f, ['run', '--src', f.src, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' }); const err = r.stderr.toString(); expect(r.exitCode, err).toBe(124); expect(err).toMatch(/watchdog: setup printed nothing for \d+s \(UBI_IDLE_TIMEOUT=2\)/); expect(err).toContain('setup failed (exit 124)'); expect(f.ssh()).not.toContain('bash -lc'); - expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/cat ~\/\.ubirun-setup-\d+\.sid/); - expect(f.ssh()).toMatch(/setsid -w bash -c 'printf %s \$\$ >~\/\.ubirun-setup-\d+\.sid; cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh'; else bash -c '[^']*'; fi$/m); - expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/rm -f ~\/\.ubirun-setup-\d+\.sid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/); + expect(err).toContain('setup start'); + // The setup runs detached in its own session: its leader's pid, both streams and the exit status are files on the VM. + expect(f.ssh()).toMatch(/setsid -f bash -c 'echo \$\$ >~\/\.ubirun-setup\.pid\.tmp && mv ~\/\.ubirun-setup\.pid\.tmp ~\/\.ubirun-setup\.pid; \( cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh \) >~\/\.ubirun-setup\.out 2>~\/\.ubirun-setup\.err <\/dev\/null; echo \$\? >~\/\.ubirun-setup\.exit\.tmp && mv ~\/\.ubirun-setup\.exit\.tmp ~\/\.ubirun-setup\.exit' <\/dev\/null/m); + expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/sid=\$\(cat ~\/\.ubirun-setup\.pid 2>\/dev\/null\); rm -f ~\/\.ubirun-setup\.pid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/); + expect(existsSync(join(f.vm, '.ubirun-setup.exit'))).toBe(false); expect(f.calls()).toContain('destroy'); expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3/); expect(f.listed()).toBe(''); @@ -477,7 +490,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( const f = fakeVm(); try { writeFileSync(join(f.root, 'command-mode'), 'for i in 1 2 3 4 5 6 7 8 9 10; do echo "tick $i"; sleep 1; done'); - const r = run(f, ['run', '--src', f.root, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' }); + const r = run(f, ['run', '--src', f.src, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' }); expect(r.exitCode).toBe(124); expect(r.stderr.toString()).toMatch(/watchdog: command ran \d+s \(UBI_RUN_TIMEOUT=3\); killing it/); expect(r.stderr.toString()).toContain('command exited 124 after'); @@ -489,7 +502,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n' }); try { const leftFile = join(f.root, 'left'); - const r = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile }); + const r = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile }); const err = r.stderr.toString(); expect(r.exitCode, err).toBe(0); expect(err).toMatch(/UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/); @@ -499,7 +512,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( const parser = read('scripts/ubicloud/pregate-remote.sh').match(/left=\$\(sed -n '([^']+)' "\$log\.left"/)![1]; expect(Bun.spawnSync(['sed', '-n', parser, leftFile], { timeout: 30_000 }).stdout.toString()).toBe('0\n'); // Value: protects=run --keep reports its live VM as UBICLOUD: left=1 ... kept=1 and never a clean left=0; fails_when=the kept line is dropped or --keep still prints the teardown line; why_new=no test runs run --keep; seam=none; no_claim=real VM lifetime (fake API and ssh) - const kept = f.run(['run', '--src', f.root, '--keep', '--', 'true'], { UBI_REAP: '0' }); + const kept = f.run(['run', '--src', f.src, '--keep', '--', 'true'], { UBI_REAP: '0' }); const keptErr = kept.stderr.toString(); expect(kept.exitCode, keptErr).toBe(0); expect(keptErr).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 kept=1\n/); @@ -512,14 +525,14 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( const f = fakeVm(); try { writeFileSync(join(f.root, 'listed.destroy-fails'), ''); - const leaked = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' }); + const leaked = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' }); const err = leaked.stderr.toString(); expect(leaked.exitCode, err).toBe(1); expect(err).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=(ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3) others=0\n[^]*VM cleanup not confirmed for \1 \(left=1\); destroy it with: \S+ down \1/); expect(f.listed()).toContain('ubirun-thrd1-'); // Value: protects=a failing command whose VM also leaks keeps its own exit status, not 1; fails_when=run_teardown exits 1 whenever left is not 0; why_new=the leak case above only runs a passing command; seam=none; no_claim=real destroy failure modes (fake API returns 500) writeFileSync(join(f.root, 'command-mode'), 'exit 7'); - const failedLeak = f.run(['run', '--src', f.root, '--', 'false'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' }); + const failedLeak = f.run(['run', '--src', f.src, '--', 'make test'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' }); expect(failedLeak.exitCode, failedLeak.stderr.toString()).toBe(7); expect(failedLeak.stderr.toString()).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/); const blind = fakeVm(); @@ -529,7 +542,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( writeFileSync(join(blind.root, 'bin/curl'), '#!/usr/bin/env bash\nif [ -f ' + JSON.stringify(join(blind.root, 'armed')) + ' ]; then touch ' + JSON.stringify(join(blind.root, 'listed.list-fails')) + '; fi\nexec ' + JSON.stringify(join(blind.root, 'bin/curl.real')) + ' "$@"\n'); chmodSync(join(blind.root, 'bin/curl'), 0o755); writeFileSync(join(blind.root, 'command-mode'), 'touch ' + JSON.stringify(join(blind.root, 'armed'))); - const r = blind.run(['run', '--src', blind.root, '--', 'true'], { UBI_REAP: '0' }); + const r = blind.run(['run', '--src', blind.src, '--', 'make test'], { UBI_REAP: '0' }); expect(r.exitCode, r.stderr.toString()).toBe(1); expect(r.stderr.toString()).toMatch(/UBICLOUD: left=unknown owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=unknown\n/); } finally { rmSync(blind.root, { recursive: true, force: true }); } @@ -551,7 +564,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( test('run: --pass forwards a named local variable only; UBICLOUD_* and non-names are refused before any API call', () => { const f = fakeVm(); try { - const ok = run(f, ['run', '--src', f.root, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' }); + const ok = run(f, ['run', '--src', f.src, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' }); expect(ok.exitCode).toBe(0); expect(ok.stderr.toString()).toContain('command exited 0 after'); const envFile = readFileSync(join(f.root, 'ubirun-env'), 'utf8'); @@ -561,7 +574,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( expect(ok.stdout.toString() + ok.stderr.toString()).not.toContain('sekret'); for (const bad of ['UBICLOUD_API_KEY', 'UBICLOUD_API_TOKEN', 'X;Y', 'HOME/..']) { const before = f.calls(); - const r = run(f, ['run', '--src', f.root, '--pass', bad, '--', 'true']); + const r = run(f, ['run', '--src', f.src, '--pass', bad, '--', 'true']); expect(r.exitCode).toBe(1); expect(r.stderr.toString()).toMatch(bad.startsWith('UBICLOUD_') ? /never forwards its own Ubicloud credentials/ : /not a variable name/); expect(f.calls()).toBe(before); @@ -753,4 +766,351 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( } } finally { rmSync(f.root, { recursive: true, force: true }); } }, 30_000); + + // GSTA-158 P3: detached phases, reattach and tier checkpoints. + const until = async (cond: () => boolean, ms: number) => { + const deadline = Date.now() + ms; + while (!cond()) { if (Date.now() > deadline) throw new Error('condition not met in time'); await Bun.sleep(100); } + }; + const vmDirs = (f: ReturnType) => (existsSync(f.state) ? readdirSync(f.state).filter(d => d.startsWith('ubirun-')) : []); + const phaseStarted = (f: ReturnType, phase: string) => vmDirs(f).some(d => existsSync(join(f.state, d, 'phase.' + phase + '.start'))); + const creates = (f: ReturnType) => f.calls().split('\n').filter(l => l.includes('"create"')).length; + const killSession = async (f: ReturnType, phase: string) => { + const pidFile = join(f.vm, '.ubirun-' + phase + '.pid'); + await until(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 10_000); + const pid = Number(readFileSync(pidFile, 'utf8').trim()); + try { process.kill(-pid, 'SIGKILL'); } catch { /* already gone */ } + await until(() => { try { process.kill(pid, 0); return false; } catch { return true; } }, 10_000); + }; + const gitSrc = (f: ReturnType, cfg?: unknown) => { + const dir = cfg === undefined ? join(f.root, 'repo') : src(f, cfg); + mkdirSync(dir, { recursive: true }); + writeFileSync(join(dir, 'README'), 'fixture repo\n'); + for (const args of [['init', '-q', '-b', 'main'], ['add', '-A'], ['commit', '-qm', 'fixture']]) { + const r = Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', ...args], { cwd: dir, timeout: 30_000 }); + if (r.exitCode !== 0) throw new Error('git ' + args.join(' ') + ': ' + r.stderr.toString()); + } + return dir; + }; + const TIERS = { setup: 'scripts/boot.sh', tiers: [{ name: 't1', command: 'make t1' }, { name: 't2', command: 'make t2' }, { name: 't3', command: 'make t3' }] }; + const bootCounts = (f: ReturnType, dir: string) => writeFileSync(join(dir, 'scripts/boot.sh'), 'echo boot >> ~/setup-count\n'); + const verifyLine = (out: string) => out.split('\n').find(l => l.startsWith('UBI_VERIFY: ')) ?? ''; + + test('run: a SIGKILLed runner leaves its VM; the same command reattaches, skips the finished setup and reports the command\'s exit', async () => { + const f = fakeVm(); + try { + const repo = gitSrc(f); + const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo setup >> ~/setup-count\n'); + writeFileSync(join(f.root, 'make-test.sh'), 'echo begin; sleep 4; echo end; exit 3\n'); + const args = ['run', '--src', repo, '--setup', setup, '--', 'make test']; + const a = f.spawn(args); + await until(() => vmDirs(f).some(d => existsSync(join(f.state, d, 'command.out')) && readFileSync(join(f.state, d, 'command.out'), 'utf8').includes('begin')), 30_000); + process.kill(a.pid, 'SIGKILL'); + await a.exited; + const name = vmDirs(f)[0]!; + expect(f.listed()).toContain(name); + const b = f.run(args); + const err = b.stderr.toString(); + expect(b.exitCode, err).toBe(3); + expect(err).toMatch(new RegExp('ubi-runner: reattached to ' + name + ' created_at=\\d{10} key=[0-9a-f]{12}\n')); + expect(b.stdout.toString().split('\n').filter(l => l === 'begin' || l === 'end')).toEqual(['begin', 'end']); + expect(err).toMatch(/command exited 3 after \d+s/); + expect(creates(f)).toBe(1); + expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('setup\n'); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 90_000); + + test('run: a second live runner of the same command refuses with UBI_ATTACH_BUSY and creates nothing', async () => { + const f = fakeVm(); + try { + const repo = gitSrc(f); + writeFileSync(join(f.root, 'make-test.sh'), 'sleep 4; echo done\n'); + const args = ['run', '--src', repo, '--', 'make test']; + const a = f.spawn(args); + await until(() => phaseStarted(f, 'command'), 30_000); + const b = f.run(args); + expect(b.exitCode).toBe(1); + expect(b.stderr.toString()).toMatch(new RegExp('UBI_ATTACH_BUSY: vm=' + vmDirs(f)[0] + ' key=[0-9a-f]{12}: runner pid ' + a.pid + ' ')); + expect(await a.exited).toBe(0); + expect(await new Response(a.stdout).text()).toContain('done'); + expect(creates(f)).toBe(1); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 90_000); + + test('run: HUP once the command is live detaches (UBI_DETACHED, exit 129, VM kept, no pass value on disk); a rerun reattaches and finishes', async () => { + const f = fakeVm(); + try { + const repo = gitSrc(f); + writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3; echo finished\n'); + const args = ['run', '--src', repo, '--pass', 'UBI_TEST_TOKEN', '--', 'make test']; + const a = f.spawn(args, { UBI_TEST_TOKEN: 'hup-secret-value' }); + await until(() => phaseStarted(f, 'command'), 30_000); + await Bun.sleep(500); + process.kill(a.pid, 'SIGHUP'); + expect(await a.exited).toBe(129); + const name = vmDirs(f)[0]!; + const err = await new Response(a.stderr).text(); + expect(err).toMatch(new RegExp('UBI_DETACHED: vm=' + name + ' key=[0-9a-f]{12} reattach: rerun the same command')); + expect(f.listed()).toContain(name); + expect(f.calls()).not.toContain('destroy'); + for (const file of readdirSync(join(f.state, name))) { + const p = join(f.state, name, file); + if (lstatSync(p).isFile()) expect(readFileSync(p, 'utf8')).not.toContain('hup-secret-value'); + } + const b = f.run(args, { UBI_TEST_TOKEN: 'hup-secret-value' }); + expect(b.exitCode, b.stderr.toString()).toBe(0); + expect(b.stderr.toString()).toContain('reattached to ' + name); + expect(b.stdout.toString()).toContain('finished'); + expect(creates(f)).toBe(1); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 90_000); + + test('run: INT and TERM during a live command still tear the VM down', async () => { + for (const sig of ['SIGTERM', 'SIGINT'] as const) { + const f = fakeVm(); + try { + const repo = gitSrc(f); + writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3\n'); + const a = f.spawn(['run', '--src', repo, '--', 'make test']); + await until(() => phaseStarted(f, 'command'), 30_000); + await Bun.sleep(500); + process.kill(a.pid, sig); + expect(await a.exited).toBe(130); + expect(await new Response(a.stderr).text()).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-/); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + } + }, 90_000); + + test('run: a remote session gone without an exit status exits 125 with UBI_ATTACH_LOST and the VM name', async () => { + const f = fakeVm(); + try { + const repo = gitSrc(f); + writeFileSync(join(f.root, 'make-test.sh'), 'sleep 30\n'); + const args = ['run', '--src', repo, '--', 'make test']; + const a = f.spawn(args); + await until(() => existsSync(join(f.vm, '.ubirun-command.pid')) && readFileSync(join(f.vm, '.ubirun-command.pid'), 'utf8').trim() !== '', 30_000); + process.kill(a.pid, 'SIGKILL'); + await a.exited; + await killSession(f, 'command'); + const name = vmDirs(f)[0]!; + const b = f.run(args); + expect(b.exitCode).toBe(125); + expect(b.stderr.toString()).toContain('UBI_ATTACH_LOST: vm=' + name + ' phase=command'); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 90_000); + + test('run: the run key covers tree, command, setup content, env, pass value, image, origin/HEAD and the runner itself; a pass value never appears in it', () => { + const f = fakeVm(); + try { + const repo = gitSrc(f); + const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo one\n'); + const runner = join(f.root, 'runner-copy.sh'); writeFileSync(runner, readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8')); + const key = (opts: { args?: string[]; env?: Record; cmd?: string; script?: string } = {}) => { + const r = Bun.spawnSync(['bash', opts.script ?? join(DIR, 'ubi-runner.sh'), 'run', '--probe', '--src', repo, '--setup', setup, '--env', 'MODE=a', '--pass', 'UBI_TEST_TOKEN', ...(opts.args ?? []), '--', opts.cmd ?? 'make test'], + { env: { ...process.env, PATH: join(f.root, 'bin') + ':' + process.env.PATH, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: f.state, UBI_OWNER: 'thrd1', UBI_TEST_TOKEN: 'v1', ...(opts.env ?? {}) }, timeout: 30_000 }); + const out = r.stdout.toString() + r.stderr.toString(); + expect(out).not.toContain('v1'); + const m = /^UBI_ATTACH_INFO: none key=([0-9a-f]{12})$/m.exec(out); + if (!m) throw new Error(out); + return m[1]!; + }; + const base = key(); + expect(key()).toBe(base); + const variants: Record = {}; + variants.command = key({ cmd: 'make check' }); + variants.env = key({ args: ['--env', 'MODE=b'] }); + variants.pass = key({ env: { UBI_TEST_TOKEN: 'v2' } }); + variants.image = key({ args: ['-b', 'ubuntu-jammy'] }); + variants.size = key({ args: ['-s', 'standard-8'] }); + writeFileSync(runner, readFileSync(runner, 'utf8') + '# edited\n'); + variants.runner = key({ script: runner }); + writeFileSync(setup, 'echo two\n'); + variants.setup = key(); + writeFileSync(setup, 'echo one\n'); + expect(key()).toBe(base); + Bun.spawnSync(['git', 'update-ref', 'refs/remotes/origin/HEAD', 'HEAD'], { cwd: repo, timeout: 10_000 }); + variants.base = key(); + writeFileSync(join(repo, 'new-file.txt'), 'untracked source\n'); + variants.tree = key(); + for (const [what, k] of Object.entries(variants)) expect({ what, same: k === base }).toEqual({ what, same: false }); + expect(new Set(Object.values(variants)).size).toBe(Object.keys(variants).length); + expect(f.calls()).not.toContain('"create"'); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 60_000); + + test('verify tiers: config refuses command with tiers, bad and duplicate tier names, before any API call', () => { + const f = fakeVm(); + try { + const cases: Array<[unknown, RegExp]> = [ + [{ command: 'make', tiers: [{ name: 'a', command: 'x' }] }, /UBI_CONFIG_TIERS_AND_COMMAND/], + [{ tiers: [{ name: 'Unit', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name 'Unit' must match/], + [{ tiers: [{ name: '-x', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name '-x'/], + [{ tiers: [{ name: 'a'.repeat(33), command: 'x' }] }, /UBI_CONFIG_TIER_NAME/], + [{ tiers: [{ name: 'a.b', command: 'x' }] }, /UBI_CONFIG_TIER_NAME/], + [{ tiers: [{ name: 'a', command: 'x' }, { name: 'a', command: 'y' }] }, /UBI_CONFIG_TIER_NAME: tier name 'a' appears twice/], + [{ tiers: [{ name: 'a' }] }, /UBI_CONFIG_TIER_NAME: tier 'a' needs a command/], + [{ tiers: [] }, /UBI_CONFIG_TIER_NAME: tiers must be a non-empty list/], + ]; + for (const [cfg, want] of cases) { + const dir = src(f, cfg); + const r = run(f, ['verify', '--src', dir]); + expect(r.exitCode).toBe(1); + expect(r.stderr.toString()).toMatch(want); + } + expect(f.calls()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 30_000); + + test('verify tiers: a run killed during tier 3 with its VM gone resumes on a new VM: setup reruns, t1 and t2 are reused, t3 reruns', async () => { + const f = fakeVm(); + try { + const dir = gitSrc(f, TIERS); + bootCounts(f, dir); + Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 }); + writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n'); + const a = f.spawn(['verify', '--src', dir]); + await until(() => existsSync(join(f.root, 't3-started')), 60_000); + process.kill(a.pid, 'SIGKILL'); + await a.exited; + await killSession(f, 'tier-t3'); + const first = vmDirs(f)[0]!; + writeFileSync(join(f.root, 'listed'), ''); + writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n'); + const b = f.run(['verify', '--src', dir]); + const err = b.stderr.toString(), out = b.stdout.toString(); + expect(b.exitCode, err).toBe(0); + expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse t1\.\d{10}\.\d+\.json \(\d+s\)\n/); + expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse t2\.\d{10}\.\d+\.json \(\d+s\)\n/); + expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t3 run: interrupted \(only a start record, t3\.\d{10}\.\d+\.json\)\n/); + expect(err).toContain('run record for ' + first + ': the VM is gone; starting a new one'); + expect(err).not.toContain('reattached to'); + expect(creates(f)).toBe(2); + expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('boot\nboot\n'); + expect(out).not.toContain('ran: make t1'); + expect(out).toContain('t3 ok'); + expect(verifyLine(out)).toMatch(/^UBI_VERIFY: tree=[0-9a-f]{12} key=[0-9a-f]{12} tiers=3 passed=3 reused=t1,t2 failed=none prior_failures=none saved_s=\d+$/); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 120_000); + + test('verify tiers: a fail record blocks a rerun until --rerun-failed (then prior_failures); passes are reused unless stale, tampered, --no-resume or the tree changed', () => { + const f = fakeVm(); + try { + const dir = gitSrc(f, TIERS); + writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 broke; exit 7\n'); + const first = run(f, ['verify', '--src', dir]); + expect(first.exitCode, first.stderr.toString()).toBe(7); + expect(first.stdout.toString()).not.toContain('ran: make t3'); + expect(verifyLine(first.stdout.toString())).toMatch(/ tiers=3 passed=1 reused=none failed=t2 prior_failures=none saved_s=0$/); + expect(creates(f)).toBe(1); + + const blocked = run(f, ['verify', '--src', dir]); + expect(blocked.exitCode).toBe(7); + expect(blocked.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 fail t2\.\d{10}\.\d+\.json \(\d+s, exit 7\): not rerun without --rerun-failed/); + expect(verifyLine(blocked.stdout.toString())).toMatch(/ failed=t2 prior_failures=none /); + expect(creates(f)).toBe(1); + + writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 fixed\n'); + const rerun = run(f, ['verify', '--src', dir, '--rerun-failed']); + expect(rerun.exitCode, rerun.stderr.toString()).toBe(0); + expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /); + expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 run: prior fail t2\.\d{10}\.\d+\.json \(exit 7; --rerun-failed\)/); + expect(verifyLine(rerun.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1 failed=none prior_failures=t2:7 saved_s=\d+$/); + expect(creates(f)).toBe(2); + + const all = run(f, ['verify', '--src', dir]); + expect(all.exitCode).toBe(0); + expect(verifyLine(all.stdout.toString())).toMatch(/ passed=3 reused=t1,t2,t3 failed=none prior_failures=none saved_s=\d+$/); + expect(all.stderr.toString()).toContain('every tier passed under this key; no VM needed'); + expect(creates(f)).toBe(2); + + const probe = (args: string[] = [], extra: Record = {}) => run(f, ['verify', '--src', dir, '--probe', ...args], extra).stderr.toString(); + expect(probe(['--no-resume'])).toMatch(/tier=t1 run: --no-resume\n[^]*tier=t2 run: --no-resume\n[^]*tier=t3 run: --no-resume\n/); + expect(probe([], { UBI_CHECKPOINT_MAX_AGE: '0' })).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json is \d+s old \(UBI_CHECKPOINT_MAX_AGE=0\)/); + const keyDir = join(f.checkpoints, readdirSync(f.checkpoints)[0]!); + const t1Log = readdirSync(keyDir).filter(n => n.startsWith('t1.') && n.endsWith('.log')).sort().at(-1)!; + writeFileSync(join(keyDir, t1Log), 'tampered\n'); + const tampered = probe(); + expect(tampered).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json: its log does not re-hash/); + expect(tampered).toMatch(/tier=t2 reuse /); + writeFileSync(join(dir, 'new-source.ts'), 'export {}\n'); + const moved = probe(); + for (const t of ['t1', 't2', 't3']) expect(moved).toContain('UBI_VERIFY_RESUME: tier=' + t + ' run: no record'); + expect(creates(f)).toBe(2); + // Records are write-once JSON; every one carries its generation. + const gens = readdirSync(keyDir).filter(n => n.endsWith('.json')).map(n => JSON.parse(readFileSync(join(keyDir, n), 'utf8')).generation as number).sort((x, y) => x - y); + expect(gens).toEqual(gens.map((_, i) => i + 1)); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 120_000); + + test('verify tiers: a reattach whose VM TTL no longer covers the run left plus UBI_REATTACH_MARGIN starts a fresh VM, destroys the old one and reuses the passed tiers', async () => { + const f = fakeVm(); + try { + const dir = gitSrc(f, TIERS); + bootCounts(f, dir); + Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 }); + writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n'); + const env = { UBI_REAP: '0', UBI_TTL_HOURS: '1' }; + const a = f.spawn(['verify', '--src', dir], env); + await until(() => existsSync(join(f.root, 't3-started')), 60_000); + process.kill(a.pid, 'SIGKILL'); + await a.exited; + const first = vmDirs(f)[0]!; + expect(first).toMatch(/-t1$/); + const probe = (extra: Record) => run(f, ['verify', '--src', dir, '--probe'], { ...env, ...extra }).stdout.toString(); + expect(probe({})).toMatch(new RegExp('UBI_ATTACH_INFO: vm=' + first + ' created_at=\\d{10} ')); + // A 1 h TTL leaves about 3600 s: t3's 1200 s run timeout plus a 2700 s margin does not fit. + const short = { UBI_RUN_TIMEOUT: '1200', UBI_REATTACH_MARGIN: '2700' }; + expect(probe(short)).toMatch(new RegExp('UBI_ATTACH_INFO: none key=[0-9a-f]{12} ttl_short vm=' + first + ' ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\)')); + expect(creates(f)).toBe(1); + writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n'); + const b = run(f, ['verify', '--src', dir], { ...env, ...short }); + const err = b.stderr.toString(); + expect(b.exitCode, err).toBe(0); + expect(err).toMatch(new RegExp('UBI_REATTACH_TTL_SHORT: vm=' + first + ' key=[0-9a-f]{12} ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\): starting a fresh VM \\(tier checkpoints carry over\\) and destroying ' + first)); + expect(err).not.toContain('reattached to'); + expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /); + expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse /); + expect(verifyLine(b.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1,t2 failed=none /); + expect(creates(f)).toBe(2); + expect(f.listed()).toBe(''); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 120_000); + + test('verify tiers: a writer whose start record is no longer the newest publishes nothing (stale writer)', () => { + const py = /\nCKPT_PY='\n([^']*)'\n/.exec(readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'))![1]!; + const root = mkdtempSync(join(tmpdir(), 'ubi-ckpt-')); + try { + const d = join(root, 'key'); + const log = join(root, 'tier.log'); writeFileSync(log, 'ok\n'); + const publish = (status: string, startgen: string, pid: string) => Bun.spawnSync(['python3', '-c', py, 'publish', d, 't1', status, 'vm', '100', '200', '0', status === 'start' ? '' : log, startgen, 'k', 'tree', pid], { timeout: 30_000 }); + expect(publish('start', '0', '1').stdout.toString().trim()).toBe('1'); + expect(publish('start', '0', '2').stdout.toString().trim()).toBe('2'); + const stale = publish('pass', '1', '1'); + expect(stale.exitCode).toBe(2); + expect(stale.stderr.toString()).toContain('stale writer: its start record is generation 1, the newest for t1 is 2'); + expect(readdirSync(d).filter(n => n.endsWith('.json')).length).toBe(2); + expect(publish('pass', '2', '2').stdout.toString().trim()).toBe('3'); + } finally { rmSync(root, { recursive: true, force: true }); } + }); + + test('verify tiers: a tier that edits the source gets no pass record (UBI_VERIFY_TREE_MOVED) and reruns next time', () => { + const f = fakeVm(); + try { + const dir = gitSrc(f, { tiers: [{ name: 'mover', command: 'make mover' }, { name: 'clean', command: 'make clean' }] }); + writeFileSync(join(f.root, 'make-mover.sh'), 'echo edited > moved.txt\n'); + const r = run(f, ['verify', '--src', dir]); + expect(r.exitCode, r.stderr.toString()).toBe(0); + expect(r.stderr.toString()).toMatch(/UBI_VERIFY_TREE_MOVED: tier=mover before=[0-9a-f]{12} after=[0-9a-f]{12}: a tier must leave the source tree as it found it; no pass record/); + expect(verifyLine(r.stdout.toString())).toMatch(/ tiers=2 passed=2 reused=none failed=none /); + const next = run(f, ['verify', '--src', dir, '--probe']).stderr.toString(); + expect(next).toMatch(/tier=mover run: interrupted \(only a start record/); + expect(next).toMatch(/tier=clean reuse /); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 90_000); }); diff --git a/ubicloud/SKILL.md b/ubicloud/SKILL.md index 4b56c9f53..4aaa63463 100644 --- a/ubicloud/SKILL.md +++ b/ubicloud/SKILL.md @@ -237,6 +237,10 @@ and for browser tests `xvfb` plus `bun run test:ubicloud` (the free suite with the CI lane's environment) and `gstack-pregate --stage tests --remote` under 8 cores. +A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]` +for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a +fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`. + --- ## Step 4: Read the result @@ -266,9 +270,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown. ## Resume checklist after a wake or a machine swap -1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is - still running without a live runner. -2. A background log without its exit sentinel after the VM is gone is an - interrupted run, not a pass; rerun it. -3. Measure before/after pairs on the same size and location, never one side +1. Rerun the same command first: a run whose runner died or detached + (`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a + finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125 + `UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over). +2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no + rerun will reattach to. +3. After a machine swap nothing reattaches; tier checkpoints on a Drive + (`UBI_CHECKPOINT_DIR`) still count. +4. A background log with no exit sentinel and no VM is an interrupted run, not + a pass; rerun it. +5. Measure before/after pairs on the same size and location, never one side local and the other remote. diff --git a/ubicloud/SKILL.md.tmpl b/ubicloud/SKILL.md.tmpl index cfcf9fa52..970627d94 100644 --- a/ubicloud/SKILL.md.tmpl +++ b/ubicloud/SKILL.md.tmpl @@ -163,6 +163,10 @@ and for browser tests `xvfb` plus `bun run test:ubicloud` (the free suite with the CI lane's environment) and `gstack-pregate --stage tests --remote` under 8 cores. +A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]` +for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a +fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`. + --- ## Step 4: Read the result @@ -192,9 +196,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown. ## Resume checklist after a wake or a machine swap -1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is - still running without a live runner. -2. A background log without its exit sentinel after the VM is gone is an - interrupted run, not a pass; rerun it. -3. Measure before/after pairs on the same size and location, never one side +1. Rerun the same command first: a run whose runner died or detached + (`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a + finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125 + `UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over). +2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no + rerun will reattach to. +3. After a machine swap nothing reattaches; tier checkpoints on a Drive + (`UBI_CHECKPOINT_DIR`) still count. +4. A background log with no exit sentinel and no VM is an interrupted run, not + a pass; rerun it. +5. Measure before/after pairs on the same size and location, never one side local and the other remote. From c830c9b8bafaf2a2327f6ef4cd3a133ad5939731 Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Sat, 31 Oct 2026 10:58:20 -0700 Subject: [PATCH 3/6] v1.91.377.0 pregate: clear the remote gate with verified Actions runs on the same tree (clear --run) (#3592) * pregate: clear the remote gate with verified Actions runs on the same tree (clear --run) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: REUSE accepted/refused lines, run-url regression test, receipt bind end to end Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: review fixes for clear --run (re-run veto, profiled lanes fail closed, no pull_request runs, user-policy keys) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: adversarial fixes for clear --run (unknown profiles fail closed, lane coverage, run re-read, base binding, paging, artifact bound, full assignment) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: pass-2 review fixes for clear --run (docstrings, tests_uncovered reason, unreadable-base and user-policy tests) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(troubleshooting): clear --run refusals, run-profile keys in the user policy, verify's foreign-tree run receipt Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * chore: CHANGELOG for clear --run (unreleased; stamped at merge) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: receipt clear --run's gh reads and head fetch in the egress ledger (sink run-evidence) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * pregate: run the remote gate inventory through runBashSync (script-invocation tripwire) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): clear --run reads origin before url.insteadOf, so Capy's git.capy.ai proxy still names the GitHub repository Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(pregate): clear --run re-reads every run before recording, ties veto, inventory runs under pipefail, freshness reads job completion, and a pregate.json rewritten mid-clear refuses Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(pregate): the veto counts runs started at or after the candidate; quote clear's real refusal line Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * docs(pregate): user policy refuses the run-profile keys; the remote gate clears one of three ways Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- CHANGELOG.md | 15 ++ VERSION | 2 +- agents-digest/gstack-AGENTS.md | 2 +- bin/gstack-pregate | 53 +++++- docs/pregate.md | 138 ++++++++++++++- docs/troubleshooting.md | 21 ++- lib/pregate/config.ts | 22 ++- lib/pregate/index.ts | 112 +++++++++++-- lib/run-evidence.ts | 261 +++++++++++++++++++++++++++++ package.json | 2 +- test/egress-receipt-wiring.test.ts | 3 + test/pregate-clear-runs.test.ts | 246 +++++++++++++++++++++++++++ test/pregate-user-policy.test.ts | 6 +- test/run-evidence.test.ts | 183 ++++++++++++++++++++ 14 files changed, 1030 insertions(+), 36 deletions(-) create mode 100644 lib/run-evidence.ts create mode 100644 test/pregate-clear-runs.test.ts create mode 100644 test/run-evidence.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 67649e356..a3ede05c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,20 @@ # Changelog +## [1.91.377.0] - 2026-10-31 + +**A passed GitHub Actions run can now clear the remote test gate for the same tree, so a repository stops running its full suite twice per slot.** gbrain paid a 30–45 minute Ubicloud gate and then an 18–20 minute full-ci dispatch on the same tree in every serial slot (GBRA-108: #7872's slot took 5.6 h). `gstack-pregate clear --remote pregate.json/remote_gate --run --run ` accepts one verified run per profile the base branch declares. gstack checks each run against the runner's own record, never against a pasted claim. It reads the GitHub API's run and jobs, the tree git computes for the run's head, and the `tree.json` the run uploaded. Nothing is skipped: the tests still run once, in CI. A repository that declares no profiles is unchanged. + +### Added +- **`gstack-pregate clear --run `** (repeatable; `lib/run-evidence.ts`, docs/pregate.md#run-evidence). A run is accepted only when it belongs to this repository, ran a `push` or `workflow_dispatch` event of the profile's workflow, and its newest attempt succeeded with every named job passing exactly once. Its `tree.json` must name the run, attempt, event and head and record the head's tree, and it must be fresh (`max_age_hours`, default 24) and on the profile's OS. It is refused if any run of that workflow on the head that started at or after it failed, was cancelled or is still running, or if the run changed attempts while being checked. All runs must be on one tree: pregate.json's, or a stamp-only predecessor under the existing tree-receipt rule. Their `files` must cover the obligation's selected tests plus every file the declared `inventory` command lists for that tree. Refusals print `REUSE: refused reason=`; an accepted clear prints `REUSE: accepted`, and the receipt records `runs=/ profiles= heads= verified=`. +- **`run_profiles` and `remote_gate_runs`** in `.gstack/pregate.json`, read only from `origin/`. A user-owned policy that sets them is refused, because nothing would read them there. `gstack-pregate clear` clears a lane named by a profile only through verified runs, so its `--run-url` is verified the same way, and `--bundle-lane` is refused. `gstack-ship-queue ready --dispatch-ci` still records a plain run URL for such a lane, and `verify` does not yet re-check it (docs/pregate.md#run-evidence). When the base config cannot be read, any lane's `--run-url` or `--bundle-lane` clear refuses, since the base may profile it. When the GitHub origin cannot be read, a profiled lane refuses rather than fall back to an unverified record. + +### Changed +- **`gstack-pregate verify`** stays offline. It also refuses a run receipt recorded for another tree unless its stamp-only substitution is recorded. + +### Tests +- `test/run-evidence.test.ts`: every refusal reason, including the re-run cases (a red re-run of an older run vetoes; re-running the candidate does not hide a failure in between), a run that changes attempts while being checked, malformed counts and paths, and profile validation. +- `test/pregate-clear-runs.test.ts`: on a fixture repository, clear on the same tree, verify offline, and bind the ship receipt. Also covers a stamp-only predecessor, a non-stamp difference, uncovered files, runs on different trees, base mismatch, repeated runs, the profile-to-run assignment, the profiled-lane matrix (`--run-url`, `--bundle-lane`, `--run`), a non-GitHub origin failing closed, an origin that `url.insteadOf` rewrites to a proxy (Capy's git.capy.ai) still read as the GitHub repository, and an unreadable base config refusing `--run-url` on every lane. The `--run-url` fallback test failed before the fix. + ## [1.91.376.0] - 2026-10-31 bump: minor diff --git a/VERSION b/VERSION index 4f0a874d2..72f0d75b2 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -1.91.376.0 +1.91.377.0 diff --git a/agents-digest/gstack-AGENTS.md b/agents-digest/gstack-AGENTS.md index b6a329420..56cb480ad 100644 --- a/agents-digest/gstack-AGENTS.md +++ b/agents-digest/gstack-AGENTS.md @@ -1,4 +1,4 @@ -# gstack digest v1.91.376.0 — regenerate/re-copy after upgrading gstack +# gstack digest v1.91.377.0 — regenerate/re-copy after upgrading gstack Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed for agent hosts without a full skill install. The full skills add workflows, diff --git a/bin/gstack-pregate b/bin/gstack-pregate index b3f5297eb..10da90e78 100755 --- a/bin/gstack-pregate +++ b/bin/gstack-pregate @@ -6,7 +6,7 @@ * [--allow-repo-commands] [--policy-from base|head] [--out ] * [--timeout ] [--stall-timeout ] [--remote --spend-cap ] [--run ] [--explain] [--json] * gstack-pregate verify [--out ] [--json] publication check: current tree, no fail/incomplete, obligations cleared - * gstack-pregate clear --remote --run-url | --bundle-lane