v1.91.376.0 feat(ubicloud): reattach to a live verify VM and checkpoint tiers (GSTA-158 P3) (#3615)

* feat(ubicloud): reattach to a live verify VM and checkpoint tiers (GSTA-158 P3) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix(ubicloud): replay on reattach, tear down TTL-less VMs on HUP, trim skill prose (GSTA-158 P3) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-10-31 22:14:25 +01:00 · 2026-10-31 17:05:40 +00:00
1 parent 7d4e71bd94
commit 5ba425af4b
@@ -1,5 +1,21 @@
 # Changelog

+## [1.91.376.0] - 2026-10-31
+
+bump: minor
+---
+**A remote verify now survives a dead local runner, and a long verify can resume from the tiers that already passed (GSTA-158 P3).** Before, a Capy machine that went to sleep or a runner that got killed lost the whole remote run, and the paid VM ran on until its TTL. Now `ubi-runner run` and `verify` run each phase detached on the VM, and running the same command again reattaches to the live VM. A verify can be split into tiers, and a rerun skips the tiers that passed.
+
+### Added
+- **Reattach by run key.** The key covers the checkout's content, the command, setup, env, a hash of each `--pass` value, size, image, origin/HEAD and the runner itself. A rerun prints `ubi-runner: reattached to <vm> created_at=<epoch> key=<k12>`, replays what the phase already printed and follows it from there. A finished setup is not rerun. `UBI_ATTACH_BUSY` means another live runner owns the VM. Exit 125 `UBI_ATTACH_LOST` means the remote session died without an exit status.
+- **Reattach only while the TTL lasts.** A reattach needs the VM's remaining TTL to cover the run still to do (each unfinished phase's `UBI_RUN_TIMEOUT` bound) plus `UBI_REATTACH_MARGIN` (900 s). Otherwise the runner prints `UBI_REATTACH_TTL_SHORT` with the numbers, destroys the old VM and starts a fresh one. Tier checkpoints carry over to it.
+- **Verify tiers.** `.gstack/ubicloud.json` can list `tiers: [{name, command}]` in place of `command`. Each tier writes a start, pass or fail checkpoint under `UBI_CHECKPOINT_DIR`. A rerun prints `UBI_VERIFY_RESUME:` per tier and reuses passes younger than `UBI_CHECKPOINT_MAX_AGE` whose logs still hash. It reruns a recorded fail only with `--rerun-failed`. The last line is `UBI_VERIFY: …`.
+- **`attach NAME`, `run --probe` and `--key-input`.**
+
+### Changed
+- **HUP once a phase is live detaches** (`UBI_DETACHED`, exit 129, `UBICLOUD: left=1 … detached=1`): the VM keeps running under its TTL. INT and TERM still tear it down.
+- **`gstack-pregate --stage tests --remote` reattaches** to the VM a killed run left. It bills `spent=` from the original create time, and a cap that is already spent refuses with `PREGATE_REMOTE_CAP_SPENT`. The `PREGATE_REMOTE:` receipt is unchanged.
+
 ## [1.91.375.0] - 2026-10-31

 **A /ship subagent now stops once its READY is sent, so CI and review callbacks on a frozen head stop waking it.** Finished /ship subagents kept getting woken by checks and reviews on a PR they had already READY'd. Each wake cost tokens, and some acted on a head that was supposed to be frozen. The parent now subscribes to the PR first, so red checks and human review still reach an owner, and then stops the subagent.
@@ -1 +1 @@
-1.91.375.0
+1.91.376.0
@@ -1,4 +1,4 @@
-# gstack digest v1.91.375.0 — regenerate/re-copy after upgrading gstack
+# gstack digest v1.91.376.0 — regenerate/re-copy after upgrading gstack

 Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
 for agent hosts without a full skill install. The full skills add workflows,
@@ -1034,6 +1034,22 @@ never forwards `UBICLOUD_*`. All of this is pinned offline in
 `test/ubicloud-runner.test.ts` against a fake API and a fake `ssh`, including
 a fixture whose fake API creates, so `run` and `verify` execute end to end.

+`run` and `verify` start their setup, command and tier phases detached on the
+VM (output, leader pid and exit status in `~/.ubirun-<phase>.*`) and key each
+run locally before any VM exists (content tree, command, setup content, env,
+pass-value hashes, size, location, image, storage, origin/HEAD, the runner's
+own sha256). A rerun of the same command after the runner died reattaches to
+the live VM and continues from the first phase without an exit file
+(`UBI_ATTACH_BUSY` while another runner owns it, exit 125 `UBI_ATTACH_LOST`
+when the session left no status); HUP detaches (`UBI_DETACHED`, exit 129), INT
+and TERM still tear down. A `.gstack/ubicloud.json` with `tiers` records
+start/pass/fail per tier under `UBI_CHECKPOINT_DIR/<run key>/`, so a resume on
+a fresh VM reruns setup and only the tiers without a pass; `pregate-remote.sh`
+adds its tree and planned-set digest to the key, so a rerun of
+`gstack-pregate --stage tests --remote` reattaches and bills from the original
+`created_at`. The fake VM in `test/ubicloud-runner.test.ts` runs these phases
+for real in a directory.
+
 Under 8 cores (`gstack-doctor`'s `cores` row says so), run the full unit and
 E2E tiers on a remote gate: `gstack-config set test_backend ubicloud` records
 the choice and the row names the runner. Resume checklist after a wake or a
@@ -1,6 +1,6 @@
 {
  "name": "gstack",
-  "version": "1.91.375",
+  "version": "1.91.376",
  "description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
  "license": "MIT",
  "type": "module",
@@ -27,10 +27,18 @@
 # teardown) times the size's price. Without a key it runs the same files
 # locally in parallel shards and reports spent=0.
 #
+# Reattach: the tree and the planned set's digest go into ubi-runner's run key
+# (--key-input), so a rerun after the local runner died (a machine stop, a
+# SIGKILL) reattaches to the VM still running this gate instead of paying for a
+# second one. Its minutes since created_at count against the cap: the wall
+# budget shrinks by them (PREGATE_REMOTE_CAP_SPENT when none is left), and
+# spent= bills from the original create.
+#
 # Exit codes: 0 pass (or a paid-only selection, tests=0); 1 test failures (or
 # a rejected file list); 2 missing input; 3 tree or plan mismatch (or the tree
 # moved during the run); 4 infrastructure failure (create, setup, SSH, stop
-# deadline); 5 the spend cap or pregate's deadline refuses the run. Every refusal line starts with a
+# deadline, another gate on this machine owning the VM to reattach to); 5 the
+# spend cap or pregate's deadline refuses the run. Every refusal line starts with a
 # PREGATE_REMOTE_* reason code (docs/pregate.md#remote-gate). Nothing is
 # uploaded or spent before the local tree check.
 set -euo pipefail
@@ -147,8 +155,6 @@ if [ -n "$outer_s" ]; then
  fi
  if [ "$outer_stop" -lt "$stop_after" ]; then stop_after=$outer_stop stop_why="$outer_from ${outer_s}s minus 120 s teardown reserve and 15 s margin"; fi
 fi
-say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"
-
 digest=$(tr -d '\r' <"$files" | grep . | LC_ALL=C sort | sha256sum | cut -d' ' -f1)
 prelude="$logs/vm-setup.sh"
 {
@@ -183,6 +189,28 @@ args=(run -s "$size" --src "$root" --setup "$prelude"
  --pull "/tmp/gstack-free-test-*:$logs"
  --pull "work/$(basename "$root")/.context/free-test-logs:$logs")
 [ -z "${GSTACK_FREE_JOBS:-}" ] || args+=(--env "GSTACK_FREE_JOBS=$GSTACK_FREE_JOBS")
+args+=(--key-input "pregate-remote tree=$tree plan=$digest")
+
+# A rerun of this gate for the same tree and planned set has the same
+# ubi-runner run key, so it reattaches to the VM a killed run left behind
+# (`ubi-runner: reattached to <vm> created_at=<epoch>`). The minutes that VM
+# has billed since created_at come off the cap's wall budget first.
+prior_vm=""
+attach=$(bash "$here/ubi-runner.sh" "${args[@]}" --probe -- "bash ~/.pregate-run.sh" 2>/dev/null \
+  | sed -n 's/^UBI_ATTACH_INFO: vm=\([^ ]*\) created_at=\([0-9]*\) key=[0-9a-f]* busy=\([01]\)$/\1 \2 \3/p' | tail -1) || attach=""
+if [ -n "$attach" ]; then
+  read -r prior_vm prior_created prior_busy <<<"$attach"
+  [ "$prior_busy" = 0 ] || stop 4 PREGATE_REMOTE_ATTACH_BUSY "another gate on this machine still runs VM $prior_vm for this tree and planned set" "wait for it; this run created nothing and spent nothing"
+  prior_min=$(( ($(date +%s) - prior_created + 59) / 60 ))
+  left_s=$(( (wall_min - 3 - prior_min) * 60 ))
+  if [ "$left_s" -le 0 ]; then
+    stop 5 PREGATE_REMOTE_CAP_SPENT "VM $prior_vm, created ${prior_min} min ago for this tree and planned set, has already used the cap's $wall_min min wall (\$$(usd "$prior_min") of \$$cap)" \
+      "nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down $prior_vm" "or rerun with a higher --spend-cap to let it finish: $rerun"
+  fi
+  if [ "$left_s" -lt "$stop_after" ]; then stop_after=$left_s stop_why="the cap's $wall_min min wall minus 3 min for teardown and $prior_min min already billed since $prior_vm was created"; fi
+  say "reattaching to VM $prior_vm, created ${prior_min} min ago by an earlier run of this gate for the same tree and planned set"
+fi
+say "$count files on one $size VM; worst case \$$(usd "$wall_min") under the cap \$$cap; the VM is told to stop after ${stop_after}s ($stop_why)"

 # The VM run gets its own session (python3, which ubi-runner already needs, for
 # setsid on Linux and macOS): pregate's --timeout kill signals this
@@ -220,7 +248,7 @@ while :; do
  rm -f "$log.pid" "$log.timedout" "$log.left"
  t0=$(date +%s)
  vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))"
-  say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
+  [ -n "$prior_vm" ] || say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
  UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
    bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" &
  vmrun=$!
@@ -236,6 +264,9 @@ while :; do
  tailer=""
  t_end=$(date +%s)
  t_create=$(at 'ubi-runner: creating ')
+  # A reattached VM is billed from its original create.
+  reattached=$(sed -n 's/^[0-9]* ubi-runner: reattached to [^ ]* created_at=\([0-9][0-9]*\) .*/\1/p' "$log" | tail -1)
+  [ -z "$reattached" ] || t_create=$reattached
  if [ -n "$t_create" ] && ! grep -q 'ubi-runner: create failed' "$log"; then minutes=$(( (t_end - t_create + 59) / 60 )); billed=$(( billed + (minutes > 0 ? minutes : 1) )); fi
  if [ "$rc" -ne 0 ] && [ "$attempt" -eq 1 ] && grep -q 'vCPU quota exhausted' "$log"; then
    say "vCPU quota refused the VM; one retry in ${PREGATE_REMOTE_QUOTA_WAIT_S:-60} s"
@@ -279,7 +310,7 @@ if [ -n "$result" ] && [ "$rc" -ne 124 ]; then
  echo "PREGATE_REMOTE: tree=$vm_tree tests=$(field tests "$result") failed=$(field failed "$result") spent=$spent"
  exit $(( rc == 0 ? 0 : 1 ))
 fi
-vm=$(sed -n 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' "$log" | tail -1)
+vm=$(sed -n -e 's/^[0-9]* ubi-runner: creating \([^ ]*\) .*/\1/p' -e 's/^[0-9]* ubi-runner: reattached to \([^ ]*\) .*/\1/p' "$log" | tail -1)
 down=$(grep -E 'ubi-runner: (destroyed|gone:|WARNING|FAILED to destroy)' "$log" | tail -1 | cut -d' ' -f2- || true)
 cause=$(grep -E 'ubi-runner: ' "$log" | grep -vE 'ubi-runner: (destroyed|gone:|destroy requested|waiting for an in-flight)' | tail -1 | cut -d' ' -f2- || true)
 [ "$rc" -ne 124 ] || cause="the VM was told to stop after ${stop_after}s ($stop_why) and torn down"
@@ -33,8 +33,9 @@
 # Teardown: `up` records each VM's state (location, create status) before it
 # sends the create request, and `down` destroys and polls until the VM is
 # confirmed gone or provably never existed, waiting out a create that may still
-# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, HUP, QUIT) runs
-# that `down`; further signals are ignored until it finishes. A create refused
+# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, QUIT, and HUP
+# until a remote phase is live) runs that `down`; further signals are ignored
+# until it finishes. A create refused
 # for vCPU quota prints the project's usage per owner.
 #
 # The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive
@@ -69,6 +70,33 @@
 # UBI_RUN_TIMEOUT seconds (0, unlimited), exiting 124 with the last 40 lines,
 # so a hung verify fails fast instead of holding the VM until its TTL.
 #
+# Detached phases and reattach: `run` and `verify` start their setup, command
+# and tier phases on the VM detached from the ssh connection (setsid; stdout to
+# ~/.ubirun-<phase>.out, stderr to .err, the session leader's pid in .pid, and
+# the exit status in .exit, written by rename once the streams are closed) and
+# stream both files back. Before creating a VM they compute a run key locally
+# (the checkout's content tree, the command, the setup script's content, every
+# --env entry, each --pass name with a sha256 of its value, size, location,
+# image, storage, origin/HEAD, tiers and this script's own sha256) and look for
+# a run record (<state dir>/<vm>/run) with the same key whose VM still exists:
+# its runner alive (lease pid and start time) refuses with UBI_ATTACH_BUSY,
+# otherwise the run reattaches, printing `reattached to <vm> created_at=<epoch>
+# key=<k12>`, and continues from the first phase without an exit file. After
+# unpack the VM recomputes the tree and refuses a mismatch (UBI_VM_TREE_MISMATCH).
+# A leader gone with no exit file exits 125 (UBI_ATTACH_LOST). Signals: INT
+# and TERM tear the VM down as before; HUP once a remote phase is live detaches
+# (UBI_DETACHED, exit 129) and leaves the VM under its TTL; SIGKILL or a machine
+# stop leaves the record and the VM, and the next run of the same command
+# reattaches. `attach NAME` resumes a recorded run explicitly. Reattach never
+# extends a TTL, and the reaper still applies.
+#
+# Tier checkpoints: a .gstack/ubicloud.json with `tiers` runs each tier as its
+# own detached phase after setup, and records start/pass/fail per tier under
+# UBI_CHECKPOINT_DIR/<run key>/ (write-once JSON plus the tier's log, published
+# under a lock with a monotonic generation). A rerun reuses a pass younger than
+# UBI_CHECKPOINT_MAX_AGE whose log re-hashes, so a resume on a fresh VM reruns
+# only setup and the tiers without a pass.
+#
 # `verify` is the one entry point for a repository's full remote verify: it
 # reads .gstack/ubicloud.json (setup, command, size, location, env, pass, pull)
 # and runs it through `run`, so lanes share one committed definition instead of
@@ -100,6 +128,11 @@ PREFIX="ubirun"
 if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD"
 elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv"
 else VM_RECORD="$STATE_ROOT/vms.tsv"; fi
+if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR"
+elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints"
+else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi
+CHECKPOINT_MAX_AGE="${UBI_CHECKPOINT_MAX_AGE:-86400}"
+REATTACH_MARGIN="${UBI_REATTACH_MARGIN:-900}"

 die() { echo "ubi-runner: $*" >&2; exit 1; }
 log() { echo "ubi-runner: $*" >&2; }
@@ -109,7 +142,7 @@ for bin in curl python3 ssh ssh-keygen tar; do
 done
 UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}"
 [[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)"
-for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT; do
+for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT CHECKPOINT_MAX_AGE REATTACH_MARGIN; do
  [[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)"
 done
 [ -z "$QUOTA_VCPUS" ] || [[ "$QUOTA_VCPUS" =~ ^[0-9]+$ ]] || die "UBI_QUOTA_VCPUS must be a whole number of vCPUs"
@@ -296,6 +329,410 @@ watched() {
  return "$rc"
 }

+proc_start() { ps -o lstart= -p "$1" 2>/dev/null | tr -s ' ' || true; }
+fsize() { if [ -f "$1" ]; then wc -c <"$1" | tr -d ' '; else echo 0; fi; }
+sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | cut -d' ' -f1; else shasum -a 256 | cut -d' ' -f1; fi; }
+
+# TREE_SH: print the content tree of the current directory, the same way here
+# and on the VM: at a checkout's root, `git add -A` into a copy of its index
+# (tracked plus untracked files that are not ignored, the files pack uploads);
+# elsewhere, the same into a scratch repository over the directory.
+TREE_SH='idx=$(mktemp -d "${TMPDIR:-/tmp}/ubirun-tree.XXXXXX") || exit 1
+if [ "$(git rev-parse --show-toplevel 2>/dev/null)" = "$(pwd -P)" ]; then
+  gi=$(git rev-parse --git-path index)
+  if [ "$(git config --bool core.splitIndex)" != true ] && [ -f "$gi" ]; then cp "$gi" "$idx/index"; fi
+else
+  git init -q --bare "$idx/git" && export GIT_DIR="$idx/git" GIT_WORK_TREE=.
+fi
+GIT_INDEX_FILE="$idx/index" git add -A >/dev/null 2>&1 && GIT_INDEX_FILE="$idx/index" git write-tree
+rc=$?; rm -rf "$idx"; exit $rc'
+
+# vm_tree NAME QDEST: the content tree of QDEST on the VM.
+vm_tree() {
+  remote "$1" "command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git; } >/dev/null 2>&1; cd $2 && bash -c $(sq "$TREE_SH")" </dev/null
+}
+
+base_sha() { git -C "$1" rev-parse -q --verify 'refs/remotes/origin/HEAD^{commit}' 2>/dev/null || echo none; }
+
+# phase_status NAME PHASE: "exit <rc>" once the phase wrote its exit file,
+# "alive" while its session leader runs, "lost" when the leader is gone with no
+# exit file, "none" when it never started.
+phase_status() {
+  local p="~/.ubirun-$2"
+  remote "$1" "if [ -f $p.exit ]; then echo exit \$(cat $p.exit); elif [ -s $p.pid ] && kill -0 \$(cat $p.pid) 2>/dev/null; then echo alive; elif [ -s $p.pid ]; then echo lost; else echo none; fi" </dev/null
+}
+
+# phase_start NAME PHASE INNER: start INNER on the VM in its own session,
+# detached from this ssh connection. The leader records its pid, runs INNER with
+# stdout and stderr in files, and writes the exit status by rename only after
+# INNER's streams are closed.
+phase_start() {
+  local p="~/.ubirun-$2" inner
+  inner=$(sq "echo \$\$ >$p.pid.tmp && mv $p.pid.tmp $p.pid; ( $3 ) >$p.out 2>$p.err </dev/null; echo \$? >$p.exit.tmp && mv $p.exit.tmp $p.exit")
+  remote "$1" "rm -f $p.exit $p.pid; : >$p.out; : >$p.err; if command -v setsid >/dev/null 2>&1; then setsid -f bash -c $inner </dev/null >/dev/null 2>&1; else nohup bash -c $inner </dev/null >/dev/null 2>&1 & fi" </dev/null >/dev/null
+}
+
+# stream_cmd PHASE OUT_OFFSET ERR_OFFSET: follow both output files from the
+# given byte offsets until the session leader exits.
+stream_cmd() {
+  local p="~/.ubirun-$1"
+  printf '%s' "i=0; while [ ! -s $p.pid ] && [ ! -f $p.exit ] && [ \$i -lt 600 ]; do sleep 0.1; i=\$((i + 1)); done; pid=\$(cat $p.pid 2>/dev/null); f=; [ -z \"\$pid\" ] || f=\"-f --pid=\$pid\"; tail -c +$(( $3 + 1 )) \$f $p.err >&2 & tail -c +$(( $2 + 1 )) \$f $p.out; wait"
+}
+
+PHASE_LIVE="" STREAM_PID="" RUN_KEY="" TREE=""
+
+# on_hup: once a remote phase is live, HUP detaches: the VM keeps running under
+# its TTL and the next run of the same command reattaches. Before that, HUP
+# tears down like INT and TERM.
+on_hup() {
+  [ -n "$PHASE_LIVE" ] || exit 130
+  [ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null || true
+  trap - EXIT
+  rm -f "$(state_dir "$RUN_VM")/lease"
+  leftover "left=1 owner=$OWNER vm=$RUN_VM detached=1"
+  echo "UBI_DETACHED: vm=$RUN_VM key=${RUN_KEY:0:12} reattach: rerun the same command" >&2
+  exit 129
+}
+
+# detached NAME PHASE INNER: run INNER as a detached phase (or reattach to it),
+# streaming its output to this process and to the VM's state dir
+# (<phase>.out, <phase>.err and the combined <phase>.log) under the same
+# heartbeat and watchdog as watched. RUN_TIMEOUT counts from the phase's
+# recorded start, not from a reattach. Returns the phase's exit status, 124
+# after a watchdog kill, 125 when the session is gone without an exit status.
+detached() {
+  local name=$1 phase=$2 inner=$3 dir st rc=0 tries=0 start now idle elapsed last_beat why old_traps spid sst oo oe
+  load "$name"
+  dir=$(state_dir "$name")
+  local lout="$dir/$phase.out" lerr="$dir/$phase.err" out="$dir/$phase.log" p="~/.ubirun-$phase"
+  st=none
+  if [ -f "$dir/phase.$phase.start" ]; then
+    st=$(phase_status "$name" "$phase") || st=""
+  fi
+  case $st in
+    none)
+      : >"$lout"; : >"$lerr"; : >"$out"
+      date +%s >"$dir/phase.$phase.start"
+      phase_start "$name" "$phase" "$inner" || { log "could not start $phase on $name"; return 255; } ;;
+    alive) log "$phase is still running on $name; following it" ;;
+    "exit "*) ;;
+    lost) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; return 125 ;;
+    *) log "could not read the state of $phase on $name"; return 255 ;;
+  esac
+  PHASE_LIVE=1
+  start=$(cat "$dir/phase.$phase.start")
+  # A runner killed on this machine leaves its stream client behind; it would
+  # append to the same local copies.
+  if [ -f "$dir/stream" ] && read -r spid sst <"$dir/stream" && [ -n "$spid" ] && [ "$(proc_start "$spid")" = "$sst" ]; then
+    kill -TERM -- "-$spid" 2>/dev/null || true
+  fi
+  oo=$(fsize "$lout") oe=$(fsize "$lerr")
+  if [ "$st" != none ]; then head -c "$oo" "$lout"; head -c "$oe" "$lerr" >&2; fi
+  old_traps=$(trap -p INT TERM)
+  trap '[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null; exit 130' INT TERM
+  last_beat=$(date +%s)
+  while :; do
+    touch "$out"
+    # shellcheck disable=SC2046
+    ( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
+        ssh $(ssh_opts) "ubi@$IP" "umask 022; $(stream_cmd "$phase" "$oo" "$oe")" ) \
+      </dev/null > >(tee -a "$lout" | tee -a "$out") 2> >(tee -a "$lerr" | tee -a "$out" >&2) &
+    STREAM_PID=$!
+    echo "$STREAM_PID $(proc_start "$STREAM_PID")" >"$dir/stream"
+    why=""
+    while :; do
+      sleep 1
+      kill -0 "$STREAM_PID" 2>/dev/null || break
+      now=$(date +%s)
+      elapsed=$(( now - start ))
+      idle=$(( now - $(mtime "$out") ))
+      if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then
+        why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)"
+      elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then
+        why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)"
+      fi
+      [ -z "$why" ] || break
+      if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then
+        log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago"
+        last_beat=$now
+      fi
+    done
+    if [ -n "$why" ]; then
+      log "watchdog: $phase $why; killing it. Last output:"
+      tail -n 40 "$out" >&2
+      kill -TERM -- "-$STREAM_PID" 2>/dev/null || kill "$STREAM_PID" 2>/dev/null || true
+      wait "$STREAM_PID" 2>/dev/null || true
+      STREAM_PID=""
+      # A phase that finished in the instant before the kill keeps its exit 0.
+      if [ "$(phase_status "$name" "$phase" 2>/dev/null)" = "exit 0" ]; then
+        log "watchdog: $phase finished before it could be killed; its exit 0 stands"
+        rc=0
+      else
+        remote_cancel "$name" "$p.pid"
+        rc=124
+      fi
+      break
+    fi
+    wait "$STREAM_PID" 2>/dev/null || true
+    STREAM_PID=""
+    st=$(phase_status "$name" "$phase") || st=""
+    case $st in
+      "exit "*) rc=${st#exit }; break ;;
+      lost|none) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; rc=125; break ;;
+    esac
+    tries=$(( tries + 1 ))
+    if [ "$tries" -ge 5 ]; then log "lost the connection to $phase on $name 5 times; giving up (the VM keeps it until teardown)"; rc=255; break; fi
+    log "the stream of $phase on $name dropped; following it again"
+    sleep "$POLL"
+    oo=$(fsize "$lout") oe=$(fsize "$lerr")
+  done
+  rm -f "$dir/stream"
+  trap - INT TERM; eval "$old_traps"
+  return "$rc"
+}
+
+# write_record DIR: the run record a later run of the same command reattaches
+# by (key, tree, and this invocation for `attach NAME`). Remote files of each
+# phase: ~/.ubirun-<phase>.{out,err,exit,pid} (setup, command, tier-<name>).
+qwords() { local x; for x in "$@"; do printf '%q ' "$x"; done; }
+write_record() {
+  {
+    printf 'RUN_KEY=%q\nTREE=%q\nPULL_ROOT=%q\nRESUME=%q\nRERUN_FAILED=%q\n' "$RUN_KEY" "$TREE" "$PULL_ROOT" "$RESUME" "$RERUN_FAILED"
+    printf 'RUN_ARGS=(%s)\n' "$(qwords "${RUN_ARGS[@]}")"
+    printf 'TIER_NAMES=(%s)\n' "$(qwords ${TIER_NAMES[@]+"${TIER_NAMES[@]}"})"
+    printf 'TIER_CMDS=(%s)\n' "$(qwords ${TIER_CMDS[@]+"${TIER_CMDS[@]}"})"
+    echo "# remote files: ~/.ubirun-<phase>.{out,err,exit,pid}; phase starts: phase.<phase>.start; lease: lease (pid start time)"
+  } >"$1/run.$$" && mv "$1/run.$$" "$1/run"
+}
+
+lease_take() { echo "$$ $(proc_start $$)" >"$1/lease.$$" && mv "$1/lease.$$" "$1/lease"; }
+lease_busy() {
+  local pid st
+  [ -f "$1/lease" ] || return 1
+  read -r pid st <"$1/lease" || true
+  [ -n "$pid" ] && [ "$pid" != "$$" ] && kill -0 "$pid" 2>/dev/null && [ "$(proc_start "$pid")" = "$st" ]
+}
+
+# vm_alive NAME: the recorded VM still answers show. One confirmed gone is
+# cleaned up (state dir and scratch record), so its record is never tried again.
+vm_alive() {
+  local n=$1 out code
+  ( load "$n"; [ "$CREATED" = yes ] && [ -n "$IP" ] ) 2>/dev/null || return 1
+  out=$(mktemp)
+  code=$(load "$n"; api "$out" vm "$LOCATION/$n" show)
+  rm -f "$out"
+  case $code in
+    200) return 0 ;;
+    404) log "run record for $n: the VM is gone; starting a new one"; cmd_down "$n" >/dev/null 2>&1 || true; return 1 ;;
+    *) return 1 ;;
+  esac
+}
+
+# ttl_short NAME SETUP: true when NAME's TTL (its created_at plus the name's
+# -t<hours>, minus now) does not cover the run still to do plus
+# REATTACH_MARGIN, with the numbers in TTL_WHY. The run still to do is the
+# UBI_RUN_TIMEOUT bound of each phase that has not finished (setup when SETUP is
+# set, then the command or every tier the plan does not reuse), counted from a
+# started phase's recorded start; it is 0 when UBI_RUN_TIMEOUT is 0.
+TTL_WHY=""
+ttl_short() {
+  local name=$1 dir created left now need=0 p i phases=()
+  [[ "$name" =~ -t([0-9]{1,3})$ ]] || return 1
+  dir=$(state_dir "$name")
+  created=$(load "$name"; echo "$CREATE_AT")
+  now=$(date +%s)
+  left=$(( created + 10#${BASH_REMATCH[1]} * 3600 - now ))
+  [ -z "$2" ] || phases+=(setup)
+  if [ ${#TIER_NAMES[@]} -gt 0 ]; then
+    for i in "${!TIER_NAMES[@]}"; do [ "${TIER_ACT[i]}" = reuse ] || phases+=("tier-${TIER_NAMES[i]}"); done
+  else
+    phases+=(command)
+  fi
+  if [ "$RUN_TIMEOUT" -gt 0 ]; then
+    for p in "${phases[@]}"; do
+      if [ -f "$dir/phase.$p.start" ]; then i=$(( RUN_TIMEOUT - (now - $(cat "$dir/phase.$p.start")) )); [ "$i" -gt 0 ] || i=0
+      else i=$RUN_TIMEOUT; fi
+      need=$(( need + i ))
+    done
+  fi
+  TTL_WHY="ttl_left=${left}s needs=$(( need + REATTACH_MARGIN ))s (run still to do ${need}s + UBI_REATTACH_MARGIN=${REATTACH_MARGIN}s)"
+  [ "$left" -lt $(( need + REATTACH_MARGIN )) ]
+}
+
+# find_record: the VM of a run record with this RUN_KEY whose VM still exists.
+find_record() {
+  local f n
+  for f in "$STATE_ROOT"/*/run; do
+    [ -f "$f" ] || continue
+    grep -qx "RUN_KEY=$RUN_KEY" "$f" || continue
+    n=$(basename "$(dirname "$f")")
+    if vm_alive "$n"; then echo "$n"; return 0; fi
+  done
+}
+
+# Tier checkpoints: write-once records <tier>.<epoch>.<pid>.json beside a copy
+# of the tier's log, under CHECKPOINT_DIR/<run key>/, published under a lock
+# with a monotonic generation; the newest generation per tier decides.
+CKPT_PY='
+import fcntl, hashlib, json, os, shutil, sys, time
+op, d = sys.argv[1], sys.argv[2]
+def records():
+    out = []
+    for f in (os.listdir(d) if os.path.isdir(d) else []):
+        if not f.endswith(".json") or f.startswith("."):
+            continue
+        try:
+            r = json.load(open(os.path.join(d, f)))
+        except (OSError, ValueError):
+            continue
+        if isinstance(r, dict) and isinstance(r.get("generation"), int):
+            r["_file"] = f
+            out.append(r)
+    return out
+def newest(rs, tier):
+    rs = [r for r in rs if r.get("tier") == tier]
+    return max(rs, key=lambda r: r["generation"]) if rs else None
+def sha(p):
+    h = hashlib.sha256()
+    with open(p, "rb") as fh:
+        for b in iter(lambda: fh.read(65536), b""):
+            h.update(b)
+    return h.hexdigest()
+if op == "decide":
+    now, maxage, resume, rerun = int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", sys.argv[6] == "1"
+    rs = records()
+    for tier in sys.argv[7:]:
+        r = newest(rs, tier)
+        f = r["_file"] if r else "-"
+        age = now - int(r.get("finished_at") or r.get("started_at") or now) if r else 0
+        dur = int(r.get("duration_s") or 0) if r else 0
+        code = str(r.get("exit", "-")) if r else "-"
+        log = os.path.join(d, str(r.get("log"))) if r and r.get("log") else ""
+        if not resume:
+            act, why = "run", "--no-resume"
+        elif r is None:
+            act, why = "run", "no record"
+        elif r.get("status") == "start":
+            act, why = "run", "interrupted (only a start record, %s)" % f
+        elif r.get("status") == "pass":
+            if age >= maxage:
+                act, why = "run", "pass %s is %ds old (UBI_CHECKPOINT_MAX_AGE=%d)" % (f, age, maxage)
+            elif not log or not os.path.isfile(log) or sha(log) != r.get("log_sha256"):
+                act, why = "run", "pass %s: its log does not re-hash" % f
+            else:
+                act, why = "reuse", "-"
+        elif r.get("status") == "fail":
+            act, why = ("rerun", "prior fail %s (exit %s; --rerun-failed)" % (f, code)) if rerun else ("blocked", "-")
+        else:
+            act, why = "run", "unreadable record %s" % f
+        print("\t".join([tier, act, f, str(age), str(dur), code, why]))
+elif op == "publish":
+    tier, status, vm, started, finished, code, log, startgen, key, tree, pid = sys.argv[3:14]
+    os.makedirs(d, exist_ok=True)
+    with open(os.path.join(d, ".lock"), "a") as lk:
+        fcntl.flock(lk, fcntl.LOCK_EX)
+        rs = records()
+        gen = max([r["generation"] for r in rs] + [0]) + 1
+        if status != "start":
+            r = newest(rs, tier)
+            if r is None or str(r["generation"]) != startgen:
+                print("stale writer: its start record is generation %s, the newest for %s is %s" % (startgen, tier, r["generation"] if r else "none"), file=sys.stderr)
+                sys.exit(2)
+        rec = {"tier": tier, "status": status, "generation": gen, "vm": vm, "key": key, "tree": tree, "started_at": int(started)}
+        if status != "start":
+            rec.update(finished_at=int(finished), duration_s=int(finished) - int(started), exit=int(code))
+        epoch = int(time.time())
+        while os.path.exists(os.path.join(d, "%s.%d.%s.json" % (tier, epoch, pid))) or os.path.exists(os.path.join(d, "%s.%d.%s.log" % (tier, epoch, pid))):
+            epoch += 1
+        stem = "%s.%d.%s" % (tier, epoch, pid)
+        if status != "start":
+            shutil.copyfile(log, os.path.join(d, stem + ".log"))
+            rec["log"], rec["log_sha256"] = stem + ".log", sha(os.path.join(d, stem + ".log"))
+        tmp = os.path.join(d, ".%s.tmp" % stem)
+        with open(tmp, "w") as fh:
+            json.dump(rec, fh, sort_keys=True)
+            fh.write("\n")
+        os.link(tmp, os.path.join(d, stem + ".json"))
+        os.unlink(tmp)
+        print(gen)
+'
+
+# ckpt_publish TIER STATUS VM STARTED FINISHED EXIT LOG STARTGEN: print the new
+# record's generation; fails (nothing published) for a stale writer.
+ckpt_publish() { python3 -c "$CKPT_PY" publish "$CHECKPOINT_DIR/$RUN_KEY" "$@" "$RUN_KEY" "$TREE" "$$"; }
+
+TIER_ACT=() TIER_TO_RUN=0 TIER_PASSED=0 TIER_REUSED="" TIER_FAILED="" TIER_PRIOR="" TIER_SAVED=0 TIER_BLOCKED="" TIER_BLOCKED_RC=1
+
+# tier_plan: decide each tier from its newest checkpoint and print one
+# UBI_VERIFY_RESUME line per tier before anything runs.
+tier_plan() {
+  local plan t act f age dur code why
+  plan=$(python3 -c "$CKPT_PY" decide "$CHECKPOINT_DIR/$RUN_KEY" "$(date +%s)" "$CHECKPOINT_MAX_AGE" "$RESUME" "$RERUN_FAILED" "${TIER_NAMES[@]}") \
+    || die "verify: could not read the tier checkpoints in $CHECKPOINT_DIR/$RUN_KEY"
+  while IFS=$'\t' read -r t act f age dur code why; do
+    case $act in
+      reuse)
+        echo "UBI_VERIFY_RESUME: tier=$t reuse $f (${age}s)" >&2
+        TIER_REUSED=${TIER_REUSED:+$TIER_REUSED,}$t TIER_SAVED=$(( TIER_SAVED + dur )) TIER_PASSED=$(( TIER_PASSED + 1 )) ;;
+      blocked)
+        echo "UBI_VERIFY_RESUME: tier=$t fail $f (${age}s, exit $code): not rerun without --rerun-failed" >&2
+        if [ -z "$TIER_BLOCKED" ]; then TIER_BLOCKED=$t; if [[ "$code" =~ ^[0-9]+$ ]]; then TIER_BLOCKED_RC=$code; fi; fi ;;
+      rerun)
+        echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
+        TIER_PRIOR=${TIER_PRIOR:+$TIER_PRIOR,}$t:$code TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
+      *)
+        echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
+        TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
+    esac
+    TIER_ACT+=("$act")
+  done <<<"$plan"
+}
+
+verify_line() {
+  echo "UBI_VERIFY: tree=${TREE:0:12} key=${RUN_KEY:0:12} tiers=${#TIER_NAMES[@]} passed=$TIER_PASSED reused=${TIER_REUSED:-none} failed=${TIER_FAILED:-none} prior_failures=${TIER_PRIOR:-none} saved_s=$TIER_SAVED"
+}
+
+# run_tiers NAME QDEST: run every tier the plan did not reuse, in order, each
+# as its own detached phase; stop at the first failure and return its status.
+# A pass is recorded only when the VM's tree after the tier equals the tree
+# before it.
+run_tiers() {
+  local name=$1 qdest=$2 dir i t phase before after trc started finished gen
+  dir=$(state_dir "$name")
+  for i in "${!TIER_NAMES[@]}"; do
+    [ "${TIER_ACT[i]}" != reuse ] || continue
+    t=${TIER_NAMES[i]} phase="tier-${TIER_NAMES[i]}"
+    if [ ! -f "$dir/phase.$phase.start" ]; then
+      before=$(vm_tree "$name" "$qdest") || before=unavailable
+      echo "$before" >"$dir/phase.$phase.tree"
+      if gen=$(ckpt_publish "$t" start "$name" "$(date +%s)" 0 0 "" 0); then echo "$gen" >"$dir/phase.$phase.gen"
+      else echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t start record (the tier still runs)" >&2; fi
+    fi
+    log "running tier $t on $name: ${TIER_CMDS[i]}"
+    trc=0
+    detached "$name" "$phase" "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "${TIER_CMDS[i]}")" || trc=$?
+    started=$(cat "$dir/phase.$phase.start" 2>/dev/null || date +%s) finished=$(date +%s)
+    log "tier $t exited $trc after $(( finished - started ))s"
+    case $trc in 125|255) TIER_FAILED=$t; return "$trc" ;; esac
+    gen=$(cat "$dir/phase.$phase.gen" 2>/dev/null || echo 0)
+    if [ "$trc" -ne 0 ]; then
+      ckpt_publish "$t" fail "$name" "$started" "$finished" "$trc" "$dir/$phase.log" "$gen" >/dev/null \
+        || echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t fail record (the result stands)" >&2
+      TIER_FAILED=$t
+      return "$trc"
+    fi
+    TIER_PASSED=$(( TIER_PASSED + 1 ))
+    after=$(vm_tree "$name" "$qdest") || after=unavailable
+    before=$(cat "$dir/phase.$phase.tree" 2>/dev/null || echo unavailable)
+    if [ "$after" != "$before" ] || [ "$after" = unavailable ]; then
+      echo "UBI_VERIFY_TREE_MOVED: tier=$t before=${before:0:12} after=${after:0:12}: a tier must leave the source tree as it found it; no pass record" >&2
+      continue
+    fi
+    ckpt_publish "$t" pass "$name" "$started" "$finished" 0 "$dir/$phase.log" "$gen" >/dev/null \
+      || echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t pass record (the result stands)" >&2
+  done
+}
+
 # vm_rows: "location name owner expires created" for every VM in the project;
 # owner is "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner
 # did not name; expires is the epoch the name's -t<hours> TTL ends at, or "-";
@@ -770,9 +1207,11 @@ run_teardown() {
  [ "$rc" -ne 0 ] || exit 1
 }

+TIER_NAMES=() TIER_CMDS=() RESUME=1 RERUN_FAILED=0 FORCE_ATTACH="" RUN_ARGS=()
 cmd_run() {
  require_owner
-  local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=()
+  local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() passes=() key_inputs=() probe=0
+  RUN_ARGS=("$@")
  while [ $# -gt 0 ]; do
    case $1 in
      -S|--storage) check_storage "$2"; up_args+=("$1" "$2"); shift 2 ;;
@@ -781,60 +1220,161 @@ cmd_run() {
      --dest) dest=$2; shift 2 ;;
      --setup) setup=$2; shift 2 ;;
      --env) envs+=("$2"); shift 2 ;;
-      --pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; envs+=("$2=${!2}"); shift 2 ;;
+      --pass) pass_name "$2"; [ -n "${!2+x}" ] || die "--pass $2: not set locally"; passes+=("$2"); shift 2 ;;
      --pull) pulls+=("$2"); shift 2 ;;
      --keep) keep=1; shift ;;
+      --key-input) key_inputs+=("$2"); shift 2 ;;
+      --probe) probe=1; shift ;;
+      --resume) RESUME=1; shift ;;
+      --no-resume) RESUME=0; shift ;;
+      --rerun-failed) RERUN_FAILED=1; shift ;;
      --) shift; break ;;
      *) die "run: unknown option $1" ;;
    esac
  done
-  [ $# -gt 0 ] || die "run: missing command after --"
+  [ $# -gt 0 ] || [ ${#TIER_NAMES[@]} -gt 0 ] || die "run: missing command after --"
  [ -z "$setup" ] || [ -f "$setup" ] || die "setup script not found: $setup"
+  command -v git >/dev/null || die "git is required: the run key fingerprints the checkout with it"
  src=$(cd "$src" && pwd)
  [ -n "$dest" ] || dest="work/$(basename "$src")"

-  local i name=""
+  local i name="" size=$DEFAULT_SIZE location=$DEFAULT_LOCATION image=ubuntu-noble storage="" e n
  for (( i = 0; i < ${#up_args[@]}; i += 2 )); do
-    case ${up_args[i]} in -n|--name) name=${up_args[i+1]} ;; esac
+    case ${up_args[i]} in
+      -n|--name) name=${up_args[i+1]} ;;
+      -s|--size) size=${up_args[i+1]} ;;
+      -l|--location) location=${up_args[i+1]} ;;
+      -b|--image) image=${up_args[i+1]} ;;
+      -S|--storage) storage=${up_args[i+1]} ;;
+    esac
  done
+  # The run key: everything that decides what the VM runs, computed before any
+  # VM exists. Pass values enter only as their sha256.
+  if [ -z "$FORCE_ATTACH" ]; then
+    TREE=$(cd "$src" && bash -c "$TREE_SH") || die "could not fingerprint $src (git add -A into a temporary index failed)"
+    RUN_KEY=$({
+      echo "ubi-runner run key v1"
+      echo "tree $TREE"
+      printf 'command %q\n' "$*"
+      if [ -n "$setup" ]; then echo "setup $(sha256 <"$setup")"; else echo "setup none"; fi
+      for e in ${envs[@]+"${envs[@]}"}; do printf 'env %q\n' "$e"; done
+      for n in ${passes[@]+"${passes[@]}"}; do printf 'pass %s %s\n' "$n" "$(printf %s "${!n}" | sha256)"; done
+      printf 'vm %q %q %q %q\n' "$size" "$location" "$image" "$storage"
+      echo "base $(base_sha "$src")"
+      echo "runner $(sha256 <"${BASH_SOURCE[0]}")"
+      for e in ${key_inputs[@]+"${key_inputs[@]}"}; do printf 'input %q\n' "$e"; done
+      for i in "${!TIER_NAMES[@]}"; do printf 'tier %q %q\n' "${TIER_NAMES[i]}" "${TIER_CMDS[i]}"; done
+    } | sha256)
+  fi
+  [ ${#TIER_NAMES[@]} -eq 0 ] || tier_plan
+
+  local attach=""
+  if [ -n "$FORCE_ATTACH" ]; then
+    vm_alive "$FORCE_ATTACH" || die "attach: $FORCE_ATTACH no longer exists; rerun the original command to start fresh"
+    attach=$FORCE_ATTACH
+  else
+    attach=$(find_record)
+  fi
+  if [ "$probe" = 1 ]; then
+    if [ -z "$attach" ]; then echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12}"
+    else
+      local busy=0
+      ! lease_busy "$(state_dir "$attach")" || busy=1
+      if [ "$busy" = 0 ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
+        echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12} ttl_short vm=$attach $TTL_WHY"
+      else
+        echo "UBI_ATTACH_INFO: vm=$attach created_at=$(load "$attach"; echo "$CREATE_AT") key=${RUN_KEY:0:12} busy=$busy"
+      fi
+    fi
+    return 0
+  fi
+  if [ ${#TIER_NAMES[@]} -gt 0 ] && { [ -n "$TIER_BLOCKED" ] || [ "$TIER_TO_RUN" -eq 0 ]; } && [ -n "$attach" ] && ! lease_busy "$(state_dir "$attach")"; then
+    log "verify: no tier needs $attach, which a dead runner left behind; destroying it"
+    cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
+  fi
+  if [ ${#TIER_NAMES[@]} -gt 0 ] && [ -n "$TIER_BLOCKED" ]; then
+    TIER_FAILED=$TIER_BLOCKED
+    verify_line
+    return "$TIER_BLOCKED_RC"
+  fi
+  if [ ${#TIER_NAMES[@]} -gt 0 ] && [ "$TIER_TO_RUN" -eq 0 ]; then
+    log "verify: every tier passed under this key; no VM needed"
+    verify_line
+    return 0
+  fi
+  if [ -n "$attach" ] && lease_busy "$(state_dir "$attach")"; then
+    die "UBI_ATTACH_BUSY: vm=$attach key=${RUN_KEY:0:12}: runner pid $(cut -d' ' -f1 <"$(state_dir "$attach")/lease") on this machine still owns it; wait for it or stop it, then rerun"
+  fi
+  if [ -n "$attach" ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
+    log "UBI_REATTACH_TTL_SHORT: vm=$attach key=${RUN_KEY:0:12} $TTL_WHY: starting a fresh VM (tier checkpoints carry over) and destroying $attach"
+    cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
+    attach=""
+  fi
+  [ -z "$attach" ] || name=$attach
  [ -n "$name" ] || { name=$(new_name); up_args+=(-n "$name"); }
  RUN_VM=$name
  # Armed before the create request: an interrupted `up` still destroys its VM.
  # A second signal must not cut teardown short, so the EXIT handler ignores them.
  trap 'run_teardown "$?"' EXIT
-  trap 'exit 130' INT TERM HUP QUIT
-  cmd_up "${up_args[@]}" >/dev/null
-  if [[ "$name" =~ -t([0-9]+)$ ]]; then
-    log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
-  elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
-    log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
+  trap 'exit 130' INT TERM QUIT
+  trap on_hup HUP
+  if [ -n "$attach" ]; then
+    load "$name"
+    log "reattached to $name created_at=$CREATE_AT key=${RUN_KEY:0:12}"
  else
-    log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
+    cmd_up "${up_args[@]}" >/dev/null
+    trap 'exit 130' INT TERM QUIT
+    trap on_hup HUP
+    if [[ "$name" =~ -t([0-9]+)$ ]]; then
+      log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
+    elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
+      log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
+    else
+      log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
+    fi
  fi
+  local dir
+  dir=$(state_dir "$name")
+  write_record "$dir"
+  lease_take "$dir"
  if [ "$keep" = 1 ]; then
-    trap - EXIT
+    # A kept VM's finished run is not reattached by the next run; a detach (HUP) keeps the record.
+    trap 'rm -f "$(state_dir "$RUN_VM")/run"' EXIT
    leftover "left=1 owner=$OWNER vm=$name kept=1"
    log "--keep: leaving $name running; destroy with: $0 down $name"
  fi

-  local e
-  for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done \
-    | remote "$name" 'umask 077 && cat > ~/.ubirun-env'
-
-  cmd_sync "$name" "$src" "$dest"
-  local qdest rc=0
+  local qdest rc=0 vmt
  qdest=$(printf %q "$dest")
+  if [ ! -f "$dir/synced" ]; then
+    { for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done
+      for n in ${passes[@]+"${passes[@]}"}; do printf 'export %s=%q\n' "$n" "${!n}"; done; } \
+      | remote "$name" 'umask 077 && cat > ~/.ubirun-env'
+    cmd_sync "$name" "$src" "$dest"
+    # A subdirectory of a checkout uploads the whole .git with only its own
+    # files, so its tree is compared VM to VM (tiers) but not to this side.
+    if [ -z "$(git -C "$src" rev-parse --show-prefix 2>/dev/null)" ]; then
+      vmt=$(vm_tree "$name" "$qdest") || vmt=unavailable
+      if [ "$vmt" != "$TREE" ]; then
+        log "UBI_VM_TREE_MISMATCH: vm=$name tree=${vmt:0:12} expected=${TREE:0:12}: the unpacked checkout is not the one this run was keyed on"
+        exit 3
+      fi
+    fi
+    [ -z "$setup" ] || remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
+    : >"$dir/synced"
+  fi
  if [ -n "$setup" ]; then
-    remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
    log "running setup $(basename "$setup")"
-    watched "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
+    detached "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
  fi

-  local start
-  start=$(date +%s)
-  log "running on $name: $*"
-  watched "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
-  log "command exited $rc after $(( $(date +%s) - start ))s"
+  if [ ${#TIER_NAMES[@]} -gt 0 ]; then
+    run_tiers "$name" "$qdest" || rc=$?
+  else
+    log "running on $name: $*"
+    detached "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
+    log "command exited $rc after $(( $(date +%s) - $(cat "$dir/phase.command.start" 2>/dev/null || date +%s) ))s"
+  fi

  local p
  for p in ${pulls[@]+"${pulls[@]}"}; do
@@ -845,9 +1385,23 @@ cmd_run() {
    fi
    cmd_pull "$name" "${p%%:*}" "${p#*:}" || log "pull failed: $p"
  done
+  [ ${#TIER_NAMES[@]} -eq 0 ] || verify_line
  return "$rc"
 }

+# attach NAME: resume the run recorded for NAME on this machine (the same
+# invocation, key and tiers), as a rerun of the same command would.
+cmd_attach() {
+  local name=${1:-}
+  [ -n "$name" ] || die "attach: missing NAME"
+  valid_name "$name" || die "invalid VM name '$name'"
+  [ -f "$(state_dir "$name")/run" ] || die "attach: no run record for $name on this machine ($(state_dir "$name")/run); a machine replacement loses it"
+  # shellcheck disable=SC1091
+  . "$(state_dir "$name")/run"
+  FORCE_ATTACH=$name
+  cmd_run "${RUN_ARGS[@]}"
+}
+
 # verify [run opts] [--trust-config] [-- COMMAND]: run the repository's
 # committed remote verify. .gstack/ubicloud.json under --src (default .)
 # supplies setup, command, size, location, storage, image, env, pass and pull;
@@ -860,7 +1414,7 @@ cmd_run() {
 # local variable it lists under pass is forwarded only when this command line
 # also names it (--pass NAME) or grants every listed name (--trust-config).
 cmd_verify() {
-  local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=()
+  local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() cfg_tier_names=() cfg_tier_cmds=()
  for (( i = 1; i <= $#; i++ )); do
    [ "${!i}" = -- ] && break
    if [ "${!i}" = --src ]; then j=$(( i + 1 )); src=${!j}; fi
@@ -869,7 +1423,7 @@ cmd_verify() {
  [ -f "$cfg" ] || die "verify: no $cfg; commit one ({\"setup\": \"scripts/ubicloud/bootstrap.sh\", \"command\": \"bash scripts/ubicloud/verify.sh\"}) so every lane runs the same verify instead of its own wrapper"
  err=$(mktemp)
  parsed=$(python3 -c '
-import json, os, shlex, sys
+import json, os, re, shlex, sys
 cfg, src = sys.argv[1], sys.argv[2]
 try:
    c = json.load(open(cfg))
@@ -895,7 +1449,25 @@ for item in c.get("pull", []):
    if not sep or not remote:
        print("pull entry %r must be REMOTE_GLOB:LOCAL_DIR" % (item,), file=sys.stderr); sys.exit(2)
    args += ["--pull", q(remote + ":" + inside(local, "pull destination"))]
+tiers = c.get("tiers")
+names = []
+if tiers is not None:
+    if "command" in c:
+        print("UBI_CONFIG_TIERS_AND_COMMAND: give either command or tiers, not both", file=sys.stderr); sys.exit(2)
+    if not isinstance(tiers, list) or not tiers:
+        print("UBI_CONFIG_TIER_NAME: tiers must be a non-empty list of {\"name\", \"command\"}", file=sys.stderr); sys.exit(2)
+    for t in tiers:
+        name = t.get("name") if isinstance(t, dict) else None
+        if not isinstance(name, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,31}", name):
+            print("UBI_CONFIG_TIER_NAME: tier name %r must match ^[a-z0-9][a-z0-9_-]{0,31}$" % (name,), file=sys.stderr); sys.exit(2)
+        if name in names:
+            print("UBI_CONFIG_TIER_NAME: tier name %r appears twice" % name, file=sys.stderr); sys.exit(2)
+        if not isinstance(t.get("command"), str) or not t["command"].strip():
+            print("UBI_CONFIG_TIER_NAME: tier %r needs a command" % name, file=sys.stderr); sys.exit(2)
+        names.append(name)
 setup = c.get("setup", "")
+print("cfg_tier_names=(" + " ".join(q(n) for n in names) + ")")
+print("cfg_tier_cmds=(" + " ".join(q(t["command"]) for t in (tiers or [])) + ")")
 print("cfg_args=(" + " ".join(args) + ")")
 print("cfg_pass=(" + " ".join(q(x) for x in c.get("pass", [])) + ")")
 print("cfg_setup=" + q(inside(setup, "setup") if setup else ""))
@@ -912,7 +1484,7 @@ print("cfg_cmd=" + q(c.get("command", "")))
    case $1 in
      --) shift; user_cmd=("$@"); break ;;
      --trust-config) trust=1; shift ;;
-      --keep) user_pre+=("$1"); shift ;;
+      --keep|--resume|--no-resume|--rerun-failed|--probe) user_pre+=("$1"); shift ;;
      --pass|-S|--storage|-s|--size|-l|--location|-n|--name|-b|--image|--src|--dest|--setup|--env|--pull)
        [ $# -ge 2 ] || die "verify: $1 needs a value"
        [ "$1" != --pass ] || granted+=("$2")
@@ -927,11 +1499,16 @@ print("cfg_cmd=" + q(c.get("command", "")))
    missing+=("$n")
  done
  [ ${#missing[@]} -eq 0 ] || die "verify: $cfg asks to forward local ${missing[*]}; grant each on this command line with --pass NAME, or every name it lists with --trust-config"
-  [ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
-  [ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
+  # A trailing -- COMMAND replaces the configured tiers as it replaces command.
+  if [ ${#user_cmd[@]} -eq 0 ] && [ ${#cfg_tier_names[@]} -gt 0 ]; then
+    TIER_NAMES=("${cfg_tier_names[@]}") TIER_CMDS=("${cfg_tier_cmds[@]}")
+  else
+    [ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
+    [ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
+  fi
  log "verify: $cfg"
  PULL_ROOT=$(cd "$src" && pwd)
-  cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- "${user_cmd[@]}"
+  cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- ${user_cmd[@]+"${user_cmd[@]}"}
 }

 usage() {
@@ -939,13 +1516,17 @@ usage() {
 usage: ubi-runner.sh <command> [args]

  run [up opts] [--src DIR] [--dest PATH] [--setup FILE] [--env K=V]...
-      [--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] -- COMMAND
-                         up + sync + setup + command + pull + destroy; exits with COMMAND's status
-  verify [run opts] [--trust-config] [-- COMMAND]
+      [--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] [--key-input S]... [--probe] -- COMMAND
+                         up + sync + setup + command + pull + destroy; exits with COMMAND's status.
+                         A rerun of the same command (same run key) reattaches to its live VM;
+                         --probe prints UBI_ATTACH_INFO (and the tier plan) and exits
+  verify [run opts] [--trust-config] [--resume|--no-resume] [--rerun-failed] [-- COMMAND]
                         run the repository's committed verify (.gstack/ubicloud.json under --src);
                         its setup script and every pull destination stay inside --src (checked
                         again at pull time), and its pass names are forwarded only when granted
-                         here (--pass NAME, or all: --trust-config)
+                         here (--pass NAME, or all: --trust-config). With tiers, a pass checkpoint
+                         under the same run key is reused (resume is the default)
+  attach NAME            resume the run recorded for NAME on this machine, as a rerun would
  up [-s SIZE] [-l LOCATION] [-n NAME] [-S STORAGE_GIB] [-b IMAGE]
                         create a VM and wait for SSH; prints its name
                         (STORAGE_GIB: one of $ALLOWED_STORAGE)
@@ -965,6 +1546,11 @@ usage: ubi-runner.sh <command> [args]
                         owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS
  cli ARGS...            raw Ubicloud CLI passthrough (e.g. cli vm list)

+signals: INT and TERM tear the VM down. HUP, once a remote phase is live, detaches
+(UBI_DETACHED, exit 129; the VM runs on under its TTL). SIGKILL or a machine stop leaves
+the run record and the VM; rerun the same command to reattach (UBI_ATTACH_BUSY while
+another runner owns it; exit 125 UBI_ATTACH_LOST when the remote session left no status).
+
 VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
 UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left=<n> line.
 defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION)
@@ -976,7 +1562,8 @@ env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the
     UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing),
     UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s),
     UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file),
-     UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640")
+     UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"),
+     UBI_CHECKPOINT_DIR (tier checkpoints; $CHECKPOINT_DIR), UBI_CHECKPOINT_MAX_AGE (${CHECKPOINT_MAX_AGE}s a pass is reused)
 EOF
 }

@@ -989,6 +1576,7 @@ esac
 case $cmd in
  run) cmd_run "$@" ;;
  verify) cmd_verify "$@" ;;
+  attach) cmd_attach "$@" ;;
  up) trap 'trap "" INT TERM HUP QUIT; [ -z "$UP_NAME" ] || [ -n "$UP_READY" ] || cmd_down "$UP_NAME" || log "WARNING: failed to destroy $UP_NAME; run: $0 down $UP_NAME"' EXIT
      trap 'exit 130' INT TERM HUP QUIT
      cmd_up "$@" ;;
@@ -37,12 +37,22 @@ const UBI_RUNNER_STUB = `#!/usr/bin/env bash
 # VM directory, runs setup and the command there, and prints ubi-runner's log
 # lines. STUB_FAIL picks a failure: create, quota, quota-once, ssh. STUB_HANG
 # sleeps before the command; TERM tears down as ubi-runner's trap does.
-# Every teardown is appended to $STUB_CALLS.down.
+# Every teardown is appended to $STUB_CALLS.down. STUB_REATTACH=<vm>:<created_at>
+# plays a live run record for this key: --probe reports it (busy per
+# STUB_REATTACH_BUSY) and run prints ubi-runner's reattached line instead of
+# creating a VM.
 set -uo pipefail
 [ "$1" != owner ] || { echo stub; exit 0; }
 left() { [ "\${STUB_LEFT:-0}" = none ] || echo "UBICLOUD: left=\${STUB_LEFT:-0} owner=stub vm=$name others=0" | tee -a "\${UBI_LEFT_FILE:-/dev/null}" >&2; }
 [ "$1" = run ] || exit 2
 shift
+case " $* " in
+  *" --probe "*)
+    printf '%s\\n' "$*" >>"$STUB_CALLS.probe"
+    if [ -n "\${STUB_REATTACH:-}" ]; then echo "UBI_ATTACH_INFO: vm=\${STUB_REATTACH%%:*} created_at=\${STUB_REATTACH#*:} key=0123456789ab busy=\${STUB_REATTACH_BUSY:-0}"
+    else echo "UBI_ATTACH_INFO: none key=0123456789ab"; fi
+    exit 0 ;;
+esac
 printf '%s\\n' "$*" >>"$STUB_CALLS"
 src=. setup="" size="" name=unnamed envs=()
 while [ $# -gt 0 ]; do
@@ -52,16 +62,20 @@ while [ $# -gt 0 ]; do
    --src) src=$2; shift 2 ;;
    --setup) setup=$2; shift 2 ;;
    --env) envs+=("$2"); shift 2 ;;
-    --pull) shift 2 ;;
+    --pull|--key-input) shift 2 ;;
    --) shift; break ;;
    *) echo "stub: unknown option $1" >&2; exit 2 ;;
  esac
 done
 cmd=$*
 log() { echo "ubi-runner: $*" >&2; }
-log "creating eu-central-h1/$name ($size)"
 calls=$(wc -l <"$STUB_CALLS")
 fail=\${STUB_FAIL:-}
+if [ -n "\${STUB_REATTACH:-}" ]; then
+  name=\${STUB_REATTACH%%:*}
+  log "reattached to $name created_at=\${STUB_REATTACH#*:} key=0123456789ab"
+else
+log "creating eu-central-h1/$name ($size)"
 if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; then
  log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64"
  log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
@@ -77,6 +91,7 @@ if [ "$fail" = create ]; then
  exit 1
 fi
 log "ready: $name (192.0.2.1)"
+fi
 trap 'log "destroyed eu-central-h1/$name"; left; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
 vm=$STUB_VM_HOME
 dest="$vm/work/$(basename "$src")"
@@ -282,6 +297,36 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
    expect(twice.out).toContain('create failed: vCPU quota exhausted');
  }, 90_000);

+  test('a rerun for the same tree and planned set reattaches: the receipt still parses, spent= bills from the original created_at, and a used-up wall budget refuses', () => {
+    const created = Math.floor(Date.now() / 1000) - 5 * 60 - 10;
+    const f = fixture();
+    const r = gate(f, ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}` });
+    expect(r.exit, r.out).toBe(0);
+    expect(r.out).toContain('reattaching to VM ubirun-stub-old, created 6 min ago by an earlier run of this gate for the same tree and planned set');
+    expect(r.out).toContain("the VM is told to stop after 2160s (the cap's 45 min wall minus 3 min for teardown and 6 min already billed since ubirun-stub-old was created)");
+    expect(r.out).not.toContain('if this run is killed before its teardown line');
+    expect(r.out.trim().split('\n').at(-1)).toMatch(/^PREGATE_REMOTE: /);
+    expect(r.report).toMatchObject({ tree: f.tree, tests: 1, failed: 0 });
+    // Billed from created_at (5 min 10 s before the gate started), not from the reattach: 6 min at $0.00831.
+    expect(r.report!.spent).toBe('0.05');
+    expect(remoteReportProblem(r.report!, f.tree)).toBeNull();
+    const plan = new RegExp(`--key-input pregate-remote tree=${f.tree} plan=[0-9a-f]{64}`);
+    expect(r.calls[0]).toMatch(plan);
+    expect(fs.readFileSync(f.calls + '.probe', 'utf8')).toMatch(plan);
+
+    const spent = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created - 40 * 60}` });
+    expect(spent.exit, spent.out).toBe(5);
+    expect(spent.out).toContain("PREGATE_REMOTE_CAP_SPENT: VM ubirun-stub-old, created 46 min ago for this tree and planned set, has already used the cap's 45 min wall ($0.39 of $2)");
+    expect(spent.out).toContain('nothing reattached; destroy it: bash scripts/ubicloud/ubi-runner.sh down ubirun-stub-old');
+    expect(spent.report).toBeNull();
+    expect(spent.calls).toEqual([]);
+
+    const busy = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_REATTACH: `ubirun-stub-old:${created}`, STUB_REATTACH_BUSY: '1' });
+    expect(busy.exit, busy.out).toBe(4);
+    expect(busy.out).toContain('PREGATE_REMOTE_ATTACH_BUSY: another gate on this machine still runs VM ubirun-stub-old');
+    expect(busy.calls).toEqual([]);
+  }, 90_000);
+
  test('the VM refuses a tree that differs after unpack, a planned set that differs from the list, and a tree that moved during the run', () => {
    const edited = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_VM_EDIT: 'test/alpha.test.ts' });
    expect(edited.exit, edited.out).toBe(3);
@@ -389,18 +389,23 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
  /**
   * Fake Ubicloud API that can create: create adds the name to the listed file
   * (or answers the quota refusal in refuse with HTTP 400), show is running at
-   * 192.0.2.1 while listed, destroy unlists. The fake ssh records every remote
-   * command in ssh.log and plays the role the command asks for: it stores
-   * ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, drains tar
-   * uploads, runs the setup phase per setup-mode (ok or hang) and echoes the
-   * command phase. No network, no VM.
+   * 192.0.2.1 while listed, destroy unlists. The fake VM is the directory vm/:
+   * the fake ssh records every remote command in ssh.log, keeps copies of
+   * ~/.ubirun-env and ~/.ubirun-setup.sh under the fixture root, serves pulls
+   * from remote-fs/, and runs every other command there for real (HOME=vm/,
+   * vmbin/ first on PATH), so uploads unpack, detached phases write their pid,
+   * output and exit files, tail --pid follows them and pkill ends them. The stub
+   * make in vmbin/ echoes `ran: make ARGS`, or runs command-mode or
+   * make-<target>.sh from the fixture root. No network, no VM.
   */
  function fakeVm(opts: { listed?: string; refuse?: string } = {}) {
    const root = mkdtempSync(join(tmpdir(), 'ubi-run-'));
    const bin = join(root, 'bin'), state = join(root, 'state'), log = join(root, 'calls.log'), sshLog = join(root, 'ssh.log'), record = join(root, 'vms.tsv');
+    const vm = join(root, 'vm'), vmbin = join(root, 'vmbin'), srcDir = join(root, 'src'), checkpoints = join(root, 'checkpoints');
+    mkdirSync(vm); mkdirSync(vmbin); mkdirSync(srcDir);
+    writeFileSync(join(srcDir, 'README'), 'fixture checkout\n');
    mkdirSync(bin); mkdirSync(state, { recursive: true });
    writeFileSync(join(root, 'listed'), opts.listed ?? '');
-    writeFileSync(join(root, 'setup-mode'), 'ok');
    writeFileSync(join(bin, 'curl'), '#!/usr/bin/env bash\n' +
      'out=""; while [ $# -gt 0 ]; do [ "$1" = -o ] && out=$2; shift; done\n' +
      'body=$(cat); echo "$body" >> ' + JSON.stringify(log) + '\n' +
@@ -415,26 +420,33 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
      '  *) : > "$out"; printf 200 ;;\n' +
      'esac\n');
    writeFileSync(join(bin, 'ssh'), '#!/usr/bin/env bash\n' +
-      'root=' + JSON.stringify(root) + '\n' +
+      'root=' + JSON.stringify(root) + '; vm="$root/vm"\n' +
      'cmd=${*: -1}\n' +
      'printf \'%s\\n\' "$cmd" >> "$root/ssh.log"\n' +
+      'export HOME="$vm" PATH="$root/vmbin:$PATH"\n' +
+      'cd "$vm" || exit 255\n' +
      'case "$cmd" in\n' +
-      '  *pkill*) echo "$cmd" >> "$root/cancel.log" ;;\n' +
+      '  *pkill*) echo "$cmd" >> "$root/cancel.log"; exec bash -c "$cmd" ;;\n' +
      '  *captured*) cat > "$root/captured" ;;\n' +
-      '  *"cat > ~/.ubirun-env"*) cat > "$root/ubirun-env" ;;\n' +
-      '  *"cat > ~/.ubirun-setup.sh"*) cat > "$root/ubirun-setup.sh" ;;\n' +
-      '  *"tar -xzf -"*) cat > /dev/null ;;\n' +
+      '  *"cat > ~/.ubirun-env"*) tee "$root/ubirun-env" > "$vm/.ubirun-env" ;;\n' +
+      '  *"cat > ~/.ubirun-setup.sh"*) tee "$root/ubirun-setup.sh" > "$vm/.ubirun-setup.sh" ;;\n' +
      '  *"tar -czf -"*) base=${cmd##* }; tar -czf - -C "$root/remote-fs" "$base" ;;\n' +
-      '  *"bash -l ~/.ubirun-setup.sh"*) echo "setup start"; if [ "$(cat "$root/setup-mode")" = hang ]; then sleep 20; fi; echo "setup done" ;;\n' +
-      '  *"bash -lc "*) if [ -f "$root/command-mode" ]; then bash -c "$(cat "$root/command-mode")"; else echo "ran: $cmd"; fi ;;\n' +
-      '  *) : ;;\n' +
+      '  *"ls -d -- "*) : ;;\n' +
+      '  *) exec bash -c "$cmd" ;;\n' +
      'esac\n');
-    chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755);
-    const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0' };
+    writeFileSync(join(vmbin, 'make'), '#!/usr/bin/env bash\n' +
+      'root=' + JSON.stringify(root) + '\n' +
+      'if [ -f "$root/command-mode" ]; then exec bash -c "$(cat "$root/command-mode")"; fi\n' +
+      'echo "ran: make $*"\n' +
+      '[ ! -f "$root/make-$1.sh" ] || exec bash "$root/make-$1.sh"\n');
+    chmodSync(join(bin, 'curl'), 0o755); chmodSync(join(bin, 'ssh'), 0o755); chmodSync(join(vmbin, 'make'), 0o755);
+    const env: Record<string, string> = { ...(process.env as Record<string, string>), PATH: `${bin}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: state, UBI_POLL_SECONDS: '1', UBI_VM_RECORD: record, UBI_OWNER: 'thrd1', UBI_HEARTBEAT: '0', UBI_CHECKPOINT_DIR: checkpoints };
    const run = (args: string[], extra: Record<string, string> = {}, stdin?: string) =>
      Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: stdin === undefined ? undefined : Buffer.from(stdin), timeout: 90_000 });
+    const spawn = (args: string[], extra: Record<string, string> = {}) =>
+      Bun.spawn(['bash', join(DIR, 'ubi-runner.sh'), ...args], { env: { ...env, ...extra }, stdin: 'ignore', stdout: 'pipe', stderr: 'pipe', timeout: 120_000 });
    return {
-      root, state, record, run,
+      root, state, record, run, spawn, vm, src: srcDir, checkpoints,
      calls: () => (existsSync(log) ? readFileSync(log, 'utf8') : ''),
      ssh: () => (existsSync(sshLog) ? readFileSync(sshLog, 'utf8') : ''),
      listed: () => readFileSync(join(root, 'listed'), 'utf8'),
@@ -455,17 +467,18 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
  test('run: a hung setup is killed by the watchdog, exits 124, never reaches the command, and the VM is destroyed', () => {
    const f = fakeVm();
    try {
-      writeFileSync(join(f.root, 'setup-mode'), 'hang');
-      const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo hi\n');
-      const r = run(f, ['run', '--src', f.root, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
+      const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo "setup start"; sleep 20\n');
+      const r = run(f, ['run', '--src', f.src, '--setup', setup, '--', 'make test'], { UBI_IDLE_TIMEOUT: '2' });
      const err = r.stderr.toString();
      expect(r.exitCode, err).toBe(124);
      expect(err).toMatch(/watchdog: setup printed nothing for \d+s \(UBI_IDLE_TIMEOUT=2\)/);
      expect(err).toContain('setup failed (exit 124)');
      expect(f.ssh()).not.toContain('bash -lc');
-      expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/cat ~\/\.ubirun-setup-\d+\.sid/);
-      expect(f.ssh()).toMatch(/setsid -w bash -c 'printf %s \$\$ >~\/\.ubirun-setup-\d+\.sid; cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh'; else bash -c '[^']*'; fi$/m);
-      expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/rm -f ~\/\.ubirun-setup-\d+\.sid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
+      expect(err).toContain('setup start');
+      // The setup runs detached in its own session: its leader's pid, both streams and the exit status are files on the VM.
+      expect(f.ssh()).toMatch(/setsid -f bash -c 'echo \$\$ >~\/\.ubirun-setup\.pid\.tmp && mv ~\/\.ubirun-setup\.pid\.tmp ~\/\.ubirun-setup\.pid; \( cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh \) >~\/\.ubirun-setup\.out 2>~\/\.ubirun-setup\.err <\/dev\/null; echo \$\? >~\/\.ubirun-setup\.exit\.tmp && mv ~\/\.ubirun-setup\.exit\.tmp ~\/\.ubirun-setup\.exit' <\/dev\/null/m);
+      expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/sid=\$\(cat ~\/\.ubirun-setup\.pid 2>\/dev\/null\); rm -f ~\/\.ubirun-setup\.pid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
+      expect(existsSync(join(f.vm, '.ubirun-setup.exit'))).toBe(false);
      expect(f.calls()).toContain('destroy');
      expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3/);
      expect(f.listed()).toBe('');
@@ -477,7 +490,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
    const f = fakeVm();
    try {
      writeFileSync(join(f.root, 'command-mode'), 'for i in 1 2 3 4 5 6 7 8 9 10; do echo "tick $i"; sleep 1; done');
-      const r = run(f, ['run', '--src', f.root, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
+      const r = run(f, ['run', '--src', f.src, '--', 'make test'], { UBI_RUN_TIMEOUT: '3' });
      expect(r.exitCode).toBe(124);
      expect(r.stderr.toString()).toMatch(/watchdog: command ran \d+s \(UBI_RUN_TIMEOUT=3\); killing it/);
      expect(r.stderr.toString()).toContain('command exited 124 after');
@@ -489,7 +502,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
    const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n' });
    try {
      const leftFile = join(f.root, 'left');
-      const r = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
+      const r = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
      const err = r.stderr.toString();
      expect(r.exitCode, err).toBe(0);
      expect(err).toMatch(/UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
@@ -499,7 +512,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
      const parser = read('scripts/ubicloud/pregate-remote.sh').match(/left=\$\(sed -n '([^']+)' "\$log\.left"/)![1];
      expect(Bun.spawnSync(['sed', '-n', parser, leftFile], { timeout: 30_000 }).stdout.toString()).toBe('0\n');
      // Value: protects=run --keep reports its live VM as UBICLOUD: left=1 ... kept=1 and never a clean left=0; fails_when=the kept line is dropped or --keep still prints the teardown line; why_new=no test runs run --keep; seam=none; no_claim=real VM lifetime (fake API and ssh)
-      const kept = f.run(['run', '--src', f.root, '--keep', '--', 'true'], { UBI_REAP: '0' });
+      const kept = f.run(['run', '--src', f.src, '--keep', '--', 'true'], { UBI_REAP: '0' });
      const keptErr = kept.stderr.toString();
      expect(kept.exitCode, keptErr).toBe(0);
      expect(keptErr).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 kept=1\n/);
@@ -512,14 +525,14 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
    const f = fakeVm();
    try {
      writeFileSync(join(f.root, 'listed.destroy-fails'), '');
-      const leaked = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
+      const leaked = f.run(['run', '--src', f.src, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
      const err = leaked.stderr.toString();
      expect(leaked.exitCode, err).toBe(1);
      expect(err).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=(ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3) others=0\n[^]*VM cleanup not confirmed for \1 \(left=1\); destroy it with: \S+ down \1/);
      expect(f.listed()).toContain('ubirun-thrd1-');
      // Value: protects=a failing command whose VM also leaks keeps its own exit status, not 1; fails_when=run_teardown exits 1 whenever left is not 0; why_new=the leak case above only runs a passing command; seam=none; no_claim=real destroy failure modes (fake API returns 500)
      writeFileSync(join(f.root, 'command-mode'), 'exit 7');
-      const failedLeak = f.run(['run', '--src', f.root, '--', 'false'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
+      const failedLeak = f.run(['run', '--src', f.src, '--', 'make test'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
      expect(failedLeak.exitCode, failedLeak.stderr.toString()).toBe(7);
      expect(failedLeak.stderr.toString()).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
      const blind = fakeVm();
@@ -529,7 +542,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
        writeFileSync(join(blind.root, 'bin/curl'), '#!/usr/bin/env bash\nif [ -f ' + JSON.stringify(join(blind.root, 'armed')) + ' ]; then touch ' + JSON.stringify(join(blind.root, 'listed.list-fails')) + '; fi\nexec ' + JSON.stringify(join(blind.root, 'bin/curl.real')) + ' "$@"\n');
        chmodSync(join(blind.root, 'bin/curl'), 0o755);
        writeFileSync(join(blind.root, 'command-mode'), 'touch ' + JSON.stringify(join(blind.root, 'armed')));
-        const r = blind.run(['run', '--src', blind.root, '--', 'true'], { UBI_REAP: '0' });
+        const r = blind.run(['run', '--src', blind.src, '--', 'make test'], { UBI_REAP: '0' });
        expect(r.exitCode, r.stderr.toString()).toBe(1);
        expect(r.stderr.toString()).toMatch(/UBICLOUD: left=unknown owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=unknown\n/);
      } finally { rmSync(blind.root, { recursive: true, force: true }); }
@@ -551,7 +564,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
  test('run: --pass forwards a named local variable only; UBICLOUD_* and non-names are refused before any API call', () => {
    const f = fakeVm();
    try {
-      const ok = run(f, ['run', '--src', f.root, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
+      const ok = run(f, ['run', '--src', f.src, '--pass', 'UBI_TEST_TOKEN', '--env', 'CI=1', '--', 'make test'], { UBI_TEST_TOKEN: 'sekret value' });
      expect(ok.exitCode).toBe(0);
      expect(ok.stderr.toString()).toContain('command exited 0 after');
      const envFile = readFileSync(join(f.root, 'ubirun-env'), 'utf8');
@@ -561,7 +574,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
      expect(ok.stdout.toString() + ok.stderr.toString()).not.toContain('sekret');
      for (const bad of ['UBICLOUD_API_KEY', 'UBICLOUD_API_TOKEN', 'X;Y', 'HOME/..']) {
        const before = f.calls();
-        const r = run(f, ['run', '--src', f.root, '--pass', bad, '--', 'true']);
+        const r = run(f, ['run', '--src', f.src, '--pass', bad, '--', 'true']);
        expect(r.exitCode).toBe(1);
        expect(r.stderr.toString()).toMatch(bad.startsWith('UBICLOUD_') ? /never forwards its own Ubicloud credentials/ : /not a variable name/);
        expect(f.calls()).toBe(before);
@@ -753,4 +766,351 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
      }
    } finally { rmSync(f.root, { recursive: true, force: true }); }
  }, 30_000);
+
+  // GSTA-158 P3: detached phases, reattach and tier checkpoints.
+  const until = async (cond: () => boolean, ms: number) => {
+    const deadline = Date.now() + ms;
+    while (!cond()) { if (Date.now() > deadline) throw new Error('condition not met in time'); await Bun.sleep(100); }
+  };
+  const vmDirs = (f: ReturnType<typeof fakeVm>) => (existsSync(f.state) ? readdirSync(f.state).filter(d => d.startsWith('ubirun-')) : []);
+  const phaseStarted = (f: ReturnType<typeof fakeVm>, phase: string) => vmDirs(f).some(d => existsSync(join(f.state, d, 'phase.' + phase + '.start')));
+  const creates = (f: ReturnType<typeof fakeVm>) => f.calls().split('\n').filter(l => l.includes('"create"')).length;
+  const killSession = async (f: ReturnType<typeof fakeVm>, phase: string) => {
+    const pidFile = join(f.vm, '.ubirun-' + phase + '.pid');
+    await until(() => existsSync(pidFile) && readFileSync(pidFile, 'utf8').trim() !== '', 10_000);
+    const pid = Number(readFileSync(pidFile, 'utf8').trim());
+    try { process.kill(-pid, 'SIGKILL'); } catch { /* already gone */ }
+    await until(() => { try { process.kill(pid, 0); return false; } catch { return true; } }, 10_000);
+  };
+  const gitSrc = (f: ReturnType<typeof fakeVm>, cfg?: unknown) => {
+    const dir = cfg === undefined ? join(f.root, 'repo') : src(f, cfg);
+    mkdirSync(dir, { recursive: true });
+    writeFileSync(join(dir, 'README'), 'fixture repo\n');
+    for (const args of [['init', '-q', '-b', 'main'], ['add', '-A'], ['commit', '-qm', 'fixture']]) {
+      const r = Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', ...args], { cwd: dir, timeout: 30_000 });
+      if (r.exitCode !== 0) throw new Error('git ' + args.join(' ') + ': ' + r.stderr.toString());
+    }
+    return dir;
+  };
+  const TIERS = { setup: 'scripts/boot.sh', tiers: [{ name: 't1', command: 'make t1' }, { name: 't2', command: 'make t2' }, { name: 't3', command: 'make t3' }] };
+  const bootCounts = (f: ReturnType<typeof fakeVm>, dir: string) => writeFileSync(join(dir, 'scripts/boot.sh'), 'echo boot >> ~/setup-count\n');
+  const verifyLine = (out: string) => out.split('\n').find(l => l.startsWith('UBI_VERIFY: ')) ?? '';
+
+  test('run: a SIGKILLed runner leaves its VM; the same command reattaches, skips the finished setup and reports the command\'s exit', async () => {
+    const f = fakeVm();
+    try {
+      const repo = gitSrc(f);
+      const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo setup >> ~/setup-count\n');
+      writeFileSync(join(f.root, 'make-test.sh'), 'echo begin; sleep 4; echo end; exit 3\n');
+      const args = ['run', '--src', repo, '--setup', setup, '--', 'make test'];
+      const a = f.spawn(args);
+      await until(() => vmDirs(f).some(d => existsSync(join(f.state, d, 'command.out')) && readFileSync(join(f.state, d, 'command.out'), 'utf8').includes('begin')), 30_000);
+      process.kill(a.pid, 'SIGKILL');
+      await a.exited;
+      const name = vmDirs(f)[0]!;
+      expect(f.listed()).toContain(name);
+      const b = f.run(args);
+      const err = b.stderr.toString();
+      expect(b.exitCode, err).toBe(3);
+      expect(err).toMatch(new RegExp('ubi-runner: reattached to ' + name + ' created_at=\\d{10} key=[0-9a-f]{12}\n'));
+      expect(b.stdout.toString().split('\n').filter(l => l === 'begin' || l === 'end')).toEqual(['begin', 'end']);
+      expect(err).toMatch(/command exited 3 after \d+s/);
+      expect(creates(f)).toBe(1);
+      expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('setup\n');
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 90_000);
+
+  test('run: a second live runner of the same command refuses with UBI_ATTACH_BUSY and creates nothing', async () => {
+    const f = fakeVm();
+    try {
+      const repo = gitSrc(f);
+      writeFileSync(join(f.root, 'make-test.sh'), 'sleep 4; echo done\n');
+      const args = ['run', '--src', repo, '--', 'make test'];
+      const a = f.spawn(args);
+      await until(() => phaseStarted(f, 'command'), 30_000);
+      const b = f.run(args);
+      expect(b.exitCode).toBe(1);
+      expect(b.stderr.toString()).toMatch(new RegExp('UBI_ATTACH_BUSY: vm=' + vmDirs(f)[0] + ' key=[0-9a-f]{12}: runner pid ' + a.pid + ' '));
+      expect(await a.exited).toBe(0);
+      expect(await new Response(a.stdout).text()).toContain('done');
+      expect(creates(f)).toBe(1);
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 90_000);
+
+  test('run: HUP once the command is live detaches (UBI_DETACHED, exit 129, VM kept, no pass value on disk); a rerun reattaches and finishes', async () => {
+    const f = fakeVm();
+    try {
+      const repo = gitSrc(f);
+      writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3; echo finished\n');
+      const args = ['run', '--src', repo, '--pass', 'UBI_TEST_TOKEN', '--', 'make test'];
+      const a = f.spawn(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
+      await until(() => phaseStarted(f, 'command'), 30_000);
+      await Bun.sleep(500);
+      process.kill(a.pid, 'SIGHUP');
+      expect(await a.exited).toBe(129);
+      const name = vmDirs(f)[0]!;
+      const err = await new Response(a.stderr).text();
+      expect(err).toMatch(new RegExp('UBI_DETACHED: vm=' + name + ' key=[0-9a-f]{12} reattach: rerun the same command'));
+      expect(f.listed()).toContain(name);
+      expect(f.calls()).not.toContain('destroy');
+      for (const file of readdirSync(join(f.state, name))) {
+        const p = join(f.state, name, file);
+        if (lstatSync(p).isFile()) expect(readFileSync(p, 'utf8')).not.toContain('hup-secret-value');
+      }
+      const b = f.run(args, { UBI_TEST_TOKEN: 'hup-secret-value' });
+      expect(b.exitCode, b.stderr.toString()).toBe(0);
+      expect(b.stderr.toString()).toContain('reattached to ' + name);
+      expect(b.stdout.toString()).toContain('finished');
+      expect(creates(f)).toBe(1);
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 90_000);
+
+  test('run: INT and TERM during a live command still tear the VM down', async () => {
+    for (const sig of ['SIGTERM', 'SIGINT'] as const) {
+      const f = fakeVm();
+      try {
+        const repo = gitSrc(f);
+        writeFileSync(join(f.root, 'make-test.sh'), 'sleep 3\n');
+        const a = f.spawn(['run', '--src', repo, '--', 'make test']);
+        await until(() => phaseStarted(f, 'command'), 30_000);
+        await Bun.sleep(500);
+        process.kill(a.pid, sig);
+        expect(await a.exited).toBe(130);
+        expect(await new Response(a.stderr).text()).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-/);
+        expect(f.listed()).toBe('');
+      } finally { rmSync(f.root, { recursive: true, force: true }); }
+    }
+  }, 90_000);
+
+  test('run: a remote session gone without an exit status exits 125 with UBI_ATTACH_LOST and the VM name', async () => {
+    const f = fakeVm();
+    try {
+      const repo = gitSrc(f);
+      writeFileSync(join(f.root, 'make-test.sh'), 'sleep 30\n');
+      const args = ['run', '--src', repo, '--', 'make test'];
+      const a = f.spawn(args);
+      await until(() => existsSync(join(f.vm, '.ubirun-command.pid')) && readFileSync(join(f.vm, '.ubirun-command.pid'), 'utf8').trim() !== '', 30_000);
+      process.kill(a.pid, 'SIGKILL');
+      await a.exited;
+      await killSession(f, 'command');
+      const name = vmDirs(f)[0]!;
+      const b = f.run(args);
+      expect(b.exitCode).toBe(125);
+      expect(b.stderr.toString()).toContain('UBI_ATTACH_LOST: vm=' + name + ' phase=command');
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 90_000);
+
+  test('run: the run key covers tree, command, setup content, env, pass value, image, origin/HEAD and the runner itself; a pass value never appears in it', () => {
+    const f = fakeVm();
+    try {
+      const repo = gitSrc(f);
+      const setup = join(f.root, 'setup.sh'); writeFileSync(setup, 'echo one\n');
+      const runner = join(f.root, 'runner-copy.sh'); writeFileSync(runner, readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'));
+      const key = (opts: { args?: string[]; env?: Record<string, string>; cmd?: string; script?: string } = {}) => {
+        const r = Bun.spawnSync(['bash', opts.script ?? join(DIR, 'ubi-runner.sh'), 'run', '--probe', '--src', repo, '--setup', setup, '--env', 'MODE=a', '--pass', 'UBI_TEST_TOKEN', ...(opts.args ?? []), '--', opts.cmd ?? 'make test'],
+          { env: { ...process.env, PATH: join(f.root, 'bin') + ':' + process.env.PATH, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: f.state, UBI_OWNER: 'thrd1', UBI_TEST_TOKEN: 'v1', ...(opts.env ?? {}) }, timeout: 30_000 });
+        const out = r.stdout.toString() + r.stderr.toString();
+        expect(out).not.toContain('v1');
+        const m = /^UBI_ATTACH_INFO: none key=([0-9a-f]{12})$/m.exec(out);
+        if (!m) throw new Error(out);
+        return m[1]!;
+      };
+      const base = key();
+      expect(key()).toBe(base);
+      const variants: Record<string, string> = {};
+      variants.command = key({ cmd: 'make check' });
+      variants.env = key({ args: ['--env', 'MODE=b'] });
+      variants.pass = key({ env: { UBI_TEST_TOKEN: 'v2' } });
+      variants.image = key({ args: ['-b', 'ubuntu-jammy'] });
+      variants.size = key({ args: ['-s', 'standard-8'] });
+      writeFileSync(runner, readFileSync(runner, 'utf8') + '# edited\n');
+      variants.runner = key({ script: runner });
+      writeFileSync(setup, 'echo two\n');
+      variants.setup = key();
+      writeFileSync(setup, 'echo one\n');
+      expect(key()).toBe(base);
+      Bun.spawnSync(['git', 'update-ref', 'refs/remotes/origin/HEAD', 'HEAD'], { cwd: repo, timeout: 10_000 });
+      variants.base = key();
+      writeFileSync(join(repo, 'new-file.txt'), 'untracked source\n');
+      variants.tree = key();
+      for (const [what, k] of Object.entries(variants)) expect({ what, same: k === base }).toEqual({ what, same: false });
+      expect(new Set(Object.values(variants)).size).toBe(Object.keys(variants).length);
+      expect(f.calls()).not.toContain('"create"');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 60_000);
+
+  test('verify tiers: config refuses command with tiers, bad and duplicate tier names, before any API call', () => {
+    const f = fakeVm();
+    try {
+      const cases: Array<[unknown, RegExp]> = [
+        [{ command: 'make', tiers: [{ name: 'a', command: 'x' }] }, /UBI_CONFIG_TIERS_AND_COMMAND/],
+        [{ tiers: [{ name: 'Unit', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name 'Unit' must match/],
+        [{ tiers: [{ name: '-x', command: 'x' }] }, /UBI_CONFIG_TIER_NAME: tier name '-x'/],
+        [{ tiers: [{ name: 'a'.repeat(33), command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
+        [{ tiers: [{ name: 'a.b', command: 'x' }] }, /UBI_CONFIG_TIER_NAME/],
+        [{ tiers: [{ name: 'a', command: 'x' }, { name: 'a', command: 'y' }] }, /UBI_CONFIG_TIER_NAME: tier name 'a' appears twice/],
+        [{ tiers: [{ name: 'a' }] }, /UBI_CONFIG_TIER_NAME: tier 'a' needs a command/],
+        [{ tiers: [] }, /UBI_CONFIG_TIER_NAME: tiers must be a non-empty list/],
+      ];
+      for (const [cfg, want] of cases) {
+        const dir = src(f, cfg);
+        const r = run(f, ['verify', '--src', dir]);
+        expect(r.exitCode).toBe(1);
+        expect(r.stderr.toString()).toMatch(want);
+      }
+      expect(f.calls()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 30_000);
+
+  test('verify tiers: a run killed during tier 3 with its VM gone resumes on a new VM: setup reruns, t1 and t2 are reused, t3 reruns', async () => {
+    const f = fakeVm();
+    try {
+      const dir = gitSrc(f, TIERS);
+      bootCounts(f, dir);
+      Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
+      writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
+      const a = f.spawn(['verify', '--src', dir]);
+      await until(() => existsSync(join(f.root, 't3-started')), 60_000);
+      process.kill(a.pid, 'SIGKILL');
+      await a.exited;
+      await killSession(f, 'tier-t3');
+      const first = vmDirs(f)[0]!;
+      writeFileSync(join(f.root, 'listed'), '');
+      writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
+      const b = f.run(['verify', '--src', dir]);
+      const err = b.stderr.toString(), out = b.stdout.toString();
+      expect(b.exitCode, err).toBe(0);
+      expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse t1\.\d{10}\.\d+\.json \(\d+s\)\n/);
+      expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse t2\.\d{10}\.\d+\.json \(\d+s\)\n/);
+      expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t3 run: interrupted \(only a start record, t3\.\d{10}\.\d+\.json\)\n/);
+      expect(err).toContain('run record for ' + first + ': the VM is gone; starting a new one');
+      expect(err).not.toContain('reattached to');
+      expect(creates(f)).toBe(2);
+      expect(readFileSync(join(f.vm, 'setup-count'), 'utf8')).toBe('boot\nboot\n');
+      expect(out).not.toContain('ran: make t1');
+      expect(out).toContain('t3 ok');
+      expect(verifyLine(out)).toMatch(/^UBI_VERIFY: tree=[0-9a-f]{12} key=[0-9a-f]{12} tiers=3 passed=3 reused=t1,t2 failed=none prior_failures=none saved_s=\d+$/);
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 120_000);
+
+  test('verify tiers: a fail record blocks a rerun until --rerun-failed (then prior_failures); passes are reused unless stale, tampered, --no-resume or the tree changed', () => {
+    const f = fakeVm();
+    try {
+      const dir = gitSrc(f, TIERS);
+      writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 broke; exit 7\n');
+      const first = run(f, ['verify', '--src', dir]);
+      expect(first.exitCode, first.stderr.toString()).toBe(7);
+      expect(first.stdout.toString()).not.toContain('ran: make t3');
+      expect(verifyLine(first.stdout.toString())).toMatch(/ tiers=3 passed=1 reused=none failed=t2 prior_failures=none saved_s=0$/);
+      expect(creates(f)).toBe(1);
+
+      const blocked = run(f, ['verify', '--src', dir]);
+      expect(blocked.exitCode).toBe(7);
+      expect(blocked.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 fail t2\.\d{10}\.\d+\.json \(\d+s, exit 7\): not rerun without --rerun-failed/);
+      expect(verifyLine(blocked.stdout.toString())).toMatch(/ failed=t2 prior_failures=none /);
+      expect(creates(f)).toBe(1);
+
+      writeFileSync(join(f.root, 'make-t2.sh'), 'echo t2 fixed\n');
+      const rerun = run(f, ['verify', '--src', dir, '--rerun-failed']);
+      expect(rerun.exitCode, rerun.stderr.toString()).toBe(0);
+      expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
+      expect(rerun.stderr.toString()).toMatch(/UBI_VERIFY_RESUME: tier=t2 run: prior fail t2\.\d{10}\.\d+\.json \(exit 7; --rerun-failed\)/);
+      expect(verifyLine(rerun.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1 failed=none prior_failures=t2:7 saved_s=\d+$/);
+      expect(creates(f)).toBe(2);
+
+      const all = run(f, ['verify', '--src', dir]);
+      expect(all.exitCode).toBe(0);
+      expect(verifyLine(all.stdout.toString())).toMatch(/ passed=3 reused=t1,t2,t3 failed=none prior_failures=none saved_s=\d+$/);
+      expect(all.stderr.toString()).toContain('every tier passed under this key; no VM needed');
+      expect(creates(f)).toBe(2);
+
+      const probe = (args: string[] = [], extra: Record<string, string> = {}) => run(f, ['verify', '--src', dir, '--probe', ...args], extra).stderr.toString();
+      expect(probe(['--no-resume'])).toMatch(/tier=t1 run: --no-resume\n[^]*tier=t2 run: --no-resume\n[^]*tier=t3 run: --no-resume\n/);
+      expect(probe([], { UBI_CHECKPOINT_MAX_AGE: '0' })).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json is \d+s old \(UBI_CHECKPOINT_MAX_AGE=0\)/);
+      const keyDir = join(f.checkpoints, readdirSync(f.checkpoints)[0]!);
+      const t1Log = readdirSync(keyDir).filter(n => n.startsWith('t1.') && n.endsWith('.log')).sort().at(-1)!;
+      writeFileSync(join(keyDir, t1Log), 'tampered\n');
+      const tampered = probe();
+      expect(tampered).toMatch(/tier=t1 run: pass t1\.\d{10}\.\d+\.json: its log does not re-hash/);
+      expect(tampered).toMatch(/tier=t2 reuse /);
+      writeFileSync(join(dir, 'new-source.ts'), 'export {}\n');
+      const moved = probe();
+      for (const t of ['t1', 't2', 't3']) expect(moved).toContain('UBI_VERIFY_RESUME: tier=' + t + ' run: no record');
+      expect(creates(f)).toBe(2);
+      // Records are write-once JSON; every one carries its generation.
+      const gens = readdirSync(keyDir).filter(n => n.endsWith('.json')).map(n => JSON.parse(readFileSync(join(keyDir, n), 'utf8')).generation as number).sort((x, y) => x - y);
+      expect(gens).toEqual(gens.map((_, i) => i + 1));
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 120_000);
+
+  test('verify tiers: a reattach whose VM TTL no longer covers the run left plus UBI_REATTACH_MARGIN starts a fresh VM, destroys the old one and reuses the passed tiers', async () => {
+    const f = fakeVm();
+    try {
+      const dir = gitSrc(f, TIERS);
+      bootCounts(f, dir);
+      Bun.spawnSync(['git', '-c', 'user.email=t@t', '-c', 'user.name=t', 'commit', '-qam', 'boot'], { cwd: dir, timeout: 10_000 });
+      writeFileSync(join(f.root, 'make-t3.sh'), 'touch ' + JSON.stringify(join(f.root, 't3-started')) + '; sleep 30\n');
+      const env = { UBI_REAP: '0', UBI_TTL_HOURS: '1' };
+      const a = f.spawn(['verify', '--src', dir], env);
+      await until(() => existsSync(join(f.root, 't3-started')), 60_000);
+      process.kill(a.pid, 'SIGKILL');
+      await a.exited;
+      const first = vmDirs(f)[0]!;
+      expect(first).toMatch(/-t1$/);
+      const probe = (extra: Record<string, string>) => run(f, ['verify', '--src', dir, '--probe'], { ...env, ...extra }).stdout.toString();
+      expect(probe({})).toMatch(new RegExp('UBI_ATTACH_INFO: vm=' + first + ' created_at=\\d{10} '));
+      // A 1 h TTL leaves about 3600 s: t3's 1200 s run timeout plus a 2700 s margin does not fit.
+      const short = { UBI_RUN_TIMEOUT: '1200', UBI_REATTACH_MARGIN: '2700' };
+      expect(probe(short)).toMatch(new RegExp('UBI_ATTACH_INFO: none key=[0-9a-f]{12} ttl_short vm=' + first + ' ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\)'));
+      expect(creates(f)).toBe(1);
+      writeFileSync(join(f.root, 'make-t3.sh'), 'echo t3 ok\n');
+      const b = run(f, ['verify', '--src', dir], { ...env, ...short });
+      const err = b.stderr.toString();
+      expect(b.exitCode, err).toBe(0);
+      expect(err).toMatch(new RegExp('UBI_REATTACH_TTL_SHORT: vm=' + first + ' key=[0-9a-f]{12} ttl_left=\\d+s needs=\\d+s \\(run still to do \\d+s \\+ UBI_REATTACH_MARGIN=2700s\\): starting a fresh VM \\(tier checkpoints carry over\\) and destroying ' + first));
+      expect(err).not.toContain('reattached to');
+      expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t1 reuse /);
+      expect(err).toMatch(/UBI_VERIFY_RESUME: tier=t2 reuse /);
+      expect(verifyLine(b.stdout.toString())).toMatch(/ tiers=3 passed=3 reused=t1,t2 failed=none /);
+      expect(creates(f)).toBe(2);
+      expect(f.listed()).toBe('');
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 120_000);
+
+  test('verify tiers: a writer whose start record is no longer the newest publishes nothing (stale writer)', () => {
+    const py = /\nCKPT_PY='\n([^']*)'\n/.exec(readFileSync(join(DIR, 'ubi-runner.sh'), 'utf8'))![1]!;
+    const root = mkdtempSync(join(tmpdir(), 'ubi-ckpt-'));
+    try {
+      const d = join(root, 'key');
+      const log = join(root, 'tier.log'); writeFileSync(log, 'ok\n');
+      const publish = (status: string, startgen: string, pid: string) => Bun.spawnSync(['python3', '-c', py, 'publish', d, 't1', status, 'vm', '100', '200', '0', status === 'start' ? '' : log, startgen, 'k', 'tree', pid], { timeout: 30_000 });
+      expect(publish('start', '0', '1').stdout.toString().trim()).toBe('1');
+      expect(publish('start', '0', '2').stdout.toString().trim()).toBe('2');
+      const stale = publish('pass', '1', '1');
+      expect(stale.exitCode).toBe(2);
+      expect(stale.stderr.toString()).toContain('stale writer: its start record is generation 1, the newest for t1 is 2');
+      expect(readdirSync(d).filter(n => n.endsWith('.json')).length).toBe(2);
+      expect(publish('pass', '2', '2').stdout.toString().trim()).toBe('3');
+    } finally { rmSync(root, { recursive: true, force: true }); }
+  });
+
+  test('verify tiers: a tier that edits the source gets no pass record (UBI_VERIFY_TREE_MOVED) and reruns next time', () => {
+    const f = fakeVm();
+    try {
+      const dir = gitSrc(f, { tiers: [{ name: 'mover', command: 'make mover' }, { name: 'clean', command: 'make clean' }] });
+      writeFileSync(join(f.root, 'make-mover.sh'), 'echo edited > moved.txt\n');
+      const r = run(f, ['verify', '--src', dir]);
+      expect(r.exitCode, r.stderr.toString()).toBe(0);
+      expect(r.stderr.toString()).toMatch(/UBI_VERIFY_TREE_MOVED: tier=mover before=[0-9a-f]{12} after=[0-9a-f]{12}: a tier must leave the source tree as it found it; no pass record/);
+      expect(verifyLine(r.stdout.toString())).toMatch(/ tiers=2 passed=2 reused=none failed=none /);
+      const next = run(f, ['verify', '--src', dir, '--probe']).stderr.toString();
+      expect(next).toMatch(/tier=mover run: interrupted \(only a start record/);
+      expect(next).toMatch(/tier=clean reuse /);
+    } finally { rmSync(f.root, { recursive: true, force: true }); }
+  }, 90_000);
 });
@@ -237,6 +237,10 @@ and for browser tests `xvfb` plus
 `bun run test:ubicloud` (the free suite with the CI lane's environment) and
 `gstack-pregate --stage tests --remote` under 8 cores.

+A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
+for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
+fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
+
 ---

 ## Step 4: Read the result
@@ -266,9 +270,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.

 ## Resume checklist after a wake or a machine swap

-1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
-   still running without a live runner.
-2. A background log without its exit sentinel after the VM is gone is an
-   interrupted run, not a pass; rerun it.
-3. Measure before/after pairs on the same size and location, never one side
+1. Rerun the same command first: a run whose runner died or detached
+   (`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
+   finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
+   `UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
+2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
+   rerun will reattach to.
+3. After a machine swap nothing reattaches; tier checkpoints on a Drive
+   (`UBI_CHECKPOINT_DIR`) still count.
+4. A background log with no exit sentinel and no VM is an interrupted run, not
+   a pass; rerun it.
+5. Measure before/after pairs on the same size and location, never one side
   local and the other remote.
@@ -163,6 +163,10 @@ and for browser tests `xvfb` plus
 `bun run test:ubicloud` (the free suite with the CI lane's environment) and
 `gstack-pregate --stage tests --remote` under 8 cores.

+A long verify can list `tiers: [{"name": "unit", "command": "make unit"}]`
+for `command`. A rerun reuses recent passes (`UBI_VERIFY_RESUME:`), reruns a
+fail only with `--rerun-failed`, and ends with `UBI_VERIFY: …`.
+
 ---

 ## Step 4: Read the result
@@ -192,9 +196,15 @@ is visible: a `standard-16` run is billed from create to confirmed teardown.

 ## Resume checklist after a wake or a machine swap

-1. `$U records` and `$U list --mine`, then `$U down` anything of yours that is
-   still running without a live runner.
-2. A background log without its exit sentinel after the VM is gone is an
-   interrupted run, not a pass; rerun it.
-3. Measure before/after pairs on the same size and location, never one side
+1. Rerun the same command first: a run whose runner died or detached
+   (`UBI_DETACHED`, exit 129 after a HUP) reattaches to its live VM and skips a
+   finished setup (`UBI_ATTACH_BUSY`: another runner owns it; exit 125
+   `UBI_ATTACH_LOST`: no status; `UBI_REATTACH_TTL_SHORT`: a fresh VM took over).
+2. `$U records` and `$U list --mine`, then `$U down` anything of yours that no
+   rerun will reattach to.
+3. After a machine swap nothing reattaches; tier checkpoints on a Drive
+   (`UBI_CHECKPOINT_DIR`) still count.
+4. A background log with no exit sentinel and no VM is an interrupted run, not
+   a pass; rerun it.
+5. Measure before/after pairs on the same size and location, never one side
   local and the other remote.