CalvinBackup/gstack
1
0 Fork 0
mirror of https://github.com/garrytan/gstack.git synced 2026-11-10 10:40:32 +01:00
Code Issues Packages Projects Releases Wiki Activity
Files
cf4d847e0169e8edb8d37abd00a29855bb2fd380
gstack/scripts/ubicloud/ubi-runner.sh
T
Garry Tanandcapy-ai[bot] 065afdfdcf fix(ubi-runner): an ambiguous create arms teardown only once this call's ssh key opens the VM (GST-1463) (#4046) 
* fix(ubi-runner): an ambiguous create arms teardown only once this call's ssh key opens the VM (GST-1463)

With an explicit -n name a second caller can take the name between the 404
name check and the create; if our create's 409 was lost (000), the by-name
lookup adopted that caller's VM and teardown destroyed it. Teardown is now
off from an ambiguous create answer on and armed only when this call's key
opens the VM within UBI_SSH_WAIT; otherwise up keeps the record and bundle,
exits UBI_CREATE_UNKNOWN and leaves the VM to its TTL, and run reports
left=unknown.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: ubi-runner ssh proof offers only this call's key (IdentitiesOnly=yes)

The ownership proof after an ambiguous create counted any ssh success, and ssh also offered agent and default ~/.ssh keys beside -i, so a holder VM that accepted one of them was armed and destroyed. Failing-first test: the fake agent holds the holder's key.

Review-Finding: p1/specialist-batched#1 reader=specialist-batched
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: ubi-runner ownership proof ignores ssh config identities (-F /dev/null)

IdentitiesOnly=yes still offers IdentityFile and CertificateFile entries from ssh config, so the proof after an ambiguous create could pass on a holder VM that accepts a configured key. The unproven proof now runs with -F /dev/null; the normal path keeps the user's config. Failing-first: the fake ssh config holds the holder's key.

Review-Finding: p2/red-team#1 reader=red-team
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: ubicloud skill says to list before down after UBI_CREATE_UNKNOWN

left=unknown can now mean another caller's VM; the runner prints list, not down, for that case. The changelog fragment notes that agent and config keys don't count for the proof.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
2026-11-05 19:24:58 +00:00

2147 lines
104 KiB
Bash
Executable File
Raw Blame History

#!/usr/bin/env bash
# ubi-runner — run a command on an ephemeral Ubicloud VM.
#
# UBICLOUD_API_KEY=... scripts/ubicloud/ubi-runner.sh run --setup S.sh -- 'make test'
#
# UBICLOUD_API_TOKEN is accepted as an alias for UBICLOUD_API_KEY. SSH
# connections are multiplexed per VM, and `pack` / `unpack` upload one prebuilt
# checkout tarball to many VMs. This is the repo copy the user-scope Capy
# `ubicloud` skill and gbrain's scripts/ubicloud/ubi-runner.sh are kept in sync
# with (plan E2); `bun run test:ubicloud` drives it through test-free.sh.
#
# Scratch record: every VM this machine creates is appended to a VM-id file
# (UBI_VM_RECORD; default ~/.capy/work/ubi-runner/vms.tsv when ~/.capy/work
# exists, else $STATE_ROOT/vms.tsv) as `name location owner created_at pid`,
# and removed once `down` confirms it gone. A resumed Capy thread reads that
# file to find VMs an earlier turn left behind (docs/TESTING_INTERNALS.md,
# "Resuming after a wake").
#
# Creates a VM (standard-16 by default), streams the checkout to it (tracked +
# untracked-unignored files + .git, so uncommitted edits are included), runs an
# optional setup script and the command, copies requested artifacts back, and
# destroys the VM on every exit path. Exits with the command's status.
# Needs only bash, curl, python3, ssh, ssh-keygen, and tar locally, so it works
# from dev boxes, containers, and cloud sandboxes.
#
# Ownership: VMs are named ubirun-<owner>-<epoch>-<suffix>-t<hours>. <owner> is UBI_OWNER
# (lowercased, letters and digits only, at most 12 characters, starting with a
# letter). `up`, `run` and `verify` refuse without it (UBI_OWNER_REQUIRED), so
# every VM names whoever can be asked about it; read-only commands fall back to a
# per-machine id kept in $STATE_ROOT/owner-id. Ubicloud names allow 63 characters
# of [a-z0-9-], so a full name stays under 45.
#
# Teardown: `up` records each VM's state (location, create status) before it
# sends the create request, and `down` destroys and polls until the VM is
# confirmed gone or provably never existed, waiting out a create that may still
# be in flight. An interrupted `up` or `run` (EXIT, INT, TERM, QUIT, and HUP
# until a remote phase is live) runs that `down`; further signals are ignored
# until it finishes. A create is sent only after a show of the name answers
# 404: a name another VM answers to, or a check with no answer, is refused
# before any request. Teardown is armed only once this call's create request is
# sent, and disarmed when Ubicloud refuses it (HTTP 4xx, a 409 name collision
# included), so it never destroys a VM this call did not create. A 5xx or no
# answer is not a refusal: `up` looks the VM up by name for UBI_CREATE_GRACE
# seconds, and when that cannot settle it either, it keeps the record and the recovery
# bundle and exits UBI_CREATE_UNKNOWN. Another caller can take an explicit -n
# name between the check and the create, so a VM found that way is adopted
# with teardown off, and teardown is armed only once this call's ssh key opens
# it (within UBI_SSH_WAIT seconds, default 300); if the key never does, `up`
# exits UBI_CREATE_UNKNOWN and leaves the VM to its TTL. `down` counts a VM as gone only
# on a readable answer: a failed or unreadable VM list or show is unknown, so
# down exits non-zero and keeps the recovery bundle. The API key reaches curl
# as a header file on a pipe, never in its argv. A create refused
# for vCPU quota prints the project's usage per owner.
#
# The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive
# number, every `up` first destroys this owner's VMs older than that; `gc HOURS`
# runs the same sweep on demand. Neither ever touches another owner's VMs.
#
# TTL and reaper: a name this script generates ends in -t<hours> (UBI_TTL_HOURS,
# 1-999, default 3), the creator's own declaration of how long the VM may live.
# `reap` destroys only this owner's VMs: its ubirun-<owner>-* names whose TTL
# expired more than ten minutes ago, and those with no TTL once they are
# UBI_LEGACY_HOURS old (default 4). Another owner's VM, an untagged one and
# names outside the ubirun- prefix are never reaped, expired or not; reap only
# counts other owners' expired VMs in its log (cloud deletes stay owner-scoped,
# GST-573). Every `up` runs it first (UBI_REAP=0 skips it), so a lane killed by
# a sleeping machine has its leftovers destroyed by its own next `up`.
#
# Leftover line: `run` and `verify` end with
# UBICLOUD: left=<0|1|unknown> owner=<owner> vm=<name> others=<n>
# after teardown: left counts this invocation's own VM, checked against the
# project's VM list, and others counts this owner's other VMs (information only).
# A left other than 0 turns a passing command into exit 1; `run --keep` reports
# its live VM as left=1 kept=1 and keeps its exit status. With UBI_LEFT_FILE set,
# the line is also appended to that file, out of reach of the command's output.
#
# Quota preflight: when the project's vCPU ceiling is known (UBI_QUOTA_VCPUS, or
# the maximum a refusal printed, cached in $STATE_ROOT/quota-max), `up` adds the
# size's vCPUs to the project's running VMs and refuses before any create when
# that exceeds it; UBI_QUOTA_WAIT=<seconds> waits for room instead.
#
# Watchdog: `run`, `verify` and `ssh NAME COMMAND` stream the remote output
# through a watchdog that logs a heartbeat every UBI_HEARTBEAT seconds (300)
# and kills a command silent for UBI_IDLE_TIMEOUT seconds (1800) or older than
# UBI_RUN_TIMEOUT seconds (0, unlimited), exiting 124 with the last 40 lines,
# so a hung verify fails fast instead of holding the VM until its TTL.
#
# Detached phases and reattach: `run` and `verify` start their setup, command
# and tier phases on the VM detached from the ssh connection (setsid; stdout to
# ~/.ubirun-<phase>.out, stderr to .err, the session leader's pid in .pid, and
# the exit status in .exit, written by rename once the streams are closed) and
# stream both files back. Before creating a VM they compute a run key locally
# (the checkout's content tree, the command, the setup script's content, every
# --env entry, each --pass name with a sha256 of its value, size, location,
# image, storage, origin/HEAD, tiers and this script's own sha256) and look for
# a run record (<state dir>/<vm>/run) with the same key whose VM still exists:
# its runner alive (lease pid and start time) refuses with UBI_ATTACH_BUSY,
# otherwise the run reattaches, printing `reattached to <vm> created_at=<epoch>
# key=<k12>`, and continues from the first phase without an exit file. After
# unpack the VM recomputes the tree and refuses a mismatch (UBI_VM_TREE_MISMATCH).
# A leader gone with no exit file exits 125 (UBI_ATTACH_LOST). Signals: INT
# and TERM tear the VM down as before; HUP once a remote phase is live detaches
# (UBI_DETACHED, exit 129) and leaves the VM under its TTL; SIGKILL or a machine
# stop leaves the record and the VM, and the next run of the same command
# reattaches. `attach NAME` resumes a recorded run explicitly. Reattach never
# extends a TTL, and the reaper still applies.
#
# Tier checkpoints: a .gstack/ubicloud.json with `tiers` runs each tier as its
# own detached phase after setup, and records start/pass/fail per tier under
# UBI_CHECKPOINT_DIR/<run key>/ (write-once JSON plus the tier's log, published
# under a lock with a monotonic generation). A rerun reuses a pass younger than
# UBI_CHECKPOINT_MAX_AGE whose log re-hashes, so a resume on a fresh VM reruns
# only setup and the tiers without a pass.
#
# `verify` is the one entry point for a repository's full remote verify: it
# reads .gstack/ubicloud.json (setup, command, size, location, env, pass, pull)
# and runs it through `run`, so lanes share one committed definition instead of
# per-lane wrappers.
#
# Surviving a dead controller (GST-1220). With UBI_RECOVERY_DIR set (on Capy,
# GSTACK_CAPY_RECOVERY_DIR names a drive folder), `up` writes an encrypted
# recovery bundle of each VM's local state (ssh key, env, tags) to
# <dir>/<owner>/<vm>.bundle, and `recover [--owner OLD]` on another machine
# rebuilds that state so pull, ssh and down work there. The bundle is AES-256
# (openssl enc, pbkdf2) under a key derived from UBI_RECOVERY_KEY or else
# UBICLOUD_API_KEY, with an HMAC-SHA256 checked before anything is decrypted;
# the secret travels only on a pipe, never in argv, a child's environment or a
# file. A counted run (--counted or UBI_COUNTED=1) refuses before any create
# without a bundle (UBI_RECOVERY_REQUIRED, UBI_RECOVERY_UNAVAILABLE). Run
# recover only once the creating machine is dead: a paused one is not detected.
# `up --sink https-put:<presigned URL> --sink-path PATH` starts a pusher on the
# VM that uploads heartbeat.json and the PATH entries as results.tgz every
# UBI_SINK_INTERVAL seconds, so results land even when every controller dies;
# the URL reaches the VM only in a 600 curl config file. The sink is off unless
# --sink or UBI_SINK names one, and `down` makes one last push first
# (SINK_FLUSH_FAILED turns a passing run into exit 1).
set -euo pipefail
# The caller's exported names and values, captured before the runner sets any
# variable of its own: --pass forwards from here, never from a live variable.
INHERITED_NAMES=() INHERITED_VALUES=()
for _ubi_n in $(compgen -e); do
[ -n "${!_ubi_n+x}" ] || continue
INHERITED_NAMES+=("$_ubi_n") INHERITED_VALUES+=("${!_ubi_n}")
done
unset _ubi_n
INHERITED_ENV=" ${INHERITED_NAMES[*]-} "
# Keep heredoc bodies on temp files, not the pipe window (test/heredoc-pipe-deadlock.test.ts).
BASH_COMPAT=50
API="${UBICLOUD_API_URL:-https://api.ubicloud.com}"
STATE_ROOT="${UBI_RUNNER_STATE:-${XDG_STATE_HOME:-$HOME/.local/state}/ubi-runner}"
DEFAULT_SIZE="${UBI_SIZE:-standard-16}"
DEFAULT_LOCATION="${UBI_LOCATION:-eu-central-h1}"
GC_HOURS="${UBI_GC_HOURS:-0}"
API_TIMEOUT="${UBI_API_TIMEOUT:-120}"
CREATE_GRACE="${UBI_CREATE_GRACE:-180}"
SSH_WAIT="${UBI_SSH_WAIT:-300}"
DOWN_TIMEOUT="${UBI_DOWN_TIMEOUT:-900}"
POLL="${UBI_POLL_SECONDS:-5}"
ALLOWED_STORAGE="${UBI_ALLOWED_STORAGE:-320 640}"
TTL_HOURS="${UBI_TTL_HOURS:-3}"
LEGACY_HOURS="${UBI_LEGACY_HOURS:-4}"
REAP="${UBI_REAP:-1}"
REAP_GRACE=600
IDLE_TIMEOUT="${UBI_IDLE_TIMEOUT:-1800}"
RUN_TIMEOUT="${UBI_RUN_TIMEOUT:-0}"
HEARTBEAT="${UBI_HEARTBEAT:-300}"
QUOTA_VCPUS="${UBI_QUOTA_VCPUS:-}"
QUOTA_WAIT="${UBI_QUOTA_WAIT:-0}"
PREFIX="ubirun"
if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD"
elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv"
else VM_RECORD="$STATE_ROOT/vms.tsv"; fi
if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR"
elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints"
else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi
CHECKPOINT_MAX_AGE="${UBI_CHECKPOINT_MAX_AGE:-86400}"
REATTACH_MARGIN="${UBI_REATTACH_MARGIN:-900}"
die() { echo "ubi-runner: $*" >&2; exit 1; }
log() { echo "ubi-runner: $*" >&2; }
for bin in curl python3 ssh ssh-keygen tar; do
command -v "$bin" >/dev/null || die "$bin is required"
done
UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}"
RECOVERY_DIR="${UBI_RECOVERY_DIR:-${GSTACK_CAPY_RECOVERY_DIR:-}}"
RECOVERY_SECRET="${UBI_RECOVERY_KEY:-$UBICLOUD_API_KEY}"
unset UBI_RECOVERY_KEY
COUNTED="${UBI_COUNTED:-0}"
SINK_INTERVAL="${UBI_SINK_INTERVAL:-300}"
SINK_MAX_MB="${UBI_SINK_MAX_MB:-512}"
SINK_FLUSH_FAILED=""
[[ "$SINK_INTERVAL" =~ ^[1-9][0-9]*$ ]] || die "UBI_SINK_INTERVAL must be a whole number of seconds, at least 1"
[[ "$SINK_MAX_MB" =~ ^[1-9][0-9]*$ ]] || die "UBI_SINK_MAX_MB must be a whole number of megabytes, at least 1"
[[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)"
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT CHECKPOINT_MAX_AGE REATTACH_MARGIN; do
[[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)"
done
[ -z "$QUOTA_VCPUS" ] || [[ "$QUOTA_VCPUS" =~ ^[0-9]+$ ]] || die "UBI_QUOTA_VCPUS must be a whole number of vCPUs"
owner_id() {
local raw=${UBI_OWNER:-} file="$STATE_ROOT/owner-id" tmp
if [ -z "$raw" ]; then
if [ ! -s "$file" ]; then
mkdir -p "$STATE_ROOT"
tmp="$file.$$"
echo "m$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')" >"$tmp"
mv -n "$tmp" "$file"
rm -f "$tmp"
fi
raw=$(cat "$file")
fi
raw=$(printf %s "$raw" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9')
[ -n "$raw" ] || die "UBI_OWNER must contain letters or digits"
[[ "$raw" =~ ^[a-z] ]] || raw="u$raw"
echo "${raw:0:12}"
}
OWNER=$(owner_id)
# require_owner: up, run and verify create VMs, and a VM must name its owner
# and carry a TTL. Only creates check UBI_TTL_HOURS, so an old value never
# blocks down, reap or list.
require_owner() {
[ -n "${UBI_OWNER:-}" ] || die "UBI_OWNER is not set: every VM must carry its owner; export UBI_OWNER=<your thread or session code> (for example gsta164) and rerun (UBI_OWNER_REQUIRED)"
[[ "$TTL_HOURS" =~ ^[1-9][0-9]{0,2}$ ]] || die "UBI_TTL_HOURS must be a whole number of hours from 1 to 999 with no leading zero (every VM carries a TTL; set it above your longest run)"
}
# leftover FIELDS...: print the UBICLOUD: leftover line, and append it to
# UBI_LEFT_FILE when set, so a caller can read it apart from the command's output.
leftover() {
echo "UBICLOUD: $*" >&2
[ -z "${UBI_LEFT_FILE:-}" ] || echo "UBICLOUD: $*" >>"$UBI_LEFT_FILE"
}
# api OUT ARGS...: send one Ubicloud CLI command, write the body to OUT, and
# print the HTTP status (000 when no response arrived).
api() {
local out=$1 code
shift
code=$(python3 -c 'import json,sys; print(json.dumps({"argv": sys.argv[1:]}))' "$@" \
| curl -sS -o "$out" -w '%{http_code}' --max-time "$API_TIMEOUT" -X POST \
-H @<(printf 'Authorization: Bearer %s\n' "$UBICLOUD_API_KEY") \
-H 'Accept: text/plain' -H 'Content-Type: application/json' \
-H 'X-Ubi-Version: 1.0.0' --data @- "$API/cli") || true
echo "${code:-000}"
}
cli() {
local out code
out=$(mktemp)
code=$(api "$out" "$@")
if [ "$code" = 000 ]; then
rm -f "$out"
die "request failed: ubi $*"
fi
if [ "$code" != 200 ]; then
cat "$out" >&2
rm -f "$out"
return 1
fi
cat "$out"
rm -f "$out"
}
state_dir() { echo "$STATE_ROOT/$1"; }
load() {
local dir
dir=$(state_dir "$1")
[ -f "$dir/env" ] || die "no local state for VM '$1' (created on another machine?)"
# shellcheck disable=SC1091
. "$dir/env"
KEY="$dir/key"
}
ssh_opts() {
echo -i "$KEY" -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-o LogLevel=ERROR -o ServerAliveInterval=30 -o ServerAliveCountMax=6 -o ConnectTimeout=10 \
-o ControlMaster=auto -o "ControlPath=$(dirname "$KEY")/ctl" -o ControlPersist=300
}
# Ubuntu's default umask (002) leaves new directories group-writable, which
# permission-checking code under test rejects; every remote command uses 022.
remote() {
local name=$1; shift
load "$name"
# shellcheck disable=SC2046
ssh $(ssh_opts) "ubi@$IP" "umask 022; $*"
}
mtime() { stat -c %Y "$1" 2>/dev/null || stat -f %m "$1"; }
# sq STRING: STRING as one single-quoted shell word.
sq() { printf "'%s'" "$(printf %s "$1" | sed "s/'/'\\\\''/g")"; }
# remote_cancel NAME SIDFILE: end the session a watched command started on the
# VM (its leader pid is in SIDFILE there), bounded to 20 seconds. Killing the
# local ssh client alone leaves a silent remote command running until it next
# writes to the closed channel.
remote_cancel() {
local name=$1 sidf=$2 cpid i=0
remote "$name" "sid=\$(cat $sidf 2>/dev/null); rm -f $sidf; [ -z \"\$sid\" ] || { pkill -TERM -s \"\$sid\"; sleep 2; pkill -KILL -s \"\$sid\"; }; true" >/dev/null 2>&1 </dev/null &
cpid=$!
while kill -0 "$cpid" 2>/dev/null && [ "$i" -lt 20 ]; do sleep 1; i=$(( i + 1 )); done
kill "$cpid" 2>/dev/null || true
wait "$cpid" 2>/dev/null || true
}
# watched NAME PHASE COMMAND: run COMMAND on the VM, streaming its output and
# keeping a copy of both streams in the VM's state dir, with a heartbeat every HEARTBEAT
# seconds and a watchdog that kills it after IDLE_TIMEOUT seconds of silence or
# RUN_TIMEOUT seconds in total: exit 124, reason and the last 40 lines on
# stderr. The ssh runs in its own session (python3's setsid) so the watchdog
# kills it and anything it started, not just the shell that launched it; on
# the VM the command runs in its own session too (setsid where the image has
# it), whose leader pid is written to ~/.ubirun-PHASE-<runner pid>.sid so
# remote_cancel can end it when the client is killed.
watched() {
local name=$1 phase=$2 cmd=$3 out pid rc=0 start now last_beat idle elapsed why="" old_traps inner sidf
sidf="~/.ubirun-$phase-$$.sid"
load "$name"
out="$(state_dir "$name")/$phase.log"
: >"$out"
inner=$(sq "printf %s \$\$ >$sidf; $cmd")
# The sid file outlives the command on purpose: a background child it left
# behind keeps the ssh channel open after the wrapper exits, and remote_cancel
# still needs the session id to end it. remote_cancel removes the file.
cmd="if command -v setsid >/dev/null 2>&1; then setsid -w bash -c $inner; else bash -c $inner; fi"
# The caller's stdin is handed to the background ssh explicitly (a background
# job in a non-interactive shell would otherwise read /dev/null), so piped
# input reaches the remote command as it did before the watchdog.
exec 3<&0
# shellcheck disable=SC2046
( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
ssh $(ssh_opts) "ubi@$IP" "umask 022; $cmd" ) <&3 > >(tee -a "$out") 2> >(tee -a "$out" >&2) &
pid=$!
exec 3<&-
# A signal to this process takes the ssh session down with it, so Ctrl-C on
# `ssh NAME COMMAND` leaves no orphaned client behind.
old_traps=$(trap -p INT TERM)
trap 'kill -TERM -- "-$pid" 2>/dev/null; exit 130' INT TERM
start=$(date +%s)
last_beat=$start
while :; do
sleep 1
kill -0 "$pid" 2>/dev/null || break
now=$(date +%s)
elapsed=$(( now - start ))
idle=$(( now - $(mtime "$out") ))
if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then
why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)"
elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then
why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)"
fi
if [ -n "$why" ]; then
# The command may have finished while the clock was read; a finished
# command is never reported as a timeout.
kill -0 "$pid" 2>/dev/null || break
log "watchdog: $phase $why; killing it. Last output:"
tail -n 40 "$out" >&2
kill -TERM -- "-$pid" 2>/dev/null || kill "$pid" 2>/dev/null || true
wait "$pid" 2>/dev/null || rc=$?
trap - INT TERM; eval "$old_traps"
# A command killed by the watchdog never exits 0; a 0 here means it
# finished in the instant before the kill, and that result stands.
if [ "$rc" -eq 0 ]; then
log "watchdog: $phase finished before it could be killed; its exit 0 stands"
return 0
fi
remote_cancel "$name" "$sidf"
return 124
fi
if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then
log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago"
last_beat=$now
fi
done
wait "$pid" || rc=$?
trap - INT TERM; eval "$old_traps"
return "$rc"
}
proc_start() { ps -o lstart= -p "$1" 2>/dev/null | tr -s ' ' || true; }
fsize() { if [ -f "$1" ]; then wc -c <"$1" | tr -d ' '; else echo 0; fi; }
sha256() { if command -v sha256sum >/dev/null 2>&1; then sha256sum | cut -d' ' -f1; else shasum -a 256 | cut -d' ' -f1; fi; }
# TREE_SH: print the content tree of the current directory, the same way here
# and on the VM: at a checkout's root, `git add -A` into a copy of its index
# (tracked plus untracked files that are not ignored, the files pack uploads);
# elsewhere, the same into a scratch repository over the directory.
TREE_SH='idx=$(mktemp -d "${TMPDIR:-/tmp}/ubirun-tree.XXXXXX") || exit 1
if [ "$(git rev-parse --show-toplevel 2>/dev/null)" = "$(pwd -P)" ]; then
gi=$(git rev-parse --git-path index)
if [ "$(git config --bool core.splitIndex)" != true ] && [ -f "$gi" ]; then cp "$gi" "$idx/index"; fi
else
git init -q --bare "$idx/git" && export GIT_DIR="$idx/git" GIT_WORK_TREE=.
fi
GIT_INDEX_FILE="$idx/index" git add -A >/dev/null 2>&1 && GIT_INDEX_FILE="$idx/index" git write-tree
rc=$?; rm -rf "$idx"; exit $rc'
# vm_tree NAME QDEST: the content tree of QDEST on the VM.
vm_tree() {
remote "$1" "command -v git >/dev/null || { sudo apt-get update -qq && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -qq git; } >/dev/null 2>&1; cd $2 && bash -c $(sq "$TREE_SH")" </dev/null
}
base_sha() { git -C "$1" rev-parse -q --verify 'refs/remotes/origin/HEAD^{commit}' 2>/dev/null || echo none; }
# phase_status NAME PHASE: "exit <rc>" once the phase wrote its exit file,
# "alive" while its session leader runs, "lost" when the leader is gone with no
# exit file, "none" when it never started.
phase_status() {
local p="~/.ubirun-$2"
remote "$1" "if [ -f $p.exit ]; then echo exit \$(cat $p.exit); elif [ -s $p.pid ] && kill -0 \$(cat $p.pid) 2>/dev/null; then echo alive; elif [ -s $p.pid ]; then echo lost; else echo none; fi" </dev/null
}
# phase_start NAME PHASE INNER: start INNER on the VM in its own session,
# detached from this ssh connection. The leader records its pid, runs INNER with
# stdout and stderr in files, and writes the exit status by rename only after
# INNER's streams are closed.
phase_start() {
local p="~/.ubirun-$2" inner
inner=$(sq "echo \$\$ >$p.pid.tmp && mv $p.pid.tmp $p.pid; ( $3 ) >$p.out 2>$p.err </dev/null; echo \$? >$p.exit.tmp && mv $p.exit.tmp $p.exit")
remote "$1" "rm -f $p.exit $p.pid; : >$p.out; : >$p.err; if command -v setsid >/dev/null 2>&1; then setsid -f bash -c $inner </dev/null >/dev/null 2>&1; else nohup bash -c $inner </dev/null >/dev/null 2>&1 & fi" </dev/null >/dev/null
}
# stream_cmd PHASE OUT_OFFSET ERR_OFFSET: follow both output files from the
# given byte offsets until the session leader exits.
stream_cmd() {
local p="~/.ubirun-$1"
printf '%s' "i=0; while [ ! -s $p.pid ] && [ ! -f $p.exit ] && [ \$i -lt 600 ]; do sleep 0.1; i=\$((i + 1)); done; pid=\$(cat $p.pid 2>/dev/null); f=; [ -z \"\$pid\" ] || f=\"-f --pid=\$pid\"; tail -c +$(( $3 + 1 )) \$f $p.err >&2 & tail -c +$(( $2 + 1 )) \$f $p.out; wait"
}
PHASE_LIVE="" STREAM_PID="" RUN_KEY="" TREE=""
# on_hup: once a remote phase is live, HUP detaches: the VM keeps running under
# its TTL and the next run of the same command reattaches. Before that, HUP
# tears down like INT and TERM.
on_hup() {
[ -n "$PHASE_LIVE" ] || exit 130
[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null || true
trap - EXIT
rm -f "$(state_dir "$RUN_VM")/lease"
leftover "left=1 owner=$OWNER vm=$RUN_VM detached=1"
echo "UBI_DETACHED: vm=$RUN_VM key=${RUN_KEY:0:12} reattach: rerun the same command" >&2
exit 129
}
# detached NAME PHASE INNER: run INNER as a detached phase (or reattach to it),
# streaming its output to this process and to the VM's state dir
# (<phase>.out, <phase>.err and the combined <phase>.log) under the same
# heartbeat and watchdog as watched. RUN_TIMEOUT counts from the phase's
# recorded start, not from a reattach. Returns the phase's exit status, 124
# after a watchdog kill, 125 when the session is gone without an exit status.
detached() {
local name=$1 phase=$2 inner=$3 dir st rc=0 tries=0 start now idle elapsed last_beat why old_traps spid sst oo oe
load "$name"
dir=$(state_dir "$name")
local lout="$dir/$phase.out" lerr="$dir/$phase.err" out="$dir/$phase.log" p="~/.ubirun-$phase"
st=none
if [ -f "$dir/phase.$phase.start" ]; then
st=$(phase_status "$name" "$phase") || st=""
fi
case $st in
none)
: >"$lout"; : >"$lerr"; : >"$out"
date +%s >"$dir/phase.$phase.start"
phase_start "$name" "$phase" "$inner" || { log "could not start $phase on $name"; return 255; } ;;
alive) log "$phase is still running on $name; following it" ;;
"exit "*) ;;
lost) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; return 125 ;;
*) log "could not read the state of $phase on $name"; return 255 ;;
esac
PHASE_LIVE=1
start=$(cat "$dir/phase.$phase.start")
# A runner killed on this machine leaves its stream client behind; it would
# append to the same local copies.
if [ -f "$dir/stream" ] && read -r spid sst <"$dir/stream" && [ -n "$spid" ] && [ "$(proc_start "$spid")" = "$sst" ]; then
kill -TERM -- "-$spid" 2>/dev/null || true
fi
oo=$(fsize "$lout") oe=$(fsize "$lerr")
if [ "$st" != none ]; then head -c "$oo" "$lout"; head -c "$oe" "$lerr" >&2; fi
old_traps=$(trap -p INT TERM)
trap '[ -z "$STREAM_PID" ] || kill -TERM -- "-$STREAM_PID" 2>/dev/null; exit 130' INT TERM
last_beat=$(date +%s)
while :; do
touch "$out"
# shellcheck disable=SC2046
( exec python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
ssh $(ssh_opts) "ubi@$IP" "umask 022; $(stream_cmd "$phase" "$oo" "$oe")" ) \
</dev/null > >(tee -a "$lout" | tee -a "$out") 2> >(tee -a "$lerr" | tee -a "$out" >&2) &
STREAM_PID=$!
echo "$STREAM_PID $(proc_start "$STREAM_PID")" >"$dir/stream"
why=""
while :; do
sleep 1
kill -0 "$STREAM_PID" 2>/dev/null || break
now=$(date +%s)
elapsed=$(( now - start ))
idle=$(( now - $(mtime "$out") ))
if [ "$IDLE_TIMEOUT" -gt 0 ] && [ "$idle" -ge "$IDLE_TIMEOUT" ]; then
why="printed nothing for ${idle}s (UBI_IDLE_TIMEOUT=$IDLE_TIMEOUT)"
elif [ "$RUN_TIMEOUT" -gt 0 ] && [ "$elapsed" -ge "$RUN_TIMEOUT" ]; then
why="ran ${elapsed}s (UBI_RUN_TIMEOUT=$RUN_TIMEOUT)"
fi
[ -z "$why" ] || break
if [ "$HEARTBEAT" -gt 0 ] && [ $(( now - last_beat )) -ge "$HEARTBEAT" ]; then
log "heartbeat: $phase ${elapsed}s elapsed, last output ${idle}s ago"
last_beat=$now
fi
done
if [ -n "$why" ]; then
log "watchdog: $phase $why; killing it. Last output:"
tail -n 40 "$out" >&2
kill -TERM -- "-$STREAM_PID" 2>/dev/null || kill "$STREAM_PID" 2>/dev/null || true
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
# A phase that finished in the instant before the kill keeps its exit 0.
if [ "$(phase_status "$name" "$phase" 2>/dev/null)" = "exit 0" ]; then
log "watchdog: $phase finished before it could be killed; its exit 0 stands"
rc=0
else
remote_cancel "$name" "$p.pid"
rc=124
fi
break
fi
wait "$STREAM_PID" 2>/dev/null || true
STREAM_PID=""
st=$(phase_status "$name" "$phase") || st=""
case $st in
"exit "*) rc=${st#exit }; break ;;
lost|none) log "UBI_ATTACH_LOST: vm=$name phase=$phase: its session is gone and left no exit status ($p.exit)"; rc=125; break ;;
esac
tries=$(( tries + 1 ))
if [ "$tries" -ge 5 ]; then log "lost the connection to $phase on $name 5 times; giving up (the VM keeps it until teardown)"; rc=255; break; fi
log "the stream of $phase on $name dropped; following it again"
sleep "$POLL"
oo=$(fsize "$lout") oe=$(fsize "$lerr")
done
rm -f "$dir/stream"
trap - INT TERM; eval "$old_traps"
return "$rc"
}
# write_record DIR: the run record a later run of the same command reattaches
# by (key, tree, and this invocation for `attach NAME`). Remote files of each
# phase: ~/.ubirun-<phase>.{out,err,exit,pid} (setup, command, tier-<name>).
qwords() { local x; for x in "$@"; do printf '%q ' "$x"; done; }
write_record() {
{
printf 'RUN_KEY=%q\nTREE=%q\nPULL_ROOT=%q\nRESUME=%q\nRERUN_FAILED=%q\n' "$RUN_KEY" "$TREE" "$PULL_ROOT" "$RESUME" "$RERUN_FAILED"
printf 'RUN_ARGS=(%s)\n' "$(qwords "${RUN_ARGS[@]}")"
printf 'TIER_NAMES=(%s)\n' "$(qwords ${TIER_NAMES[@]+"${TIER_NAMES[@]}"})"
printf 'TIER_CMDS=(%s)\n' "$(qwords ${TIER_CMDS[@]+"${TIER_CMDS[@]}"})"
echo "# remote files: ~/.ubirun-<phase>.{out,err,exit,pid}; phase starts: phase.<phase>.start; lease: lease (pid start time)"
} >"$1/run.$$" && mv "$1/run.$$" "$1/run"
}
lease_take() { echo "$$ $(proc_start $$)" >"$1/lease.$$" && mv "$1/lease.$$" "$1/lease"; }
lease_busy() {
local pid st
[ -f "$1/lease" ] || return 1
read -r pid st <"$1/lease" || true
[ -n "$pid" ] && [ "$pid" != "$$" ] && kill -0 "$pid" 2>/dev/null && [ "$(proc_start "$pid")" = "$st" ]
}
# vm_alive NAME: the recorded VM still answers show. One confirmed gone is
# cleaned up (state dir and scratch record), so its record is never tried again.
# Returns 2 when the API gives no answer either way: the caller must not start a
# second VM beside one that may still be running.
vm_alive() {
local n=$1 out code
( load "$n"; [ "$CREATED" = yes ] && [ -n "$IP" ] ) 2>/dev/null || return 1
out=$(mktemp)
code=$(load "$n"; api "$out" vm "$LOCATION/$n" show)
rm -f "$out"
case $code in
200) return 0 ;;
404) log "run record for $n: the VM is gone; starting a new one"; cmd_down "$n" >/dev/null 2>&1 || true; return 1 ;;
*) log "run record for $n: could not check the VM (HTTP $code)"; return 2 ;;
esac
}
# ttl_short NAME SETUP: true when NAME's TTL (its created_at plus the name's
# -t<hours>, minus now) does not cover the run still to do plus
# REATTACH_MARGIN, with the numbers in TTL_WHY. The run still to do is the
# UBI_RUN_TIMEOUT bound of each phase that has not finished (setup when SETUP is
# set, then the command or every tier the plan does not reuse), counted from a
# started phase's recorded start; it is 0 when UBI_RUN_TIMEOUT is 0.
TTL_WHY=""
ttl_short() {
local name=$1 dir created left now need=0 p i phases=()
[[ "$name" =~ -t([0-9]{1,3})$ ]] || return 1
dir=$(state_dir "$name")
created=$(load "$name"; echo "$CREATE_AT")
now=$(date +%s)
left=$(( created + 10#${BASH_REMATCH[1]} * 3600 - now ))
[ -z "$2" ] || phases+=(setup)
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
for i in "${!TIER_NAMES[@]}"; do [ "${TIER_ACT[i]}" = reuse ] || phases+=("tier-${TIER_NAMES[i]}"); done
else
phases+=(command)
fi
if [ "$RUN_TIMEOUT" -gt 0 ]; then
for p in "${phases[@]}"; do
if [ -f "$dir/phase.$p.start" ]; then i=$(( RUN_TIMEOUT - (now - $(cat "$dir/phase.$p.start")) )); [ "$i" -gt 0 ] || i=0
else i=$RUN_TIMEOUT; fi
need=$(( need + i ))
done
fi
TTL_WHY="ttl_left=${left}s needs=$(( need + REATTACH_MARGIN ))s (run still to do ${need}s + UBI_REATTACH_MARGIN=${REATTACH_MARGIN}s)"
[ "$left" -lt $(( need + REATTACH_MARGIN )) ]
}
# find_record: the VM of a run record with this RUN_KEY whose VM still exists.
find_record() {
local f n
for f in "$STATE_ROOT"/*/run; do
[ -f "$f" ] || continue
grep -qx "RUN_KEY=$RUN_KEY" "$f" || continue
n=$(basename "$(dirname "$f")")
vm_alive "$n" && { echo "$n"; return 0; }
[ $? -ne 2 ] || return 2
done
}
# Tier checkpoints: write-once records <tier>.<epoch>.<pid>.json beside a copy
# of the tier's log, under CHECKPOINT_DIR/<run key>/, published under a lock
# with a monotonic generation; the newest generation per tier decides.
CKPT_PY='
import fcntl, hashlib, json, os, shutil, sys, time
op, d = sys.argv[1], sys.argv[2]
def records():
out = []
for f in (os.listdir(d) if os.path.isdir(d) else []):
if not f.endswith(".json") or f.startswith("."):
continue
try:
r = json.load(open(os.path.join(d, f)))
except (OSError, ValueError):
continue
if isinstance(r, dict) and isinstance(r.get("generation"), int):
r["_file"] = f
out.append(r)
return out
def newest(rs, tier):
rs = [r for r in rs if r.get("tier") == tier]
return max(rs, key=lambda r: r["generation"]) if rs else None
def sha(p):
h = hashlib.sha256()
with open(p, "rb") as fh:
for b in iter(lambda: fh.read(65536), b""):
h.update(b)
return h.hexdigest()
if op == "decide":
now, maxage, resume, rerun = int(sys.argv[3]), int(sys.argv[4]), sys.argv[5] == "1", sys.argv[6] == "1"
rs = records()
for tier in sys.argv[7:]:
r = newest(rs, tier)
f = r["_file"] if r else "-"
age = now - int(r.get("finished_at") or r.get("started_at") or now) if r else 0
dur = int(r.get("duration_s") or 0) if r else 0
code = str(r.get("exit", "-")) if r else "-"
log = os.path.join(d, str(r.get("log"))) if r and r.get("log") else ""
if not resume:
act, why = "run", "--no-resume"
elif r is None:
act, why = "run", "no record"
elif r.get("status") == "start":
act, why = "run", "interrupted (only a start record, %s)" % f
elif r.get("status") == "pass":
if age >= maxage:
act, why = "run", "pass %s is %ds old (UBI_CHECKPOINT_MAX_AGE=%d)" % (f, age, maxage)
elif not log or not os.path.isfile(log) or sha(log) != r.get("log_sha256"):
act, why = "run", "pass %s: its log does not re-hash" % f
else:
act, why = "reuse", "-"
elif r.get("status") == "fail":
act, why = ("rerun", "prior fail %s (exit %s; --rerun-failed)" % (f, code)) if rerun else ("blocked", "-")
else:
act, why = "run", "unreadable record %s" % f
print("\t".join([tier, act, f, str(age), str(dur), code, why]))
elif op == "publish":
tier, status, vm, started, finished, code, log, startgen, key, tree, pid = sys.argv[3:14]
os.makedirs(d, exist_ok=True)
with open(os.path.join(d, ".lock"), "a") as lk:
fcntl.flock(lk, fcntl.LOCK_EX)
rs = records()
gen = max([r["generation"] for r in rs] + [0]) + 1
if status != "start":
r = newest(rs, tier)
if r is None or str(r["generation"]) != startgen:
print("stale writer: its start record is generation %s, the newest for %s is %s" % (startgen, tier, r["generation"] if r else "none"), file=sys.stderr)
sys.exit(2)
rec = {"tier": tier, "status": status, "generation": gen, "vm": vm, "key": key, "tree": tree, "started_at": int(started)}
if status != "start":
rec.update(finished_at=int(finished), duration_s=int(finished) - int(started), exit=int(code))
epoch = int(time.time())
while os.path.exists(os.path.join(d, "%s.%d.%s.json" % (tier, epoch, pid))) or os.path.exists(os.path.join(d, "%s.%d.%s.log" % (tier, epoch, pid))):
epoch += 1
stem = "%s.%d.%s" % (tier, epoch, pid)
if status != "start":
shutil.copyfile(log, os.path.join(d, stem + ".log"))
rec["log"], rec["log_sha256"] = stem + ".log", sha(os.path.join(d, stem + ".log"))
tmp = os.path.join(d, ".%s.tmp" % stem)
with open(tmp, "w") as fh:
json.dump(rec, fh, sort_keys=True)
fh.write("\n")
os.link(tmp, os.path.join(d, stem + ".json"))
os.unlink(tmp)
print(gen)
'
# ckpt_publish TIER STATUS VM STARTED FINISHED EXIT LOG STARTGEN: print the new
# record's generation; fails (nothing published) for a stale writer.
ckpt_publish() { python3 -c "$CKPT_PY" publish "$CHECKPOINT_DIR/$RUN_KEY" "$@" "$RUN_KEY" "$TREE" "$$"; }
TIER_ACT=() TIER_TO_RUN=0 TIER_PASSED=0 TIER_REUSED="" TIER_FAILED="" TIER_PRIOR="" TIER_SAVED=0 TIER_BLOCKED="" TIER_BLOCKED_RC=1
# tier_plan: decide each tier from its newest checkpoint and print one
# UBI_VERIFY_RESUME line per tier before anything runs.
tier_plan() {
local plan t act f age dur code why
plan=$(python3 -c "$CKPT_PY" decide "$CHECKPOINT_DIR/$RUN_KEY" "$(date +%s)" "$CHECKPOINT_MAX_AGE" "$RESUME" "$RERUN_FAILED" "${TIER_NAMES[@]}") \
|| die "verify: could not read the tier checkpoints in $CHECKPOINT_DIR/$RUN_KEY"
while IFS=$'\t' read -r t act f age dur code why; do
case $act in
reuse)
echo "UBI_VERIFY_RESUME: tier=$t reuse $f (${age}s)" >&2
TIER_REUSED=${TIER_REUSED:+$TIER_REUSED,}$t TIER_SAVED=$(( TIER_SAVED + dur )) TIER_PASSED=$(( TIER_PASSED + 1 )) ;;
blocked)
echo "UBI_VERIFY_RESUME: tier=$t fail $f (${age}s, exit $code): not rerun without --rerun-failed" >&2
if [ -z "$TIER_BLOCKED" ]; then TIER_BLOCKED=$t; if [[ "$code" =~ ^[0-9]+$ ]]; then TIER_BLOCKED_RC=$code; fi; fi ;;
rerun)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_PRIOR=${TIER_PRIOR:+$TIER_PRIOR,}$t:$code TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
*)
echo "UBI_VERIFY_RESUME: tier=$t run: $why" >&2
TIER_TO_RUN=$(( TIER_TO_RUN + 1 )) ;;
esac
TIER_ACT+=("$act")
done <<<"$plan"
}
verify_line() {
echo "UBI_VERIFY: tree=${TREE:0:12} key=${RUN_KEY:0:12} tiers=${#TIER_NAMES[@]} passed=$TIER_PASSED reused=${TIER_REUSED:-none} failed=${TIER_FAILED:-none} prior_failures=${TIER_PRIOR:-none} saved_s=$TIER_SAVED"
}
# run_tiers NAME QDEST: run every tier the plan did not reuse, in order, each
# as its own detached phase; stop at the first failure and return its status.
# A pass is recorded only when the VM's tree after the tier equals the tree
# before it.
run_tiers() {
local name=$1 qdest=$2 dir i t phase before after trc started finished gen
dir=$(state_dir "$name")
for i in "${!TIER_NAMES[@]}"; do
[ "${TIER_ACT[i]}" != reuse ] || continue
t=${TIER_NAMES[i]} phase="tier-${TIER_NAMES[i]}"
if [ ! -f "$dir/phase.$phase.start" ]; then
before=$(vm_tree "$name" "$qdest") || before=unavailable
echo "$before" >"$dir/phase.$phase.tree"
if gen=$(ckpt_publish "$t" start "$name" "$(date +%s)" 0 0 "" 0); then echo "$gen" >"$dir/phase.$phase.gen"
else echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t start record (the tier still runs)" >&2; fi
fi
log "running tier $t on $name: ${TIER_CMDS[i]}"
trc=0
detached "$name" "$phase" "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "${TIER_CMDS[i]}")" || trc=$?
started=$(cat "$dir/phase.$phase.start" 2>/dev/null || date +%s) finished=$(date +%s)
log "tier $t exited $trc after $(( finished - started ))s"
case $trc in 125|255) TIER_FAILED=$t; return "$trc" ;; esac
gen=$(cat "$dir/phase.$phase.gen" 2>/dev/null || echo 0)
if [ "$trc" -ne 0 ]; then
ckpt_publish "$t" fail "$name" "$started" "$finished" "$trc" "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t fail record (the result stands)" >&2
TIER_FAILED=$t
return "$trc"
fi
TIER_PASSED=$(( TIER_PASSED + 1 ))
after=$(vm_tree "$name" "$qdest") || after=unavailable
before=$(cat "$dir/phase.$phase.tree" 2>/dev/null || echo unavailable)
if [ "$after" != "$before" ] || [ "$after" = unavailable ]; then
echo "UBI_VERIFY_TREE_MOVED: tier=$t before=${before:0:12} after=${after:0:12}: a tier must leave the source tree as it found it; no pass record" >&2
continue
fi
ckpt_publish "$t" pass "$name" "$started" "$finished" 0 "$dir/$phase.log" "$gen" >/dev/null \
|| echo "UBI_CHECKPOINT_WRITE_FAILED: tier=$t pass record (the result stands)" >&2
done
}
# vm_rows: "location name owner expires created" for every VM in the project;
# owner is "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner
# did not name; expires is the epoch the name's -t<hours> TTL ends at, or "-";
# created is the name's epoch, or "-".
vm_rows() {
local loc name owner created expires list
list=$(cli vm list -N -f location,name) || return 1
if grep -Evq '^[[:blank:]]*([a-z0-9-]+[[:blank:]]+[a-z0-9-]+)?[[:blank:]]*$' <<<"$list"; then
log "the VM list answer is unreadable (a line is not 'location name')"
return 1
fi
while read -r loc name; do
[ -n "$name" ] || continue
owner=. created="" expires=-
if [[ "$name" =~ ^${PREFIX}-([a-z][a-z0-9]{0,11})-([0-9]{10})- ]]; then
owner=${BASH_REMATCH[1]} created=${BASH_REMATCH[2]}
elif [[ "$name" =~ ^${PREFIX}-([0-9]{10})- ]]; then
owner=- created=${BASH_REMATCH[1]}
fi
if [ -n "$created" ] && [[ "$name" =~ -t([0-9]{1,3})$ ]]; then
expires=$(( 10#$created + 10#${BASH_REMATCH[1]} * 3600 ))
fi
echo "$loc $name $owner $expires ${created:--}"
done <<<"$list"
}
# reap: destroy the caller's own VMs whose TTL ended more than REAP_GRACE
# seconds ago, and its TTL-less ubirun-<UBI_OWNER>-* names at least LEGACY_HOURS
# old. Every other VM is never touched, whatever its TTL says: a delete stays
# with the owner that created the VM. Other owners' expired VMs are only counted.
cmd_reap() {
local now rows loc name owner expires created n=0 failed=0 others=0
[[ "$LEGACY_HOURS" =~ ^[1-9][0-9]{0,3}$ ]] || die "UBI_LEGACY_HOURS must be a whole number of hours from 1 to 9999 with no leading zero (how old a TTL-less ubirun-* VM must be before reap destroys it)"
now=$(date +%s)
rows=$(vm_rows) || { log "reap: could not list the project's VMs; nothing destroyed"; return 1; }
while read -r loc name owner expires created; do
[ -n "$name" ] || continue
if [ "$owner" != "$OWNER" ]; then
[ "$expires" = - ] || [ "$now" -lt $(( expires + REAP_GRACE )) ] || others=$(( others + 1 ))
continue
fi
if [ "$expires" != - ]; then
[ "$now" -ge $(( expires + REAP_GRACE )) ] || continue
log "reaping expired $loc/$name (owner $owner; its TTL ended $(( (now - expires) / 60 )) min ago)"
else
[ -n "${UBI_OWNER:-}" ] && [ "$created" != - ] || continue
[ $(( now - 10#$created )) -ge $(( LEGACY_HOURS * 3600 )) ] || continue
log "reaping TTL-less $loc/$name (owner $owner; $(( (now - 10#$created) / 3600 ))h old, legacy sweep at ${LEGACY_HOURS}h)"
fi
n=$(( n + 1 ))
cmd_down "$name" -l "$loc" || { failed=$(( failed + 1 )); log "failed to reap $loc/$name"; }
done <<<"$rows"
[ "$n" -gt 0 ] || log "reap: no expired VMs"
[ "$others" -eq 0 ] || log "reap: left $others expired VMs of other owners; only their owners reap them"
[ "$failed" -eq 0 ] || { log "reap: $failed of $n expired VMs could not be destroyed"; return 1; }
}
cmd_gc() {
local hours=${1:-$GC_HOURS} now rows loc name owner failed=0
[[ "$hours" =~ ^[0-9]+$ ]] || die "gc: HOURS must be a whole number (or set UBI_GC_HOURS)"
if [ "$hours" -eq 0 ]; then log "gc: no sweep (give HOURS > 0 or set UBI_GC_HOURS); nothing listed, nothing destroyed"; return 0; fi
now=$(date +%s)
rows=$(vm_rows) || { log "gc: could not list the project's VMs; nothing destroyed"; return 1; }
while read -r loc name owner _; do
[ "$owner" = "$OWNER" ] || continue
[[ "$name" =~ -([0-9]{10})- ]] || continue
if [ $(( (now - BASH_REMATCH[1]) / 3600 )) -ge "$hours" ]; then
log "destroying stale $loc/$name (owner $OWNER, older than ${hours}h)"
cmd_down "$name" -l "$loc" || { failed=$(( failed + 1 )); log "failed to destroy $loc/$name"; }
fi
done <<<"$rows"
[ "$failed" -eq 0 ] || { log "gc: $failed stale VMs could not be destroyed"; return 1; }
}
cmd_list() {
case ${1:-} in
"") cli vm list ;;
--mine) vm_rows | awk -v o="$OWNER" '$3==o {print $1 " " $2}' ;;
*) die "list: unknown option $1 (use --mine)" ;;
esac
}
# usage: VMs and vCPUs per owner, from each VM's size (standard-16 = 16 vCPUs).
# Exits 1 with no table when the VM list or any VM's size cannot be read, so a
# partial count is never taken for the project's usage. A VM destroyed between
# the list and its size read (404) no longer counts and is not a failure.
cmd_usage() {
local tmp rows loc name owner f i=0 unread=0
rows=$(vm_rows) || { log "usage: could not list the project's VMs"; return 1; }
tmp=$(mktemp -d)
while read -r loc name owner _; do
[ -n "$name" ] || continue
i=$(( i + 1 ))
{
case $(api "$tmp/$i.show" vm "$loc/$name" show) in
200) sed -n "s/^size: /$owner /p" "$tmp/$i.show" >"$tmp/$i" ;;
404) ;;
*) : >"$tmp/$i" ;;
esac
rm -f "$tmp/$i.show"
} &
[ $(( i % 8 )) -ne 0 ] || wait
done <<<"$rows"
wait
for f in "$tmp"/*; do [ ! -e "$f" ] || [ -s "$f" ] || unread=$(( unread + 1 )); done
if [ "$unread" -gt 0 ]; then
rm -rf "$tmp"
log "usage: could not read the size of $unread of $i VMs"
return 1
fi
find "$tmp" -type f -exec cat {} + | awk -v me="$OWNER" '
{ n = $2; sub(/^.*-/, "", n); o = $1 == "-" ? "(untagged)" : $1 == "." ? "(other)" : $1
vms[o]++; cpu[o] += n; total += n; count++ }
END {
printf "%-14s %4s %6s\n", "OWNER", "VMS", "VCPUS"
for (o in cpu) printf "%-14s %4d %6d%s\n", o, vms[o], cpu[o], (o == me ? " (you)" : "") | "sort -k3,3nr"
close("sort -k3,3nr")
printf "%-14s %4d %6d\n", "total", count, total
}'
rm -rf "$tmp"
}
write_state() {
printf 'NAME=%q\nLOCATION=%q\nSIZE=%q\nIP=%q\nCREATED=%q\nCREATE_AT=%q\nCREATE_PID=%q\n' \
"$1" "$2" "$3" "$4" "$5" "$6" "$$" >"$(state_dir "$1")/env"
}
# record_vm NAME LOCATION [OWNER]: append to the scratch VM-id file (record-before-create).
record_vm() {
mkdir -p "$(dirname "$VM_RECORD")" 2>/dev/null || return 0
printf '%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "${3:-$OWNER}" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$$" >>"$VM_RECORD"
}
# forget_vm NAME: drop the VM from the scratch file once it is confirmed gone.
forget_vm() {
[ -f "$VM_RECORD" ] || return 0
local tmp="$VM_RECORD.$$"
awk -F'\t' -v n="$1" '$1!=n' "$VM_RECORD" >"$tmp" && mv "$tmp" "$VM_RECORD"
}
# records: print the scratch file (VMs this machine created and has not confirmed gone).
cmd_records() {
echo "record: $VM_RECORD"
[ -s "$VM_RECORD" ] || { echo "(no VMs recorded)"; return 0; }
printf '%-40s %-14s %-12s %-20s %s\n' NAME LOCATION OWNER CREATED_AT PID
awk -F'\t' '{ printf "%-40s %-14s %-12s %-20s %s\n", $1, $2, $3, $4, $5 }' "$VM_RECORD"
}
# vm_gone_final SEEN CREATED CREATE_AT CREATE_PID: a "not found" for the VM is
# final: it was seen, its create was answered, or the create was sent more than
# UBI_CREATE_GRACE seconds ago by a process that is gone (or is this one).
vm_gone_final() {
[ "$1" = 1 ] || [ "$2" != unknown ] || { [ $(( $(date +%s) - $3 )) -ge "$CREATE_GRACE" ] \
&& { [ "$4" = "$$" ] || ! kill -0 "$4" 2>/dev/null; }; }
}
# Recovery bundles. RB_PY reads the secret on stdin and stretches it with
# PBKDF2-SHA256 (200000 rounds, a random salt per bundle) into an encryption key
# and a MAC key; the encryption key reaches openssl on a pipe fd (-pass fd:N).
# A bundle is UBIRB1, the salt, the ciphertext of a JSON map of the state files
# with the VM name and owner, and an HMAC over all of it, checked with
# compare_digest before anything is decrypted. open NAME OWNER exits 3 on any
# failure, a sealed name or owner other than NAME and OWNER included.
RB_PY='
import base64, hashlib, hmac, json, os, subprocess, sys
MAGIC = b"UBIRB1\n"
FILES = ("key", "key.pub", "env", "tags", "sink")
secret = sys.stdin.buffer.read()
if not secret:
sys.exit(2)
def keys(salt):
k = hashlib.pbkdf2_hmac("sha256", secret, b"ubi-runner recovery bundle v1" + salt, 200000, 64)
return k[:32].hex().encode(), k[32:]
def ossl(mode, data, enc_key):
r, w = os.pipe()
os.write(w, enc_key + b"\n")
os.close(w)
try:
p = subprocess.run(["openssl", "enc", mode, "-aes-256-cbc", "-pbkdf2", "-iter", "1000", "-salt", "-pass", "fd:%d" % r],
input=data, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, pass_fds=(r,))
finally:
os.close(r)
return p.returncode, p.stdout
if sys.argv[1] == "seal":
src, out, name, owner = sys.argv[2:6]
salt = os.urandom(16)
enc_key, mac_key = keys(salt)
files = {}
for n in FILES:
if os.path.isfile(os.path.join(src, n)):
with open(os.path.join(src, n), "rb") as f:
files[n] = base64.b64encode(f.read()).decode()
rc, ct = ossl("-e", json.dumps({"v": 1, "name": name, "owner": owner, "files": files}).encode(), enc_key)
if rc != 0 or not ct:
sys.exit(1)
body = MAGIC + salt + ct
os.makedirs(os.path.dirname(out), mode=0o700, exist_ok=True)
tmp = "%s.tmp.%d" % (out, os.getpid())
with os.fdopen(os.open(tmp, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as f:
f.write(body + hmac.new(mac_key, body, hashlib.sha256).digest())
os.rename(tmp, out)
else:
bundle, dest, name, owner = sys.argv[2:6]
with open(bundle, "rb") as f:
data = f.read()
body, mac = data[:-32], data[-32:]
if len(data) < len(MAGIC) + 16 + 33 or not body.startswith(MAGIC):
sys.exit(3)
enc_key, mac_key = keys(body[len(MAGIC):len(MAGIC) + 16])
if not hmac.compare_digest(mac, hmac.new(mac_key, body, hashlib.sha256).digest()):
sys.exit(3)
rc, plain = ossl("-d", body[len(MAGIC) + 16:], enc_key)
try:
sealed = json.loads(plain) if rc == 0 else {}
except ValueError:
sealed = {}
files = sealed.get("files") if isinstance(sealed, dict) else None
if sealed.get("name") != name or sealed.get("owner") != owner or not isinstance(files, dict) \
or "env" not in files or "key" not in files or any(n not in FILES for n in files):
sys.exit(3)
for n, b in files.items():
with os.fdopen(os.open(os.path.join(dest, n), os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600), "wb") as f:
f.write(base64.b64decode(b))
'
# bundle_save NAME: (re)write NAME's recovery bundle when UBI_RECOVERY_DIR is
# set. A counted run stops on a failed write; any other run warns and goes on.
bundle_save() {
[ -n "$RECOVERY_DIR" ] || return 0
printf %s "$RECOVERY_SECRET" | python3 -c "$RB_PY" seal "$(state_dir "$1")" "$RECOVERY_DIR/$OWNER/$1.bundle" "$1" "$OWNER" && return 0
[ "$COUNTED" != 1 ] || die "UBI_RECOVERY_UNAVAILABLE: could not write the recovery bundle for $1 under $RECOVERY_DIR (needs openssl and a writable directory); a counted run needs it"
log "WARNING: could not write the recovery bundle for $1 under $RECOVERY_DIR (UBI_RECOVERY_UNAVAILABLE); continuing without one"
}
# recovery_preflight: a counted run needs a bundle another controller can open.
recovery_preflight() {
[ -n "$RECOVERY_DIR" ] || die "UBI_RECOVERY_REQUIRED: a counted run (--counted or UBI_COUNTED=1) needs a recovery bundle; set UBI_RECOVERY_DIR to a directory another controller can read (on Capy, a drive folder), then rerun"
command -v openssl >/dev/null && [ -n "$RECOVERY_SECRET" ] \
|| die "UBI_RECOVERY_UNAVAILABLE: a counted run needs openssl and a secret (UBI_RECOVERY_KEY or UBICLOUD_API_KEY) to write its recovery bundle"
}
# recover_failed NAME CODE FIX: one RECOVER_FAILED line; the bundle is kept.
recover_failed() { echo "RECOVER_FAILED: vm=$1 code=$2 fix=$3"; RECOVER_FAILS=$(( RECOVER_FAILS + 1 )); }
# recover [--owner OLD] [--dir DIR]: rebuild local state for every VM with a
# bundle under DIR/OLD (default UBI_RECOVERY_DIR and this owner). A live VM
# prints RECOVERED, a VM confirmed gone drops its bundle (RECOVER_GONE), and
# anything else prints RECOVER_FAILED and keeps the bundle. State already on
# this machine is never overwritten.
RECOVER_FAILS=0
cmd_recover() {
local from=$OWNER dir=$RECOVERY_DIR b name dest tmp code out ip tags
while [ $# -gt 0 ]; do
case $1 in
--owner) from=$2; shift 2 ;;
--dir) dir=$2; shift 2 ;;
*) die "recover: unknown option $1" ;;
esac
done
[[ "$from" =~ ^[a-z][a-z0-9]{0,11}$ ]] || die "recover: --owner must be an owner tag as the owner command prints it"
[ -n "$dir" ] || die "UBI_RECOVERY_REQUIRED: recover reads bundles from UBI_RECOVERY_DIR (or --dir DIR), and neither is set"
command -v openssl >/dev/null || die "UBI_RECOVERY_UNAVAILABLE: openssl is required to open recovery bundles"
mkdir -p "$STATE_ROOT"
out=$(mktemp)
for b in "$dir/$from"/*.bundle; do
[ -f "$b" ] || { log "recover: no bundles under $dir/$from"; continue; }
name=$(basename "$b" .bundle)
valid_name "$name" || continue
dest=$(state_dir "$name")
if [ -e "$dest" ]; then
recover_failed "$name" RECOVER_EXISTS "this machine already has state for $name; use it, or run down $name"
continue
fi
tmp=$(mktemp -d "$STATE_ROOT/.recover.XXXXXX")
local NAME="" LOCATION="" SIZE="" IP="" CREATED="" CREATE_AT=0 CREATE_PID="" line
if printf %s "$RECOVERY_SECRET" | python3 -c "$RB_PY" open "$b" "$tmp" "$name" "$from"; then
while IFS= read -r line; do
[[ "$line" =~ ^(NAME|LOCATION|SIZE|IP|CREATED|CREATE_AT|CREATE_PID)=([A-Za-z0-9._:/-]*)$ ]] || continue
printf -v "${BASH_REMATCH[1]}" %s "${BASH_REMATCH[2]}"
done <"$tmp/env"
fi
if [ "$NAME" != "$name" ] || ! [[ "$CREATE_AT" =~ ^[0-9]+$ && "$LOCATION" =~ ^[a-z0-9-]+$ ]]; then
rm -rf "$tmp"
recover_failed "$name" RECOVER_DECRYPT_FAILED "the bundle does not authenticate under this secret, or it was sealed for another VM or owner; set UBI_RECOVERY_KEY (or UBICLOUD_API_KEY) to the creating controller's"
continue
fi
code=$(api "$out" vm "$LOCATION/$name" show)
case $code in
200)
ip=$(sed -n 's/^ip4: //p' "$out")
printf 'NAME=%q\nLOCATION=%q\nSIZE=%q\nIP=%q\nCREATED=yes\nCREATE_AT=%q\nCREATE_PID=\nADOPTED_BY=%q\n' \
"$name" "$LOCATION" "$SIZE" "${ip:-$IP}" "$CREATE_AT" "$OWNER" >"$tmp/env"
if [ -e "$dest" ]; then
rm -rf "$tmp"
recover_failed "$name" RECOVER_EXISTS "state for $name appeared on this machine while recovering; use it"
continue
fi
mv "$tmp" "$dest"
record_vm "$name" "$LOCATION" "$from"
tags=$(grep -E '^[A-Za-z0-9_.-]+=[A-Za-z0-9_.:/@+-]*$' "$dest/tags" 2>/dev/null | paste -s -d, - || true)
echo "RECOVERED: vm=$name ip=${ip:-$IP} owner=$from tags=$tags" ;;
404)
rm -rf "$tmp"
if vm_gone_final 0 "$CREATED" "$CREATE_AT" ""; then
rm -f "$b"
echo "RECOVER_GONE: vm=$name"
else
recover_failed "$name" RECOVER_API_UNAVAILABLE "its create may still be in flight; rerun recover after UBI_CREATE_GRACE (${CREATE_GRACE}s)"
fi ;;
*)
rm -rf "$tmp"
recover_failed "$name" RECOVER_API_UNAVAILABLE "the Ubicloud API answered HTTP $code; nothing changed, rerun recover" ;;
esac
done
rm -f "$out"
[ "$RECOVER_FAILS" -eq 0 ]
}
# sink_check SPEC: https-put:<https URL> is the only network sink; file:<dir>
# only under UBI_SINK_ALLOW_FILE=1 (tests). The URL is never printed.
sink_check() {
case $1 in
https-put:https://*) [[ "${1#https-put:}" =~ ^https://[^[:space:]]+$ ]] && return 0 ;;
file:/*) [ "${UBI_SINK_ALLOW_FILE:-}" = 1 ] && return 0 ;;
esac
die "UBI_SINK_INVALID: a sink is https-put:<presigned https URL> (file:<absolute dir> only with UBI_SINK_ALLOW_FILE=1); got a '${1%%:*}:' spec. Nothing was created"
}
# SINK_SH: the VM-side pusher. Every SINK_INTERVAL seconds it packs
# heartbeat.json and the SINK_PATHS entries under ~ (never ~/.ubirun-*, which
# hold --pass values and the URL) into results.tgz and pushes it when they
# changed, or every fifth interval regardless. A snapshot tar cannot read, or
# one over UBI_SINK_MAX_MB, is skipped, so the last good one stays in the sink.
# A failed or skipped push backs off and retries; it never ends the loop.
# --once pushes one time and exits 1 with "seq= last_push= [sink_skipped=]".
SINK_SH='#!/usr/bin/env bash
. ~/.ubirun-sink.conf || exit 1
once=0
[ "${1:-}" != --once ] || once=1
[ "$once" = 1 ] || echo $$ > ~/.ubirun-sink.pid
seq=$(cat ~/.ubirun-sink.seq 2>/dev/null || echo 0)
last=$(cat ~/.ubirun-sink.last 2>/dev/null || echo never)
present() { P=(); local p; for p in ${SINK_PATHS[@]+"${SINK_PATHS[@]}"}; do [ ! -e "$HOME/$p" ] || P+=("$p"); done; }
digest() { present; [ ${#P[@]} -gt 0 ] || { echo none; return 0; }; tar -cf - --exclude=".ubirun-*" -C "$HOME" "${P[@]}" 2>/dev/null | cksum; }
beat() { printf "{\"vm\": \"%s\", \"owner\": \"%s\", \"seq\": %d, \"time\": \"%s\", \"last_push\": \"%s\", \"sink_skipped\": \"%s\"}\n" \
"$SINK_VM" "$SINK_OWNER" "$seq" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$last" "$2" > "$1/heartbeat.json"; }
push() {
local d rc=0
skipped=""
d=$(mktemp -d "$HOME/.ubirun-sink.XXXXXX") || return 1
seq=$((seq + 1)); echo "$seq" > ~/.ubirun-sink.seq
present
beat "$d" ""
if [ ${#P[@]} -gt 0 ]; then
tar -czf "$d/results.tgz" --exclude=".ubirun-*" -C "$d" heartbeat.json -C "$HOME" "${P[@]}" 2>/dev/null || rc=$?
[ "$rc" -le 1 ] || skipped=tar_failed
[ -n "$skipped" ] || [ "$(wc -c < "$d/results.tgz")" -le $((SINK_MAX_MB * 1048576)) ] || skipped=size
else
tar -czf "$d/results.tgz" -C "$d" heartbeat.json
fi
if [ -n "$skipped" ]; then rm -rf "$d"; return 2; fi
rc=0
case $SINK_KIND in
file) { mkdir -p "$SINK_DIR/$SINK_VM" && cp "$d/results.tgz" "$SINK_DIR/$SINK_VM/.results.tgz.$$" \
&& mv -f "$SINK_DIR/$SINK_VM/.results.tgz.$$" "$SINK_DIR/$SINK_VM/results.tgz"; } 2>/dev/null || rc=1 ;;
https-put) curl -sS --fail --max-time 120 -K "$HOME/.ubirun-sink.curl" -T "$d/results.tgz" >/dev/null 2>&1 || rc=1 ;;
*) rc=1 ;;
esac
rm -rf "$d"
[ "$rc" -ne 0 ] || { last=$(date +%s); echo "$last" > ~/.ubirun-sink.last; }
return "$rc"
}
if [ "$once" = 1 ]; then push && exit 0; echo "seq=$seq last_push=$last${skipped:+ sink_skipped=$skipped}"; exit 1; fi
h="" beats=0 backoff=0
while :; do
n=$(digest)
if [ "$n" != "$h" ] || [ "$beats" -ge 5 ]; then
if push; then h=$n; beats=0; backoff=0
else backoff=$(( backoff ? backoff * 2 : SINK_INTERVAL )); [ "$backoff" -le 900 ] || backoff=900; fi
else
beats=$((beats + 1))
fi
sleep $(( SINK_INTERVAL + backoff ))
done
'
# SINK_STOP: end the running pusher (its process group, so an upload in
# flight dies with it) and wait for it, so no older snapshot lands later.
SINK_STOP='p=$(cat ~/.ubirun-sink.pid 2>/dev/null); if [ -n "$p" ]; then kill -TERM -- "-$p" 2>/dev/null || kill -TERM "$p" 2>/dev/null; for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do kill -0 "$p" 2>/dev/null || break; sleep 0.5; done; kill -KILL -- "-$p" 2>/dev/null; rm -f ~/.ubirun-sink.pid; fi;'
# sink_start NAME SPEC PATH...: write the pusher's config (600; the URL only in
# ~/.ubirun-sink.curl, over ssh stdin), upload it and start it outside any ssh
# session, and mark the state dir so down flushes it.
sink_start() {
local name=$1 spec=$2 kind interval=$SINK_INTERVAL url sdir=""
shift 2
kind=${spec%%:*}
[ "$kind" != file ] || sdir=${spec#file:}
[ "$kind" = file ] || [ "$interval" -ge 60 ] || interval=60
{ printf 'SINK_KIND=%q\nSINK_DIR=%q\nSINK_VM=%q\nSINK_OWNER=%q\nSINK_INTERVAL=%q\nSINK_MAX_MB=%q\n' \
"$kind" "$sdir" "$name" "$OWNER" "$interval" "$SINK_MAX_MB"
printf 'SINK_PATHS=(%s)\n' "$(qwords "$@")"; } | remote "$name" 'umask 077 && cat > ~/.ubirun-sink.conf && chmod 600 ~/.ubirun-sink.conf'
if [ "$kind" = https-put ]; then
url=${spec#https-put:}; url=${url//\\/\\\\}; url=${url//\"/\\\"}
printf 'url = "%s"\n' "$url" | remote "$name" 'umask 077 && cat > ~/.ubirun-sink.curl && chmod 600 ~/.ubirun-sink.curl'
fi
printf %s "$SINK_SH" | remote "$name" 'umask 077 && cat > ~/.ubirun-sink.sh'
remote "$name" "$SINK_STOP"' if command -v setsid >/dev/null 2>&1; then setsid -f bash ~/.ubirun-sink.sh </dev/null >>~/.ubirun-sink.log 2>&1; else nohup bash ~/.ubirun-sink.sh </dev/null >>~/.ubirun-sink.log 2>&1 & fi' </dev/null >/dev/null
: >"$(state_dir "$name")/sink"
log "sink: pushing $# path(s) to a $kind sink every ${interval}s (pid in ~/.ubirun-sink.pid on $name)"
}
# sink_flush NAME: stop the pusher, then one last push before down destroys
# NAME (150 s cap, above curl's 120). A failure prints SINK_FLUSH_FAILED and
# adds NAME to SINK_FLUSH_FAILED, which turns a passing run or verify of that
# VM into exit 1; the VM is destroyed either way.
sink_flush() {
local out
if out=$(remote "$1" "$SINK_STOP"' timeout 150 bash ~/.ubirun-sink.sh --once' </dev/null 2>/dev/null); then
log "sink: final push done for $1"
return 0
fi
SINK_FLUSH_FAILED="$SINK_FLUSH_FAILED $1 "
out=$(tail -n 1 <<<"$out")
echo "SINK_FLUSH_FAILED: vm=$1 ${out:-seq=unknown last_push=unknown} (results in the sink may be incomplete)" >&2
}
new_name() {
echo "$PREFIX-$OWNER-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$TTL_HOURS"
}
# vcpus SIZE: the vCPU count a Ubicloud size name carries (standard-16 -> 16).
vcpus() { [[ "$1" =~ -([0-9]+)$ ]] && echo "${BASH_REMATCH[1]}"; }
# quota_preflight LOCATION NAME SIZE: refuse (or wait UBI_QUOTA_WAIT seconds)
# before the create request when the project's running VMs plus this size
# exceed the known vCPU ceiling. The ceiling comes from UBI_QUOTA_VCPUS or the
# maximum the last refusal printed; without either the API's own refusal rules.
QUOTA_CACHE_MAX_AGE=86400
quota_preflight() {
local location=$1 name=$2 size=$3 need max used remaining step source="UBI_QUOTA_VCPUS" cache="$STATE_ROOT/quota-max"
need=$(vcpus "$size") || return 0
max=$QUOTA_VCPUS
if [ -z "$max" ] && [ -f "$cache" ]; then
# A refusal's maximum is remembered for a day: after that (a raised quota,
# another project's key) the API's own answer decides again.
if [ $(( $(date +%s) - $(mtime "$cache") )) -lt "$QUOTA_CACHE_MAX_AGE" ]; then
max=$(cat "$cache" 2>/dev/null || true); source="the last refusal, cached in $cache"
else
rm -f "$cache"
fi
fi
[[ "$max" =~ ^[0-9]+$ ]] || return 0
# The budget counts seconds spent waiting for room, not the usage calls
# between them: a slow first usage read must not shave the printed budget.
remaining=$QUOTA_WAIT
while :; do
if ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then
if [ "$remaining" -le 0 ]; then
log "quota preflight: could not read the project's vCPU usage; the API's own answer decides"
return 0
fi
log "quota: could not read the project's vCPU usage; retrying, up to ${remaining}s more"
step=$(( remaining < 30 ? remaining : 30 ))
sleep "$step"
remaining=$(( remaining - step ))
continue
fi
[ $(( used + need )) -gt "$max" ] || return 0
if [ "$remaining" -le 0 ]; then
log "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)"
cmd_usage >&2 || true
die "create refused before any request: vCPU quota exhausted (quota full); wait for running VMs to finish (UBI_QUOTA_WAIT=<seconds> waits here), run $0 reap, use a smaller size, or set UBI_QUOTA_VCPUS to the project's real ceiling"
fi
log "quota: $used of $max vCPUs in use and $need needed; waiting up to ${remaining}s more for room"
step=$(( remaining < 30 ? remaining : 30 ))
sleep "$step"
remaining=$(( remaining - step ))
done
}
valid_name() { [[ "$1" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]]; }
# check_storage GIB: Ubicloud accepts only these disk sizes; refuse before any API call.
check_storage() {
case " $ALLOWED_STORAGE " in
*" $1 "*) [ -n "$1" ] && return 0 ;;
esac
die "invalid storage '$1' GiB (allowed: $ALLOWED_STORAGE; set UBI_ALLOWED_STORAGE when Ubicloud adds sizes)"
}
UP_NAME=""
UP_READY=""
cmd_up() {
require_owner
local size=$DEFAULT_SIZE location=$DEFAULT_LOCATION name="" storage="" image=ubuntu-noble sink=${UBI_SINK:-} tags=() sink_paths=()
while [ $# -gt 0 ]; do
case $1 in
-s|--size) size=$2; shift 2 ;;
-l|--location) location=$2; shift 2 ;;
-n|--name) name=$2; shift 2 ;;
-S|--storage) check_storage "$2"; storage=$2; shift 2 ;;
-b|--image) image=$2; shift 2 ;;
--tag) [[ "$2" =~ ^[A-Za-z0-9_.-]+=[A-Za-z0-9_.:/@+-]*$ ]] || die "up: --tag $2: expected key=value (letters, digits and _.:/@+- only)"; tags+=("$2"); shift 2 ;;
--sink) sink=$2; shift 2 ;;
--sink-path) [[ "$2" != /* && "/$2/" != */../* && -n "$2" ]] || die "up: --sink-path $2: must be a path under ~ on the VM"; sink_paths+=("$2"); shift 2 ;;
--counted) COUNTED=1; shift ;;
*) die "up: unknown option $1" ;;
esac
done
[ -n "$name" ] || name=$(new_name)
valid_name "$name" || die "invalid VM name '$name' (Ubicloud allows 1-63 of a-z, 0-9 and '-', starting and ending with a letter or digit)"
[ "$COUNTED" != 1 ] || recovery_preflight
[ -z "$sink" ] || sink_check "$sink"
if [ "$REAP" != 0 ]; then
cmd_reap || log "reap failed; continuing"
fi
if [ "$GC_HOURS" -gt 0 ]; then
cmd_gc "$GC_HOURS" || log "stale-VM cleanup failed; continuing"
fi
quota_preflight "$location" "$name" "$size"
local pre probe
probe=$(mktemp)
pre=$(api "$probe" vm "$location/$name" show)
rm -f "$probe"
case $pre in
404) ;;
200) die "create refused: $location/$name already exists; nothing was created, and that VM is not this call's to adopt or destroy" ;;
*) die "create refused: could not check that $location/$name is free (HTTP $pre); nothing was created, rerun once the API answers" ;;
esac
local dir create_at
dir=$(state_dir "$name")
mkdir -p "$dir"
chmod 700 "$dir"
ssh-keygen -q -t ed25519 -N '' -C "$name" -f "$dir/key"
[ ${#tags[@]} -eq 0 ] || printf '%s\n' "${tags[@]}" >"$dir/tags"
local args=(vm "$location/$name" create -s "$size" -b "$image")
[ -z "$storage" ] || args+=(-S "$storage")
args+=("$(cat "$dir/key.pub")")
create_at=$(date +%s)
log "creating $location/$name ($size)"
# Signals are held from before the create request until its answer is
# recorded. Teardown is armed (UP_NAME) only once the request is on its way
# and disarmed when Ubicloud refuses it (HTTP 4xx): a refused create made
# nothing, and a VM that answers to the name (a 409) is not this call's.
local interrupted="" created code err refusal need used max
trap 'interrupted=1' INT TERM HUP QUIT
write_state "$name" "$location" "$size" "" unknown "$create_at"
bundle_save "$name"
record_vm "$name" "$location"
UP_NAME=$name
code=$(api "$dir/create.err" "${args[@]}")
case $code in
200) created=yes ;;
4[0-9][0-9]) created=no ;;
*) created=unknown UP_NAME="" ;;
esac
if [ "$created" = no ]; then
UP_NAME=""
err=$(cat "$dir/create.err")
rm -rf "$dir"
forget_vm "$name"
[ -z "$RECOVERY_DIR" ] || rm -f "$RECOVERY_DIR/$OWNER/$name.bundle"
else
write_state "$name" "$location" "$size" "" "$created" "$create_at"
[ "$created" = unknown ] || bundle_save "$name"
fi
trap 'exit 130' INT TERM HUP QUIT
[ -z "$interrupted" ] || exit 130
if [ "$created" = unknown ]; then
# A 5xx or no answer (000) is not a refusal: the VM may exist. Look it up
# by name until it shows or UBI_CREATE_GRACE has passed. Another caller can
# take the name between the check and the create (a lost 409), so teardown
# stays off until this call's key opens the VM.
log "create of $location/$name answered HTTP $code; looking it up by name before deciding"
local seen_code
while :; do
seen_code=$(api "$dir/lookup.out" vm "$location/$name" show)
[ "$seen_code" != 200 ] || break
[ $(( $(date +%s) - create_at )) -lt "$CREATE_GRACE" ] || break
sleep "$POLL"
done
rm -f "$dir/lookup.out"
case $seen_code in
200) created=yes ;;
404) created=no ;;
esac
write_state "$name" "$location" "$size" "" "$created" "$create_at"
case $created in
yes) bundle_save "$name"; log "create of $location/$name went through; teardown waits until this call's key opens it" ;;
no) cat "$dir/create.err" >&2; rm -rf "$dir"; forget_vm "$name"; [ -z "$RECOVERY_DIR" ] || rm -f "$RECOVERY_DIR/$OWNER/$name.bundle"; die "create failed (HTTP $code): $location/$name was still absent ${CREATE_GRACE}s after the request" ;;
unknown) die "UBI_CREATE_UNKNOWN: could not tell whether $location/$name exists (create answered HTTP $code, its lookup HTTP $seen_code); its record and recovery bundle stay, nothing destroys it, and this owner's reap destroys it once its -t<hours> TTL ends; check with: $0 list" ;;
esac
fi
if [ "$created" = no ]; then
refusal=$(sed -n 's/.*Requested vCPU count: \([0-9]*\), currently used vCPU count: \([0-9]*\), maximum allowed vCPU count: \([0-9]*\).*/\1 \2 \3/p' <<<"$err")
if [ -z "$refusal" ]; then
printf '%s\n' "$err" >&2
die "create failed (HTTP $code); nothing was created"
fi
read -r need used max <<<"$refusal"
echo "$max" >"$STATE_ROOT/quota-max"
log "quota refused $location/$name ($size): it needs $need vCPUs and the project already uses $used of $max"
log "VMs in the project by owner (the rest of the used count is other usage, such as managed GitHub runners):"
cmd_usage >&2 || true
die "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
fi
local deadline=$(( $(date +%s) + 600 )) show state ip
while :; do
show=$(cli vm "$location/$name" show 2>/dev/null || true)
state=$(sed -n 's/^state: //p' <<<"$show")
ip=$(sed -n 's/^ip4: //p' <<<"$show")
[ "$state" = running ] && [ -n "$ip" ] && break
[ "$(date +%s)" -lt "$deadline" ] || die "VM did not reach running in 10 minutes (last state: ${state:-unknown})"
sleep "$POLL"
done
write_state "$name" "$location" "$size" "$ip" yes "$create_at"
bundle_save "$name"
load "$name"
deadline=$(( $(date +%s) + SSH_WAIT ))
local proof=()
[ -n "$UP_NAME" ] || proof=(-F /dev/null)
# shellcheck disable=SC2046
until ssh ${proof[@]+"${proof[@]}"} $(ssh_opts) "ubi@$IP" true 2>/dev/null; do
if [ "$(date +%s)" -ge "$deadline" ]; then
[ -z "$UP_NAME" ] || die "SSH did not come up on $IP"
die "UBI_CREATE_UNKNOWN: $location/$name answers its name after a create that answered HTTP $code, but this call's key did not open it in ${SSH_WAIT}s, so it may be another caller's; nothing destroys it, its record and recovery bundle stay, and this owner's reap destroys it once its -t<hours> TTL ends; check with: $0 list"
fi
sleep "$POLL"
done
[ -n "$UP_NAME" ] || { UP_NAME=$name; log "this call's key opened $name; teardown armed"; }
remote "$name" 'cloud-init status --wait >/dev/null 2>&1 || true'
if [ -n "$sink" ]; then
sink_start "$name" "$sink" ${sink_paths[@]+"${sink_paths[@]}"}
bundle_save "$name"
fi
UP_READY=1
log "ready: $name ($IP)"
echo "$name"
}
# down NAME [-l LOCATION]: destroy NAME and return 0 only once it is confirmed
# gone or never existed. While this machine's create request for NAME may
# still be in flight (its `up` process is alive, or the create was sent less
# than UBI_CREATE_GRACE seconds ago with no answer), "not found" is not final.
cmd_down() {
local name=${1:-} loc="" dir
[ -n "$name" ] || die "down: missing NAME"
shift
while [ $# -gt 0 ]; do
case $1 in
-l|--location) loc=$2; shift 2 ;;
*) die "down: unknown option $1" ;;
esac
done
valid_name "$name" || die "invalid VM name '$name'"
dir=$(state_dir "$name")
local CREATED=none CREATE_AT=0 CREATE_PID="" IP="" LOCATION="" KEY=""
if [ -f "$dir/env" ]; then
load "$name"
loc=$LOCATION
[ ! -f "$dir/sink" ] || [ -z "$IP" ] || sink_flush "$name"
if [ -S "$dir/ctl" ] && [ -n "$IP" ]; then
# shellcheck disable=SC2046
ssh $(ssh_opts) -O exit "ubi@$IP" >/dev/null 2>&1 || true
fi
fi
local deadline=$(( $(date +%s) + DOWN_TIMEOUT )) seen=0 asked=0 out code now rows unlisted=0
out=$(mktemp)
while :; do
now=$(date +%s)
if [ -z "$loc" ]; then
if rows=$(vm_rows 2>/dev/null); then
loc=$(awk -v n="$name" '$2==n {print $1}' <<<"$rows")
code=404
[ -z "$loc" ] || code=$(api "$out" vm "$loc/$name" show)
else
code=unlisted
[ "$unlisted" = 1 ] || log "could not read the project's VM list; $name is not counted as gone"
unlisted=1
fi
else
code=$(api "$out" vm "$loc/$name" show)
fi
if [ "$code" = 200 ]; then
seen=1
if [ $(( now - asked )) -ge 60 ]; then
code=$(api "$out" vm "$loc/$name" destroy -f)
if [ "$code" != 200 ]; then
cat "$out" >&2
rm -f "$out"
log "FAILED to destroy $loc/$name (HTTP $code); retry: $0 down $name"
return 1
fi
asked=$now
log "destroy requested: $loc/$name"
fi
elif [ "$code" = 404 ]; then
if vm_gone_final "$seen" "$CREATED" "$CREATE_AT" "$CREATE_PID"; then
rm -f "$out"
rm -rf "$dir"
forget_vm "$name"
[ -z "$RECOVERY_DIR" ] || rm -f "$RECOVERY_DIR"/*/"$name.bundle"
if [ "$seen" = 1 ]; then log "destroyed ${loc:-?}/$name"; else log "gone: ${loc:-?}/$name (never existed or already destroyed)"; fi
return 0
fi
log "waiting for an in-flight create of $name to resolve"
fi
if [ "$now" -ge "$deadline" ]; then
rm -f "$out"
log "WARNING: could not confirm ${loc:-?}/$name is gone after ${DOWN_TIMEOUT}s; rerun: $0 down $name"
return 1
fi
sleep "$POLL"
done
}
# pack SRC: write a gzipped checkout tarball to stdout (tracked + untracked
# files that are not ignored, plus a .git snapshot; a non-git directory is
# taken whole). The live .git is never tarred: another process running git in
# the checkout (a review subagent's diff, a trial merge's worktree add) writes
# index.lock, packed-refs and objects while tar reads, and GNU tar exits 1
# on a file that changed under it (GSTA-121's PREGATE_REMOTE_INFRA). Instead
# the refs are fetched into a fresh repository through git's own transport,
# which reads a consistent view, the remote.* config is replayed so
# origin/<base> still resolves, and the index is copied (git writes it
# atomically through a rename). A linked worktree's .git is a one-line
# pointer into the main repository, useless on the VM; the same snapshot
# replaces it, on the worktree's own branch and index. Not carried: submodule
# repositories (.git/modules) and a partial clone's promisor (the fetch pulls
# every missing blob first); gstack has neither. The archive is built in two
# passes inside the snapshot directory because macOS's bsdtar stops reading
# options at the first operand and resolves -T names after a -C, so no single
# tar command line works on both it and GNU tar (#3680). COPYFILE_DISABLE=1
# keeps Apple's tar from adding ._* AppleDouble entries for files with
# extended attributes, which the VM would see as untracked files. A
# subdirectory of a work tree ships the same file set without a .git: git's
# local transport fetches only from a repository's root, and the whole
# repository's .git would show the VM every file outside the subdirectory as
# deleted (GST-935). A subdirectory git lists no files under (one the parent
# ignores) is refused rather than shipped empty: cmd_run keys the run on the
# local files and skips the tree check for a subdirectory, so an empty upload
# would pass on content that never ran.
cmd_pack() {
local src tmp gitdir branch head entry
src=$(cd "${1:-.}" && pwd)
if git -C "$src" rev-parse --is-inside-work-tree >/dev/null 2>&1 && [ -n "$(git -C "$src" rev-parse --show-prefix)" ]; then
[ -n "$(git -C "$src" ls-files -co --exclude-standard | head -n 1)" ] \
|| die "pack: git lists no files under $src (ignored by its checkout, or empty); nothing to ship"
(
cd "$src"
export COPYFILE_DISABLE=1
git ls-files -z -co --exclude-standard \
| while IFS= read -r -d '' f; do
if [ -e "$f" ] || [ -L "$f" ]; then printf '%s\0' "$f"; fi
done \
| tar --null -T - -cf - \
| gzip -c
)
elif git -C "$src" rev-parse --is-inside-work-tree >/dev/null 2>&1; then
head=$(git -C "$src" rev-parse --verify -q HEAD) || die "pack: $src has no commits; the VM needs a HEAD to fingerprint"
branch=$(git -C "$src" symbolic-ref -q --short HEAD || true)
tmp=$(mktemp -d)
# The path is baked in: a set -e failure unwinds the function before the trap runs, and a local $tmp is unset there.
trap "rm -rf $(printf %q "$tmp")" EXIT
# No template hooks from a global init.templateDir, and no auto-gc under the tar.
git -c gc.auto=0 init -q --bare --template= "$tmp/.git"
git -C "$tmp/.git" -c gc.auto=0 fetch -q --no-tags --update-shallow "$src" '+refs/*:refs/*' \
|| die "pack: could not snapshot the refs of $src"
if [ -z "$branch" ]; then
branch="detached-${head:0:12}"
git -C "$tmp/.git" fetch -q --no-tags --update-shallow "$src" "+HEAD:refs/heads/$branch"
fi
git -C "$tmp/.git" symbolic-ref HEAD "refs/heads/$branch"
git -C "$tmp/.git" config core.bare false
gitdir=$(cd "$src" && cd "$(git rev-parse --git-common-dir)" && pwd)
while IFS= read -r -d '' entry; do
# A key with no value (`[remote "x"] mirror`) is boolean true.
case $entry in *$'\n'*) ;; *) entry="$entry"$'\n'true ;; esac
git -C "$tmp/.git" config --add "${entry%%$'\n'*}" "${entry#*$'\n'}"
done < <(git -C "$gitdir" config --local --null --get-regexp '^(remote|branch)\.' 2>/dev/null || true)
# The checkout's index rides along so the VM's status matches, except a
# split index, whose shared half lives only in the original git dir. The
# index's blobs that HEAD's tree lacks (staged, intent-to-add, the stages
# of an unmerged entry) are behind no ref, so they travel as a pack of
# their own; without them the VM's write-tree (gstack-wtree) cannot read a
# staged file. cat, not cp: git renames index.lock over the index, which
# cp reports as "replaced while being copied"; one open() reads one whole index.
if [ "$(git -C "$src" config --bool core.splitIndex 2>/dev/null)" != true ]; then
cat "$(cd "$src" && cd "$(dirname "$(git rev-parse --git-path index)")" && pwd)/index" > "$tmp/.git/index"
# Read the copy, not the live index, so the pack matches it; a gitlink's
# sha (mode 160000) is a submodule commit this object store lacks.
GIT_INDEX_FILE="$tmp/.git/index" git -C "$src" ls-files -s | awk '$1 != "160000" {print $2}' | sort -u \
| comm -23 - <(git -C "$src" ls-tree -r HEAD | awk '{print $3}' | sort -u) \
| git -C "$src" pack-objects -q --stdout \
| git -C "$tmp/.git" unpack-objects -q \
|| die "pack: could not snapshot the staged blobs of $src"
fi
(
cd "$src"
export COPYFILE_DISABLE=1
git ls-files -z -co --exclude-standard \
| while IFS= read -r -d '' f; do
if [ -e "$f" ] || [ -L "$f" ]; then printf '%s\0' "$f"; fi
done \
| tar --null -T - -cf "$tmp/files.tar" \
&& tar -rf "$tmp/files.tar" -C "$tmp" .git \
&& gzip -c "$tmp/files.tar"
)
rm -rf "$tmp"; trap - EXIT
else
COPYFILE_DISABLE=1 tar -C "$src" -czf - .
fi
}
# unpack NAME DEST: extract a tarball read from stdin into DEST on the VM.
cmd_unpack() {
local qdest
qdest=$(printf %q "$2")
remote "$1" "mkdir -p $qdest && tar -xzf - -C $qdest"
}
cmd_sync() {
local name=$1 src=${2:-.} dest=${3:-}
src=$(cd "$src" && pwd)
[ -n "$dest" ] || dest="work/$(basename "$src")"
cmd_pack "$src" | cmd_unpack "$name" "$dest"
log "synced $src -> $name:$dest"
}
# pull NAME REMOTE_GLOB LOCAL_DIR: copy matching remote entries into LOCAL_DIR.
# Relative remote paths start at /home/ubi. A glob that matches nothing (for
# example an optional flake ledger) is reported and skipped, not a failure.
cmd_pull() {
local name=$1 from=$2 to=$3 dir rc=0
dir=$(printf %q "$(dirname "$from")")
remote "$name" "cd $dir 2>/dev/null || exit 3; ls -d -- $(basename "$from") >/dev/null 2>&1 || exit 3" || rc=$?
case $rc in
0) ;;
3) log "pull: nothing matches $from"; return 0 ;;
*) log "pull failed: could not check $from on $name (exit $rc)"; return 1 ;;
esac
mkdir -p "$to"
remote "$name" "cd $dir && tar -czf - $(basename "$from")" | tar -xzf - -C "$to"
}
# pass_name NAME: a variable this runner may forward to a VM: one the caller's
# environment held when the runner started. Its own credentials (UBICLOUD_*),
# its recovery secret and sink URL, and every variable it sets itself
# (RECOVERY_SECRET holds the Ubicloud key by default) never leave this machine.
pass_name() {
[[ "$1" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || die "--pass $1: not a variable name"
case $1 in UBICLOUD_*) die "--pass $1: the runner never forwards its own Ubicloud credentials to a VM" ;; esac
case $1 in RECOVERY_SECRET|UBI_RECOVERY_KEY|UBI_SINK) die "--pass $1: the runner never forwards its recovery secret or sink URL to a VM" ;; esac
case $1 in INHERITED_NAMES|INHERITED_VALUES|INHERITED_ENV|PASS_VALUE|_ubi_n) die "--pass $1: the runner never forwards its own variables" ;; esac
[[ "$INHERITED_ENV" == *" $1 "* ]] || die "--pass $1: not in this shell's environment; the runner forwards only variables you exported, never its own"
}
# inherited NAME: PASS_VALUE becomes NAME's value as the caller exported it.
inherited() {
local i
for i in "${!INHERITED_NAMES[@]}"; do
[ "${INHERITED_NAMES[i]}" = "$1" ] || continue
PASS_VALUE=${INHERITED_VALUES[i]}
return 0
done
die "--pass $1: not in this shell's environment"
}
# inside_dir DIR ROOT: DIR resolves, symlinks followed, to ROOT or below it.
# Its nearest existing ancestor is checked first, so a refused destination
# creates nothing (not even outside the root); only then is DIR created and
# checked again.
inside_dir() {
local a b near=$1
b=$(cd "$2" 2>/dev/null && pwd -P) || return 1
while [ ! -d "$near" ]; do
[ "$near" != "$(dirname "$near")" ] || return 1
near=$(dirname "$near")
done
a=$(cd "$near" 2>/dev/null && pwd -P) || return 1
[ "$a" = "$b" ] || [[ "$a" = "$b"/* ]] || return 1
mkdir -p "$1" 2>/dev/null || return 1
a=$(cd "$1" 2>/dev/null && pwd -P) || return 1
[ "$a" = "$b" ] || [[ "$a" = "$b"/* ]]
}
# PULL_ROOT: set by verify; every pull destination must resolve inside it at
# pull time (an earlier pull may have placed a symlink where a later one lands).
PULL_ROOT=""
# run_teardown RC: run's EXIT handler. Destroys RUN_VM, ignoring further
# signals, then checks the project's VM list for it and prints the leftover
# line; a VM still listed, or a list that cannot be read, fails the run.
run_teardown() {
local rc=$1 rows left others
trap "" INT TERM HUP QUIT
[ -z "$UP_NAME" ] || cmd_down "$RUN_VM" || log "WARNING: failed to destroy $RUN_VM; run: $0 down $RUN_VM"
if rows=$(vm_rows 2>/dev/null); then
left=$(awk -v n="$RUN_VM" '$2==n' <<<"$rows" | grep -c . || true)
others=$(awk -v n="$RUN_VM" -v o="$OWNER" '$3==o && $2!=n' <<<"$rows" | grep -c . || true)
else
left=unknown others=unknown
fi
if [ -z "$UP_NAME" ]; then
left=0
[ ! -f "$(state_dir "$RUN_VM")/env" ] || left=unknown
fi
leftover "left=$left owner=$OWNER vm=$RUN_VM others=$others"
if [ "$left" = 0 ]; then
[[ "$SINK_FLUSH_FAILED" != *" $RUN_VM "* ]] || [ "$rc" -ne 0 ] || exit 1
return 0
fi
[ -z "$UP_NAME" ] || log "VM cleanup not confirmed for $RUN_VM (left=$left); destroy it with: $0 down $RUN_VM"
[ "$rc" -ne 0 ] || exit 1
}
TIER_NAMES=() TIER_CMDS=() RESUME=1 RERUN_FAILED=0 FORCE_ATTACH="" RUN_ARGS=()
cmd_run() {
require_owner
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=() passes=() key_inputs=() probe=0
# The run record replays RUN_ARGS on attach. Each option's words are copied as
# it is parsed, so a --sink URL stays out of it (pass it as UBI_SINK).
local all=("$@") pos=0 opt left
RUN_ARGS=()
while [ $# -gt 0 ]; do
opt=$1 left=$#
case $1 in
-S|--storage) check_storage "$2"; up_args+=("$1" "$2"); shift 2 ;;
-s|--size|-l|--location|-n|--name|-b|--image|--tag|--sink|--sink-path) up_args+=("$1" "$2"); shift 2 ;;
--counted) COUNTED=1; shift ;;
--src) src=$2; shift 2 ;;
--dest) dest=$2; shift 2 ;;
--setup) setup=$2; shift 2 ;;
--env) envs+=("$2"); shift 2 ;;
--pass) pass_name "$2"; passes+=("$2"); shift 2 ;;
--pull) pulls+=("$2"); shift 2 ;;
--keep) keep=1; shift ;;
--key-input) key_inputs+=("$2"); shift 2 ;;
--probe) probe=1; shift ;;
--resume) RESUME=1; shift ;;
--no-resume) RESUME=0; shift ;;
--rerun-failed) RERUN_FAILED=1; shift ;;
--) shift; break ;;
*) die "run: unknown option $1" ;;
esac
[ "$opt" = --sink ] || RUN_ARGS+=("${all[@]:pos:left-$#}")
pos=$(( pos + left - $# ))
done
[ "$pos" -ge ${#all[@]} ] || RUN_ARGS+=("${all[@]:pos}")
[ $# -gt 0 ] || [ ${#TIER_NAMES[@]} -gt 0 ] || die "run: missing command after --"
[ -z "$setup" ] || [ -f "$setup" ] || die "setup script not found: $setup"
command -v git >/dev/null || die "git is required: the run key fingerprints the checkout with it"
src=$(cd "$src" && pwd)
[ -n "$dest" ] || dest="work/$(basename "$src")"
local i name="" size=$DEFAULT_SIZE location=$DEFAULT_LOCATION image=ubuntu-noble storage="" e n sink=${UBI_SINK:-} sink_paths=()
for (( i = 0; i < ${#up_args[@]}; i += 2 )); do
case ${up_args[i]} in
--sink) sink=${up_args[i+1]} ;;
--sink-path) sink_paths+=("${up_args[i+1]}") ;;
-n|--name) name=${up_args[i+1]} ;;
-s|--size) size=${up_args[i+1]} ;;
-l|--location) location=${up_args[i+1]} ;;
-b|--image) image=${up_args[i+1]} ;;
-S|--storage) storage=${up_args[i+1]} ;;
esac
done
# The run key: everything that decides what the VM runs, computed before any
# VM exists. Pass values enter only as their sha256.
if [ -z "$FORCE_ATTACH" ]; then
TREE=$(cd "$src" && bash -c "$TREE_SH") || die "could not fingerprint $src (git add -A into a temporary index failed)"
RUN_KEY=$({
echo "ubi-runner run key v1"
echo "tree $TREE"
printf 'command %q\n' "$*"
if [ -n "$setup" ]; then echo "setup $(sha256 <"$setup")"; else echo "setup none"; fi
for e in ${envs[@]+"${envs[@]}"}; do printf 'env %q\n' "$e"; done
for n in ${passes[@]+"${passes[@]}"}; do inherited "$n"; printf 'pass %s %s\n' "$n" "$(printf %s "$PASS_VALUE" | sha256)"; done
printf 'vm %q %q %q %q\n' "$size" "$location" "$image" "$storage"
echo "base $(base_sha "$src")"
echo "runner $(sha256 <"${BASH_SOURCE[0]}")"
for e in ${key_inputs[@]+"${key_inputs[@]}"}; do printf 'input %q\n' "$e"; done
for i in "${!TIER_NAMES[@]}"; do printf 'tier %q %q\n' "${TIER_NAMES[i]}" "${TIER_CMDS[i]}"; done
} | sha256)
fi
[ ${#TIER_NAMES[@]} -eq 0 ] || tier_plan
local attach=""
if [ -n "$FORCE_ATTACH" ]; then
vm_alive "$FORCE_ATTACH" || { [ $? -ne 2 ] || die "UBI_ATTACH_UNKNOWN: could not check whether $FORCE_ATTACH still exists; nothing created, rerun"; die "attach: $FORCE_ATTACH no longer exists; rerun the original command to start fresh"; }
attach=$FORCE_ATTACH
else
attach=$(find_record) || die "UBI_ATTACH_UNKNOWN: a run record for this key names a VM the Ubicloud API could not check; nothing created (a new VM would leave that one billing), rerun"
fi
if [ "$probe" = 1 ]; then
if [ -z "$attach" ]; then echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12}"
else
local busy=0
! lease_busy "$(state_dir "$attach")" || busy=1
if [ "$busy" = 0 ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
echo "UBI_ATTACH_INFO: none key=${RUN_KEY:0:12} ttl_short vm=$attach $TTL_WHY"
else
echo "UBI_ATTACH_INFO: vm=$attach created_at=$(load "$attach"; echo "$CREATE_AT") key=${RUN_KEY:0:12} busy=$busy"
fi
fi
return 0
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && { [ -n "$TIER_BLOCKED" ] || [ "$TIER_TO_RUN" -eq 0 ]; } && [ -n "$attach" ] && ! lease_busy "$(state_dir "$attach")"; then
log "verify: no tier needs $attach, which a dead runner left behind; destroying it"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ -n "$TIER_BLOCKED" ]; then
TIER_FAILED=$TIER_BLOCKED
verify_line
return "$TIER_BLOCKED_RC"
fi
if [ ${#TIER_NAMES[@]} -gt 0 ] && [ "$TIER_TO_RUN" -eq 0 ]; then
log "verify: every tier passed under this key; no VM needed"
verify_line
return 0
fi
if [ -n "$attach" ] && lease_busy "$(state_dir "$attach")"; then
die "UBI_ATTACH_BUSY: vm=$attach key=${RUN_KEY:0:12}: runner pid $(cut -d' ' -f1 <"$(state_dir "$attach")/lease") on this machine still owns it; wait for it or stop it, then rerun"
fi
if [ -n "$attach" ] && [ -z "$FORCE_ATTACH" ] && ttl_short "$attach" "$setup"; then
log "UBI_REATTACH_TTL_SHORT: vm=$attach key=${RUN_KEY:0:12} $TTL_WHY: starting a fresh VM (tier checkpoints carry over) and destroying $attach"
cmd_down "$attach" >/dev/null || log "WARNING: failed to destroy $attach; run: $0 down $attach"
attach=""
fi
if [ -n "$attach" ] && [ "$COUNTED" = 1 ]; then
recovery_preflight
local opened rbt
rbt=$(mktemp -d)
opened=0
printf %s "$RECOVERY_SECRET" | python3 -c "$RB_PY" open "$RECOVERY_DIR/$OWNER/$attach.bundle" "$rbt" "$attach" "$OWNER" 2>/dev/null && opened=1
rm -rf "$rbt"
[ "$opened" = 1 ] \
|| die "UBI_RECOVERY_REQUIRED: a counted run would reattach to $attach, which has no recovery bundle under $RECOVERY_DIR/$OWNER that opens under this secret; destroy it ($0 down $attach) and rerun"
fi
[ -z "$attach" ] || name=$attach
[ -n "$name" ] || { name=$(new_name); up_args+=(-n "$name"); }
RUN_VM=$name
# Armed before the create request: an interrupted `up` still destroys its VM.
# A second signal must not cut teardown short, so the EXIT handler ignores them.
trap 'run_teardown "$?"' EXIT
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [ -n "$attach" ]; then
UP_NAME=$name
load "$name"
log "reattached to $name created_at=$CREATE_AT key=${RUN_KEY:0:12}"
if [ -n "$sink" ]; then
sink_check "$sink"
sink_start "$name" "$sink" ${sink_paths[@]+"${sink_paths[@]}"}
bundle_save "$name"
fi
else
cmd_up "${up_args[@]}" >/dev/null
trap 'exit 130' INT TERM QUIT
trap on_hup HUP
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (this owner's next up or reap destroys it once it expires); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
else
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
fi
fi
local dir
dir=$(state_dir "$name")
write_record "$dir"
lease_take "$dir"
if [ "$keep" = 1 ]; then
# A kept VM's finished run is not reattached by the next run; a detach (HUP) keeps the record.
trap 'rm -f "$(state_dir "$RUN_VM")/run"' EXIT
leftover "left=1 owner=$OWNER vm=$name kept=1"
log "--keep: leaving $name running; destroy with: $0 down $name"
fi
local qdest rc=0 vmt
qdest=$(printf %q "$dest")
if [ ! -f "$dir/synced" ]; then
{ for e in ${envs[@]+"${envs[@]}"}; do printf 'export %s=%q\n' "${e%%=*}" "${e#*=}"; done
for n in ${passes[@]+"${passes[@]}"}; do inherited "$n"; printf 'export %s=%q\n' "$n" "$PASS_VALUE"; done; } \
| remote "$name" 'umask 077 && cat > ~/.ubirun-env'
cmd_sync "$name" "$src" "$dest"
# A subdirectory of a checkout ships without a .git, and its local tree
# (a scratch repository, blind to the parent's ignore rules) can count
# files pack leaves out, so it is compared VM to VM (tiers) but not to
# this side.
if [ -z "$(git -C "$src" rev-parse --show-prefix 2>/dev/null)" ]; then
vmt=$(vm_tree "$name" "$qdest") || vmt=unavailable
if [ "$vmt" != "$TREE" ]; then
log "UBI_VM_TREE_MISMATCH: vm=$name tree=${vmt:0:12} expected=${TREE:0:12}: the unpacked checkout is not the one this run was keyed on"
exit 3
fi
fi
[ -z "$setup" ] || remote "$name" 'cat > ~/.ubirun-setup.sh' <"$setup"
: >"$dir/synced"
fi
if [ -n "$setup" ]; then
log "running setup $(basename "$setup")"
detached "$name" setup "cd $qdest && . ~/.ubirun-env && bash -l ~/.ubirun-setup.sh" || { rc=$?; log "setup failed (exit $rc)"; exit "$rc"; }
fi
if [ ${#TIER_NAMES[@]} -gt 0 ]; then
run_tiers "$name" "$qdest" || rc=$?
else
log "running on $name: $*"
detached "$name" command "cd $qdest && . ~/.ubirun-env && bash -lc $(printf %q "$*")" || rc=$?
log "command exited $rc after $(( $(date +%s) - $(cat "$dir/phase.command.start" 2>/dev/null || date +%s) ))s"
fi
local p
for p in ${pulls[@]+"${pulls[@]}"}; do
if [ -n "$PULL_ROOT" ] && ! inside_dir "${p#*:}" "$PULL_ROOT"; then
log "pull refused: ${p#*:} resolves outside $PULL_ROOT"
[ "$rc" -ne 0 ] || rc=1
continue
fi
cmd_pull "$name" "${p%%:*}" "${p#*:}" || { log "pull failed: $p"; [ "$rc" -ne 0 ] || rc=1; }
done
[ ${#TIER_NAMES[@]} -eq 0 ] || verify_line
return "$rc"
}
# attach NAME: resume the run recorded for NAME on this machine (the same
# invocation, key and tiers), as a rerun of the same command would.
cmd_attach() {
local name=${1:-}
[ -n "$name" ] || die "attach: missing NAME"
valid_name "$name" || die "invalid VM name '$name'"
[ -f "$(state_dir "$name")/run" ] || die "attach: no run record for $name on this machine ($(state_dir "$name")/run); a machine replacement loses it"
# shellcheck disable=SC1091
. "$(state_dir "$name")/run"
FORCE_ATTACH=$name
cmd_run "${RUN_ARGS[@]}"
}
# verify [run opts] [--trust-config] [-- COMMAND]: run the repository's
# committed remote verify. .gstack/ubicloud.json under --src (default .)
# supplies setup, command, size, location, storage, image, env, pass and pull;
# flags given here and a trailing -- COMMAND override it. Without the file it
# refuses, so every lane runs the one committed definition instead of its own
# wrapper.
#
# The file is repository content and gets no reach beyond the checkout on its
# own: its setup script and pull destinations must resolve inside --src, and a
# local variable it lists under pass is forwarded only when this command line
# also names it (--pass NAME) or grants every listed name (--trust-config).
cmd_verify() {
local src=. i j g n cfg cfg_setup cfg_cmd parsed err rc=0 trust=0 cfg_args=() cfg_pass=() user_pre=() user_cmd=() granted=() missing=() cfg_tier_names=() cfg_tier_cmds=()
for (( i = 1; i <= $#; i++ )); do
[ "${!i}" = -- ] && break
if [ "${!i}" = --src ]; then j=$(( i + 1 )); src=${!j}; fi
done
cfg="$src/.gstack/ubicloud.json"
[ -f "$cfg" ] || die "verify: no $cfg; commit one ({\"setup\": \"scripts/ubicloud/bootstrap.sh\", \"command\": \"bash scripts/ubicloud/verify.sh\"}) so every lane runs the same verify instead of its own wrapper"
err=$(mktemp)
parsed=$(python3 -c '
import json, os, re, shlex, sys
cfg, src = sys.argv[1], sys.argv[2]
try:
c = json.load(open(cfg))
except ValueError:
sys.exit(3)
if not isinstance(c, dict):
sys.exit(3)
base, root = os.path.abspath(src), os.path.realpath(src)
def inside(rel, what):
if not isinstance(rel, str) or not rel or os.path.isabs(rel) or rel.startswith("~") or ".." in rel.replace("\\", "/").split("/"):
print("%s %r must be a relative path inside the checkout" % (what, rel), file=sys.stderr); sys.exit(2)
full = os.path.realpath(os.path.join(root, rel))
if full != root and not full.startswith(root + os.sep):
print("%s %r resolves outside the checkout (%s)" % (what, rel, full), file=sys.stderr); sys.exit(2)
return os.path.join(base, rel)
q = lambda s: shlex.quote(str(s))
args = []
for key, flag in (("size", "-s"), ("location", "-l"), ("storage", "-S"), ("image", "-b")):
if key in c: args += [flag, q(c[key])]
for item in c.get("env", []): args += ["--env", q(item)]
for item in c.get("pull", []):
remote, sep, local = str(item).partition(":")
if not sep or not remote:
print("pull entry %r must be REMOTE_GLOB:LOCAL_DIR" % (item,), file=sys.stderr); sys.exit(2)
args += ["--pull", q(remote + ":" + inside(local, "pull destination"))]
tiers = c.get("tiers")
names = []
if tiers is not None:
if "command" in c:
print("UBI_CONFIG_TIERS_AND_COMMAND: give either command or tiers, not both", file=sys.stderr); sys.exit(2)
if not isinstance(tiers, list) or not tiers:
print("UBI_CONFIG_TIER_NAME: tiers must be a non-empty list of {\"name\", \"command\"}", file=sys.stderr); sys.exit(2)
for t in tiers:
name = t.get("name") if isinstance(t, dict) else None
if not isinstance(name, str) or not re.fullmatch(r"[a-z0-9][a-z0-9_-]{0,31}", name):
print("UBI_CONFIG_TIER_NAME: tier name %r must match ^[a-z0-9][a-z0-9_-]{0,31}$" % (name,), file=sys.stderr); sys.exit(2)
if name in names:
print("UBI_CONFIG_TIER_NAME: tier name %r appears twice" % name, file=sys.stderr); sys.exit(2)
if not isinstance(t.get("command"), str) or not t["command"].strip():
print("UBI_CONFIG_TIER_NAME: tier %r needs a command" % name, file=sys.stderr); sys.exit(2)
names.append(name)
setup = c.get("setup", "")
print("cfg_tier_names=(" + " ".join(q(n) for n in names) + ")")
print("cfg_tier_cmds=(" + " ".join(q(t["command"]) for t in (tiers or [])) + ")")
print("cfg_args=(" + " ".join(args) + ")")
print("cfg_pass=(" + " ".join(q(x) for x in c.get("pass", [])) + ")")
print("cfg_setup=" + q(inside(setup, "setup") if setup else ""))
print("cfg_cmd=" + q(c.get("command", "")))
' "$cfg" "$src" 2>"$err") || rc=$?
case $rc in
0) ;;
2) local why; why=$(cat "$err"); rm -f "$err"; die "verify: $cfg: $why" ;;
*) rm -f "$err"; die "verify: $cfg is not valid JSON" ;;
esac
rm -f "$err"
eval "$parsed"
while [ $# -gt 0 ]; do
case $1 in
--) shift; user_cmd=("$@"); break ;;
--trust-config) trust=1; shift ;;
--keep|--resume|--no-resume|--rerun-failed|--probe|--counted) user_pre+=("$1"); shift ;;
--pass|-S|--storage|-s|--size|-l|--location|-n|--name|-b|--image|--src|--dest|--setup|--env|--pull|--tag|--sink|--sink-path)
[ $# -ge 2 ] || die "verify: $1 needs a value"
[ "$1" != --pass ] || granted+=("$2")
user_pre+=("$1" "$2"); shift 2 ;;
*) die "verify: unknown option $1" ;;
esac
done
for n in ${cfg_pass[@]+"${cfg_pass[@]}"}; do
pass_name "$n"
if [ "$trust" = 1 ]; then cfg_args+=(--pass "$n"); continue; fi
for g in ${granted[@]+"${granted[@]}"}; do [ "$g" = "$n" ] && continue 2; done
missing+=("$n")
done
[ ${#missing[@]} -eq 0 ] || die "verify: $cfg asks to forward local ${missing[*]}; grant each on this command line with --pass NAME, or every name it lists with --trust-config"
# A trailing -- COMMAND replaces the configured tiers as it replaces command.
if [ ${#user_cmd[@]} -eq 0 ] && [ ${#cfg_tier_names[@]} -gt 0 ]; then
TIER_NAMES=("${cfg_tier_names[@]}") TIER_CMDS=("${cfg_tier_cmds[@]}")
else
[ ${#user_cmd[@]} -gt 0 ] || [ -n "$cfg_cmd" ] || die "verify: $cfg names no command and none was given after --"
[ ${#user_cmd[@]} -gt 0 ] || user_cmd=("$cfg_cmd")
fi
log "verify: $cfg"
PULL_ROOT=$(cd "$src" && pwd)
cmd_run ${cfg_args[@]+"${cfg_args[@]}"} ${cfg_setup:+--setup "$cfg_setup"} --src "$src" ${user_pre[@]+"${user_pre[@]}"} -- ${user_cmd[@]+"${user_cmd[@]}"}
}
usage() {
cat <<EOF
usage: ubi-runner.sh <command> [args]
run [up opts] [--src DIR] [--dest PATH] [--setup FILE] [--env K=V]...
[--pass NAME]... [--pull REMOTE_GLOB:LOCAL_DIR]... [--keep] [--key-input S]... [--probe] -- COMMAND
up + sync + setup + command + pull + destroy; exits with COMMAND's status.
A rerun of the same command (same run key) reattaches to its live VM;
--probe prints UBI_ATTACH_INFO (and the tier plan) and exits
verify [run opts] [--trust-config] [--resume|--no-resume] [--rerun-failed] [-- COMMAND]
run the repository's committed verify (.gstack/ubicloud.json under --src);
its setup script and every pull destination stay inside --src (checked
again at pull time), and its pass names are forwarded only when granted
here (--pass NAME, or all: --trust-config). With tiers, a pass checkpoint
under the same run key is reused (resume is the default)
attach NAME resume the run recorded for NAME on this machine, as a rerun would
up [-s SIZE] [-l LOCATION] [-n NAME] [-S STORAGE_GIB] [-b IMAGE] [--tag K=V]...
[--sink https-put:URL] [--sink-path PATH]... [--counted]
create a VM and wait for SSH; prints its name
(STORAGE_GIB: one of $ALLOWED_STORAGE). --tag is kept in the VM's
state and recovery bundle; --sink starts a pusher on the VM that
uploads heartbeat.json and the PATH entries under ~ as results.tgz;
--counted (or UBI_COUNTED=1) refuses without a recovery bundle
(UBI_RECOVERY_REQUIRED, UBI_RECOVERY_UNAVAILABLE). run and verify
take the same four options
recover [--owner OLD] [--dir DIR]
rebuild this machine's state for the VMs with a bundle under
DIR/OLD (UBI_RECOVERY_DIR, this owner); prints RECOVERED, RECOVER_GONE
or RECOVER_FAILED per VM. Run it only once the creating machine is dead
ssh NAME [COMMAND] shell or login-shell command on the VM (user ubi, passwordless sudo)
sync NAME [SRC] [DEST] stream a checkout (tracked + untracked-unignored + .git)
pack [SRC] write that checkout tarball to stdout
unpack NAME DEST extract a tarball from stdin into DEST on the VM
pull NAME REMOTE_GLOB LOCAL_DIR
copy matching remote entries into LOCAL_DIR
down NAME [-l LOC] destroy the VM; returns once it is confirmed gone or never existed
list [--mine] list all VMs in the project, or only this owner's
usage VMs and vCPUs per owner across the project
owner print this caller's owner tag ($OWNER)
records VMs this machine created and has not confirmed gone ($VM_RECORD)
gc HOURS destroy this owner's VMs older than HOURS (never other owners')
reap destroy this owner's VMs whose -t<hours> TTL has ended and its
TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS (never other owners')
cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)
signals: INT and TERM tear the VM down. HUP, once a remote phase is live, detaches
(UBI_DETACHED, exit 129; the VM runs on under its TTL). SIGKILL or a machine stop leaves
the run record and the VM; rerun the same command to reattach (UBI_ATTACH_BUSY while
another runner owns it; exit 125 UBI_ATTACH_LOST when the remote session left no status).
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left=<n> line.
defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION)
env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the reap),
UBI_LEGACY_HOURS (${LEGACY_HOURS}h: reap's bound for this owner's TTL-less ubirun-* names),
UBI_IDLE_TIMEOUT (${IDLE_TIMEOUT}s of silence kills the remote command, exit 124; 0 disables),
UBI_RUN_TIMEOUT (${RUN_TIMEOUT}s wall clock; 0 unlimited), UBI_HEARTBEAT (${HEARTBEAT}s between heartbeat lines),
UBI_QUOTA_VCPUS (project vCPU ceiling for the preflight; else the last refusal's maximum),
UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing),
UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s),
UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file),
UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"),
UBI_CHECKPOINT_DIR (tier checkpoints; $CHECKPOINT_DIR), UBI_CHECKPOINT_MAX_AGE (${CHECKPOINT_MAX_AGE}s a pass is reused),
UBI_RECOVERY_DIR (recovery bundles; GSTACK_CAPY_RECOVERY_DIR on Capy), UBI_RECOVERY_KEY (bundle secret;
else UBICLOUD_API_KEY), UBI_COUNTED, UBI_SINK (a sink spec), UBI_SINK_INTERVAL (${SINK_INTERVAL}s, at least 60
for https-put), UBI_SINK_MAX_MB (${SINK_MAX_MB}: a bigger results.tgz is skipped, sink_skipped=size)
EOF
}
cmd=${1:-help}
[ $# -eq 0 ] || shift
case $cmd in
owner|pack|records|help|-h|--help) ;;
*) [ -n "${UBICLOUD_API_KEY:-}" ] || die "UBICLOUD_API_KEY is not set (create a token under your Ubicloud project's Tokens page)" ;;
esac
case $cmd in
run) cmd_run "$@" ;;
verify) cmd_verify "$@" ;;
attach) cmd_attach "$@" ;;
up) trap 'trap "" INT TERM HUP QUIT; [ -z "$UP_NAME" ] || [ -n "$UP_READY" ] || cmd_down "$UP_NAME" || log "WARNING: failed to destroy $UP_NAME; run: $0 down $UP_NAME"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "$@" ;;
ssh) name=$1; shift; load "$name"
# shellcheck disable=SC2046
if [ $# -eq 0 ]; then exec ssh -t $(ssh_opts) "ubi@$IP"; else watched "$name" ssh "bash -lc $(printf %q "$*")"; fi ;;
sync) cmd_sync "$@" ;;
pack) cmd_pack "$@" ;;
unpack) cmd_unpack "$@" ;;
pull) cmd_pull "$@" ;;
down) cmd_down "$@" ;;
recover) cmd_recover "$@" ;;
list) cmd_list "$@" ;;
usage) cmd_usage ;;
owner) echo "$OWNER" ;;
records) cmd_records ;;
gc) cmd_gc "$@" ;;
reap) cmd_reap ;;
cli) cli "$@" ;;
help|-h|--help) usage ;;
*) usage >&2; exit 2 ;;
esac
Reference in new issue View Git Blame Copy Permalink