From 67f1f82fbb4027d478f01778b3b2c9a0b266236b Mon Sep 17 00:00:00 2001 From: Garry Tan Date: Tue, 17 Nov 2026 19:11:34 +0800 Subject: [PATCH] Stop Ubicloud fan-outs overshooting the vCPU limit and owners deleting each other's VMs (GST-2852) (#4430) * Stop Ubicloud fan-outs overshooting the vCPU limit and owners deleting each other's VMs (GST-2852) The runner's quota check now reads usage and books the new VM's vCPUs under one lock per machine, so shards started together see each other instead of all passing on the same reading. A new room command prints how many VMs of a size fit now, so a fan-out can size each batch. When usage can't be read, a UBI_QUOTA_VCPUS ceiling refuses the create instead of letting it through. VM names keep the whole owner id (up to 31 characters) instead of its first 12, so owners sharing a prefix no longer reap each other's VMs (#3748). A VM named under the old cut form is still reaped by the machine that recorded it. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Add the changelog entry for the Ubicloud quota and owner-id fixes (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Review-Finding: p1/specialist-batched#1 reader=testing Review-Finding: p1/specialist-batched#2 reader=red-team Review-Finding: p1/specialist-batched#3 reader=testing Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Say which old-form VMs a long owner still reaps in the changelog entry (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Name the room command in the Ubicloud skill summary (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --- .../ubicloud-quota-booking-full-owner.md | 1 + docs/TESTING_INTERNALS.md | 15 +- docs/skills.md | 3 +- scripts/ubicloud/ubi-runner.sh | 211 +++++++++++++++--- test/helpers/ubi-runner-fake-vm.ts | 3 +- test/ubicloud-runner-verify-quota.test.ts | 73 +++++- test/ubicloud-runner.test.ts | 28 ++- 7 files changed, 284 insertions(+), 50 deletions(-) create mode 100644 changelog.d/ubicloud-quota-booking-full-owner.md diff --git a/changelog.d/ubicloud-quota-booking-full-owner.md b/changelog.d/ubicloud-quota-booking-full-owner.md new file mode 100644 index 000000000..c38a11286 --- /dev/null +++ b/changelog.d/ubicloud-quota-booking-full-owner.md @@ -0,0 +1 @@ +- **Ubicloud fan-outs can no longer overshoot the vCPU limit, and owners no longer delete each other's VMs.** `ubi-runner.sh up` now reads the project's vCPU usage and books its new VM under one lock per machine, so shards started together see each other's bookings instead of all passing on the same reading. A new `ubi-runner.sh room [-s SIZE]` command prints how many more VMs fit now, so a fan-out can size each batch to the room left. When usage can't be read and `UBI_QUOTA_VCPUS` is set, the create is refused (`UBI_QUOTA_UNREADABLE`) instead of going ahead. VM names now carry the whole owner id (up to 31 characters; longer is refused with `UBI_OWNER_TOO_LONG`) instead of its first 12, so two owners sharing a prefix no longer reap each other's expired VMs (#3748); a VM an older runner copy named under the old cut form is still reaped by the long owner on the machine whose VM record lists it, and `recover` accepts the longer owner ids. Merge checks and canaries set no `UBI_QUOTA_VCPUS` and behave as before. diff --git a/docs/TESTING_INTERNALS.md b/docs/TESTING_INTERNALS.md index 49f6c7069..806b6640e 100644 --- a/docs/TESTING_INTERNALS.md +++ b/docs/TESTING_INTERNALS.md @@ -1036,8 +1036,10 @@ namespaces Chromium's sandbox needs, and `python3-venv` is absent (required by the Python runner tests). VMs are named `ubirun---`, plus a `-t` TTL suffix -(below); `` is `UBI_OWNER` (the thread id on a Capy machine; lowercase -letters and digits, at most 12), which `up`, `run` and `verify` require. Every exit +(below); `` is `UBI_OWNER` in full (the thread id on a Capy machine; lowercase +letters and digits, at most 31, longer is refused with `UBI_OWNER_TOO_LONG`), which `up`, `run` and `verify` require. Older runner copies cut the owner to 12 +characters, so owners sharing that prefix reaped each other's VMs (#3748); a VM named under the cut form +counts as a longer owner's only when this machine's VM record lists it under that form. Every exit path destroys the client's own VM (`trap` on EXIT, INT, TERM, HUP and QUIT), and `down` fails loudly (`FAILED to destroy / (HTTP )`) unless the VM is confirmed gone. `list --mine` shows this owner's VMs, `usage` the @@ -1069,7 +1071,14 @@ before the create request when that exceeds the known ceiling (`UBI_QUOTA_VCPUS`, or the maximum the last refusal printed, remembered in the state root; `UBI_QUOTA_WAIT=` waits for room instead). When the usage read fails (`usage` exits 1 on an unreadable VM list or size and skips a VM that 404s), the preflight retries -within `UBI_QUOTA_WAIT`, then logs and leaves the decision to the API. `run`, `verify` +within `UBI_QUOTA_WAIT`, then refuses under a `UBI_QUOTA_VCPUS` ceiling +(`UBI_QUOTA_UNREADABLE`, nothing launched) and leaves the decision to the API under a +cached refusal maximum, which is the API's own ceiling. The usage read and the +booking of the new VM's vCPUs run under one lock per machine (`UBI_QUOTA_LOCK`), and a +booking counts until its create is answered, so shards a fan-out starts together +on one machine cannot all pass on the same reading; `room [-s SIZE]` prints how many +VMs fit now (`UBICLOUD_ROOM: room= ...`) for a fan-out to size each batch. Callers +on different machines still race, which the project's 48 spare vCPUs absorb. `run`, `verify` and `ssh NAME COMMAND` stream the remote output through a watchdog: a heartbeat line every `UBI_HEARTBEAT` seconds (300), and a command silent for `UBI_IDLE_TIMEOUT` seconds (1800) or older than `UBI_RUN_TIMEOUT` (0, unlimited) diff --git a/docs/skills.md b/docs/skills.md index ac86c53b4..dfc68690c 100644 --- a/docs/skills.md +++ b/docs/skills.md @@ -1064,7 +1064,8 @@ and those with no TTL once they are 4 hours old, never another owner's; every ru ends with `UBICLOUD: left=0` or fails; the quota preflight refuses before the create request when the project's running VMs plus the new size exceed the known ceiling (`UBI_QUOTA_VCPUS`, or the -maximum the last refusal printed; `UBI_QUOTA_WAIT` waits for room instead); a +maximum the last refusal printed; `UBI_QUOTA_WAIT` waits for room instead, and +`gstack-ubi room` prints how many more VMs fit so a fan-out can size each batch); a watchdog kills a remote command silent for `UBI_IDLE_TIMEOUT` seconds (1800) with exit 124 and its last 40 lines, and prints a heartbeat every `UBI_HEARTBEAT` seconds; and `gstack-ubi verify` runs the repository's diff --git a/scripts/ubicloud/ubi-runner.sh b/scripts/ubicloud/ubi-runner.sh index 5216c048b..371c1ab66 100755 --- a/scripts/ubicloud/ubi-runner.sh +++ b/scripts/ubicloud/ubi-runner.sh @@ -24,11 +24,16 @@ # from dev boxes, containers, and cloud sandboxes. # # Ownership: VMs are named ubirun----t. is UBI_OWNER -# (lowercased, letters and digits only, at most 12 characters, starting with a -# letter). `up`, `run` and `verify` refuse without it (UBI_OWNER_REQUIRED), so -# every VM names whoever can be asked about it; read-only commands fall back to a -# per-machine id kept in $STATE_ROOT/owner-id. Ubicloud names allow 63 characters -# of [a-z0-9-], so a full name stays under 45. +# in full (lowercased, letters and digits only, starting with a letter), and an +# owner longer than 31 characters is refused (UBI_OWNER_TOO_LONG): Ubicloud names +# allow 63 characters of [a-z0-9-], and the rest of the name takes 32. `up`, `run` +# and `verify` refuse without it (UBI_OWNER_REQUIRED), so every VM names whoever +# can be asked about it; read-only commands fall back to a per-machine id kept in +# $STATE_ROOT/owner-id. Runner copies before GST-2852 cut the owner to its first +# 12 characters, so two owners sharing that prefix reaped each other's VMs +# (#3748). A VM named under that cut form of a longer owner still counts as this +# owner's only when this machine's VM record lists it under the cut form in a +# row an older copy wrote (five columns; this copy adds a sixth, full). # # Teardown: `up` records each VM's state (location, create status) before it # sends the create request, and `down` destroys and polls until the VM is @@ -77,7 +82,15 @@ # Quota preflight: when the project's vCPU ceiling is known (UBI_QUOTA_VCPUS, or # the maximum a refusal printed, cached in $STATE_ROOT/quota-max), `up` adds the # size's vCPUs to the project's running VMs and refuses before any create when -# that exceeds it; UBI_QUOTA_WAIT= waits for room instead. +# that exceeds it; UBI_QUOTA_WAIT= waits for room instead. The read and +# the booking happen under one lock per machine (UBI_QUOTA_LOCK, beside the VM +# record), and each booking counts until its create is answered, so shards a +# fan-out starts together on one machine share the room instead of all passing +# on the same reading. `room [-s SIZE]` prints how many VMs of SIZE fit now, for +# a fan-out to size each batch. When usage can't be read, a UBI_QUOTA_VCPUS +# ceiling refuses the create (UBI_QUOTA_UNREADABLE); a cached refusal maximum is +# the API's own ceiling, so there the API's answer decides. Merge checks and +# canaries set no UBI_QUOTA_VCPUS, so the eval limit never holds them back. # # Watchdog: `run`, `verify` and `ssh NAME COMMAND` stream the remote output # through a watchdog that logs a heartbeat every UBI_HEARTBEAT seconds (300) @@ -171,6 +184,8 @@ PREFIX="ubirun" if [ -n "${UBI_VM_RECORD:-}" ]; then VM_RECORD="$UBI_VM_RECORD" elif [ -d "$HOME/.capy/work" ]; then VM_RECORD="$HOME/.capy/work/ubi-runner/vms.tsv" else VM_RECORD="$STATE_ROOT/vms.tsv"; fi +QUOTA_LOCK="${UBI_QUOTA_LOCK:-$(dirname "$VM_RECORD")/quota.lock}" +QUOTA_LEDGER="$QUOTA_LOCK.booked" if [ -n "${UBI_CHECKPOINT_DIR:-}" ]; then CHECKPOINT_DIR="$UBI_CHECKPOINT_DIR" elif [ -d "$HOME/.capy/work" ]; then CHECKPOINT_DIR="$HOME/.capy/work/ubi-runner/checkpoints" else CHECKPOINT_DIR="$STATE_ROOT/checkpoints"; fi @@ -214,9 +229,14 @@ owner_id() { raw=$(printf %s "$raw" | tr '[:upper:]' '[:lower:]' | tr -cd 'a-z0-9') [ -n "$raw" ] || die "UBI_OWNER must contain letters or digits" [[ "$raw" =~ ^[a-z] ]] || raw="u$raw" - echo "${raw:0:12}" + [ ${#raw} -le 31 ] || die "UBI_OWNER is ${#raw} characters once lowercased to letters and digits ($raw), and a VM name has room for 31; use a shorter owner such as your thread code (UBI_OWNER_TOO_LONG)" + echo "$raw" } OWNER=$(owner_id) +# Runner copies before GST-2852 cut the owner to 12 characters; vm_rows reads +# that cut form as this owner only for VMs an older copy recorded under it here. +LEGACY_OWNER="" +[ ${#OWNER} -le 12 ] || LEGACY_OWNER=${OWNER:0:12} # require_owner: up, run and verify create VMs, and a VM must name its owner # and carry a TTL. Only creates check UBI_TTL_HOURS, so an old value never @@ -802,8 +822,12 @@ vm_rows() { while read -r loc name; do [ -n "$name" ] || continue owner=. created="" expires=- - if [[ "$name" =~ ^${PREFIX}-([a-z][a-z0-9]{0,11})-([0-9]{10})- ]]; then + if [[ "$name" =~ ^${PREFIX}-([a-z][a-z0-9]{0,30})-([0-9]{10})- ]]; then owner=${BASH_REMATCH[1]} created=${BASH_REMATCH[2]} + if [ -n "$LEGACY_OWNER" ] && [ "$owner" = "$LEGACY_OWNER" ] && [ -f "$VM_RECORD" ] \ + && awk -F'\t' -v n="$name" -v o="$LEGACY_OWNER" '$1 == n && $3 == o && NF == 5 { f = 1 } END { exit !f }' "$VM_RECORD"; then + owner=$OWNER + fi elif [[ "$name" =~ ^${PREFIX}-([0-9]{10})- ]]; then owner=- created=${BASH_REMATCH[1]} fi @@ -916,9 +940,11 @@ write_state() { } # record_vm NAME LOCATION [OWNER]: append to the scratch VM-id file (record-before-create). +# The sixth column, full, marks a row whose owner was never cut to 12 characters, +# so vm_rows never reads it as a longer owner's cut form; older copies write five. record_vm() { mkdir -p "$(dirname "$VM_RECORD")" 2>/dev/null || return 0 - printf '%s\t%s\t%s\t%s\t%s\n' "$1" "$2" "${3:-$OWNER}" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$$" >>"$VM_RECORD" + printf '%s\t%s\t%s\t%s\t%s\tfull\n' "$1" "$2" "${3:-$OWNER}" "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$$" >>"$VM_RECORD" } # forget_vm NAME: drop the VM from the scratch file once it is confirmed gone. @@ -1047,7 +1073,7 @@ cmd_recover() { *) die "recover: unknown option $1" ;; esac done - [[ "$from" =~ ^[a-z][a-z0-9]{0,11}$ ]] || die "recover: --owner must be an owner tag as the owner command prints it" + [[ "$from" =~ ^[a-z][a-z0-9]{0,30}$ ]] || die "recover: --owner must be an owner tag as the owner command prints it" [ -n "$dir" ] || die "UBI_RECOVERY_REQUIRED: recover reads bundles from UBI_RECOVERY_DIR (or --dir DIR), and neither is set" command -v openssl >/dev/null || die "UBI_RECOVERY_UNAVAILABLE: openssl is required to open recovery bundles" mkdir -p "$STATE_ROOT" @@ -1222,53 +1248,167 @@ new_name() { # vcpus SIZE: the vCPU count a Ubicloud size name carries (standard-16 -> 16). vcpus() { [[ "$1" =~ -([0-9]+)$ ]] && echo "${BASH_REMATCH[1]}"; } -# quota_preflight LOCATION NAME SIZE: refuse (or wait UBI_QUOTA_WAIT seconds) -# before the create request when the project's running VMs plus this size -# exceed the known vCPU ceiling. The ceiling comes from UBI_QUOTA_VCPUS or the -# maximum the last refusal printed; without either the API's own refusal rules. +# quota_ceiling: set QUOTA_MAX and QUOTA_SOURCE to the known vCPU ceiling, from +# UBI_QUOTA_VCPUS or the maximum the last refusal printed; QUOTA_MAX stays empty +# when neither is known, and the API's own refusal rules then. QUOTA_CACHE_MAX_AGE=86400 -quota_preflight() { - local location=$1 name=$2 size=$3 need max used remaining step source="UBI_QUOTA_VCPUS" cache="$STATE_ROOT/quota-max" - need=$(vcpus "$size") || return 0 - max=$QUOTA_VCPUS - if [ -z "$max" ] && [ -f "$cache" ]; then +QUOTA_MAX="" QUOTA_SOURCE="" QUOTA_BOOKED="" +quota_ceiling() { + local cache="$STATE_ROOT/quota-max" + QUOTA_MAX=$QUOTA_VCPUS QUOTA_SOURCE=UBI_QUOTA_VCPUS + if [ -z "$QUOTA_MAX" ] && [ -f "$cache" ]; then # A refusal's maximum is remembered for a day: after that (a raised quota, # another project's key) the API's own answer decides again. if [ $(( $(date +%s) - $(mtime "$cache") )) -lt "$QUOTA_CACHE_MAX_AGE" ]; then - max=$(cat "$cache" 2>/dev/null || true); source="the last refusal, cached in $cache" + QUOTA_MAX=$(cat "$cache" 2>/dev/null || true) QUOTA_SOURCE="the last refusal, cached in $cache" else rm -f "$cache" fi fi - [[ "$max" =~ ^[0-9]+$ ]] || return 0 + [[ "$QUOTA_MAX" =~ ^[0-9]+$ ]] || QUOTA_MAX="" +} + +# quota_lock SECONDS: take this machine's quota lock on fd 9, waiting up to +# SECONDS. It is flock(2) through python3, which the runner needs anyway and +# which macOS has where flock(1) is missing; the lock belongs to the open file, +# so it ends with this process however the process exits. quota_unlock drops it. +quota_lock() { + mkdir -p "$(dirname "$QUOTA_LOCK")" 2>/dev/null && exec 9>>"$QUOTA_LOCK" || return 1 + python3 -c 'import fcntl, sys, time +end = time.time() + float(sys.argv[1]) +while True: + try: + fcntl.flock(9, fcntl.LOCK_EX | fcntl.LOCK_NB) + break + except BlockingIOError: + if time.time() >= end: + sys.exit(1) + time.sleep(0.2)' "$1" 2>/dev/null || { exec 9>&-; return 1; } +} +quota_unlock() { exec 9>&-; } + +# quota_booked: under the lock, print the vCPUs this machine's creates have +# booked and not yet had answered. A booking whose process is gone, or older +# than any create can take (two API timeouts and the create grace), is dropped. +quota_booked() { + local name n pid at now total=0 keep + [ -s "$QUOTA_LEDGER" ] || { echo 0; return 0; } + now=$(date +%s) keep="$QUOTA_LEDGER.$$" + : >"$keep" + while read -r name n pid at; do + [[ "$n" =~ ^[0-9]+$ && "$pid" =~ ^[0-9]+$ && "$at" =~ ^[0-9]+$ ]] || continue + kill -0 "$pid" 2>/dev/null && [ $(( now - at )) -lt $(( 2 * API_TIMEOUT + CREATE_GRACE + 60 )) ] || continue + echo "$name $n $pid $at" >>"$keep" + total=$(( total + n )) + done <"$QUOTA_LEDGER" + mv "$keep" "$QUOTA_LEDGER" + echo "$total" +} + +# quota_release: drop this call's booking once its create is answered (the VM +# is then in the project's list, or was never made). A booking the lock can't +# reach here still ends with its process or its age. +quota_release() { + [ -n "$QUOTA_BOOKED" ] || return 0 + if quota_lock 30; then + awk -v n="$QUOTA_BOOKED" -v p="$$" '!($1 == n && $3 == p)' "$QUOTA_LEDGER" >"$QUOTA_LEDGER.$$" 2>/dev/null \ + && mv "$QUOTA_LEDGER.$$" "$QUOTA_LEDGER" || rm -f "$QUOTA_LEDGER.$$" + quota_unlock + fi + QUOTA_BOOKED="" +} + +# quota_preflight LOCATION NAME SIZE: under this machine's quota lock, read the +# project's usage and this machine's open bookings, and book SIZE's vCPUs when +# they fit the known ceiling; otherwise refuse (or wait UBI_QUOTA_WAIT seconds) +# before the create request. Shards started together on one machine therefore +# see each other's bookings instead of all passing on the same reading. When +# usage can't be read, a UBI_QUOTA_VCPUS ceiling refuses (it is ours, and the API +# would let the create through); a cached refusal maximum is the API's own, so +# there the API's answer decides. +quota_preflight() { + local location=$1 name=$2 size=$3 need used held remaining step why booked_note + need=$(vcpus "$size") || return 0 + quota_ceiling + [ -n "$QUOTA_MAX" ] || return 0 # The budget counts seconds spent waiting for room, not the usage calls # between them: a slow first usage read must not shave the printed budget. remaining=$QUOTA_WAIT while :; do - if ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then - if [ "$remaining" -le 0 ]; then - log "quota preflight: could not read the project's vCPU usage; the API's own answer decides" + why="" held=0 + if ! quota_lock 300; then + why="could not take this machine's quota lock ($QUOTA_LOCK) within 300s" + elif ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then + why="could not read the project's vCPU usage" + else + held=$(quota_booked) + if [ $(( used + held + need )) -le "$QUOTA_MAX" ]; then + echo "$name $need $$ $(date +%s)" >>"$QUOTA_LEDGER" + QUOTA_BOOKED=$name + quota_unlock return 0 fi - log "quota: could not read the project's vCPU usage; retrying, up to ${remaining}s more" - step=$(( remaining < 30 ? remaining : 30 )) - sleep "$step" - remaining=$(( remaining - step )) - continue fi - [ $(( used + need )) -gt "$max" ] || return 0 + quota_unlock + booked_note="" + [ "$held" -eq 0 ] || booked_note=", with $held more booked by creates in flight on this machine" if [ "$remaining" -le 0 ]; then - log "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $max (ceiling from $source; other usage such as managed runners counts on top)" + if [ -n "$why" ]; then + if [ "$QUOTA_SOURCE" != UBI_QUOTA_VCPUS ]; then + log "quota preflight: $why; the API's own answer decides" + return 0 + fi + die "create refused before any request: $why, so the UBI_QUOTA_VCPUS=$QUOTA_MAX ceiling can't be checked and nothing was launched (UBI_QUOTA_UNREADABLE); rerun once the API answers, or set UBI_QUOTA_WAIT= to retry for that long" + fi + log "quota preflight refused $location/$name ($size): it needs $need vCPUs and the project's VMs already use $used of $QUOTA_MAX$booked_note (ceiling from $QUOTA_SOURCE; other usage such as managed runners counts on top)" cmd_usage >&2 || true die "create refused before any request: vCPU quota exhausted (quota full); wait for running VMs to finish (UBI_QUOTA_WAIT= waits here), run $0 reap, use a smaller size, or set UBI_QUOTA_VCPUS to the project's real ceiling" fi - log "quota: $used of $max vCPUs in use and $need needed; waiting up to ${remaining}s more for room" + if [ -n "$why" ]; then + log "quota: $why; retrying, up to ${remaining}s more" + else + log "quota: $used of $QUOTA_MAX vCPUs in use$booked_note and $need needed; waiting up to ${remaining}s more for room" + fi step=$(( remaining < 30 ? remaining : 30 )) sleep "$step" remaining=$(( remaining - step )) done } +# room [-s SIZE]: how many more VMs of SIZE fit under the known ceiling now, from +# one usage read plus this machine's open bookings, so a fan-out can size each +# batch to the room left. It books nothing; each up still books its own VM. +cmd_room() { + local size=$DEFAULT_SIZE need used held=0 why="" room + while [ $# -gt 0 ]; do + case $1 in + -s|--size) size=$2; shift 2 ;; + *) die "room: unknown option $1" ;; + esac + done + need=$(vcpus "$size") || die "room: cannot tell how many vCPUs size $size has" + quota_ceiling + if [ -z "$QUOTA_MAX" ]; then + echo "UBICLOUD_ROOM: room=unknown size=$size reason=no-ceiling (set UBI_QUOTA_VCPUS)" + return 1 + fi + if ! quota_lock 300; then + why=lock-busy + elif ! used=$(cmd_usage 2>/dev/null | awk '$1=="total" {print $3}') || ! [[ "$used" =~ ^[0-9]+$ ]]; then + why=usage-unreadable + else + held=$(quota_booked) + fi + quota_unlock + if [ -n "$why" ]; then + echo "UBICLOUD_ROOM: room=unknown size=$size reason=$why max=$QUOTA_MAX" + return 1 + fi + room=$(( (QUOTA_MAX - used - held) / need )) + [ "$room" -gt 0 ] || room=0 + echo "UBICLOUD_ROOM: room=$room size=$size used=$used booked=$held max=$QUOTA_MAX" +} + valid_name() { [[ "$1" =~ ^[a-z0-9]([a-z0-9-]{0,61}[a-z0-9])?$ ]]; } # check_storage GIB: Ubicloud accepts only these disk sizes; refuse before any API call. @@ -1358,6 +1498,7 @@ cmd_up() { write_state "$name" "$location" "$size" "" "$created" "$create_at" [ "$created" = unknown ] || bundle_save "$name" fi + [ "$created" = unknown ] || quota_release trap 'exit 130' INT TERM HUP QUIT [ -z "$interrupted" ] || exit 130 if [ "$created" = unknown ]; then @@ -1378,6 +1519,7 @@ cmd_up() { 200) created=yes ;; 404) created=no ;; esac + quota_release write_state "$name" "$location" "$size" "" "$created" "$create_at" case $created in yes) bundle_save "$name"; log "create of $location/$name went through; teardown waits until this call's key opens it" ;; @@ -2081,6 +2223,9 @@ usage: ubi-runner.sh [args] down NAME [-l LOC] destroy the VM; returns once it is confirmed gone or never existed list [--mine] list all VMs in the project, or only this owner's usage VMs and vCPUs per owner across the project + room [-s SIZE] how many more VMs of SIZE fit under the quota ceiling now + (UBICLOUD_ROOM: room= ...; exit 1 with room=unknown), so + a fan-out can size each batch to the room left owner print this caller's owner tag ($OWNER) records VMs this machine created and has not confirmed gone ($VM_RECORD) gc HOURS destroy this owner's VMs older than HOURS (never other owners') @@ -2102,6 +2247,7 @@ env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the UBI_RUN_TIMEOUT (${RUN_TIMEOUT}s wall clock; 0 unlimited), UBI_HEARTBEAT (${HEARTBEAT}s between heartbeat lines), UBI_QUOTA_VCPUS (project vCPU ceiling for the preflight; else the last refusal's maximum), UBI_QUOTA_WAIT (${QUOTA_WAIT}s to wait for quota room instead of refusing), + UBI_QUOTA_LOCK (this machine's quota lock; its bookings sit beside it in .booked), UBI_GC_HOURS (unset/0: no sweep on up), UBI_CREATE_GRACE (${CREATE_GRACE}s), UBI_DOWN_TIMEOUT (${DOWN_TIMEOUT}s), UBI_API_TIMEOUT (${API_TIMEOUT}s), UBI_VM_RECORD (scratch VM-id file), UBI_ALLOWED_STORAGE (the -S sizes Ubicloud accepts; default "320 640"), @@ -2136,6 +2282,7 @@ case $cmd in recover) cmd_recover "$@" ;; list) cmd_list "$@" ;; usage) cmd_usage ;; + room) cmd_room "$@" ;; owner) echo "$OWNER" ;; records) cmd_records ;; gc) cmd_gc "$@" ;; diff --git a/test/helpers/ubi-runner-fake-vm.ts b/test/helpers/ubi-runner-fake-vm.ts index 7954e2c53..9ea5f7bc8 100644 --- a/test/helpers/ubi-runner-fake-vm.ts +++ b/test/helpers/ubi-runner-fake-vm.ts @@ -19,6 +19,7 @@ export const read = (rel: string) => readFileSync(join(ROOT, rel), 'utf8'); * with listed.create-lands (a count N) the VM still lands, listed from the Nth * show on (0: listed at once, as a holder that took the name first), and with listed.show-fails-after-create every later show answers 503; * listed.list-fails, listed.show-fails and listed.destroy-fails answer 503 or 500; listed.list-body is the list's 200 body. + * FAKE_CREATE_DELAY (seconds) holds every create's answer that long, so concurrent callers overlap. * With listed.holder-key (a public key line), the VM at the name is another caller's: the fake ssh * refuses every key but that one with exit 255, as sshd answers a key the VM was not created with; with listed.agent-holds-holder-key * the controller's ssh agent holds the holder's key, which ssh offers unless IdentitiesOnly=yes; with listed.config-holds-holder-key @@ -47,7 +48,7 @@ export function fakeVm(opts: { listed?: string; refuse?: string } = {}) { 'loc=$(printf \'%s\' "$body" | sed -n \'s/.*"vm", *"\\([^/"]*\\)\\/[^"]*".*/\\1/p\')\n' + 'name=$(printf \'%s\' "$body" | sed -n \'s/.*"vm", *"[^/"]*\\/\\([^"]*\\)".*/\\1/p\')\n' + 'case "$body" in\n' + - ' *\'"create"\'*) if [ -f "$listed.create-status" ]; then : > "$out"; if [ -f "$listed.create-lands" ]; then if [ "$(cat "$listed.create-lands")" = 0 ]; then echo "$loc $name" >> "$listed"; else echo "$loc $name" > "$listed.pending"; fi; fi; [ ! -f "$listed.show-fails-after-create" ] || : > "$listed.show-fails"; cat "$listed.create-status"; elif [ -n ' + JSON.stringify(opts.refuse ?? '') + ' ]; then printf \'%s\' ' + JSON.stringify(opts.refuse ?? '') + ' > "$out"; printf 400; else echo "$loc $name" >> "$listed"; : > "$out"; printf 200; fi ;;\n' + + ' *\'"create"\'*) [ -z "${FAKE_CREATE_DELAY:-}" ] || sleep "$FAKE_CREATE_DELAY"; if [ -f "$listed.create-status" ]; then : > "$out"; if [ -f "$listed.create-lands" ]; then if [ "$(cat "$listed.create-lands")" = 0 ]; then echo "$loc $name" >> "$listed"; else echo "$loc $name" > "$listed.pending"; fi; fi; [ ! -f "$listed.show-fails-after-create" ] || : > "$listed.show-fails"; cat "$listed.create-status"; elif [ -n ' + JSON.stringify(opts.refuse ?? '') + ' ]; then printf \'%s\' ' + JSON.stringify(opts.refuse ?? '') + ' > "$out"; printf 400; else echo "$loc $name" >> "$listed"; : > "$out"; printf 200; fi ;;\n' + ' *destroy*) : > "$out"; if [ -f "$listed.destroy-fails" ]; then printf 500; else grep -v " $name$" "$listed" > "$listed.tmp"; mv "$listed.tmp" "$listed"; printf 200; fi ;;\n' + ' *\'"list"\'*) if [ -f "$listed.list-fails" ]; then echo down > "$out"; printf 503; elif [ -f "$listed.list-body" ]; then cat "$listed.list-body" > "$out"; printf 200; else cat "$listed" "$listed.ghost" > "$out" 2>/dev/null; printf 200; fi ;;\n' + ' *\'"show"\'*) [ -z "${FAKE_SHOW_DELAY:-}" ] || sleep "$FAKE_SHOW_DELAY"; if [ -f "$listed.pending" ]; then n=$(( $(cat "$listed.shows" 2>/dev/null || echo 0) + 1 )); echo "$n" > "$listed.shows"; if [ "$n" -ge "$(cat "$listed.create-lands")" ]; then cat "$listed.pending" >> "$listed"; rm -f "$listed.pending"; fi; fi; if [ -f "$listed.show-fails" ]; then echo down > "$out"; printf 503; elif grep -q " $name$" "$listed"; then printf \'state: running\\nsize: standard-16\\nip4: 192.0.2.1\\n\' > "$out"; printf 200; else : > "$out"; printf 404; fi ;;\n' + diff --git a/test/ubicloud-runner-verify-quota.test.ts b/test/ubicloud-runner-verify-quota.test.ts index a7cbeea6b..ce0361f2b 100644 --- a/test/ubicloud-runner-verify-quota.test.ts +++ b/test/ubicloud-runner-verify-quota.test.ts @@ -5,7 +5,7 @@ import { describe, expect, test } from 'bun:test'; import { chmodSync, existsSync, lstatSync, mkdirSync, readFileSync, readdirSync, rmSync, symlinkSync, utimesSync, writeFileSync } from 'node:fs'; import { join } from 'node:path'; -import { read, fakeVm, src, run } from './helpers/ubi-runner-fake-vm'; +import { read, fakeVm, src, run, creates } from './helpers/ubi-runner-fake-vm'; describe('ubi-runner run, verify and quota against a fake API and a fake ssh', () => { test('verify forwards a configured pass name only when the command line grants it, and never UBICLOUD_*', () => { @@ -140,8 +140,8 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( } finally { rmSync(f.root, { recursive: true, force: true }); } }, 90_000); - // Value: protects=usage never reports a partial or empty count as the project's real usage, and the quota preflight says when it cannot read usage instead of passing silently; fails_when=a failed vm list or vm show prints a total again (the empty-project fix's `|| true` did that) or the preflight drops its log line; why_new=only successful reads were tested, and #3679 showed an unreadable usage switching the preflight off without a word; seam=none; no_claim=the real API's failure bodies - test('usage exits 1 without a total when the VM list or a size read fails, skips a VM destroyed since the list, and the quota preflight logs an unreadable usage and leaves the call to the API', () => { + // Value: protects=usage never reports a partial or empty count as the project's real usage, and an unreadable usage launches nothing under a UBI_QUOTA_VCPUS ceiling, while callers held to the API's own ceiling (a cached refusal maximum) or to none (merge checks, canaries) still create; fails_when=a failed vm list or vm show prints a total again (the empty-project fix's `|| true` did that), the preflight lets a create through when it cannot check UBI_QUOTA_VCPUS (GST-2852), or the refusal spreads to callers with no eval ceiling; why_new=the preflight failed open under UBI_QUOTA_WAIT=0 and this case pinned that; seam=none; no_claim=the real API's failure bodies + test('usage exits 1 without a total when the VM list or a size read fails, skips a VM destroyed since the list; an unreadable usage refuses a UBI_QUOTA_VCPUS create and leaves cached-ceiling and ceiling-free creates to the API', () => { const f = fakeVm({ listed: 'eu-central-h1 ubirun-peer-9999999999-cccc\n' }); const listed = join(f.root, 'listed'); try { @@ -150,16 +150,31 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( expect(noList.exitCode).toBe(1); expect(noList.stdout.toString()).not.toContain('total'); expect(noList.stderr.toString()).toContain("usage: could not list the project's VMs"); - const up = run(f, ['up'], { UBI_REAP: '0', UBI_QUOTA_VCPUS: '16' }); - expect(up.stderr.toString()).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides"); - expect(f.calls()).toContain('"create"'); - // With a wait budget, an unreadable usage is retried until the budget is spent, as a full project is. - const waited = run(f, ['up'], { UBI_REAP: '0', UBI_QUOTA_VCPUS: '16', UBI_QUOTA_WAIT: '2' }); + const up = run(f, ['up'], { UBI_REAP: '0', UBI_QUOTA_VCPUS: '464' }); + expect(up.exitCode).toBe(1); + expect(up.stderr.toString()).toContain("create refused before any request: could not read the project's vCPU usage, so the UBI_QUOTA_VCPUS=464 ceiling can't be checked and nothing was launched (UBI_QUOTA_UNREADABLE)"); + expect(f.calls()).not.toContain('"create"'); + expect(existsSync(f.record)).toBe(false); + // With a wait budget, an unreadable usage is retried until the budget is spent, then refused the same way. + const waited = run(f, ['up'], { UBI_REAP: '0', UBI_QUOTA_VCPUS: '464', UBI_QUOTA_WAIT: '2' }); + expect(waited.exitCode).toBe(1); expect(waited.stderr.toString()).toContain("quota: could not read the project's vCPU usage; retrying, up to 2s more"); - expect(waited.stderr.toString()).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides"); - const waitedName = waited.stdout.toString().trim(); + expect(waited.stderr.toString()).toContain('(UBI_QUOTA_UNREADABLE)'); + expect(f.calls()).not.toContain('"create"'); + const room = run(f, ['room'], { UBI_QUOTA_VCPUS: '464' }); + expect(room.exitCode).toBe(1); + expect(room.stdout.toString()).toBe('UBICLOUD_ROOM: room=unknown size=standard-16 reason=usage-unreadable max=464\n'); + // A cached refusal maximum is the API's own ceiling, so the API's answer decides; with no ceiling there is no preflight. + writeFileSync(join(f.state, 'quota-max'), '512\n'); + const cached = run(f, ['up'], { UBI_REAP: '0' }); + expect(cached.exitCode, cached.stderr.toString()).toBe(0); + expect(cached.stderr.toString()).toContain("quota preflight: could not read the project's vCPU usage; the API's own answer decides"); + rmSync(join(f.state, 'quota-max')); + const free = run(f, ['up'], { UBI_REAP: '0' }); + expect(free.exitCode, free.stderr.toString()).toBe(0); + expect(free.stderr.toString()).not.toContain('quota'); rmSync(`${listed}.list-fails`); - for (const name of [up.stdout.toString().trim(), waitedName]) { + for (const name of [cached.stdout.toString().trim(), free.stdout.toString().trim()]) { expect(name).toMatch(/^ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3$/); expect(run(f, ['down', name]).exitCode).toBe(0); } @@ -209,6 +224,42 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', ( } finally { rmSync(f.root, { recursive: true, force: true }); } }, 90_000); + // Value: protects=two shards a fan-out starts together cannot both pass the preflight on the same usage reading and take the project past UBI_QUOTA_VCPUS; room reports the VMs that fit for a fan-out to size its batch; fails_when=the usage read and the create are no longer serialised per machine, a booking stops counting before its create is answered, or a booking outlives its create and keeps counting; why_new=HARN-1 found the check-then-create race at 448 of 464 (GST-2852) and no case ran two callers at once; seam=FAKE_CREATE_DELAY holds the fake create's answer so both callers read usage before either VM lands; no_claim=callers on different machines (the lock is per machine) and the real API's create latency + test('two ups started together at 448 of 464 vCPUs create one VM: the second sees the first one\'s booking and is refused; room reports what fits', async () => { + const listed = Array.from({ length: 28 }, (_, i) => `eu-central-h1 ubirun-peer-9999999999-${String(i).padStart(4, '0')}\n`).join(''); + const f = fakeVm({ listed }); + try { + // A booking whose process is gone, and one older than any create can take, no longer count. + writeFileSync(join(f.root, 'quota.lock.booked'), `ubirun-dead 16 2147483646 ${Math.floor(Date.now() / 1000)}\nubirun-old 16 ${process.pid} 1000000000\n`); + expect(run(f, ['room'], { UBI_QUOTA_VCPUS: '464' }).stdout.toString()).toBe('UBICLOUD_ROOM: room=1 size=standard-16 used=448 booked=0 max=464\n'); + expect(readFileSync(join(f.root, 'quota.lock.booked'), 'utf8')).toBe(''); + const extra = { UBI_REAP: '0', UBI_QUOTA_VCPUS: '464', FAKE_CREATE_DELAY: '3' }; + const shards = [f.spawn(['up'], extra), f.spawn(['up'], extra)]; + const results = await Promise.all(shards.map(async (p) => ({ code: await p.exited, out: await new Response(p.stdout).text(), err: await new Response(p.stderr).text() }))); + const made = results.filter(r => r.code === 0), refused = results.filter(r => r.code !== 0); + expect(made, results.map(r => r.err).join('\n---\n')).toHaveLength(1); + expect(refused).toHaveLength(1); + expect(refused[0].err).toMatch(/quota preflight refused .*: it needs 16 vCPUs and the project's VMs already use (448 of 464, with 16 more booked by creates in flight on this machine|464 of 464) \(ceiling from UBI_QUOTA_VCPUS/); + expect(creates(f)).toBe(1); + expect(readFileSync(join(f.root, 'quota.lock.booked'), 'utf8')).toBe(''); + expect(run(f, ['room'], { UBI_QUOTA_VCPUS: '464' }).stdout.toString()).toBe('UBICLOUD_ROOM: room=0 size=standard-16 used=464 booked=0 max=464\n'); + expect(run(f, ['down', made[0].out.trim()]).exitCode).toBe(0); + expect(run(f, ['room'], { UBI_QUOTA_VCPUS: '464' }).stdout.toString()).toBe('UBICLOUD_ROOM: room=1 size=standard-16 used=448 booked=0 max=464\n'); + // A create answered 503 that the lookup then finds still drops its booking. + writeFileSync(join(f.root, 'listed.create-status'), '503'); + writeFileSync(join(f.root, 'listed.create-lands'), '0'); + const unsure = run(f, ['up'], { UBI_REAP: '0', UBI_QUOTA_VCPUS: '464' }); + expect(unsure.exitCode, unsure.stderr.toString()).toBe(0); + expect(unsure.stderr.toString()).toContain('answered HTTP 503; looking it up by name before deciding'); + expect(readFileSync(join(f.root, 'quota.lock.booked'), 'utf8')).toBe(''); + rmSync(join(f.root, 'listed.create-status')); rmSync(join(f.root, 'listed.create-lands')); + expect(run(f, ['down', unsure.stdout.toString().trim()]).exitCode).toBe(0); + const unknown = run(f, ['room']); + expect(unknown.exitCode).toBe(1); + expect(unknown.stdout.toString()).toBe('UBICLOUD_ROOM: room=unknown size=standard-16 reason=no-ceiling (set UBI_QUOTA_VCPUS)\n'); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }, 120_000); + test('reap reports a failed list and a failed destroy as failures; UBI_TTL_HOURS rejects leading zeros and more than three digits', () => { const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-1000000000-bbbb-t1\n' }); try { diff --git a/test/ubicloud-runner.test.ts b/test/ubicloud-runner.test.ts index 11f6872a6..4ce8d5d2c 100644 --- a/test/ubicloud-runner.test.ts +++ b/test/ubicloud-runner.test.ts @@ -249,14 +249,38 @@ esac } finally { rmSync(f.root, { recursive: true, force: true }); } }); - test('list --mine shows this owner’s VMs only; owner tags are lowercase letters and digits, 12 max, starting with a letter', () => { + test('list --mine shows this owner’s VMs only; owner tags are lowercase letters and digits, kept in full up to 31, starting with a letter', () => { const f = fakeApi({ destroy: 200, listed: 'eu-central-h1 ubirun-thrd1-1000000000-aaaa\neu-north-h1 ubirun-other-1000000000-bbbb\neu-central-h1 ubirun-1000000000-cccc\n' }); try { const mine = f.run('list', '--mine'); expect(mine.exitCode).toBe(0); expect(mine.stdout.toString()).toBe('eu-central-h1 ubirun-thrd1-1000000000-aaaa\n'); const owner = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'owner'], { env: { ...process.env, UBI_OWNER: '12-Capy_Thread-ABCDEFGH', UBI_RUNNER_STATE: f.root }, timeout: 10_000 }); - expect(owner.stdout.toString().trim()).toBe('u12capythrea'); + expect(owner.stdout.toString().trim()).toBe('u12capythreadabcdefgh'); + const long = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'owner'], { env: { ...process.env, UBI_OWNER: 'a'.repeat(32), UBI_RUNNER_STATE: f.root }, timeout: 10_000 }); + expect(long.exitCode).toBe(1); + expect(long.stderr.toString()).toContain('UBI_OWNER is 32 characters once lowercased to letters and digits'); + expect(long.stderr.toString()).toContain('(UBI_OWNER_TOO_LONG)'); + } finally { rmSync(f.root, { recursive: true, force: true }); } + }); + + // Value: protects=owners that share their first 12 characters never reap each other's VMs, and a longer owner's VMs from runner copies that cut the owner to 12 still get reaped by the machine that recorded them; fails_when=owner_id cuts the owner again (#3748), the name parser stops at 12 characters, or the cut form is read as this owner without this machine's record; why_new=the old owner case pinned the cut itself and no case had two owners sharing a prefix; seam=none; no_claim=VMs a machine created under the cut form and whose record it lost (they stay to their owner's TTL, counted as another owner's) + test('owners sharing a 12-character prefix reap only their own expired VMs; a VM named under the old cut form is reaped only when this machine recorded it (#3748)', () => { + const listed = ['ubirun-gstackminder1-1000000000-aaaa-t1', 'ubirun-gstackminder2-1000000000-bbbb-t1', 'ubirun-gstackminder-1000000000-cccc-t1', 'ubirun-gstackminder-1000000000-dddd-t1'].map(n => `eu-central-h1 ${n}\n`).join(''); + const f = fakeApi({ destroy: 200, listed, owner: 'gstackminder1' }); + try { + // cccc: an older copy (five columns) recorded it under the cut owner; dddd: this copy recorded it for the 12-character owner gstackminder (sixth column full). + writeFileSync(f.record, 'ubirun-gstackminder-1000000000-cccc-t1\teu-central-h1\tgstackminder\t2001-09-09T01:46:40Z\t1\nubirun-gstackminder-1000000000-dddd-t1\teu-central-h1\tgstackminder\t2001-09-09T01:46:40Z\t2\tfull\n'); + expect(f.run('list', '--mine').stdout.toString()).toBe('eu-central-h1 ubirun-gstackminder1-1000000000-aaaa-t1\neu-central-h1 ubirun-gstackminder-1000000000-cccc-t1\n'); + const reap = f.run('reap'); + expect(reap.exitCode).toBe(0); + const destroyed = f.calls().split('\n').filter(l => l.includes('destroy')).map(l => l.match(/ubirun-[a-z0-9-]+/)?.[0]).sort(); + expect(destroyed).toEqual(['ubirun-gstackminder-1000000000-cccc-t1', 'ubirun-gstackminder1-1000000000-aaaa-t1']); + expect(reap.stderr.toString()).toContain('reap: left 2 expired VMs of other owners; only their owners reap them'); + const recover = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'recover', '--dir', join(f.root, 'bundles')], { env: { ...process.env, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: join(f.root, 'state'), UBI_OWNER: 'gstackminder1' }, timeout: 10_000 }); + expect(recover.stderr.toString()).not.toContain('--owner must be an owner tag'); + expect(recover.stderr.toString()).toContain('recover: no bundles under'); + expect(recover.exitCode).toBe(0); } finally { rmSync(f.root, { recursive: true, force: true }); } });