v1.91.341.0 fix(ubicloud): no zombie VMs: owner required, 3h TTL, legacy sweep, leftover line (GST-118) (#3652)

* fix(ubicloud): no zombie VMs — owner required, 3h TTL, legacy sweep, UBICLOUD left line (GST-118)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: extend ubicloud runner and pregate-remote coverage (legacy-hours override, --keep, leak exit status, missing left= line)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix: pre-landing review fixes (TTL checks only on create, leftover line via UBI_LEFT_FILE, pregate TTL from its deadline)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* docs: name UBI_OWNER as a prerequisite for test:ubicloud and the keyed remote pregate

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* chore: changelog for GST-118 (unreleased)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* test: give the left-file parser spawnSync a timeout (spawnsync tripwire)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* fix(ubicloud): legacy sweep reaps only the caller's own TTL-less VMs (GST-324)

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

* v1.91.341.0 stamp

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>

---------

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
Garry Tanandcapy-ai[bot] authored and GitHub committed 2026-10-30 21:12:32 +00:00
1 parent 7bc2034724
commit 5532345a10
14 files changed
+316 -93

No files matched your search

+14
View File
@@ -1,5 +1,19 @@
# Changelog
## [1.91.341.0] - 2026-10-30
**gstack no longer leaves Ubicloud VMs running after a lane dies (GST-118).** A runner killed mid-run, a copy of the runner without a TTL, or a VM with no owner could keep billing until someone noticed. Now every VM names its owner and its lifetime, the next `up` anywhere destroys anything past its lifetime, and every run says in one line whether its VM is gone.
### Fixed
- **`scripts/ubicloud/ubi-runner.sh`**: `up`, `run` and `verify` refuse without `UBI_OWNER` (`UBI_OWNER_REQUIRED`), so no VM is created that nobody can name. The default TTL drops from 4 hours to 3 (`UBI_TTL_HOURS`, 1-999), checked only when a VM is created, so an old setting never blocks `down` or `reap`. `reap`, which every `up` runs first, now also destroys the caller's own `ubirun-<UBI_OWNER>-*` VMs that carry no TTL (made by older runner copies) once they are 4 hours old (`UBI_LEGACY_HOURS`). Another owner's TTL-less VM, an untagged one and names outside the `ubirun-` prefix are never touched, because only a TTL lets reap judge another lane's VM.
- **`UBICLOUD: left=<0|1|unknown> owner=<o> vm=<name> others=<n>`**: `run` and `verify` end with this line, checked against the project's VM list after teardown. Anything but `left=0` turns a passing command into exit 1 and prints the `down` command to run; `run --keep` prints `left=1 ... kept=1` and keeps its exit status.
- **`scripts/ubicloud/pregate-remote.sh`**: the remote pregate refuses without `UBI_OWNER` (`PREGATE_REMOTE_OWNER_REQUIRED`), names its VM with a TTL taken from its own stop deadline plus 15 minutes (`-t1` at today's 2,700-second timeout), and fails with `PREGATE_REMOTE_INFRA` unless the runner reports `left=0`. It reads that line from the file named by `UBI_LEFT_FILE`, never from the remote command's output.
- docs/TESTING_INTERNALS.md, docs/pregate.md, docs/skills.md and the `/ubicloud` skill describe the owner rule, the new TTL default, the legacy sweep and the `left=` line.
### Tests
- `test/ubicloud-runner.test.ts`: refusal without an owner, the TTL bounds, the legacy sweep at the default and at `UBI_LEGACY_HOURS=2` (never another owner's, an untagged or an owner-less caller's), `left=0`, `left=1` on a failed destroy (with a failing command keeping exit 7), `run --keep`, and read-only commands accepting an old `UBI_TTL_HOURS=0`.
- `test/ubicloud-pregate-remote.test.ts`: the owner refusal, the `-t1` TTL at today's timeout, and `left=1`, `left=unknown` and a missing line each failing closed with the `down` hint.
## [1.91.340.0] - 2026-10-30
### Changed
+2 -2
View File
@@ -190,8 +190,8 @@ the new defaults.
| Check an ordinary edit quickly | `bun run test:quick` | Free, about a minute | Bun 1.4.2 |
| Run one free test file while repairing | `bun test <file>` | Free, seconds | Never bare `bun test` for the suite |
| Run full free acceptance before publishing | `bun run test` | Free, a few minutes | Bun 1.4.2 |
| Run the full free suite from a small machine | `bun run test:ubicloud` | Free suite on a billed 16-vCPU VM, about 5 minutes | `UBICLOUD_API_KEY` |
| Clear pregate's test tier on a machine under 8 cores ([remote gate](docs/pregate.md#remote-gate)) | `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands` (same command keyed or keyless) | Keyed: one billed 16-vCPU VM, about 7 minutes and $0.06 for the whole free tier, worst case $0.38; keyless: free, local parallel shards, practical for small selections only (98 files took 103 s on 4 cores) | Keyed: `UBICLOUD_API_KEY`; keyless: nothing |
| Run the full free suite from a small machine | `bun run test:ubicloud` | Free suite on a billed 16-vCPU VM, about 5 minutes | `UBICLOUD_API_KEY`, `UBI_OWNER` |
| Clear pregate's test tier on a machine under 8 cores ([remote gate](docs/pregate.md#remote-gate)) | `gstack-pregate --stage tests --remote --spend-cap 2 --timeout 2700 --allow-repo-commands` (same command keyed or keyless) | Keyed: one billed 16-vCPU VM, about 7 minutes and $0.06 for the whole free tier, worst case $0.38; keyless: free, local parallel shards, practical for small selections only (98 files took 103 s on 4 cores) | Keyed: `UBICLOUD_API_KEY`, `UBI_OWNER`; keyless: nothing |
| Run the curated Windows-safe subset | `bun run test:windows` (CI: six `windows-latest` jobs) | Free | Windows, Git Bash |
| Refresh the Windows duration seed | `gh workflow run windows-free-tests.yml --ref <branch> -f record_durations=true`, then commit the `free-test-durations-windows` artifact | Free CI runners | Pushed branch, `gh` with workflow rights |
| Run a native Windows qualification campaign | `gh workflow run native-qualification.yml --ref <branch> -f mode=<mode>` ([modes](docs/TESTING_INTERNALS.md#windows-free-lane)) | Free CI runners | Pushed branch, `gh` with workflow rights |
+1 -1
View File
@@ -1 +1 @@
1.91.340.0
1.91.341.0
+1 -1
View File
@@ -1,4 +1,4 @@
# gstack digest v1.91.340.0 — regenerate/re-copy after upgrading gstack
# gstack digest v1.91.341.0 — regenerate/re-copy after upgrading gstack
Behavioral rules from gstack (https://github.com/garrytan/gstack), compressed
for agent hosts without a full skill install. The full skills add workflows,
+17 -11
View File
@@ -979,28 +979,34 @@ namespaces Chromium's sandbox needs, and `python3-venv` is absent (required by
the Python runner tests).
VMs are named `ubirun-<owner>-<epoch>-<hex>`, plus a `-t<hours>` TTL suffix
(below); `<owner>` is `UBI_OWNER` (set it to the thread id on a Capy machine;
lowercase letters and digits, at most 12), else a stable per-machine id. Every exit
(below); `<owner>` is `UBI_OWNER` (the thread id on a Capy machine; lowercase
letters and digits, at most 12), which `up`, `run` and `verify` require. Every exit
path destroys the client's own VM (`trap` on EXIT, INT, TERM, HUP and QUIT), and
`down` fails loudly (`FAILED to destroy <location>/<name> (HTTP <code>)`) unless
the VM is confirmed gone. `list --mine` shows this owner's VMs, `usage` the
project's, and `records` the VM-id scratch file (`UBI_VM_RECORD`, default
`~/.capy/work/ubi-runner/vms.tsv`, else `<state root>/vms.tsv`): every VM is
recorded there before the create call and forgotten only after a confirmed
destroy, so a killed runner leaves a name to clean up. No age-based sweep runs
by default, because the project quota is shared with other agents' runners (the
TTL reaper below destroys only a VM whose own `-t<hours>` TTL has ended); set
destroy, so a killed runner leaves a name to clean up. `run` and `verify` end
with `UBICLOUD: left=<0|1|unknown> owner=<o> vm=<name> others=<n>`, checked
against the project's VM list after teardown; anything but `left=0` turns a
passing command into exit 1. pregate-remote.sh reads that line from the file
named by `UBI_LEFT_FILE`, never from the command's output. No owner-age sweep runs by default (the TTL reaper
below covers every owner); set
`UBI_GC_HOURS` to a positive number (or run `gc HOURS`; `gc 0` is a no-op) to
sweep only this owner's VMs older than that. For other commands, use the runner
directly: `scripts/ubicloud/ubi-runner.sh run --setup <script> -- '<command>'`,
or its `up` / `ssh` / `sync` / `pull` / `down` steps (`--help` lists them).
The runner is also the `/ubicloud` skill's engine (`bin/gstack-ubi` execs it).
A generated name ends in `-t<hours>` (`UBI_TTL_HOURS`, default 4; 0 omits it),
the creator's own statement of how long the VM may live: `reap` destroys every
VM in the project whose TTL ended more than ten minutes ago, whichever owner
made it, and every `up` runs it first (`UBI_REAP=0` skips it); names without a
TTL, such as the ones pregate-remote.sh builds itself, are never reaped. The
A generated name ends in `-t<hours>` (`UBI_TTL_HOURS`, 1-999, default 3; pregate-remote.sh
uses its stop deadline plus 15 minutes, rounded up: `-t1` today), the creator's own statement of how long the VM may live: `reap`
destroys every VM in the project whose TTL ended more than ten minutes ago,
whichever owner made it, and the caller's own `ubirun-<UBI_OWNER>-*` names
without a TTL (older runner copies) once they are `UBI_LEGACY_HOURS` old
(default 4). Another owner's TTL-less VM and an untagged one are never reaped,
because only a TTL lets reap judge another lane's VM. Every `up` runs it
first (`UBI_REAP=0` skips it). Names outside the `ubirun-` prefix are never reaped. The
quota preflight adds the size's vCPUs to the project's running VMs and refuses
before the create request when that exceeds the known ceiling
(`UBI_QUOTA_VCPUS`, or the maximum the last refusal printed, remembered in the
@@ -1036,7 +1042,7 @@ machine swap, in this order:
one side local and the other remote.
4. Profiling very large imports (a whole-repo ingest, the full free suite under
a profiler) is warned about before it starts: it holds the VM for hours, and
any other lane's `up` reaps it once its TTL (`UBI_TTL_HOURS`, default 4)
any other lane's `up` reaps it once its TTL (`UBI_TTL_HOURS`, default 3)
plus ten minutes has passed, so set `UBI_TTL_HOURS` above the expected
duration and name that duration in the thread before launching.
+3 -2
View File
@@ -527,7 +527,7 @@ gstack-pregate verify # PREGATE_VERIFY: ok …; preg
gstack's own example: `.gstack/pregate.json` declares
`"remote_gate": "bash scripts/ubicloud/pregate-remote.sh"`. With
`UBICLOUD_API_KEY` set the script runs the selected files on one Ubicloud VM
`UBICLOUD_API_KEY` and `UBI_OWNER` set the script runs the selected files on one Ubicloud VM
with `bun run test:ubicloud`'s environment, refuses before creating the VM when
the cap cannot buy a 20-minute run at the size's list price (`UBI_SIZE`,
default `standard-16`), checks the tree and the planned file set on the VM
@@ -578,7 +578,8 @@ Every refusal line starts with a reason code:
| `PREGATE_REMOTE_CAP_TOO_LOW` | 5 | the cap cannot buy a 20-minute run at the size's price; nothing created |
| `PREGATE_REMOTE_NO_PRICE` | 5 | `UBI_SIZE` has no list price in the script |
| `PREGATE_REMOTE_TIMEOUT_TOO_SHORT` | 5 | pregate's deadline leaves the VM under 300 s after the 135 s teardown reserve; nothing created |
| `PREGATE_REMOTE_INFRA` | 4 | create, setup, SSH or the stop deadline failed; the VM, its teardown line and `spent=` follow |
| `PREGATE_REMOTE_OWNER_REQUIRED` | 2 | `UBI_OWNER` is unset; nothing created (every VM must name its owner) |
| `PREGATE_REMOTE_INFRA` | 4 | create, setup, SSH or the stop deadline failed, or the runner's `UBICLOUD: left=` line is not `0`; the VM, its teardown line and `spent=` follow |
| `PREGATE_REMOTE_INTERRUPTED` | 130 | TERM or INT; ubi-runner tears the VM down on its own |
<a id="guards"></a>
+8 -5
View File
@@ -989,8 +989,10 @@ A suite that takes an hour on a 4-core sandbox takes minutes on a `standard-16`
VM. `/ubicloud` runs it there through one entry point, `bin/gstack-ubi`
(`scripts/ubicloud/ubi-runner.sh`), with the discipline a shared quota needs:
every VM is named `ubirun-<owner>-<epoch>-<suffix>-t<hours>`, so its owner
(`UBI_OWNER`) and TTL (`UBI_TTL_HOURS`, default 4) are visible to everyone;
every `up` first reaps VMs whose TTL ended, whichever owner made them; the
(`UBI_OWNER`, required to create one) and TTL (`UBI_TTL_HOURS`, default 3) are
visible to everyone; every `up` first reaps VMs whose TTL ended, whichever owner
made them, and its own owner's VMs with no TTL once they are 4 hours old; every run
ends with `UBICLOUD: left=0` or fails; the
quota preflight refuses before the create request when the project's running
VMs plus the new size exceed the known ceiling (`UBI_QUOTA_VCPUS`, or the
maximum the last refusal printed; `UBI_QUOTA_WAIT` waits for room instead); a
@@ -1011,11 +1013,12 @@ You: /ubicloud run the full verify
Claude: UBICLOUD_API_KEY: set; UBI_OWNER=lane7
gstack-ubi verify (.gstack/ubicloud.json: bootstrap.sh, "bash scripts/verify.sh", standard-16)
ubi-runner: reap: no expired VMs
ubi-runner: creating eu-central-h1/ubirun-lane7-1761500000-9f3a2c10-t4 (standard-16)
ubi-runner: ubirun-lane7-1761500000-9f3a2c10-t4 carries a 4h TTL (any later run of this script reaps it); if this process dies first: gstack-ubi down ubirun-lane7-1761500000-9f3a2c10-t4
ubi-runner: creating eu-central-h1/ubirun-lane7-1761500000-9f3a2c10-t3 (standard-16)
ubi-runner: ubirun-lane7-1761500000-9f3a2c10-t3 carries a 3h TTL (any later run of this script reaps it); if this process dies first: gstack-ubi down ubirun-lane7-1761500000-9f3a2c10-t3
ubi-runner: heartbeat: command 300s elapsed, last output 2s ago
ubi-runner: command exited 0 after 1140s
ubi-runner: destroyed eu-central-h1/ubirun-lane7-1761500000-9f3a2c10-t4
ubi-runner: destroyed eu-central-h1/ubirun-lane7-1761500000-9f3a2c10-t3
UBICLOUD: left=0 owner=lane7 vm=ubirun-lane7-1761500000-9f3a2c10-t3 others=0
```
---
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "gstack",
"version": "1.91.340",
"version": "1.91.341",
"description": "Garry's Stack — Claude Code skills + fast headless browser. One repo, one install, entire AI engineering workflow.",
"license": "MIT",
"type": "module",
+11 -3
View File
@@ -102,6 +102,7 @@ if [ -z "${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}" ]; then
exit "$rc"
fi
[ -n "${UBI_OWNER:-}" ] || stop 2 PREGATE_REMOTE_OWNER_REQUIRED "UBI_OWNER is not set: every VM must carry its owner" "nothing created, nothing spent" "rerun with UBI_OWNER=<your thread code>: $rerun"
size=${UBI_SIZE:-standard-16}
case $size in
standard-2) ppm=0.00104 ;; standard-4) ppm=0.00208 ;; standard-8) ppm=0.00416 ;;
@@ -215,11 +216,11 @@ while :; do
attempt=$(( attempt + 1 ))
: >"$log"
: >"$stream"
rm -f "$log.pid" "$log.timedout"
rm -f "$log.pid" "$log.timedout" "$log.left"
t0=$(date +%s)
vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')"
vm_name="ubirun-$(bash "$here/ubi-runner.sh" owner)-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$(( (stop_after + 900 + 3599) / 3600 ))"
say "VM $vm_name; if this run is killed before its teardown line, destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name"
python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
UBI_LEFT_FILE="$log.left" python3 -c 'import os, sys; os.setsid(); os.execvp(sys.argv[1], sys.argv[1:])' \
bash -c "$vm_session" pregate-remote-vm "$stop_after" "$log" "$stream" "$here/ubi-runner.sh" "${args[@]}" -n "$vm_name" &
vmrun=$!
tail -n +1 -f "$stream" &
@@ -253,6 +254,13 @@ span() { if [ -n "$1" ] && [ -n "$2" ]; then echo "$(( $2 - $1 ))s"; else echo "
say "phases: create $(span "$t0" "$t_ready"), setup $(span "$t_setup" "$t_run"), run $(span "$t_run" "$t_exit"), teardown $(span "$t_exit" "$t_end"); total $(( t_end - started ))s (target 600s)"
say "log: $log"
left=""
[ ! -f "$log.left" ] || left=$(sed -n 's/^UBICLOUD: left=\([^ ]*\) .*/\1/p' "$log.left" | tail -1)
if [ "$left" != 0 ]; then
stop 4 PREGATE_REMOTE_INFRA "VM cleanup not confirmed (UBICLOUD: left=${left:-missing}); no result reported" \
"VM: $vm_name; destroy it with: bash scripts/ubicloud/ubi-runner.sh down $vm_name" "log: $log" "spent=$spent ($billed billed min on $size)"
fi
vm_tree=$(sed -n 's/^[0-9]* PREGATE_VM_TREE \([0-9a-f]*\)$/\1/p' "$log" | tail -1)
mismatch=$(grep -m1 -oE 'PREGATE_VM_(TREE|PLAN)_MISMATCH .*' "$log" || true)
if [ -n "$mismatch" ]; then
+91 -30
View File
@@ -23,10 +23,12 @@
# Needs only bash, curl, python3, ssh, ssh-keygen, and tar locally, so it works
# from dev boxes, containers, and cloud sandboxes.
#
# Ownership: VMs are named ubirun-<owner>-<epoch>-<suffix>[-t<hours>]. <owner> is UBI_OWNER
# Ownership: VMs are named ubirun-<owner>-<epoch>-<suffix>-t<hours>. <owner> is UBI_OWNER
# (lowercased, letters and digits only, at most 12 characters, starting with a
# letter) or a random per-machine id kept in $STATE_ROOT/owner-id. Ubicloud names
# allow 63 characters of [a-z0-9-], so a full name stays under 40.
# letter). `up`, `run` and `verify` refuse without it (UBI_OWNER_REQUIRED), so
# every VM names whoever can be asked about it; read-only commands fall back to a
# per-machine id kept in $STATE_ROOT/owner-id. Ubicloud names allow 63 characters
# of [a-z0-9-], so a full name stays under 45.
#
# Teardown: `up` records each VM's state (location, create status) before it
# sends the create request, and `down` destroys and polls until the VM is
@@ -35,16 +37,26 @@
# that `down`; further signals are ignored until it finishes. A create refused
# for vCPU quota prints the project's usage per owner.
#
# Stale-VM sweeps are off by default. With UBI_GC_HOURS set to a positive
# The owner-age sweep is off by default (the TTL reap below always runs). With UBI_GC_HOURS set to a positive
# number, every `up` first destroys this owner's VMs older than that; `gc HOURS`
# runs the same sweep on demand. Neither ever touches another owner's VMs.
#
# TTL and reaper: a name this script generates ends in -t<hours> (UBI_TTL_HOURS,
# default 4; 0 omits it), the creator's own declaration of how long the VM may
# live. `reap` destroys every VM in the project whose TTL expired more than ten
# minutes ago, whichever owner made it, and every `up` runs it first (UBI_REAP=0
# skips it), so a runner killed by a sleeping machine leaves nothing behind for
# longer than its TTL. Names without a TTL are never reaped.
# 1-999, default 3), the creator's own declaration of how long the VM may live.
# `reap` destroys every VM in the project whose TTL expired more than ten
# minutes ago, whichever owner made it, and this owner's own ubirun-<owner>-*
# names with no TTL once they are UBI_LEGACY_HOURS old (default 4; another
# owner's or an untagged TTL-less VM is never reaped); every `up` runs it first (UBI_REAP=0 skips it), so a runner
# killed by a sleeping machine leaves nothing behind for longer than its TTL.
# Names outside the ubirun- prefix are never reaped.
#
# Leftover line: `run` and `verify` end with
# UBICLOUD: left=<0|1|unknown> owner=<owner> vm=<name> others=<n>
# after teardown: left counts this invocation's own VM, checked against the
# project's VM list, and others counts this owner's other VMs (information only).
# A left other than 0 turns a passing command into exit 1; `run --keep` reports
# its live VM as left=1 kept=1 and keeps its exit status. With UBI_LEFT_FILE set,
# the line is also appended to that file, out of reach of the command's output.
#
# Quota preflight: when the project's vCPU ceiling is known (UBI_QUOTA_VCPUS, or
# the maximum a refusal printed, cached in $STATE_ROOT/quota-max), `up` adds the
@@ -75,7 +87,8 @@ CREATE_GRACE="${UBI_CREATE_GRACE:-180}"
DOWN_TIMEOUT="${UBI_DOWN_TIMEOUT:-900}"
POLL="${UBI_POLL_SECONDS:-5}"
ALLOWED_STORAGE="${UBI_ALLOWED_STORAGE:-320 640}"
TTL_HOURS="${UBI_TTL_HOURS:-4}"
TTL_HOURS="${UBI_TTL_HOURS:-3}"
LEGACY_HOURS="${UBI_LEGACY_HOURS:-4}"
REAP="${UBI_REAP:-1}"
REAP_GRACE=600
IDLE_TIMEOUT="${UBI_IDLE_TIMEOUT:-1800}"
@@ -96,7 +109,6 @@ for bin in curl python3 ssh ssh-keygen tar; do
done
UBICLOUD_API_KEY="${UBICLOUD_API_KEY:-${UBICLOUD_API_TOKEN:-}}"
[[ "$GC_HOURS" =~ ^[0-9]+$ ]] || die "UBI_GC_HOURS must be a whole number of hours (unset or 0 disables the sweep)"
[[ "$TTL_HOURS" =~ ^(0|[1-9][0-9]{0,2})$ ]] || die "UBI_TTL_HOURS must be a whole number of hours from 0 to 999 with no leading zero (0 names VMs without a TTL)"
for v in IDLE_TIMEOUT RUN_TIMEOUT HEARTBEAT QUOTA_WAIT; do
[[ "${!v}" =~ ^[0-9]+$ ]] || die "UBI_$v must be a whole number of seconds (0 disables it)"
done
@@ -121,6 +133,21 @@ owner_id() {
}
OWNER=$(owner_id)
# require_owner: up, run and verify create VMs, and a VM must name its owner
# and carry a TTL. Only creates check UBI_TTL_HOURS, so an old value never
# blocks down, reap or list.
require_owner() {
[ -n "${UBI_OWNER:-}" ] || die "UBI_OWNER is not set: every VM must carry its owner; export UBI_OWNER=<your thread or session code> (for example gsta164) and rerun (UBI_OWNER_REQUIRED)"
[[ "$TTL_HOURS" =~ ^[1-9][0-9]{0,2}$ ]] || die "UBI_TTL_HOURS must be a whole number of hours from 1 to 999 with no leading zero (every VM carries a TTL; set it above your longest run)"
}
# leftover FIELDS...: print the UBICLOUD: leftover line, and append it to
# UBI_LEFT_FILE when set, so a caller can read it apart from the command's output.
leftover() {
echo "UBICLOUD: $*" >&2
[ -z "${UBI_LEFT_FILE:-}" ] || echo "UBICLOUD: $*" >>"$UBI_LEFT_FILE"
}
# api OUT ARGS...: send one Ubicloud CLI command, write the body to OUT, and
# print the HTTP status (000 when no response arrived).
api() {
@@ -269,9 +296,10 @@ watched() {
return "$rc"
}
# vm_rows: "location name owner expires" for every VM in the project; owner is
# "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner did not
# name; expires is the epoch the name's -t<hours> TTL ends at, or "-".
# vm_rows: "location name owner expires created" for every VM in the project;
# owner is "-" for untagged ubirun-<epoch>-* names and "." for VMs this runner
# did not name; expires is the epoch the name's -t<hours> TTL ends at, or "-";
# created is the name's epoch, or "-".
vm_rows() {
local loc name owner created expires
cli vm list -N -f location,name | while read -r loc name; do
@@ -284,23 +312,31 @@ vm_rows() {
if [ -n "$created" ] && [[ "$name" =~ -t([0-9]{1,3})$ ]]; then
expires=$(( 10#$created + 10#${BASH_REMATCH[1]} * 3600 ))
fi
echo "$loc $name $owner $expires"
echo "$loc $name $owner $expires ${created:--}"
done
}
# reap: destroy every VM whose name carries a TTL that ended more than
# REAP_GRACE seconds ago, whichever owner created it (the TTL is that owner's
# own declaration). Names without a TTL are never touched.
# own declaration), and the caller's own TTL-less ubirun-<UBI_OWNER>-* names at
# least LEGACY_HOURS old. Another owner's TTL-less VM, an untagged one, and names
# outside the prefix are never touched: only a TTL lets reap judge another lane's VM.
cmd_reap() {
local now rows loc name owner expires n=0 failed=0
local now rows loc name owner expires created n=0 failed=0
[[ "$LEGACY_HOURS" =~ ^[1-9][0-9]{0,3}$ ]] || die "UBI_LEGACY_HOURS must be a whole number of hours from 1 to 9999 with no leading zero (how old a TTL-less ubirun-* VM must be before reap destroys it)"
now=$(date +%s)
rows=$(vm_rows) || { log "reap: could not list the project's VMs; nothing destroyed"; return 1; }
while read -r loc name owner expires; do
while read -r loc name owner expires created; do
[ -n "$name" ] || continue
[ "$expires" != - ] || continue
[ "$now" -ge $(( expires + REAP_GRACE )) ] || continue
if [ "$expires" != - ]; then
[ "$now" -ge $(( expires + REAP_GRACE )) ] || continue
log "reaping expired $loc/$name (owner $owner; its TTL ended $(( (now - expires) / 60 )) min ago)"
else
[ -n "${UBI_OWNER:-}" ] && [ "$owner" = "$OWNER" ] && [ "$created" != - ] || continue
[ $(( now - 10#$created )) -ge $(( LEGACY_HOURS * 3600 )) ] || continue
log "reaping TTL-less $loc/$name (owner $owner; $(( (now - 10#$created) / 3600 ))h old, legacy sweep at ${LEGACY_HOURS}h)"
fi
n=$(( n + 1 ))
log "reaping expired $loc/$name (owner $owner; its TTL ended $(( (now - expires) / 60 )) min ago)"
cmd_down "$name" -l "$loc" || { failed=$(( failed + 1 )); log "failed to reap $loc/$name"; }
done <<<"$rows"
[ "$n" -gt 0 ] || log "reap: no expired VMs"
@@ -379,9 +415,7 @@ cmd_records() {
}
new_name() {
local ttl=""
[ "$TTL_HOURS" -eq 0 ] || ttl="-t$TTL_HOURS"
echo "$PREFIX-$OWNER-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')$ttl"
echo "$PREFIX-$OWNER-$(date +%s)-$(head -c4 /dev/urandom | od -An -tx1 | tr -d ' \n')-t$TTL_HOURS"
}
# vcpus SIZE: the vCPU count a Ubicloud size name carries (standard-16 -> 16).
@@ -438,6 +472,7 @@ check_storage() {
UP_NAME=""
UP_READY=""
cmd_up() {
require_owner
local size=$DEFAULT_SIZE location=$DEFAULT_LOCATION name="" storage="" image=ubuntu-noble
while [ $# -gt 0 ]; do
case $1 in
@@ -715,7 +750,28 @@ inside_dir() {
# PULL_ROOT: set by verify; every pull destination must resolve inside it at
# pull time (an earlier pull may have placed a symlink where a later one lands).
PULL_ROOT=""
# run_teardown RC: run's EXIT handler. Destroys RUN_VM, ignoring further
# signals, then checks the project's VM list for it and prints the leftover
# line; a VM still listed, or a list that cannot be read, fails the run.
run_teardown() {
local rc=$1 rows left others
trap "" INT TERM HUP QUIT
cmd_down "$RUN_VM" || log "WARNING: failed to destroy $RUN_VM; run: $0 down $RUN_VM"
if rows=$(vm_rows 2>/dev/null); then
left=$(awk -v n="$RUN_VM" '$2==n' <<<"$rows" | grep -c . || true)
others=$(awk -v n="$RUN_VM" -v o="$OWNER" '$3==o && $2!=n' <<<"$rows" | grep -c . || true)
else
left=unknown others=unknown
fi
leftover "left=$left owner=$OWNER vm=$RUN_VM others=$others"
[ "$left" = 0 ] && return 0
log "VM cleanup not confirmed for $RUN_VM (left=$left); destroy it with: $0 down $RUN_VM"
[ "$rc" -ne 0 ] || exit 1
}
cmd_run() {
require_owner
local up_args=() src=. dest="" setup="" keep=0 pulls=() envs=()
while [ $# -gt 0 ]; do
case $1 in
@@ -745,16 +801,19 @@ cmd_run() {
RUN_VM=$name
# Armed before the create request: an interrupted `up` still destroys its VM.
# A second signal must not cut teardown short, so the EXIT handler ignores them.
trap 'trap "" INT TERM HUP QUIT; cmd_down "$RUN_VM" || log "WARNING: failed to destroy $RUN_VM; run: $0 down $RUN_VM"' EXIT
trap 'run_teardown "$?"' EXIT
trap 'exit 130' INT TERM HUP QUIT
cmd_up "${up_args[@]}" >/dev/null
if [[ "$name" =~ -t([0-9]+)$ ]]; then
log "$name carries a ${BASH_REMATCH[1]}h TTL (any later run of this script reaps it); if this process dies first: $0 down $name"
elif [[ "$name" =~ ^$PREFIX-$OWNER-[0-9]{10}- ]]; then
log "$name has no TTL (reap destroys it once it is ${LEGACY_HOURS}h old); if this process dies before its teardown line, destroy it with: $0 down $name"
else
log "$name has no TTL; if this process dies before its teardown line, destroy it with: $0 down $name"
log "$name has no TTL and no reapable name, so nothing reaps it; if this process dies before its teardown line, destroy it with: $0 down $name"
fi
if [ "$keep" = 1 ]; then
trap - EXIT
leftover "left=1 owner=$OWNER vm=$name kept=1"
log "--keep: leaving $name running; destroy with: $0 down $name"
fi
@@ -902,13 +961,15 @@ usage: ubi-runner.sh <command> [args]
owner print this caller's owner tag ($OWNER)
records VMs this machine created and has not confirmed gone ($VM_RECORD)
gc HOURS destroy this owner's VMs older than HOURS (never other owners')
reap destroy every VM whose -t<hours> TTL has ended (any owner; TTL-less names never)
reap destroy every VM whose -t<hours> TTL has ended (any owner) and this
owner's TTL-less ubirun-* VMs older than UBI_LEGACY_HOURS
cli ARGS... raw Ubicloud CLI passthrough (e.g. cli vm list)
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; set UBI_OWNER (e.g. your
thread code) to tag them, otherwise a per-machine id is used.
VMs are named $PREFIX-<owner>-<epoch>-<suffix>-t<ttl hours>; up, run and verify need
UBI_OWNER (e.g. your thread code); run and verify end with a UBICLOUD: left=<n> line.
defaults: size=$DEFAULT_SIZE location=$DEFAULT_LOCATION (env UBI_SIZE, UBI_LOCATION)
env: UBI_TTL_HOURS (${TTL_HOURS}h in the name; 0 omits it), UBI_REAP (0: up skips the reap),
env: UBI_TTL_HOURS (${TTL_HOURS}h in the name, 1-999), UBI_REAP (0: up skips the reap),
UBI_LEGACY_HOURS (${LEGACY_HOURS}h: reap's bound for this owner's TTL-less ubirun-* names),
UBI_IDLE_TIMEOUT (${IDLE_TIMEOUT}s of silence kills the remote command, exit 124; 0 disables),
UBI_RUN_TIMEOUT (${RUN_TIMEOUT}s wall clock; 0 unlimited), UBI_HEARTBEAT (${HEARTBEAT}s between heartbeat lines),
UBI_QUOTA_VCPUS (project vCPU ceiling for the preflight; else the last refusal's maximum),
+32 -7
View File
@@ -40,6 +40,7 @@ const UBI_RUNNER_STUB = `#!/usr/bin/env bash
# Every teardown is appended to $STUB_CALLS.down.
set -uo pipefail
[ "$1" != owner ] || { echo stub; exit 0; }
left() { [ "\${STUB_LEFT:-0}" = none ] || echo "UBICLOUD: left=\${STUB_LEFT:-0} owner=stub vm=$name others=0" | tee -a "\${UBI_LEFT_FILE:-/dev/null}" >&2; }
[ "$1" = run ] || exit 2
shift
printf '%s\\n' "$*" >>"$STUB_CALLS"
@@ -65,16 +66,18 @@ if [ "$fail" = quota ] || { [ "$fail" = quota-once ] && [ "$calls" -eq 1 ]; }; t
log "quota refused eu-central-h1/$name ($size): it needs 16 vCPUs and the project already uses 64 of 64"
log "create failed: vCPU quota exhausted; wait for running VMs to finish, or use fewer or smaller VMs (ci:ubicloud --vms N)"
log "gone: eu-central-h1/$name (never existed or already destroyed)"
left
exit 1
fi
if [ "$fail" = create ]; then
echo "Validation failed for following fields: size" >&2
log "create failed"
log "gone: eu-central-h1/$name (never existed or already destroyed)"
left
exit 1
fi
log "ready: $name (192.0.2.1)"
trap 'log "destroyed eu-central-h1/$name"; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
trap 'log "destroyed eu-central-h1/$name"; left; echo "term $name" >>"$STUB_CALLS.down"; exit 130' TERM
vm=$STUB_VM_HOME
dest="$vm/work/$(basename "$src")"
rm -rf "$dest" && mkdir -p "$dest"
@@ -88,6 +91,7 @@ if [ -n "$setup" ]; then
if ! (cd "$dest" && HOME="$vm" bash -c '. ~/.ubirun-env && bash ~/.ubirun-setup.sh'); then
log "setup failed"
log "destroyed eu-central-h1/$name"
left
exit 1
fi
fi
@@ -102,6 +106,7 @@ else
fi
log "command exited $rc after 1s"
log "destroyed eu-central-h1/$name"
left
echo "exit $name" >>"$STUB_CALLS.down"
exit "$rc"
`;
@@ -150,7 +155,7 @@ function gate(f: Fixture, files: string[] | null, env: Record<string, string | u
const out = r.stdout.toString() + r.stderr.toString();
return { exit: r.exitCode, out, report: parseRemoteReport(out), calls: fs.existsSync(f.calls) ? fs.readFileSync(f.calls, 'utf8').trim().split('\n') : [] };
}
const KEY = { UBICLOUD_API_KEY: 'offline-stub' };
const KEY = { UBICLOUD_API_KEY: 'offline-stub', UBI_OWNER: 'stub' };
describe('pregate-remote.sh inputs and the local tree check', () => {
test('run bare it exits 2 before any work, naming the pregate command', () => {
@@ -208,6 +213,26 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
expect(remoteReportProblem(empty.report!, empty.report!.tree)).toBe('the remote gate reported 0 tests');
}, 90_000);
// Value: protects=a runner that prints no UBICLOUD: left= line fails closed as PREGATE_REMOTE_INFRA (left=missing); fails_when=the leftover check treats an empty left as clean; why_new=the 1 and unknown rows never exercise a missing line; seam=none; no_claim=real ubi-runner output (stub runner)
test('a passing run whose VM is not confirmed gone, or whose runner printed no leftover line, is PREGATE_REMOTE_INFRA, never a test result', () => {
for (const [stubLeft, shown] of [['1', 'left=1'], ['unknown', 'left=unknown'], ['none', 'left=missing']] as const) {
const r = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_LEFT: stubLeft });
expect(r.exit, r.out).toBe(4);
expect(r.out).toContain(`PREGATE_REMOTE_INFRA: VM cleanup not confirmed (UBICLOUD: ${shown}); no result reported`);
expect(r.out).toMatch(/VM: (ubirun-stub-\d+-[0-9a-f]{8}-t1); destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1/);
expect(r.report).toBeNull();
}
}, 90_000);
test('without UBI_OWNER the keyed path refuses with exit 2 before any VM is created', () => {
const f = fixture();
const r = gate(f, ['test/alpha.test.ts'], { ...KEY, UBI_OWNER: '' });
expect(r.exit, r.out).toBe(2);
expect(r.out).toContain('PREGATE_REMOTE_OWNER_REQUIRED: UBI_OWNER is not set: every VM must carry its owner');
expect(r.out).toContain('nothing created, nothing spent');
expect(fs.existsSync(f.calls)).toBe(false);
}, 60_000);
test('the cap refuses before any VM is created, at the price of the size actually used', () => {
const f = fixture();
const low = gate(f, ['test/alpha.test.ts'], { ...KEY, PREGATE_SPEND_CAP_USD: '0.1' });
@@ -231,14 +256,14 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
const create = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'create' });
expect(create.exit, create.out).toBe(4);
expect(create.out).toContain('PREGATE_REMOTE_INFRA: infrastructure failure (ubi-runner exit 1): ubi-runner: create failed');
expect(create.out).toMatch(/VM: eu-central-h1\/(ubirun-stub-\d+-[0-9a-f]{8}); ubi-runner: gone: eu-central-h1\/\1 \(never existed or already destroyed\)/);
expect(create.out).toMatch(/VM (ubirun-stub-\d+-[0-9a-f]{8}); if this run is killed before its teardown line, destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1\n/);
expect(create.out).toMatch(/VM: eu-central-h1\/(ubirun-stub-\d+-[0-9a-f]{8}-t1); ubi-runner: gone: eu-central-h1\/\1 \(never existed or already destroyed\)/);
expect(create.out).toMatch(/VM (ubirun-stub-\d+-[0-9a-f]{8}-t1); if this run is killed before its teardown line, destroy it with: bash scripts\/ubicloud\/ubi-runner\.sh down \1\n/);
expect(create.out).toMatch(/log: \S+\/\.context\/pregate-remote\/\S+\/run\.log/);
expect(create.out).toContain('spent=0.00 (0 billed min on standard-16)');
const setup = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_SETUP_FAIL: '1' });
expect(setup.exit, setup.out).toBe(4);
expect(setup.out).toContain('ubi-runner: setup failed');
expect(setup.out).toMatch(/; ubi-runner: destroyed eu-central-h1\/ubirun-stub-\d+-[0-9a-f]{8}/);
expect(setup.out).toMatch(/; ubi-runner: destroyed eu-central-h1\/ubirun-stub-\d+-[0-9a-f]{8}-t1/);
expect(setup.out).toContain('spent=0.01 (1 billed min on standard-16)');
const ssh = gate(fixture(), ['test/alpha.test.ts'], { ...KEY, STUB_FAIL: 'ssh' });
expect(ssh.exit, ssh.out).toBe(4);
@@ -293,7 +318,7 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
process.kill(-a.pid, 'SIGKILL');
await a.exited;
await until(() => down(killed).endsWith('\n'), 30_000);
expect(down(killed)).toMatch(/^exit ubirun-stub-\d+-[0-9a-f]{8}\n$/);
expect(down(killed)).toMatch(/^exit ubirun-stub-\d+-[0-9a-f]{8}-t1\n$/);
const termed = fixture();
const b = run(termed);
await until(() => vmLog(termed).includes('ubi-runner: running on'), 30_000);
@@ -301,7 +326,7 @@ describe('pregate-remote.sh keyed path (stub ubi-runner)', () => {
expect(await b.exited).toBe(130);
expect(await new Response(b.stdout).text()).toContain('interrupted: ubi-runner is tearing the VM down on its own');
await until(() => down(termed).endsWith('\n'), 10_000);
expect(down(termed)).toMatch(/^term ubirun-stub-\d+-[0-9a-f]{8}\n$/);
expect(down(termed)).toMatch(/^term ubirun-stub-\d+-[0-9a-f]{8}-t1\n$/);
const edited = fixture();
const c = run(edited);
await until(() => vmLog(edited).includes('ubi-runner: running on'), 30_000);
+103 -18
View File
@@ -54,8 +54,10 @@ describe('ubicloud free-suite runner', () => {
const runner = read('scripts/ubicloud/ubi-runner.sh');
expect(runner).toContain('"umask 022; $*"');
// Armed before the create request and shielded from a second signal (record-before-create teardown, plan E2).
expect(runner).toMatch(/trap 'trap "" INT TERM HUP QUIT; cmd_down "\$RUN_VM"[^']*' EXIT/);
expect(runner.indexOf("trap 'trap \"\" INT TERM HUP QUIT; cmd_down \"$RUN_VM\"")).toBeLessThan(runner.indexOf('cmd_up "${up_args[@]}" >/dev/null'));
expect(runner).toContain(`trap 'run_teardown "$?"' EXIT`);
expect(runner.indexOf(`trap 'run_teardown "$?"' EXIT`)).toBeLessThan(runner.indexOf('cmd_up "${up_args[@]}" >/dev/null'));
const teardown = runner.slice(runner.indexOf('run_teardown() {'), runner.indexOf('\n}\n', runner.indexOf('run_teardown() {')));
expect(teardown.indexOf('trap "" INT TERM HUP QUIT')).toBeLessThan(teardown.indexOf('cmd_down "$RUN_VM"'));
});
test('pull retrieves the retained free-test logs and skips a glob that matches nothing', () => {
@@ -210,7 +212,7 @@ esac
expect(r.exitCode).toBe(1);
const err = r.stderr.toString();
expect(err).toMatch(/reaping expired eu-central-h1\/ubirun-other-1000000000-bbbb-t1 \(owner other; its TTL ended \d+ min ago\)/);
expect(err).toMatch(/quota preflight refused eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t4 \(standard-16\): it needs 16 vCPUs and the project's VMs already use 16 of 31/);
expect(err).toMatch(/quota preflight refused eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 \(standard-16\): it needs 16 vCPUs and the project's VMs already use 16 of 31/);
expect(err).toContain('create refused before any request: vCPU quota exhausted');
const destroyed = f.calls().split('\n').filter(l => l.includes('destroy'));
expect(destroyed).toHaveLength(1);
@@ -220,20 +222,46 @@ esac
} finally { rmSync(f.root, { recursive: true, force: true }); }
});
test('UBI_REAP=0 skips the reap and UBI_TTL_HOURS=0 names the VM without a TTL; reap alone never touches unexpired or TTL-less names', () => {
const listed = 'eu-central-h1 ubirun-other-1000000000-bbbb-t1\neu-central-h1 ubirun-thrd1-9999999999-dddd-t1\neu-central-h1 ubirun-thrd1-1000000000-aaaa\neu-central-h1 ubirun-1000000000-cccc\n';
test('UBI_REAP=0 skips the reap and UBI_TTL_HOURS sets the name\'s TTL; reap destroys expired TTLs and this owner\'s old TTL-less names, never unexpired, recent, another owner\'s TTL-less, untagged or foreign ones', () => {
const now = Math.floor(Date.now() / 1000);
const listed = `eu-central-h1 ubirun-other-1000000000-bbbb-t1\neu-central-h1 ubirun-thrd1-9999999999-dddd-t1\neu-central-h1 ubirun-thrd1-1000000000-aaaa\neu-central-h1 ubirun-1000000000-cccc\neu-central-h1 ubirun-thrd1-${now - 3 * 3600}-eeee\neu-central-h1 ubirun-gbra240-1000000000-gggg\neu-central-h1 team-db-1000000000-ffff\n`;
const f = fakeApi({ destroy: 200, listed });
try {
const up = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'up'], { env: { ...process.env, PATH: `${join(f.root, 'bin')}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: join(f.root, 'state'), UBI_POLL_SECONDS: '1', UBI_VM_RECORD: f.record, UBI_OWNER: 'thrd1', UBI_QUOTA_VCPUS: '64', UBI_REAP: '0', UBI_TTL_HOURS: '0' }, timeout: 90_000 });
const up = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'up'], { env: { ...process.env, PATH: `${join(f.root, 'bin')}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: join(f.root, 'state'), UBI_POLL_SECONDS: '1', UBI_VM_RECORD: f.record, UBI_OWNER: 'thrd1', UBI_QUOTA_VCPUS: '112', UBI_REAP: '0', UBI_TTL_HOURS: '7' }, timeout: 90_000 });
expect(up.exitCode).toBe(1);
expect(up.stderr.toString()).toMatch(/quota preflight refused eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8} \(standard-16\): it needs 16 vCPUs and the project's VMs already use 64 of 64/);
expect(up.stderr.toString()).toMatch(/quota preflight refused eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t7 \(standard-16\): it needs 16 vCPUs and the project's VMs already use 112 of 112/);
expect(f.calls()).not.toContain('destroy');
const reap = f.run('reap');
expect(reap.exitCode).toBe(0);
const destroyed = f.calls().split('\n').filter(l => l.includes('destroy'));
expect(destroyed).toHaveLength(1);
expect(destroyed[0]).toContain('ubirun-other-1000000000-bbbb-t1');
expect(reap.stderr.toString()).toMatch(/reaping TTL-less eu-central-h1\/ubirun-thrd1-1000000000-aaaa \(owner thrd1; \d+h old, legacy sweep at 4h\)/);
const destroyed = f.calls().split('\n').filter(l => l.includes('destroy')).map(l => l.match(/ubirun-[a-z0-9-]+|team-db-[a-z0-9-]+/)?.[0]).sort();
expect(destroyed).toEqual(['ubirun-other-1000000000-bbbb-t1', 'ubirun-thrd1-1000000000-aaaa']);
expect(f.run('reap').stderr.toString()).toContain('reap: no expired VMs');
// Value: protects=UBI_LEGACY_HOURS sets reap's bound for TTL-less ubirun names (a 3h-old name is spared at 4h, reaped at 2h); fails_when=reap ignores UBI_LEGACY_HOURS and keeps a fixed bound; why_new=only the default bound and the validation are tested; seam=none; no_claim=real Ubicloud list and destroy (fake curl)
const lowered = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'reap'], { env: { ...process.env, PATH: `${join(f.root, 'bin')}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: join(f.root, 'state'), UBI_POLL_SECONDS: '1', UBI_VM_RECORD: f.record, UBI_OWNER: 'thrd1', UBI_LEGACY_HOURS: '2' }, timeout: 90_000 });
expect(lowered.exitCode).toBe(0);
expect(lowered.stderr.toString()).toMatch(/reaping TTL-less eu-central-h1\/ubirun-thrd1-\d{10}-eeee \(owner thrd1; 3h old, legacy sweep at 2h\)/);
expect(f.calls()).toMatch(/ubirun-thrd1-\d{10}-eeee[^\n]*destroy/);
// Value: protects=the legacy sweep never destroys another lane's or an untagged TTL-less VM (GST-324: 22 gbra240 eval VMs lost); fails_when=the sweep matches any owner again; why_new=the cross-owner sweep destroyed a paid eval; seam=none; no_claim=real Ubicloud list and destroy (fake curl)
expect(f.calls()).not.toMatch(/(team-db|dddd|gggg|cccc)[^\n]*destroy/);
const ownerless = Bun.spawnSync(['bash', join(DIR, 'ubi-runner.sh'), 'reap'], { env: { ...process.env, PATH: `${join(f.root, 'bin')}:${process.env.PATH}`, UBICLOUD_API_KEY: 'offline', UBI_RUNNER_STATE: join(f.root, 'state'), UBI_POLL_SECONDS: '1', UBI_VM_RECORD: f.record, UBI_OWNER: '', UBI_LEGACY_HOURS: '1' }, timeout: 90_000 });
expect(ownerless.exitCode).toBe(0);
expect(ownerless.stderr.toString()).not.toContain('reaping TTL-less');
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
test('up and run refuse without UBI_OWNER before any API call; list, usage, reap and down still work without it', () => {
const f = fakeApi({ destroy: 200, listed: 'eu-central-h1 vm-a\n', owner: '' });
try {
for (const args of [['up'], ['run', '--', 'true']]) {
const r = f.run(...args);
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toContain('UBI_OWNER is not set: every VM must carry its owner; export UBI_OWNER=<your thread or session code>');
expect(r.stderr.toString()).toContain('(UBI_OWNER_REQUIRED)');
}
expect(f.calls()).toBe('');
for (const args of [['list'], ['usage'], ['reap']]) expect(f.run(...args).exitCode).toBe(0);
expect(f.run('down', 'vm-a').exitCode).toBe(0);
} finally { rmSync(f.root, { recursive: true, force: true }); }
});
@@ -381,8 +409,8 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
'name=$(printf \'%s\' "$body" | sed -n \'s/.*"vm", *"[^/"]*\\/\\([^"]*\\)".*/\\1/p\')\n' +
'case "$body" in\n' +
' *\'"create"\'*) if [ -n ' + JSON.stringify(opts.refuse ?? '') + ' ]; then printf \'%s\' ' + JSON.stringify(opts.refuse ?? '') + ' > "$out"; printf 400; else echo "$loc $name" >> "$listed"; : > "$out"; printf 200; fi ;;\n' +
' *destroy*) : > "$out"; grep -v " $name$" "$listed" > "$listed.tmp"; mv "$listed.tmp" "$listed"; printf 200 ;;\n' +
' *\'"list"\'*) cat "$listed" > "$out"; printf 200 ;;\n' +
' *destroy*) : > "$out"; if [ -f "$listed.destroy-fails" ]; then printf 500; else grep -v " $name$" "$listed" > "$listed.tmp"; mv "$listed.tmp" "$listed"; printf 200; fi ;;\n' +
' *\'"list"\'*) if [ -f "$listed.list-fails" ]; then echo down > "$out"; printf 503; else cat "$listed" > "$out"; printf 200; fi ;;\n' +
' *\'"show"\'*) [ -z "${FAKE_SHOW_DELAY:-}" ] || sleep "$FAKE_SHOW_DELAY"; if grep -q " $name$" "$listed"; then printf \'state: running\\nsize: standard-16\\nip4: 192.0.2.1\\n\' > "$out"; printf 200; else : > "$out"; printf 404; fi ;;\n' +
' *) : > "$out"; printf 200 ;;\n' +
'esac\n');
@@ -439,7 +467,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
expect(f.ssh()).toMatch(/setsid -w bash -c 'printf %s \$\$ >~\/\.ubirun-setup-\d+\.sid; cd work\/[^ ]+ && \. ~\/\.ubirun-env && bash -l ~\/\.ubirun-setup\.sh'; else bash -c '[^']*'; fi$/m);
expect(readFileSync(join(f.root, 'cancel.log'), 'utf8')).toMatch(/rm -f ~\/\.ubirun-setup-\d+\.sid; \[ -z "\$sid" \] \|\| \{ pkill -TERM -s "\$sid"; sleep 2; pkill -KILL -s "\$sid"; \}/);
expect(f.calls()).toContain('destroy');
expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t4/);
expect(err).toMatch(/destroyed eu-central-h1\/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3/);
expect(f.listed()).toBe('');
expect(readFileSync(f.record, 'utf8')).toBe('');
} finally { rmSync(f.root, { recursive: true, force: true }); }
@@ -457,6 +485,57 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 60_000);
test('run: ends with UBICLOUD: left=0 for its own VM; a same-owner sibling is counted as information and does not fail the run', () => {
const f = fakeVm({ listed: 'eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n' });
try {
const leftFile = join(f.root, 'left');
const r = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_LEFT_FILE: leftFile });
const err = r.stderr.toString();
expect(r.exitCode, err).toBe(0);
expect(err).toMatch(/UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
expect(f.listed()).toBe('eu-central-h1 ubirun-thrd1-9999999999-sib1-t3\n');
// Value: protects=UBI_LEFT_FILE carries the same leftover line pregate-remote.sh parses, apart from the command's output; fails_when=the file is not written or its format drifts from pregate-remote.sh's parser; why_new=the pregate stub hand-copies the format; seam=none; no_claim=real VM teardown (fake API and ssh)
expect(readFileSync(leftFile, 'utf8')).toMatch(/^UBICLOUD: left=0 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n$/);
const parser = read('scripts/ubicloud/pregate-remote.sh').match(/left=\$\(sed -n '([^']+)' "\$log\.left"/)![1];
expect(Bun.spawnSync(['sed', '-n', parser, leftFile], { timeout: 30_000 }).stdout.toString()).toBe('0\n');
// Value: protects=run --keep reports its live VM as UBICLOUD: left=1 ... kept=1 and never a clean left=0; fails_when=the kept line is dropped or --keep still prints the teardown line; why_new=no test runs run --keep; seam=none; no_claim=real VM lifetime (fake API and ssh)
const kept = f.run(['run', '--src', f.root, '--keep', '--', 'true'], { UBI_REAP: '0' });
const keptErr = kept.stderr.toString();
expect(kept.exitCode, keptErr).toBe(0);
expect(keptErr).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 kept=1\n/);
expect(keptErr).not.toContain('left=0');
expect(f.listed()).toMatch(/ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3\n/);
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
test('run: a passing command whose teardown fails exits 1 with UBICLOUD: left=1 and the down command; an unreadable VM list fails closed', () => {
const f = fakeVm();
try {
writeFileSync(join(f.root, 'listed.destroy-fails'), '');
const leaked = f.run(['run', '--src', f.root, '--', 'true'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
const err = leaked.stderr.toString();
expect(leaked.exitCode, err).toBe(1);
expect(err).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=(ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3) others=0\n[^]*VM cleanup not confirmed for \1 \(left=1\); destroy it with: \S+ down \1/);
expect(f.listed()).toContain('ubirun-thrd1-');
// Value: protects=a failing command whose VM also leaks keeps its own exit status, not 1; fails_when=run_teardown exits 1 whenever left is not 0; why_new=the leak case above only runs a passing command; seam=none; no_claim=real destroy failure modes (fake API returns 500)
writeFileSync(join(f.root, 'command-mode'), 'exit 7');
const failedLeak = f.run(['run', '--src', f.root, '--', 'false'], { UBI_REAP: '0', UBI_DOWN_TIMEOUT: '3' });
expect(failedLeak.exitCode, failedLeak.stderr.toString()).toBe(7);
expect(failedLeak.stderr.toString()).toMatch(/UBICLOUD: left=1 owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=1\n/);
const blind = fakeVm();
try {
writeFileSync(join(blind.root, 'bin/curl.real'), readFileSync(join(blind.root, 'bin/curl')));
chmodSync(join(blind.root, 'bin/curl.real'), 0o755);
writeFileSync(join(blind.root, 'bin/curl'), '#!/usr/bin/env bash\nif [ -f ' + JSON.stringify(join(blind.root, 'armed')) + ' ]; then touch ' + JSON.stringify(join(blind.root, 'listed.list-fails')) + '; fi\nexec ' + JSON.stringify(join(blind.root, 'bin/curl.real')) + ' "$@"\n');
chmodSync(join(blind.root, 'bin/curl'), 0o755);
writeFileSync(join(blind.root, 'command-mode'), 'touch ' + JSON.stringify(join(blind.root, 'armed')));
const r = blind.run(['run', '--src', blind.root, '--', 'true'], { UBI_REAP: '0' });
expect(r.exitCode, r.stderr.toString()).toBe(1);
expect(r.stderr.toString()).toMatch(/UBICLOUD: left=unknown owner=thrd1 vm=ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3 others=unknown\n/);
} finally { rmSync(blind.root, { recursive: true, force: true }); }
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 60_000);
test('run: the fake ssh dispatches on the remote command, so a checkout whose name holds -O still syncs and runs (GSTA-139)', () => {
const f = fakeVm();
try {
@@ -612,7 +691,7 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
expect(f.calls()).toContain('"create"');
expect(existsSync(join(f.state, 'quota-max'))).toBe(false);
const name = stale.stdout.toString().trim();
expect(name).toMatch(/^ubirun-thrd1-\d{10}-[0-9a-f]{8}-t4$/);
expect(name).toMatch(/^ubirun-thrd1-\d{10}-[0-9a-f]{8}-t3$/);
expect(run(f, ['down', name]).exitCode).toBe(0);
expect(f.listed()).toBe('eu-central-h1 ubirun-peer-9999999999-cccc\n');
} finally { rmSync(f.root, { recursive: true, force: true }); }
@@ -660,12 +739,18 @@ describe('ubi-runner run, verify and quota against a fake API and a fake ssh', (
const listFails = run(f, ['reap']);
expect(listFails.exitCode).toBe(1);
expect(listFails.stderr.toString()).toContain("reap: could not list the project's VMs; nothing destroyed");
for (const ttl of ['08', '1000', 'abc']) {
const r = run(f, ['owner'], { UBI_TTL_HOURS: ttl });
for (const ttl of ['0', '08', '1000', 'abc']) {
const r = run(f, ['up'], { UBI_TTL_HOURS: ttl });
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toContain('UBI_TTL_HOURS must be a whole number of hours from 0 to 999');
expect(r.stderr.toString()).toContain('UBI_TTL_HOURS must be a whole number of hours from 1 to 999');
}
// Value: protects=an old UBI_TTL_HOURS=0 or bad UBI_LEGACY_HOURS never blocks commands that create nothing (down, list, owner); fails_when=the TTL checks move back to startup; why_new=the checks were global and blocked cleanup; seam=none; no_claim=real down against the API (owner stands in for every read-only command)
expect(run(f, ['owner'], { UBI_TTL_HOURS: '0', UBI_LEGACY_HOURS: '0' }).exitCode).toBe(0);
for (const legacy of ['0', '04', 'x']) {
const r = run(f, ['reap'], { UBI_LEGACY_HOURS: legacy });
expect(r.exitCode).toBe(1);
expect(r.stderr.toString()).toContain('UBI_LEGACY_HOURS must be a whole number of hours from 1 to 9999');
}
expect(run(f, ['owner'], { UBI_TTL_HOURS: '999' }).exitCode).toBe(0);
} finally { rmSync(f.root, { recursive: true, force: true }); }
}, 30_000);
});
+16 -6
View File
@@ -126,8 +126,10 @@ $U owner
`UBICLOUD_API_KEY` (or `UBICLOUD_API_TOKEN`) is a project-scoped token. When it
is unset, ask for it as an environment variable and stop; never echo it, write
it to a file or commit, and never run `env`, `printenv` or `set -x` while it is
in the environment. `UBI_OWNER` is what `usage` and `list --mine` key on, so set
it before the first `up` of every session and on every machine.
in the environment. `UBI_OWNER` is required: `up`, `run` and `verify` refuse
without it (`UBI_OWNER_REQUIRED`), because a VM nobody can name is a VM nobody
cleans up. `usage` and `list --mine` key on it, so set it before the first
`up` of every session and on every machine.
---
@@ -143,8 +145,10 @@ $U usage # VMs and vCPUs per owner: who holds the shared quota
A VM in `records` but absent from `list` is already gone: `$U down <name>`
forgets it. A VM in the project under another owner is theirs; leave it (its
TTL will reap it). Every `up` runs `$U reap` first, destroying any VM whose
`-t<hours>` TTL ended more than ten minutes ago, whichever owner made it; a name
without a TTL is never reaped.
`-t<hours>` TTL ended more than ten minutes ago, whichever owner made it, and
this owner's own `ubirun-<UBI_OWNER>-*` names with no TTL once they are
`UBI_LEGACY_HOURS` old (default 4). Another owner's TTL-less VM, an untagged
one and names outside the `ubirun-` prefix are never reaped.
Quota: with `UBI_QUOTA_VCPUS=<project ceiling>` exported (or after one refusal,
whose maximum the runner remembers), `up` adds the size's vCPUs to the project's
@@ -165,8 +169,9 @@ background operation leaves its VM until the reaper. So:
detach it, and never leave it unwatched when the thread goes idle.
- In Claude Code, run it under `gstack-detach` and poll the log for its
`### gstack-detach EXIT=<code> ###` sentinel.
- Set `UBI_TTL_HOURS` above the longest plausible run (default 4) and name the
expected duration in the thread before launching anything over an hour.
- Set `UBI_TTL_HOURS` above the longest plausible run (default 3; 1-999) and
name the expected duration in the thread before launching anything over an
hour. A run that outlives its TTL is reaped by the next `up` anywhere.
The runner prints `<name> carries a <n>h TTL ...; if this process dies first:
<self> down <name>` right after the VM is ready; quote that line in your report
@@ -238,6 +243,11 @@ and for browser tests `xvfb` plus
- `ubi-runner: command exited <rc> after <n>s` is the verdict line; the exit
status is the command's.
- `UBICLOUD: left=<n> owner=<o> vm=<name> others=<n>` follows teardown. Report
it with every result: `left=0` means this run's VM is confirmed gone. `left=1`
or `left=unknown` turns a passing command into exit 1; run the printed `down`
command now. `others` counts this owner's other VMs, for information.
`run --keep` prints `left=1 ... kept=1` instead and keeps its exit status.
- Exit 124 with `watchdog: <phase> printed nothing for <n>s` (or `ran <n>s`)
means the remote hung or exceeded its budget; the last 40 lines follow.
Diagnose from them before rerunning; a hang is a failure, not infrastructure.
+16 -6
View File
@@ -52,8 +52,10 @@ $U owner
`UBICLOUD_API_KEY` (or `UBICLOUD_API_TOKEN`) is a project-scoped token. When it
is unset, ask for it as an environment variable and stop; never echo it, write
it to a file or commit, and never run `env`, `printenv` or `set -x` while it is
in the environment. `UBI_OWNER` is what `usage` and `list --mine` key on, so set
it before the first `up` of every session and on every machine.
in the environment. `UBI_OWNER` is required: `up`, `run` and `verify` refuse
without it (`UBI_OWNER_REQUIRED`), because a VM nobody can name is a VM nobody
cleans up. `usage` and `list --mine` key on it, so set it before the first
`up` of every session and on every machine.
---
@@ -69,8 +71,10 @@ $U usage # VMs and vCPUs per owner: who holds the shared quota
A VM in `records` but absent from `list` is already gone: `$U down <name>`
forgets it. A VM in the project under another owner is theirs; leave it (its
TTL will reap it). Every `up` runs `$U reap` first, destroying any VM whose
`-t<hours>` TTL ended more than ten minutes ago, whichever owner made it; a name
without a TTL is never reaped.
`-t<hours>` TTL ended more than ten minutes ago, whichever owner made it, and
this owner's own `ubirun-<UBI_OWNER>-*` names with no TTL once they are
`UBI_LEGACY_HOURS` old (default 4). Another owner's TTL-less VM, an untagged
one and names outside the `ubirun-` prefix are never reaped.
Quota: with `UBI_QUOTA_VCPUS=<project ceiling>` exported (or after one refusal,
whose maximum the runner remembers), `up` adds the size's vCPUs to the project's
@@ -91,8 +95,9 @@ background operation leaves its VM until the reaper. So:
detach it, and never leave it unwatched when the thread goes idle.
- In Claude Code, run it under `gstack-detach` and poll the log for its
`### gstack-detach EXIT=<code> ###` sentinel.
- Set `UBI_TTL_HOURS` above the longest plausible run (default 4) and name the
expected duration in the thread before launching anything over an hour.
- Set `UBI_TTL_HOURS` above the longest plausible run (default 3; 1-999) and
name the expected duration in the thread before launching anything over an
hour. A run that outlives its TTL is reaped by the next `up` anywhere.
The runner prints `<name> carries a <n>h TTL ...; if this process dies first:
<self> down <name>` right after the VM is ready; quote that line in your report
@@ -164,6 +169,11 @@ and for browser tests `xvfb` plus
- `ubi-runner: command exited <rc> after <n>s` is the verdict line; the exit
status is the command's.
- `UBICLOUD: left=<n> owner=<o> vm=<name> others=<n>` follows teardown. Report
it with every result: `left=0` means this run's VM is confirmed gone. `left=1`
or `left=unknown` turns a passing command into exit 1; run the printed `down`
command now. `others` counts this owner's other VMs, for information.
`run --keep` prints `left=1 ... kept=1` instead and keeps its exit status.
- Exit 124 with `watchdog: <phase> printed nothing for <n>s` (or `ran <n>s`)
means the remote hung or exceeded its budget; the last 40 lines follow.
Diagnose from them before rerunning; a hang is a failure, not infrastructure.