mirror of
https://github.com/garrytan/gstack.git
synced 2026-11-18 06:04:33 +01:00
Stop Ubicloud fan-outs overshooting the vCPU limit and owners deleting each other's VMs (GST-2852) (#4430)
* Stop Ubicloud fan-outs overshooting the vCPU limit and owners deleting each other's VMs (GST-2852) The runner's quota check now reads usage and books the new VM's vCPUs under one lock per machine, so shards started together see each other instead of all passing on the same reading. A new room command prints how many VMs of a size fit now, so a fan-out can size each batch. When usage can't be read, a UBI_QUOTA_VCPUS ceiling refuses the create instead of letting it through. VM names keep the whole owner id (up to 31 characters) instead of its first 12, so owners sharing a prefix no longer reap each other's VMs (#3748). A VM named under the old cut form is still reaped by the machine that recorded it. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Add the changelog entry for the Ubicloud quota and owner-id fixes (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * fix: pre-landing review fixes Review-Finding: p1/specialist-batched#1 reader=testing Review-Finding: p1/specialist-batched#2 reader=red-team Review-Finding: p1/specialist-batched#3 reader=testing Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Say which old-form VMs a long owner still reaps in the changelog entry (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> * Name the room command in the Ubicloud skill summary (GST-2852) Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com> --------- Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
This commit is contained in:
1 parent
dc366816ef
commit
67f1f82fbb
7 files changed
+284
-50
No files matched your search
@@ -1036,8 +1036,10 @@ namespaces Chromium's sandbox needs, and `python3-venv` is absent (required by
|
||||
the Python runner tests).
|
||||
|
||||
VMs are named `ubirun-<owner>-<epoch>-<hex>`, plus a `-t<hours>` TTL suffix
|
||||
(below); `<owner>` is `UBI_OWNER` (the thread id on a Capy machine; lowercase
|
||||
letters and digits, at most 12), which `up`, `run` and `verify` require. Every exit
|
||||
(below); `<owner>` is `UBI_OWNER` in full (the thread id on a Capy machine; lowercase
|
||||
letters and digits, at most 31, longer is refused with `UBI_OWNER_TOO_LONG`), which `up`, `run` and `verify` require. Older runner copies cut the owner to 12
|
||||
characters, so owners sharing that prefix reaped each other's VMs (#3748); a VM named under the cut form
|
||||
counts as a longer owner's only when this machine's VM record lists it under that form. Every exit
|
||||
path destroys the client's own VM (`trap` on EXIT, INT, TERM, HUP and QUIT), and
|
||||
`down` fails loudly (`FAILED to destroy <location>/<name> (HTTP <code>)`) unless
|
||||
the VM is confirmed gone. `list --mine` shows this owner's VMs, `usage` the
|
||||
@@ -1069,7 +1071,14 @@ before the create request when that exceeds the known ceiling
|
||||
(`UBI_QUOTA_VCPUS`, or the maximum the last refusal printed, remembered in the
|
||||
state root; `UBI_QUOTA_WAIT=<seconds>` waits for room instead). When the usage read fails (`usage` exits 1 on an
|
||||
unreadable VM list or size and skips a VM that 404s), the preflight retries
|
||||
within `UBI_QUOTA_WAIT`, then logs and leaves the decision to the API. `run`, `verify`
|
||||
within `UBI_QUOTA_WAIT`, then refuses under a `UBI_QUOTA_VCPUS` ceiling
|
||||
(`UBI_QUOTA_UNREADABLE`, nothing launched) and leaves the decision to the API under a
|
||||
cached refusal maximum, which is the API's own ceiling. The usage read and the
|
||||
booking of the new VM's vCPUs run under one lock per machine (`UBI_QUOTA_LOCK`), and a
|
||||
booking counts until its create is answered, so shards a fan-out starts together
|
||||
on one machine cannot all pass on the same reading; `room [-s SIZE]` prints how many
|
||||
VMs fit now (`UBICLOUD_ROOM: room=<n> ...`) for a fan-out to size each batch. Callers
|
||||
on different machines still race, which the project's 48 spare vCPUs absorb. `run`, `verify`
|
||||
and `ssh NAME COMMAND` stream the remote output through a watchdog: a heartbeat
|
||||
line every `UBI_HEARTBEAT` seconds (300), and a command silent for
|
||||
`UBI_IDLE_TIMEOUT` seconds (1800) or older than `UBI_RUN_TIMEOUT` (0, unlimited)
|
||||
|
||||
+2
-1
@@ -1064,7 +1064,8 @@ and those with no TTL once they are 4 hours old, never another owner's; every ru
|
||||
ends with `UBICLOUD: left=0` or fails; the
|
||||
quota preflight refuses before the create request when the project's running
|
||||
VMs plus the new size exceed the known ceiling (`UBI_QUOTA_VCPUS`, or the
|
||||
maximum the last refusal printed; `UBI_QUOTA_WAIT` waits for room instead); a
|
||||
maximum the last refusal printed; `UBI_QUOTA_WAIT` waits for room instead, and
|
||||
`gstack-ubi room` prints how many more VMs fit so a fan-out can size each batch); a
|
||||
watchdog kills a remote command silent for `UBI_IDLE_TIMEOUT` seconds (1800)
|
||||
with exit 124 and its last 40 lines, and prints a heartbeat every
|
||||
`UBI_HEARTBEAT` seconds; and `gstack-ubi verify` runs the repository's
|
||||
|
||||
Reference in new issue
Block a user