fix: gate GPU allocation on supervisor admission

This commit is contained in:
Joseph Magly
2026-08-24 21:02:58 -04:00
parent ee6492d26a
commit 31ceb7041c
4 changed files with 246 additions and 18 deletions
+43 -10
View File
@@ -56,11 +56,40 @@ lifecycle protocol. When it is unset, publication is a no-op and conservative
wrappers should retain their lease for the full process lifetime.
The application atomically replaces `current.json` and appends ordered JSON
objects to `events.jsonl`. Schema version 1 publishes `loading`, `resize`,
objects to `events.jsonl`. Before any accelerator allocation it publishes a
`loading` request in the `intent_published` phase and waits for a correlated
`ack.json`. Only a valid grant advances through `admission_granted` to
`allocation_started`; denial, timeout, a stale run/request ID, changed lease
identity, or an invalid grant fails closed. Subsequent events include `resize`,
`ready`, `heartbeat`, and `release`, with run/model identity, a monotonic
sequence, process ID, timestamp, and measured allocator bytes. Readiness occurs
after model allocation, never at UI startup. Set
`OBLITERATUS_GPU_HEARTBEAT_SECONDS` to change the default 15-second heartbeat.
sequence, process ID, timestamp, and measured allocator bytes.
The supervisor acknowledgment schema is:
```json
{
"schema_version": 1,
"run_id": "copied from the loading request",
"request_event_id": "copied from the loading request",
"decision": "grant",
"lease_id": "stable supervisor-owned lease identity",
"granted_vram_bytes": 137438953472
}
```
For denial, set `decision` to `deny` and optionally include a non-sensitive
`reason`. Write the acknowledgment to a temporary file and atomically rename it
to `ack.json`; never update it in place. The application accepts only an exact
run and request correlation. A model reload or CPU-to-GPU move repeats the
barrier and must retain the same `lease_id`. The grant is a reservation ceiling:
measured growth beyond `granted_vram_bytes` fails the lifecycle contract rather
than becoming ready. Memory decreases are published after release and need no
new admission.
Readiness occurs after model allocation, never at UI startup. Set
`OBLITERATUS_GPU_HEARTBEAT_SECONDS` to change the default 15-second heartbeat,
and `OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS` to change the default 30-second
fail-closed acknowledgment timeout.
A minimal systemd setup lets the supervisor create the local boundary without
giving the application scheduling authority:
@@ -73,15 +102,19 @@ RuntimeDirectory=obliteratus-gpu-lifecycle
RuntimeDirectoryMode=0770
Environment=OBLITERATUS_GPU_LIFECYCLE_DIR=/run/obliteratus-gpu-lifecycle
Environment=OBLITERATUS_GPU_HEARTBEAT_SECONDS=15
Environment=OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS=30
ExecStart=/srv/obliteratus/current/.venv/bin/obliteratus ui --host 127.0.0.1
```
The host supervisor should acquire capacity before starting this unit, consume
the runtime files as untrusted structured data, deduplicate by `event_id`, and
release or resize its reservation only after the corresponding application
event. It should treat a stale heartbeat or process exit as a failed lease and
must not pass its control socket, administrative API, or scheduling permissions
to the application account.
The runtime directory must be writable by the application group and readable by
the root supervisor; `ack.json` should be root-owned and group-readable. The
supervisor watches `current.json`, completes acquire/prepare, atomically writes
the correlated acknowledgment, and only then permits the application wait to
finish. On supervisor restart it must recover or explicitly deny the current
intent before writing a new acknowledgment. It should consume runtime files as
untrusted structured data, deduplicate by `event_id`, treat a stale heartbeat or
process exit as a failed lease, and never pass its control API or scheduling
permissions to the application account.
The requested VRAM must include weights, activation collection, verification,
checkpoint snapshots, CUDA context, and a safety margin. Multi-GPU sharding is