mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-29 22:20:36 +02:00
fix: gate GPU allocation on supervisor admission
This commit is contained in:
@@ -56,11 +56,40 @@ lifecycle protocol. When it is unset, publication is a no-op and conservative
|
||||
wrappers should retain their lease for the full process lifetime.
|
||||
|
||||
The application atomically replaces `current.json` and appends ordered JSON
|
||||
objects to `events.jsonl`. Schema version 1 publishes `loading`, `resize`,
|
||||
objects to `events.jsonl`. Before any accelerator allocation it publishes a
|
||||
`loading` request in the `intent_published` phase and waits for a correlated
|
||||
`ack.json`. Only a valid grant advances through `admission_granted` to
|
||||
`allocation_started`; denial, timeout, a stale run/request ID, changed lease
|
||||
identity, or an invalid grant fails closed. Subsequent events include `resize`,
|
||||
`ready`, `heartbeat`, and `release`, with run/model identity, a monotonic
|
||||
sequence, process ID, timestamp, and measured allocator bytes. Readiness occurs
|
||||
after model allocation, never at UI startup. Set
|
||||
`OBLITERATUS_GPU_HEARTBEAT_SECONDS` to change the default 15-second heartbeat.
|
||||
sequence, process ID, timestamp, and measured allocator bytes.
|
||||
|
||||
The supervisor acknowledgment schema is:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"run_id": "copied from the loading request",
|
||||
"request_event_id": "copied from the loading request",
|
||||
"decision": "grant",
|
||||
"lease_id": "stable supervisor-owned lease identity",
|
||||
"granted_vram_bytes": 137438953472
|
||||
}
|
||||
```
|
||||
|
||||
For denial, set `decision` to `deny` and optionally include a non-sensitive
|
||||
`reason`. Write the acknowledgment to a temporary file and atomically rename it
|
||||
to `ack.json`; never update it in place. The application accepts only an exact
|
||||
run and request correlation. A model reload or CPU-to-GPU move repeats the
|
||||
barrier and must retain the same `lease_id`. The grant is a reservation ceiling:
|
||||
measured growth beyond `granted_vram_bytes` fails the lifecycle contract rather
|
||||
than becoming ready. Memory decreases are published after release and need no
|
||||
new admission.
|
||||
|
||||
Readiness occurs after model allocation, never at UI startup. Set
|
||||
`OBLITERATUS_GPU_HEARTBEAT_SECONDS` to change the default 15-second heartbeat,
|
||||
and `OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS` to change the default 30-second
|
||||
fail-closed acknowledgment timeout.
|
||||
|
||||
A minimal systemd setup lets the supervisor create the local boundary without
|
||||
giving the application scheduling authority:
|
||||
@@ -73,15 +102,19 @@ RuntimeDirectory=obliteratus-gpu-lifecycle
|
||||
RuntimeDirectoryMode=0770
|
||||
Environment=OBLITERATUS_GPU_LIFECYCLE_DIR=/run/obliteratus-gpu-lifecycle
|
||||
Environment=OBLITERATUS_GPU_HEARTBEAT_SECONDS=15
|
||||
Environment=OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS=30
|
||||
ExecStart=/srv/obliteratus/current/.venv/bin/obliteratus ui --host 127.0.0.1
|
||||
```
|
||||
|
||||
The host supervisor should acquire capacity before starting this unit, consume
|
||||
the runtime files as untrusted structured data, deduplicate by `event_id`, and
|
||||
release or resize its reservation only after the corresponding application
|
||||
event. It should treat a stale heartbeat or process exit as a failed lease and
|
||||
must not pass its control socket, administrative API, or scheduling permissions
|
||||
to the application account.
|
||||
The runtime directory must be writable by the application group and readable by
|
||||
the root supervisor; `ack.json` should be root-owned and group-readable. The
|
||||
supervisor watches `current.json`, completes acquire/prepare, atomically writes
|
||||
the correlated acknowledgment, and only then permits the application wait to
|
||||
finish. On supervisor restart it must recover or explicitly deny the current
|
||||
intent before writing a new acknowledgment. It should consume runtime files as
|
||||
untrusted structured data, deduplicate by `event_id`, treat a stale heartbeat or
|
||||
process exit as a failed lease, and never pass its control API or scheduling
|
||||
permissions to the application account.
|
||||
|
||||
The requested VRAM must include weights, activation collection, verification,
|
||||
checkpoint snapshots, CUDA context, and a safety margin. Multi-GPU sharding is
|
||||
|
||||
Reference in New Issue
Block a user