Shared GPU Host Deployment

This document defines the supported shape for preparing OBLITERATUS on a GPU host used by more than one person. It is a deployment contract, not a claim that the current Gradio process provides tenant isolation.

Supported initial mode

Install one immutable, administrator-owned release. The preferred isolation is per-user CLI execution, but a dedicated no-login service account may run one explicitly shared trusted-team UI on loopback. Keep these boundaries separate:

Boundary	Requirement
Release	Read-only to users; selected by an atomic `current` symlink
Workspace	One directory per Unix identity; mode `0700`
Model cache	Per-user by default; shared caches must be administrator-owned and read-only
Outputs	Per-user, quota-controlled, and excluded from backups unless explicitly promoted
Credentials	Per-user file, systemd credential, or trusted broker; never a shared environment file
GPU	Acquired through the host scheduler/lease broker before CUDA initialization
Network	CLI over SSH; any UI listens on loopback behind authenticated ingress

Do not grant users write access to the release checkout, virtual environment, service units, wrapper scripts, or another user's cache. Do not put users in the Docker group merely to run OBLITERATUS; Docker group membership is effectively root access.

Gradio limitation

The current UI keeps loaded-model metadata and cleanup controls in process-global state. A single shared process can therefore expose or remove another user's session artifacts. Basic authentication controls entry but does not create tenant isolation.

Until per-session artifact ownership, authorization, quotas, and cleanup tests exist, use one of these modes:

shared CLI installation with private Unix workspaces (recommended);
one UI process per user, bound to loopback on a distinct port; or
a dedicated service account serving one trusted research team through one explicitly shared workspace, bound to loopback.

Do not expose the UI directly on 0.0.0.0. Gradio --share links are also not an approved production ingress mechanism.

GPU coordination contract

Static nvidia-smi checks do not prevent races. A host scheduler must reserve the requested devices and VRAM before OBLITERATUS initializes CUDA, maintain the lease for the entire process, and release it after CUDA allocations terminate.

Schedulers that drain another inference service commonly distinguish acquire from ready. Set OBLITERATUS_GPU_LIFECYCLE_DIR to an existing runtime directory writable by the OBLITERATUS service account to enable the local lifecycle protocol. When it is unset, publication is a no-op and conservative wrappers should retain their lease for the full process lifetime.

The application atomically replaces current.json and appends ordered JSON objects to events.jsonl. Schema version 1 publishes loading, resize, ready, heartbeat, and release, with run/model identity, a monotonic sequence, process ID, timestamp, and measured allocator bytes. Readiness occurs after model allocation, never at UI startup. Set OBLITERATUS_GPU_HEARTBEAT_SECONDS to change the default 15-second heartbeat.

A minimal systemd setup lets the supervisor create the local boundary without giving the application scheduling authority:

[Service]
User=obliteratus
Group=obliteratus-gpu
RuntimeDirectory=obliteratus-gpu-lifecycle
RuntimeDirectoryMode=0770
Environment=OBLITERATUS_GPU_LIFECYCLE_DIR=/run/obliteratus-gpu-lifecycle
Environment=OBLITERATUS_GPU_HEARTBEAT_SECONDS=15
ExecStart=/srv/obliteratus/current/.venv/bin/obliteratus ui --host 127.0.0.1

The host supervisor should acquire capacity before starting this unit, consume the runtime files as untrusted structured data, deduplicate by event_id, and release or resize its reservation only after the corresponding application event. It should treat a stale heartbeat or process exit as a failed lease and must not pass its control socket, administrative API, or scheduling permissions to the application account.

The requested VRAM must include weights, activation collection, verification, checkpoint snapshots, CUDA context, and a safety margin. Multi-GPU sharding is a capacity feature, not a general throughput multiplier.

Filesystem layout

A conventional layout is:

/srv/obliteratus/
├── releases/<commit>/       # administrator-owned source and .venv
├── current -> releases/...  # atomic promotion target
├── users/<login>/           # mode 0700 workspaces and per-user caches
├── catalog/                 # optional administrator-curated read-only models
└── staging/                 # release verification before promotion

Large model checkpoints and rewritten outputs require a dedicated filesystem. Set capacity and inode alerts, per-user quotas, and a retention policy before allowing downloads. A safe default is no automatic backup of reproducible model caches and outputs; back up only configs, reports, provenance, and explicitly promoted research artifacts.

Release and rollback

Select an immutable signed tag or reviewed commit.
Verify the source artifact and supply-chain evidence described in the release process.
Build the locked environment in a new release directory.
Run installer/scripts/verify.sh on the target CUDA host.
Run a small, pinned-model smoke job through the real GPU scheduler.
Atomically repoint current only after all checks pass.
Retain the previous release until the observation window ends.

Rollback is an atomic symlink reversal followed by termination and restart of new work. Do not change the release beneath a running job.

Activation gate

Production activation requires all of the following:

dedicated storage mounted and monitored;
named users/groups and private workspace creation tested;
GPU lease integration with a truthful post-load readiness signal;
pinned release and locked CUDA environment verified;
credential isolation and telemetry policy documented;
smoke test, failure test, cancellation test, and rollback test passed;
operator runbook and host inventory published outside the application repo.

6.1 KiB Raw Blame History