OBLITERATUS/docs/deployment/shared-gpu-host.md

# Shared GPU Host Deployment

This document defines the supported shape for preparing OBLITERATUS on a GPU
host used by more than one person. It is a deployment contract, not a claim
that the current Gradio process provides tenant isolation.

## Supported initial mode

Install one immutable, administrator-owned release. The preferred isolation is
per-user CLI execution, but a dedicated no-login service account may run one
explicitly shared trusted-team UI on loopback. Keep these boundaries separate:

| Boundary | Requirement |
|---|---|
| Release | Read-only to users; selected by an atomic `current` symlink |
| Workspace | One directory per Unix identity; mode `0700` |
| Model cache | Per-user by default; shared caches must be administrator-owned and read-only |
| Outputs | Per-user, quota-controlled, and excluded from backups unless explicitly promoted |
| Credentials | Per-user file, systemd credential, or trusted broker; never a shared environment file |
| GPU | Acquired through the host scheduler/lease broker before CUDA initialization |
| Network | CLI over SSH; any UI listens on loopback behind authenticated ingress |

Do not grant users write access to the release checkout, virtual environment,
service units, wrapper scripts, or another user's cache. Do not put users in the
Docker group merely to run OBLITERATUS; Docker group membership is effectively
root access.

## Gradio limitation

The current UI keeps loaded-model metadata and cleanup controls in process-global
state. A single shared process can therefore expose or remove another user's
session artifacts. Basic authentication controls entry but does not create
tenant isolation.

Until per-session artifact ownership, authorization, quotas, and cleanup tests
exist, use one of these modes:

1. shared CLI installation with private Unix workspaces (recommended);
2. one UI process per user, bound to loopback on a distinct port; or
3. a dedicated service account serving one trusted research team through one
   explicitly shared workspace, bound to loopback.

Do not expose the UI directly on `0.0.0.0`. Gradio `--share` links are also not
an approved production ingress mechanism.

## GPU coordination contract

Static `nvidia-smi` checks do not prevent races. A host scheduler must reserve
the requested devices and VRAM before OBLITERATUS initializes CUDA, maintain the
lease for the entire process, and release it after CUDA allocations terminate.

Schedulers that drain another inference service commonly distinguish `acquire`
from `ready`. Set `OBLITERATUS_GPU_LIFECYCLE_DIR` to an existing runtime
directory writable by the OBLITERATUS service account to enable the local
lifecycle protocol. When it is unset, publication is a no-op and conservative
wrappers should retain their lease for the full process lifetime.

The application atomically replaces `current.json` and appends ordered JSON
objects to `events.jsonl`. Before any accelerator allocation it publishes a
`loading` request in the `intent_published` phase and waits for a correlated
`ack.json`. Only a valid grant advances through `admission_granted` to
`allocation_started`; denial, timeout, a stale run/request ID, changed lease
identity, or an invalid grant fails closed. Subsequent events include `resize`,
`ready`, `heartbeat`, and `release`, with run/model identity, a monotonic
sequence, process ID, timestamp, and measured allocator bytes.

The host supervisor must finish its broker operation before the application's
admission deadline. Reserve a safety margin for publishing a correlated denial
and cleaning up a broker request that times out. Admission errors shown in the
UI include safe run/event identifiers and deadline details; the service journal
records the matching request, decision, and duration without lease credentials.

The supervisor acknowledgment schema is:

```json
{
  "schema_version": 1,
  "run_id": "copied from the loading request",
  "request_event_id": "copied from the loading request",
  "decision": "grant",
  "lease_id": "stable supervisor-owned lease identity",
  "granted_vram_bytes": 137438953472
}
```

For denial, set `decision` to `deny` and optionally include a non-sensitive
`reason`. Write the acknowledgment to a temporary file and atomically rename it
to `ack.json`; never update it in place. The application accepts only an exact
run and request correlation. A model reload or CPU-to-GPU move repeats the
barrier and must retain the same `lease_id`. The grant is a reservation ceiling:
measured growth beyond `granted_vram_bytes` fails the lifecycle contract rather
than becoming ready. Memory decreases are published after release and need no
new admission.

Readiness occurs after model allocation, never at UI startup. Set
`OBLITERATUS_GPU_HEARTBEAT_SECONDS` to change the default 15-second heartbeat,
and `OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS` to change the default 30-second
fail-closed acknowledgment timeout. Shared hosts that must drain another model
server should use a larger measured deadline (120 seconds in the example below)
and require the supervisor's broker timeout plus heartbeat handshake to finish
several seconds before it.

A minimal systemd setup lets the supervisor create the local boundary without
giving the application scheduling authority:

```ini
[Service]
User=obliteratus
Group=obliteratus-gpu
RuntimeDirectory=obliteratus-gpu-lifecycle
RuntimeDirectoryMode=0770
Environment=OBLITERATUS_GPU_LIFECYCLE_DIR=/run/obliteratus-gpu-lifecycle
Environment=OBLITERATUS_GPU_HEARTBEAT_SECONDS=15
Environment=OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS=120
Environment=OBLITERATUS_RUN_ARCHIVE=/srv/obliteratus/run-archive
ExecStart=/srv/obliteratus/current/.venv/bin/obliteratus ui --host 127.0.0.1
```

Create the archive before starting the service and make it writable only by the
service account (for example, owner `obliteratus`, mode `0750`). Every UI and
headless run receives a stable `run-*` ID and writes `manifest.json`,
`events.jsonl`, `run.log`, `notes.md`, checkpoint hashes, and an atomic
`COMPLETE` marker below this root. The manifest is committed before GPU
admission, so a disconnect or process restart remains diagnosable.

The same contract is available without the UI:

```bash
obliteratus runs --archive-root /srv/obliteratus/run-archive launch -- \
  Qwen/Qwen3.8-27B --method advanced --dtype bfloat16
obliteratus runs --archive-root /srv/obliteratus/run-archive status RUN_ID
obliteratus runs --archive-root /srv/obliteratus/run-archive cancel RUN_ID
obliteratus runs --archive-root /srv/obliteratus/run-archive result RUN_ID
```

Checkpoint retention is deliberately separate from evidence retention. Under
verified disk pressure, an operator may run `runs prune-checkpoint RUN_ID
--reason TEXT`; this hashes and inventories the checkpoint first, then removes
only the model payload. Notes, configuration, logs, metrics, failure details,
events, and hashes remain in the archive. Do not automate this command without
a policy that ranks and selects dominated runs.

The runtime directory must be writable by the application group and readable by
the root supervisor; `ack.json` should be root-owned and group-readable. The
supervisor watches `current.json`, completes acquire/prepare, atomically writes
the correlated acknowledgment, and only then permits the application wait to
finish. On supervisor restart it must recover or explicitly deny the current
intent before writing a new acknowledgment. It should consume runtime files as
untrusted structured data, deduplicate by `event_id`, treat a stale heartbeat or
process exit as a failed lease, and never pass its control API or scheduling
permissions to the application account.

The requested VRAM must include weights, activation collection, verification,
checkpoint snapshots, CUDA context, and a safety margin. Multi-GPU sharding is
a capacity feature, not a general throughput multiplier.

## Filesystem layout

A conventional layout is:

```text
/srv/obliteratus/
├── releases/<commit>/       # administrator-owned source and .venv
├── current -> releases/...  # atomic promotion target
├── users/<login>/           # mode 0700 workspaces and per-user caches
├── catalog/                 # optional administrator-curated read-only models
└── staging/                 # release verification before promotion
```

Large model checkpoints and rewritten outputs require a dedicated filesystem.
Set capacity and inode alerts, per-user quotas, and a retention policy before
allowing downloads. A safe default is no automatic backup of reproducible model
caches and outputs; back up only configs, reports, provenance, and explicitly
promoted research artifacts.

## Release and rollback

1. Select an immutable signed tag or reviewed commit.
2. Verify the source artifact and supply-chain evidence described in
   [the release process](../RELEASE_PROCESS.md).
3. Build the locked environment in a new release directory.
4. Run `installer/scripts/verify.sh` on the target CUDA host.
5. Run a small, pinned-model smoke job through the real GPU scheduler.
6. Atomically repoint `current` only after all checks pass.
7. Retain the previous release until the observation window ends.

Rollback is an atomic symlink reversal followed by termination and restart of
new work. Do not change the release beneath a running job.

## Activation gate

Production activation requires all of the following:

- dedicated storage mounted and monitored;
- named users/groups and private workspace creation tested;
- GPU lease integration with a truthful post-load readiness signal;
- pinned release and locked CUDA environment verified;
- credential isolation and telemetry policy documented;
- smoke test, failure test, cancellation test, and rollback test passed;
- operator runbook and host inventory published outside the application repo.