mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-30 22:50:46 +02:00
105 lines
4.8 KiB
Markdown
105 lines
4.8 KiB
Markdown
# Shared GPU Host Deployment
|
|
|
|
This document defines the supported shape for preparing OBLITERATUS on a GPU
|
|
host used by more than one person. It is a deployment contract, not a claim
|
|
that the current Gradio process provides tenant isolation.
|
|
|
|
## Supported initial mode
|
|
|
|
Install one immutable, administrator-owned release. The preferred isolation is
|
|
per-user CLI execution, but a dedicated no-login service account may run one
|
|
explicitly shared trusted-team UI on loopback. Keep these boundaries separate:
|
|
|
|
| Boundary | Requirement |
|
|
|---|---|
|
|
| Release | Read-only to users; selected by an atomic `current` symlink |
|
|
| Workspace | One directory per Unix identity; mode `0700` |
|
|
| Model cache | Per-user by default; shared caches must be administrator-owned and read-only |
|
|
| Outputs | Per-user, quota-controlled, and excluded from backups unless explicitly promoted |
|
|
| Credentials | Per-user file, systemd credential, or trusted broker; never a shared environment file |
|
|
| GPU | Acquired through the host scheduler/lease broker before CUDA initialization |
|
|
| Network | CLI over SSH; any UI listens on loopback behind authenticated ingress |
|
|
|
|
Do not grant users write access to the release checkout, virtual environment,
|
|
service units, wrapper scripts, or another user's cache. Do not put users in the
|
|
Docker group merely to run OBLITERATUS; Docker group membership is effectively
|
|
root access.
|
|
|
|
## Gradio limitation
|
|
|
|
The current UI keeps loaded-model metadata and cleanup controls in process-global
|
|
state. A single shared process can therefore expose or remove another user's
|
|
session artifacts. Basic authentication controls entry but does not create
|
|
tenant isolation.
|
|
|
|
Until per-session artifact ownership, authorization, quotas, and cleanup tests
|
|
exist, use one of these modes:
|
|
|
|
1. shared CLI installation with private Unix workspaces (recommended);
|
|
2. one UI process per user, bound to loopback on a distinct port; or
|
|
3. a dedicated service account serving one trusted research team through one
|
|
explicitly shared workspace, bound to loopback.
|
|
|
|
Do not expose the UI directly on `0.0.0.0`. Gradio `--share` links are also not
|
|
an approved production ingress mechanism.
|
|
|
|
## GPU coordination contract
|
|
|
|
Static `nvidia-smi` checks do not prevent races. A host scheduler must reserve
|
|
the requested devices and VRAM before OBLITERATUS initializes CUDA, maintain the
|
|
lease for the entire process, and release it after CUDA allocations terminate.
|
|
|
|
Schedulers that drain another inference service commonly distinguish `acquire`
|
|
from `ready`. OBLITERATUS does not yet emit a portable post-model-load readiness
|
|
signal. Such deployments must not mark a lease ready immediately after process
|
|
start; add and test an application readiness hook first.
|
|
|
|
The requested VRAM must include weights, activation collection, verification,
|
|
checkpoint snapshots, CUDA context, and a safety margin. Multi-GPU sharding is
|
|
a capacity feature, not a general throughput multiplier.
|
|
|
|
## Filesystem layout
|
|
|
|
A conventional layout is:
|
|
|
|
```text
|
|
/srv/obliteratus/
|
|
├── releases/<commit>/ # administrator-owned source and .venv
|
|
├── current -> releases/... # atomic promotion target
|
|
├── users/<login>/ # mode 0700 workspaces and per-user caches
|
|
├── catalog/ # optional administrator-curated read-only models
|
|
└── staging/ # release verification before promotion
|
|
```
|
|
|
|
Large model checkpoints and rewritten outputs require a dedicated filesystem.
|
|
Set capacity and inode alerts, per-user quotas, and a retention policy before
|
|
allowing downloads. A safe default is no automatic backup of reproducible model
|
|
caches and outputs; back up only configs, reports, provenance, and explicitly
|
|
promoted research artifacts.
|
|
|
|
## Release and rollback
|
|
|
|
1. Select an immutable signed tag or reviewed commit.
|
|
2. Verify the source artifact and supply-chain evidence described in
|
|
[the release process](../RELEASE_PROCESS.md).
|
|
3. Build the locked environment in a new release directory.
|
|
4. Run `installer/scripts/verify.sh` on the target CUDA host.
|
|
5. Run a small, pinned-model smoke job through the real GPU scheduler.
|
|
6. Atomically repoint `current` only after all checks pass.
|
|
7. Retain the previous release until the observation window ends.
|
|
|
|
Rollback is an atomic symlink reversal followed by termination and restart of
|
|
new work. Do not change the release beneath a running job.
|
|
|
|
## Activation gate
|
|
|
|
Production activation requires all of the following:
|
|
|
|
- dedicated storage mounted and monitored;
|
|
- named users/groups and private workspace creation tested;
|
|
- GPU lease integration with a truthful post-load readiness signal;
|
|
- pinned release and locked CUDA environment verified;
|
|
- credential isolation and telemetry policy documented;
|
|
- smoke test, failure test, cancellation test, and rollback test passed;
|
|
- operator runbook and host inventory published outside the application repo.
|