CalvinBackup/OBLITERATUS
1
0
Fork 0
mirror of https://github.com/elder-plinius/OBLITERATUS.git synced 2026-08-30 14:40:38 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: add durable reconnect-safe experiment runs (#184)

Browse Source
This commit is contained in:
Joseph Magly
2026-08-28 23:50:45 -04:00
parent b36a5f7d3d
commit e8a36d796c
9 changed files with 1204 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/deployment/shared-gpu-host.md
+25
View File
@@ -112,9 +112,34 @@ RuntimeDirectoryMode=0770
Environment=OBLITERATUS_GPU_LIFECYCLE_DIR=/run/obliteratus-gpu-lifecycle
Environment=OBLITERATUS_GPU_HEARTBEAT_SECONDS=15
Environment=OBLITERATUS_GPU_ADMISSION_TIMEOUT_SECONDS=120
Environment=OBLITERATUS_RUN_ARCHIVE=/srv/obliteratus/run-archive
ExecStart=/srv/obliteratus/current/.venv/bin/obliteratus ui --host 127.0.0.1
```
Create the archive before starting the service and make it writable only by the
service account (for example, owner `obliteratus`, mode `0750`). Every UI and
headless run receives a stable `run-*` ID and writes `manifest.json`,
`events.jsonl`, `run.log`, `notes.md`, checkpoint hashes, and an atomic
`COMPLETE` marker below this root. The manifest is committed before GPU
admission, so a disconnect or process restart remains diagnosable.
The same contract is available without the UI:
```bash
obliteratus runs --archive-root /srv/obliteratus/run-archive launch -- \
Qwen/Qwen3.8-27B --method advanced --dtype bfloat16
obliteratus runs --archive-root /srv/obliteratus/run-archive status RUN_ID
obliteratus runs --archive-root /srv/obliteratus/run-archive cancel RUN_ID
obliteratus runs --archive-root /srv/obliteratus/run-archive result RUN_ID
```
Checkpoint retention is deliberately separate from evidence retention. Under
verified disk pressure, an operator may run `runs prune-checkpoint RUN_ID
--reason TEXT`; this hashes and inventories the checkpoint first, then removes
only the model payload. Notes, configuration, logs, metrics, failure details,
events, and hashes remain in the archive. Do not automate this command without
a policy that ranks and selects dominated runs.
The runtime directory must be writable by the application group and readable by
the root supervisor; `ack.json` should be root-owned and group-readable. The
supervisor watches `current.json`, completes acquire/prepare, atomically writes