OBLITERATUS/docs/checkpoints/support-runbook.md

# Support runbook: checkpoints, placement, and distributed-state reports

**Artifact ID:** SUPPORT-DCI-001
**Version:** 0.3.0
**Status:** Wave 2 safe structural inspection available; producer conversion deferred
**Owner:** OBLITERATUS maintainers
**Tracking:** Use the repository issue and pull-request workflow; include sanitized evidence only.

## 1. Service overview

This runbook supports the current HF-compatible loader, process-local device
placement/offload, one-host remote runner, and reports involving distributed
checkpoint formats. The bounded structural inspector is current; producer
readers, adapters, trusted execution, and producer-backed conversion are not.
See the [guide](distributed-checkpoint-intake.md) and [support
matrix](support-matrix-v1.json).

## 2. Escalation ownership

| Report class | Primary owner | Escalate when |
|---|---|---|
| Model-capacity / “multi node offset” report | Distributed-runtime triage | Workflow exceeds one qualified host or exact model/topology is unknown |
| Descriptor/format/topology contract | Architecture | New format or overloaded term appears |
| Trusted-reader or filesystem boundary | Security | Report requires vendor/Python metadata reader |
| Reconstruction correctness | Test and data | Gap/overlap/replica/tie/padding behavior is unclear |
| Writer/recovery/resources | Persistence | ENOSPC, partial staging, or whole-state memory risk |
| Producer adapter | Adapter owner | Exact supported version/model mapping is identified |
| Provenance/PEFT | Data | Base/config/tokenizer/adapter identity is incomplete |
| Support claim | Release and documentation | A matrix status or public claim would change |
| Live multi-node runtime | Architecture and security | Reporter needs multiple participating hosts/processes |

## 3. Operational readiness checklist

- [ ] Record exact OBLITERATUS commit and installed package versions.
- [ ] Classify the request using the glossary before suggesting remediation.
- [ ] Run `obliteratus checkpoint inspect SOURCE --json` only when the source can
      be handled under the local structural-inspection policy.
- [ ] Confirm whether the input is ordinary HF, DCP/FSDP, Megatron, DeepSpeed,
  PEFT, or ambiguous.
- [ ] Confirm whether the goal is loading, conversion, surgery, export,
  inference, or exact resume.
- [ ] Check the machine support row and its limitations.
- [ ] Do not describe a deferred/planned row as available.
- [ ] Do not cross the trusted-reader boundary during triage; no exact profile,
      residual-risk acceptance, reader, adapter, or payload execution is part
      of the current capability.
- [ ] Link evidence to the exact public issue or pull request and candidate commit.

## 4. Monitoring and alerts

There is no persistent intake service in version 1. For inspection, monitor the
CLI exit status and descriptor blockers; stable `DCI_*` codes identify the first
failed structural phase. For current model runs, monitor existing CLI logs,
stage transitions, host RAM/disk/VRAM, source and output paths, and validation
results. Alert thresholds and producer conversion metrics are not claimed.

## 5. Common scenarios

### A. “Model needs multi node offset”

1. Route the complaint primarily to live model-capacity triage; record that
   this remains an interpretation,
   not a reporter-defined standard term.
2. Request the exact model/revision, workflow stage, current memory/topology,
   desired host count, failure output, and sanitized environment evidence.
3. Do not infer a checkpoint producer or adapter from the phrase. Treat a DCP,
   Megatron, or ZeRO/UCP artifact as a separate request that requires independent
   exact producer evidence.
4. Treat an explicit affine/reference-mean activation offset hypothesis as a
   separate algorithm question.
5. State that current model support remains single-process placement/offload or
   one process on one remote host; the distributed command is preflight only.

### B. HF model does not fit on one device

1. Confirm model-family placement restrictions and current CLI help.
2. For compatible families, use existing single-process placement/offload or a
   supported quantization mode with sufficient headroom.
3. Do not call `device_map` rank sharding or multi-node execution.
4. Preserve the Qwen hybrid one-device restriction.

### C. DCP, Megatron, or DeepSpeed directory supplied today

1. State that no current OBLITERATUS producer adapter is qualified.
2. Run the structural inspector only; a successful classification is not a load
   or conversion result.
3. Do not invoke an unfamiliar vendor reader as a diagnostic shortcut.
4. Collect only sanitized structural evidence from the descriptor.
5. Record the deferred adapter capability and offer external producer-supported
   conversion only as an operator-controlled workaround.

### D. Conversion or save runs out of RAM/disk

1. Preserve the source and any prior valid output.
2. Record stage, normalized error, host resource totals/free capacity, source
   logical size estimate, and staging status without private local identifiers.
3. Do not promote or reuse incomplete staging as valid output.
4. Classify full-state export pressure separately from common-writer failures.

### E. Request for exact training resume

Explain that the planned v1 output is model weights only. Exact resume normally
requires producer-specific optimizer, scheduler, RNG/scaler, progress, and data-
position state [R19–R20](references.md#primary-and-upstream-sources).

### F. Trusted metadata is requested

1. State that the planned trusted-reader path is not currently implemented or
   accepted for production use.
2. Do not treat locality, filenames, prior structural inspection, a checksum,
   or `weights_only=True` as trust [R27–R29](references.md#security-and-containment-sources).
3. When implemented, require fresh explicit intent plus an exact single-use
   source/operation/runtime/isolation/resource-bound policy.
4. Refuse when any isolation/runtime capability is unavailable; never suggest a
   weaker subprocess/container fallback or integrity override.
5. Prefer a producer-side safetensors export performed in the operator's already
   trusted environment when an approved reader profile is unavailable.
6. Require security review of the exact source/runtime/profile/fixture decision;
   keep adapter rows deferred until the complete evidence boundary passes.

## 6. Troubleshooting procedure

1. Capture `git rev-parse HEAD` and `python3 --version`.
2. Capture current CLI syntax with `python3 -m obliteratus --help`; do not rely on
   examples that the parser rejects.
3. Identify the first failing stage: structural inventory/probe, current load,
   placement, pristine check, surgery, save, reload, or a gated intake phase.
4. Compare the report to the support matrix and glossary.
5. Check architecture/model restrictions before changing placement.
6. Record normalized relative paths or opaque IDs, file sizes, safe digests,
   versions, topology facts/provenance, and resource estimates.
7. Reproduce only with project-owned or explicitly approved fixtures.
8. Escalate to the owner table with exact evidence and a no-mutation statement.

## 7. Recovery and rollback

Current producer-neutral writes use sibling staging, validate all outputs,
recheck source identity, then promote. Existing model runs retain their current
persistence behavior.

- If inspection fails: no output should exist.
- If an exact registered capability reports a missing or incompatible optional
  dependency: retain its `adapter_resolution.reason` diagnostic, install only
  the named exact reviewed extra/version in the intended disposable profile,
  and retry from an unchanged source. A dependency match does not satisfy the
  separate trust-policy or profile-approval gates.
- If materialization fails: source/prior output remain unchanged; staging is not
  success.
- If promotion validation fails: the common writer restores the prior
  destination and removes owned staging; never overwrite evidence silently.
- If trust, source identity, runtime, containment, redaction, evidence, cleanup,
  or security-baseline validation fails: promote nothing, allow no bypass, and
  create a fresh attempt only after the cause is corrected.
- If an adapter support regression appears: remove/defer its static support row
  and registry entry; ordinary HF loading remains available.

## 8. Change management

- Contract/schema changes require architecture and traceability review plus a
  new schema version when incompatible.
- Adapter changes require exact producer-version fixtures and retained evidence.
- Trusted-reader/profile changes require an exact threat model, degraded-mode
  review, security tests, and explicit residual-risk acceptance.
- Support status changes require release approval and offline contract validation.
- Live multi-node claims require architecture and security approval plus exact
  multi-host evidence.
- Delivery follows repository PR, signed-commit, and CI policy.

## 9. Communication templates

### Unsupported format

> OBLITERATUS identified this as `{format}`, which is `{status}` in support
> matrix v1. No source or prior output was changed. The blocking contract is
> `{code}` at `{phase}`. Continue with `{sanitized next action}`.

### Evidence pending

> The upstream framework documents this capability, but OBLITERATUS has no
> exact-version/model/topology qualification at the candidate commit. The row
> remains deferred until the required evidence passes.

### Conversion versus runtime

> Converting rank-sharded model weights into HF safetensors is an offline input
> step. It does not make the OBLITERATUS surgery process multi-node; that runtime
> remains a separate, unsupported capability.

## 10. Post-incident activities

- Preserve descriptor/manifest and test result digests.
- Record exact commit, versions, topology, resource admission/actuals, failure
  code/phase, source-immutability result, and output-promotion result.
- Add a minimal project-owned regression fixture.
- Update risk, traceability, support matrix, and runbook if the contract changed.
- Never turn one successful case into an unqualified universal support claim.

## 11. Runbook maintenance

Review with each adapter version-band change, schema version, release candidate,
and incident. Security owns trust-boundary text; architecture owns terminology;
test/release own evidence status; support owns scenario clarity. Offline checks
must continue validating local links, CLI examples, and support-matrix contracts.