OBLITERATUS/docs/checkpoints/distributed-checkpoint-intake.md

# Checkpoint formats, placement, and safe distributed-checkpoint inspection

This guide separates capabilities that are often called “sharding” but have
different contracts. The machine-readable source of truth is
[support-matrix-v1.json](support-matrix-v1.json). The current implementation provides bounded,
non-executing structural inspection plus producer-neutral validation and writing.
Producer readers, adapters, exact trusted-reader profiles, trusted payload
execution, and live multi-node model surgery remain unavailable.

## Short answer

Current OBLITERATUS can load ordinary Hugging Face-compatible checkpoints and
can use Accelerate to place complete modules across devices visible to one
process on one host. It does not currently reconstruct PyTorch DCP/FSDP,
Megatron, or DeepSpeed rank fragments, and it does not run one surgery job across
multiple hosts. It can now classify the inert structure of those checkpoint
directories without invoking their framework readers.

No producer-backed conversion is current. The common writer accepts only
already-normalized, validated fragments through its Python API; it does not read
a DCP, Megatron, or DeepSpeed payload. Future explicit offline conversion of one
narrowly qualified producer/model/version case remains Wave 3/4 work. Such a
conversion would not be exact training resume and would not make surgery
multi-node.

## Terms that must remain separate

| Term | Meaning | Current OBLITERATUS relevance |
|---|---|---|
| HF file shard | Complete named tensors distributed across files; index maps tensor name to file | Current loader can consume compatible inputs |
| safetensors byte offset | Half-open byte range inside one safetensors data buffer | File-format metadata, not rank placement |
| Accelerate `device_map` | Complete modules placed/offloaded by one process | Current, model-family dependent |
| CPU/disk offload | Process-local backing for complete model parameters | Current, not a distributed checkpoint |
| DCP/FSDP state | Framework-defined distributed state/chunks | Current structural detection only; no reader or adapter |
| Megatron offset | Logical tensor element placement and axis/chunk metadata | Current structural detection only; model-aware mapping remains required |
| DeepSpeed ZeRO/Universal | Partitioned or topology-neutral DeepSpeed state | Current structural detection only; trusted reader/resource gates remain |
| Live multi-node execution | Multiple processes/hosts with rendezvous, collectives, ownership, failure coordination, and save | Not supported; separate research decision |

The terms in this document and the machine-readable
[schemas](schemas/support-matrix-v1.schema.json) are normative for this feature. Upstream semantics and
qualifications cite the public
[source register](references.md),
including PyTorch DCP/FSDP [R01–R03], Megatron/Bridge [R04–R07], DeepSpeed
[R08–R10], HF/safetensors [R11–R13], Accelerate placement/launch [R14–R15],
and serialization/containment guidance [R27–R35].

## Current support matrix

| Subject | Load/surgery | Structural inspect | Convert to HF | Exact resume | Live multi-node |
|---|---|---|---|---|---|
| Existing compatible HF safetensors | Conditional on model/runtime gates | Conditional, header-only | Already canonical | Out of scope | Out of scope |
| Accelerate `device_map`/offload | Conditional, one process/host | Not applicable | Not applicable | Out of scope | Out of scope |
| PyTorch DCP/FSDP | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
| Megatron distributed state | No payload load | Conditional structural classification | Deferred, model-aware | Out of scope | Out of scope |
| DeepSpeed ZeRO/Universal | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
| PEFT LoRA safetensors | Conditional exact-base Python export | Conditional, header/JSON only | Not a rank-fragment conversion | Out of scope | Out of scope |
| Live multi-node surgery | Preflight only; no model payload | Not applicable | Separate offline concern | Out of scope | Deferred and unqualified |

“Conditional” means the behavior depends on an exact model architecture,
runtime, kernels, dtype/quantization, memory, and quality gates. It is not a
universal compatibility claim. “Deferred” means planned and unimplemented.

## Current safe structural inspection

Run the inspector against one local file or directory:

```bash
obliteratus checkpoint inspect ./checkpoint --json
```

The command inventories regular files without following links, streams digests,
parses bounded JSON and safetensors headers, and emits a strict descriptor with
classification evidence, confidence, resource estimates, and stable blockers.
It does not read tensor payloads, import producer frameworks, unpickle metadata,
initialize a process group, discover plugins, execute remote code, or access the
network. Treat a `conditional` classification as structural evidence only—not a
promise that the checkpoint can be loaded or converted.

Legacy HF `.bin`/`.pt` files and DCP `.metadata` may be recognized by safe names
and companion structure, but remain opaque and trust-gated. Ambiguous layouts,
links, non-regular files, races, malformed bounded metadata, and resource-limit
violations fail closed. Inspection does not mutate the source or create output.

The product capability registry is deliberately empty until one producer,
version, model family, and adapter is separately selected and qualified. The
safe plane nevertheless implements the closed-registry dependency diagnostic:
an explicitly supplied exact capability can be identified as a format candidate,
installed distribution metadata is checked without importing the distribution,
and a missing or mismatched dependency reports the exact OBLITERATUS extra,
project version, required distribution versions, and sanitized observed
versions. A candidate becomes an exact match only when independently observed
producer and version evidence also agree. Multiple matching capabilities fail
closed. This metadata-only resolution does not install anything, authorize
trust, invoke a reader, or make the capability a support claim.

## Existing single-host placement

For compatible model families, `device_map="auto"` may place complete modules
across CUDA devices visible to the current process and may use CPU/disk offload.
This is a capacity mechanism, not saved checkpoint topology and not rank-based
pipeline parallelism. Upstream Accelerate documents the placement/offload model
[R14](references.md#primary-and-upstream-sources); the project boundary is
confirmed by local code [R21](references.md#project-evidence).

Qwen hybrid models retain the complete-text-model-on-one-device restriction.
Generic multi-device layer placement is rejected for those paths because the
restriction is a correctness gate, not an unimplemented distributed-checkpoint
feature.

`--remote` selects one SSH host and launches one OBLITERATUS process there. It
does not coordinate ranks across hosts.

## Gated intake workflow

Only step 1 and the producer-neutral portions of steps 4–5 are current. The
remaining actions require separate design, security review, and qualification;
there is no producer-conversion CLI:

1. **Inspect:** bounded local inventory plus JSON/safetensors-header parsing;
   classify format, components, producer evidence, topology facts, state scope,
   trust requirement, resources, adapter match, and blockers.
2. **Escalate only when reviewed:** some vendor metadata requires a trusted
   reader. Default inspection stops and reports that requirement. Any future
   policy must require fresh per-invocation intent plus a strict single-use
   source/operation/runtime/isolation/resource-bound record and an exact approved
   disposable worker profile. A checksum, familiar local filename, prior scan,
   or `weights_only=True` is not trust. This remains unimplemented and
   unauthorized.
3. **Adapt:** one exact producer/version/model adapter emits neutral tensor
   fragments. Megatron requires a supported Bridge/provider mapping; offsets
   alone do not define fused tensor semantics [R04, R07].
4. **Validate:** prove shape/range coverage, replicas, padding, ties/shared state,
   topology, and resource budgets.
5. **Materialize:** the current Python writer accepts already-normalized neutral
   fragments, writes bounded safetensors staging, index, configuration, and
   conversion manifest, then validates and promotes only on complete success.
   It is not a producer reader or adapter.
6. **Load:** pass the canonical path into the unchanged existing HF loader.

Version 1 intentionally emits model weights only. Model weights do not include
all optimizer, scheduler, RNG/scaler, progress, and data-position state needed
for producer-compatible exact resume [R19–R20](references.md#primary-and-upstream-sources).

## What to provide with an unsupported-checkpoint report

Provide only sanitized structural evidence:

- exact model identifier and immutable revision when shareable;
- producer framework and exact version;
- checkpoint type and normalized relative file tree with sizes and safe SHA-256
  digests, excluding tensor values and sensitive local identifiers;
- saved node/world and TP/PP/DP/CP/EP/ETP/ZeRO topology when known;
- exact metadata field/API meant by “offset”;
- OBLITERATUS commit, command/config, OS, Python, PyTorch, Transformers,
  Accelerate, and optional producer versions;
- complete normalized error and first failing stage;
- desired result: inspect, convert, run surgery, export, infer, or resume.

Do not open an unfamiliar `.pt`, DCP `.metadata`, or vendor checkpoint merely to
collect a report. PyTorch documents serialization trust risks and an upstream
DCP issue identifies pickle use in `.metadata`; a 2026 advisory also shows why
weights-only loading is not a permanent safe-parser boundary [R17–R18,
R27–R29](references.md#security-and-containment-sources).

## Resource and recovery expectations

The producer-neutral writer first estimates source/logical/output/temporary
bytes and peak RAM. It enforces actual staged output/temporary bytes before
promotion; actual peak RAM and temporary-byte measurements remain `null` when
the process has not instrumented them, rather than being populated with
estimates. A denied or unknown admission does not start
materialization. Source artifacts remain immutable. Output is written into
sibling staging, validated, and promoted only when complete. Failure or
cancellation does not replace a prior valid output. The existing full-model
REBIRTH scaling bottleneck remains; the common writer API is not a practical
large-model conversion claim.

DeepSpeed warns that fp32 consolidation can require substantial CPU memory
[R08](references.md#primary-and-upstream-sources).
No general memory multiplier or GPU-count promise is made without exact retained
evidence.

## Support and escalation

Use the [support runbook](support-runbook.md) for current triage, evidence
collection, recovery, and escalation. A future `supported` matrix row requires
exact producer/adapter versions, fixture digest, candidate commit, environment,
topology, retained result, and known limits. The offline contract validator is
current; no distributed producer row is promoted to `supported` by this work.