feat: add safe distributed checkpoint intake and preflight

docs/checkpoints/distributed-checkpoint-intake.md

@@ -0,0 +1,189 @@
+# Checkpoint formats, placement, and safe distributed-checkpoint inspection
+
+This guide separates capabilities that are often called “sharding” but have
+different contracts. The machine-readable source of truth is
+[support-matrix-v1.json](support-matrix-v1.json). The current implementation provides bounded,
+non-executing structural inspection plus producer-neutral validation and writing.
+Producer readers, adapters, exact trusted-reader profiles, trusted payload
+execution, and live multi-node model surgery remain unavailable.
+
+## Short answer
+
+Current OBLITERATUS can load ordinary Hugging Face-compatible checkpoints and
+can use Accelerate to place complete modules across devices visible to one
+process on one host. It does not currently reconstruct PyTorch DCP/FSDP,
+Megatron, or DeepSpeed rank fragments, and it does not run one surgery job across
+multiple hosts. It can now classify the inert structure of those checkpoint
+directories without invoking their framework readers.
+
+No producer-backed conversion is current. The common writer accepts only
+already-normalized, validated fragments through its Python API; it does not read
+a DCP, Megatron, or DeepSpeed payload. Future explicit offline conversion of one
+narrowly qualified producer/model/version case remains Wave 3/4 work. Such a
+conversion would not be exact training resume and would not make surgery
+multi-node.
+
+## Terms that must remain separate
+
+| Term | Meaning | Current OBLITERATUS relevance |
+|---|---|---|
+| HF file shard | Complete named tensors distributed across files; index maps tensor name to file | Current loader can consume compatible inputs |
+| safetensors byte offset | Half-open byte range inside one safetensors data buffer | File-format metadata, not rank placement |
+| Accelerate `device_map` | Complete modules placed/offloaded by one process | Current, model-family dependent |
+| CPU/disk offload | Process-local backing for complete model parameters | Current, not a distributed checkpoint |
+| DCP/FSDP state | Framework-defined distributed state/chunks | Current structural detection only; no reader or adapter |
+| Megatron offset | Logical tensor element placement and axis/chunk metadata | Current structural detection only; model-aware mapping remains required |
+| DeepSpeed ZeRO/Universal | Partitioned or topology-neutral DeepSpeed state | Current structural detection only; trusted reader/resource gates remain |
+| Live multi-node execution | Multiple processes/hosts with rendezvous, collectives, ownership, failure coordination, and save | Not supported; separate research decision |
+
+The terms in this document and the machine-readable
+[schemas](schemas/support-matrix-v1.schema.json) are normative for this feature. Upstream semantics and
+qualifications cite the public
+[source register](references.md),
+including PyTorch DCP/FSDP [R01–R03], Megatron/Bridge [R04–R07], DeepSpeed
+[R08–R10], HF/safetensors [R11–R13], Accelerate placement/launch [R14–R15],
+and serialization/containment guidance [R27–R35].
+
+## Current support matrix
+
+| Subject | Load/surgery | Structural inspect | Convert to HF | Exact resume | Live multi-node |
+|---|---|---|---|---|---|
+| Existing compatible HF safetensors | Conditional on model/runtime gates | Conditional, header-only | Already canonical | Out of scope | Out of scope |
+| Accelerate `device_map`/offload | Conditional, one process/host | Not applicable | Not applicable | Out of scope | Out of scope |
+| PyTorch DCP/FSDP | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
+| Megatron distributed state | No payload load | Conditional structural classification | Deferred, model-aware | Out of scope | Out of scope |
+| DeepSpeed ZeRO/Universal | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
+| PEFT LoRA safetensors | Conditional exact-base Python export | Conditional, header/JSON only | Not a rank-fragment conversion | Out of scope | Out of scope |
+| Live multi-node surgery | Preflight only; no model payload | Not applicable | Separate offline concern | Out of scope | Deferred and unqualified |
+
+“Conditional” means the behavior depends on an exact model architecture,
+runtime, kernels, dtype/quantization, memory, and quality gates. It is not a
+universal compatibility claim. “Deferred” means planned and unimplemented.
+
+## Current safe structural inspection
+
+Run the inspector against one local file or directory:
+
+```bash
+obliteratus checkpoint inspect ./checkpoint --json
+```
+
+The command inventories regular files without following links, streams digests,
+parses bounded JSON and safetensors headers, and emits a strict descriptor with
+classification evidence, confidence, resource estimates, and stable blockers.
+It does not read tensor payloads, import producer frameworks, unpickle metadata,
+initialize a process group, discover plugins, execute remote code, or access the
+network. Treat a `conditional` classification as structural evidence only—not a
+promise that the checkpoint can be loaded or converted.
+
+Legacy HF `.bin`/`.pt` files and DCP `.metadata` may be recognized by safe names
+and companion structure, but remain opaque and trust-gated. Ambiguous layouts,
+links, non-regular files, races, malformed bounded metadata, and resource-limit
+violations fail closed. Inspection does not mutate the source or create output.
+
+The product capability registry is deliberately empty until one producer,
+version, model family, and adapter is separately selected and qualified. The
+safe plane nevertheless implements the closed-registry dependency diagnostic:
+an explicitly supplied exact capability can be identified as a format candidate,
+installed distribution metadata is checked without importing the distribution,
+and a missing or mismatched dependency reports the exact OBLITERATUS extra,
+project version, required distribution versions, and sanitized observed
+versions. A candidate becomes an exact match only when independently observed
+producer and version evidence also agree. Multiple matching capabilities fail
+closed. This metadata-only resolution does not install anything, authorize
+trust, invoke a reader, or make the capability a support claim.
+
+## Existing single-host placement
+
+For compatible model families, `device_map="auto"` may place complete modules
+across CUDA devices visible to the current process and may use CPU/disk offload.
+This is a capacity mechanism, not saved checkpoint topology and not rank-based
+pipeline parallelism. Upstream Accelerate documents the placement/offload model
+[R14](references.md#primary-and-upstream-sources); the project boundary is
+confirmed by local code [R21](references.md#project-evidence).
+
+Qwen hybrid models retain the complete-text-model-on-one-device restriction.
+Generic multi-device layer placement is rejected for those paths because the
+restriction is a correctness gate, not an unimplemented distributed-checkpoint
+feature.
+
+`--remote` selects one SSH host and launches one OBLITERATUS process there. It
+does not coordinate ranks across hosts.
+
+## Gated intake workflow
+
+Only step 1 and the producer-neutral portions of steps 4–5 are current. The
+remaining actions require separate design, security review, and qualification;
+there is no producer-conversion CLI:
+
+1. **Inspect:** bounded local inventory plus JSON/safetensors-header parsing;
+   classify format, components, producer evidence, topology facts, state scope,
+   trust requirement, resources, adapter match, and blockers.
+2. **Escalate only when reviewed:** some vendor metadata requires a trusted
+   reader. Default inspection stops and reports that requirement. Any future
+   policy must require fresh per-invocation intent plus a strict single-use
+   source/operation/runtime/isolation/resource-bound record and an exact approved
+   disposable worker profile. A checksum, familiar local filename, prior scan,
+   or `weights_only=True` is not trust. This remains unimplemented and
+   unauthorized.
+3. **Adapt:** one exact producer/version/model adapter emits neutral tensor
+   fragments. Megatron requires a supported Bridge/provider mapping; offsets
+   alone do not define fused tensor semantics [R04, R07].
+4. **Validate:** prove shape/range coverage, replicas, padding, ties/shared state,
+   topology, and resource budgets.
+5. **Materialize:** the current Python writer accepts already-normalized neutral
+   fragments, writes bounded safetensors staging, index, configuration, and
+   conversion manifest, then validates and promotes only on complete success.
+   It is not a producer reader or adapter.
+6. **Load:** pass the canonical path into the unchanged existing HF loader.
+
+Version 1 intentionally emits model weights only. Model weights do not include
+all optimizer, scheduler, RNG/scaler, progress, and data-position state needed
+for producer-compatible exact resume [R19–R20](references.md#primary-and-upstream-sources).
+
+## What to provide with an unsupported-checkpoint report
+
+Provide only sanitized structural evidence:
+
+- exact model identifier and immutable revision when shareable;
+- producer framework and exact version;
+- checkpoint type and normalized relative file tree with sizes and safe SHA-256
+  digests, excluding tensor values and sensitive local identifiers;
+- saved node/world and TP/PP/DP/CP/EP/ETP/ZeRO topology when known;
+- exact metadata field/API meant by “offset”;
+- OBLITERATUS commit, command/config, OS, Python, PyTorch, Transformers,
+  Accelerate, and optional producer versions;
+- complete normalized error and first failing stage;
+- desired result: inspect, convert, run surgery, export, infer, or resume.
+
+Do not open an unfamiliar `.pt`, DCP `.metadata`, or vendor checkpoint merely to
+collect a report. PyTorch documents serialization trust risks and an upstream
+DCP issue identifies pickle use in `.metadata`; a 2026 advisory also shows why
+weights-only loading is not a permanent safe-parser boundary [R17–R18,
+R27–R29](references.md#security-and-containment-sources).
+
+## Resource and recovery expectations
+
+The producer-neutral writer first estimates source/logical/output/temporary
+bytes and peak RAM. It enforces actual staged output/temporary bytes before
+promotion; actual peak RAM and temporary-byte measurements remain `null` when
+the process has not instrumented them, rather than being populated with
+estimates. A denied or unknown admission does not start
+materialization. Source artifacts remain immutable. Output is written into
+sibling staging, validated, and promoted only when complete. Failure or
+cancellation does not replace a prior valid output. The existing full-model
+REBIRTH scaling bottleneck remains; the common writer API is not a practical
+large-model conversion claim.
+
+DeepSpeed warns that fp32 consolidation can require substantial CPU memory
+[R08](references.md#primary-and-upstream-sources).
+No general memory multiplier or GPU-count promise is made without exact retained
+evidence.
+
+## Support and escalation
+
+Use the [support runbook](support-runbook.md) for current triage, evidence
+collection, recovery, and escalation. A future `supported` matrix row requires
+exact producer/adapter versions, fixture digest, candidate commit, environment,
+topology, retained result, and known limits. The offline contract validator is
+current; no distributed producer row is promoted to `supported` by this work.