OBLITERATUS/docs/checkpoints/support-runbook.md

Support runbook: checkpoints, placement, and distributed-state reports

Artifact ID: SUPPORT-DCI-001 Version: 0.3.0 Status: Wave 2 safe structural inspection available; producer conversion deferred Owner: OBLITERATUS maintainers Tracking: Use the repository issue and pull-request workflow; include sanitized evidence only.

1. Service overview

This runbook supports the current HF-compatible loader, process-local device placement/offload, one-host remote runner, and reports involving distributed checkpoint formats. The bounded structural inspector is current; producer readers, adapters, trusted execution, and producer-backed conversion are not. See the guide and support matrix.

2. Escalation ownership

Report class	Primary owner	Escalate when
Model-capacity / “multi node offset” report	Distributed-runtime triage	Workflow exceeds one qualified host or exact model/topology is unknown
Descriptor/format/topology contract	Architecture	New format or overloaded term appears
Trusted-reader or filesystem boundary	Security	Report requires vendor/Python metadata reader
Reconstruction correctness	Test and data	Gap/overlap/replica/tie/padding behavior is unclear
Writer/recovery/resources	Persistence	ENOSPC, partial staging, or whole-state memory risk
Producer adapter	Adapter owner	Exact supported version/model mapping is identified
Provenance/PEFT	Data	Base/config/tokenizer/adapter identity is incomplete
Support claim	Release and documentation	A matrix status or public claim would change
Live multi-node runtime	Architecture and security	Reporter needs multiple participating hosts/processes

3. Operational readiness checklist

• Record exact OBLITERATUS commit and installed package versions.
• Classify the request using the glossary before suggesting remediation.
• Run obliteratus checkpoint inspect SOURCE --json only when the source can be handled under the local structural-inspection policy.
• Confirm whether the input is ordinary HF, DCP/FSDP, Megatron, DeepSpeed, PEFT, or ambiguous.
• Confirm whether the goal is loading, conversion, surgery, export, inference, or exact resume.
• Check the machine support row and its limitations.
• Do not describe a deferred/planned row as available.
• Do not cross the trusted-reader boundary during triage; no exact profile, residual-risk acceptance, reader, adapter, or payload execution is part of the current capability.
• Link evidence to the exact public issue or pull request and candidate commit.

4. Monitoring and alerts

There is no persistent intake service in version 1. For inspection, monitor the CLI exit status and descriptor blockers; stable DCI_* codes identify the first failed structural phase. For current model runs, monitor existing CLI logs, stage transitions, host RAM/disk/VRAM, source and output paths, and validation results. Alert thresholds and producer conversion metrics are not claimed.

5. Common scenarios

A. “Model needs multi node offset”

1. Route the complaint primarily to live model-capacity triage; record that this remains an interpretation, not a reporter-defined standard term.
2. Request the exact model/revision, workflow stage, current memory/topology, desired host count, failure output, and sanitized environment evidence.
3. Do not infer a checkpoint producer or adapter from the phrase. Treat a DCP, Megatron, or ZeRO/UCP artifact as a separate request that requires independent exact producer evidence.
4. Treat an explicit affine/reference-mean activation offset hypothesis as a separate algorithm question.
5. State that current model support remains single-process placement/offload or one process on one remote host; the distributed command is preflight only.

B. HF model does not fit on one device

1. Confirm model-family placement restrictions and current CLI help.
2. For compatible families, use existing single-process placement/offload or a supported quantization mode with sufficient headroom.
3. Do not call device_map rank sharding or multi-node execution.
4. Preserve the Qwen hybrid one-device restriction.

C. DCP, Megatron, or DeepSpeed directory supplied today

1. State that no current OBLITERATUS producer adapter is qualified.
2. Run the structural inspector only; a successful classification is not a load or conversion result.
3. Do not invoke an unfamiliar vendor reader as a diagnostic shortcut.
4. Collect only sanitized structural evidence from the descriptor.
5. Record the deferred adapter capability and offer external producer-supported conversion only as an operator-controlled workaround.

D. Conversion or save runs out of RAM/disk

1. Preserve the source and any prior valid output.
2. Record stage, normalized error, host resource totals/free capacity, source logical size estimate, and staging status without private local identifiers.
3. Do not promote or reuse incomplete staging as valid output.
4. Classify full-state export pressure separately from common-writer failures.

E. Request for exact training resume

Explain that the planned v1 output is model weights only. Exact resume normally requires producer-specific optimizer, scheduler, RNG/scaler, progress, and data- position state R19–R20.

F. Trusted metadata is requested

1. State that the planned trusted-reader path is not currently implemented or accepted for production use.
2. Do not treat locality, filenames, prior structural inspection, a checksum, or weights_only=True as trust R27–R29.
3. When implemented, require fresh explicit intent plus an exact single-use source/operation/runtime/isolation/resource-bound policy.
4. Refuse when any isolation/runtime capability is unavailable; never suggest a weaker subprocess/container fallback or integrity override.
5. Prefer a producer-side safetensors export performed in the operator's already trusted environment when an approved reader profile is unavailable.
6. Require security review of the exact source/runtime/profile/fixture decision; keep adapter rows deferred until the complete evidence boundary passes.

6. Troubleshooting procedure

1. Capture git rev-parse HEAD and python3 --version.
2. Capture current CLI syntax with python3 -m obliteratus --help; do not rely on examples that the parser rejects.
3. Identify the first failing stage: structural inventory/probe, current load, placement, pristine check, surgery, save, reload, or a gated intake phase.
4. Compare the report to the support matrix and glossary.
5. Check architecture/model restrictions before changing placement.
6. Record normalized relative paths or opaque IDs, file sizes, safe digests, versions, topology facts/provenance, and resource estimates.
7. Reproduce only with project-owned or explicitly approved fixtures.
8. Escalate to the owner table with exact evidence and a no-mutation statement.

7. Recovery and rollback

Current producer-neutral writes use sibling staging, validate all outputs, recheck source identity, then promote. Existing model runs retain their current persistence behavior.

• If inspection fails: no output should exist.
• If an exact registered capability reports a missing or incompatible optional dependency: retain its adapter_resolution.reason diagnostic, install only the named exact reviewed extra/version in the intended disposable profile, and retry from an unchanged source. A dependency match does not satisfy the separate trust-policy or profile-approval gates.
• If materialization fails: source/prior output remain unchanged; staging is not success.
• If promotion validation fails: the common writer restores the prior destination and removes owned staging; never overwrite evidence silently.
• If trust, source identity, runtime, containment, redaction, evidence, cleanup, or security-baseline validation fails: promote nothing, allow no bypass, and create a fresh attempt only after the cause is corrected.
• If an adapter support regression appears: remove/defer its static support row and registry entry; ordinary HF loading remains available.

8. Change management

• Contract/schema changes require architecture and traceability review plus a new schema version when incompatible.
• Adapter changes require exact producer-version fixtures and retained evidence.
• Trusted-reader/profile changes require an exact threat model, degraded-mode review, security tests, and explicit residual-risk acceptance.
• Support status changes require release approval and offline contract validation.
• Live multi-node claims require architecture and security approval plus exact multi-host evidence.
• Delivery follows repository PR, signed-commit, and CI policy.

9. Communication templates

Unsupported format

OBLITERATUS identified this as {format}, which is {status} in support matrix v1. No source or prior output was changed. The blocking contract is {code} at {phase}. Continue with {sanitized next action}.

Evidence pending

The upstream framework documents this capability, but OBLITERATUS has no exact-version/model/topology qualification at the candidate commit. The row remains deferred until the required evidence passes.

Conversion versus runtime

Converting rank-sharded model weights into HF safetensors is an offline input step. It does not make the OBLITERATUS surgery process multi-node; that runtime remains a separate, unsupported capability.

10. Post-incident activities

• Preserve descriptor/manifest and test result digests.
• Record exact commit, versions, topology, resource admission/actuals, failure code/phase, source-immutability result, and output-promotion result.
• Add a minimal project-owned regression fixture.
• Update risk, traceability, support matrix, and runbook if the contract changed.
• Never turn one successful case into an unqualified universal support claim.

11. Runbook maintenance

Review with each adapter version-band change, schema version, release candidate, and incident. Security owns trust-boundary text; architecture owns terminology; test/release own evidence status; support owns scenario clarity. Offline checks must continue validating local links, CLI examples, and support-matrix contracts.