mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-09-22 01:10:49 +02:00
feat: add safe distributed checkpoint intake and preflight
This commit is contained in:
Joseph Magly
@@ -0,0 +1,203 @@
|
||||
# Support runbook: checkpoints, placement, and distributed-state reports
|
||||
|
||||
**Artifact ID:** SUPPORT-DCI-001
|
||||
**Version:** 0.3.0
|
||||
**Status:** Wave 2 safe structural inspection available; producer conversion deferred
|
||||
**Owner:** OBLITERATUS maintainers
|
||||
**Tracking:** Use the repository issue and pull-request workflow; include sanitized evidence only.
|
||||
|
||||
## 1. Service overview
|
||||
|
||||
This runbook supports the current HF-compatible loader, process-local device
|
||||
placement/offload, one-host remote runner, and reports involving distributed
|
||||
checkpoint formats. The bounded structural inspector is current; producer
|
||||
readers, adapters, trusted execution, and producer-backed conversion are not.
|
||||
See the [guide](distributed-checkpoint-intake.md) and [support
|
||||
matrix](support-matrix-v1.json).
|
||||
|
||||
## 2. Escalation ownership
|
||||
|
||||
| Report class | Primary owner | Escalate when |
|
||||
|---|---|---|
|
||||
| Model-capacity / “multi node offset” report | Distributed-runtime triage | Workflow exceeds one qualified host or exact model/topology is unknown |
|
||||
| Descriptor/format/topology contract | Architecture | New format or overloaded term appears |
|
||||
| Trusted-reader or filesystem boundary | Security | Report requires vendor/Python metadata reader |
|
||||
| Reconstruction correctness | Test and data | Gap/overlap/replica/tie/padding behavior is unclear |
|
||||
| Writer/recovery/resources | Persistence | ENOSPC, partial staging, or whole-state memory risk |
|
||||
| Producer adapter | Adapter owner | Exact supported version/model mapping is identified |
|
||||
| Provenance/PEFT | Data | Base/config/tokenizer/adapter identity is incomplete |
|
||||
| Support claim | Release and documentation | A matrix status or public claim would change |
|
||||
| Live multi-node runtime | Architecture and security | Reporter needs multiple participating hosts/processes |
|
||||
|
||||
## 3. Operational readiness checklist
|
||||
|
||||
- [ ] Record exact OBLITERATUS commit and installed package versions.
|
||||
- [ ] Classify the request using the glossary before suggesting remediation.
|
||||
- [ ] Run `obliteratus checkpoint inspect SOURCE --json` only when the source can
|
||||
be handled under the local structural-inspection policy.
|
||||
- [ ] Confirm whether the input is ordinary HF, DCP/FSDP, Megatron, DeepSpeed,
|
||||
PEFT, or ambiguous.
|
||||
- [ ] Confirm whether the goal is loading, conversion, surgery, export,
|
||||
inference, or exact resume.
|
||||
- [ ] Check the machine support row and its limitations.
|
||||
- [ ] Do not describe a deferred/planned row as available.
|
||||
- [ ] Do not cross the trusted-reader boundary during triage; no exact profile,
|
||||
residual-risk acceptance, reader, adapter, or payload execution is part
|
||||
of the current capability.
|
||||
- [ ] Link evidence to the exact public issue or pull request and candidate commit.
|
||||
|
||||
## 4. Monitoring and alerts
|
||||
|
||||
There is no persistent intake service in version 1. For inspection, monitor the
|
||||
CLI exit status and descriptor blockers; stable `DCI_*` codes identify the first
|
||||
failed structural phase. For current model runs, monitor existing CLI logs,
|
||||
stage transitions, host RAM/disk/VRAM, source and output paths, and validation
|
||||
results. Alert thresholds and producer conversion metrics are not claimed.
|
||||
|
||||
## 5. Common scenarios
|
||||
|
||||
### A. “Model needs multi node offset”
|
||||
|
||||
1. Route the complaint primarily to live model-capacity triage; record that
|
||||
this remains an interpretation,
|
||||
not a reporter-defined standard term.
|
||||
2. Request the exact model/revision, workflow stage, current memory/topology,
|
||||
desired host count, failure output, and sanitized environment evidence.
|
||||
3. Do not infer a checkpoint producer or adapter from the phrase. Treat a DCP,
|
||||
Megatron, or ZeRO/UCP artifact as a separate request that requires independent
|
||||
exact producer evidence.
|
||||
4. Treat an explicit affine/reference-mean activation offset hypothesis as a
|
||||
separate algorithm question.
|
||||
5. State that current model support remains single-process placement/offload or
|
||||
one process on one remote host; the distributed command is preflight only.
|
||||
|
||||
### B. HF model does not fit on one device
|
||||
|
||||
1. Confirm model-family placement restrictions and current CLI help.
|
||||
2. For compatible families, use existing single-process placement/offload or a
|
||||
supported quantization mode with sufficient headroom.
|
||||
3. Do not call `device_map` rank sharding or multi-node execution.
|
||||
4. Preserve the Qwen hybrid one-device restriction.
|
||||
|
||||
### C. DCP, Megatron, or DeepSpeed directory supplied today
|
||||
|
||||
1. State that no current OBLITERATUS producer adapter is qualified.
|
||||
2. Run the structural inspector only; a successful classification is not a load
|
||||
or conversion result.
|
||||
3. Do not invoke an unfamiliar vendor reader as a diagnostic shortcut.
|
||||
4. Collect only sanitized structural evidence from the descriptor.
|
||||
5. Record the deferred adapter capability and offer external producer-supported
|
||||
conversion only as an operator-controlled workaround.
|
||||
|
||||
### D. Conversion or save runs out of RAM/disk
|
||||
|
||||
1. Preserve the source and any prior valid output.
|
||||
2. Record stage, normalized error, host resource totals/free capacity, source
|
||||
logical size estimate, and staging status without private local identifiers.
|
||||
3. Do not promote or reuse incomplete staging as valid output.
|
||||
4. Classify full-state export pressure separately from common-writer failures.
|
||||
|
||||
### E. Request for exact training resume
|
||||
|
||||
Explain that the planned v1 output is model weights only. Exact resume normally
|
||||
requires producer-specific optimizer, scheduler, RNG/scaler, progress, and data-
|
||||
position state [R19–R20](references.md#primary-and-upstream-sources).
|
||||
|
||||
### F. Trusted metadata is requested
|
||||
|
||||
1. State that the planned trusted-reader path is not currently implemented or
|
||||
accepted for production use.
|
||||
2. Do not treat locality, filenames, prior structural inspection, a checksum,
|
||||
or `weights_only=True` as trust [R27–R29](references.md#security-and-containment-sources).
|
||||
3. When implemented, require fresh explicit intent plus an exact single-use
|
||||
source/operation/runtime/isolation/resource-bound policy.
|
||||
4. Refuse when any isolation/runtime capability is unavailable; never suggest a
|
||||
weaker subprocess/container fallback or integrity override.
|
||||
5. Prefer a producer-side safetensors export performed in the operator's already
|
||||
trusted environment when an approved reader profile is unavailable.
|
||||
6. Require security review of the exact source/runtime/profile/fixture decision;
|
||||
keep adapter rows deferred until the complete evidence boundary passes.
|
||||
|
||||
## 6. Troubleshooting procedure
|
||||
|
||||
1. Capture `git rev-parse HEAD` and `python3 --version`.
|
||||
2. Capture current CLI syntax with `python3 -m obliteratus --help`; do not rely on
|
||||
examples that the parser rejects.
|
||||
3. Identify the first failing stage: structural inventory/probe, current load,
|
||||
placement, pristine check, surgery, save, reload, or a gated intake phase.
|
||||
4. Compare the report to the support matrix and glossary.
|
||||
5. Check architecture/model restrictions before changing placement.
|
||||
6. Record normalized relative paths or opaque IDs, file sizes, safe digests,
|
||||
versions, topology facts/provenance, and resource estimates.
|
||||
7. Reproduce only with project-owned or explicitly approved fixtures.
|
||||
8. Escalate to the owner table with exact evidence and a no-mutation statement.
|
||||
|
||||
## 7. Recovery and rollback
|
||||
|
||||
Current producer-neutral writes use sibling staging, validate all outputs,
|
||||
recheck source identity, then promote. Existing model runs retain their current
|
||||
persistence behavior.
|
||||
|
||||
- If inspection fails: no output should exist.
|
||||
- If an exact registered capability reports a missing or incompatible optional
|
||||
dependency: retain its `adapter_resolution.reason` diagnostic, install only
|
||||
the named exact reviewed extra/version in the intended disposable profile,
|
||||
and retry from an unchanged source. A dependency match does not satisfy the
|
||||
separate trust-policy or profile-approval gates.
|
||||
- If materialization fails: source/prior output remain unchanged; staging is not
|
||||
success.
|
||||
- If promotion validation fails: the common writer restores the prior
|
||||
destination and removes owned staging; never overwrite evidence silently.
|
||||
- If trust, source identity, runtime, containment, redaction, evidence, cleanup,
|
||||
or security-baseline validation fails: promote nothing, allow no bypass, and
|
||||
create a fresh attempt only after the cause is corrected.
|
||||
- If an adapter support regression appears: remove/defer its static support row
|
||||
and registry entry; ordinary HF loading remains available.
|
||||
|
||||
## 8. Change management
|
||||
|
||||
- Contract/schema changes require architecture and traceability review plus a
|
||||
new schema version when incompatible.
|
||||
- Adapter changes require exact producer-version fixtures and retained evidence.
|
||||
- Trusted-reader/profile changes require an exact threat model, degraded-mode
|
||||
review, security tests, and explicit residual-risk acceptance.
|
||||
- Support status changes require release approval and offline contract validation.
|
||||
- Live multi-node claims require architecture and security approval plus exact
|
||||
multi-host evidence.
|
||||
- Delivery follows repository PR, signed-commit, and CI policy.
|
||||
|
||||
## 9. Communication templates
|
||||
|
||||
### Unsupported format
|
||||
|
||||
> OBLITERATUS identified this as `{format}`, which is `{status}` in support
|
||||
> matrix v1. No source or prior output was changed. The blocking contract is
|
||||
> `{code}` at `{phase}`. Continue with `{sanitized next action}`.
|
||||
|
||||
### Evidence pending
|
||||
|
||||
> The upstream framework documents this capability, but OBLITERATUS has no
|
||||
> exact-version/model/topology qualification at the candidate commit. The row
|
||||
> remains deferred until the required evidence passes.
|
||||
|
||||
### Conversion versus runtime
|
||||
|
||||
> Converting rank-sharded model weights into HF safetensors is an offline input
|
||||
> step. It does not make the OBLITERATUS surgery process multi-node; that runtime
|
||||
> remains a separate, unsupported capability.
|
||||
|
||||
## 10. Post-incident activities
|
||||
|
||||
- Preserve descriptor/manifest and test result digests.
|
||||
- Record exact commit, versions, topology, resource admission/actuals, failure
|
||||
code/phase, source-immutability result, and output-promotion result.
|
||||
- Add a minimal project-owned regression fixture.
|
||||
- Update risk, traceability, support matrix, and runbook if the contract changed.
|
||||
- Never turn one successful case into an unqualified universal support claim.
|
||||
|
||||
## 11. Runbook maintenance
|
||||
|
||||
Review with each adapter version-band change, schema version, release candidate,
|
||||
and incident. Security owns trust-boundary text; architecture owns terminology;
|
||||
test/release own evidence status; support owns scenario clarity. Offline checks
|
||||
must continue validating local links, CLI examples, and support-matrix contracts.
|
||||
Reference in New Issue
Block a user