mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-09-21 17:00:50 +02:00
feat: add safe distributed checkpoint intake and preflight
This commit is contained in:
Joseph Magly
@@ -0,0 +1,189 @@
|
||||
# Checkpoint formats, placement, and safe distributed-checkpoint inspection
|
||||
|
||||
This guide separates capabilities that are often called “sharding” but have
|
||||
different contracts. The machine-readable source of truth is
|
||||
[support-matrix-v1.json](support-matrix-v1.json). The current implementation provides bounded,
|
||||
non-executing structural inspection plus producer-neutral validation and writing.
|
||||
Producer readers, adapters, exact trusted-reader profiles, trusted payload
|
||||
execution, and live multi-node model surgery remain unavailable.
|
||||
|
||||
## Short answer
|
||||
|
||||
Current OBLITERATUS can load ordinary Hugging Face-compatible checkpoints and
|
||||
can use Accelerate to place complete modules across devices visible to one
|
||||
process on one host. It does not currently reconstruct PyTorch DCP/FSDP,
|
||||
Megatron, or DeepSpeed rank fragments, and it does not run one surgery job across
|
||||
multiple hosts. It can now classify the inert structure of those checkpoint
|
||||
directories without invoking their framework readers.
|
||||
|
||||
No producer-backed conversion is current. The common writer accepts only
|
||||
already-normalized, validated fragments through its Python API; it does not read
|
||||
a DCP, Megatron, or DeepSpeed payload. Future explicit offline conversion of one
|
||||
narrowly qualified producer/model/version case remains Wave 3/4 work. Such a
|
||||
conversion would not be exact training resume and would not make surgery
|
||||
multi-node.
|
||||
|
||||
## Terms that must remain separate
|
||||
|
||||
| Term | Meaning | Current OBLITERATUS relevance |
|
||||
|---|---|---|
|
||||
| HF file shard | Complete named tensors distributed across files; index maps tensor name to file | Current loader can consume compatible inputs |
|
||||
| safetensors byte offset | Half-open byte range inside one safetensors data buffer | File-format metadata, not rank placement |
|
||||
| Accelerate `device_map` | Complete modules placed/offloaded by one process | Current, model-family dependent |
|
||||
| CPU/disk offload | Process-local backing for complete model parameters | Current, not a distributed checkpoint |
|
||||
| DCP/FSDP state | Framework-defined distributed state/chunks | Current structural detection only; no reader or adapter |
|
||||
| Megatron offset | Logical tensor element placement and axis/chunk metadata | Current structural detection only; model-aware mapping remains required |
|
||||
| DeepSpeed ZeRO/Universal | Partitioned or topology-neutral DeepSpeed state | Current structural detection only; trusted reader/resource gates remain |
|
||||
| Live multi-node execution | Multiple processes/hosts with rendezvous, collectives, ownership, failure coordination, and save | Not supported; separate research decision |
|
||||
|
||||
The terms in this document and the machine-readable
|
||||
[schemas](schemas/support-matrix-v1.schema.json) are normative for this feature. Upstream semantics and
|
||||
qualifications cite the public
|
||||
[source register](references.md),
|
||||
including PyTorch DCP/FSDP [R01–R03], Megatron/Bridge [R04–R07], DeepSpeed
|
||||
[R08–R10], HF/safetensors [R11–R13], Accelerate placement/launch [R14–R15],
|
||||
and serialization/containment guidance [R27–R35].
|
||||
|
||||
## Current support matrix
|
||||
|
||||
| Subject | Load/surgery | Structural inspect | Convert to HF | Exact resume | Live multi-node |
|
||||
|---|---|---|---|---|---|
|
||||
| Existing compatible HF safetensors | Conditional on model/runtime gates | Conditional, header-only | Already canonical | Out of scope | Out of scope |
|
||||
| Accelerate `device_map`/offload | Conditional, one process/host | Not applicable | Not applicable | Out of scope | Out of scope |
|
||||
| PyTorch DCP/FSDP | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
|
||||
| Megatron distributed state | No payload load | Conditional structural classification | Deferred, model-aware | Out of scope | Out of scope |
|
||||
| DeepSpeed ZeRO/Universal | No payload load | Conditional structural classification | Deferred | Out of scope | Out of scope |
|
||||
| PEFT LoRA safetensors | Conditional exact-base Python export | Conditional, header/JSON only | Not a rank-fragment conversion | Out of scope | Out of scope |
|
||||
| Live multi-node surgery | Preflight only; no model payload | Not applicable | Separate offline concern | Out of scope | Deferred and unqualified |
|
||||
|
||||
“Conditional” means the behavior depends on an exact model architecture,
|
||||
runtime, kernels, dtype/quantization, memory, and quality gates. It is not a
|
||||
universal compatibility claim. “Deferred” means planned and unimplemented.
|
||||
|
||||
## Current safe structural inspection
|
||||
|
||||
Run the inspector against one local file or directory:
|
||||
|
||||
```bash
|
||||
obliteratus checkpoint inspect ./checkpoint --json
|
||||
```
|
||||
|
||||
The command inventories regular files without following links, streams digests,
|
||||
parses bounded JSON and safetensors headers, and emits a strict descriptor with
|
||||
classification evidence, confidence, resource estimates, and stable blockers.
|
||||
It does not read tensor payloads, import producer frameworks, unpickle metadata,
|
||||
initialize a process group, discover plugins, execute remote code, or access the
|
||||
network. Treat a `conditional` classification as structural evidence only—not a
|
||||
promise that the checkpoint can be loaded or converted.
|
||||
|
||||
Legacy HF `.bin`/`.pt` files and DCP `.metadata` may be recognized by safe names
|
||||
and companion structure, but remain opaque and trust-gated. Ambiguous layouts,
|
||||
links, non-regular files, races, malformed bounded metadata, and resource-limit
|
||||
violations fail closed. Inspection does not mutate the source or create output.
|
||||
|
||||
The product capability registry is deliberately empty until one producer,
|
||||
version, model family, and adapter is separately selected and qualified. The
|
||||
safe plane nevertheless implements the closed-registry dependency diagnostic:
|
||||
an explicitly supplied exact capability can be identified as a format candidate,
|
||||
installed distribution metadata is checked without importing the distribution,
|
||||
and a missing or mismatched dependency reports the exact OBLITERATUS extra,
|
||||
project version, required distribution versions, and sanitized observed
|
||||
versions. A candidate becomes an exact match only when independently observed
|
||||
producer and version evidence also agree. Multiple matching capabilities fail
|
||||
closed. This metadata-only resolution does not install anything, authorize
|
||||
trust, invoke a reader, or make the capability a support claim.
|
||||
|
||||
## Existing single-host placement
|
||||
|
||||
For compatible model families, `device_map="auto"` may place complete modules
|
||||
across CUDA devices visible to the current process and may use CPU/disk offload.
|
||||
This is a capacity mechanism, not saved checkpoint topology and not rank-based
|
||||
pipeline parallelism. Upstream Accelerate documents the placement/offload model
|
||||
[R14](references.md#primary-and-upstream-sources); the project boundary is
|
||||
confirmed by local code [R21](references.md#project-evidence).
|
||||
|
||||
Qwen hybrid models retain the complete-text-model-on-one-device restriction.
|
||||
Generic multi-device layer placement is rejected for those paths because the
|
||||
restriction is a correctness gate, not an unimplemented distributed-checkpoint
|
||||
feature.
|
||||
|
||||
`--remote` selects one SSH host and launches one OBLITERATUS process there. It
|
||||
does not coordinate ranks across hosts.
|
||||
|
||||
## Gated intake workflow
|
||||
|
||||
Only step 1 and the producer-neutral portions of steps 4–5 are current. The
|
||||
remaining actions require separate design, security review, and qualification;
|
||||
there is no producer-conversion CLI:
|
||||
|
||||
1. **Inspect:** bounded local inventory plus JSON/safetensors-header parsing;
|
||||
classify format, components, producer evidence, topology facts, state scope,
|
||||
trust requirement, resources, adapter match, and blockers.
|
||||
2. **Escalate only when reviewed:** some vendor metadata requires a trusted
|
||||
reader. Default inspection stops and reports that requirement. Any future
|
||||
policy must require fresh per-invocation intent plus a strict single-use
|
||||
source/operation/runtime/isolation/resource-bound record and an exact approved
|
||||
disposable worker profile. A checksum, familiar local filename, prior scan,
|
||||
or `weights_only=True` is not trust. This remains unimplemented and
|
||||
unauthorized.
|
||||
3. **Adapt:** one exact producer/version/model adapter emits neutral tensor
|
||||
fragments. Megatron requires a supported Bridge/provider mapping; offsets
|
||||
alone do not define fused tensor semantics [R04, R07].
|
||||
4. **Validate:** prove shape/range coverage, replicas, padding, ties/shared state,
|
||||
topology, and resource budgets.
|
||||
5. **Materialize:** the current Python writer accepts already-normalized neutral
|
||||
fragments, writes bounded safetensors staging, index, configuration, and
|
||||
conversion manifest, then validates and promotes only on complete success.
|
||||
It is not a producer reader or adapter.
|
||||
6. **Load:** pass the canonical path into the unchanged existing HF loader.
|
||||
|
||||
Version 1 intentionally emits model weights only. Model weights do not include
|
||||
all optimizer, scheduler, RNG/scaler, progress, and data-position state needed
|
||||
for producer-compatible exact resume [R19–R20](references.md#primary-and-upstream-sources).
|
||||
|
||||
## What to provide with an unsupported-checkpoint report
|
||||
|
||||
Provide only sanitized structural evidence:
|
||||
|
||||
- exact model identifier and immutable revision when shareable;
|
||||
- producer framework and exact version;
|
||||
- checkpoint type and normalized relative file tree with sizes and safe SHA-256
|
||||
digests, excluding tensor values and sensitive local identifiers;
|
||||
- saved node/world and TP/PP/DP/CP/EP/ETP/ZeRO topology when known;
|
||||
- exact metadata field/API meant by “offset”;
|
||||
- OBLITERATUS commit, command/config, OS, Python, PyTorch, Transformers,
|
||||
Accelerate, and optional producer versions;
|
||||
- complete normalized error and first failing stage;
|
||||
- desired result: inspect, convert, run surgery, export, infer, or resume.
|
||||
|
||||
Do not open an unfamiliar `.pt`, DCP `.metadata`, or vendor checkpoint merely to
|
||||
collect a report. PyTorch documents serialization trust risks and an upstream
|
||||
DCP issue identifies pickle use in `.metadata`; a 2026 advisory also shows why
|
||||
weights-only loading is not a permanent safe-parser boundary [R17–R18,
|
||||
R27–R29](references.md#security-and-containment-sources).
|
||||
|
||||
## Resource and recovery expectations
|
||||
|
||||
The producer-neutral writer first estimates source/logical/output/temporary
|
||||
bytes and peak RAM. It enforces actual staged output/temporary bytes before
|
||||
promotion; actual peak RAM and temporary-byte measurements remain `null` when
|
||||
the process has not instrumented them, rather than being populated with
|
||||
estimates. A denied or unknown admission does not start
|
||||
materialization. Source artifacts remain immutable. Output is written into
|
||||
sibling staging, validated, and promoted only when complete. Failure or
|
||||
cancellation does not replace a prior valid output. The existing full-model
|
||||
REBIRTH scaling bottleneck remains; the common writer API is not a practical
|
||||
large-model conversion claim.
|
||||
|
||||
DeepSpeed warns that fp32 consolidation can require substantial CPU memory
|
||||
[R08](references.md#primary-and-upstream-sources).
|
||||
No general memory multiplier or GPU-count promise is made without exact retained
|
||||
evidence.
|
||||
|
||||
## Support and escalation
|
||||
|
||||
Use the [support runbook](support-runbook.md) for current triage, evidence
|
||||
collection, recovery, and escalation. A future `supported` matrix row requires
|
||||
exact producer/adapter versions, fixture digest, candidate commit, environment,
|
||||
topology, retained result, and known limits. The offline contract validator is
|
||||
current; no distributed producer row is promoted to `supported` by this work.
|
||||
@@ -0,0 +1,76 @@
|
||||
# Distributed checkpoint intake source register
|
||||
|
||||
**Artifact ID:** RESEARCH-DCI-001
|
||||
**Version:** 0.3.0
|
||||
**Status:** Reviewed technical evidence
|
||||
**Accessed:** 2026-09-02
|
||||
**Repository baseline:** `5cc43c6e52903497574d80e08dff856028bc47f7`
|
||||
|
||||
This register is the citation authority for the distributed-checkpoint feature
|
||||
documentation. It records what each source supports and the limits on how the source may
|
||||
be used. Vendor documentation establishes upstream behavior; it does not by
|
||||
itself prove OBLITERATUS compatibility. Project support claims require exact-head
|
||||
tests, immutable fixtures, and retained evidence at an exact candidate commit.
|
||||
|
||||
## Primary and upstream sources
|
||||
|
||||
| ID | Source | Evidence class | Supported use | Required qualification |
|
||||
|---|---|---|---|---|
|
||||
| R01 | [PyTorch Distributed Checkpoint API](https://docs.pytorch.org/docs/stable/distributed.checkpoint.html) | Primary project documentation; moderate confidence | DCP save/load planning, multi-rank storage, load-time resharding, preallocated state | DCP documents no general saved-state backward-compatibility guarantee; version-gate adapters |
|
||||
| R02 | [PyTorch DCP recipe](https://docs.pytorch.org/tutorials/recipes/distributed_checkpoint_recipe.html) | Primary project tutorial; moderate confidence | Multi-rank examples and topology-change behavior | Tutorial behavior is illustrative, not a universal format contract |
|
||||
| R03 | [PyTorch FSDP API](https://docs.pytorch.org/docs/stable/fsdp.html) | Primary project documentation; moderate confidence | Full/local/sharded state-dict distinctions and rank-zero CPU-offload semantics | FSDP state mode is not synonymous with DCP serialization |
|
||||
| R04 | [Megatron Core sharded-tensor mapping](https://docs.nvidia.com/megatron-core/developer-guide/latest/apidocs/core/core.dist_checkpointing.mapping.html) | Primary vendor API documentation; moderate confidence | `global_offset`, `rank_offsets`, replica identity, element-coordinate semantics | Offsets establish placement, not model-family semantic mapping |
|
||||
| R05 | [Megatron Core distributed checkpointing](https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/core/dist_checkpointing.html) | Primary vendor documentation; moderate confidence | Model-weight resharding across supported topology changes | Optimizer resharding is format/version-dependent and must be separately qualified |
|
||||
| R06 | [Megatron Core parallelism guide](https://docs.nvidia.com/megatron-core/developer-guide/latest/user-guide/parallelism-guide.html) | Primary vendor documentation; moderate confidence | TP, PP, DP, CP, EP, ETP, FSDP terminology | Axis products and parameter sharding behavior must follow the producer contract, not inference |
|
||||
| R07 | [Megatron Bridge AutoBridge](https://docs.nvidia.com/nemo/megatron-bridge/latest/apidocs/bridge/bridge.models.conversion.auto_bridge.html) | Primary vendor API documentation; moderate confidence | Model-aware bidirectional conversion and provider mapping | Bridge availability does not imply every model family or checkpoint version is supported |
|
||||
| R08 | [DeepSpeed model checkpointing](https://deepspeed.readthedocs.io/en/stable/model-checkpointing.html) | Primary project documentation; moderate confidence | ZeRO-2/3 fp32 consolidation, CPU-memory warning, `safe_serialization` output option | Input consolidation uses framework serialization and belongs behind the explicit trust gate |
|
||||
| R09 | [DeepSpeed Universal Checkpointing](https://www.deepspeed.ai/tutorials/universal-checkpointing/) | Primary project tutorial; moderate confidence | Topology-neutral DeepSpeed model/optimizer representation for compatible mappings | Universal Checkpointing is neither HF safetensors nor a generic architecture converter |
|
||||
| R10 | [DeepSpeed `zero_to_fp32.py`](https://github.com/deepspeedai/DeepSpeed/blob/master/deepspeed/utils/zero_to_fp32.py) | Upstream implementation; moderate confidence, version-volatile | Confirms current consolidation path and `weights_only=False` input loading | Pin the exact source revision used by an adapter qualification |
|
||||
| R11 | [safetensors metadata parsing](https://huggingface.co/docs/safetensors/metadata_parsing) | Primary project documentation; moderate confidence | Header dtype, shape, and half-open `data_offsets` relative to the tensor data buffer | Safe parsing does not establish authenticity, path safety, or provenance |
|
||||
| R12 | [Hugging Face serialization helpers](https://huggingface.co/docs/huggingface_hub/en/package_reference/serialization) | Primary project documentation; moderate confidence | Sharded safetensors writing, indexes, tied/shared-tensor handling | Atomic publication is an OBLITERATUS responsibility, not an upstream guarantee |
|
||||
| R13 | [Transformers model loading and sharded checkpoints](https://huggingface.co/docs/transformers/main/models) | Primary project documentation; moderate confidence | Named tensors distributed across files and index-based loading | A file-shard index contains no rank-fragment coordinate contract |
|
||||
| R14 | [Accelerate big-model inference](https://huggingface.co/docs/accelerate/main/en/concept_guides/big_model_inference) | Primary project documentation; moderate confidence | Single-process module placement and CPU/disk offload via `device_map` | `device_map` is not checkpoint topology or a multi-node launcher |
|
||||
| R15 | [Accelerate multi-node launch](https://huggingface.co/docs/accelerate/main/en/basic_tutorials/launch) | Primary project documentation; moderate confidence | Per-node machine rank, common rendezvous, launcher invocation on every node | Launch documentation does not prove OBLITERATUS has a distributed runtime |
|
||||
| R16 | [PEFT checkpoint format](https://huggingface.co/docs/peft/main/developer_guides/checkpoint) | Primary project documentation; moderate confidence | Standard adapter files, adapter-only state, dependency on a base model | Immutable base revision/digest is an OBLITERATUS provenance rule and may be absent upstream |
|
||||
| R17 | [PyTorch serialization notes](https://docs.pytorch.org/docs/main/notes/serialization.html) and [`torch.load`](https://docs.pytorch.org/docs/stable/generated/torch.load.html) | Primary project documentation; moderate confidence | Serialization trust warning and `weights_only` behavior | `weights_only` narrows risk but does not turn arbitrary input into verified data |
|
||||
| R18 | [PyTorch issue: DCP `.metadata` uses pickle](https://github.com/pytorch/pytorch/issues/189308) | Upstream issue and source-linked observation; low-to-moderate confidence | Establishes a concrete reason default inspection must not treat `.metadata` as inert | Recheck against the exact PyTorch version before implementing a trusted reader |
|
||||
| R19 | [Transformers Trainer resume recipes](https://huggingface.co/docs/transformers/main/trainer_recipes) | Primary project documentation; moderate confidence | Resume includes more than model weights, such as optimizer/scheduler/RNG state | Exact state varies by trainer/framework/version |
|
||||
| R20 | [Accelerate training migration](https://huggingface.co/docs/accelerate/basic_tutorials/migration) | Primary project documentation; moderate confidence | Accelerator state can include model, optimizer, scheduler, RNG, and data position | This does not define a portable cross-framework resume format |
|
||||
|
||||
## Security and containment sources
|
||||
|
||||
| ID | Source | Evidence class | Supported use | Required qualification |
|
||||
|---|---|---|---|---|
|
||||
| R27 | [Python `pickle` documentation](https://docs.python.org/3/library/pickle.html) | Primary language documentation; high confidence | Establishes that malicious pickle can execute arbitrary code and untrusted/tampered data must not be unpickled | A signature or digest establishes integrity only under a separately trusted provenance/key decision; it does not make arbitrary objects semantically safe |
|
||||
| R28 | [PyTorch security policy](https://github.com/pytorch/pytorch/blob/main/SECURITY.md) | Primary project security guidance; high confidence | Treat untrusted models as programs, prefer isolated execution, validate even safer formats, and do not expose distributed primitives to untrusted networks | Security guidance is not proof that a container or any individual loader/profile is safe; qualify the exact runtime and isolation |
|
||||
| R29 | [PyTorch advisory GHSA-63cw-57p8-fm3p / CVE-2026-24747](https://github.com/pytorch/pytorch/security/advisories/GHSA-63cw-57p8-fm3p) | Primary project advisory; high confidence | Demonstrates code-execution risk in affected `weights_only=True` loading and supports rejecting it as a permanent safe-plane boundary | A patched version fixes the named defect only; future/parser/resource risks and trust requirements remain |
|
||||
| R30 | [NIST SP 800-190, Application Container Security Guide](https://csrc.nist.gov/pubs/sp/800/190/final) | Primary government security guidance; high confidence | Container-specific threat/mitigation context and the need to secure images, registries, orchestrators, hosts, and runtime configuration | Published in 2017; apply principles to the exact current runtime and do not equate containers with complete sandboxing |
|
||||
| R31 | [Linux kernel `no_new_privs` documentation](https://docs.kernel.org/userspace-api/no_new_privs.html) | Primary kernel documentation; high confidence | Prevent privilege gains through `execve` and support unprivileged seccomp-filter use | The flag does not prevent all privilege changes or provide filesystem/network/resource isolation by itself |
|
||||
| R32 | [Linux kernel seccomp-filter documentation](https://docs.kernel.org/userspace-api/seccomp_filter.html) | Primary kernel documentation; high confidence | Reduce the syscall surface of a constrained worker and layer filters after `no_new_privs` | Syscall filtering is one containment layer, not a semantic validator or full sandbox |
|
||||
| R33 | [Linux kernel cgroup v2 documentation](https://docs.kernel.org/admin-guide/cgroup-v2.html) | Primary kernel documentation; high confidence | Bound and observe worker memory/CPU/process resource use | Controller availability/configuration and kernel behavior must be preflighted and recorded on the exact host/profile |
|
||||
| R34 | [OWASP Deserialization Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Deserialization_Cheat_Sheet.html) | Primary security-community guidance; moderate-to-high confidence | Prefer pure data formats, avoid native deserialization for untrusted data, and validate before object construction | General guidance; project controls must follow the Python/PyTorch and exact adapter/runtime behavior |
|
||||
| R35 | [Linux `openat2(2)` manual](https://man7.org/linux/man-pages/man2/openat2.2.html) | Authoritative Linux interface documentation; high confidence | Root-relative path resolution with `RESOLVE_BENEATH`, `RESOLVE_NO_SYMLINKS`, and `RESOLVE_NO_MAGICLINKS` for untrusted paths | Linux-specific and kernel-version-dependent; other platforms need reviewed equivalent semantics or must refuse trusted-reader use |
|
||||
|
||||
## Project evidence
|
||||
|
||||
| ID | Source | Supported use | Limitation |
|
||||
|---|---|---|---|
|
||||
| R21 | OBLITERATUS `origin/main` at `5cc43c6e52903497574d80e08dff856028bc47f7`; see `obliteratus/models/loader.py`, `obliteratus/models/offload_surgery.py`, `obliteratus/abliterate.py`, `obliteratus/persistence_contracts.py`, `obliteratus/remote.py`, and `README.md` | Establishes current local loader, process-local placement/offload, complete-state export, atomic helper, one-host remote runner, and documentation boundary | Line references must be refreshed when implementation changes |
|
||||
|
||||
## Claim rules
|
||||
|
||||
1. Use “documents,” “defines,” or “currently implements” for vendor behavior;
|
||||
do not convert vendor documentation into an OBLITERATUS support claim.
|
||||
2. Mark project interpretations explicitly, especially the distinction between
|
||||
`device_map` and checkpoint-rank topology.
|
||||
3. A `supported` matrix row requires an exact producer version, adapter version,
|
||||
immutable fixture digest, candidate commit, environment, topology, and retained
|
||||
result at the exact candidate commit.
|
||||
4. Archived/versioned documentation remains historical evidence only. Current
|
||||
contracts use the latest cited primary documentation plus exact-version source.
|
||||
5. Security claims remain bounded: subprocess isolation and resource limits are
|
||||
containment controls, not proof that vendor deserialization is safe.
|
||||
6. `weights_only=True`, a checksum, and a recognized local filename are never
|
||||
represented as sufficient trust or authenticity controls [R27–R29].
|
||||
7. OS controls are cited as exact-profile containment mechanisms, not a portable
|
||||
universal sandbox claim [R30–R35].
|
||||
@@ -0,0 +1,148 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/adapter-capability-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint Adapter Capability v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_id",
|
||||
"schema_version",
|
||||
"adapter_id",
|
||||
"adapter_version",
|
||||
"contract_version",
|
||||
"supported_producer_versions",
|
||||
"tested_producer_versions",
|
||||
"required_extras",
|
||||
"formats",
|
||||
"model_families",
|
||||
"state_scopes",
|
||||
"topology_capabilities",
|
||||
"quantization_capabilities",
|
||||
"safe_inspection",
|
||||
"trusted_inspection",
|
||||
"conversion",
|
||||
"evidence"
|
||||
],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.adapter-capability"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"adapter_id": {"$ref": "#/$defs/id"},
|
||||
"adapter_version": {"$ref": "#/$defs/version"},
|
||||
"contract_version": {"const": "1.0.0"},
|
||||
"supported_producer_versions": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {"$ref": "#/$defs/versionBand"}
|
||||
},
|
||||
"tested_producer_versions": {
|
||||
"type": "array",
|
||||
"items": {"$ref": "#/$defs/testedVersion"}
|
||||
},
|
||||
"required_extras": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/id"}
|
||||
},
|
||||
"formats": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {
|
||||
"enum": [
|
||||
"hf_safetensors",
|
||||
"pytorch_dcp",
|
||||
"fsdp_state_dict",
|
||||
"megatron_torch_dist",
|
||||
"megatron_torch_dcp",
|
||||
"megatron_fsdp_dtensor",
|
||||
"deepspeed_zero",
|
||||
"deepspeed_universal",
|
||||
"peft_safetensors"
|
||||
]
|
||||
}
|
||||
},
|
||||
"model_families": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/id"}
|
||||
},
|
||||
"state_scopes": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/id"}
|
||||
},
|
||||
"topology_capabilities": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["axes", "saved_to_canonical", "saved_to_changed_topology"],
|
||||
"properties": {
|
||||
"axes": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"enum": ["nodes", "world", "tp", "pp", "dp", "cp", "ep", "etp", "fsdp", "zero"]}
|
||||
},
|
||||
"saved_to_canonical": {"type": "boolean"},
|
||||
"saved_to_changed_topology": {"type": "boolean"}
|
||||
}
|
||||
},
|
||||
"quantization_capabilities": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["format", "status", "evidence_refs"],
|
||||
"properties": {
|
||||
"format": {"$ref": "#/$defs/id"},
|
||||
"status": {"enum": ["supported", "refused", "deferred"]},
|
||||
"evidence_refs": {"type": "array", "items": {"$ref": "#/$defs/id"}}
|
||||
}
|
||||
}
|
||||
},
|
||||
"safe_inspection": {"type": "boolean"},
|
||||
"trusted_inspection": {"type": "boolean"},
|
||||
"conversion": {"type": "boolean"},
|
||||
"evidence": {
|
||||
"type": "array",
|
||||
"items": {"$ref": "#/$defs/evidence"}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"version": {"type": "string", "minLength": 1, "maxLength": 128},
|
||||
"versionBand": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["producer", "specifier", "rationale"],
|
||||
"properties": {
|
||||
"producer": {"$ref": "#/$defs/id"},
|
||||
"specifier": {"$ref": "#/$defs/version"},
|
||||
"rationale": {"type": "string", "minLength": 1, "maxLength": 2048}
|
||||
}
|
||||
},
|
||||
"testedVersion": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["producer", "version", "fixture_digest", "candidate_commit", "environment", "topology"],
|
||||
"properties": {
|
||||
"producer": {"$ref": "#/$defs/id"},
|
||||
"version": {"$ref": "#/$defs/version"},
|
||||
"fixture_digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"candidate_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||
"environment": {"$ref": "#/$defs/id"},
|
||||
"topology": {"$ref": "#/$defs/id"}
|
||||
}
|
||||
},
|
||||
"evidence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "reference", "status"],
|
||||
"properties": {
|
||||
"kind": {"enum": ["primary_documentation", "upstream_source", "fixture", "test_result", "review"]},
|
||||
"reference": {"$ref": "#/$defs/id"},
|
||||
"status": {"enum": ["required", "present", "expired", "missing"]}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,145 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/artifact-provenance-v1.schema.json",
|
||||
"title": "OBLITERATUS Artifact Provenance v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_id",
|
||||
"schema_version",
|
||||
"artifact_id",
|
||||
"record_digest",
|
||||
"sources",
|
||||
"converter",
|
||||
"obliteratus_commit",
|
||||
"configuration_digest",
|
||||
"tokenizer",
|
||||
"base_model",
|
||||
"command",
|
||||
"environment",
|
||||
"source_topology",
|
||||
"lineage",
|
||||
"input_digests",
|
||||
"output_digests",
|
||||
"transformations",
|
||||
"state",
|
||||
"adapter",
|
||||
"dataset",
|
||||
"training",
|
||||
"unknowns"
|
||||
],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.artifact-provenance"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"artifact_id": {"type": "string", "pattern": "^artifact-sha256:[0-9a-f]{64}$"},
|
||||
"record_digest": {"$ref": "#/$defs/digest"},
|
||||
"sources": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/artifactIdentity"}},
|
||||
"converter": {"$ref": "#/$defs/toolIdentity"},
|
||||
"obliteratus_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||
"configuration_digest": {"anyOf": [{"$ref": "#/$defs/digest"}, {"type": "null"}]},
|
||||
"tokenizer": {"anyOf": [{"$ref": "#/$defs/artifactIdentity"}, {"type": "null"}]},
|
||||
"base_model": {"anyOf": [{"$ref": "#/$defs/artifactIdentity"}, {"type": "null"}]},
|
||||
"command": {"type": "array", "maxItems": 256, "items": {"type": "string", "maxLength": 4096}},
|
||||
"environment": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["packages", "platform", "python"],
|
||||
"properties": {
|
||||
"python": {"type": ["string", "null"], "maxLength": 128},
|
||||
"platform": {"type": ["string", "null"], "maxLength": 256},
|
||||
"packages": {"type": "object", "additionalProperties": {"type": "string", "maxLength": 128}}
|
||||
}
|
||||
},
|
||||
"source_topology": {"type": "object"},
|
||||
"lineage": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/lineageEvent"}},
|
||||
"input_digests": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/digest"}},
|
||||
"output_digests": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/digest"}},
|
||||
"transformations": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/id"}},
|
||||
"state": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["classification", "observed_scopes", "lost_state"],
|
||||
"properties": {
|
||||
"classification": {"enum": ["weights_only", "model_and_optimizer", "exact_resume", "unknown"]},
|
||||
"observed_scopes": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/id"}},
|
||||
"lost_state": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/id"}}
|
||||
}
|
||||
},
|
||||
"adapter": {"anyOf": [{"$ref": "#/$defs/adapterIdentity"}, {"type": "null"}]},
|
||||
"dataset": {"anyOf": [{"$ref": "#/$defs/datasetIdentity"}, {"type": "null"}]},
|
||||
"training": {"anyOf": [{"$ref": "#/$defs/trainingIdentity"}, {"type": "null"}]},
|
||||
"unknowns": {"type": "array", "uniqueItems": true, "items": {"type": "string", "minLength": 1, "maxLength": 1024}}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"artifactIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "identity", "revision", "digest"],
|
||||
"properties": {
|
||||
"kind": {"enum": ["local", "hub", "generated"]},
|
||||
"identity": {"$ref": "#/$defs/id"},
|
||||
"revision": {"type": ["string", "null"], "maxLength": 256},
|
||||
"digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
},
|
||||
"toolIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "version", "commit"],
|
||||
"properties": {
|
||||
"name": {"$ref": "#/$defs/id"},
|
||||
"version": {"$ref": "#/$defs/id"},
|
||||
"commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"}
|
||||
}
|
||||
},
|
||||
"lineageEvent": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["event_id", "event_type", "parent_artifact_ids", "tool", "transformations"],
|
||||
"properties": {
|
||||
"event_id": {"$ref": "#/$defs/id"},
|
||||
"event_type": {"enum": ["consolidation", "reshard", "pretrain", "full_finetune", "adapter_train", "adapter_merge", "quantization", "dequantization", "surgery"]},
|
||||
"parent_artifact_ids": {"type": "array", "uniqueItems": true, "items": {"type": "string", "pattern": "^artifact-sha256:[0-9a-f]{64}$"}},
|
||||
"tool": {"$ref": "#/$defs/id"},
|
||||
"transformations": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/id"}}
|
||||
}
|
||||
},
|
||||
"adapterIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["adapter_type", "base_model", "config_digest", "key_map_digest"],
|
||||
"properties": {
|
||||
"adapter_type": {"$ref": "#/$defs/id"},
|
||||
"base_model": {"$ref": "#/$defs/artifactIdentity"},
|
||||
"config_digest": {"$ref": "#/$defs/digest"},
|
||||
"key_map_digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
},
|
||||
"datasetIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["identifier", "revision", "digest", "split", "subset", "record_count"],
|
||||
"properties": {
|
||||
"identifier": {"$ref": "#/$defs/id"},
|
||||
"revision": {"type": ["string", "null"], "maxLength": 256},
|
||||
"digest": {"$ref": "#/$defs/digest"},
|
||||
"split": {"type": ["string", "null"], "maxLength": 256},
|
||||
"subset": {"type": ["string", "null"], "maxLength": 256},
|
||||
"record_count": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807}
|
||||
}
|
||||
},
|
||||
"trainingIdentity": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["method", "framework", "framework_version", "hyperparameters_digest"],
|
||||
"properties": {
|
||||
"method": {"enum": ["pretrain", "full_finetune", "adapter_train", "unknown"]},
|
||||
"framework": {"type": ["string", "null"], "maxLength": 256},
|
||||
"framework_version": {"type": ["string", "null"], "maxLength": 128},
|
||||
"hyperparameters_digest": {"anyOf": [{"$ref": "#/$defs/digest"}, {"type": "null"}]}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,541 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/checkpoint-descriptor-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint Descriptor v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_id",
|
||||
"schema_version",
|
||||
"descriptor_id",
|
||||
"primary_format",
|
||||
"classification_confidence",
|
||||
"components",
|
||||
"producer",
|
||||
"evidence",
|
||||
"source_inventory",
|
||||
"safety",
|
||||
"state",
|
||||
"topologies",
|
||||
"tensor_fragments",
|
||||
"adapter_resolution",
|
||||
"conversion_plan",
|
||||
"resource_estimate",
|
||||
"support_decision",
|
||||
"blockers"
|
||||
],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.checkpoint-descriptor"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"descriptor_id": {"type": "string", "minLength": 1, "maxLength": 160},
|
||||
"primary_format": {"$ref": "#/$defs/checkpointFormat"},
|
||||
"classification_confidence": {
|
||||
"enum": ["verified", "declared", "inferred", "unknown"]
|
||||
},
|
||||
"components": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"maxItems": 1024,
|
||||
"items": {"$ref": "#/$defs/component"}
|
||||
},
|
||||
"producer": {"$ref": "#/$defs/producer"},
|
||||
"evidence": {
|
||||
"type": "array",
|
||||
"maxItems": 100000,
|
||||
"items": {"$ref": "#/$defs/evidence"}
|
||||
},
|
||||
"source_inventory": {"$ref": "#/$defs/sourceInventory"},
|
||||
"safety": {"$ref": "#/$defs/safety"},
|
||||
"state": {"$ref": "#/$defs/state"},
|
||||
"topologies": {
|
||||
"type": "array",
|
||||
"maxItems": 128,
|
||||
"items": {"$ref": "#/$defs/topology"}
|
||||
},
|
||||
"tensor_fragments": {
|
||||
"type": "array",
|
||||
"maxItems": 10000000,
|
||||
"items": {"$ref": "#/$defs/tensorFragment"}
|
||||
},
|
||||
"adapter_resolution": {"$ref": "#/$defs/adapterResolution"},
|
||||
"conversion_plan": {"$ref": "#/$defs/conversionPlan"},
|
||||
"resource_estimate": {"$ref": "#/$defs/resourceEstimate"},
|
||||
"support_decision": {
|
||||
"enum": [
|
||||
"canonical_hf_ready",
|
||||
"conversion_supported",
|
||||
"trusted_inspection_required",
|
||||
"blocked"
|
||||
]
|
||||
},
|
||||
"blockers": {
|
||||
"type": "array",
|
||||
"maxItems": 10000,
|
||||
"items": {"$ref": "#/$defs/blocker"}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"nonNegativeInt64": {
|
||||
"type": "integer",
|
||||
"minimum": 0,
|
||||
"maximum": 9223372036854775807
|
||||
},
|
||||
"digest": {
|
||||
"type": "string",
|
||||
"pattern": "^sha256:[0-9a-f]{64}$"
|
||||
},
|
||||
"identifier": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 512
|
||||
},
|
||||
"checkpointFormat": {
|
||||
"enum": [
|
||||
"hf_safetensors",
|
||||
"hf_pytorch_pickle",
|
||||
"pytorch_dcp",
|
||||
"fsdp_state_dict",
|
||||
"megatron_torch_dist",
|
||||
"megatron_torch_dcp",
|
||||
"megatron_fsdp_dtensor",
|
||||
"deepspeed_zero",
|
||||
"deepspeed_universal",
|
||||
"peft_safetensors",
|
||||
"unknown",
|
||||
"ambiguous"
|
||||
]
|
||||
},
|
||||
"component": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"component_id",
|
||||
"kind",
|
||||
"format",
|
||||
"producer",
|
||||
"state_scopes",
|
||||
"inventory_refs",
|
||||
"tensor_fragment_refs"
|
||||
],
|
||||
"properties": {
|
||||
"component_id": {"$ref": "#/$defs/identifier"},
|
||||
"kind": {
|
||||
"enum": [
|
||||
"model",
|
||||
"optimizer",
|
||||
"scheduler",
|
||||
"rng",
|
||||
"scaler",
|
||||
"progress",
|
||||
"dataloader",
|
||||
"peft_adapter",
|
||||
"configuration",
|
||||
"tokenizer",
|
||||
"unknown"
|
||||
]
|
||||
},
|
||||
"format": {"$ref": "#/$defs/checkpointFormat"},
|
||||
"producer": {"$ref": "#/$defs/producer"},
|
||||
"state_scopes": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
},
|
||||
"topology_ref": {"type": ["string", "null"], "maxLength": 512},
|
||||
"inventory_refs": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
},
|
||||
"tensor_fragment_refs": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"producer": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["name", "version", "format_version", "evidence_refs"],
|
||||
"properties": {
|
||||
"name": {"type": ["string", "null"], "maxLength": 256},
|
||||
"version": {"type": ["string", "null"], "maxLength": 128},
|
||||
"format_version": {"type": ["string", "null"], "maxLength": 128},
|
||||
"evidence_refs": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"evidence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"evidence_id",
|
||||
"subject",
|
||||
"kind",
|
||||
"file_ref",
|
||||
"location",
|
||||
"confidence",
|
||||
"verifier"
|
||||
],
|
||||
"properties": {
|
||||
"evidence_id": {"$ref": "#/$defs/identifier"},
|
||||
"subject": {"type": "string", "pattern": "^/"},
|
||||
"kind": {
|
||||
"enum": ["explicit_metadata", "header", "json", "filename", "operator", "derived"]
|
||||
},
|
||||
"file_ref": {"type": ["string", "null"], "maxLength": 512},
|
||||
"location": {"type": "string", "maxLength": 1024},
|
||||
"confidence": {"enum": ["verified", "declared", "inferred", "unknown"]},
|
||||
"verifier": {"type": "string", "maxLength": 256}
|
||||
}
|
||||
},
|
||||
"sourceInventory": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["inventory_id", "files", "total_bytes", "observation_complete"],
|
||||
"properties": {
|
||||
"inventory_id": {"$ref": "#/$defs/identifier"},
|
||||
"files": {
|
||||
"type": "array",
|
||||
"maxItems": 1000000,
|
||||
"items": {"$ref": "#/$defs/inventoryFile"}
|
||||
},
|
||||
"total_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"observation_complete": {"type": "boolean"}
|
||||
}
|
||||
},
|
||||
"inventoryFile": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"file_id",
|
||||
"relative_path",
|
||||
"role",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"regular_file",
|
||||
"observation_id"
|
||||
],
|
||||
"properties": {
|
||||
"file_id": {"$ref": "#/$defs/identifier"},
|
||||
"relative_path": {
|
||||
"type": "string",
|
||||
"minLength": 1,
|
||||
"maxLength": 4096,
|
||||
"not": {"pattern": "(^/|(^|/)\\.\\.(/|$))"}
|
||||
},
|
||||
"role": {"type": "string", "minLength": 1, "maxLength": 128},
|
||||
"size_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"sha256": {"$ref": "#/$defs/digest"},
|
||||
"regular_file": {"const": true},
|
||||
"observation_id": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
},
|
||||
"safety": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"inspection_level",
|
||||
"trust_required",
|
||||
"inventory_revalidated",
|
||||
"unsafe_serialization_findings",
|
||||
"violations"
|
||||
],
|
||||
"properties": {
|
||||
"inspection_level": {"enum": ["safe_structure", "trusted_metadata"]},
|
||||
"trust_required": {"type": "boolean"},
|
||||
"inventory_revalidated": {"type": "boolean"},
|
||||
"unsafe_serialization_findings": {
|
||||
"type": "array",
|
||||
"items": {"type": "string", "maxLength": 1024}
|
||||
},
|
||||
"violations": {
|
||||
"type": "array",
|
||||
"items": {"type": "string", "maxLength": 1024}
|
||||
}
|
||||
}
|
||||
},
|
||||
"state": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["observed_scopes", "classification"],
|
||||
"properties": {
|
||||
"observed_scopes": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
},
|
||||
"classification": {
|
||||
"enum": ["weights_only", "model_and_optimizer", "exact_resume", "unknown"]
|
||||
}
|
||||
}
|
||||
},
|
||||
"fact": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["value", "provenance", "evidence_refs"],
|
||||
"properties": {
|
||||
"value": {"anyOf": [{"$ref": "#/$defs/nonNegativeInt64"}, {"type": "null"}]},
|
||||
"provenance": {
|
||||
"enum": ["explicit", "filename_inferred", "operator_supplied", "unknown"]
|
||||
},
|
||||
"evidence_refs": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"topology": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"topology_id",
|
||||
"kind",
|
||||
"node_count",
|
||||
"global_world_size",
|
||||
"machine_rank",
|
||||
"global_rank",
|
||||
"local_rank",
|
||||
"tp_size",
|
||||
"pp_size",
|
||||
"dp_size",
|
||||
"cp_size",
|
||||
"ep_size",
|
||||
"etp_size",
|
||||
"zero_stage"
|
||||
],
|
||||
"properties": {
|
||||
"topology_id": {"$ref": "#/$defs/identifier"},
|
||||
"kind": {"enum": ["saved", "target"]},
|
||||
"node_count": {"$ref": "#/$defs/fact"},
|
||||
"global_world_size": {"$ref": "#/$defs/fact"},
|
||||
"machine_rank": {"$ref": "#/$defs/fact"},
|
||||
"global_rank": {"$ref": "#/$defs/fact"},
|
||||
"local_rank": {"$ref": "#/$defs/fact"},
|
||||
"tp_size": {"$ref": "#/$defs/fact"},
|
||||
"pp_size": {"$ref": "#/$defs/fact"},
|
||||
"dp_size": {"$ref": "#/$defs/fact"},
|
||||
"cp_size": {"$ref": "#/$defs/fact"},
|
||||
"ep_size": {"$ref": "#/$defs/fact"},
|
||||
"etp_size": {"$ref": "#/$defs/fact"},
|
||||
"zero_stage": {"$ref": "#/$defs/fact"}
|
||||
}
|
||||
},
|
||||
"tensorFragment": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"fragment_id",
|
||||
"component_id",
|
||||
"fqn",
|
||||
"role",
|
||||
"dtype",
|
||||
"global_shape",
|
||||
"local_shape",
|
||||
"element_offset",
|
||||
"element_extent",
|
||||
"padding",
|
||||
"shard_file_id",
|
||||
"shard_digest_ref",
|
||||
"replica",
|
||||
"partition_axes",
|
||||
"logical_tensor_id",
|
||||
"storage_locations",
|
||||
"evidence_refs"
|
||||
],
|
||||
"properties": {
|
||||
"fragment_id": {"$ref": "#/$defs/identifier"},
|
||||
"component_id": {"$ref": "#/$defs/identifier"},
|
||||
"fqn": {"$ref": "#/$defs/identifier"},
|
||||
"role": {"enum": ["parameter", "persistent_buffer"]},
|
||||
"dtype": {"$ref": "#/$defs/identifier"},
|
||||
"global_shape": {"$ref": "#/$defs/shape"},
|
||||
"local_shape": {"$ref": "#/$defs/shape"},
|
||||
"element_offset": {"$ref": "#/$defs/shape"},
|
||||
"element_extent": {"$ref": "#/$defs/shape"},
|
||||
"padding": {"$ref": "#/$defs/padding"},
|
||||
"shard_file_id": {"$ref": "#/$defs/identifier"},
|
||||
"shard_digest_ref": {"$ref": "#/$defs/identifier"},
|
||||
"fragment_digest": {"anyOf": [{"$ref": "#/$defs/digest"}, {"type": "null"}]},
|
||||
"replica": {"$ref": "#/$defs/replica"},
|
||||
"partition_axes": {
|
||||
"type": "array",
|
||||
"items": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"uniqueItems": true
|
||||
},
|
||||
"logical_tensor_id": {"$ref": "#/$defs/identifier"},
|
||||
"tie_group_id": {"type": ["string", "null"], "maxLength": 512},
|
||||
"shared_storage_id": {"type": ["string", "null"], "maxLength": 512},
|
||||
"storage_locations": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {"$ref": "#/$defs/storageLocation"}
|
||||
},
|
||||
"evidence_refs": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"shape": {
|
||||
"type": "array",
|
||||
"maxItems": 32,
|
||||
"items": {"$ref": "#/$defs/nonNegativeInt64"}
|
||||
},
|
||||
"padding": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["before", "after", "semantic"],
|
||||
"properties": {
|
||||
"before": {"$ref": "#/$defs/shape"},
|
||||
"after": {"$ref": "#/$defs/shape"},
|
||||
"semantic": {"enum": ["none", "producer_declared", "model_mapping_declared"]}
|
||||
}
|
||||
},
|
||||
"replica": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["group_id", "member_index", "member_count"],
|
||||
"properties": {
|
||||
"group_id": {"type": ["string", "null"], "maxLength": 512},
|
||||
"member_index": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"member_count": {"$ref": "#/$defs/nonNegativeInt64"}
|
||||
}
|
||||
},
|
||||
"storageLocation": {
|
||||
"oneOf": [
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "element_offset", "element_extent"],
|
||||
"properties": {
|
||||
"kind": {"const": "logical_element_range"},
|
||||
"element_offset": {"$ref": "#/$defs/shape"},
|
||||
"element_extent": {"$ref": "#/$defs/shape"}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "global_offset", "rank_offsets"],
|
||||
"properties": {
|
||||
"kind": {"const": "megatron_rank_offsets"},
|
||||
"global_offset": {"$ref": "#/$defs/shape"},
|
||||
"rank_offsets": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "array",
|
||||
"prefixItems": [
|
||||
{"$ref": "#/$defs/nonNegativeInt64"},
|
||||
{"$ref": "#/$defs/nonNegativeInt64"},
|
||||
{"$ref": "#/$defs/nonNegativeInt64"}
|
||||
],
|
||||
"items": false,
|
||||
"minItems": 3,
|
||||
"maxItems": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "begin", "end"],
|
||||
"properties": {
|
||||
"kind": {"const": "safetensors_byte_range"},
|
||||
"begin": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"end": {"$ref": "#/$defs/nonNegativeInt64"}
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["kind", "tensor_name", "file_id"],
|
||||
"properties": {
|
||||
"kind": {"const": "hf_weight_map"},
|
||||
"tensor_name": {"$ref": "#/$defs/identifier"},
|
||||
"file_id": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"adapterResolution": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["status", "adapter_id", "adapter_version", "capability_digest", "reason"],
|
||||
"properties": {
|
||||
"status": {"enum": ["matched", "not_required", "missing", "ambiguous", "unsupported"]},
|
||||
"adapter_id": {"type": ["string", "null"], "maxLength": 256},
|
||||
"adapter_version": {"type": ["string", "null"], "maxLength": 128},
|
||||
"capability_digest": {"anyOf": [{"$ref": "#/$defs/digest"}, {"type": "null"}]},
|
||||
"reason": {"type": "string", "maxLength": 2048}
|
||||
}
|
||||
},
|
||||
"conversionPlan": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["eligible", "target_format", "state_scope", "dropped_scopes"],
|
||||
"properties": {
|
||||
"eligible": {"type": "boolean"},
|
||||
"target_format": {"const": "hf_safetensors"},
|
||||
"state_scope": {"const": "weights_only"},
|
||||
"dropped_scopes": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {"$ref": "#/$defs/identifier"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"resourceEstimate": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"source_bytes",
|
||||
"logical_bytes",
|
||||
"output_bytes",
|
||||
"temporary_bytes",
|
||||
"peak_ram_bytes",
|
||||
"peak_vram_bytes",
|
||||
"file_count",
|
||||
"tensor_count",
|
||||
"shard_count",
|
||||
"assumptions",
|
||||
"confidence",
|
||||
"admission"
|
||||
],
|
||||
"properties": {
|
||||
"source_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"logical_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"output_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"temporary_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"peak_ram_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"peak_vram_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"file_count": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"tensor_count": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"shard_count": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"assumptions": {"type": "array", "items": {"type": "string", "maxLength": 1024}},
|
||||
"confidence": {"enum": ["verified", "declared", "inferred", "unknown"]},
|
||||
"admission": {"enum": ["admitted", "denied", "unknown"]}
|
||||
}
|
||||
},
|
||||
"blocker": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["code", "category", "phase", "affected_refs", "retryable", "next_action"],
|
||||
"properties": {
|
||||
"code": {"type": "string", "pattern": "^DCI_[A-Z0-9_]+$"},
|
||||
"category": {"enum": ["cleanup", "concurrency", "evidence", "output", "protocol", "resource", "runtime", "source", "trust", "unsupported", "validation"]},
|
||||
"phase": {"enum": ["admission", "classification", "cleanup", "evidence", "materialization", "policy", "preflight", "promotion", "protocol", "reader", "source", "validation"]},
|
||||
"affected_refs": {"type": "array", "items": {"$ref": "#/$defs/identifier"}},
|
||||
"retryable": {"type": "boolean"},
|
||||
"next_action": {"type": "string", "maxLength": 2048}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-error-registry",
|
||||
"schema_version": "1.0.0",
|
||||
"entries": [
|
||||
{"code": "DCI_UNSUPPORTED_FORMAT_OR_VERSION", "phase": "classification", "category": "unsupported", "degraded_modes": ["F01"]},
|
||||
{"code": "DCI_TRUST_POLICY_REQUIRED", "phase": "policy", "category": "trust", "degraded_modes": ["F02"]},
|
||||
{"code": "DCI_TRUST_POLICY_MISMATCH", "phase": "policy", "category": "trust", "degraded_modes": ["F02", "F18"]},
|
||||
{"code": "DCI_SOURCE_BOUNDARY_VIOLATION", "phase": "source", "category": "source", "degraded_modes": ["F03"]},
|
||||
{"code": "DCI_SOURCE_CHANGED", "phase": "source", "category": "source", "degraded_modes": ["F04"]},
|
||||
{"code": "DCI_TRUST_RUNTIME_UNAVAILABLE", "phase": "preflight", "category": "runtime", "degraded_modes": ["F05", "F19"]},
|
||||
{"code": "DCI_RUNTIME_IDENTITY_MISMATCH", "phase": "preflight", "category": "runtime", "degraded_modes": ["F06", "F20"]},
|
||||
{"code": "DCI_FORBIDDEN_READER_CAPABILITY", "phase": "reader", "category": "runtime", "degraded_modes": ["F07"]},
|
||||
{"code": "DCI_RESOURCE_LIMIT", "phase": "reader", "category": "resource", "degraded_modes": ["F08", "F09"]},
|
||||
{"code": "DCI_TRUSTED_READER_FAILED", "phase": "reader", "category": "runtime", "degraded_modes": ["F09"]},
|
||||
{"code": "DCI_WORKER_PROTOCOL_INVALID", "phase": "protocol", "category": "protocol", "degraded_modes": ["F10"]},
|
||||
{"code": "DCI_VALIDATION_FAILED", "phase": "validation", "category": "validation", "degraded_modes": ["F11", "F14"]},
|
||||
{"code": "DCI_ADMISSION_DENIED", "phase": "admission", "category": "resource", "degraded_modes": ["F12"]},
|
||||
{"code": "DCI_MATERIALIZE_FAILED", "phase": "materialization", "category": "output", "degraded_modes": ["F13"]},
|
||||
{"code": "DCI_PROMOTION_FAILED", "phase": "promotion", "category": "output", "degraded_modes": ["F13", "F14"]},
|
||||
{"code": "DCI_EVIDENCE_UNAVAILABLE", "phase": "evidence", "category": "evidence", "degraded_modes": ["F15"]},
|
||||
{"code": "DCI_DIAGNOSTIC_REDACTION_FAILED", "phase": "evidence", "category": "evidence", "degraded_modes": ["F16"]},
|
||||
{"code": "DCI_CLEANUP_INCOMPLETE", "phase": "cleanup", "category": "cleanup", "degraded_modes": ["F17"]},
|
||||
{"code": "DCI_CONCURRENT_OPERATION_CONFLICT", "phase": "admission", "category": "concurrency", "degraded_modes": ["F18"]},
|
||||
{"code": "DCI_HOST_TRUST_UNSATISFIED", "phase": "preflight", "category": "trust", "degraded_modes": ["F19"]},
|
||||
{"code": "DCI_SECURITY_BASELINE_REVOKED", "phase": "preflight", "category": "runtime", "degraded_modes": ["F20"]}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/checkpoint-error-registry-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint Error Registry v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_id", "schema_version", "entries"],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.checkpoint-error-registry"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"entries": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["code", "phase", "category", "degraded_modes"],
|
||||
"properties": {
|
||||
"code": {"type": "string", "pattern": "^DCI_[A-Z0-9_]+$"},
|
||||
"phase": {"enum": ["classification", "policy", "source", "preflight", "reader", "protocol", "validation", "admission", "materialization", "promotion", "evidence", "cleanup"]},
|
||||
"category": {"enum": ["unsupported", "trust", "source", "runtime", "resource", "protocol", "validation", "output", "evidence", "cleanup", "concurrency"]},
|
||||
"degraded_modes": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"type": "string", "pattern": "^F(?:0[1-9]|1[0-9]|20)$"}}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/checkpoint-trust-policy-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint Trust Policy v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_id", "schema_version", "policy_id", "operation_level", "requested_state_scope", "adapter", "source", "runtime", "isolation", "resources", "output", "authorization", "evidence"],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.checkpoint-trust-policy"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"policy_id": {"$ref": "#/$defs/id"},
|
||||
"operation_level": {"enum": ["trusted_metadata", "trusted_conversion"]},
|
||||
"requested_state_scope": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/id"}},
|
||||
"adapter": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["capability_id", "adapter_id", "adapter_version", "adapter_digest"],
|
||||
"properties": {
|
||||
"capability_id": {"$ref": "#/$defs/id"},
|
||||
"adapter_id": {"$ref": "#/$defs/id"},
|
||||
"adapter_version": {"$ref": "#/$defs/id"},
|
||||
"adapter_digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
},
|
||||
"source": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["root_id", "inventory_digest", "files"],
|
||||
"properties": {
|
||||
"root_id": {"$ref": "#/$defs/id"},
|
||||
"inventory_digest": {"$ref": "#/$defs/digest"},
|
||||
"files": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"maxItems": 1000000,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["file_id", "relative_path", "size_bytes", "sha256", "observation_id"],
|
||||
"properties": {
|
||||
"file_id": {"$ref": "#/$defs/id"},
|
||||
"relative_path": {"type": "string", "minLength": 1, "maxLength": 4096, "not": {"pattern": "(^/|(^|/)\\.\\.(/|$))"}},
|
||||
"size_bytes": {"$ref": "#/$defs/nonNegativeInt64"},
|
||||
"sha256": {"$ref": "#/$defs/digest"},
|
||||
"observation_id": {"$ref": "#/$defs/id"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"runtime": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["framework", "framework_version", "dependency_identity", "runtime_digest"],
|
||||
"properties": {
|
||||
"framework": {"$ref": "#/$defs/id"},
|
||||
"framework_version": {"$ref": "#/$defs/id"},
|
||||
"dependency_identity": {"$ref": "#/$defs/digest"},
|
||||
"runtime_digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
},
|
||||
"isolation": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["profile_id", "profile_version", "profile_digest", "required_capabilities"],
|
||||
"properties": {
|
||||
"profile_id": {"$ref": "#/$defs/id"},
|
||||
"profile_version": {"$ref": "#/$defs/id"},
|
||||
"profile_digest": {"$ref": "#/$defs/digest"},
|
||||
"required_capabilities": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"uniqueItems": true,
|
||||
"items": {"enum": ["unprivileged", "no_new_privs", "read_only_source", "root_isolation", "private_scratch", "network_denied", "ipc_isolated", "environment_allowlist", "fd_allowlist", "syscall_policy", "resource_limits", "device_denied", "core_dumps_disabled", "bounded_cleanup"]}
|
||||
}
|
||||
}
|
||||
},
|
||||
"resources": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["wall_time_seconds", "cpu_seconds", "memory_bytes", "processes", "threads", "open_files", "input_bytes", "header_bytes", "message_bytes", "scratch_bytes", "output_bytes"],
|
||||
"properties": {
|
||||
"wall_time_seconds": {"$ref": "#/$defs/positiveInt64"},
|
||||
"cpu_seconds": {"$ref": "#/$defs/positiveInt64"},
|
||||
"memory_bytes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"processes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"threads": {"$ref": "#/$defs/positiveInt64"},
|
||||
"open_files": {"$ref": "#/$defs/positiveInt64"},
|
||||
"input_bytes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"header_bytes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"message_bytes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"scratch_bytes": {"$ref": "#/$defs/positiveInt64"},
|
||||
"output_bytes": {"$ref": "#/$defs/nonNegativeInt64"}
|
||||
}
|
||||
},
|
||||
"output": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["root_id", "staging_policy"],
|
||||
"properties": {
|
||||
"root_id": {"type": ["string", "null"], "minLength": 1, "maxLength": 512},
|
||||
"staging_policy": {"enum": ["none", "private_sibling_atomic"]}
|
||||
}
|
||||
},
|
||||
"authorization": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["event_id", "actor_role", "created_at", "expires_at", "single_use_nonce", "single_use"],
|
||||
"properties": {
|
||||
"event_id": {"$ref": "#/$defs/id"},
|
||||
"actor_role": {"$ref": "#/$defs/id"},
|
||||
"created_at": {"type": "string", "format": "date-time"},
|
||||
"expires_at": {"type": "string", "format": "date-time"},
|
||||
"single_use_nonce": {"type": "string", "minLength": 32, "maxLength": 256},
|
||||
"single_use": {"const": true}
|
||||
}
|
||||
},
|
||||
"evidence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["candidate_commit", "policy_digest"],
|
||||
"properties": {
|
||||
"candidate_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||
"policy_digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"nonNegativeInt64": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807},
|
||||
"positiveInt64": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,167 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/conversion-manifest-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint Conversion Manifest v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"schema_id",
|
||||
"schema_version",
|
||||
"manifest_id",
|
||||
"descriptor",
|
||||
"source_inventory_digest",
|
||||
"source_files",
|
||||
"adapter",
|
||||
"source_topology",
|
||||
"state",
|
||||
"dropped_scopes",
|
||||
"canonical_output",
|
||||
"resource_usage",
|
||||
"validation",
|
||||
"provenance",
|
||||
"publication"
|
||||
],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.conversion-manifest"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"manifest_id": {"$ref": "#/$defs/id"},
|
||||
"descriptor": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_version", "digest"],
|
||||
"properties": {
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
},
|
||||
"source_inventory_digest": {"$ref": "#/$defs/digest"},
|
||||
"source_files": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {"$ref": "#/$defs/file"}
|
||||
},
|
||||
"adapter": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["id", "version", "capability_digest", "producer", "producer_version"],
|
||||
"properties": {
|
||||
"id": {"$ref": "#/$defs/id"},
|
||||
"version": {"$ref": "#/$defs/id"},
|
||||
"capability_digest": {"$ref": "#/$defs/digest"},
|
||||
"producer": {"$ref": "#/$defs/id"},
|
||||
"producer_version": {"$ref": "#/$defs/id"}
|
||||
}
|
||||
},
|
||||
"source_topology": {"type": "object"},
|
||||
"state": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["source_classification", "output_classification", "observed_scopes"],
|
||||
"properties": {
|
||||
"source_classification": {"enum": ["weights_only", "model_and_optimizer", "exact_resume", "unknown"]},
|
||||
"output_classification": {"const": "weights_only"},
|
||||
"observed_scopes": {"type": "array", "uniqueItems": true, "items": {"$ref": "#/$defs/id"}}
|
||||
}
|
||||
},
|
||||
"dropped_scopes": {
|
||||
"type": "array",
|
||||
"uniqueItems": true,
|
||||
"items": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["scope", "reason"],
|
||||
"properties": {
|
||||
"scope": {"$ref": "#/$defs/id"},
|
||||
"reason": {"type": "string", "minLength": 1, "maxLength": 2048}
|
||||
}
|
||||
}
|
||||
},
|
||||
"canonical_output": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["format", "dtype_policy", "files", "hf_index", "logical_tensor_count", "logical_bytes"],
|
||||
"properties": {
|
||||
"format": {"const": "hf_safetensors"},
|
||||
"dtype_policy": {"$ref": "#/$defs/id"},
|
||||
"files": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/file"}},
|
||||
"hf_index": {"type": ["string", "null"], "maxLength": 4096},
|
||||
"logical_tensor_count": {"$ref": "#/$defs/int64"},
|
||||
"logical_bytes": {"$ref": "#/$defs/int64"}
|
||||
}
|
||||
},
|
||||
"resource_usage": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["estimated_peak_ram_bytes", "actual_peak_ram_bytes", "estimated_temp_bytes", "actual_temp_bytes"],
|
||||
"properties": {
|
||||
"estimated_peak_ram_bytes": {"$ref": "#/$defs/int64"},
|
||||
"actual_peak_ram_bytes": {"anyOf": [{"$ref": "#/$defs/int64"}, {"type": "null"}]},
|
||||
"estimated_temp_bytes": {"$ref": "#/$defs/int64"},
|
||||
"actual_temp_bytes": {"anyOf": [{"$ref": "#/$defs/int64"}, {"type": "null"}]}
|
||||
}
|
||||
},
|
||||
"validation": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["coverage", "replicas", "ties", "hashes", "index", "safe_reload", "source_unchanged", "result"],
|
||||
"properties": {
|
||||
"coverage": {"type": "boolean"},
|
||||
"replicas": {"type": "boolean"},
|
||||
"ties": {"type": "boolean"},
|
||||
"hashes": {"type": "boolean"},
|
||||
"index": {"type": "boolean"},
|
||||
"safe_reload": {"type": "boolean"},
|
||||
"source_unchanged": {"type": "boolean"},
|
||||
"result": {"enum": ["passed", "failed"]}
|
||||
}
|
||||
},
|
||||
"provenance": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["obliteratus_commit", "configuration_digest", "tokenizer_digest", "base_model", "transformation_log", "unknowns"],
|
||||
"properties": {
|
||||
"obliteratus_commit": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||
"configuration_digest": {"type": ["string", "null"], "pattern": "^(sha256:[0-9a-f]{64})?$"},
|
||||
"tokenizer_digest": {"type": ["string", "null"], "pattern": "^(sha256:[0-9a-f]{64})?$"},
|
||||
"base_model": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["identity", "revision", "digest"],
|
||||
"properties": {
|
||||
"identity": {"type": ["string", "null"], "maxLength": 512},
|
||||
"revision": {"type": ["string", "null"], "maxLength": 256},
|
||||
"digest": {"anyOf": [{"$ref": "#/$defs/digest"}, {"type": "null"}]}
|
||||
}
|
||||
},
|
||||
"transformation_log": {"type": "array", "items": {"type": "string", "maxLength": 2048}},
|
||||
"unknowns": {"type": "array", "items": {"type": "string", "maxLength": 2048}}
|
||||
}
|
||||
},
|
||||
"publication": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["staging_validated", "promoted", "atomic_strategy", "rollback_result"],
|
||||
"properties": {
|
||||
"staging_validated": {"type": "boolean"},
|
||||
"promoted": {"type": "boolean"},
|
||||
"atomic_strategy": {"$ref": "#/$defs/id"},
|
||||
"rollback_result": {"enum": ["not_required", "succeeded", "failed"]}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"int64": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807},
|
||||
"digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"file": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["relative_path", "size_bytes", "sha256"],
|
||||
"properties": {
|
||||
"relative_path": {"type": "string", "minLength": 1, "maxLength": 4096, "not": {"pattern": "(^/|(^|/)\\.\\.(/|$))"}},
|
||||
"size_bytes": {"$ref": "#/$defs/int64"},
|
||||
"sha256": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
{
|
||||
"schema_id": "obliteratus.adapter-capability",
|
||||
"schema_version": "1.0.0",
|
||||
"adapter_id": "invalid-adapter",
|
||||
"adapter_version": "1.0.0",
|
||||
"contract_version": "1.0.0",
|
||||
"supported_producer_versions": [],
|
||||
"tested_producer_versions": [
|
||||
{"producer": "unknown", "version": "latest"}
|
||||
],
|
||||
"required_extras": [],
|
||||
"formats": ["pytorch_dcp"],
|
||||
"model_families": ["any"],
|
||||
"state_scopes": ["exact_resume"],
|
||||
"topology_capabilities": {"axes": [], "saved_to_canonical": true, "saved_to_changed_topology": true},
|
||||
"quantization_capabilities": [],
|
||||
"safe_inspection": true,
|
||||
"trusted_inspection": true,
|
||||
"conversion": true,
|
||||
"evidence": []
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-descriptor",
|
||||
"schema_version": "1.0.0",
|
||||
"descriptor_id": "invalid-generic-offset",
|
||||
"offset": [0, 4]
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-trust-policy",
|
||||
"schema_version": "1.0.0",
|
||||
"environment": {"HOME": "/private/path", "TOKEN": "must-not-persist"}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"schema_id": "obliteratus.trusted-worker-message",
|
||||
"schema_version": "1.0.0",
|
||||
"message_type": "error",
|
||||
"sequence": 1,
|
||||
"source_inventory_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"adapter_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"policy_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"payload": {"code": "DCI_TRUSTED_READER_FAILED", "phase": "reader", "category": "runtime", "subject_ids": [], "limit": null, "actual": null, "raw_message": "private path or secret"}
|
||||
}
|
||||
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"schema_id": "obliteratus.adapter-capability",
|
||||
"schema_version": "1.0.0",
|
||||
"adapter_id": "megatron-bridge-planned",
|
||||
"adapter_version": "0.1.0-planned",
|
||||
"contract_version": "1.0.0",
|
||||
"supported_producer_versions": [
|
||||
{
|
||||
"producer": "megatron-core",
|
||||
"specifier": "unselected",
|
||||
"rationale": "Version band remains blocked on exact fixture and adapter selection."
|
||||
}
|
||||
],
|
||||
"tested_producer_versions": [],
|
||||
"required_extras": ["checkpoint-megatron"],
|
||||
"formats": ["megatron_torch_dist"],
|
||||
"model_families": ["unselected"],
|
||||
"state_scopes": ["model_weights"],
|
||||
"topology_capabilities": {
|
||||
"axes": ["world", "tp", "pp", "dp"],
|
||||
"saved_to_canonical": true,
|
||||
"saved_to_changed_topology": false
|
||||
},
|
||||
"quantization_capabilities": [
|
||||
{"format": "native_distributed", "status": "refused", "evidence_refs": []}
|
||||
],
|
||||
"safe_inspection": false,
|
||||
"trusted_inspection": false,
|
||||
"conversion": false,
|
||||
"evidence": [
|
||||
{"kind": "primary_documentation", "reference": "R04,R07", "status": "present"},
|
||||
{"kind": "fixture", "reference": "exact-version-producer-fixture", "status": "missing"}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-trust-policy",
|
||||
"schema_version": "1.0.0",
|
||||
"policy_id": "policy-fixture-1",
|
||||
"operation_level": "trusted_metadata",
|
||||
"requested_state_scope": ["model_weights"],
|
||||
"adapter": {"capability_id": "dcp-metadata", "adapter_id": "fixture-reader", "adapter_version": "1.0.0", "adapter_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"},
|
||||
"source": {"root_id": "source-1", "inventory_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", "files": [{"file_id": "file-1", "relative_path": ".metadata", "size_bytes": 32, "sha256": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", "observation_id": "observation-1"}]},
|
||||
"runtime": {"framework": "pytorch", "framework_version": "exact-fixture-version", "dependency_identity": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", "runtime_digest": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"},
|
||||
"isolation": {"profile_id": "disposable-linux-fixture", "profile_version": "1", "profile_digest": "sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", "required_capabilities": ["unprivileged", "no_new_privs", "read_only_source", "network_denied", "resource_limits", "bounded_cleanup"]},
|
||||
"resources": {"wall_time_seconds": 30, "cpu_seconds": 20, "memory_bytes": 1073741824, "processes": 8, "threads": 32, "open_files": 128, "input_bytes": 1048576, "header_bytes": 65536, "message_bytes": 1048576, "scratch_bytes": 1048576, "output_bytes": 0},
|
||||
"output": {"root_id": null, "staging_policy": "none"},
|
||||
"authorization": {"event_id": "authorization-fixture-1", "actor_role": "checkpoint-security-operator", "created_at": "2026-09-02T12:00:00Z", "expires_at": "2026-09-02T12:05:00Z", "single_use_nonce": "0123456789abcdef0123456789abcdef", "single_use": true},
|
||||
"evidence": {"candidate_commit": "1111111111111111111111111111111111111111", "policy_digest": "sha256:9999999999999999999999999999999999999999999999999999999999999999"}
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
{
|
||||
"schema_id": "obliteratus.trusted-worker-message",
|
||||
"schema_version": "1.0.0",
|
||||
"message_type": "complete",
|
||||
"sequence": 4,
|
||||
"source_inventory_digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"adapter_digest": "sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"policy_digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"payload": {"record_count": 4, "encoded_bytes": 4096, "transcript_digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd"}
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-descriptor",
|
||||
"schema_version": "1.0.0",
|
||||
"descriptor_id": "fixture-typed-offset-namespaces",
|
||||
"primary_format": "megatron_torch_dist",
|
||||
"classification_confidence": "verified",
|
||||
"components": [
|
||||
{
|
||||
"component_id": "model",
|
||||
"kind": "model",
|
||||
"format": "megatron_torch_dist",
|
||||
"producer": {
|
||||
"name": "megatron-core",
|
||||
"version": "fixture-version",
|
||||
"format_version": "torch_dist",
|
||||
"evidence_refs": ["ev-producer"]
|
||||
},
|
||||
"state_scopes": ["model_weights"],
|
||||
"topology_ref": "saved-topology",
|
||||
"inventory_refs": ["shard-0"],
|
||||
"tensor_fragment_refs": ["fragment-0"]
|
||||
}
|
||||
],
|
||||
"producer": {
|
||||
"name": "megatron-core",
|
||||
"version": "fixture-version",
|
||||
"format_version": "torch_dist",
|
||||
"evidence_refs": ["ev-producer"]
|
||||
},
|
||||
"evidence": [
|
||||
{
|
||||
"evidence_id": "ev-producer",
|
||||
"subject": "/producer",
|
||||
"kind": "explicit_metadata",
|
||||
"file_ref": "shard-0",
|
||||
"location": "fixture metadata producer",
|
||||
"confidence": "verified",
|
||||
"verifier": "neutral-fixture-generator-v1"
|
||||
}
|
||||
],
|
||||
"source_inventory": {
|
||||
"inventory_id": "inventory-0",
|
||||
"files": [
|
||||
{
|
||||
"file_id": "shard-0",
|
||||
"relative_path": "rank-0/model.safetensors",
|
||||
"role": "model_fragment",
|
||||
"size_bytes": 16,
|
||||
"sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000",
|
||||
"regular_file": true,
|
||||
"observation_id": "dev-1:ino-1:size-16:mtime-0"
|
||||
}
|
||||
],
|
||||
"total_bytes": 16,
|
||||
"observation_complete": true
|
||||
},
|
||||
"safety": {
|
||||
"inspection_level": "trusted_metadata",
|
||||
"trust_required": true,
|
||||
"inventory_revalidated": true,
|
||||
"unsafe_serialization_findings": [],
|
||||
"violations": []
|
||||
},
|
||||
"state": {
|
||||
"observed_scopes": ["model_weights"],
|
||||
"classification": "weights_only"
|
||||
},
|
||||
"topologies": [
|
||||
{
|
||||
"topology_id": "saved-topology",
|
||||
"kind": "saved",
|
||||
"node_count": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"global_world_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"machine_rank": {"value": 0, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"global_rank": {"value": 0, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"local_rank": {"value": 0, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"tp_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"pp_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"dp_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"cp_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"ep_size": {"value": 1, "provenance": "explicit", "evidence_refs": ["ev-producer"]},
|
||||
"etp_size": {"value": null, "provenance": "unknown", "evidence_refs": []},
|
||||
"zero_stage": {"value": null, "provenance": "unknown", "evidence_refs": []}
|
||||
}
|
||||
],
|
||||
"tensor_fragments": [
|
||||
{
|
||||
"fragment_id": "fragment-0",
|
||||
"component_id": "model",
|
||||
"fqn": "layer.weight",
|
||||
"role": "parameter",
|
||||
"dtype": "F32",
|
||||
"global_shape": [2, 2],
|
||||
"local_shape": [2, 2],
|
||||
"element_offset": [0, 0],
|
||||
"element_extent": [2, 2],
|
||||
"padding": {"before": [0, 0], "after": [0, 0], "semantic": "none"},
|
||||
"shard_file_id": "shard-0",
|
||||
"shard_digest_ref": "shard-0",
|
||||
"fragment_digest": null,
|
||||
"replica": {"group_id": null, "member_index": 0, "member_count": 1},
|
||||
"partition_axes": [],
|
||||
"logical_tensor_id": "layer.weight",
|
||||
"tie_group_id": null,
|
||||
"shared_storage_id": null,
|
||||
"storage_locations": [
|
||||
{"kind": "logical_element_range", "element_offset": [0, 0], "element_extent": [2, 2]},
|
||||
{"kind": "megatron_rank_offsets", "global_offset": [0, 0], "rank_offsets": [[0, 0, 1]]},
|
||||
{"kind": "safetensors_byte_range", "begin": 0, "end": 16},
|
||||
{"kind": "hf_weight_map", "tensor_name": "layer.weight", "file_id": "shard-0"}
|
||||
],
|
||||
"evidence_refs": ["ev-producer"]
|
||||
}
|
||||
],
|
||||
"adapter_resolution": {
|
||||
"status": "matched",
|
||||
"adapter_id": "fixture-megatron",
|
||||
"adapter_version": "0.0.0-fixture",
|
||||
"capability_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"reason": "Neutral schema fixture only; not a product support claim."
|
||||
},
|
||||
"conversion_plan": {
|
||||
"eligible": true,
|
||||
"target_format": "hf_safetensors",
|
||||
"state_scope": "weights_only",
|
||||
"dropped_scopes": []
|
||||
},
|
||||
"resource_estimate": {
|
||||
"source_bytes": 16,
|
||||
"logical_bytes": 16,
|
||||
"output_bytes": 16,
|
||||
"temporary_bytes": 32,
|
||||
"peak_ram_bytes": 16,
|
||||
"peak_vram_bytes": 0,
|
||||
"file_count": 1,
|
||||
"tensor_count": 1,
|
||||
"shard_count": 1,
|
||||
"assumptions": ["neutral fixture"],
|
||||
"confidence": "verified",
|
||||
"admission": "admitted"
|
||||
},
|
||||
"support_decision": "conversion_supported",
|
||||
"blockers": []
|
||||
}
|
||||
@@ -0,0 +1,77 @@
|
||||
{
|
||||
"schema_id": "obliteratus.conversion-manifest",
|
||||
"schema_version": "1.0.0",
|
||||
"manifest_id": "fixture-manifest-weights-only",
|
||||
"descriptor": {
|
||||
"schema_version": "1.0.0",
|
||||
"digest": "sha256:2222222222222222222222222222222222222222222222222222222222222222"
|
||||
},
|
||||
"source_inventory_digest": "sha256:3333333333333333333333333333333333333333333333333333333333333333",
|
||||
"source_files": [
|
||||
{
|
||||
"relative_path": "rank-0/model.safetensors",
|
||||
"size_bytes": 16,
|
||||
"sha256": "sha256:0000000000000000000000000000000000000000000000000000000000000000"
|
||||
}
|
||||
],
|
||||
"adapter": {
|
||||
"id": "fixture-adapter",
|
||||
"version": "0.0.0-fixture",
|
||||
"capability_digest": "sha256:1111111111111111111111111111111111111111111111111111111111111111",
|
||||
"producer": "fixture",
|
||||
"producer_version": "0"
|
||||
},
|
||||
"source_topology": {"topology_id": "saved-topology"},
|
||||
"state": {
|
||||
"source_classification": "model_and_optimizer",
|
||||
"output_classification": "weights_only",
|
||||
"observed_scopes": ["model_weights", "optimizer"]
|
||||
},
|
||||
"dropped_scopes": [
|
||||
{"scope": "optimizer", "reason": "Version 1 canonical output is model weights only."}
|
||||
],
|
||||
"canonical_output": {
|
||||
"format": "hf_safetensors",
|
||||
"dtype_policy": "preserve",
|
||||
"files": [
|
||||
{
|
||||
"relative_path": "model.safetensors",
|
||||
"size_bytes": 16,
|
||||
"sha256": "sha256:4444444444444444444444444444444444444444444444444444444444444444"
|
||||
}
|
||||
],
|
||||
"hf_index": null,
|
||||
"logical_tensor_count": 1,
|
||||
"logical_bytes": 16
|
||||
},
|
||||
"resource_usage": {
|
||||
"estimated_peak_ram_bytes": 16,
|
||||
"actual_peak_ram_bytes": 16,
|
||||
"estimated_temp_bytes": 32,
|
||||
"actual_temp_bytes": 16
|
||||
},
|
||||
"validation": {
|
||||
"coverage": true,
|
||||
"replicas": true,
|
||||
"ties": true,
|
||||
"hashes": true,
|
||||
"index": true,
|
||||
"safe_reload": true,
|
||||
"source_unchanged": true,
|
||||
"result": "passed"
|
||||
},
|
||||
"provenance": {
|
||||
"obliteratus_commit": "e39f908832405ccad89cb2a5111e7c2576741d94",
|
||||
"configuration_digest": null,
|
||||
"tokenizer_digest": null,
|
||||
"base_model": {"identity": null, "revision": null, "digest": null},
|
||||
"transformation_log": ["neutral fixture canonicalization"],
|
||||
"unknowns": ["base model identity intentionally absent in neutral fixture"]
|
||||
},
|
||||
"publication": {
|
||||
"staging_validated": true,
|
||||
"promoted": true,
|
||||
"atomic_strategy": "sibling-staging-and-rename",
|
||||
"rollback_result": "not_required"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/peft-adapter-manifest-v1.schema.json",
|
||||
"title": "OBLITERATUS PEFT Adapter Manifest v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_id", "schema_version", "adapter_name", "adapter_type", "adapter_format_version", "base_model", "rank", "alpha", "scaling", "dropout", "bias", "modules_to_save", "target_modules", "tie_policy", "merged", "key_map_digest", "model_card_digest", "key_map"],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.peft-adapter-manifest"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"adapter_name": {"$ref": "#/$defs/id"},
|
||||
"adapter_type": {"const": "lora"},
|
||||
"adapter_format_version": {"const": "peft-lora-v1"},
|
||||
"base_model": {"$ref": "#/$defs/baseModel"},
|
||||
"rank": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807},
|
||||
"alpha": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807},
|
||||
"scaling": {"type": "number", "exclusiveMinimum": 0},
|
||||
"dropout": {"type": "number", "minimum": 0, "exclusiveMaximum": 1},
|
||||
"bias": {"const": "none"},
|
||||
"modules_to_save": {"type": "array", "maxItems": 0},
|
||||
"target_modules": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/id"}},
|
||||
"tie_policy": {"const": "base_model_declared"},
|
||||
"merged": {"const": false},
|
||||
"key_map_digest": {"$ref": "#/$defs/digest"},
|
||||
"model_card_digest": {"$ref": "#/$defs/digest"},
|
||||
"key_map": {"type": "array", "minItems": 1, "uniqueItems": true, "items": {"$ref": "#/$defs/keyMap"}}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"shape": {"type": "array", "minItems": 2, "maxItems": 2, "items": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807}},
|
||||
"baseModel": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["repo_id", "revision", "weights_digest", "tokenizer_digest", "vocab_size", "architecture", "tied_embeddings"],
|
||||
"properties": {
|
||||
"repo_id": {"$ref": "#/$defs/id"},
|
||||
"revision": {"type": "string", "pattern": "^[0-9a-f]{40,64}$"},
|
||||
"weights_digest": {"$ref": "#/$defs/digest"},
|
||||
"tokenizer_digest": {"$ref": "#/$defs/digest"},
|
||||
"vocab_size": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807},
|
||||
"architecture": {"$ref": "#/$defs/id"},
|
||||
"tied_embeddings": {"type": "boolean"}
|
||||
}
|
||||
},
|
||||
"keyMap": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["module_name", "target_module", "lora_A_key", "lora_B_key", "lora_A_shape", "lora_B_shape", "rank"],
|
||||
"properties": {
|
||||
"module_name": {"$ref": "#/$defs/id"},
|
||||
"target_module": {"$ref": "#/$defs/id"},
|
||||
"lora_A_key": {"$ref": "#/$defs/id"},
|
||||
"lora_B_key": {"$ref": "#/$defs/id"},
|
||||
"lora_A_shape": {"$ref": "#/$defs/shape"},
|
||||
"lora_B_shape": {"$ref": "#/$defs/shape"},
|
||||
"rank": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/support-matrix-v1.schema.json",
|
||||
"title": "OBLITERATUS Checkpoint and Runtime Support Matrix v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_id", "schema_version", "generated_from", "status_vocabulary", "rows"],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.checkpoint-support-matrix"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"generated_from": {"type": "string", "pattern": "^[0-9a-f]{40}$"},
|
||||
"status_vocabulary": {
|
||||
"type": "array",
|
||||
"const": ["supported", "conditional", "deferred", "out_of_scope"]
|
||||
},
|
||||
"rows": {
|
||||
"type": "array",
|
||||
"minItems": 1,
|
||||
"items": {"$ref": "#/$defs/row"}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"status": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["value", "basis"],
|
||||
"properties": {
|
||||
"value": {"enum": ["supported", "conditional", "deferred", "out_of_scope"]},
|
||||
"basis": {"type": "string", "minLength": 1, "maxLength": 2048}
|
||||
}
|
||||
},
|
||||
"evidence": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["references", "candidate_commit", "fixture_digest", "environment", "topology", "retained_result"],
|
||||
"properties": {
|
||||
"references": {"type": "array", "minItems": 1, "items": {"$ref": "#/$defs/id"}},
|
||||
"candidate_commit": {"type": ["string", "null"], "pattern": "^[0-9a-f]{40}$"},
|
||||
"fixture_digest": {"type": ["string", "null"], "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"environment": {"type": ["string", "null"], "maxLength": 1024},
|
||||
"topology": {"type": ["string", "null"], "maxLength": 1024},
|
||||
"retained_result": {"type": ["string", "null"], "maxLength": 2048}
|
||||
}
|
||||
},
|
||||
"row": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": [
|
||||
"id",
|
||||
"subject",
|
||||
"format",
|
||||
"producer_versions",
|
||||
"adapter",
|
||||
"model_mapping",
|
||||
"state_scopes",
|
||||
"safety_level",
|
||||
"optional_extras",
|
||||
"capabilities",
|
||||
"canonical_output",
|
||||
"evidence",
|
||||
"limits"
|
||||
],
|
||||
"properties": {
|
||||
"id": {"type": "string", "pattern": "^[a-z0-9][a-z0-9-]*$"},
|
||||
"subject": {"$ref": "#/$defs/id"},
|
||||
"format": {"$ref": "#/$defs/id"},
|
||||
"producer_versions": {"type": "array", "items": {"$ref": "#/$defs/id"}},
|
||||
"adapter": {"type": ["string", "null"], "maxLength": 512},
|
||||
"model_mapping": {"type": "string", "maxLength": 2048},
|
||||
"state_scopes": {"type": "array", "items": {"$ref": "#/$defs/id"}},
|
||||
"safety_level": {"enum": ["ordinary_hf_load", "safe_structure", "trusted_metadata", "not_applicable"]},
|
||||
"optional_extras": {"type": "array", "items": {"$ref": "#/$defs/id"}},
|
||||
"capabilities": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["detect", "safe_inspect", "trusted_inspect", "weights_canonicalize", "topology_reshard", "surgery", "exact_resume", "live_multi_node"],
|
||||
"properties": {
|
||||
"detect": {"$ref": "#/$defs/status"},
|
||||
"safe_inspect": {"$ref": "#/$defs/status"},
|
||||
"trusted_inspect": {"$ref": "#/$defs/status"},
|
||||
"weights_canonicalize": {"$ref": "#/$defs/status"},
|
||||
"topology_reshard": {"$ref": "#/$defs/status"},
|
||||
"surgery": {"$ref": "#/$defs/status"},
|
||||
"exact_resume": {"$ref": "#/$defs/status"},
|
||||
"live_multi_node": {"$ref": "#/$defs/status"}
|
||||
}
|
||||
},
|
||||
"canonical_output": {"type": ["string", "null"], "maxLength": 512},
|
||||
"evidence": {"$ref": "#/$defs/evidence"},
|
||||
"limits": {"type": "array", "minItems": 1, "items": {"type": "string", "maxLength": 2048}}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://obliteratus.dev/schemas/trusted-worker-message-v1.schema.json",
|
||||
"title": "OBLITERATUS Trusted Worker Message v1",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_id", "schema_version", "message_type", "sequence", "source_inventory_digest", "adapter_digest", "policy_digest", "payload"],
|
||||
"properties": {
|
||||
"schema_id": {"const": "obliteratus.trusted-worker-message"},
|
||||
"schema_version": {"const": "1.0.0"},
|
||||
"message_type": {"enum": ["metadata_record", "tensor_fragment_record", "error", "complete"]},
|
||||
"sequence": {"type": "integer", "minimum": 0, "maximum": 100000000},
|
||||
"source_inventory_digest": {"$ref": "#/$defs/digest"},
|
||||
"adapter_digest": {"$ref": "#/$defs/digest"},
|
||||
"policy_digest": {"$ref": "#/$defs/digest"},
|
||||
"payload": {"type": "object"}
|
||||
},
|
||||
"allOf": [
|
||||
{"if": {"properties": {"message_type": {"const": "metadata_record"}}}, "then": {"properties": {"payload": {"$ref": "#/$defs/metadata"}}}},
|
||||
{"if": {"properties": {"message_type": {"const": "tensor_fragment_record"}}}, "then": {"properties": {"payload": {"$ref": "#/$defs/fragment"}}}},
|
||||
{"if": {"properties": {"message_type": {"const": "error"}}}, "then": {"properties": {"payload": {"$ref": "#/$defs/error"}}}},
|
||||
{"if": {"properties": {"message_type": {"const": "complete"}}}, "then": {"properties": {"payload": {"$ref": "#/$defs/complete"}}}}
|
||||
],
|
||||
"$defs": {
|
||||
"id": {"type": "string", "minLength": 1, "maxLength": 512},
|
||||
"digest": {"type": "string", "pattern": "^sha256:[0-9a-f]{64}$"},
|
||||
"metadata": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["record_id", "record_type", "subject", "value", "evidence_refs"],
|
||||
"properties": {
|
||||
"record_id": {"$ref": "#/$defs/id"},
|
||||
"record_type": {"enum": ["producer", "topology", "state_scope", "tensor_metadata"]},
|
||||
"subject": {"type": "string", "pattern": "^/", "maxLength": 1024},
|
||||
"value": {"type": ["string", "integer", "boolean", "null"], "maxLength": 4096},
|
||||
"evidence_refs": {"type": "array", "maxItems": 1024, "uniqueItems": true, "items": {"$ref": "#/$defs/id"}}
|
||||
}
|
||||
},
|
||||
"fragment": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["fragment_id", "tensor_name", "dtype", "shape", "source_file_id", "source_byte_offset", "source_byte_length", "source_digest", "logical_axis", "logical_start", "logical_end"],
|
||||
"properties": {
|
||||
"fragment_id": {"$ref": "#/$defs/id"},
|
||||
"tensor_name": {"$ref": "#/$defs/id"},
|
||||
"dtype": {"$ref": "#/$defs/id"},
|
||||
"shape": {"type": "array", "maxItems": 64, "items": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807}},
|
||||
"source_file_id": {"$ref": "#/$defs/id"},
|
||||
"source_byte_offset": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807},
|
||||
"source_byte_length": {"type": "integer", "minimum": 1, "maximum": 9223372036854775807},
|
||||
"source_digest": {"$ref": "#/$defs/digest"},
|
||||
"logical_axis": {"type": ["integer", "null"], "minimum": 0, "maximum": 63},
|
||||
"logical_start": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807},
|
||||
"logical_end": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807}
|
||||
}
|
||||
},
|
||||
"error": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["code", "phase", "category", "subject_ids", "limit", "actual"],
|
||||
"properties": {
|
||||
"code": {"type": "string", "pattern": "^DCI_[A-Z0-9_]+$"},
|
||||
"phase": {"enum": ["policy", "preflight", "reader", "protocol", "validation", "materialization", "promotion", "evidence", "cleanup"]},
|
||||
"category": {"enum": ["unsupported", "trust", "source", "runtime", "resource", "protocol", "validation", "output", "evidence", "cleanup"]},
|
||||
"subject_ids": {"type": "array", "maxItems": 128, "uniqueItems": true, "items": {"$ref": "#/$defs/id"}},
|
||||
"limit": {"type": ["integer", "null"], "minimum": 0},
|
||||
"actual": {"type": ["integer", "null"], "minimum": 0}
|
||||
}
|
||||
},
|
||||
"complete": {
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["record_count", "encoded_bytes", "transcript_digest"],
|
||||
"properties": {
|
||||
"record_count": {"type": "integer", "minimum": 0, "maximum": 100000000},
|
||||
"encoded_bytes": {"type": "integer", "minimum": 0, "maximum": 9223372036854775807},
|
||||
"transcript_digest": {"$ref": "#/$defs/digest"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,237 @@
|
||||
{
|
||||
"schema_id": "obliteratus.checkpoint-support-matrix",
|
||||
"schema_version": "1.0.0",
|
||||
"generated_from": "5cc43c6e52903497574d80e08dff856028bc47f7",
|
||||
"status_vocabulary": ["supported", "conditional", "deferred", "out_of_scope"],
|
||||
"rows": [
|
||||
{
|
||||
"id": "hf-safetensors-existing-loader",
|
||||
"subject": "Existing Hugging Face-compatible safetensors input",
|
||||
"format": "hf_safetensors",
|
||||
"producer_versions": ["Transformers/Hugging Face compatible; exact model-family behavior varies"],
|
||||
"adapter": null,
|
||||
"model_mapping": "Existing Transformers model class and OBLITERATUS architecture gates",
|
||||
"state_scopes": ["model_weights"],
|
||||
"safety_level": "ordinary_hf_load",
|
||||
"optional_extras": [],
|
||||
"capabilities": {
|
||||
"detect": {"value": "conditional", "basis": "The standalone inspector classifies bounded HF safetensors structure without loading payloads; model compatibility is separate."},
|
||||
"safe_inspect": {"value": "conditional", "basis": "Current bounded JSON and safetensors-header inspection emits structural evidence only."},
|
||||
"trusted_inspect": {"value": "out_of_scope", "basis": "Ordinary supported HF safetensors do not need the planned vendor metadata path."},
|
||||
"weights_canonicalize": {"value": "out_of_scope", "basis": "Input is already the planned canonical format."},
|
||||
"topology_reshard": {"value": "out_of_scope", "basis": "HF file indexes do not encode rank-fragment topology."},
|
||||
"surgery": {"value": "conditional", "basis": "Current behavior depends on model architecture, dtype, quantization, kernels, memory, and quality gates."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "OBLITERATUS is not a training-resume system."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "The current loader/runner has no multi-host process topology."}
|
||||
},
|
||||
"canonical_output": "Hugging Face-compatible safetensors from the current save path",
|
||||
"evidence": {
|
||||
"references": ["R11", "R12", "R13", "R21"],
|
||||
"candidate_commit": "5cc43c6e52903497574d80e08dff856028bc47f7",
|
||||
"fixture_digest": null,
|
||||
"environment": "Local baseline source and existing repository tests; not a universal model-family qualification",
|
||||
"topology": "One OBLITERATUS process; visible devices may receive complete modules",
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": [
|
||||
"A registry entry is not proof of a successful model run.",
|
||||
"Architecture-specific restrictions, including Qwen hybrid complete-model placement, still apply.",
|
||||
"Structural inspection does not establish model-load, surgery, or producer compatibility."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "accelerate-device-map-process-local",
|
||||
"subject": "Accelerate device_map placement and CPU/disk offload",
|
||||
"format": "runtime_placement_not_checkpoint_format",
|
||||
"producer_versions": ["Existing project dependency lock"],
|
||||
"adapter": null,
|
||||
"model_mapping": "Complete modules placed across devices visible to one process",
|
||||
"state_scopes": ["runtime_model_placement"],
|
||||
"safety_level": "not_applicable",
|
||||
"optional_extras": [],
|
||||
"capabilities": {
|
||||
"detect": {"value": "out_of_scope", "basis": "This is runtime placement, not an input format."},
|
||||
"safe_inspect": {"value": "out_of_scope", "basis": "This is runtime placement, not checkpoint inspection."},
|
||||
"trusted_inspect": {"value": "out_of_scope", "basis": "This is runtime placement, not checkpoint inspection."},
|
||||
"weights_canonicalize": {"value": "out_of_scope", "basis": "Placement does not reconstruct rank fragments."},
|
||||
"topology_reshard": {"value": "out_of_scope", "basis": "device_map has no saved rank-fragment contract."},
|
||||
"surgery": {"value": "conditional", "basis": "Current process-local path depends on model-family compatibility and resource headroom."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "No training state is restored."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "One process and one host; device placement is not a launcher."}
|
||||
},
|
||||
"canonical_output": null,
|
||||
"evidence": {
|
||||
"references": ["R14", "R21"],
|
||||
"candidate_commit": "5cc43c6e52903497574d80e08dff856028bc47f7",
|
||||
"fixture_digest": null,
|
||||
"environment": "Existing OBLITERATUS single-process implementation",
|
||||
"topology": "Single host, single OBLITERATUS process, one or more visible devices",
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": [
|
||||
"Not rank-based pipeline parallelism.",
|
||||
"Not distributed checkpoint intake.",
|
||||
"Qwen hybrid models reject generic layer placement across devices."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "pytorch-dcp-fsdp-planned",
|
||||
"subject": "PyTorch DCP and FSDP model-weight intake",
|
||||
"format": "pytorch_dcp_or_fsdp_state",
|
||||
"producer_versions": ["Unselected; must be exact-version qualified"],
|
||||
"adapter": "Deferred exact-version PyTorch adapter",
|
||||
"model_mapping": "Requires a predeclared target model state schema",
|
||||
"state_scopes": ["model_weights"],
|
||||
"safety_level": "safe_structure",
|
||||
"optional_extras": ["checkpoint-pytorch-planned"],
|
||||
"capabilities": {
|
||||
"detect": {"value": "conditional", "basis": "Current bounded inventory recognizes DCP/FSDP marker structure without reading DCP metadata."},
|
||||
"safe_inspect": {"value": "conditional", "basis": "Current inventory-only classification keeps DCP metadata opaque and trust-gated."},
|
||||
"trusted_inspect": {"value": "deferred", "basis": "Blocked on security-owner acceptance, exact-profile controls, and an exact PyTorch version."},
|
||||
"weights_canonicalize": {"value": "deferred", "basis": "No exact-version adapter has been qualified."},
|
||||
"topology_reshard": {"value": "deferred", "basis": "Only exact model-weight cases with retained fixture evidence may qualify."},
|
||||
"surgery": {"value": "deferred", "basis": "Requires successful canonical output then existing OBLITERATUS gates."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "Version 1 emits weights only."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "Checkpoint intake is not distributed execution."}
|
||||
},
|
||||
"canonical_output": "Planned HF safetensors plus conversion manifest",
|
||||
"evidence": {
|
||||
"references": ["R01", "R02", "R03", "R17", "R18", "R27", "R28", "R29"],
|
||||
"candidate_commit": null,
|
||||
"fixture_digest": null,
|
||||
"environment": null,
|
||||
"topology": null,
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": ["No producer reader, adapter, conversion implementation, or qualified version band exists.", "Structural classification alone is not payload compatibility.", "Exact resume and arbitrary stateful/planner objects are excluded."]
|
||||
},
|
||||
{
|
||||
"id": "megatron-bridge-planned",
|
||||
"subject": "Megatron distributed model-weight intake through a model-aware Bridge",
|
||||
"format": "megatron_torch_dist_with_newer_forms_explicitly_deferred",
|
||||
"producer_versions": ["Unselected; must be exact-version and format qualified"],
|
||||
"adapter": "Deferred model-aware Megatron Bridge adapter",
|
||||
"model_mapping": "Requires supported Megatron Bridge/provider mapping and reference configuration",
|
||||
"state_scopes": ["model_weights"],
|
||||
"safety_level": "safe_structure",
|
||||
"optional_extras": ["checkpoint-megatron-planned"],
|
||||
"capabilities": {
|
||||
"detect": {"value": "conditional", "basis": "Current bounded inventory recognizes declared Megatron torch_dist marker structure without a vendor import."},
|
||||
"safe_inspect": {"value": "conditional", "basis": "Current inspection reports bounded structural facts only and does not interpret model semantics."},
|
||||
"trusted_inspect": {"value": "deferred", "basis": "Blocked on security-owner acceptance, exact-profile controls, and an exact vendor stack."},
|
||||
"weights_canonicalize": {"value": "deferred", "basis": "Blocked on common infrastructure and model-aware mapping."},
|
||||
"topology_reshard": {"value": "deferred", "basis": "Must be proven for exact model-weight topology and format."},
|
||||
"surgery": {"value": "deferred", "basis": "Requires canonical output and existing model-family gates."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "Optimizer/resume compatibility is version/format-specific and outside v1."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "Bridge conversion is not an OBLITERATUS distributed runtime."}
|
||||
},
|
||||
"canonical_output": "Planned HF safetensors plus conversion manifest",
|
||||
"evidence": {
|
||||
"references": ["R04", "R05", "R06", "R07", "R27", "R28"],
|
||||
"candidate_commit": null,
|
||||
"fixture_digest": null,
|
||||
"environment": null,
|
||||
"topology": null,
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": ["Offsets alone never authorize generic QKV/MLP/expert reconstruction.", "No producer reader, adapter, conversion implementation, or qualified model/version exists.", "Structural classification alone is not payload compatibility."]
|
||||
},
|
||||
{
|
||||
"id": "deepspeed-zero-universal-planned",
|
||||
"subject": "DeepSpeed ZeRO or Universal model-weight intake",
|
||||
"format": "deepspeed_zero_or_universal",
|
||||
"producer_versions": ["Unselected; must be exact-version qualified"],
|
||||
"adapter": "Deferred exact-version DeepSpeed adapter",
|
||||
"model_mapping": "Official consolidation or Universal mapping for an exact compatible case",
|
||||
"state_scopes": ["model_weights"],
|
||||
"safety_level": "safe_structure",
|
||||
"optional_extras": ["checkpoint-deepspeed-planned"],
|
||||
"capabilities": {
|
||||
"detect": {"value": "conditional", "basis": "Current bounded inventory recognizes declared ZeRO or Universal marker structure without a DeepSpeed import."},
|
||||
"safe_inspect": {"value": "conditional", "basis": "Current inspection reports bounded structural facts only and keeps framework serialization opaque."},
|
||||
"trusted_inspect": {"value": "deferred", "basis": "Blocked on security-owner acceptance and exact-profile controls; official consolidation reads framework serialization."},
|
||||
"weights_canonicalize": {"value": "deferred", "basis": "No adapter has passed the resource and trust gates."},
|
||||
"topology_reshard": {"value": "deferred", "basis": "Universal behavior must be proven for the exact mapping/version."},
|
||||
"surgery": {"value": "deferred", "basis": "Requires canonical output and existing model gates."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "Version 1 emits weights only."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "Checkpoint consolidation is not live OBLITERATUS execution."}
|
||||
},
|
||||
"canonical_output": "Planned HF safetensors plus conversion manifest",
|
||||
"evidence": {
|
||||
"references": ["R08", "R09", "R10", "R17", "R27", "R28"],
|
||||
"candidate_commit": null,
|
||||
"fixture_digest": null,
|
||||
"environment": null,
|
||||
"topology": null,
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": ["No producer reader, adapter, conversion implementation, or qualified version band exists.", "Structural classification alone is not payload compatibility.", "Input is trust-gated and peak CPU RAM/disk must be admitted."]
|
||||
},
|
||||
{
|
||||
"id": "peft-lora-safe-artifacts",
|
||||
"subject": "PEFT LoRA safetensors inspection and exact-base artifact export",
|
||||
"format": "peft_lora_safetensors",
|
||||
"producer_versions": ["PEFT format contract; live compatibility remains version-conditional"],
|
||||
"adapter": null,
|
||||
"model_mapping": "Exact base revision, weights/tokenizer digests, architecture, vocabulary, ties, and full target-module paths are required",
|
||||
"state_scopes": ["adapter_weights"],
|
||||
"safety_level": "safe_structure",
|
||||
"optional_extras": [],
|
||||
"capabilities": {
|
||||
"detect": {"value": "conditional", "basis": "Current inspector recognizes adapter_config.json with safetensors structure without loading payloads."},
|
||||
"safe_inspect": {"value": "conditional", "basis": "Current bounded JSON/header inspection preserves adapter and base identity evidence."},
|
||||
"trusted_inspect": {"value": "out_of_scope", "basis": "Canonical PEFT safetensors artifacts do not require a vendor metadata reader."},
|
||||
"weights_canonicalize": {"value": "out_of_scope", "basis": "LoRA artifact export is not distributed rank-fragment canonicalization."},
|
||||
"topology_reshard": {"value": "out_of_scope", "basis": "PEFT adapter tensors do not establish distributed checkpoint topology."},
|
||||
"surgery": {"value": "conditional", "basis": "The Python exporter writes canonical PEFT files only when exact base identity is supplied; the default pipeline otherwise writes a truthfully unsupported safe artifact."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "Adapter artifacts do not contain complete training-resume state."},
|
||||
"live_multi_node": {"value": "out_of_scope", "basis": "Adapter export is an offline artifact operation."}
|
||||
},
|
||||
"canonical_output": "adapter_model.safetensors, adapter_config.json, adapter_manifest.json, provenance, and model card",
|
||||
"evidence": {
|
||||
"references": ["R16", "R27"],
|
||||
"candidate_commit": null,
|
||||
"fixture_digest": null,
|
||||
"environment": "Mandatory CPU artifact tests plus a conditional upstream PEFT loader check; not a universal base-model qualification",
|
||||
"topology": "Offline single-process adapter artifact export",
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": [
|
||||
"Exact base and tokenizer identity are mandatory for a canonical claim.",
|
||||
"The default pipeline does not invent missing identity and emits a safely serialized unsupported artifact instead.",
|
||||
"Live base-model behavior remains conditional on the exact PEFT, Transformers, model, and runtime versions."
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": "live-multi-node-surgery-research",
|
||||
"subject": "Live multi-node OBLITERATUS surgery",
|
||||
"format": "runtime_not_checkpoint_format",
|
||||
"producer_versions": [],
|
||||
"adapter": null,
|
||||
"model_mapping": "Fixed-membership preflight only; model execution mapping is unimplemented",
|
||||
"state_scopes": ["runtime_execution"],
|
||||
"safety_level": "not_applicable",
|
||||
"optional_extras": [],
|
||||
"capabilities": {
|
||||
"detect": {"value": "out_of_scope", "basis": "Runtime capability is not a source format."},
|
||||
"safe_inspect": {"value": "out_of_scope", "basis": "Runtime capability is not checkpoint inspection."},
|
||||
"trusted_inspect": {"value": "out_of_scope", "basis": "Runtime capability is not checkpoint inspection."},
|
||||
"weights_canonicalize": {"value": "out_of_scope", "basis": "Offline conversion is a separate subsystem."},
|
||||
"topology_reshard": {"value": "out_of_scope", "basis": "Runtime ownership does not establish checkpoint conversion."},
|
||||
"surgery": {"value": "deferred", "basis": "Preflight and CPU/Gloo protocol contracts are available; model loading and surgery payloads are not."},
|
||||
"exact_resume": {"value": "out_of_scope", "basis": "No training-resume product is planned."},
|
||||
"live_multi_node": {"value": "deferred", "basis": "Candidate preflight only; no exact physical profile, multi-host qualification, or supported model workflow exists."}
|
||||
},
|
||||
"canonical_output": null,
|
||||
"evidence": {
|
||||
"references": ["R06", "R15", "R21"],
|
||||
"candidate_commit": null,
|
||||
"fixture_digest": null,
|
||||
"environment": null,
|
||||
"topology": null,
|
||||
"retained_result": null
|
||||
},
|
||||
"limits": ["Current --remote runs one OBLITERATUS process on one SSH host.", "Canonicalizing a distributed checkpoint does not make surgery distributed.", "The distributed command validates fixed membership and exits before model allocation."]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,203 @@
|
||||
# Support runbook: checkpoints, placement, and distributed-state reports
|
||||
|
||||
**Artifact ID:** SUPPORT-DCI-001
|
||||
**Version:** 0.3.0
|
||||
**Status:** Wave 2 safe structural inspection available; producer conversion deferred
|
||||
**Owner:** OBLITERATUS maintainers
|
||||
**Tracking:** Use the repository issue and pull-request workflow; include sanitized evidence only.
|
||||
|
||||
## 1. Service overview
|
||||
|
||||
This runbook supports the current HF-compatible loader, process-local device
|
||||
placement/offload, one-host remote runner, and reports involving distributed
|
||||
checkpoint formats. The bounded structural inspector is current; producer
|
||||
readers, adapters, trusted execution, and producer-backed conversion are not.
|
||||
See the [guide](distributed-checkpoint-intake.md) and [support
|
||||
matrix](support-matrix-v1.json).
|
||||
|
||||
## 2. Escalation ownership
|
||||
|
||||
| Report class | Primary owner | Escalate when |
|
||||
|---|---|---|
|
||||
| Model-capacity / “multi node offset” report | Distributed-runtime triage | Workflow exceeds one qualified host or exact model/topology is unknown |
|
||||
| Descriptor/format/topology contract | Architecture | New format or overloaded term appears |
|
||||
| Trusted-reader or filesystem boundary | Security | Report requires vendor/Python metadata reader |
|
||||
| Reconstruction correctness | Test and data | Gap/overlap/replica/tie/padding behavior is unclear |
|
||||
| Writer/recovery/resources | Persistence | ENOSPC, partial staging, or whole-state memory risk |
|
||||
| Producer adapter | Adapter owner | Exact supported version/model mapping is identified |
|
||||
| Provenance/PEFT | Data | Base/config/tokenizer/adapter identity is incomplete |
|
||||
| Support claim | Release and documentation | A matrix status or public claim would change |
|
||||
| Live multi-node runtime | Architecture and security | Reporter needs multiple participating hosts/processes |
|
||||
|
||||
## 3. Operational readiness checklist
|
||||
|
||||
- [ ] Record exact OBLITERATUS commit and installed package versions.
|
||||
- [ ] Classify the request using the glossary before suggesting remediation.
|
||||
- [ ] Run `obliteratus checkpoint inspect SOURCE --json` only when the source can
|
||||
be handled under the local structural-inspection policy.
|
||||
- [ ] Confirm whether the input is ordinary HF, DCP/FSDP, Megatron, DeepSpeed,
|
||||
PEFT, or ambiguous.
|
||||
- [ ] Confirm whether the goal is loading, conversion, surgery, export,
|
||||
inference, or exact resume.
|
||||
- [ ] Check the machine support row and its limitations.
|
||||
- [ ] Do not describe a deferred/planned row as available.
|
||||
- [ ] Do not cross the trusted-reader boundary during triage; no exact profile,
|
||||
residual-risk acceptance, reader, adapter, or payload execution is part
|
||||
of the current capability.
|
||||
- [ ] Link evidence to the exact public issue or pull request and candidate commit.
|
||||
|
||||
## 4. Monitoring and alerts
|
||||
|
||||
There is no persistent intake service in version 1. For inspection, monitor the
|
||||
CLI exit status and descriptor blockers; stable `DCI_*` codes identify the first
|
||||
failed structural phase. For current model runs, monitor existing CLI logs,
|
||||
stage transitions, host RAM/disk/VRAM, source and output paths, and validation
|
||||
results. Alert thresholds and producer conversion metrics are not claimed.
|
||||
|
||||
## 5. Common scenarios
|
||||
|
||||
### A. “Model needs multi node offset”
|
||||
|
||||
1. Route the complaint primarily to live model-capacity triage; record that
|
||||
this remains an interpretation,
|
||||
not a reporter-defined standard term.
|
||||
2. Request the exact model/revision, workflow stage, current memory/topology,
|
||||
desired host count, failure output, and sanitized environment evidence.
|
||||
3. Do not infer a checkpoint producer or adapter from the phrase. Treat a DCP,
|
||||
Megatron, or ZeRO/UCP artifact as a separate request that requires independent
|
||||
exact producer evidence.
|
||||
4. Treat an explicit affine/reference-mean activation offset hypothesis as a
|
||||
separate algorithm question.
|
||||
5. State that current model support remains single-process placement/offload or
|
||||
one process on one remote host; the distributed command is preflight only.
|
||||
|
||||
### B. HF model does not fit on one device
|
||||
|
||||
1. Confirm model-family placement restrictions and current CLI help.
|
||||
2. For compatible families, use existing single-process placement/offload or a
|
||||
supported quantization mode with sufficient headroom.
|
||||
3. Do not call `device_map` rank sharding or multi-node execution.
|
||||
4. Preserve the Qwen hybrid one-device restriction.
|
||||
|
||||
### C. DCP, Megatron, or DeepSpeed directory supplied today
|
||||
|
||||
1. State that no current OBLITERATUS producer adapter is qualified.
|
||||
2. Run the structural inspector only; a successful classification is not a load
|
||||
or conversion result.
|
||||
3. Do not invoke an unfamiliar vendor reader as a diagnostic shortcut.
|
||||
4. Collect only sanitized structural evidence from the descriptor.
|
||||
5. Record the deferred adapter capability and offer external producer-supported
|
||||
conversion only as an operator-controlled workaround.
|
||||
|
||||
### D. Conversion or save runs out of RAM/disk
|
||||
|
||||
1. Preserve the source and any prior valid output.
|
||||
2. Record stage, normalized error, host resource totals/free capacity, source
|
||||
logical size estimate, and staging status without private local identifiers.
|
||||
3. Do not promote or reuse incomplete staging as valid output.
|
||||
4. Classify full-state export pressure separately from common-writer failures.
|
||||
|
||||
### E. Request for exact training resume
|
||||
|
||||
Explain that the planned v1 output is model weights only. Exact resume normally
|
||||
requires producer-specific optimizer, scheduler, RNG/scaler, progress, and data-
|
||||
position state [R19–R20](references.md#primary-and-upstream-sources).
|
||||
|
||||
### F. Trusted metadata is requested
|
||||
|
||||
1. State that the planned trusted-reader path is not currently implemented or
|
||||
accepted for production use.
|
||||
2. Do not treat locality, filenames, prior structural inspection, a checksum,
|
||||
or `weights_only=True` as trust [R27–R29](references.md#security-and-containment-sources).
|
||||
3. When implemented, require fresh explicit intent plus an exact single-use
|
||||
source/operation/runtime/isolation/resource-bound policy.
|
||||
4. Refuse when any isolation/runtime capability is unavailable; never suggest a
|
||||
weaker subprocess/container fallback or integrity override.
|
||||
5. Prefer a producer-side safetensors export performed in the operator's already
|
||||
trusted environment when an approved reader profile is unavailable.
|
||||
6. Require security review of the exact source/runtime/profile/fixture decision;
|
||||
keep adapter rows deferred until the complete evidence boundary passes.
|
||||
|
||||
## 6. Troubleshooting procedure
|
||||
|
||||
1. Capture `git rev-parse HEAD` and `python3 --version`.
|
||||
2. Capture current CLI syntax with `python3 -m obliteratus --help`; do not rely on
|
||||
examples that the parser rejects.
|
||||
3. Identify the first failing stage: structural inventory/probe, current load,
|
||||
placement, pristine check, surgery, save, reload, or a gated intake phase.
|
||||
4. Compare the report to the support matrix and glossary.
|
||||
5. Check architecture/model restrictions before changing placement.
|
||||
6. Record normalized relative paths or opaque IDs, file sizes, safe digests,
|
||||
versions, topology facts/provenance, and resource estimates.
|
||||
7. Reproduce only with project-owned or explicitly approved fixtures.
|
||||
8. Escalate to the owner table with exact evidence and a no-mutation statement.
|
||||
|
||||
## 7. Recovery and rollback
|
||||
|
||||
Current producer-neutral writes use sibling staging, validate all outputs,
|
||||
recheck source identity, then promote. Existing model runs retain their current
|
||||
persistence behavior.
|
||||
|
||||
- If inspection fails: no output should exist.
|
||||
- If an exact registered capability reports a missing or incompatible optional
|
||||
dependency: retain its `adapter_resolution.reason` diagnostic, install only
|
||||
the named exact reviewed extra/version in the intended disposable profile,
|
||||
and retry from an unchanged source. A dependency match does not satisfy the
|
||||
separate trust-policy or profile-approval gates.
|
||||
- If materialization fails: source/prior output remain unchanged; staging is not
|
||||
success.
|
||||
- If promotion validation fails: the common writer restores the prior
|
||||
destination and removes owned staging; never overwrite evidence silently.
|
||||
- If trust, source identity, runtime, containment, redaction, evidence, cleanup,
|
||||
or security-baseline validation fails: promote nothing, allow no bypass, and
|
||||
create a fresh attempt only after the cause is corrected.
|
||||
- If an adapter support regression appears: remove/defer its static support row
|
||||
and registry entry; ordinary HF loading remains available.
|
||||
|
||||
## 8. Change management
|
||||
|
||||
- Contract/schema changes require architecture and traceability review plus a
|
||||
new schema version when incompatible.
|
||||
- Adapter changes require exact producer-version fixtures and retained evidence.
|
||||
- Trusted-reader/profile changes require an exact threat model, degraded-mode
|
||||
review, security tests, and explicit residual-risk acceptance.
|
||||
- Support status changes require release approval and offline contract validation.
|
||||
- Live multi-node claims require architecture and security approval plus exact
|
||||
multi-host evidence.
|
||||
- Delivery follows repository PR, signed-commit, and CI policy.
|
||||
|
||||
## 9. Communication templates
|
||||
|
||||
### Unsupported format
|
||||
|
||||
> OBLITERATUS identified this as `{format}`, which is `{status}` in support
|
||||
> matrix v1. No source or prior output was changed. The blocking contract is
|
||||
> `{code}` at `{phase}`. Continue with `{sanitized next action}`.
|
||||
|
||||
### Evidence pending
|
||||
|
||||
> The upstream framework documents this capability, but OBLITERATUS has no
|
||||
> exact-version/model/topology qualification at the candidate commit. The row
|
||||
> remains deferred until the required evidence passes.
|
||||
|
||||
### Conversion versus runtime
|
||||
|
||||
> Converting rank-sharded model weights into HF safetensors is an offline input
|
||||
> step. It does not make the OBLITERATUS surgery process multi-node; that runtime
|
||||
> remains a separate, unsupported capability.
|
||||
|
||||
## 10. Post-incident activities
|
||||
|
||||
- Preserve descriptor/manifest and test result digests.
|
||||
- Record exact commit, versions, topology, resource admission/actuals, failure
|
||||
code/phase, source-immutability result, and output-promotion result.
|
||||
- Add a minimal project-owned regression fixture.
|
||||
- Update risk, traceability, support matrix, and runbook if the contract changed.
|
||||
- Never turn one successful case into an unqualified universal support claim.
|
||||
|
||||
## 11. Runbook maintenance
|
||||
|
||||
Review with each adapter version-band change, schema version, release candidate,
|
||||
and incident. Security owns trust-boundary text; architecture owns terminology;
|
||||
test/release own evidence status; support owns scenario clarity. Offline checks
|
||||
must continue validating local links, CLI examples, and support-matrix contracts.
|
||||
@@ -0,0 +1,182 @@
|
||||
# Distributed preflight contract
|
||||
|
||||
`obliteratus distributed preflight PROFILE.json` is the only distributed entry
|
||||
point currently implemented. It consumes a worker group that an external
|
||||
trusted scheduler has already launched. It does not launch `torchrun`, SSH to a
|
||||
peer, provision hosts, install packages, relay credentials, load model weights,
|
||||
or perform surgery.
|
||||
|
||||
The command is an admission gate, not a supported multi-node workflow. Native
|
||||
Llama tensor-parallel loading, distributed PROBE/DISTILL/EXCISE/VERIFY, export,
|
||||
and physical two-host qualification remain unimplemented and unqualified.
|
||||
NCCL, Gloo, and rendezvous authentication or encryption are not claimed.
|
||||
|
||||
## Invocation boundary
|
||||
|
||||
The scheduler must provide all of the following to every rank:
|
||||
|
||||
- `RANK`, `LOCAL_RANK`, `WORLD_SIZE`, and `LOCAL_WORLD_SIZE`;
|
||||
- `GROUP_RANK`, `ROLE_RANK`, and `ROLE_WORLD_SIZE`;
|
||||
- `MASTER_ADDR` and `MASTER_PORT`;
|
||||
- `TORCHELASTIC_RUN_ID`, `TORCHELASTIC_RESTART_COUNT=0`, and
|
||||
`TORCHELASTIC_MAX_RESTARTS=0`;
|
||||
- a fresh `OBLITERATUS_RUN_ID` distinct from the rendezvous ID; and
|
||||
- exact `GLOO_SOCKET_IFNAME` and `NCCL_SOCKET_IFNAME` values matching the
|
||||
reviewed profile.
|
||||
|
||||
Ordinary `run`, `obliterate`, and one-host SSH commands never inspect these
|
||||
variables to infer distributed intent. The distributed command accepts only
|
||||
the local profile path and optional `--json`; secrets and operational overrides
|
||||
are not CLI inputs.
|
||||
|
||||
## Closed profile schema
|
||||
|
||||
The profile is bounded UTF-8 JSON. Every section and field is required; unknown
|
||||
or duplicate fields fail closed. The following uses non-operational example
|
||||
values and is not an accepted physical-host profile:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 1,
|
||||
"run": {
|
||||
"run_id": "11111111111111111111111111111111",
|
||||
"rendezvous_id": "22222222222222222222222222222222",
|
||||
"world_size": 2,
|
||||
"local_world_size": 1
|
||||
},
|
||||
"identity": {
|
||||
"source_digest": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
|
||||
"model_digest": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
|
||||
"tokenizer_digest": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
|
||||
"commit_sha": "dddddddddddddddddddddddddddddddddddddddd",
|
||||
"code_digest": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee"
|
||||
},
|
||||
"topology": {
|
||||
"tensor_parallel_size": 2,
|
||||
"coordinator_rank": 0,
|
||||
"placement_plan_digest": "eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
|
||||
"dimension_divisors": [4096, 11008]
|
||||
},
|
||||
"network": {
|
||||
"master_addr": "10.10.0.10",
|
||||
"master_port": 29500,
|
||||
"interface": "eth0",
|
||||
"allowed_master_cidrs": ["10.10.0.0/24"]
|
||||
},
|
||||
"source": {"path": "/srv/immutable/model"},
|
||||
"staging": {
|
||||
"path": "/srv/obliteratus/staging",
|
||||
"storage_digest": "ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
|
||||
},
|
||||
"resources": {
|
||||
"min_free_device_memory_bytes": 8589934592,
|
||||
"min_free_host_memory_bytes": 17179869184,
|
||||
"min_free_staging_bytes": 107374182400,
|
||||
"max_source_files": 10000,
|
||||
"max_source_bytes": 1099511627776,
|
||||
"max_source_file_bytes": 274877906944
|
||||
},
|
||||
"timeouts": {
|
||||
"source_seconds": 1800,
|
||||
"init_seconds": 60,
|
||||
"collective_seconds": 60,
|
||||
"teardown_seconds": 15
|
||||
},
|
||||
"software": {
|
||||
"python": "3.12.11",
|
||||
"platform": "Linux-6.18.0-x86_64-with-glibc2.42",
|
||||
"machine": "x86_64",
|
||||
"torch": "2.13.0",
|
||||
"transformers": "5.15.0",
|
||||
"accelerate": "1.12.0",
|
||||
"safetensors": "0.7.0",
|
||||
"cuda": "13.0",
|
||||
"nccl": "2.28.3",
|
||||
"driver": "13000"
|
||||
},
|
||||
"execution": {
|
||||
"device_kind": "cuda",
|
||||
"device_name": "REVIEWED-EXACT-GPU-NAME",
|
||||
"compute_capability": "REVIEWED-EXACT-CAPABILITY",
|
||||
"evidence_tier": "candidate_preflight",
|
||||
"allowed_environment_keys": ["HOME", "LANG", "PATH", "CUDA_VISIBLE_DEVICES"],
|
||||
"local_files_only": true,
|
||||
"trust_remote_code": false,
|
||||
"allow_runtime_install": false,
|
||||
"allow_plugins": false,
|
||||
"allow_compilation": false,
|
||||
"allow_adapters": false,
|
||||
"allow_quantization": false
|
||||
},
|
||||
"evidence": {"path": "/srv/obliteratus/staging/11111111111111111111111111111111/preflight.json"}
|
||||
}
|
||||
```
|
||||
|
||||
The source tree must be an immutable, local, non-symlink directory containing
|
||||
a structurally valid canonical Hugging Face safetensors checkpoint, immutable
|
||||
tokenizer metadata, and only closed inert file types. The existing bounded
|
||||
safe-structure inspector validates safetensors headers and rejects adapters,
|
||||
pickle formats, aliases, links, and mutable files without deserializing tensor
|
||||
payloads. Content hashes are streamed without loading tensor data. The
|
||||
three source digests are canonical hashes of the complete file inventory, the
|
||||
safetensors subset, and tokenizer-named files respectively.
|
||||
|
||||
`commit_sha` binds Git history while `code_digest` independently binds the
|
||||
Python files actually available to the process, including dirty-tree changes.
|
||||
`storage_digest` is not a label: each worker derives it from the exact Linux
|
||||
mountinfo record backing the staging path. The selected interface must own an
|
||||
address inside the private allowlist, and the coordinator's master address must
|
||||
be bound to that interface. All inherited environment keys must appear in the
|
||||
closed allowlist; credential-, cloud-, token-, and proxy-shaped keys can never
|
||||
be allowlisted.
|
||||
|
||||
## Admission and failure behavior
|
||||
|
||||
Before returning `preflighted`, every rank must agree on the run, config,
|
||||
source, model, tokenizer, commit, software, storage, placement, topology, and
|
||||
network-interface identities. Global devices and per-host local ranks must be
|
||||
unique. Device, host RAM, and staging capacity must meet the exact integer
|
||||
floors. Rank zero exclusively reserves `staging/<run_id>` and every rank must
|
||||
observe an atomic shared marker there.
|
||||
|
||||
The selected interface must own an address inside the private allowlist before
|
||||
Gloo initialization is attempted; the coordinator must also own the configured
|
||||
numeric rendezvous address on that interface. All local preflight probes are
|
||||
bounded by the smaller configured source/collective deadline. Source traversal
|
||||
and structural inspection nest under that same process-level wall-clock timer,
|
||||
including payload hashing and revalidation. A runtime that cannot enforce that
|
||||
deadline refuses the attempt.
|
||||
|
||||
Any missing rank, disagreement, timeout, backend error, rank loss, or uncertain
|
||||
cleanup fails the whole attempt. Restarts and in-memory resume are forbidden;
|
||||
retry requires fresh run and rendezvous IDs and a complete new preflight.
|
||||
Cleanup uncertainty is `quarantined`.
|
||||
|
||||
Evidence is bounded, mode `0600`, atomically created, and never overwritten.
|
||||
The path is fixed to `staging/<run_id>/preflight.json`. While Gloo is live,
|
||||
every rank reads and votes on one canonical `PREFLIGHTED/PREPARED` lifecycle
|
||||
record. This record has a distinct stage-message schema and cannot be decoded
|
||||
as terminal success evidence. After bounded group destruction, each rank
|
||||
creates a private `PREFLIGHTED/COMMITTED` teardown acknowledgement; rank zero
|
||||
publishes terminal success only after validating the complete fixed-rank set.
|
||||
Failures emit `ABORTING` and then `ABORTED` or `QUARANTINED` lifecycle receipts.
|
||||
Missing or invalid receipts produce `LMS_CLEANUP_INCOMPLETE`, never success.
|
||||
|
||||
An FD-level guard is active across backend initialization, collectives, and
|
||||
teardown. Raw native/backend stderr is discarded; any emitted bytes fail the
|
||||
attempt as `LMS_DIAGNOSTIC_REDACTION_FAILED`. Terminal evidence contains only
|
||||
allowlisted state, counts, stable error codes, the mandatory `protocol_cpu` or
|
||||
`candidate_preflight` scope label, and opaque digests—not raw endpoints,
|
||||
hostnames, device identifiers, paths, environment mappings, exceptions,
|
||||
prompts, tensors, or credentials.
|
||||
|
||||
If process-group destruction exceeds its explicit bound, the affected
|
||||
externally launched worker exits with fixed status `70` before the FD guard is
|
||||
restored. A Python teardown thread is never allowed to outlive containment.
|
||||
Missing teardown acknowledgement then forces coordinator-side
|
||||
`LMS_CLEANUP_INCOMPLETE` and `QUARANTINED`; torchrun restart remains disabled.
|
||||
|
||||
The implementation and tests define a fail-closed candidate boundary only. No
|
||||
exact physical-host profile, residual-risk acceptance, authenticated or encrypted
|
||||
transport, GPU/NCCL qualification, model payload, or production support claim is
|
||||
included.
|
||||
Reference in New Issue
Block a user