Point obliteratus at an FP8 or NVFP4 checkpoint and it just works:
the loader detects the format from config.json + safetensors metadata
(no weight loads), dequantizes shard-by-shard to a temporary BF16 copy,
then runs the normal float pipeline and saves BF16.
Supported layouts:
- FP8 DeepSeek-style block-wise (weight_scale_inv + weight_block_size)
- FP8 per-channel / per-tensor (compressed-tensors, ModelOpt)
- NVFP4 ModelOpt (uint8 nibbles + FP8 group scales + FP32 global),
including MIXED_PRECISION checkpoints (FP8 mixer + NVFP4 experts)
- NVFP4 compressed-tensors (reciprocal scales)
Design:
- New pure-torch obliteratus/models/quant_dequant.py; no new deps.
NVFP4 unpack uses torch.float4_e2m1fn_x2 when a runtime probe proves
it works, else a chunked nibble LUT (bounds transient int64 index
memory; a naive implementation OOMed at 96GB on a 30B model).
- Scale keys are dropped only when their base weight exists in the
same shard, so legitimate params ending in _scale (logit_scale et al.)
survive.
- Unsupported schemes (fbgemm, quanto, W4A4, ...) fail loudly at load,
naming the scheme.
- Surgery guards: float8 or packed uint8 reaching _dequantize_weight or
any fused-MoE path raises RuntimeError instead of silently upcasting
(bitsandbytes quant_state params are explicitly excluded).
- Save path strips quantization metadata and logs that output is BF16;
re-quantization for serving is out of scope (llm-compressor/modelopt).
- CLI: new --trust-remote-code flag; help text documents auto-detection.
Validated end-to-end on 1x A100-80GB (see PR description):
Nemotron-3-Nano-Omni-30B NVFP4 (mixed) and FP8, Qwen3-8B-FP8
(block-wise) vs Qwen3-8B BF16 baseline (perplexity 4.23 vs 4.33).
New `obliteratus gpu-calc` subcommand estimates minimum GPU count from
model params, dtype, and GPU VRAM. Auto-detects param counts from HF
configs including MoE expert structure.
README now covers --dtype, --quantization flags, the gpu-calc command,
and references both in the "Choosing the right setup" table.
Add a comprehensive section covering:
- How model sharding (pipeline parallelism) works and its limitations
- GPU selection via --gpus flag
- Pipeline parallel benchmarks on GPT-OSS-120B across 3-8 A100-80GB GPUs
- Stage-by-stage timing breakdown
- When data parallelism helps (and when it doesn't)
- Remote SSH execution with CLI and YAML examples
- Decision table for choosing the right setup
When --data-parallel is passed and the model fits on a single GPU,
wraps it with nn.DataParallel to split prompt batches across all
available GPUs during activation collection. Batch size scales by
GPU count. Hooks already move activations to CPU so they work
correctly across replicas.
Joseph Magly