mirror of
https://github.com/elder-plinius/OBLITERATUS.git
synced 2026-08-18 00:47:23 +02:00
feat: FP8 and NVFP4 checkpoint support (dequantize, surgery in float, BF16 output)
Point obliteratus at an FP8 or NVFP4 checkpoint and it just works: the loader detects the format from config.json + safetensors metadata (no weight loads), dequantizes shard-by-shard to a temporary BF16 copy, then runs the normal float pipeline and saves BF16. Supported layouts: - FP8 DeepSeek-style block-wise (weight_scale_inv + weight_block_size) - FP8 per-channel / per-tensor (compressed-tensors, ModelOpt) - NVFP4 ModelOpt (uint8 nibbles + FP8 group scales + FP32 global), including MIXED_PRECISION checkpoints (FP8 mixer + NVFP4 experts) - NVFP4 compressed-tensors (reciprocal scales) Design: - New pure-torch obliteratus/models/quant_dequant.py; no new deps. NVFP4 unpack uses torch.float4_e2m1fn_x2 when a runtime probe proves it works, else a chunked nibble LUT (bounds transient int64 index memory; a naive implementation OOMed at 96GB on a 30B model). - Scale keys are dropped only when their base weight exists in the same shard, so legitimate params ending in _scale (logit_scale et al.) survive. - Unsupported schemes (fbgemm, quanto, W4A4, ...) fail loudly at load, naming the scheme. - Surgery guards: float8 or packed uint8 reaching _dequantize_weight or any fused-MoE path raises RuntimeError instead of silently upcasting (bitsandbytes quant_state params are explicitly excluded). - Save path strips quantization metadata and logs that output is BF16; re-quantization for serving is out of scope (llm-compressor/modelopt). - CLI: new --trust-remote-code flag; help text documents auto-detection. Validated end-to-end on 1x A100-80GB (see PR description): Nemotron-3-Nano-Omni-30B NVFP4 (mixed) and FP8, Qwen3-8B-FP8 (block-wise) vs Qwen3-8B BF16 baseline (perplexity 4.23 vs 4.33).
This commit is contained in:
@@ -479,6 +479,15 @@ obliteratus obliterate meta-llama/Llama-3.1-405B-Instruct \
|
||||
|
||||
Quantization roughly halves the GPU count at each step down. A 70B model that needs 3x A100-80GB in bf16 fits on 2 in int8 or 1 in int4.
|
||||
|
||||
**FP8 and NVFP4 checkpoints are supported automatically.** No flag needed — the loader detects the format from the checkpoint's `quantization_config`, dequantizes the weights to float (BF16 by default) shard-by-shard, runs the normal pipeline, and saves the output as plain BF16:
|
||||
|
||||
| Format | Schemes detected |
|
||||
|--------|------------------|
|
||||
| FP8 | DeepSeek-style block-wise (`weight_scale_inv` + `weight_block_size`), compressed-tensors per-channel, ModelOpt FP8 |
|
||||
| NVFP4 | ModelOpt (`weight` + `weight_scale` + `weight_scale_2`), compressed-tensors NVFP4 |
|
||||
|
||||
Two things to know: peak VRAM is the **BF16 size** of the model (not the quantized size), and the output is saved as BF16 — re-quantize afterward with llm-compressor or modelopt if you want a quantized serving artifact. Other quantization schemes (fbgemm, quanto, W4A4) fail loudly with a message naming the scheme.
|
||||
|
||||
### GPU calculator
|
||||
|
||||
Not sure how many GPUs you need? The `gpu-calc` command estimates the minimum GPU count for any model, accounting for weight memory, activation overhead, and CUDA context:
|
||||
|
||||
Reference in New Issue
Block a user