Point obliteratus at an FP8 or NVFP4 checkpoint and it just works:
the loader detects the format from config.json + safetensors metadata
(no weight loads), dequantizes shard-by-shard to a temporary BF16 copy,
then runs the normal float pipeline and saves BF16.
Supported layouts:
- FP8 DeepSeek-style block-wise (weight_scale_inv + weight_block_size)
- FP8 per-channel / per-tensor (compressed-tensors, ModelOpt)
- NVFP4 ModelOpt (uint8 nibbles + FP8 group scales + FP32 global),
including MIXED_PRECISION checkpoints (FP8 mixer + NVFP4 experts)
- NVFP4 compressed-tensors (reciprocal scales)
Design:
- New pure-torch obliteratus/models/quant_dequant.py; no new deps.
NVFP4 unpack uses torch.float4_e2m1fn_x2 when a runtime probe proves
it works, else a chunked nibble LUT (bounds transient int64 index
memory; a naive implementation OOMed at 96GB on a 30B model).
- Scale keys are dropped only when their base weight exists in the
same shard, so legitimate params ending in _scale (logit_scale et al.)
survive.
- Unsupported schemes (fbgemm, quanto, W4A4, ...) fail loudly at load,
naming the scheme.
- Surgery guards: float8 or packed uint8 reaching _dequantize_weight or
any fused-MoE path raises RuntimeError instead of silently upcasting
(bitsandbytes quant_state params are explicitly excluded).
- Save path strips quantization metadata and logs that output is BF16;
re-quantization for serving is out of scope (llm-compressor/modelopt).
- CLI: new --trust-remote-code flag; help text documents auto-detection.
Validated end-to-end on 1x A100-80GB (see PR description):
Nemotron-3-Nano-Omni-30B NVFP4 (mixed) and FP8, Qwen3-8B-FP8
(block-wise) vs Qwen3-8B BF16 baseline (perplexity 4.23 vs 4.33).
Loading custom-architecture checkpoints (Nemotron Omni family) against
transformers 5.14 surfaced API drift and wrapper mismatches:
- flash_attention_2 pinned in config but flash-attn not installed:
recursively fall back to eager attention, incl. nested sub-configs
(multimodal llm_config).
- all_tied_weights_keys shim: expected by the accelerate device-map
integration and assigned by 5.x post_init, but absent on older remote
code. Settable property aggregating legacy _tied_weights_keys,
filtering keys that don't resolve on multimodal wrappers.
- prepare_inputs_for_generation: 5.x may pass cache_position=None;
older remote code assumes a tensor. Wrap and synthesize it.
- Legacy list-style _tied_weights_keys normalized post-load so
save_pretrained works (emptied when tie_word_embeddings is false).
- Multimodal wrappers whose forward() requires media inputs are
unwrapped to their language_model submodule for text-only surgery.
- Register nemotron_h architecture (hybrid Mamba/attention/MoE, all
layer content under layer.mixer) in strategies/utils.py.
Add a --gpu-memory-utilization flag (0.0-1.0, default 0.85) that controls
the fraction of GPU VRAM available for model loading. Plumbed from CLI
through AbliterationPipeline to load_model's max_memory calculation.
Useful on dedicated GPU setups where the default 15% reserve is wasteful
and causes unnecessary CPU offloading on models that would otherwise fit.
Four bugs prevented bitsandbytes 4-bit quantized models from completing
ablation studies on GPUs with 16GB VRAM:
1. runner.py: quantization parameter was never passed from StudyConfig
to load_model(), so the loader had no idea quantization was enabled.
2. loader.py (max_memory): GPU memory budget was calculated against the
unquantized model size, causing accelerate to offload layers to meta
device even though the quantized model fits comfortably.
Now divides estimate by 4 (4-bit) or 2 (8-bit) before deciding.
3. evaluator.py: empty strings in wikitext dataset caused zero-length
tensors that crashed the forward pass with a reshape error.
Now filters empty/whitespace-only texts and skips empty batches.
4. loader.py (snapshot/restore): snapshot skip decision used unquantized
size estimate, and restore used strict=True which rejects bitsandbytes
metadata keys (.absmax, .quant_map, .quant_state). Now uses quantized
estimate and strict=False.
Tested on RTX 5060 Ti (16GB) with Qwen2.5-Coder-7B-Instruct in 4-bit.
Quick Scan (layer_removal + ffn_ablation) completes all 56 specs.
The snapshot() deepcopy was cloning tensors on their original GPU
devices, doubling VRAM usage. For a 234GB model sharded across 6
A100-80GB GPUs (~39GB each), this left no room for the copy.
Now snapshot stores tensors on CPU and restore() moves them back
to each parameter's current device.
Brian Bell