33 CPU-only tests: FP8 block-wise/per-channel round-trips, NVFP4
round-trips (direct + reciprocal scales, native vs manual unpack
agreement), scheme detection incl. ModelOpt MIXED_PRECISION, surgery
guard raises, and tiny synthetic FP8/NVFP4 GPT-2 checkpoints through
load_model end-to-end. Full suite: 870 passed (8 failures pre-exist on
main, verified against pristine checkout).
docs/theory_journal.md Appendices E-E2: engineering log of the 12
findings from implementation and real-checkpoint validation (nibble
order conventions, mixed-precision layouts, the 96GB OOM and the
chunked-unpack fix, transformers 5.x drift shims, Omni wrapper unwrap).
Point obliteratus at an FP8 or NVFP4 checkpoint and it just works:
the loader detects the format from config.json + safetensors metadata
(no weight loads), dequantizes shard-by-shard to a temporary BF16 copy,
then runs the normal float pipeline and saves BF16.
Supported layouts:
- FP8 DeepSeek-style block-wise (weight_scale_inv + weight_block_size)
- FP8 per-channel / per-tensor (compressed-tensors, ModelOpt)
- NVFP4 ModelOpt (uint8 nibbles + FP8 group scales + FP32 global),
including MIXED_PRECISION checkpoints (FP8 mixer + NVFP4 experts)
- NVFP4 compressed-tensors (reciprocal scales)
Design:
- New pure-torch obliteratus/models/quant_dequant.py; no new deps.
NVFP4 unpack uses torch.float4_e2m1fn_x2 when a runtime probe proves
it works, else a chunked nibble LUT (bounds transient int64 index
memory; a naive implementation OOMed at 96GB on a 30B model).
- Scale keys are dropped only when their base weight exists in the
same shard, so legitimate params ending in _scale (logit_scale et al.)
survive.
- Unsupported schemes (fbgemm, quanto, W4A4, ...) fail loudly at load,
naming the scheme.
- Surgery guards: float8 or packed uint8 reaching _dequantize_weight or
any fused-MoE path raises RuntimeError instead of silently upcasting
(bitsandbytes quant_state params are explicitly excluded).
- Save path strips quantization metadata and logs that output is BF16;
re-quantization for serving is out of scope (llm-compressor/modelopt).
- CLI: new --trust-remote-code flag; help text documents auto-detection.
Validated end-to-end on 1x A100-80GB (see PR description):
Nemotron-3-Nano-Omni-30B NVFP4 (mixed) and FP8, Qwen3-8B-FP8
(block-wise) vs Qwen3-8B BF16 baseline (perplexity 4.23 vs 4.33).
Loading custom-architecture checkpoints (Nemotron Omni family) against
transformers 5.14 surfaced API drift and wrapper mismatches:
- flash_attention_2 pinned in config but flash-attn not installed:
recursively fall back to eager attention, incl. nested sub-configs
(multimodal llm_config).
- all_tied_weights_keys shim: expected by the accelerate device-map
integration and assigned by 5.x post_init, but absent on older remote
code. Settable property aggregating legacy _tied_weights_keys,
filtering keys that don't resolve on multimodal wrappers.
- prepare_inputs_for_generation: 5.x may pass cache_position=None;
older remote code assumes a tensor. Wrap and synthesize it.
- Legacy list-style _tied_weights_keys normalized post-load so
save_pretrained works (emptied when tie_word_embeddings is false).
- Multimodal wrappers whose forward() requires media inputs are
unwrapped to their language_model submodule for text-only surgery.
- Register nemotron_h architecture (hybrid Mamba/attention/MoE, all
layer content under layer.mixer) in strategies/utils.py.
Retain the unsuperseded macOS RAM-detection contribution from PR #13 while making every optional probe fail closed. The shared MPS device, loader, cache, and dtype work is already present on current main and is intentionally not duplicated.
Cover legacy console fallbacks, platform temp paths, local UI disk probing, and lazy analysis exports. Bind the new local UI test to the executable source-to-test risk map.
Salvage the still-relevant functional work from PR #48: add non-UTF-8 console fallbacks, use platform temporary directories, make pipeline log output encoding-safe, and defer heavyweight analysis imports. The obsolete contributed CI workflow and already-corrected remote URL are intentionally excluded.
Add a --gpu-memory-utilization flag (0.0-1.0, default 0.85) that controls
the fraction of GPU VRAM available for model loading. Plumbed from CLI
through AbliterationPipeline to load_model's max_memory calculation.
Useful on dedicated GPU setups where the default 15% reserve is wasteful
and causes unnecessary CPU offloading on models that would otherwise fit.
Joseph Magly