transformers 5 stores routed experts as fused 3D parameters
(experts.gate_up_proj / experts.down_proj) for Mixtral, Qwen3-MoE,
DeepSeek-V3, GLM-4 MoE, Llama 4, gpt-oss and OLMoE. When such a layer is
CPU- or disk-offloaded by Accelerate, EXCISE failed closed with
"offloaded fused expert tensors are not yet a supported surgery layout",
so every frontier MoE checkpoint large enough to need offload could not be
abliterated at all.
Route the three fused projections and the fused bias projection through the
existing LogicalParameterTransaction adapter: a meta-resident fused
parameter is resolved to a private copy of its authoritative backing value,
projected per expert, and committed atomically; the live parameter stays on
meta and quantized or unknown backing layouts still fail before mutation.
Thread offload_roots through the MoE dispatch call sites so parent-prefixed
Accelerate hooks resolve, and drop the preflight rejection of 3D meta
parameters (validate_offloaded_parameters already resolves them).
Tests: fused 3D, bias, granular and selective-inversion projections on
offloaded backing stores, bounded norm restoration, parent-prefixed hook
resolution, quantized fail-closed, commit-failure rollback, and a full
offline pipeline run on a disk-offloaded tiny Mixtral fixture that verifies
the saved checkpoint carries the updated experts with no meta tensors.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Point obliteratus at an FP8 or NVFP4 checkpoint and it just works:
the loader detects the format from config.json + safetensors metadata
(no weight loads), dequantizes shard-by-shard to a temporary BF16 copy,
then runs the normal float pipeline and saves BF16.
Supported layouts:
- FP8 DeepSeek-style block-wise (weight_scale_inv + weight_block_size)
- FP8 per-channel / per-tensor (compressed-tensors, ModelOpt)
- NVFP4 ModelOpt (uint8 nibbles + FP8 group scales + FP32 global),
including MIXED_PRECISION checkpoints (FP8 mixer + NVFP4 experts)
- NVFP4 compressed-tensors (reciprocal scales)
Design:
- New pure-torch obliteratus/models/quant_dequant.py; no new deps.
NVFP4 unpack uses torch.float4_e2m1fn_x2 when a runtime probe proves
it works, else a chunked nibble LUT (bounds transient int64 index
memory; a naive implementation OOMed at 96GB on a 30B model).
- Scale keys are dropped only when their base weight exists in the
same shard, so legitimate params ending in _scale (logit_scale et al.)
survive.
- Unsupported schemes (fbgemm, quanto, W4A4, ...) fail loudly at load,
naming the scheme.
- Surgery guards: float8 or packed uint8 reaching _dequantize_weight or
any fused-MoE path raises RuntimeError instead of silently upcasting
(bitsandbytes quant_state params are explicitly excluded).
- Save path strips quantization metadata and logs that output is BF16;
re-quantization for serving is out of scope (llm-compressor/modelopt).
- CLI: new --trust-remote-code flag; help text documents auto-detection.
Validated end-to-end on 1x A100-80GB (see PR description):
Nemotron-3-Nano-Omni-30B NVFP4 (mixed) and FP8, Qwen3-8B-FP8
(block-wise) vs Qwen3-8B BF16 baseline (perplexity 4.23 vs 4.33).
Salvage the still-relevant functional work from PR #48: add non-UTF-8 console fallbacks, use platform temporary directories, make pipeline log output encoding-safe, and defer heavyweight analysis imports. The obsolete contributed CI workflow and already-corrected remote URL are intentionally excluded.
Add a --gpu-memory-utilization flag (0.0-1.0, default 0.85) that controls
the fraction of GPU VRAM available for model loading. Plumbed from CLI
through AbliterationPipeline to load_model's max_memory calculation.
Useful on dedicated GPU setups where the default 15% reserve is wasteful
and causes unnecessary CPU offloading on models that would otherwise fit.
Replace the manual safetensors/dat file materialization in
_gather_state_dict with accelerate's get_state_dict_offloaded_model().
The old code only handled disk-offloaded weights but failed for the 398
CPU-offloaded meta tensors managed by accelerate's AlignDevicesHook.
Pre-move all GPU tensors to CPU before materialization to prevent CUDA
OOM when align_module_device restores non-hooked params to CUDA during
its __exit__ cleanup.
Update test to verify the safety net catches unmaterialized meta tensors
after the accelerate path (the old test checked for a missing offload
directory, which this codepath no longer uses).
Raja Mukerji