transformers 5 stores routed experts as fused 3D parameters
(experts.gate_up_proj / experts.down_proj) for Mixtral, Qwen3-MoE,
DeepSeek-V3, GLM-4 MoE, Llama 4, gpt-oss and OLMoE. When such a layer is
CPU- or disk-offloaded by Accelerate, EXCISE failed closed with
"offloaded fused expert tensors are not yet a supported surgery layout",
so every frontier MoE checkpoint large enough to need offload could not be
abliterated at all.
Route the three fused projections and the fused bias projection through the
existing LogicalParameterTransaction adapter: a meta-resident fused
parameter is resolved to a private copy of its authoritative backing value,
projected per expert, and committed atomically; the live parameter stays on
meta and quantized or unknown backing layouts still fail before mutation.
Thread offload_roots through the MoE dispatch call sites so parent-prefixed
Accelerate hooks resolve, and drop the preflight rejection of 3D meta
parameters (validate_offloaded_parameters already resolves them).
Tests: fused 3D, bias, granular and selective-inversion projections on
offloaded backing stores, bounded norm restoration, parent-prefixed hook
resolution, quantized fail-closed, commit-failure rollback, and a full
offline pipeline run on a disk-offloaded tiny Mixtral fixture that verifies
the saved checkpoint carries the updated experts with no meta tensors.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Joseph Magly