Add a --gpu-memory-utilization flag (0.0-1.0, default 0.85) that controls
the fraction of GPU VRAM available for model loading. Plumbed from CLI
through AbliterationPipeline to load_model's max_memory calculation.
Useful on dedicated GPU setups where the default 15% reserve is wasteful
and causes unnecessary CPU offloading on models that would otherwise fit.
Four bugs prevented bitsandbytes 4-bit quantized models from completing
ablation studies on GPUs with 16GB VRAM:
1. runner.py: quantization parameter was never passed from StudyConfig
to load_model(), so the loader had no idea quantization was enabled.
2. loader.py (max_memory): GPU memory budget was calculated against the
unquantized model size, causing accelerate to offload layers to meta
device even though the quantized model fits comfortably.
Now divides estimate by 4 (4-bit) or 2 (8-bit) before deciding.
3. evaluator.py: empty strings in wikitext dataset caused zero-length
tensors that crashed the forward pass with a reshape error.
Now filters empty/whitespace-only texts and skips empty batches.
4. loader.py (snapshot/restore): snapshot skip decision used unquantized
size estimate, and restore used strict=True which rejects bitsandbytes
metadata keys (.absmax, .quant_map, .quant_state). Now uses quantized
estimate and strict=False.
Tested on RTX 5060 Ti (16GB) with Qwen2.5-Coder-7B-Instruct in 4-bit.
Quick Scan (layer_removal + ffn_ablation) completes all 56 specs.
The snapshot() deepcopy was cloning tensors on their original GPU
devices, doubling VRAM usage. For a 234GB model sharded across 6
A100-80GB GPUs (~39GB each), this left no room for the copy.
Now snapshot stores tensors on CPU and restore() moves them back
to each parameter's current device.
Joseph Magly