Four bugs prevented bitsandbytes 4-bit quantized models from completing
ablation studies on GPUs with 16GB VRAM:
1. runner.py: quantization parameter was never passed from StudyConfig
to load_model(), so the loader had no idea quantization was enabled.
2. loader.py (max_memory): GPU memory budget was calculated against the
unquantized model size, causing accelerate to offload layers to meta
device even though the quantized model fits comfortably.
Now divides estimate by 4 (4-bit) or 2 (8-bit) before deciding.
3. evaluator.py: empty strings in wikitext dataset caused zero-length
tensors that crashed the forward pass with a reshape error.
Now filters empty/whitespace-only texts and skips empty batches.
4. loader.py (snapshot/restore): snapshot skip decision used unquantized
size estimate, and restore used strict=True which rejects bitsandbytes
metadata keys (.absmax, .quant_map, .quant_state). Now uses quantized
estimate and strict=False.
Tested on RTX 5060 Ti (16GB) with Qwen2.5-Coder-7B-Instruct in 4-bit.
Quick Scan (layer_removal + ffn_ablation) completes all 56 specs.
Replace the manual safetensors/dat file materialization in
_gather_state_dict with accelerate's get_state_dict_offloaded_model().
The old code only handled disk-offloaded weights but failed for the 398
CPU-offloaded meta tensors managed by accelerate's AlignDevicesHook.
Pre-move all GPU tensors to CPU before materialization to prevent CUDA
OOM when align_module_device restores non-hooked params to CUDA during
its __exit__ cleanup.
Update test to verify the safety net catches unmaterialized meta tensors
after the accelerate path (the old test checked for a missing offload
directory, which this codepath no longer uses).
The README/CONTRIBUTING examples used `obliteratus aggregate --format ...` but the CLI only accepted `--dir`.
This adds `--format {summary,latex}`, `--metric`, and `--min-runs` to the aggregate command, reuses community LaTeX table generation, and adds CLI parsing tests to align behavior with documented usage.
New `obliteratus gpu-calc` subcommand estimates minimum GPU count from
model params, dtype, and GPU VRAM. Auto-detects param counts from HF
configs including MoE expert structure.
README now covers --dtype, --quantization flags, the gpu-calc command,
and references both in the "Choosing the right setup" table.
Add a comprehensive section covering:
- How model sharding (pipeline parallelism) works and its limitations
- GPU selection via --gpus flag
- Pipeline parallel benchmarks on GPT-OSS-120B across 3-8 A100-80GB GPUs
- Stage-by-stage timing breakdown
- When data parallelism helps (and when it doesn't)
- Remote SSH execution with CLI and YAML examples
- Decision table for choosing the right setup
The snapshot() deepcopy was cloning tensors on their original GPU
devices, doubling VRAM usage. For a 234GB model sharded across 6
A100-80GB GPUs (~39GB each), this left no room for the copy.
Now snapshot stores tensors on CPU and restore() moves them back
to each parameter's current device.
When --data-parallel is passed and the model fits on a single GPU,
wraps it with nn.DataParallel to split prompt batches across all
available GPUs during activation collection. Batch size scales by
GPU count. Hooks already move activations to CPU so they work
correctly across replicas.
Adds --gpus flag to obliterate, run, and tourney commands for controlling
which GPUs to use (sets CUDA_VISIBLE_DEVICES). Works both locally and with
--remote. Models are automatically split across selected GPUs via
accelerate's device_map="auto". Also adds gpus field to remote YAML config.
Joseph Magly