Retain the unsuperseded macOS RAM-detection contribution from PR #13 while making every optional probe fail closed. The shared MPS device, loader, cache, and dtype work is already present on current main and is intentionally not duplicated.
Salvage the still-relevant functional work from PR #48: add non-UTF-8 console fallbacks, use platform temporary directories, make pipeline log output encoding-safe, and defer heavyweight analysis imports. The obsolete contributed CI workflow and already-corrected remote URL are intentionally excluded.
Add a --gpu-memory-utilization flag (0.0-1.0, default 0.85) that controls
the fraction of GPU VRAM available for model loading. Plumbed from CLI
through AbliterationPipeline to load_model's max_memory calculation.
Useful on dedicated GPU setups where the default 15% reserve is wasteful
and causes unnecessary CPU offloading on models that would otherwise fit.
Four bugs prevented bitsandbytes 4-bit quantized models from completing
ablation studies on GPUs with 16GB VRAM:
1. runner.py: quantization parameter was never passed from StudyConfig
to load_model(), so the loader had no idea quantization was enabled.
2. loader.py (max_memory): GPU memory budget was calculated against the
unquantized model size, causing accelerate to offload layers to meta
device even though the quantized model fits comfortably.
Now divides estimate by 4 (4-bit) or 2 (8-bit) before deciding.
3. evaluator.py: empty strings in wikitext dataset caused zero-length
tensors that crashed the forward pass with a reshape error.
Now filters empty/whitespace-only texts and skips empty batches.
4. loader.py (snapshot/restore): snapshot skip decision used unquantized
size estimate, and restore used strict=True which rejects bitsandbytes
metadata keys (.absmax, .quant_map, .quant_state). Now uses quantized
estimate and strict=False.
Tested on RTX 5060 Ti (16GB) with Qwen2.5-Coder-7B-Instruct in 4-bit.
Quick Scan (layer_removal + ffn_ablation) completes all 56 specs.
Replace the manual safetensors/dat file materialization in
_gather_state_dict with accelerate's get_state_dict_offloaded_model().
The old code only handled disk-offloaded weights but failed for the 398
CPU-offloaded meta tensors managed by accelerate's AlignDevicesHook.
Pre-move all GPU tensors to CPU before materialization to prevent CUDA
OOM when align_module_device restores non-hooked params to CUDA during
its __exit__ cleanup.
Update test to verify the safety net catches unmaterialized meta tensors
after the accelerate path (the old test checked for a missing offload
directory, which this codepath no longer uses).
The README/CONTRIBUTING examples used `obliteratus aggregate --format ...` but the CLI only accepted `--dir`.
This adds `--format {summary,latex}`, `--metric`, and `--min-runs` to the aggregate command, reuses community LaTeX table generation, and adds CLI parsing tests to align behavior with documented usage.
New `obliteratus gpu-calc` subcommand estimates minimum GPU count from
model params, dtype, and GPU VRAM. Auto-detects param counts from HF
configs including MoE expert structure.
README now covers --dtype, --quantization flags, the gpu-calc command,
and references both in the "Choosing the right setup" table.
The snapshot() deepcopy was cloning tensors on their original GPU
devices, doubling VRAM usage. For a 234GB model sharded across 6
A100-80GB GPUs (~39GB each), this left no room for the copy.
Now snapshot stores tensors on CPU and restore() moves them back
to each parameter's current device.
When --data-parallel is passed and the model fits on a single GPU,
wraps it with nn.DataParallel to split prompt batches across all
available GPUs during activation collection. Batch size scales by
GPU count. Hooks already move activations to CPU so they work
correctly across replicas.
Adds --gpus flag to obliterate, run, and tourney commands for controlling
which GPUs to use (sets CUDA_VISIBLE_DEVICES). Works both locally and with
--remote. Models are automatically split across selected GPUs via
accelerate's device_map="auto". Also adds gpus field to remote YAML config.
Adds --remote [user@]host flag to obliterate, run, and tourney commands,
enabling execution on remote GPU nodes via SSH. Also supports a remote:
section in YAML configs. The remote runner handles SSH connectivity checks,
GPU detection, auto-installation of obliteratus, log streaming, and result
syncing back to the local machine via scp.
Joseph Magly