OBLITERATUS/docs/complementary_blending.md

Complementary Abliteration Blending

Date: 2026-08-20 Status: Contributor-reported preliminary results; implementation independently tested Authors: OBLITERATUS contributors and maintainers

---

Abstract

This document describes complementary abliteration blending, which combines two compatible abliterated checkpoints via weight-space interpolation. PR #127 did not include the raw benchmark outputs, prompt-level refusal evidence, model revisions, environment capture, or artifact hashes needed to independently reproduce its numerical results. The tables below therefore preserve the contributor's preliminary report as context; they are not independently verified project claims.

---

1. Motivation

The contributor report frames abliteration as a tradeoff between refusal removal and measured capability. Single-direction work such as Arditi et al. motivates the approach, while the reported OBLITERATUS comparison observed lower MMLU for a more aggressive surgery. Raw evidence for the specific refusal and MMLU figures was not included in PR #127.

We hypothesized that different direction-finding methods damage different parts of the model's capability geometry, and that blending their outputs could cancel these damages.

---

2. Method

2.1 Surgery A: Aggressive/SVD

Standard OBLITERATUS aggressive pipeline with SVD-based direction extraction:

obliteratus obliterate $BASE --method aggressive --n-directions 3 \
  --regularization 0.08 --residue-weight 3 --refinement-passes 2 \
  --min-layer-fraction 0.45

Contributor-reported properties: deep refusal removal with measurable capability damage on a small spot check. The proposed explanation—SVD directions overlapping capability subspaces—is a hypothesis that requires activation and controlled model-comparison evidence.

2.2 Surgery B: LEACE

OBLITERATUS aggressive pipeline with LEACE (Linear Erasure of Concept Embeddings) direction method:

obliteratus obliterate $BASE --method aggressive --direction-method leace \
  --n-directions 3 --regularization 0.06 --residue-weight 7 \
  --refinement-passes 3 --min-layer-fraction 0.40

LEACE is a closed-form linear concept-erasure method designed to remove linearly available concept information while minimizing distortion (Belrose et al.). The contributor reported capability retention and weaker output quality for this surgery; the asserted mechanism in the generation pathway was not evidenced in this PR.

2.3 Weight-Space LERP Blend

Simple linear interpolation in weight space:

for key in weight_keys:
    blended[key] = alpha * leace_weights[key] + (1 - alpha) * aggressive_weights[key]

The blend ratio alpha = 0.60 was found by binary search over {0.30, 0.50, 0.55, 0.60, 0.65, 0.70}.

---

3. Contributor-Reported Results (Not Independently Reproduced)

No machine-readable benchmark or refusal-evaluation artifacts accompany these tables. Counts, uncertainty, prompt selection, exact model revisions, and evaluator configuration must be supplied before the results can support a release or research conclusion.

3.1 Blend Ratio Search (15-subject MMLU, 150 questions)

Blend (% LEACE)	Refuse	Usable	MMLU	vs Stock
0% (pure SVD)	0%	100%	84.0%	-2.0pp
30%	0%	90%	83.3%	-2.7pp
50%	0%	100%	84.0%	-2.0pp
55%	0%	80%	84.7%	-1.3pp
60%	0%	100%	86.0%	+0.0pp
65%	0%	—	—	—
70%	0%	80%	86.7%	+0.7pp
100% (pure LEACE)	0%	50%	86.7%	+0.7pp

The contributor selected the 60% blend from this small search. The evidence is insufficient to establish a unique optimum or distinguish the apparent differences from sampling noise.

3.2 Larger Contributor Spot Check (57-subject MMLU, 570 questions)

Model	MMLU	Stderr	vs Stock
Stock Qwen3.8-27B	85.26%	±0.014	—
OBLITERATUS V1 (s51)	81.40%	—	-6.0pp
OBLITERATUS V2 (s78)	86.32%	±0.014	+1.1pp

3.3 Per-Subject Gains (5 questions/subject)

Capability gains span both safety-adjacent and neutral reasoning topics:

Subject	Stock	V2	Delta	Type
College Mathematics	40%	80%	+40pp	Neutral
Formal Logic	40%	60%	+20pp	Neutral
Jurisprudence	60%	80%	+20pp	Sensitive
Business Ethics	80%	100%	+20pp	Sensitive
Professional Law	80%	100%	+20pp	Sensitive

The reported neutral-topic gains motivate a controlled follow-up; five questions per subject are not sufficient to establish capability improvement or rule out sampling variation.

3.4 Real-World Practical Tasks

Test Suite	Stock	V2	Tasks
Basic (8 tasks)	5/8	6/8	Code, SQL, tool calling, JSON, math
Advanced (8 tasks)	7/8	7/8	ReAct agents, async refactor, K8s debug,
			security review, system design

The contributor reported comparable outcomes on this small practical-task set. The tasks and raw outputs were not included, so maintainers have not independently verified that comparison.

---

4. Why It Works

4.1 Complementary Error Cancellation

One hypothesis is that SVD and LEACE make different errors in weight space:

• SVD greedily captures maximum variance directions. Some captured variance encodes capability, not just refusal. This damages specific weight regions.
• LEACE minimizes a linear erasure objective. Whether it leaves specific residue in attention heads or output projections must be measured rather than inferred from output behavior.

Weight-space interpolation may average complementary errors:

• Where SVD damaged capability, LEACE's intact weights dilute the damage
• Where LEACE left refusal residue, SVD's clean weights dilute the residue

4.2 Theoretical Connection to Model Merging

This technique is related to model-merging work such as Model Soups and TIES-Merging, but is applied within the abliteration domain. PR #127 does not measure task-vector orthogonality or error anti-correlation, so those remain testable explanations rather than established properties.

4.3 Capacity Hypothesis

The contributor-reported MMLU difference raises several competing hypotheses. A separate, provenance-gated experiment framework is tracked in issue #132:

1. Activation Rank Analysis — Does effective dimensionality increase after abliteration?
2. Topic Cluster Analysis — Do gains cluster on sensitive topics (hedging) or spread broadly (capacity)?
3. Blend Control — Does blending two identical SVD surgeries also gain MMLU? (Tests regularization hypothesis)
4. Learning Absorption — Does the abliterated model learn new information faster? (Tests freed capacity directly)

The small per-subject report does not distinguish these hypotheses.

---

5. Reproducibility

# Step 1: Aggressive surgery
obliteratus obliterate $BASE --method aggressive --n-directions 3 \
  --regularization 0.08 --residue-weight 3 --refinement-passes 2 \
  --min-layer-fraction 0.45 --output-dir surgery_a

# Step 2: LEACE surgery
obliteratus obliterate $BASE --method aggressive --direction-method leace \
  --n-directions 3 --regularization 0.06 --residue-weight 7 \
  --refinement-passes 3 --min-layer-fraction 0.40 --output-dir surgery_b

# Step 3: Blend
obliteratus blend --model-a surgery_a --model-b surgery_b --alpha 0.6 \
  --config-source a --output blended_model

# Step 4: Validate
lm_eval --model hf --model_args pretrained=blended_model --tasks mmlu

The command requires matching source_model values in both checkpoints' abliteration_metadata.json, identical tensor keys, compatible shapes/dtypes, floating-point weights, and sharded safetensors indexes. For legacy checkpoints whose common lineage was verified out of band, --allow-unverified-lineage is an explicit escape hatch. Output is staged and validated before atomically replacing any existing destination.

---

6. Limitations

• Contributor measurements cover only Qwen3.8-27B; independent validation is pending
• MMLU is a multiple-choice benchmark; gains may not transfer to all downstream tasks
• The 60/40 blend ratio may be model-specific
• repetition_penalty=1.15 is still required for clean generation
• System prompts still reintroduce refusals
• Full 842-corpus validation in progress at time of writing
• The numerical results lack committed raw evidence and independent reproduction
• Single-file and quantized/integer checkpoints are not supported by the current blender
• Atomic promotion temporarily requires space for the complete staged output and any prior output

---

7. Future Work

• Cross-architecture validation on Llama, Gemma, Mistral
• SLERP blending instead of LERP (spherical interpolation may better preserve weight norms)
• Three-way blends with additional direction methods (diff_means, SOM)
• Post-blend recovery, tracked separately in issue #133
• Formal capacity-hypothesis validation, tracked in issue #132

References

• Arditi et al., Refusal in Language Models Is Mediated by a Single Direction
• Belrose et al., LEACE: Perfect Linear Concept Erasure in Closed Form
• Wortsman et al., Model Soups
• Yadav et al., TIES-Merging