Merge pull request #137 from elder-plinius/docs/3-winsorization-rationale

docs(research): record BOS winsorization rationale
This commit is contained in:
Joseph Magly
2026-08-21 18:14:04 -04:00
committed by GitHub
+10
View File
@@ -266,6 +266,16 @@ This decomposes weight matrices into **magnitude and direction**, modifies only
- **32-bit floating point** for all intermediate calculations, even for models stored in bfloat16. Using bfloat16 for intermediates led to suboptimal results.
- Winsorization strength was determined empirically.
**Post-publication mechanistic note:** After Gemma Scope 2 12B became
available, grimjim reported that the beginning-of-sequence (BOS) token was
strongly activated and could drown out the other measured activations in Gemma
3 12B. This offers a plausible model-specific explanation for the earlier
empirical result: winsorization limits the leverage of that high-magnitude
outlier so weaker harmful-versus-harmless activation differences remain visible
to the direction estimator. This is an author-reported historical observation,
not an independently reproduced OBLITERATUS result; see [issue
#3](https://github.com/elder-plinius/OBLITERATUS/issues/3).
### 3.6 Multi-Layer Intervention Rationale (The Ouroboros Effect)
When individual layers are ablated, other layers **adaptively compensate to restore approximately 70%** of the original computation (per McGrath et al.'s self-repair findings). This self-repair mechanism — the Ouroboros effect, named for the serpent that consumes itself to be reborn — explains why single-layer interventions are insufficient.