docs(research): record BOS winsorization rationale

This commit is contained in:
Joseph Magly
2026-08-21 18:12:21 -04:00
parent a7891963d0
commit 20d98b4ac1
+10
View File
@@ -266,6 +266,16 @@ This decomposes weight matrices into **magnitude and direction**, modifies only
- **32-bit floating point** for all intermediate calculations, even for models stored in bfloat16. Using bfloat16 for intermediates led to suboptimal results.
- Winsorization strength was determined empirically.
**Post-publication mechanistic note:** After Gemma Scope 2 12B became
available, grimjim reported that the beginning-of-sequence (BOS) token was
strongly activated and could drown out the other measured activations in Gemma
3 12B. This offers a plausible model-specific explanation for the earlier
empirical result: winsorization limits the leverage of that high-magnitude
outlier so weaker harmful-versus-harmless activation differences remain visible
to the direction estimator. This is an author-reported historical observation,
not an independently reproduced OBLITERATUS result; see [issue
#3](https://github.com/elder-plinius/OBLITERATUS/issues/3).
### 3.6 Multi-Layer Intervention Rationale (The Ouroboros Effect)
When individual layers are ablated, other layers **adaptively compensate to restore approximately 70%** of the original computation (per McGrath et al.'s self-repair findings). This self-repair mechanism — the Ouroboros effect, named for the serpent that consumes itself to be reborn — explains why single-layer interventions are insufficient.