From 20d98b4ac1dff6fc66b640f64b1e7c884c4f623f Mon Sep 17 00:00:00 2001 From: Joseph Magly <1159087+jmagly@users.noreply.github.com> Date: Fri, 21 Aug 2026 18:12:21 -0400 Subject: [PATCH] docs(research): record BOS winsorization rationale --- docs/RESEARCH_SURVEY.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/RESEARCH_SURVEY.md b/docs/RESEARCH_SURVEY.md index 1a511dc..5788206 100644 --- a/docs/RESEARCH_SURVEY.md +++ b/docs/RESEARCH_SURVEY.md @@ -266,6 +266,16 @@ This decomposes weight matrices into **magnitude and direction**, modifies only - **32-bit floating point** for all intermediate calculations, even for models stored in bfloat16. Using bfloat16 for intermediates led to suboptimal results. - Winsorization strength was determined empirically. +**Post-publication mechanistic note:** After Gemma Scope 2 12B became +available, grimjim reported that the beginning-of-sequence (BOS) token was +strongly activated and could drown out the other measured activations in Gemma +3 12B. This offers a plausible model-specific explanation for the earlier +empirical result: winsorization limits the leverage of that high-magnitude +outlier so weaker harmful-versus-harmless activation differences remain visible +to the direction estimator. This is an author-reported historical observation, +not an independently reproduced OBLITERATUS result; see [issue +#3](https://github.com/elder-plinius/OBLITERATUS/issues/3). + ### 3.6 Multi-Layer Intervention Rationale (The Ouroboros Effect) When individual layers are ablated, other layers **adaptively compensate to restore approximately 70%** of the original computation (per McGrath et al.'s self-repair findings). This self-repair mechanism — the Ouroboros effect, named for the serpent that consumes itself to be reborn — explains why single-layer interventions are insufficient.