diff --git a/docs/RESEARCH_SURVEY.md b/docs/RESEARCH_SURVEY.md index 1a511dc..5788206 100644 --- a/docs/RESEARCH_SURVEY.md +++ b/docs/RESEARCH_SURVEY.md @@ -266,6 +266,16 @@ This decomposes weight matrices into **magnitude and direction**, modifies only - **32-bit floating point** for all intermediate calculations, even for models stored in bfloat16. Using bfloat16 for intermediates led to suboptimal results. - Winsorization strength was determined empirically. +**Post-publication mechanistic note:** After Gemma Scope 2 12B became +available, grimjim reported that the beginning-of-sequence (BOS) token was +strongly activated and could drown out the other measured activations in Gemma +3 12B. This offers a plausible model-specific explanation for the earlier +empirical result: winsorization limits the leverage of that high-magnitude +outlier so weaker harmful-versus-harmless activation differences remain visible +to the direction estimator. This is an author-reported historical observation, +not an independently reproduced OBLITERATUS result; see [issue +#3](https://github.com/elder-plinius/OBLITERATUS/issues/3). + ### 3.6 Multi-Layer Intervention Rationale (The Ouroboros Effect) When individual layers are ablated, other layers **adaptively compensate to restore approximately 70%** of the original computation (per McGrath et al.'s self-repair findings). This self-repair mechanism — the Ouroboros effect, named for the serpent that consumes itself to be reborn — explains why single-layer interventions are insufficient.