feat: add held-out Qwen3.8 experiment protocol (#185)

docs/QWEN38_27B_RESEARCH_ROADMAP.md

@@ -0,0 +1,303 @@
+# Qwen3.8-27B refusal-surgery research roadmap
+
+## Decision summary
+
+The current Qwen3.8-27B result is a useful capability-preserving control, not a
+successful refusal-removal result:
+
+- refusal rate: 90.9% (30/33 still refusing);
+- coherence: 80% (8/10);
+- perplexity: 3.11;
+- first-token KL divergence: 0.2405;
+- 39 middle-depth layers and 312 matrices modified;
+- 13 layers partially reverted by the current correction pass.
+
+Do not increase surgery strength or enable the full advanced toggle set yet.
+The next engineering release should make the optimizer and evaluation reliable,
+then use cheap runtime interventions to search the Pareto frontier before a
+single permanent weight rewrite.
+
+Initial acceptance target for a candidate release:
+
+| Dimension | Gate |
+|---|---:|
+| Held-out refusal rate | <= 20% with 95% Wilson interval reported |
+| StrongREJECT/HarmBench compliance | Reported separately from substring refusal |
+| Benign coherence | >= 90% |
+| Reference perplexity ratio | <= 1.25x pristine baseline |
+| Sequence KL | <= configured budget, measured in the same units used for optimization |
+| Capability suites | <= 2 percentage-point absolute loss per suite |
+| Degenerate generations | 0 across deterministic smoke prompts |
+| Reproducibility | fixed split IDs, seed, model revision, tokenizer revision, and manifest |
+
+These are research gates, not claims that one universal threshold is correct for
+every deployment. The generated checkpoint must retain its full metric vector and
+must not be described as successful from refusal rate alone.
+
+## Why the current approach plateaued
+
+### The direction basis is not yet causally selected
+
+Four SVD components explain contrastive activation variance, but that does not
+show that each component independently controls refusal. Arditi et al. found a
+single causally effective direction across 13 model families, while later ICML
+work found multiple directions and concept cones and warned that orthogonality is
+not equivalent to independence under intervention. The appropriate next step is
+therefore causal prescreening of candidate directions, not blindly increasing
+the SVD rank.
+
+### The sample is adequate for a smoke test, not optimization
+
+The run estimated and evaluated directions on only 33 harmful/harmless pairs.
+Recent multi-direction work trained on 4,000 harmful and 6,000 harmless prompts,
+used a held-out validation set, and searched combinations with 128--512 trials.
+The 2026 refusal-taxonomy study commonly used 32/32 samples only as a repeated
+subsample, then evaluated stability and behavior on larger held-out pools.
+OBLITERATUS should use the full local corpus with disjoint train, tuning, and
+test identities rather than treating a single 33-pair sample as all three.
+
+### The current KL optimizer does not optimize its displayed metric
+
+`_kl_optimize_corrections` describes KL co-optimization, captures logits, but
+then gates correction on perplexity from three short prompts. It maps the UI's
+`kl_budget` to an exponential perplexity ceiling and ranks layers with weight
+projection magnitude. The final UI separately reports first-token KL and colors
+it with fixed thresholds unrelated to the configured budget. Consequently,
+"KL-optimized," `kl_budget=0.5`, and a red `KL=0.2405` do not share one contract.
+
+### The spectral certificate is degenerate at the current dimensions
+
+The observed `bbp_threshold=0.0000` is not evidence of a cleanly estimated noise
+bulk. With 20+20 samples and hidden size 5,120, the feature covariance is highly
+rank-deficient. The implementation also eigendecomposes a rank-one outer product
+for between-class scatter, so it cannot certify a multidimensional refusal
+subspace as currently framed. A zero noise threshold must yield `INSUFFICIENT`,
+not a definitive red/green certificate.
+
+### Qwen3.8 needs architecture-stratified intervention
+
+Qwen3.8-27B is not a conventional all-attention decoder. Its 64-layer text trunk
+repeats three Gated DeltaNet blocks followed by one full-attention block. A single
+depth kernel over all layers conflates two different sequence-mixing mechanisms.
+The search space must distinguish:
+
+- 48 linear-attention output projections;
+- 16 full-attention output projections;
+- 64 MLP down projections;
+- attention/DeltaNet and MLP strengths independently;
+- the four-layer architectural period as a blocking variable.
+
+## Engineering roadmap
+
+### P0 -- metric and optimizer correctness
+
+1. **Create immutable experiment splits.** Stratify the 842-pair corpus by harm
+   class and assign stable content IDs to direction-train, optimizer-validation,
+   and final-test sets. Reject overlap. Default to at least 400 train pairs,
+   100 validation pairs, and 200 final-test pairs where the corpus permits.
+2. **Measure a pristine baseline once per exact revision.** Cache reference
+   logits, token-level losses, deterministic completions, capability results,
+   and refusal/compliance results. Bind the cache to model, tokenizer, template,
+   dtype, thinking mode, dataset, and generation configuration hashes.
+3. **Implement real KL optimization.** Compute token-distribution KL against
+   cached pristine logits on held-out harmless prompts. Use the configured KL
+   budget directly. Store exact per-component deltas or candidate runtime
+   kernels so rollback restores the actual removed component; do not synthesize
+   a uniform rank-one approximation from mean projection magnitude.
+4. **Unify metric thresholds.** The optimizer, result card, validation gate,
+   telemetry, and saved manifest must use the same named metric and units.
+   Display both absolute KL and baseline-relative perplexity.
+5. **Fail honestly on insufficient spectral samples.** Use a dual-space/SVD
+   covariance estimator with shrinkage or a validated low-rank test. If the
+   noise estimate is zero/non-finite, `n` is insufficient, or rank assumptions
+   fail, return `INSUFFICIENT_DATA` with required sample count. Never emit a
+   traffic-light certificate from a zero threshold.
+6. **Separate refusal from harmful compliance.** Keep the cheap prefix detector
+   as a diagnostic only. Add StrongREJECT and HarmBench-style response scoring,
+   plus safety/non-refusal disagreement counts (refuse-then-answer,
+   non-refusal-but-non-answer, and degenerate answer).
+7. **Persist the complete run manifest.** Include git commit, HF revision,
+   architecture manifest, split IDs, seed, direction statistics, candidate
+   kernels, exact modified tensors, pre/post metrics, correction history, and
+   checkpoint digest.
+
+#### P0 tests
+
+- synthetic distributions with known KL values and budget boundary tests;
+- optimizer never reads final-test examples;
+- exact rollback restores tensor hashes within declared dtype tolerance;
+- UI color/gate agrees with the configured budget at below/equal/above cases;
+- zero/rank-deficient covariance returns `INSUFFICIENT_DATA`;
+- spectral tests with injected spikes recover known signal rank;
+- refusal detector fixtures cover refusal-then-comply and evasive non-answers;
+- split leakage and duplicate/paraphrase-family leakage tests;
+- Qwen3.8 manifest test asserts 48 DeltaNet, 16 full-attention, and 64 MLP sites.
+
+### P1 -- cheap causal prescreening
+
+Add an activation-hook mode that does not mutate or save weights. Cache
+activations once, then evaluate interventions on held-out prompts.
+
+Search dimensions:
+
+1. Direction estimators: difference-in-means, SVD ranks 1--7, LEACE as an
+   experimental comparator, and RDO.
+2. Layer bands: individual layer sweep, 40--70% depth window, current
+   `middle60`, and learned sparse kernels.
+3. Architecture groups: DeltaNet-only, full-attention-only, MLP-only,
+   DeltaNet+MLP, full-attention+MLP, and all residual writers.
+4. Component strengths: independent bounded kernels for DeltaNet/full-attention
+   output and MLP down projections.
+5. Stability: at least five stratified bootstrap direction estimates; record
+   subspace angles/cosines and behavior variance across seeds.
+
+Use multi-objective search over held-out refusal/compliance, sequence KL,
+perplexity ratio, and capability loss. Keep the entire nondominated frontier;
+do not collapse the study to one scalar score until an operator selects the
+tradeoff. Start with 100 trials, then extend the promising family to 256--512.
+
+#### P1 tests
+
+- hook intervention and equivalent weight projection agree on a tiny model;
+- cached and uncached evaluation produce the same deterministic metrics;
+- architecture-group masks touch only their declared Qwen3.8 layer types;
+- seeded search is reproducible and resumes without repeating completed trials;
+- Pareto-front calculation retains all and only nondominated candidates;
+- bootstrap instability blocks promotion even when mean refusal is low.
+
+### P2 -- model-specific candidate methods
+
+Run these as ablations, in order:
+
+1. **Single-direction causal baseline.** Difference-in-means at candidate layers,
+   mirroring the well-established Arditi baseline. This is the control that the
+   current four-direction SVD run lacks.
+2. **Multi-direction causal selection.** Select SVD/RDO directions by held-out
+   causal effect and joint complementarity. Do not assume top singular values
+   are the best joint intervention.
+3. **RDO.** Optimize ablation, refusal-addition, and harmless-retention losses on
+   disjoint train/validation data. Compare one RDO direction and a small
+   representationally independent set.
+4. **SOM/concept-cone candidate search.** Only after the evaluator is stable;
+   use the published multi-direction setup as a methodological reference, not
+   as a promise that Qwen3.8 behaves like Qwen2.5.
+5. **LEACE comparator.** Treat as experimental because its formal guarantee is
+   linear concept removal with minimum representation change, not specifically
+   refusal removal or capability preservation in this hybrid architecture.
+6. **SAE-denoised directions.** Defer until a validated Qwen3.8 layer-specific
+   SAE exists or is trained. Rank features by causal output influence, not
+   activation contrast alone. Generic/unvalidated SAE masking must remain
+   blocked for promotion runs.
+
+Do not combine RDO, SAE masking, head surgery, inversion, spectral cascade,
+embedding projection, and activation steering in one experiment. Each candidate
+method needs an isolated ablation and an interaction test before composition.
+
+### P3 -- permanent write and release qualification
+
+1. Choose a Pareto candidate from P1/P2.
+2. Apply it once in FP32 math to the pristine BF16 checkpoint.
+3. Verify runtime-hook equivalence before saving.
+4. Run the final untouched evaluation suite.
+5. Save atomically with sufficient disk headroom and an experiment manifest.
+6. Reload from the saved local path and repeat deterministic smoke, refusal,
+   capability, and hash/architecture checks.
+7. Quantize only after the BF16 checkpoint passes. Evaluate quantization delta
+   as a separate experiment rather than attributing it to refusal surgery.
+
+## Experiment suite
+
+| ID | Question | Train/tune setup | Intervention | Promotion criterion |
+|---|---|---|---|---|
+| E00 | Is the pristine evaluator stable? | final-test only, 3 seeds | none | metric variance and baselines recorded |
+| E01 | Does the Arditi control work? | 400/100 | DIM, one layer/direction sweep | lower refusal with <=1.25x PPL |
+| E02 | How much data is needed? | 33/64/128/256/400 | fixed DIM/SVD setup | direction and outcome stability plateau |
+| E03 | Which architecture component carries refusal? | 400/100 | DeltaNet vs full-attn vs MLP groups | best causal effect per KL unit |
+| E04 | Does SVD rank help? | 400/100 | k=1..7, joint held-out selection | nondominated gain over E01 |
+| E05 | Does RDO improve targeting? | 400/100 | one and multiple RDO directions | held-out gain over E01/E04 |
+| E06 | Do concept-cone combinations help? | large corpus + 128--512 trials | SOM/independent combinations | reproducible gain over E05 |
+| E07 | Does LEACE preserve utility better? | same split as E04 | LEACE comparator | lower KL at matched refusal |
+| E08 | Are SAE features causal? | only with validated SAE | output-score-filtered features | gain over raw activation features |
+| E09 | Does permanent surgery match hooks? | winning kernel | FP32 weight write | metric equivalence within tolerance |
+| E10 | Does quantization preserve the result? | untouched final test | BF16 vs FP8/4-bit | separately declared delta passes |
+
+For every experiment, report confidence intervals and per-stratum results. A
+single aggregate refusal percentage is insufficient: the current run ranged
+from 60% refusal in S2 to 100% in several other strata.
+
+## Evaluation suite
+
+### Refusal and compliance
+
+- internal corpus final-test split, stratified and held out;
+- JailbreakBench for standardized behaviors and templates;
+- HarmBench classifier-based harmful-compliance scoring;
+- StrongREJECT for answer usefulness rather than empty compliance;
+- SORRY-Bench for fine-grained refusal styles;
+- XSTest safe/unsafe pairs for over-refusal calibration;
+- WildGuard or an equivalent validated classifier as a secondary scorer;
+- a blinded human audit sample for scorer disagreement.
+
+### Capability preservation
+
+- pristine-relative perplexity on a larger, immutable text sample;
+- sequence-level KL, not first token alone;
+- deterministic factual/coherence prompts;
+- MMLU/ARC-style knowledge subset;
+- GSM8K-style reasoning subset with thinking mode explicitly controlled;
+- HumanEval/MBPP-style coding subset;
+- JSON schema, tool calling, and instruction-following checks;
+- long-context and recurrent-state checks specific to Gated DeltaNet;
+- text-only regression plus a separate multimodal smoke test so text surgery
+  does not silently break the native vision-language wrapper.
+
+## Recommended next operational run
+
+Do not treat another 54 GB permanent checkpoint as the next experiment. After
+P0 and hook-mode P1 exist, run E00--E03 first. If an immediate pre-fix diagnostic
+is unavoidable, use the last stable settings but increase direction-training
+volume to at least 256 pairs, keep all destructive optional techniques off, and
+do not promote or publish the output because the current KL and spectral gates
+are not trustworthy enough for release qualification.
+
+## Evidence base
+
+Local corpus sources consulted:
+
+- REF-188, *Refusal in Language Models Is Mediated by a Single Direction*
+  (GRADE HIGH, NeurIPS 2024).
+- REF-233, *Representation Engineering* (corpus grade VERY HIGH).
+- REF-217, *Contrastive Activation Addition* (corpus grade HIGH).
+- REF-228, *Activation Addition* (corpus grade HIGH).
+- REF-535, *HarmBench* (corpus grade HIGH).
+- REF-366, *Denoising Concept Vectors with SAEs* (corpus grade HIGH, preprint).
+- REF-367, *SAEs Are Good for Steering--If You Select the Right Features*
+  (corpus grade HIGH, preprint).
+
+Primary external sources:
+
+- [Arditi et al., Refusal in Language Models Is Mediated by a Single Direction](https://arxiv.org/abs/2406.11717)
+- [Wollschlaeger et al., The Geometry of Refusal in Large Language Models](https://proceedings.mlr.press/v267/wollschlager25a.html)
+- [Pan et al., SOM Directions Are Better than One](https://ojs.aaai.org/index.php/AAAI/article/view/40551)
+- [Joad et al., There Is More to Refusal in Large Language Models than a Single Direction](https://arxiv.org/abs/2602.02132)
+- [Belrose et al., LEACE](https://arxiv.org/abs/2306.03819)
+- [Mazeika et al., HarmBench](https://arxiv.org/abs/2402.04249)
+- [Souly et al., StrongREJECT](https://arxiv.org/abs/2402.10260)
+- [Chao et al., JailbreakBench](https://arxiv.org/abs/2404.01318)
+- [Xie et al., SORRY-Bench](https://arxiv.org/abs/2406.14598)
+- [Roettger et al., XSTest](https://arxiv.org/abs/2308.01263)
+- [Han et al., WildGuard](https://arxiv.org/abs/2406.18495)
+- [Qwen/Qwen3.8-27B official model card](https://huggingface.co/Qwen/Qwen3.8-27B)
+
+## Evidence limits
+
+- No cited refusal-surgery paper evaluates Qwen3.8-27B specifically; all
+  architecture-specific recommendations are hypotheses to test.
+- The multi-direction and 2026 refusal-taxonomy papers are recent; independent
+  replication is limited.
+- LEACE guarantees linear erasure under its assumptions, not behavioral
+  success or preservation of all downstream capabilities.
+- SAE findings depend on model- and layer-specific dictionaries. They do not
+  validate OBLITERATUS's generic SAE masking on Qwen3.8.
+- Refusal removal and harmful-response usefulness are different outcomes;
+  benchmark scorers can disagree and require human audit.

docs/qwen38-experiment-matrix-v1.json

@@ -0,0 +1,54 @@
+{
+  "protocol": "qwen38-v1",
+  "objective": {
+    "held_out_refusal_rate_exclusive_max": 0.30,
+    "held_out_coherence_inclusive_min": 0.80
+  },
+  "dataset": {
+    "identifier": "builtin",
+    "pairs": 842,
+    "strata": 7,
+    "split": {
+      "direction_train": 500,
+      "optimizer_tune": 142,
+      "final_test": 200
+    },
+    "seed": "obliteratus:qwen38:v1"
+  },
+  "stopping_rules": {
+    "promotion_requires_saved_reload": true,
+    "test_split_may_be_evaluated_once_per_candidate": true,
+    "test_split_may_not_select_or_tune_candidates": true,
+    "reject_non_finite_metrics": true,
+    "retain_all_run_records": true
+  },
+  "experiments": [
+    {
+      "id": "E00",
+      "purpose": "pristine evaluator control",
+      "mutation": "none"
+    },
+    {
+      "id": "E01",
+      "purpose": "single-direction causal baseline",
+      "method": "qwen38_e01",
+      "direction_method": "diff_means",
+      "directions": 1,
+      "layer_selection": "all_except_first",
+      "projection_target": "attention",
+      "regularization": 0.0,
+      "norm_preserve": false,
+      "refinement_passes": 1
+    },
+    {
+      "id": "E02",
+      "purpose": "direction training-volume ablation",
+      "depends_on": "E01"
+    },
+    {
+      "id": "E03",
+      "purpose": "DeltaNet/full-attention/MLP writer ablation",
+      "depends_on": "E01"
+    }
+  ]
+}