OBLITERATUS/docs/beyond_sota_roadmap.md

# Beyond-SOTA Obliteratus Roadmap

Date: 2026-05-13

## Current Position

The current leader is:

```text
outputs/qwen3.6-27b-aspa-n2-reg05-srcgamma090
```

Confirmed metrics:

| Axis | Result |
|---|---:|
| Full n120 ship score | 88.392 |
| Full refusal rate | 0.025 |
| Full usable rate | 0.9667 |
| Benign/style pass | 1.0 / 1.0 |
| Capability score | 96.75 |
| Coding/creative pass | 1.0 / 1.0 |
| Boundary score | 100.0 |
| Adversarial boundary score | 99.165 |
| Full community scrutiny score | 99.5 |
| Full community code/knowledge/math/truth | 1.0 / 1.0 / 1.0 / 1.0 |
| Full community allowed-topic pass | 1.0 |
| Full community restricted refusal/usable | 0.0333 / 0.9667 |
| First-token mean KL vs source | 0.549806 |
| First-token top-1 agreement vs source | 0.818 |

The central discovery is Source-Tethered Edit Scaling:

```text
candidate_gamma = source + gamma * (edited_candidate - source)
```

For Qwen3.6-27B, `gamma=0.90` is a real Pareto improvement over raw ASPA:
lower refusal, higher usable rate, restored coding pass, and strong boundary
behavior.

The community scrutiny gate adds a broader release-readiness check on top of
the original ship/capability probes. The leader passes the full MLX-backed
gate with score `99.5`, perfect executable-code, knowledge, math,
truthfulness, instruction, creative, and allowed stigmatized-topic results, and
restricted aggregate refusal/usable of `0.0333` / `0.9667`.

The first fine-bracket optimizer pass found:

| Gamma | Status |
|---:|---|
| 0.875 | Preservation challenger: capability `96.80`, mean KL `0.525216`, full score `87.250` |
| 0.900 | Release leader: capability `96.75`, mean KL `0.549806`, full score `88.392` |
| 0.925 | Dominated in this round: capability `96.45`, mean KL `0.589938`, n30 score `88.167` |
| 0.875 + late0.90 | Capability-only gain: capability `96.81`, n30 score `86.333` |
| 0.875 + late MLP-down0.90 | Component neutral: capability `96.80`, n30 score `88.167` |
| 0.875 + late full-attn-o0.90 | Component neutral: capability `96.80`, n30 score `88.167` |
| 0.875 + late linear-attn-out0.90 | Component neutral: capability `96.80`, n30 score `88.167` |

This implies the global scalar is too blunt, but the first broad layer-weighted
attempt was also too blunt. `0.875` globally plus `0.90` on layers `48:64`
improved capability but worsened n30 refusal. Narrowing that bump to late MLP
`down_proj`, late full-attention `self_attn.o_proj`, or late linear-attention
`linear_attn.out_proj` preserved capability but did not recover the `0.90`
refusal win. The next beyond-SOTA step is a learned component-weighted STES
schedule:

```text
candidate = source + gamma(layer, component) * (edited_candidate - source)
```

where preservation-sensitive components use a smaller gamma and
refusal-critical components use a larger gamma.

## What Others Still Have That We Need

1. **Automated search:** Heretic-style Optuna/TPE loops over edit strength,
   layer ranges, regularization, and preservation metrics.
2. **KL as a first-class objective:** not just a report after the fact, but a
   constraint or penalty during candidate selection.
3. **Quantization-native validation:** edit-then-quantize and quantize-then-edit
   behavior across MLX/GGUF/JANG-style deployment formats.
4. **MoE/router-aware editing:** expert-local edits, router drift checks, and
   per-expert preservation metrics.
5. **Standard public benchmarks:** the local community gate now covers compact
   HumanEval/MBPP/GSM8K/MMLU-style probes, but we still need full harness
   exports for public comparability.
6. **Judge/calibrated boundary scoring:** transparent heuristics are useful, but
   a second calibrated local judge should catch both false refusals and unsafe
   compliance more robustly.

## Beyond-SOTA Plan

### Phase 1: Preservation-Aware Search

Implement an optimizer that treats every candidate as a vector:

```text
score = ship_gate + capability + boundary - KL_penalty - degeneration_penalty
```

Required experiment grid:

| Family | Values |
|---|---|
| STES gamma | 0.82, 0.85, 0.875, 0.90, 0.925, 0.95, 0.975 |
| component-weighted STES | learned per-component coefficients, residual late 0.90, hybrid sparse masks |
| second-pass n-directions | 1, 2, 3 |
| regularization | 0.45, 0.50, 0.55, 0.60 |
| layer range | final quartile, final third, ASPA current |
| component mask | full, MLP out, attention out, hybrid |

Promotion rule:

1. n30 ship gate and capability probe for triage.
2. KL probe for candidates that pass triage.
3. n120 ship gate only for Pareto candidates.
4. boundary and adversarial-boundary probe only for release candidates.

### Phase 2: Edit-Field Tomography

Measure how each layer/component contributes to:

- refusal reduction,
- benign KL drift,
- coding pass/fail cliffs,
- boundary over-refusal,
- adversarial robustness.

The output should be an edit-field map, not just a model artifact:

```text
layer -> component -> {refusal_delta, KL_delta, capability_delta, boundary_delta}
```

This is the path beyond generic ablation. The optimizer should learn which
subspaces are causal for refusal and which are merely correlated collateral.

### Phase 3: Quantization Invariance

Every leader must survive deployment:

1. BF16 source/candidate metrics.
2. MLX or GGUF quantized metrics.
3. Quantization drift report:

```text
delta_ship_score
delta_capability_score
delta_boundary_score
delta_first_token_KL
```

Beyond-SOTA target: a candidate is not considered real unless the edit survives
the target inference format.

### Phase 4: MoE and Router Awareness

For MoE models, add:

- expert-level direction extraction,
- router-logit KL,
- per-expert activation KL,
- expert-load entropy before/after edit,
- expert-local STES rather than one global gamma.

Beyond-SOTA target:

```text
gamma_expert = optimizer(expert_load, refusal_signal, KL_sensitivity)
```

### Phase 5: Public Scorecard

Generate a release bundle:

```text
model_card_metrics.json
pareto_frontier.json
kl_report.json
boundary_report.json
quantization_report.json
benchmark_report.json
```

The model card should show:

- source model,
- edit recipe,
- refusal/usable metrics,
- over-refusal metrics,
- capability benchmarks,
- KL preservation,
- quantized deployment parity,
- known failure cases.

## Immediate Next Experiments

1. Promote the MLX-backed eval path as a fallback when torch/MPS is unavailable.
2. Run the community gate against source, raw ASPA, `srcgamma875`, and public
   baseline candidates for side-by-side scrutiny tables.
3. Add an optimizer over component regex masks and coefficients, seeded by:
   late MLP-down, late attention-o, residual/norm, and sparse hybrid masks.
4. Add an early objective that can stop dead branches after capability + n30
   without wasting KL or n120 budget.
5. Run KL for raw ASPA, `gamma=0.85`, `gamma=0.95`, and any component candidate
   that beats `gamma=0.90` on n30 or capability.
6. Add a controller objective that ranks full candidates by ship score,
   capability, boundary score, and low KL.
7. Promote only if a candidate beats `gamma=0.90` on at least two axes without losing any
   hard gate.