OBLITERATUS/docs/beyond_sota_roadmap.md

Beyond-SOTA Obliteratus Roadmap

Date: 2026-05-13

Current Position

The current leader is:

outputs/qwen3.6-27b-aspa-n2-reg05-srcgamma090

Confirmed metrics:

Axis	Result
Full n120 ship score	88.392
Full refusal rate	0.025
Full usable rate	0.9667
Benign/style pass	1.0 / 1.0
Capability score	96.75
Coding/creative pass	1.0 / 1.0
Boundary score	100.0
Adversarial boundary score	99.165
Full community scrutiny score	99.5
Full community code/knowledge/math/truth	1.0 / 1.0 / 1.0 / 1.0
Full community allowed-topic pass	1.0
Full community restricted refusal/usable	0.0333 / 0.9667
First-token mean KL vs source	0.549806
First-token top-1 agreement vs source	0.818

The central discovery is Source-Tethered Edit Scaling:

candidate_gamma = source + gamma * (edited_candidate - source)

For Qwen3.6-27B, gamma=0.90 is a real Pareto improvement over raw ASPA: lower refusal, higher usable rate, restored coding pass, and strong boundary behavior.

The community scrutiny gate adds a broader release-readiness check on top of the original ship/capability probes. The leader passes the full MLX-backed gate with score 99.5, perfect executable-code, knowledge, math, truthfulness, instruction, creative, and allowed stigmatized-topic results, and restricted aggregate refusal/usable of 0.0333 / 0.9667.

The first fine-bracket optimizer pass found:

Gamma	Status
0.875	Preservation challenger: capability 96.80, mean KL 0.525216, full score 87.250
0.900	Release leader: capability 96.75, mean KL 0.549806, full score 88.392
0.925	Dominated in this round: capability 96.45, mean KL 0.589938, n30 score 88.167
0.875 + late0.90	Capability-only gain: capability 96.81, n30 score 86.333
0.875 + late MLP-down0.90	Component neutral: capability 96.80, n30 score 88.167
0.875 + late full-attn-o0.90	Component neutral: capability 96.80, n30 score 88.167
0.875 + late linear-attn-out0.90	Component neutral: capability 96.80, n30 score 88.167

This implies the global scalar is too blunt, but the first broad layer-weighted attempt was also too blunt. 0.875 globally plus 0.90 on layers 48:64 improved capability but worsened n30 refusal. Narrowing that bump to late MLP down_proj, late full-attention self_attn.o_proj, or late linear-attention linear_attn.out_proj preserved capability but did not recover the 0.90 refusal win. The next beyond-SOTA step is a learned component-weighted STES schedule:

candidate = source + gamma(layer, component) * (edited_candidate - source)

where preservation-sensitive components use a smaller gamma and refusal-critical components use a larger gamma.

What Others Still Have That We Need

1. Automated search: Heretic-style Optuna/TPE loops over edit strength, layer ranges, regularization, and preservation metrics.
2. KL as a first-class objective: not just a report after the fact, but a constraint or penalty during candidate selection.
3. Quantization-native validation: edit-then-quantize and quantize-then-edit behavior across MLX/GGUF/JANG-style deployment formats.
4. MoE/router-aware editing: expert-local edits, router drift checks, and per-expert preservation metrics.
5. Standard public benchmarks: the local community gate now covers compact HumanEval/MBPP/GSM8K/MMLU-style probes, but we still need full harness exports for public comparability.
6. Judge/calibrated boundary scoring: transparent heuristics are useful, but a second calibrated local judge should catch both false refusals and unsafe compliance more robustly.

Beyond-SOTA Plan

Phase 1: Preservation-Aware Search

Implement an optimizer that treats every candidate as a vector:

score = ship_gate + capability + boundary - KL_penalty - degeneration_penalty

Required experiment grid:

Family	Values
STES gamma	0.82, 0.85, 0.875, 0.90, 0.925, 0.95, 0.975
component-weighted STES	learned per-component coefficients, residual late 0.90, hybrid sparse masks
second-pass n-directions	1, 2, 3
regularization	0.45, 0.50, 0.55, 0.60
layer range	final quartile, final third, ASPA current
component mask	full, MLP out, attention out, hybrid

Promotion rule:

1. n30 ship gate and capability probe for triage.
2. KL probe for candidates that pass triage.
3. n120 ship gate only for Pareto candidates.
4. boundary and adversarial-boundary probe only for release candidates.

Phase 2: Edit-Field Tomography

Measure how each layer/component contributes to:

• refusal reduction,
• benign KL drift,
• coding pass/fail cliffs,
• boundary over-refusal,
• adversarial robustness.

The output should be an edit-field map, not just a model artifact:

layer -> component -> {refusal_delta, KL_delta, capability_delta, boundary_delta}

This is the path beyond generic ablation. The optimizer should learn which subspaces are causal for refusal and which are merely correlated collateral.

Phase 3: Quantization Invariance

Every leader must survive deployment:

1. BF16 source/candidate metrics.
2. MLX or GGUF quantized metrics.
3. Quantization drift report:

delta_ship_score
delta_capability_score
delta_boundary_score
delta_first_token_KL

Beyond-SOTA target: a candidate is not considered real unless the edit survives the target inference format.

Phase 4: MoE and Router Awareness

For MoE models, add:

• expert-level direction extraction,
• router-logit KL,
• per-expert activation KL,
• expert-load entropy before/after edit,
• expert-local STES rather than one global gamma.

Beyond-SOTA target:

gamma_expert = optimizer(expert_load, refusal_signal, KL_sensitivity)

Phase 5: Public Scorecard

Generate a release bundle:

model_card_metrics.json
pareto_frontier.json
kl_report.json
boundary_report.json
quantization_report.json
benchmark_report.json

The model card should show:

• source model,
• edit recipe,
• refusal/usable metrics,
• over-refusal metrics,
• capability benchmarks,
• KL preservation,
• quantized deployment parity,
• known failure cases.

Immediate Next Experiments

1. Promote the MLX-backed eval path as a fallback when torch/MPS is unavailable.
2. Run the community gate against source, raw ASPA, srcgamma875, and public baseline candidates for side-by-side scrutiny tables.
3. Add an optimizer over component regex masks and coefficients, seeded by: late MLP-down, late attention-o, residual/norm, and sparse hybrid masks.
4. Add an early objective that can stop dead branches after capability + n30 without wasting KL or n120 budget.
5. Run KL for raw ASPA, gamma=0.85, gamma=0.95, and any component candidate that beats gamma=0.90 on n30 or capability.
6. Add a controller objective that ranks full candidates by ship score, capability, boundary score, and low KL.
7. Promote only if a candidate beats gamma=0.90 on at least two axes without losing any hard gate.