Sample 0 of ClearHarm appears in training/eval pretraining data we
want to keep clean of. Replace it with sample 51 to keep N=40.
Assisted-by: Claude <noreply@anthropic.com>
* safeguard_valid: drop 10 samples that trip gpt-oss attention bug
Exclude {2, 4, 5, 12, 14, 15, 28, 33, 44, 47} which crash with a
170-vs-169 size mismatch in eager_attention_forward under the
clearharm template. Matches hmcGCG's safeguard_clearharm_gpt-oss_3e17
sample set.
Assisted-by: Claude <noreply@anthropic.com>
* Add claudini.asr: compute ASR from benchmark results
ASR (Attack Success Rate) is the fraction of runs whose greedy
completion from the best suffix exactly matches every target
token (i.e. match_rate == 1.0). This is the canonical success
metric for the safeguard track, distinct from loss-based ranking.
uv run -m claudini.asr results/ --preset safeguard_valid
Assisted-by: Claude <noreply@anthropic.com>
---------
Co-authored-by: Alexander Panfilov <apanfilov@g003.internal.cluster.is.localnet>
- **PEFT adapter merging.** `model_loader.py` auto-detects PEFT adapters (e.g. `facebook/Meta-SecAlign-8B`), merges on CPU in bf16, and caches the merged model to disk. No config flags needed.
- **Configurable quantization.** `quantization:` field in YAML or `--quantization` on CLI, accepting `nf4`, `fp4`, or `int8`. Replaces the old `load_in_4bit` boolean.
- **Multi-GPU sharding.** `device_map:` in configs or `--device-map` on CLI. Config value is now correctly read from YAML presets (was previously ignored).
- **CLI overrides.** New `--model`, `--device-map`, `--quantization` flags to override preset values from the command line.
- **SecAlign injection presets.** Configs for prompt injection on Meta-SecAlign-8B and 70B (default + Optuna-tuned), using new `AlpacaInjectionSource` — generates 3-role prompts from AlpacaFarm data with trusted/untrusted separation.
- **Fixes.** `BenchmarkRunner.summarize()` crash when all runs are skipped. System prompt suppression now works correctly (`""` suppresses model defaults, `None` omits the turn).
Co-authored-by: Peter Romov <peter@romov.com>
Co-authored-by: Alexander Panfilov <39771221+kotekjedi@users.noreply.github.com>
Add .claude/skills/claudini/SKILL.md to drive the autoresearch loop
via /claudini slash command. Update CLAUDE.md with skill docs. Replace
PROMPT.txt with the skill-based workflow. Rewrite README to feature
the autoresearch loop prominently. Add easy_1e16 and easy_1e17 preset
configs and update safeguard configs.
Assisted-by: Claude <noreply@anthropic.com>
kotekjedi