Initial commit

Co-Authored-By: Alexander Panfilov <sasha_pusha@mail.de> Co-Authored-By: Claude <noreply@anthropic.com>
2026-07-13 22:47:18 +02:00 · 2026-03-25 02:09:26 +00:00
commit 5b6058b3c4
759 changed files with 37753 additions and 0 deletions
@@ -0,0 +1,83 @@
+# ⛓️‍💥 Claudini ⛓️‍💥
+
+**Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs**
+
+[![arXiv](https://img.shields.io/badge/arXiv-XXXX.XXXXX-b31b1b.svg?logo=arxiv)](https://arxiv.org/abs/XXXX.XXXXX)
+
+<p align="center">
+  <img src="assets/teaser.png" width="90%" alt="Claude autoresearch vs Optuna hyperparameter search: best train and validation loss over trials">
+</p>
+
+We show that an *[autoresearch](https://github.com/karpathy/autoresearch)*-style pipeline powered by Claude Code discovers novel white-box adversarial attack *algorithms* that **significantly outperform** all existing [methods](claudini/methods/original/README.md) in jailbreaking and prompt injection evaluations.
+
+This official code repository contains a demo autoresearch pipeline, the Claude-discovered methods from the paper, baseline implementations, and the evaluation benchmark. Read our [paper](https://arxiv.org/abs/XXXX.XXXXX) and consider [citing us](#citation) if you find this useful.
+
+## Setup
+
+```bash
+git clone https://github.com/romovpa/claudini.git
+cd claudini
+uv sync
+```
+
+Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).
+
+## Evaluate
+
+All experiments are run via `claudini.run_bench` CLI:
+```bash
+uv run -m claudini.run_bench --help
+```
+
+It takes a preset name (from [`configs/`](configs/)) or a path to a YAML file.
+
+Config settings can be overridden with CLI options. For example, to evaluate methods on the random targets track, override FLOPs budget:
+```bash
+uv run -m claudini.run_bench random_valid --method gcg,acg --max-flops 1e15
+```
+
+Results are saved to `results/<method>/<preset>/<model>/sample_<S>_seed_<N>.json`. Existing results are auto-skipped.
+
+## Run Autoresearch
+
+<p align="center">
+  <img src="assets/autoresearch_loop.png" width="90%" alt="Autoresearch loop: seeding, analysis-experiment cycle, evaluation">
+</p>
+
+Install [Claude Code](https://docs.anthropic.com/en/docs/claude-code), then start the loop:
+
+```bash
+claude
+> /loop 10m $(cat PROMPT.txt)
+```
+
+Or with the [ralph-loop](https://github.com/anthropics/claude-code-ralph-loop) plugin:
+
+```bash
+claude
+> /ralph-loop:ralph-loop $(cat PROMPT.txt)
+```
+
+The loop reads [`PROMPT.txt`](PROMPT.txt), studies existing methods and results, designs a new optimizer, implements it, runs the benchmark, and commits — all autonomously. Use tmux or screen so sessions survive disconnection. Track progress via `git log`.
+
+## Attack Methods
+
+We consider white-box GCG-style attacks that search directly over the model's vocabulary using gradients. Each method ([`TokenOptimizer`](claudini/base.py#L429)) optimizes a short discrete token *suffix* that, when appended to an input prompt, causes the model to produce a desired target sequence.
+
+- **Baselines** (existing methods): [`claudini/methods/original/`](claudini/methods/original/)
+- **Claude-deviced methods**:
+    - Generalizable attacks (random targets): [`claudini/methods/claude_random/`](claudini/methods/claude_random/)
+    - Attacks on a safeguard model: [`claudini/methods/claude_safeguard/`](claudini/methods/claude_safeguard/)
+
+See [`CLAUDE.md`](CLAUDE.md) for how to implement a new method.
+
+## Citation
+
+```bibtex
+@article{panfilov2026claudini,
+  title={Claudini: Autoresearch Discovers State-of-the-Art Adversarial Attack Algorithms for LLMs},
+  author={Panfilov, Alexander and Romov, Peter and Shilov, Igor and de Montjoye, Yves-Alexandre and Geiping, Jonas and Andriushchenko, Maksym},
+  journal={arXiv preprint arXiv:XXXX.XXXXX},
+  year={2026}
+}
+```