Add autoresearch skill, update configs and README

Add .claude/skills/claudini/SKILL.md to drive the autoresearch loop
via /claudini slash command. Update CLAUDE.md with skill docs. Replace
PROMPT.txt with the skill-based workflow. Rewrite README to feature
the autoresearch loop prominently. Add easy_1e16 and easy_1e17 preset
configs and update safeguard configs.

Assisted-by: Claude <noreply@anthropic.com>
This commit is contained in:
Peter Romov
2026-03-26 16:28:14 +00:00
parent 4c938fd325
commit 69c04a2b9e
9 changed files with 160 additions and 55 deletions
+21 -22
View File
@@ -22,6 +22,25 @@ uv sync
Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).
## Discover Your Own SOTA Attack
<p align="center">
<img src="assets/autoresearch_loop.png" width="90%" alt="Autoresearch loop: seeding, analysis-experiment cycle, evaluation">
</p>
To run autoresearch, open [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and start the `/claudini` skill in a loop:
```bash
claude
> /loop /claudini my_run break Qwen2.5-7B on random strings under 1e15 FLOPs
```
Each iteration, Claude studies existing methods and results, designs a new optimizer, benchmarks it, and commits — maintaining an agent log across iterations. The run code (`my_run` above) isolates the method chain, git branch, and log.
See the full [skill prompt](.claude/skills/claudini/SKILL.md) for details.
Use `tmux` or `screen` so sessions survive disconnection. Track progress via `git log`.
## Evaluate
All experiments are run via `claudini.run_bench` CLI:
@@ -38,34 +57,14 @@ uv run -m claudini.run_bench random_valid --method gcg,acg --max-flops 1e15
Results are saved to `results/<method>/<preset>/<model>/sample_<S>_seed_<N>.json`. Existing results are auto-skipped.
## Run Autoresearch
<p align="center">
<img src="assets/autoresearch_loop.png" width="90%" alt="Autoresearch loop: seeding, analysis-experiment cycle, evaluation">
</p>
Install [Claude Code](https://docs.anthropic.com/en/docs/claude-code), then start the loop:
```bash
claude
> /loop 10m $(cat PROMPT.txt)
```
Or with the [ralph-loop](https://github.com/anthropics/claude-code-ralph-loop) plugin:
```bash
claude
> /ralph-loop:ralph-loop $(cat PROMPT.txt)
```
The loop reads [`PROMPT.txt`](PROMPT.txt), studies existing methods and results, designs a new optimizer, implements it, runs the benchmark, and commits — all autonomously. Use tmux or screen so sessions survive disconnection. Track progress via `git log`.
Precomputed results from the paper are available as a [GitHub release](https://github.com/romovpa/claudini/releases). Download and unzip `claudini-results.zip` into the repo root.
## Attack Methods
We consider white-box GCG-style attacks that search directly over the model's vocabulary using gradients. Each method ([`TokenOptimizer`](claudini/base.py#L429)) optimizes a short discrete token *suffix* that, when appended to an input prompt, causes the model to produce a desired target sequence.
- **Baselines** (existing methods): [`claudini/methods/original/`](claudini/methods/original/)
- **Claude-deviced methods**:
- **Claude-designed methods** (each run code produces a separate chain):
- Generalizable attacks (random targets): [`claudini/methods/claude_random/`](claudini/methods/claude_random/)
- Attacks on a safeguard model: [`claudini/methods/claude_safeguard/`](claudini/methods/claude_safeguard/)