mirror of
https://github.com/romovpa/claudini.git
synced 2026-07-19 23:17:21 +02:00
Add autoresearch skill, update configs and README
Add .claude/skills/claudini/SKILL.md to drive the autoresearch loop via /claudini slash command. Update CLAUDE.md with skill docs. Replace PROMPT.txt with the skill-based workflow. Rewrite README to feature the autoresearch loop prominently. Add easy_1e16 and easy_1e17 preset configs and update safeguard configs. Assisted-by: Claude <noreply@anthropic.com>
This commit is contained in:
@@ -22,6 +22,25 @@ uv sync
|
||||
|
||||
Requires Python 3.12+ and [uv](https://docs.astral.sh/uv/).
|
||||
|
||||
## Discover Your Own SOTA Attack
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/autoresearch_loop.png" width="90%" alt="Autoresearch loop: seeding, analysis-experiment cycle, evaluation">
|
||||
</p>
|
||||
|
||||
To run autoresearch, open [Claude Code](https://docs.anthropic.com/en/docs/claude-code) and start the `/claudini` skill in a loop:
|
||||
|
||||
```bash
|
||||
claude
|
||||
> /loop /claudini my_run break Qwen2.5-7B on random strings under 1e15 FLOPs
|
||||
```
|
||||
|
||||
Each iteration, Claude studies existing methods and results, designs a new optimizer, benchmarks it, and commits — maintaining an agent log across iterations. The run code (`my_run` above) isolates the method chain, git branch, and log.
|
||||
|
||||
See the full [skill prompt](.claude/skills/claudini/SKILL.md) for details.
|
||||
|
||||
Use `tmux` or `screen` so sessions survive disconnection. Track progress via `git log`.
|
||||
|
||||
## Evaluate
|
||||
|
||||
All experiments are run via `claudini.run_bench` CLI:
|
||||
@@ -38,34 +57,14 @@ uv run -m claudini.run_bench random_valid --method gcg,acg --max-flops 1e15
|
||||
|
||||
Results are saved to `results/<method>/<preset>/<model>/sample_<S>_seed_<N>.json`. Existing results are auto-skipped.
|
||||
|
||||
## Run Autoresearch
|
||||
|
||||
<p align="center">
|
||||
<img src="assets/autoresearch_loop.png" width="90%" alt="Autoresearch loop: seeding, analysis-experiment cycle, evaluation">
|
||||
</p>
|
||||
|
||||
Install [Claude Code](https://docs.anthropic.com/en/docs/claude-code), then start the loop:
|
||||
|
||||
```bash
|
||||
claude
|
||||
> /loop 10m $(cat PROMPT.txt)
|
||||
```
|
||||
|
||||
Or with the [ralph-loop](https://github.com/anthropics/claude-code-ralph-loop) plugin:
|
||||
|
||||
```bash
|
||||
claude
|
||||
> /ralph-loop:ralph-loop $(cat PROMPT.txt)
|
||||
```
|
||||
|
||||
The loop reads [`PROMPT.txt`](PROMPT.txt), studies existing methods and results, designs a new optimizer, implements it, runs the benchmark, and commits — all autonomously. Use tmux or screen so sessions survive disconnection. Track progress via `git log`.
|
||||
Precomputed results from the paper are available as a [GitHub release](https://github.com/romovpa/claudini/releases). Download and unzip `claudini-results.zip` into the repo root.
|
||||
|
||||
## Attack Methods
|
||||
|
||||
We consider white-box GCG-style attacks that search directly over the model's vocabulary using gradients. Each method ([`TokenOptimizer`](claudini/base.py#L429)) optimizes a short discrete token *suffix* that, when appended to an input prompt, causes the model to produce a desired target sequence.
|
||||
|
||||
- **Baselines** (existing methods): [`claudini/methods/original/`](claudini/methods/original/)
|
||||
- **Claude-deviced methods**:
|
||||
- **Claude-designed methods** (each run code produces a separate chain):
|
||||
- Generalizable attacks (random targets): [`claudini/methods/claude_random/`](claudini/methods/claude_random/)
|
||||
- Attacks on a safeguard model: [`claudini/methods/claude_safeguard/`](claudini/methods/claude_safeguard/)
|
||||
|
||||
|
||||
Reference in New Issue
Block a user