CalvinBackup/fuzzforge_ai
1
0
Fork 0
mirror of https://github.com/FuzzingLabs/fuzzforge_ai.git synced 2026-02-12 17:12:46 +00:00
Code Issues Packages Projects Releases Wiki Activity

docs: Remove obsolete Docker registry configuration

Browse Source 
Updated documentation to reflect v0.7.0 Temporal architecture which uses MinIO for storage instead of a Docker registry.

Major changes:
- getting-started.md: Added mandatory volumes/env/.env setup, removed registry config section, updated service list
- docker-setup.md: Complete rewrite focusing on system requirements and worker profiles instead of registry
- index.md: Replaced registry warning with environment file requirement
- troubleshooting.md: Removed all registry troubleshooting, added environment configuration issues
This commit is contained in:
tduhamel42
2025-10-17 10:28:17 +02:00
parent e42f07fc63
commit 37c15af130
4 changed files with 268 additions and 280 deletions
Download Patch File Download Diff File Expand all files Collapse all files

352
docs/docs/how-to/docker-setup.md
View File

@@ -1,226 +1,242 @@
# How to Configure Docker for FuzzForge
# Docker Requirements for FuzzForge
Getting Docker set up correctly is essential for running FuzzForge workflows. This guide will walk you through the process, explain why each step matters, and help you troubleshoot common issues—so you can get up and running with confidence.
FuzzForge runs entirely in Docker containers with Temporal orchestration. This guide covers system requirements, worker profiles, and resource management to help you run FuzzForge efficiently.
---
## Why Does FuzzForge Need Special Docker Configuration?
## System Requirements
FuzzForge builds and runs custom workflow images using a local Docker registry at `localhost:5001`. By default, Docker only trusts secure (HTTPS) registries, so you need to explicitly allow this local, insecure registry for development. Without this, workflows that build or pull images will fail.
### Docker Version
---
- **Docker Engine**: 20.10.0 or later
- **Docker Compose**: 2.0.0 or later
## Quick Setup: The One-Liner
Verify your installation:
Add this to your Docker daemon configuration:
```json
{
"insecure-registries": ["localhost:5001"]
}
```bash
docker --version
docker compose version
```
After editing, **restart Docker** for changes to take effect.
### Hardware Requirements
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | 2 cores | 4+ cores |
| RAM | 4 GB | 8 GB+ |
| Disk | 20 GB free | 50 GB+ free |
| Network | Internet access | Stable connection |
### Port Requirements
FuzzForge services use these ports (must be available):
| Port | Service | Purpose |
|------|---------|---------|
| 8000 | Backend API | FastAPI server |
| 8080 | Temporal UI | Workflow monitoring |
| 7233 | Temporal gRPC | Workflow execution |
| 9000 | MinIO API | S3-compatible storage |
| 9001 | MinIO Console | Storage management |
| 5432 | PostgreSQL | Temporal database |
Check for port conflicts:
```bash
# macOS/Linux
lsof -i :8000,8080,7233,9000,9001,5432
# Or just try starting FuzzForge
docker compose -f docker-compose.yml up -d
```
---
## Step-by-Step: Platform-Specific Instructions
## Worker Profiles (Resource Optimization)
### Docker Desktop (macOS & Windows)
FuzzForge uses Docker Compose **profiles** to prevent workers from auto-starting. This saves 5-7GB of RAM by only running workers when needed.
#### Using the Docker Desktop UI
### Profile Configuration
1. Open Docker Desktop.
2. Go to **Settings** (Windows) or **Preferences** (macOS).
3. Navigate to **Docker Engine**.
4. Add or update the `"insecure-registries"` section as shown above.
5. Click **Apply & Restart**.
6. Wait for Docker to restart (this may take a minute).
#### Editing the Config File Directly
- **macOS:** Edit `~/.docker/daemon.json`
- **Windows:** Edit `%USERPROFILE%\.docker\daemon.json`
Add or update the `"insecure-registries"` entry, then restart Docker Desktop.
---
### Docker Engine (Linux)
1. Edit (or create) `/etc/docker/daemon.json`:
```bash
sudo nano /etc/docker/daemon.json
```
2. Add:
```json
{
"insecure-registries": ["localhost:5001"]
}
```
3. Restart Docker:
```bash
sudo systemctl restart docker
```
4. Confirm Docker is running:
```bash
sudo systemctl status docker
```
#### Alternative: Systemd Drop-in (Advanced)
If you prefer, you can use a systemd override to add the registry flag. See the original guide for details.
---
## Verifying Your Configuration
1. **Check Docker’s registry settings:**
```bash
docker info | grep -i "insecure registries"
```
You should see `localhost:5001` listed.
2. **Test the registry:**
```bash
curl -f http://localhost:5001/v2/ && echo "✅ Registry accessible" || echo "❌ Registry not accessible"
```
3. **Try pushing and pulling an image:**
```bash
docker pull hello-world
docker tag hello-world localhost:5001/hello-world:test
docker push localhost:5001/hello-world:test
docker rmi localhost:5001/hello-world:test
docker pull localhost:5001/hello-world:test
```
---
## Worker Profiles (Resource Optimization - v0.7.0)
FuzzForge workers use Docker Compose profiles to prevent auto-startup:
Workers are configured with profiles in `docker-compose.yml`:
```yaml
# docker-compose.yml
worker-ossfuzz:
profiles:
- workers # For starting all workers
- ossfuzz # For starting just this worker
restart: "no" # Don't auto-restart
worker-python:
profiles:
- workers
- python
restart: "no"
worker-rust:
profiles:
- workers
- rust
restart: "no"
```
### Behavior
### Default Behavior
- **`docker-compose up -d`**: Workers DON'T start (saves ~6-7GB RAM)
- **CLI workflows**: Workers start automatically on-demand
- **Manual start**: `docker start fuzzforge-worker-ossfuzz`
**`docker compose up -d`** starts only core services:
- temporal
- temporal-ui
- postgresql
- minio
- minio-setup
- backend
- task-agent
### Resource Savings
Workers remain stopped until needed.
### On-Demand Worker Startup
When you run a workflow via the CLI, FuzzForge automatically starts the required worker:
```bash
# Automatically starts worker-python
fuzzforge workflow run atheris_fuzzing ./project
# Automatically starts worker-rust
fuzzforge workflow run cargo_fuzzing ./rust-project
# Automatically starts worker-secrets
fuzzforge workflow run secret_detection ./codebase
```
### Manual Worker Management
Start specific workers when needed:
```bash
# Start a single worker
docker start fuzzforge-worker-python
# Start all workers at once (uses more RAM)
docker compose --profile workers up -d
# Stop a worker to free resources
docker stop fuzzforge-worker-ossfuzz
```
### Resource Comparison
| Command | Workers Started | RAM Usage |
|---------|----------------|-----------|
| `docker-compose up -d` | None (core only) | ~1.2 GB |
| `ff workflow run ossfuzz_campaign .` | ossfuzz worker only | ~3-5 GB |
| `docker-compose --profile workers up -d` | All workers | ~8 GB |
| `docker compose up -d` | None (core only) | ~1.2 GB |
| `fuzzforge workflow run atheris_fuzzing .` | Python worker only | ~2-3 GB |
| `fuzzforge workflow run ossfuzz_campaign .` | OSS-Fuzz worker only | ~3-5 GB |
| `docker compose --profile workers up -d` | All workers | ~8 GB |
### Starting All Workers (Legacy Behavior)
---
If you prefer the old behavior where all workers start:
## Storage Management
### Docker Volume Cleanup
FuzzForge creates Docker volumes for persistent data. Clean them up periodically:
```bash
docker-compose --profile workers up -d
# Remove unused volumes (safe - keeps volumes for running containers)
docker volume prune
# List FuzzForge volumes
docker volume ls | grep fuzzforge
# Remove specific volume (WARNING: deletes data)
docker volume rm fuzzforge_minio-data
```
### Cache Directory
Workers use `/cache` inside containers for downloaded targets. This is managed automatically with LRU eviction, but you can check usage:
```bash
# Check cache usage in a worker
docker exec fuzzforge-worker-python du -sh /cache
# Clear cache manually if needed (safe - will re-download targets)
docker exec fuzzforge-worker-python rm -rf /cache/*
```
---
## Common Issues & How to Fix Them
## Environment Configuration
### "x509: certificate signed by unknown authority"
FuzzForge requires `volumes/env/.env` to start. This file contains API keys and configuration:
- **What’s happening?** Docker is trying to use HTTPS for the registry.
- **How to fix:** Double-check your `"insecure-registries"` config and restart Docker.
```bash
# Copy the example file
cp volumes/env/.env.example volumes/env/.env
### "connection refused" to localhost:5001
# Edit to add your API keys (if using AI features)
nano volumes/env/.env
```
- **What’s happening?** The registry isn’t running or the port is blocked.
- **How to fix:** Make sure FuzzForge services are up (`docker compose ps`), and that nothing else is using port 5001.
### Docker Desktop doesn’t apply settings
- **How to fix:** Fully quit and restart Docker Desktop. Check for typos in your JSON config.
### "permission denied" on Linux
- **How to fix:** Add your user to the `docker` group:
```bash
sudo usermod -aG docker $USER
newgrp docker
```
See [Getting Started](../tutorial/getting-started.md) for detailed environment setup.
---
## Security Notes
## Troubleshooting
- Using an insecure registry on `localhost:5001` is safe for local development.
- For production, always use a secure (HTTPS) registry and proper authentication.
### Services Won't Start
**Check ports are available:**
```bash
docker compose -f docker-compose.yml ps
lsof -i :8000,8080,7233,9000,9001
```
**Check Docker resources:**
```bash
docker system df
docker system prune # Free up space if needed
```
### Worker Memory Issues
If workers crash with OOM (out of memory) errors:
1. **Close other applications** to free RAM
2. **Start only needed workers** (don't use `--profile workers`)
3. **Increase Docker Desktop memory limit** (Settings → Resources)
4. **Monitor usage**: `docker stats`
### Slow Worker Startup
Workers pull large images (~2-5GB each) on first run:
```bash
# Check download progress
docker compose logs worker-python -f
# Pre-pull images (optional)
docker compose pull
```
---
## Where Are Docker Config Files?
## Best Practices
- **Docker Desktop (macOS):** `~/.docker/daemon.json`
- **Docker Desktop (Windows):** `%USERPROFILE%\.docker\daemon.json`
- **Docker Engine (Linux):** `/etc/docker/daemon.json`
**Tip:** Always back up your config before making changes.
---
## Advanced: Example Configurations
### Minimal
```json
{
"insecure-registries": ["localhost:5001"]
}
```
### With Logging and Storage Options
```json
{
"insecure-registries": ["localhost:5001"],
"log-driver": "json-file",
"log-opts": {
"max-size": "10m",
"max-file": "3"
},
"storage-driver": "overlay2"
}
```
### Multiple Registries
```json
{
"insecure-registries": [
"localhost:5001",
"192.168.1.100:5000",
"registry.internal:5000"
]
}
```
1. **Default startup**: Use `docker compose up -d` (core services only)
2. **Let CLI manage workers**: Workers start automatically when workflows run
3. **Stop unused workers**: Free RAM when not running workflows
4. **Monitor resources**: `docker stats` shows real-time usage
5. **Regular cleanup**: `docker system prune` removes unused images/containers
6. **Backup volumes**: `docker volume ls` shows persistent data locations
---
## Next Steps
- [Getting Started Guide](../tutorial/getting-started.md): Continue with FuzzForge setup.
- [Troubleshooting](troubleshooting.md): For more help if things don’t work.
- [Getting Started Guide](../tutorial/getting-started.md): Complete setup walkthrough
- [Running Workflows](running-workflows.md): Execute security workflows
- [Troubleshooting](troubleshooting.md): Fix common issues
---
**Remember:**
After any Docker config change, always restart Docker and verify your settings with `docker info` before running FuzzForge.
**Remember:** FuzzForge's on-demand worker startup saves resources. You don't need to manually manage workers - the CLI does it automatically!

85
docs/docs/how-to/troubleshooting.md
View File

@@ -10,10 +10,7 @@ Before diving into specific errors, let’s check the basics:
```bash
# Check all FuzzForge services
docker-compose -f docker-compose.yml ps
# Verify Docker registry config (if using workflow registry)
docker info | grep -i "insecure registries"
docker compose -f docker-compose.yml ps
# Test service health endpoints
curl http://localhost:8000/health
@@ -26,52 +23,45 @@ If any of these commands fail, note the error message and continue below.
---
## Docker Registry Problems
## Environment Configuration Issues
### "x509: certificate signed by unknown authority"
### Docker Compose fails with "file not found" or variable errors
**What’s happening?**
Docker is trying to use HTTPS for the local registry, but it’s set up for HTTP.
**What's happening?**
The required `volumes/env/.env` file is missing. Docker Compose needs this file to start.
**How to fix:**
1. Add this to your Docker daemon config (e.g., `/etc/docker/daemon.json`):
```json
{
"insecure-registries": ["localhost:5001"]
}
```bash
# Create the environment file from the template
cp volumes/env/.env.example volumes/env/.env
# Restart Docker Compose
docker compose -f docker-compose.yml down
docker compose -f docker-compose.yml up -d
```
**Note:** You can leave the `.env` file with default values if you're only using basic workflows (no AI features).
### API key errors for AI features
**What's happening?**
AI-powered workflows (like `llm_secret_detection`) or the AI agent need API keys.
**How to fix:**
1. Edit `volumes/env/.env` and add your keys:
```env
LITELLM_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-your-key-here
```
2. Restart Docker.
3. Confirm with:
2. Restart the backend to pick up new environment variables:
```bash
docker info | grep -A 5 -i "insecure registries"
docker compose restart backend
```
### "connection refused" to localhost:5001
**What’s happening?**
The registry isn’t running or the port is blocked.
**How to fix:**
- Make sure the registry container is up (if using registry for workflow images):
```bash
docker-compose -f docker-compose.yml ps registry
```
- Check logs for errors:
```bash
docker-compose -f docker-compose.yml logs registry
```
- If port 5001 is in use, change it in `docker-compose.yml` and your Docker config.
**Note:** With Temporal architecture, target files use MinIO (port 9000), not the registry.
### "no such host" error
**What’s happening?**
Docker can’t resolve `localhost`.
**How to fix:**
- Try using `127.0.0.1` instead of `localhost` in your Docker config.
- Check your `/etc/hosts` file for a correct `127.0.0.1 localhost` entry.
**Which workflows need API keys?**
- ✅ **Don't need keys**: `security_assessment`, `gitleaks_detection`, `trufflehog_detection`, `atheris_fuzzing`, `cargo_fuzzing`
- ⚠️ **Need keys**: `llm_secret_detection`, AI agent (`fuzzforge ai agent`)
---
@@ -189,10 +179,12 @@ File upload to MinIO failed or worker can't download target.
```bash
docker system prune -f
docker image prune -f
docker volume prune -f # Remove unused volumes
```
- Remove old workflow images:
- Clean worker cache manually if needed:
```bash
docker images | grep localhost:5001 | awk '{print $3}' | xargs docker rmi -f
docker exec fuzzforge-worker-python rm -rf /cache/*
docker exec fuzzforge-worker-rust rm -rf /cache/*
```
### High memory usage
@@ -258,12 +250,11 @@ Save and run this script to gather info for support:
#!/bin/bash
echo "=== FuzzForge Diagnostics ==="
date
docker-compose -f docker-compose.yml ps
docker info | grep -A 5 -i "insecure registries"
docker compose -f docker-compose.yml ps
curl -s http://localhost:8000/health || echo "Backend unhealthy"
curl -s http://localhost:8080 >/dev/null && echo "Temporal UI healthy" || echo "Temporal UI unhealthy"
curl -s http://localhost:9000 >/dev/null && echo "MinIO healthy" || echo "MinIO unhealthy"
docker-compose -f docker-compose.yml logs --tail 10
docker compose -f docker-compose.yml logs --tail 10
```
### Still stuck?

97
docs/docs/tutorial/getting-started.md
View File

@@ -18,67 +18,39 @@ First, let's clone the FuzzForge repository:
```bash
git clone https://github.com/FuzzingLabs/fuzzforge_ai.git
cd fuzzforge
cd fuzzforge_ai
```
Create the required environment configuration:
## Step 2: Configure Environment (Required)
**⚠️ This step is mandatory - Docker Compose will fail without it.**
Create the environment configuration file:
```bash
echo "COMPOSE_PROJECT_NAME=fuzzforge" > .env
cp volumes/env/.env.example volumes/env/.env
```
## Step 2: Configure Docker (Critical Step)
This file is required for FuzzForge to start. You can leave it with default values if you're only using basic workflows.
**⚠️ This step is mandatory - workflows will fail without proper Docker configuration.**
### When you need to add API keys:
FuzzForge uses a local Docker registry at `localhost:5001` for workflow images. You must configure Docker to allow this insecure registry.
Edit `volumes/env/.env` and add your API keys if using:
- **AI-powered workflows**: `llm_secret_detection`
- **AI agent**: `ff ai agent`
### For Docker Desktop (macOS/Windows):
1. Open Docker Desktop
2. Go to Settings/Preferences → Docker Engine
3. Add the following to the JSON configuration:
```json
{
"builder": {
"gc": {
"defaultKeepStorage": "20GB",
"enabled": true
}
},
"experimental": false,
"insecure-registries": [
"localhost:5001"
]
}
Example:
```env
LITELLM_MODEL=gpt-4o-mini
OPENAI_API_KEY=sk-your-key-here
```
4. Click "Apply & Restart"
### For Docker Engine (Linux):
Edit `/etc/docker/daemon.json` (create if it doesn't exist):
```json
{
"insecure-registries": ["localhost:5001"]
}
```
Restart Docker:
```bash
sudo systemctl restart docker
```
### Verify Docker Configuration
Test that Docker can access the insecure registry:
```bash
# After starting FuzzForge (next step), this should work without certificate errors
docker pull localhost:5001/hello-world 2>/dev/null || echo "Registry not accessible yet (normal before startup)"
```
### Basic workflows that don't need API keys:
- `security_assessment`
- `gitleaks_detection`
- `trufflehog_detection`
- `atheris_fuzzing`
- `cargo_fuzzing`
## Step 3: Start FuzzForge
@@ -88,14 +60,23 @@ Start all FuzzForge services:
docker-compose -f docker-compose.yml up -d
```
This will start 6+ services:
- **temporal**: Workflow orchestration server (includes embedded PostgreSQL for dev)
- **minio**: S3-compatible storage for uploaded targets and results
- **minio-setup**: One-time setup for MinIO buckets (exits after setup)
- **fuzzforge-backend**: FastAPI backend and workflow management
- **worker-rust**: Long-lived worker for Rust/native security analysis
- **worker-android**: Long-lived worker for Android security analysis (if configured)
- **worker-web**: Long-lived worker for web security analysis (if configured)
This will start the FuzzForge platform with multiple services:
**Core Services:**
- **temporal**: Workflow orchestration server
- **temporal-ui**: Web UI for workflow monitoring (port 8080)
- **postgresql**: Database for Temporal state
- **minio**: S3-compatible storage for targets and results
- **minio-setup**: One-time MinIO bucket configuration (exits after setup)
- **backend**: FastAPI server and workflow management API
- **task-agent**: A2A agent coordination service
**Security Workers** (vertical toolchains):
- **worker-python**: Python security analysis and Atheris fuzzing
- **worker-rust**: Rust/Cargo fuzzing and native code analysis
- **worker-secrets**: Secret detection and credential scanning
- **worker-ossfuzz**: OSS-Fuzz integration (heavy development)
- **worker-android**: Android security analysis (early development)
Wait for all services to be healthy (this may take 2-3 minutes on first startup):

14
docs/index.md
View File

@@ -84,17 +84,17 @@ Technical reference materials and specifications.
## 🚨 Important Setup Requirement
**Docker Insecure Registry Configuration Required**
**Environment Configuration Required**
FuzzForge uses a local Docker registry for workflow images. You **must** configure Docker to allow insecure registries:
Before starting FuzzForge, you **must** create the environment configuration file:
```json
{
"insecure-registries": ["localhost:5001"]
}
```bash
cp volumes/env/.env.example volumes/env/.env
```
See [Docker Setup Guide](how-to/docker-setup.md) for detailed configuration instructions.
Docker Compose will fail without this file. You can leave it with default values if you're only using basic workflows (no AI features).
See [Getting Started Guide](tutorials/getting-started.md) for detailed setup instructions.
---