From e42f07fc63fa233da3066e5852eec300840c5a5d Mon Sep 17 00:00:00 2001 From: tduhamel42 Date: Fri, 17 Oct 2025 10:21:47 +0200 Subject: [PATCH 1/5] docs: Apply global fixes for v0.7.0 Temporal architecture MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Replace docker-compose.temporal.yaml β†’ docker-compose.yml - Replace Temporal UI port :8233 β†’ :8080 - Replace repository URL fuzzforge.git β†’ fuzzforge_ai.git Affected files: - concept/docker-containers.md - concept/resource-management.md - concept/sarif-format.md - how-to/create-workflow.md - how-to/debugging.md - how-to/troubleshooting.md - tutorial/getting-started.md --- ...5-01-16-v0.7.0-temporal-workers-release.md | 321 ++++++++++++++++++ docs/docs/concept/docker-containers.md | 2 +- docs/docs/concept/resource-management.md | 24 +- docs/docs/concept/sarif-format.md | 2 +- docs/docs/how-to/create-workflow.md | 4 +- docs/docs/how-to/debugging.md | 54 +-- docs/docs/how-to/troubleshooting.md | 50 +-- docs/docs/tutorial/getting-started.md | 8 +- 8 files changed, 393 insertions(+), 72 deletions(-) create mode 100644 docs/blog/2025-01-16-v0.7.0-temporal-workers-release.md diff --git a/docs/blog/2025-01-16-v0.7.0-temporal-workers-release.md b/docs/blog/2025-01-16-v0.7.0-temporal-workers-release.md new file mode 100644 index 0000000..ef8a641 --- /dev/null +++ b/docs/blog/2025-01-16-v0.7.0-temporal-workers-release.md @@ -0,0 +1,321 @@ +# FuzzForge v0.7.0: Temporal Orchestration & Vertical Workers Architecture + +We're excited to announce **FuzzForge v0.7.0**, a major release featuring two significant improvements: + +1. **Architectural Foundation**: Complete migration from Prefect to **Temporal** orchestration with **vertical workers** - pre-built containers for instant deployment +2. **AI-Powered Secret Detection**: New workflows achieving 84% recall on obfuscated secrets using LLM semantic analysis + +This release transforms how security workflows are built, deployed, and scaled. + + + +## πŸš€ Flagship Features + +### Temporal Orchestration: Production-Ready Workflow Engine + +We've fully migrated from Prefect to [Temporal](https://temporal.io), bringing enterprise-grade workflow orchestration to FuzzForge: + +**Why Temporal?** + +- βœ… **Reliability**: Automatic retries, timeouts, and failure handling built-in +- βœ… **Observability**: World-class UI for monitoring workflow execution, logs, and debugging +- βœ… **Scalability**: Horizontal scaling across workers with intelligent load balancing +- βœ… **Developer Experience**: Type-safe workflows, versioning, and zero downtime deployments + +**What This Means for You:** + +```bash +# Start FuzzForge with Temporal +docker compose up -d + +# Monitor workflows in real-time +open http://localhost:8080 # Temporal UI + +# Submit workflows - everything just works +cd your_project/ +ff workflow run security_assessment . +``` + +The Temporal UI gives you complete visibility into workflow execution: + +- Live activity timelines +- Detailed logs for every step +- Retry history and failure analysis +- Performance metrics and bottleneck detection + +### Vertical Workers: Pre-Built Security Toolchains + +FuzzForge now uses **vertical workers** - long-lived containers pre-built with security toolchains for different languages and platforms: + +| Worker | Toolchain | Status | Available Workflows | +| ----------- | ----------------------------- | ------------- | ------------------------------------- | +| **python** | Gitleaks, TruffleHog, Atheris | βœ… Production | Secret detection, security assessment | +| **rust** | cargo-fuzz | ⚠️ Early Dev | Rust fuzzing | +| **ossfuzz** | OSS-Fuzz infrastructure | ⚠️ Heavy Dev | Continuous fuzzing campaigns | + +**Note:** Additional workers (web, android, Go) are planned but not yet available. + +**Key Benefits:** + +1. **Zero Build Time**: Workflows start instantly - no container builds per workflow +2. **Instant Code Changes**: Modify workflow code, restart worker, done +3. **Consistent Environment**: Same toolchain versions across all runs +4. **Resource Efficiency**: Share workers across multiple concurrent workflows + +**Example: Running Secret Detection** + +```bash +# Worker is already running with Gitleaks, TruffleHog installed +ff workflow run gitleaks_detection . + +# Behind the scenes: +# 1. CLI uploads project to MinIO +# 2. Temporal schedules on python-worker +# 3. Worker downloads from MinIO +# 4. Gitleaks runs (already installed!) +# 5. Results returned as SARIF +``` + +### MinIO Storage: Unified File Handling + +We've replaced volume mounts with **MinIO** (S3-compatible object storage): + +**Old Way (Volume Mounts):** + +```yaml +# Had to mount directories, manage paths, cleanup manually +volumes: + - ./my_project:/target +``` + +**New Way (MinIO):** + +```bash +# CLI handles everything automatically +ff workflow run security_assessment . +# βœ“ Creates tarball +# βœ“ Uploads to MinIO +# βœ“ Passes target_id to workflow +# βœ“ Worker downloads and extracts +# βœ“ Cleanup handled automatically +``` + +**Benefits:** + +- βœ… No path conflicts or permissions issues +- βœ… Works seamlessly with remote Temporal clusters +- βœ… Automatic cleanup and caching +- βœ… Supports large targets (GB+) + +## πŸ” AI-Powered Secret Detection: Also in v0.7.0 + +Alongside the architectural improvements, we're releasing a comprehensive **secret detection** system with three workflows: + +### Benchmark Results + +We tested on a controlled dataset of **32 documented secrets** (12 Easy, 10 Medium, 10 Hard): + +| Tool | Recall | Secrets Found | Speed | Best For | +| --------------------- | --------- | ------------- | ----- | --------------------------- | +| **LLM (gpt-5-mini)** | **84.4%** | 41 | 618s | Obfuscated & hidden secrets | +| **LLM (gpt-4o-mini)** | 56.2% | 30 | 297s | Balanced speed/accuracy | +| **Gitleaks** | 37.5% | 12 | 5s | Fast pattern-based scanning | +| **TruffleHog** | 0.0% | 1 | 5s | Entropy analysis | + +πŸ“Š [Full benchmark methodology and results β†’](https://github.com/FuzzingLabs/fuzzforge_ai/blob/dev/backend/benchmarks/by_category/secret_detection/results/comparison_report.md) + +### Why LLM-Based Detection Wins + +**Obfuscated Secrets (Medium Difficulty):** + +```python +# Gitleaks: ❌ Missed (no pattern match) +# LLM: βœ… Found (semantic understanding) +aws_key = base64.b64decode("QUtJQUlPU0ZPRE5ON0VYQU1QTEU=").decode() +``` + +**Well-Hidden Secrets (Hard Difficulty):** + +```python +# Gitleaks: ❌ Missed (no pattern) +# LLM: βœ… Found (understands XOR + join) +secret = ''.join(chr(ord(c) ^ 0x42) for c in "\x0b\x15\x04\x1b...") +``` + +**Standard Secrets (Easy Difficulty):** + +```python +# Both find these: +AWS_ACCESS_KEY = "AKIAIOSFODNN7EXAMPLE" +``` + +### Try It Yourself + +```bash +# Start FuzzForge +docker compose up -d + +# Run secret detection on your code +cd your_project/ +ff workflow run gitleaks_detection . # Fast pattern-based +ff workflow run trufflehog_detection . # Entropy analysis +ff workflow run llm_secret_detection . # AI semantic analysis + +# Get SARIF output +ff finding +``` + +## πŸ“Š Real-World Impact + +**Before v0.7.0 (Pattern-Only Detection):** + +- Found: Standard API keys, simple patterns +- Missed: Base64-encoded secrets, obfuscated credentials, split secrets + +**After v0.7.0 (LLM + Patterns):** + +- **84% recall** on comprehensive benchmark +- Detects novel obfuscation techniques +- Understands code context (not just regex) +- Catches secrets in: + - Base64/hex encoding + - String concatenation + - XOR/ROT13 obfuscation + - Template strings + - Binary literals + +## πŸ”„ Migration Guide + +### What Changed + +**Docker Compose:** + +```bash +# Old (Prefect) +docker-compose up + +# New (Temporal) +docker compose up -d +``` + +**Workflow Submission:** + +```bash +# Old (volume mounts) +ff workflow run security_assessment --volume ./project + +# New (automatic upload) +ff workflow run security_assessment . +# CLI handles upload automatically! +``` + +**Worker Management:** + +```bash +# Old (per-workflow containers) +# Each workflow built its own container + +# New (vertical workers) +docker compose up -d # All workers start +# Workflows share workers - much faster! +``` + +### Configuration + +Set up AI workflows with API keys: + +```bash +cp volumes/env/.env.example volumes/env/.env +# Edit .env and add your API keys (OpenAI, Anthropic, etc.) +``` + +Required for: + +- `llm_secret_detection` workflow +- AI agent features (`ff ai agent`) + +Basic security workflows (gitleaks, trufflehog, security_assessment) work without this. + +## πŸ—οΈ Architecture Overview + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ User CLI β”‚ Upload β†’ MinIO +β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + ↓ Submit +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Temporal β”‚ Schedule β†’ Task Queue +β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜ + ↓ Execute +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Vertical β”‚ Download from MinIO β†’ Run Tools β†’ Upload Results +β”‚ Workers β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + rust, python, web, android, ossfuzz +``` + +**Benefits:** + +- πŸ”„ Automatic retries and timeouts (Temporal) +- πŸ“¦ No file path management (MinIO) +- ⚑ Zero container build time (Vertical Workers) +- πŸ“ˆ Horizontal scaling ready (Temporal + Workers) + +## 🎯 Workflow Stability Status + +### βœ… Stable & Production-Ready + +- **Secret Detection**: `gitleaks_detection`, `trufflehog_detection`, `llm_secret_detection` +- **Security Assessment**: `security_assessment` +- Temporal orchestration with python worker +- MinIO file storage + +### ⚠️ Early Development (Functional but not production-ready) + +- **Fuzzing workflows**: + - `atheris_fuzzing` - Python fuzzing with Atheris + - `cargo_fuzzing` - Rust fuzzing with cargo-fuzz +- **OSS-Fuzz integration**: `ossfuzz_campaign` (under heavy active development) + +**Important:** Fuzzing workflows are functional for testing but not recommended for production use yet. + +## πŸ“š Resources + +- 🌐 [Website](https://fuzzforge.ai) +- πŸ“– [Documentation](https://docs.fuzzforge.ai) +- πŸ’¬ [Discord Community](https://discord.gg/8XEX33UUwZ) +- πŸŽ“ [FuzzingLabs Academy](https://academy.fuzzinglabs.com/?coupon=GITHUB_FUZZFORGE) +- πŸ“Š [Secret Detection Benchmarks](https://github.com/FuzzingLabs/fuzzforge_ai/blob/dev/backend/benchmarks/by_category/secret_detection/results/comparison_report.md) + +## πŸ™ Acknowledgments + +Special thanks to: + +- [Temporal](https://temporal.io) for the amazing workflow engine +- Our community for feedback during the migration + +## πŸš€ Get Started + +```bash +# Clone and install +git clone https://github.com/fuzzinglabs/fuzzforge_ai.git +cd fuzzforge_ai +uv tool install --python python3.12 . + +# Start FuzzForge with Temporal +docker compose up -d + +# Run your first workflow +cd test_projects/vulnerable_app/ +fuzzforge init +ff workflow run security_assessment . + +# Check Temporal UI +open http://localhost:8080 +``` + +--- + +**FuzzForge v0.7.0** is a foundational release that sets the stage for scalable, production-ready security automation. Try it today and let us know what you think! + +Star us on [GitHub](https://github.com/FuzzingLabs/fuzzforge_ai) ⭐ diff --git a/docs/docs/concept/docker-containers.md b/docs/docs/concept/docker-containers.md index d010f44..e5a8bc6 100644 --- a/docs/docs/concept/docker-containers.md +++ b/docs/docs/concept/docker-containers.md @@ -141,7 +141,7 @@ Example build: cd workers/rust docker build -t fuzzforge-worker-rust:latest . # Or via docker-compose -docker-compose -f docker-compose.temporal.yaml build worker-rust +docker-compose -f docker-compose.yml build worker-rust ``` --- diff --git a/docs/docs/concept/resource-management.md b/docs/docs/concept/resource-management.md index ab56599..e189fa7 100644 --- a/docs/docs/concept/resource-management.md +++ b/docs/docs/concept/resource-management.md @@ -124,7 +124,7 @@ docker ps -a --filter "name=fuzzforge-worker" ## Level 1: Docker Container Limits (Primary) -Docker container limits are the **primary enforcement mechanism** for CPU and memory resources. These are configured in `docker-compose.temporal.yaml` and enforced by the Docker runtime. +Docker container limits are the **primary enforcement mechanism** for CPU and memory resources. These are configured in `docker-compose.yml` and enforced by the Docker runtime. ### Configuration @@ -267,10 +267,10 @@ Check how many workflows are running: ```bash # View worker logs -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "Starting" +docker-compose -f docker-compose.yml logs worker-rust | grep "Starting" # Check Temporal UI -# Open http://localhost:8233 +# Open http://localhost:8080 # Navigate to "Task Queues" β†’ "rust" β†’ See pending/running counts ``` @@ -324,7 +324,7 @@ class SecurityAssessmentWorkflow: ``` **Check timeout in Temporal UI:** -1. Open http://localhost:8233 +1. Open http://localhost:8080 2. Navigate to workflow execution 3. See "Timeout" in workflow details 4. If exceeded, status shows "TIMED_OUT" @@ -383,7 +383,7 @@ Watch for these warning signs: ```bash # Check for OOM kills -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep -i "oom\|killed" +docker-compose -f docker-compose.yml logs worker-rust | grep -i "oom\|killed" # Check for CPU throttling docker stats fuzzforge-worker-rust @@ -413,7 +413,7 @@ To handle more workflows, scale worker containers horizontally: ```bash # Scale rust worker to 3 instances -docker-compose -f docker-compose.temporal.yaml up -d --scale worker-rust=3 +docker-compose -f docker-compose.yml up -d --scale worker-rust=3 # Now you can run: # - 3 workers Γ— 5 concurrent activities = 15 workflows simultaneously @@ -435,13 +435,13 @@ docker-compose -f docker-compose.temporal.yaml up -d --scale worker-rust=3 **Diagnosis:** ```bash # Check worker is alive -docker-compose -f docker-compose.temporal.yaml ps worker-rust +docker-compose -f docker-compose.yml ps worker-rust # Check worker resource usage docker stats fuzzforge-worker-rust # Check for OOM kills -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep -i oom +docker-compose -f docker-compose.yml logs worker-rust | grep -i oom ``` **Solution:** @@ -459,7 +459,7 @@ docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep -i oom docker exec fuzzforge-worker-rust env | grep MAX_CONCURRENT_ACTIVITIES # Check current workload -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "Starting" +docker-compose -f docker-compose.yml logs worker-rust | grep "Starting" ``` **Solution:** @@ -557,10 +557,10 @@ Check cache size and cleanup logs: docker exec fuzzforge-worker-rust du -sh /cache # Monitor cache evictions -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "Evicted from cache" +docker-compose -f docker-compose.yml logs worker-rust | grep "Evicted from cache" # Check download vs cache hit rate -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep -E "Cache (HIT|MISS)" +docker-compose -f docker-compose.yml logs worker-rust | grep -E "Cache (HIT|MISS)" ``` See the [Workspace Isolation](/docs/concept/workspace-isolation) guide for complete details on isolation modes and when to use each. @@ -588,7 +588,7 @@ FuzzForge's resource management strategy: --- **Next Steps:** -- Review `docker-compose.temporal.yaml` resource configuration +- Review `docker-compose.yml` resource configuration - Profile your workflows to determine actual resource usage - Adjust limits based on monitoring data - Set up alerts for resource exhaustion diff --git a/docs/docs/concept/sarif-format.md b/docs/docs/concept/sarif-format.md index 058de88..05ef8ca 100644 --- a/docs/docs/concept/sarif-format.md +++ b/docs/docs/concept/sarif-format.md @@ -45,7 +45,7 @@ Each FuzzForge workflow produces a SARIF "run" containing: "driver": { "name": "FuzzForge", "version": "1.0.0", - "informationUri": "https://github.com/FuzzingLabs/fuzzforge", + "informationUri": "https://github.com/FuzzingLabs/fuzzforge_ai", "organization": "FuzzingLabs", "rules": [ /* Security rules applied */ ] }, diff --git a/docs/docs/how-to/create-workflow.md b/docs/docs/how-to/create-workflow.md index 3a5c814..3c46620 100644 --- a/docs/docs/how-to/create-workflow.md +++ b/docs/docs/how-to/create-workflow.md @@ -397,7 +397,7 @@ class DependencyAnalysisWorkflow: ```bash # Start FuzzForge with Temporal -docker-compose -f docker-compose.temporal.yaml up -d +docker-compose -f docker-compose.yml up -d # Wait for services to initialize sleep 10 @@ -445,7 +445,7 @@ client.close() ### Check Temporal UI -Open http://localhost:8233 to see: +Open http://localhost:8080 to see: - Workflow execution timeline - Activity results - Logs and errors diff --git a/docs/docs/how-to/debugging.md b/docs/docs/how-to/debugging.md index fd94e73..3a86b24 100644 --- a/docs/docs/how-to/debugging.md +++ b/docs/docs/how-to/debugging.md @@ -8,10 +8,10 @@ This guide shows you how to debug FuzzForge workflows and modules using Temporal When something goes wrong: -1. **Check worker logs** - `docker-compose -f docker-compose.temporal.yaml logs worker-rust -f` -2. **Check Temporal UI** - http://localhost:8233 (visual execution history) +1. **Check worker logs** - `docker-compose -f docker-compose.yml logs worker-rust -f` +2. **Check Temporal UI** - http://localhost:8080 (visual execution history) 3. **Check MinIO console** - http://localhost:9001 (inspect uploaded files) -4. **Check backend logs** - `docker-compose -f docker-compose.temporal.yaml logs fuzzforge-backend -f` +4. **Check backend logs** - `docker-compose -f docker-compose.yml logs fuzzforge-backend -f` --- @@ -41,12 +41,12 @@ When something goes wrong: 4. **Check worker logs for discovery errors:** ```bash - docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "my_workflow" + docker-compose -f docker-compose.yml logs worker-rust | grep "my_workflow" ``` **Solution:** - Ensure `metadata.yaml` has correct `vertical` field -- Restart worker to reload: `docker-compose -f docker-compose.temporal.yaml restart worker-rust` +- Restart worker to reload: `docker-compose -f docker-compose.yml restart worker-rust` - Check worker logs for discovery confirmation --- @@ -55,10 +55,10 @@ When something goes wrong: ### Using Temporal Web UI -The Temporal UI at http://localhost:8233 is your primary debugging tool. +The Temporal UI at http://localhost:8080 is your primary debugging tool. **Navigate to a workflow:** -1. Open http://localhost:8233 +1. Open http://localhost:8080 2. Click "Workflows" in left sidebar 3. Find your workflow by `run_id` or workflow name 4. Click to see detailed execution @@ -86,13 +86,13 @@ The Temporal UI at http://localhost:8233 is your primary debugging tool. ```bash # Follow logs from rust worker -docker-compose -f docker-compose.temporal.yaml logs worker-rust -f +docker-compose -f docker-compose.yml logs worker-rust -f # Follow logs from all workers -docker-compose -f docker-compose.temporal.yaml logs worker-rust worker-android -f +docker-compose -f docker-compose.yml logs worker-rust worker-android -f # Show last 100 lines -docker-compose -f docker-compose.temporal.yaml logs worker-rust --tail 100 +docker-compose -f docker-compose.yml logs worker-rust --tail 100 ``` ### What Worker Logs Show @@ -125,16 +125,16 @@ IndentationError: expected an indented block ```bash # Show only errors -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep ERROR +docker-compose -f docker-compose.yml logs worker-rust | grep ERROR # Show workflow discovery -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "Discovered workflow" +docker-compose -f docker-compose.yml logs worker-rust | grep "Discovered workflow" # Show specific workflow execution -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "security_assessment-abc123" +docker-compose -f docker-compose.yml logs worker-rust | grep "security_assessment-abc123" # Show activity execution -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "activity" +docker-compose -f docker-compose.yml logs worker-rust | grep "activity" ``` --- @@ -156,7 +156,7 @@ docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep "activity curl http://localhost:9000 # List backend logs for upload -docker-compose -f docker-compose.temporal.yaml logs fuzzforge-backend | grep "upload" +docker-compose -f docker-compose.yml logs fuzzforge-backend | grep "upload" ``` ### Check Worker Cache @@ -230,12 +230,12 @@ Since toolbox is mounted as a volume, you can edit code on your host and reload: 2. **Restart worker to reload:** ```bash - docker-compose -f docker-compose.temporal.yaml restart worker-rust + docker-compose -f docker-compose.yml restart worker-rust ``` 3. **Check discovery logs:** ```bash - docker-compose -f docker-compose.temporal.yaml logs worker-rust | tail -50 + docker-compose -f docker-compose.yml logs worker-rust | tail -50 ``` ### Add Debug Logging @@ -266,14 +266,14 @@ class MyWorkflow: Set debug logging: ```bash -# Edit docker-compose.temporal.yaml +# Edit docker-compose.yml services: worker-rust: environment: LOG_LEVEL: DEBUG # Change from INFO to DEBUG # Restart -docker-compose -f docker-compose.temporal.yaml restart worker-rust +docker-compose -f docker-compose.yml restart worker-rust ``` --- @@ -285,7 +285,7 @@ docker-compose -f docker-compose.temporal.yaml restart worker-rust **Debug:** 1. Check Temporal UI for last completed activity 2. Check worker logs for errors -3. Check if worker is still running: `docker-compose -f docker-compose.temporal.yaml ps worker-rust` +3. Check if worker is still running: `docker-compose -f docker-compose.yml ps worker-rust` **Solution:** - Worker may have crashed - restart it @@ -315,7 +315,7 @@ docker-compose -f docker-compose.temporal.yaml restart worker-rust **Solution:** - Re-upload file via CLI -- Check MinIO is running: `docker-compose -f docker-compose.temporal.yaml ps minio` +- Check MinIO is running: `docker-compose -f docker-compose.yml ps minio` - Check MinIO credentials in worker environment --- @@ -337,7 +337,7 @@ docker-compose -f docker-compose.temporal.yaml restart worker-rust docker stats fuzzforge-worker-rust # Check worker logs for memory warnings -docker-compose -f docker-compose.temporal.yaml logs worker-rust | grep -i "memory\|oom" +docker-compose -f docker-compose.yml logs worker-rust | grep -i "memory\|oom" ``` ### Profile Workflow Execution @@ -368,7 +368,7 @@ class MyWorkflow: ### Enable Temporal Worker Debug Logs ```bash -# Edit docker-compose.temporal.yaml +# Edit docker-compose.yml services: worker-rust: environment: @@ -376,7 +376,7 @@ services: LOG_LEVEL: DEBUG # Restart -docker-compose -f docker-compose.temporal.yaml restart worker-rust +docker-compose -f docker-compose.yml restart worker-rust ``` ### Inspect Temporal Workflows via CLI @@ -405,7 +405,7 @@ docker exec fuzzforge-worker-rust ping temporal docker exec fuzzforge-worker-rust curl http://minio:9000 # From host to services -curl http://localhost:8233 # Temporal UI +curl http://localhost:8080 # Temporal UI curl http://localhost:9000 # MinIO curl http://localhost:8000/health # Backend ``` @@ -431,10 +431,10 @@ If you're still stuck: 1. **Collect diagnostic info:** ```bash # Save all logs - docker-compose -f docker-compose.temporal.yaml logs > fuzzforge-logs.txt + docker-compose -f docker-compose.yml logs > fuzzforge-logs.txt # Check service status - docker-compose -f docker-compose.temporal.yaml ps > service-status.txt + docker-compose -f docker-compose.yml ps > service-status.txt ``` 2. **Check Temporal UI** and take screenshots of: diff --git a/docs/docs/how-to/troubleshooting.md b/docs/docs/how-to/troubleshooting.md index 76cac2b..2c46e88 100644 --- a/docs/docs/how-to/troubleshooting.md +++ b/docs/docs/how-to/troubleshooting.md @@ -10,14 +10,14 @@ Before diving into specific errors, let’s check the basics: ```bash # Check all FuzzForge services -docker-compose -f docker-compose.temporal.yaml ps +docker-compose -f docker-compose.yml ps # Verify Docker registry config (if using workflow registry) docker info | grep -i "insecure registries" # Test service health endpoints curl http://localhost:8000/health -curl http://localhost:8233 # Temporal Web UI +curl http://localhost:8080 # Temporal Web UI curl http://localhost:9000 # MinIO API curl http://localhost:9001 # MinIO Console ``` @@ -54,13 +54,13 @@ The registry isn’t running or the port is blocked. **How to fix:** - Make sure the registry container is up (if using registry for workflow images): ```bash - docker-compose -f docker-compose.temporal.yaml ps registry + docker-compose -f docker-compose.yml ps registry ``` - Check logs for errors: ```bash - docker-compose -f docker-compose.temporal.yaml logs registry + docker-compose -f docker-compose.yml logs registry ``` -- If port 5001 is in use, change it in `docker-compose.temporal.yaml` and your Docker config. +- If port 5001 is in use, change it in `docker-compose.yml` and your Docker config. **Note:** With Temporal architecture, target files use MinIO (port 9000), not the registry. @@ -85,11 +85,11 @@ File upload to MinIO failed or worker can't download target. **How to fix:** - Check MinIO is running: ```bash - docker-compose -f docker-compose.temporal.yaml ps minio + docker-compose -f docker-compose.yml ps minio ``` - Check MinIO logs: ```bash - docker-compose -f docker-compose.temporal.yaml logs minio + docker-compose -f docker-compose.yml logs minio ``` - Verify MinIO is accessible: ```bash @@ -106,13 +106,13 @@ File upload to MinIO failed or worker can't download target. **How to fix:** - Check worker logs for details: ```bash - docker-compose -f docker-compose.temporal.yaml logs worker-rust | tail -50 + docker-compose -f docker-compose.yml logs worker-rust | tail -50 ``` -- Check Temporal Web UI at http://localhost:8233 for detailed execution history +- Check Temporal Web UI at http://localhost:8080 for detailed execution history - Restart services: ```bash - docker-compose -f docker-compose.temporal.yaml down - docker-compose -f docker-compose.temporal.yaml up -d + docker-compose -f docker-compose.yml down + docker-compose -f docker-compose.yml up -d ``` - Reduce the number of concurrent workflows if your system is resource-constrained. @@ -125,18 +125,18 @@ File upload to MinIO failed or worker can't download target. **How to fix:** - Check if the service is running: ```bash - docker-compose -f docker-compose.temporal.yaml ps fuzzforge-backend - docker-compose -f docker-compose.temporal.yaml ps temporal + docker-compose -f docker-compose.yml ps fuzzforge-backend + docker-compose -f docker-compose.yml ps temporal ``` - View logs for errors: ```bash - docker-compose -f docker-compose.temporal.yaml logs fuzzforge-backend --tail 50 - docker-compose -f docker-compose.temporal.yaml logs temporal --tail 20 + docker-compose -f docker-compose.yml logs fuzzforge-backend --tail 50 + docker-compose -f docker-compose.yml logs temporal --tail 20 ``` - Restart the affected service: ```bash - docker-compose -f docker-compose.temporal.yaml restart fuzzforge-backend - docker-compose -f docker-compose.temporal.yaml restart temporal + docker-compose -f docker-compose.yml restart fuzzforge-backend + docker-compose -f docker-compose.yml restart temporal ``` --- @@ -216,9 +216,9 @@ File upload to MinIO failed or worker can't download target. ``` - Recreate the network: ```bash - docker-compose -f docker-compose.temporal.yaml down + docker-compose -f docker-compose.yml down docker network prune -f - docker-compose -f docker-compose.temporal.yaml up -d + docker-compose -f docker-compose.yml up -d ``` --- @@ -245,9 +245,9 @@ File upload to MinIO failed or worker can't download target. ```bash export TEMPORAL_LOGGING_LEVEL=DEBUG -docker-compose -f docker-compose.temporal.yaml down -docker-compose -f docker-compose.temporal.yaml up -d -docker-compose -f docker-compose.temporal.yaml logs fuzzforge-backend -f +docker-compose -f docker-compose.yml down +docker-compose -f docker-compose.yml up -d +docker-compose -f docker-compose.yml logs fuzzforge-backend -f ``` ### Collect diagnostic info @@ -258,12 +258,12 @@ Save and run this script to gather info for support: #!/bin/bash echo "=== FuzzForge Diagnostics ===" date -docker-compose -f docker-compose.temporal.yaml ps +docker-compose -f docker-compose.yml ps docker info | grep -A 5 -i "insecure registries" curl -s http://localhost:8000/health || echo "Backend unhealthy" -curl -s http://localhost:8233 >/dev/null && echo "Temporal UI healthy" || echo "Temporal UI unhealthy" +curl -s http://localhost:8080 >/dev/null && echo "Temporal UI healthy" || echo "Temporal UI unhealthy" curl -s http://localhost:9000 >/dev/null && echo "MinIO healthy" || echo "MinIO unhealthy" -docker-compose -f docker-compose.temporal.yaml logs --tail 10 +docker-compose -f docker-compose.yml logs --tail 10 ``` ### Still stuck? diff --git a/docs/docs/tutorial/getting-started.md b/docs/docs/tutorial/getting-started.md index 426b1c6..939e2ad 100644 --- a/docs/docs/tutorial/getting-started.md +++ b/docs/docs/tutorial/getting-started.md @@ -17,7 +17,7 @@ Before we begin, ensure you have the following installed: First, let's clone the FuzzForge repository: ```bash -git clone https://github.com/FuzzingLabs/fuzzforge.git +git clone https://github.com/FuzzingLabs/fuzzforge_ai.git cd fuzzforge ``` @@ -85,7 +85,7 @@ docker pull localhost:5001/hello-world 2>/dev/null || echo "Registry not accessi Start all FuzzForge services: ```bash -docker-compose -f docker-compose.temporal.yaml up -d +docker-compose -f docker-compose.yml up -d ``` This will start 6+ services: @@ -101,7 +101,7 @@ Wait for all services to be healthy (this may take 2-3 minutes on first startup) ```bash # Check service health -docker-compose -f docker-compose.temporal.yaml ps +docker-compose -f docker-compose.yml ps # Verify FuzzForge is ready curl http://localhost:8000/health @@ -228,7 +228,7 @@ Example output: You can monitor workflow execution in real-time using the Temporal Web UI: -1. Open http://localhost:8233 in your browser +1. Open http://localhost:8080 in your browser 2. Navigate to "Workflows" to see workflow executions 3. Click on a workflow to see detailed execution history and activity results From 37c15af1304a4918efa49ff4846678ccbf2ef97f Mon Sep 17 00:00:00 2001 From: tduhamel42 Date: Fri, 17 Oct 2025 10:28:17 +0200 Subject: [PATCH 2/5] docs: Remove obsolete Docker registry configuration Updated documentation to reflect v0.7.0 Temporal architecture which uses MinIO for storage instead of a Docker registry. Major changes: - getting-started.md: Added mandatory volumes/env/.env setup, removed registry config section, updated service list - docker-setup.md: Complete rewrite focusing on system requirements and worker profiles instead of registry - index.md: Replaced registry warning with environment file requirement - troubleshooting.md: Removed all registry troubleshooting, added environment configuration issues --- docs/docs/how-to/docker-setup.md | 352 ++++++++++++++------------ docs/docs/how-to/troubleshooting.md | 85 +++---- docs/docs/tutorial/getting-started.md | 97 +++---- docs/index.md | 14 +- 4 files changed, 268 insertions(+), 280 deletions(-) diff --git a/docs/docs/how-to/docker-setup.md b/docs/docs/how-to/docker-setup.md index 6448726..993aa12 100644 --- a/docs/docs/how-to/docker-setup.md +++ b/docs/docs/how-to/docker-setup.md @@ -1,226 +1,242 @@ -# How to Configure Docker for FuzzForge +# Docker Requirements for FuzzForge -Getting Docker set up correctly is essential for running FuzzForge workflows. This guide will walk you through the process, explain why each step matters, and help you troubleshoot common issuesβ€”so you can get up and running with confidence. +FuzzForge runs entirely in Docker containers with Temporal orchestration. This guide covers system requirements, worker profiles, and resource management to help you run FuzzForge efficiently. --- -## Why Does FuzzForge Need Special Docker Configuration? +## System Requirements -FuzzForge builds and runs custom workflow images using a local Docker registry at `localhost:5001`. By default, Docker only trusts secure (HTTPS) registries, so you need to explicitly allow this local, insecure registry for development. Without this, workflows that build or pull images will fail. +### Docker Version ---- +- **Docker Engine**: 20.10.0 or later +- **Docker Compose**: 2.0.0 or later -## Quick Setup: The One-Liner +Verify your installation: -Add this to your Docker daemon configuration: - -```json -{ - "insecure-registries": ["localhost:5001"] -} +```bash +docker --version +docker compose version ``` -After editing, **restart Docker** for changes to take effect. +### Hardware Requirements + +| Component | Minimum | Recommended | +|-----------|---------|-------------| +| CPU | 2 cores | 4+ cores | +| RAM | 4 GB | 8 GB+ | +| Disk | 20 GB free | 50 GB+ free | +| Network | Internet access | Stable connection | + +### Port Requirements + +FuzzForge services use these ports (must be available): + +| Port | Service | Purpose | +|------|---------|---------| +| 8000 | Backend API | FastAPI server | +| 8080 | Temporal UI | Workflow monitoring | +| 7233 | Temporal gRPC | Workflow execution | +| 9000 | MinIO API | S3-compatible storage | +| 9001 | MinIO Console | Storage management | +| 5432 | PostgreSQL | Temporal database | + +Check for port conflicts: + +```bash +# macOS/Linux +lsof -i :8000,8080,7233,9000,9001,5432 + +# Or just try starting FuzzForge +docker compose -f docker-compose.yml up -d +``` --- -## Step-by-Step: Platform-Specific Instructions +## Worker Profiles (Resource Optimization) -### Docker Desktop (macOS & Windows) +FuzzForge uses Docker Compose **profiles** to prevent workers from auto-starting. This saves 5-7GB of RAM by only running workers when needed. -#### Using the Docker Desktop UI +### Profile Configuration -1. Open Docker Desktop. -2. Go to **Settings** (Windows) or **Preferences** (macOS). -3. Navigate to **Docker Engine**. -4. Add or update the `"insecure-registries"` section as shown above. -5. Click **Apply & Restart**. -6. Wait for Docker to restart (this may take a minute). - -#### Editing the Config File Directly - -- **macOS:** Edit `~/.docker/daemon.json` -- **Windows:** Edit `%USERPROFILE%\.docker\daemon.json` - -Add or update the `"insecure-registries"` entry, then restart Docker Desktop. - ---- - -### Docker Engine (Linux) - -1. Edit (or create) `/etc/docker/daemon.json`: - ```bash - sudo nano /etc/docker/daemon.json - ``` -2. Add: - ```json - { - "insecure-registries": ["localhost:5001"] - } - ``` -3. Restart Docker: - ```bash - sudo systemctl restart docker - ``` -4. Confirm Docker is running: - ```bash - sudo systemctl status docker - ``` - -#### Alternative: Systemd Drop-in (Advanced) - -If you prefer, you can use a systemd override to add the registry flag. See the original guide for details. - ---- - -## Verifying Your Configuration - -1. **Check Docker’s registry settings:** - ```bash - docker info | grep -i "insecure registries" - ``` - You should see `localhost:5001` listed. - -2. **Test the registry:** - ```bash - curl -f http://localhost:5001/v2/ && echo "βœ… Registry accessible" || echo "❌ Registry not accessible" - ``` - -3. **Try pushing and pulling an image:** - ```bash - docker pull hello-world - docker tag hello-world localhost:5001/hello-world:test - docker push localhost:5001/hello-world:test - docker rmi localhost:5001/hello-world:test - docker pull localhost:5001/hello-world:test - ``` - ---- - -## Worker Profiles (Resource Optimization - v0.7.0) - -FuzzForge workers use Docker Compose profiles to prevent auto-startup: +Workers are configured with profiles in `docker-compose.yml`: ```yaml -# docker-compose.yml worker-ossfuzz: profiles: - workers # For starting all workers - ossfuzz # For starting just this worker restart: "no" # Don't auto-restart + +worker-python: + profiles: + - workers + - python + restart: "no" + +worker-rust: + profiles: + - workers + - rust + restart: "no" ``` -### Behavior +### Default Behavior -- **`docker-compose up -d`**: Workers DON'T start (saves ~6-7GB RAM) -- **CLI workflows**: Workers start automatically on-demand -- **Manual start**: `docker start fuzzforge-worker-ossfuzz` +**`docker compose up -d`** starts only core services: +- temporal +- temporal-ui +- postgresql +- minio +- minio-setup +- backend +- task-agent -### Resource Savings +Workers remain stopped until needed. + +### On-Demand Worker Startup + +When you run a workflow via the CLI, FuzzForge automatically starts the required worker: + +```bash +# Automatically starts worker-python +fuzzforge workflow run atheris_fuzzing ./project + +# Automatically starts worker-rust +fuzzforge workflow run cargo_fuzzing ./rust-project + +# Automatically starts worker-secrets +fuzzforge workflow run secret_detection ./codebase +``` + +### Manual Worker Management + +Start specific workers when needed: + +```bash +# Start a single worker +docker start fuzzforge-worker-python + +# Start all workers at once (uses more RAM) +docker compose --profile workers up -d + +# Stop a worker to free resources +docker stop fuzzforge-worker-ossfuzz +``` + +### Resource Comparison | Command | Workers Started | RAM Usage | |---------|----------------|-----------| -| `docker-compose up -d` | None (core only) | ~1.2 GB | -| `ff workflow run ossfuzz_campaign .` | ossfuzz worker only | ~3-5 GB | -| `docker-compose --profile workers up -d` | All workers | ~8 GB | +| `docker compose up -d` | None (core only) | ~1.2 GB | +| `fuzzforge workflow run atheris_fuzzing .` | Python worker only | ~2-3 GB | +| `fuzzforge workflow run ossfuzz_campaign .` | OSS-Fuzz worker only | ~3-5 GB | +| `docker compose --profile workers up -d` | All workers | ~8 GB | -### Starting All Workers (Legacy Behavior) +--- -If you prefer the old behavior where all workers start: +## Storage Management + +### Docker Volume Cleanup + +FuzzForge creates Docker volumes for persistent data. Clean them up periodically: ```bash -docker-compose --profile workers up -d +# Remove unused volumes (safe - keeps volumes for running containers) +docker volume prune + +# List FuzzForge volumes +docker volume ls | grep fuzzforge + +# Remove specific volume (WARNING: deletes data) +docker volume rm fuzzforge_minio-data +``` + +### Cache Directory + +Workers use `/cache` inside containers for downloaded targets. This is managed automatically with LRU eviction, but you can check usage: + +```bash +# Check cache usage in a worker +docker exec fuzzforge-worker-python du -sh /cache + +# Clear cache manually if needed (safe - will re-download targets) +docker exec fuzzforge-worker-python rm -rf /cache/* ``` --- -## Common Issues & How to Fix Them +## Environment Configuration -### "x509: certificate signed by unknown authority" +FuzzForge requires `volumes/env/.env` to start. This file contains API keys and configuration: -- **What’s happening?** Docker is trying to use HTTPS for the registry. -- **How to fix:** Double-check your `"insecure-registries"` config and restart Docker. +```bash +# Copy the example file +cp volumes/env/.env.example volumes/env/.env -### "connection refused" to localhost:5001 +# Edit to add your API keys (if using AI features) +nano volumes/env/.env +``` -- **What’s happening?** The registry isn’t running or the port is blocked. -- **How to fix:** Make sure FuzzForge services are up (`docker compose ps`), and that nothing else is using port 5001. - -### Docker Desktop doesn’t apply settings - -- **How to fix:** Fully quit and restart Docker Desktop. Check for typos in your JSON config. - -### "permission denied" on Linux - -- **How to fix:** Add your user to the `docker` group: - ```bash - sudo usermod -aG docker $USER - newgrp docker - ``` +See [Getting Started](../tutorial/getting-started.md) for detailed environment setup. --- -## Security Notes +## Troubleshooting -- Using an insecure registry on `localhost:5001` is safe for local development. -- For production, always use a secure (HTTPS) registry and proper authentication. +### Services Won't Start + +**Check ports are available:** +```bash +docker compose -f docker-compose.yml ps +lsof -i :8000,8080,7233,9000,9001 +``` + +**Check Docker resources:** +```bash +docker system df +docker system prune # Free up space if needed +``` + +### Worker Memory Issues + +If workers crash with OOM (out of memory) errors: + +1. **Close other applications** to free RAM +2. **Start only needed workers** (don't use `--profile workers`) +3. **Increase Docker Desktop memory limit** (Settings β†’ Resources) +4. **Monitor usage**: `docker stats` + +### Slow Worker Startup + +Workers pull large images (~2-5GB each) on first run: + +```bash +# Check download progress +docker compose logs worker-python -f + +# Pre-pull images (optional) +docker compose pull +``` --- -## Where Are Docker Config Files? +## Best Practices -- **Docker Desktop (macOS):** `~/.docker/daemon.json` -- **Docker Desktop (Windows):** `%USERPROFILE%\.docker\daemon.json` -- **Docker Engine (Linux):** `/etc/docker/daemon.json` - -**Tip:** Always back up your config before making changes. - ---- - -## Advanced: Example Configurations - -### Minimal - -```json -{ - "insecure-registries": ["localhost:5001"] -} -``` - -### With Logging and Storage Options - -```json -{ - "insecure-registries": ["localhost:5001"], - "log-driver": "json-file", - "log-opts": { - "max-size": "10m", - "max-file": "3" - }, - "storage-driver": "overlay2" -} -``` - -### Multiple Registries - -```json -{ - "insecure-registries": [ - "localhost:5001", - "192.168.1.100:5000", - "registry.internal:5000" - ] -} -``` +1. **Default startup**: Use `docker compose up -d` (core services only) +2. **Let CLI manage workers**: Workers start automatically when workflows run +3. **Stop unused workers**: Free RAM when not running workflows +4. **Monitor resources**: `docker stats` shows real-time usage +5. **Regular cleanup**: `docker system prune` removes unused images/containers +6. **Backup volumes**: `docker volume ls` shows persistent data locations --- ## Next Steps -- [Getting Started Guide](../tutorial/getting-started.md): Continue with FuzzForge setup. -- [Troubleshooting](troubleshooting.md): For more help if things don’t work. +- [Getting Started Guide](../tutorial/getting-started.md): Complete setup walkthrough +- [Running Workflows](running-workflows.md): Execute security workflows +- [Troubleshooting](troubleshooting.md): Fix common issues --- -**Remember:** -After any Docker config change, always restart Docker and verify your settings with `docker info` before running FuzzForge. +**Remember:** FuzzForge's on-demand worker startup saves resources. You don't need to manually manage workers - the CLI does it automatically! diff --git a/docs/docs/how-to/troubleshooting.md b/docs/docs/how-to/troubleshooting.md index 2c46e88..8784ef3 100644 --- a/docs/docs/how-to/troubleshooting.md +++ b/docs/docs/how-to/troubleshooting.md @@ -10,10 +10,7 @@ Before diving into specific errors, let’s check the basics: ```bash # Check all FuzzForge services -docker-compose -f docker-compose.yml ps - -# Verify Docker registry config (if using workflow registry) -docker info | grep -i "insecure registries" +docker compose -f docker-compose.yml ps # Test service health endpoints curl http://localhost:8000/health @@ -26,52 +23,45 @@ If any of these commands fail, note the error message and continue below. --- -## Docker Registry Problems +## Environment Configuration Issues -### "x509: certificate signed by unknown authority" +### Docker Compose fails with "file not found" or variable errors -**What’s happening?** -Docker is trying to use HTTPS for the local registry, but it’s set up for HTTP. +**What's happening?** +The required `volumes/env/.env` file is missing. Docker Compose needs this file to start. **How to fix:** -1. Add this to your Docker daemon config (e.g., `/etc/docker/daemon.json`): - ```json - { - "insecure-registries": ["localhost:5001"] - } +```bash +# Create the environment file from the template +cp volumes/env/.env.example volumes/env/.env + +# Restart Docker Compose +docker compose -f docker-compose.yml down +docker compose -f docker-compose.yml up -d +``` + +**Note:** You can leave the `.env` file with default values if you're only using basic workflows (no AI features). + +### API key errors for AI features + +**What's happening?** +AI-powered workflows (like `llm_secret_detection`) or the AI agent need API keys. + +**How to fix:** +1. Edit `volumes/env/.env` and add your keys: + ```env + LITELLM_MODEL=gpt-4o-mini + OPENAI_API_KEY=sk-your-key-here ``` -2. Restart Docker. -3. Confirm with: + +2. Restart the backend to pick up new environment variables: ```bash - docker info | grep -A 5 -i "insecure registries" + docker compose restart backend ``` -### "connection refused" to localhost:5001 - -**What’s happening?** -The registry isn’t running or the port is blocked. - -**How to fix:** -- Make sure the registry container is up (if using registry for workflow images): - ```bash - docker-compose -f docker-compose.yml ps registry - ``` -- Check logs for errors: - ```bash - docker-compose -f docker-compose.yml logs registry - ``` -- If port 5001 is in use, change it in `docker-compose.yml` and your Docker config. - -**Note:** With Temporal architecture, target files use MinIO (port 9000), not the registry. - -### "no such host" error - -**What’s happening?** -Docker can’t resolve `localhost`. - -**How to fix:** -- Try using `127.0.0.1` instead of `localhost` in your Docker config. -- Check your `/etc/hosts` file for a correct `127.0.0.1 localhost` entry. +**Which workflows need API keys?** +- βœ… **Don't need keys**: `security_assessment`, `gitleaks_detection`, `trufflehog_detection`, `atheris_fuzzing`, `cargo_fuzzing` +- ⚠️ **Need keys**: `llm_secret_detection`, AI agent (`fuzzforge ai agent`) --- @@ -189,10 +179,12 @@ File upload to MinIO failed or worker can't download target. ```bash docker system prune -f docker image prune -f + docker volume prune -f # Remove unused volumes ``` -- Remove old workflow images: +- Clean worker cache manually if needed: ```bash - docker images | grep localhost:5001 | awk '{print $3}' | xargs docker rmi -f + docker exec fuzzforge-worker-python rm -rf /cache/* + docker exec fuzzforge-worker-rust rm -rf /cache/* ``` ### High memory usage @@ -258,12 +250,11 @@ Save and run this script to gather info for support: #!/bin/bash echo "=== FuzzForge Diagnostics ===" date -docker-compose -f docker-compose.yml ps -docker info | grep -A 5 -i "insecure registries" +docker compose -f docker-compose.yml ps curl -s http://localhost:8000/health || echo "Backend unhealthy" curl -s http://localhost:8080 >/dev/null && echo "Temporal UI healthy" || echo "Temporal UI unhealthy" curl -s http://localhost:9000 >/dev/null && echo "MinIO healthy" || echo "MinIO unhealthy" -docker-compose -f docker-compose.yml logs --tail 10 +docker compose -f docker-compose.yml logs --tail 10 ``` ### Still stuck? diff --git a/docs/docs/tutorial/getting-started.md b/docs/docs/tutorial/getting-started.md index 939e2ad..6c0e9a5 100644 --- a/docs/docs/tutorial/getting-started.md +++ b/docs/docs/tutorial/getting-started.md @@ -18,67 +18,39 @@ First, let's clone the FuzzForge repository: ```bash git clone https://github.com/FuzzingLabs/fuzzforge_ai.git -cd fuzzforge +cd fuzzforge_ai ``` -Create the required environment configuration: +## Step 2: Configure Environment (Required) + +**⚠️ This step is mandatory - Docker Compose will fail without it.** + +Create the environment configuration file: ```bash -echo "COMPOSE_PROJECT_NAME=fuzzforge" > .env +cp volumes/env/.env.example volumes/env/.env ``` -## Step 2: Configure Docker (Critical Step) +This file is required for FuzzForge to start. You can leave it with default values if you're only using basic workflows. -**⚠️ This step is mandatory - workflows will fail without proper Docker configuration.** +### When you need to add API keys: -FuzzForge uses a local Docker registry at `localhost:5001` for workflow images. You must configure Docker to allow this insecure registry. +Edit `volumes/env/.env` and add your API keys if using: +- **AI-powered workflows**: `llm_secret_detection` +- **AI agent**: `ff ai agent` -### For Docker Desktop (macOS/Windows): - -1. Open Docker Desktop -2. Go to Settings/Preferences β†’ Docker Engine -3. Add the following to the JSON configuration: - -```json -{ - "builder": { - "gc": { - "defaultKeepStorage": "20GB", - "enabled": true - } - }, - "experimental": false, - "insecure-registries": [ - "localhost:5001" - ] -} +Example: +```env +LITELLM_MODEL=gpt-4o-mini +OPENAI_API_KEY=sk-your-key-here ``` -4. Click "Apply & Restart" - -### For Docker Engine (Linux): - -Edit `/etc/docker/daemon.json` (create if it doesn't exist): - -```json -{ - "insecure-registries": ["localhost:5001"] -} -``` - -Restart Docker: -```bash -sudo systemctl restart docker -``` - -### Verify Docker Configuration - -Test that Docker can access the insecure registry: - -```bash -# After starting FuzzForge (next step), this should work without certificate errors -docker pull localhost:5001/hello-world 2>/dev/null || echo "Registry not accessible yet (normal before startup)" -``` +### Basic workflows that don't need API keys: +- `security_assessment` +- `gitleaks_detection` +- `trufflehog_detection` +- `atheris_fuzzing` +- `cargo_fuzzing` ## Step 3: Start FuzzForge @@ -88,14 +60,23 @@ Start all FuzzForge services: docker-compose -f docker-compose.yml up -d ``` -This will start 6+ services: -- **temporal**: Workflow orchestration server (includes embedded PostgreSQL for dev) -- **minio**: S3-compatible storage for uploaded targets and results -- **minio-setup**: One-time setup for MinIO buckets (exits after setup) -- **fuzzforge-backend**: FastAPI backend and workflow management -- **worker-rust**: Long-lived worker for Rust/native security analysis -- **worker-android**: Long-lived worker for Android security analysis (if configured) -- **worker-web**: Long-lived worker for web security analysis (if configured) +This will start the FuzzForge platform with multiple services: + +**Core Services:** +- **temporal**: Workflow orchestration server +- **temporal-ui**: Web UI for workflow monitoring (port 8080) +- **postgresql**: Database for Temporal state +- **minio**: S3-compatible storage for targets and results +- **minio-setup**: One-time MinIO bucket configuration (exits after setup) +- **backend**: FastAPI server and workflow management API +- **task-agent**: A2A agent coordination service + +**Security Workers** (vertical toolchains): +- **worker-python**: Python security analysis and Atheris fuzzing +- **worker-rust**: Rust/Cargo fuzzing and native code analysis +- **worker-secrets**: Secret detection and credential scanning +- **worker-ossfuzz**: OSS-Fuzz integration (heavy development) +- **worker-android**: Android security analysis (early development) Wait for all services to be healthy (this may take 2-3 minutes on first startup): diff --git a/docs/index.md b/docs/index.md index fc2a6a8..a5a3a24 100644 --- a/docs/index.md +++ b/docs/index.md @@ -84,17 +84,17 @@ Technical reference materials and specifications. ## 🚨 Important Setup Requirement -**Docker Insecure Registry Configuration Required** +**Environment Configuration Required** -FuzzForge uses a local Docker registry for workflow images. You **must** configure Docker to allow insecure registries: +Before starting FuzzForge, you **must** create the environment configuration file: -```json -{ - "insecure-registries": ["localhost:5001"] -} +```bash +cp volumes/env/.env.example volumes/env/.env ``` -See [Docker Setup Guide](how-to/docker-setup.md) for detailed configuration instructions. +Docker Compose will fail without this file. You can leave it with default values if you're only using basic workflows (no AI features). + +See [Getting Started Guide](tutorials/getting-started.md) for detailed setup instructions. --- From f14bec9410ea731aed596e3f1fddae29e585984a Mon Sep 17 00:00:00 2001 From: tduhamel42 Date: Fri, 17 Oct 2025 10:29:38 +0200 Subject: [PATCH 3/5] docs: Update architecture diagram to reflect Temporal/MinIO architecture - Removed Docker Registry from execution layer diagram - Updated diagram to show vertical workers with MinIO storage - Removed obsolete COMPOSE_PROJECT_NAME from example configuration --- docs/docs/concept/architecture.md | 19 +++++++++---------- 1 file changed, 9 insertions(+), 10 deletions(-) diff --git a/docs/docs/concept/architecture.md b/docs/docs/concept/architecture.md index a7e7429..fcd922f 100644 --- a/docs/docs/concept/architecture.md +++ b/docs/docs/concept/architecture.md @@ -52,14 +52,14 @@ graph TB end subgraph "Execution Layer" - Docker[Docker Engine] - Containers[Workflow Containers] - Registry[Docker Registry] + VerticalWorkers[Vertical Worker Containers] + Tools[Pre-installed Toolchains] + WorkerCache[Worker Cache /cache] end subgraph "Storage Layer" PostgreSQL[PostgreSQL Database] - Volumes[Docker Volumes] + MinIO[MinIO S3 Storage] Cache[Result Cache] end @@ -73,15 +73,15 @@ graph TB Temporal --> Workers Workers --> Scheduler - Scheduler --> Docker + Scheduler --> VerticalWorkers - Docker --> Containers - Docker --> Registry - Containers --> Volumes + VerticalWorkers --> Tools + VerticalWorkers --> WorkerCache + VerticalWorkers --> MinIO FastAPI --> PostgreSQL Workers --> PostgreSQL - Containers --> Cache + FastAPI --> MinIO ``` ## What Are the Main Components? @@ -201,7 +201,6 @@ services: Example configuration: ```bash -COMPOSE_PROJECT_NAME=fuzzforge DATABASE_URL=postgresql://postgres:postgres@postgres:5432/fuzzforge TEMPORAL_ADDRESS=temporal:7233 S3_ENDPOINT=http://minio:9000 From 187b1713606c9ae1f5b8bd2edb17f90f3eee296d Mon Sep 17 00:00:00 2001 From: tduhamel42 Date: Fri, 17 Oct 2025 10:48:48 +0200 Subject: [PATCH 4/5] docs: Fix workflow references and module paths for v0.7.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Updated all documentation to reflect actual v0.7.0 workflow implementation: Workflow name changes: - Removed all references to non-existent workflows (static_analysis_scan, secret_detection_scan, infrastructure_scan, penetration_testing_scan) - Updated examples to use actual workflows (security_assessment, gitleaks_detection, trufflehog_detection, llm_secret_detection) - Deleted docs/docs/reference/workflows/static-analysis.md (described non-existent workflow) Content corrections: - Fixed workflow tool descriptions (removed incorrect Semgrep/Bandit references, documented actual SecurityAnalyzer and FileScanner modules) - Updated all workflow lists to show production-ready vs development status - Fixed all example configurations to match actual workflow parameters Module creation guide fixes: - Fixed 4 path references: backend/src/toolbox β†’ backend/toolbox - Updated import statements in example code Files updated: - docs/index.md - workflow list, CLI example, broken tutorial links - docs/docs/tutorial/getting-started.md - workflow list, example output, tool descriptions - docs/docs/how-to/create-module.md - module paths and imports - docs/docs/how-to/mcp-integration.md - workflow examples and list - docs/docs/ai/prompts.md - workflow example - docs/docs/reference/cli-ai.md - 3 workflow references --- docs/docs/ai/prompts.md | 2 +- docs/docs/how-to/create-module.md | 8 +- docs/docs/how-to/mcp-integration.md | 29 +- docs/docs/reference/cli-ai.md | 29 +- .../reference/workflows/static-analysis.md | 257 ------------------ docs/docs/tutorial/getting-started.md | 25 +- docs/index.md | 28 +- 7 files changed, 69 insertions(+), 309 deletions(-) delete mode 100644 docs/docs/reference/workflows/static-analysis.md diff --git a/docs/docs/ai/prompts.md b/docs/docs/ai/prompts.md index 7ac5859..5669cad 100644 --- a/docs/docs/ai/prompts.md +++ b/docs/docs/ai/prompts.md @@ -20,7 +20,7 @@ Use the `fuzzforge ai agent` shell to mix structured slash commands with natural You> list available fuzzforge workflows Assistant> [returns workflow names, descriptions, and required parameters] -You> run fuzzforge workflow static_analysis_scan on ./backend with target_branch=main +You> run fuzzforge workflow security_assessment on ./backend Assistant> Submits the run, emits TaskStatusUpdateEvent entries, and links the SARIF artifact when complete. You> show findings for that run once it finishes diff --git a/docs/docs/how-to/create-module.md b/docs/docs/how-to/create-module.md index 7e9d7c4..8d88f47 100644 --- a/docs/docs/how-to/create-module.md +++ b/docs/docs/how-to/create-module.md @@ -25,7 +25,7 @@ All FuzzForge modules inherit from a common `BaseModule` interface and use Pydan - `ModuleResult`: Standardized result format for module execution - `ModuleMetadata`: Describes module capabilities and requirements -Modules are located in `backend/src/toolbox/modules/`. +Modules are located in `backend/toolbox/modules/`. --- @@ -34,7 +34,7 @@ Modules are located in `backend/src/toolbox/modules/`. Let’s create a simple example: a **License Scanner** module that detects license files and extracts license information. Create a new file: -`backend/src/toolbox/modules/license_scanner.py` +`backend/toolbox/modules/license_scanner.py` ```python import re @@ -98,7 +98,7 @@ class LicenseScanner(BaseModule): ## Step 3: Register Your Module -Add your module to `backend/src/toolbox/modules/__init__.py`: +Add your module to `backend/toolbox/modules/__init__.py`: ```python from .license_scanner import LicenseScanner @@ -115,7 +115,7 @@ Create a test file (e.g., `test_license_scanner.py`) and run your module against ```python import asyncio from pathlib import Path -from backend.src.toolbox.modules.license_scanner import LicenseScanner +from toolbox.modules.license_scanner import LicenseScanner async def main(): workspace = Path("/path/to/your/test/project") diff --git a/docs/docs/how-to/mcp-integration.md b/docs/docs/how-to/mcp-integration.md index 3470bd9..f22123b 100644 --- a/docs/docs/how-to/mcp-integration.md +++ b/docs/docs/how-to/mcp-integration.md @@ -81,14 +81,17 @@ You should see status responses and endpoint listings. { "tool": "submit_security_scan_mcp", "parameters": { - "workflow_name": "infrastructure_scan", + "workflow_name": "security_assessment", "target_path": "/path/to/your/project", "parameters": { - "checkov_config": { - "severity": ["HIGH", "MEDIUM", "LOW"] + "scanner_config": { + "patterns": ["*"], + "check_sensitive": true }, - "hadolint_config": { - "severity": ["error", "warning", "info", "style"] + "analyzer_config": { + "file_extensions": [".py", ".js", ".java"], + "check_secrets": true, + "check_sql": true } } } @@ -110,13 +113,17 @@ You should see status responses and endpoint listings. ## 6. Available Workflows -You can trigger these workflows via MCP: +You can trigger these production-ready workflows via MCP: -1. **infrastructure_scan** β€” Docker/Kubernetes/Terraform security analysis -2. **static_analysis_scan** β€” Code vulnerability detection -3. **secret_detection_scan** β€” Credential and secret scanning -4. **penetration_testing_scan** β€” Network and web app testing -5. **security_assessment** β€” Comprehensive security evaluation +1. **security_assessment** β€” Comprehensive security analysis (secrets, SQL, dangerous functions) +2. **gitleaks_detection** β€” Pattern-based secret scanning +3. **trufflehog_detection** β€” Pattern-based secret scanning +4. **llm_secret_detection** β€” AI-powered secret detection (requires API key) + +Development workflows (early stages): +- **atheris_fuzzing** β€” Python fuzzing +- **cargo_fuzzing** β€” Rust fuzzing +- **ossfuzz_campaign** β€” OSS-Fuzz integration List all workflows: diff --git a/docs/docs/reference/cli-ai.md b/docs/docs/reference/cli-ai.md index a821235..2e69613 100644 --- a/docs/docs/reference/cli-ai.md +++ b/docs/docs/reference/cli-ai.md @@ -134,14 +134,16 @@ FuzzForge supports the Model Context Protocol (MCP), allowing LLM clients and AI { "tool": "submit_security_scan_mcp", "parameters": { - "workflow_name": "infrastructure_scan", + "workflow_name": "security_assessment", "target_path": "/path/to/your/project", "parameters": { - "checkov_config": { - "severity": ["HIGH", "MEDIUM", "LOW"] + "scanner_config": { + "patterns": ["*"], + "check_sensitive": true }, - "hadolint_config": { - "severity": ["error", "warning", "info", "style"] + "analyzer_config": { + "file_extensions": [".py", ".js"], + "check_secrets": true } } } @@ -161,11 +163,16 @@ FuzzForge supports the Model Context Protocol (MCP), allowing LLM clients and AI ### Available Workflows -1. **infrastructure_scan** β€” Docker/Kubernetes/Terraform security analysis -2. **static_analysis_scan** β€” Code vulnerability detection -3. **secret_detection_scan** β€” Credential and secret scanning -4. **penetration_testing_scan** β€” Network and web app testing -5. **security_assessment** β€” Comprehensive security evaluation +**Production-ready:** +1. **security_assessment** β€” Comprehensive security analysis (secrets, SQL, dangerous functions) +2. **gitleaks_detection** β€” Pattern-based secret scanning +3. **trufflehog_detection** β€” Pattern-based secret scanning +4. **llm_secret_detection** β€” AI-powered secret detection (requires API key) + +**In development:** +- **atheris_fuzzing** β€” Python fuzzing +- **cargo_fuzzing** β€” Rust fuzzing +- **ossfuzz_campaign** β€” OSS-Fuzz integration ### MCP Client Configuration Example @@ -192,7 +199,7 @@ FuzzForge supports the Model Context Protocol (MCP), allowing LLM clients and AI `curl http://localhost:8000/workflows/` - **Scan Submission Errors:** - `curl -X POST http://localhost:8000/workflows/infrastructure_scan/submit -H "Content-Type: application/json" -d '{"target_path": "/your/path"}'` + `curl -X POST http://localhost:8000/workflows/security_assessment/submit -H "Content-Type: application/json" -d '{"target_path": "/your/path"}'` - **General Support:** - Check Docker Compose logs: `docker compose logs fuzzforge-backend` diff --git a/docs/docs/reference/workflows/static-analysis.md b/docs/docs/reference/workflows/static-analysis.md deleted file mode 100644 index 48624aa..0000000 --- a/docs/docs/reference/workflows/static-analysis.md +++ /dev/null @@ -1,257 +0,0 @@ -# Static Analysis Workflow Reference - -The Static Analysis workflow in FuzzForge helps you find vulnerabilities, code quality issues, and compliance problemsβ€”before they reach production. This workflow uses multiple Static Application Security Testing (SAST) tools to analyze your source code without executing it, providing fast, actionable feedback in a standardized format. - ---- - -## What Does This Workflow Do? - -- **Workflow ID:** `static_analysis_scan` -- **Primary Tools:** Semgrep (multi-language), Bandit (Python) -- **Supported Languages:** Python, JavaScript, Java, Go, C/C++, PHP, Ruby, and more -- **Typical Duration:** 1–5 minutes (varies by codebase size) -- **Output Format:** SARIF 2.1.0 (industry standard) - ---- - -## How Does It Work? - -The workflow orchestrates multiple SAST tools in a containerized environment: - -- **Semgrep:** Pattern-based static analysis for 30+ languages, with rule sets for OWASP Top 10, CWE Top 25, and more. -- **Bandit:** Python-specific security scanner, focused on issues like hardcoded secrets, injection, and unsafe code patterns. - -Each tool runs independently, and their findings are merged and normalized into a single SARIF report. - ---- - -## How to Use the Static Analysis Workflow - -### Basic Usage - -**CLI:** -```bash -fuzzforge runs submit static_analysis_scan /path/to/your/project -``` - -**API:** -```bash -curl -X POST "http://localhost:8000/workflows/static_analysis_scan/submit" \ - -H "Content-Type: application/json" \ - -d '{"target_path": "/path/to/your/project"}' -``` - -### Advanced Configuration - -You can fine-tune the workflow by passing parameters for each tool: - -**CLI:** -```bash -fuzzforge runs submit static_analysis_scan /path/to/project \ - --parameters '{ - "semgrep_config": { - "rules": ["p/security-audit", "owasp-top-ten"], - "severity": ["ERROR", "WARNING"], - "exclude_patterns": ["test/*", "vendor/*", "node_modules/*"] - }, - "bandit_config": { - "confidence": "MEDIUM", - "severity": "MEDIUM", - "exclude_dirs": ["tests", "migrations"] - } - }' -``` - -**API:** -```json -{ - "target_path": "/path/to/project", - "parameters": { - "semgrep_config": { - "rules": ["p/security-audit"], - "languages": ["python", "javascript"], - "severity": ["ERROR", "WARNING"], - "exclude_patterns": ["*.test.js", "test_*.py", "vendor/*"] - }, - "bandit_config": { - "confidence": "MEDIUM", - "severity": "LOW", - "tests": ["B201", "B301"], - "exclude_dirs": ["tests", ".git"] - } - } -} -``` - ---- - -## Configuration Reference - -### Semgrep Parameters - -| Parameter | Type | Default | Description | -|-------------------|-----------|-------------------------------|---------------------------------------------| -| `rules` | array | `"auto"` | Rule sets to use (e.g., `"p/security-audit"`)| -| `languages` | array | `null` | Languages to analyze | -| `severity` | array | `["ERROR", "WARNING", "INFO"]`| Severities to include | -| `exclude_patterns`| array | `[]` | File patterns to exclude | -| `include_patterns`| array | `[]` | File patterns to include | -| `max_target_bytes`| integer | `1000000` | Max file size to analyze (bytes) | -| `timeout` | integer | `300` | Tool timeout (seconds) | - -### Bandit Parameters - -| Parameter | Type | Default | Description | -|-------------------|-----------|-------------------------------|---------------------------------------------| -| `confidence` | string | `"LOW"` | Minimum confidence (`"LOW"`, `"MEDIUM"`, `"HIGH"`) | -| `severity` | string | `"LOW"` | Minimum severity (`"LOW"`, `"MEDIUM"`, `"HIGH"`) | -| `tests` | array | `null` | Specific test IDs to run | -| `exclude_dirs` | array | `["tests", ".git"]` | Directories to exclude | -| `aggregate` | string | `"file"` | Aggregation mode (`"file"`, `"vuln"`) | -| `context_lines` | integer | `3` | Context lines around findings | - ---- - -## What Can It Detect? - -### Vulnerability Categories - -- **OWASP Top 10:** Broken Access Control, Injection, Security Misconfiguration, etc. -- **CWE Top 25:** SQL Injection, XSS, Command Injection, Information Exposure, etc. -- **Language-Specific:** Python (unsafe eval, Django/Flask issues), JavaScript (XSS, prototype pollution), Java (deserialization), Go (race conditions), C/C++ (buffer overflows). - -### Example Detections - -**SQL Injection (Python)** -```python -query = f"SELECT * FROM users WHERE id = {user_id}" # CWE-89 -``` -*Recommendation: Use parameterized queries.* - -**Command Injection (Python)** -```python -os.system(f"cp {filename} backup/") # CWE-78 -``` -*Recommendation: Use subprocess with argument arrays.* - -**XSS (JavaScript)** -```javascript -element.innerHTML = userInput; // CWE-79 -``` -*Recommendation: Use textContent or sanitize input.* - ---- - -## Output Format - -All results are returned in SARIF 2.1.0 format, which is supported by many IDEs and security tools. - -**Summary Example:** -```json -{ - "workflow": "static_analysis_scan", - "status": "completed", - "total_findings": 18, - "severity_counts": { - "critical": 0, - "high": 6, - "medium": 5, - "low": 7 - }, - "tool_counts": { - "semgrep": 12, - "bandit": 6 - } -} -``` - -**Finding Example:** -```json -{ - "ruleId": "bandit.B608", - "level": "error", - "message": { - "text": "Possible SQL injection vector through string-based query construction" - }, - "locations": [ - { - "physicalLocation": { - "artifactLocation": { - "uri": "src/database.py" - }, - "region": { - "startLine": 42, - "startColumn": 15, - "endLine": 42, - "endColumn": 65 - } - } - } - ], - "properties": { - "severity": "high", - "category": "sql_injection", - "cwe": "CWE-89", - "confidence": "high", - "tool": "bandit" - } -} -``` - ---- - -## Performance Tips - -- For large codebases, increase `max_target_bytes` and `timeout` as needed. -- Exclude large generated or dependency directories (`vendor/`, `node_modules/`, `dist/`). -- Run focused scans on changed files for faster CI/CD feedback. - ---- - -## Integration Examples - -### GitHub Actions - -```yaml -- name: Run Static Analysis - run: | - curl -X POST "${{ secrets.FUZZFORGE_URL }}/workflows/static_analysis_scan/submit" \ - -H "Content-Type: application/json" \ - -d '{ - "target_path": "${{ github.workspace }}", - "parameters": { - "semgrep_config": {"severity": ["ERROR", "WARNING"]}, - "bandit_config": {"confidence": "MEDIUM"} - } - }' -``` - -### Pre-commit Hook - -```bash -fuzzforge runs submit static_analysis_scan . --wait --json > /tmp/analysis.json -HIGH_ISSUES=$(jq '.sarif.severity_counts.high // 0' /tmp/analysis.json) -if [ "$HIGH_ISSUES" -gt 0 ]; then - echo "❌ Found $HIGH_ISSUES high-severity security issues. Commit blocked." - exit 1 -fi -``` - ---- - -## Best Practices - -- **Target the right code:** Focus on your main source directories, not dependencies or build artifacts. -- **Start broad, then refine:** Use default rule sets first, then add exclusions or custom rules as needed. -- **Triage findings:** Address high-severity issues first, and document false positives for future runs. -- **Monitor trends:** Track your security posture over time to measure improvement. -- **Optimize for speed:** Use file size limits and timeouts for very large projects. - ---- - -## Troubleshooting - -- **No Python files found:** Bandit will report zero findings if your project isn’t Python, this is normal. -- **High memory usage:** Exclude large files and directories, or increase Docker memory limits. -- **Slow scans:** Use inclusion/exclusion patterns and increase timeouts for big repos. -- **Workflow errors:** See the [Troubleshooting Guide](/how-to/troubleshooting.md) for help with registry, Docker, or workflow issues. diff --git a/docs/docs/tutorial/getting-started.md b/docs/docs/tutorial/getting-started.md index 6c0e9a5..135bb2a 100644 --- a/docs/docs/tutorial/getting-started.md +++ b/docs/docs/tutorial/getting-started.md @@ -124,13 +124,16 @@ fuzzforge workflows list curl http://localhost:8000/workflows | jq . ``` -You should see 6 production workflows: -- `static_analysis_scan` -- `secret_detection_scan` -- `infrastructure_scan` -- `penetration_testing_scan` -- `language_fuzzing` -- `security_assessment` +You should see production-ready workflows: +- `security_assessment` - Comprehensive security analysis (secrets, SQL, dangerous functions) +- `gitleaks_detection` - Pattern-based secret detection +- `trufflehog_detection` - Pattern-based secret detection +- `llm_secret_detection` - AI-powered secret detection (requires API key) + +And development workflows: +- `atheris_fuzzing` - Python fuzzing (early development) +- `cargo_fuzzing` - Rust fuzzing (early development) +- `ossfuzz_campaign` - OSS-Fuzz integration (heavy development) ## Step 6: Run Your First Workflow @@ -183,19 +186,19 @@ curl "http://localhost:8000/runs/{run-id}/findings" The workflow will complete in 30-60 seconds and return results in SARIF format. For the test project, you should see: - **Total findings**: 15-20 security issues -- **Tools used**: OpenGrep (Semgrep) and Bandit +- **Analysis modules**: FileScanner and SecurityAnalyzer (regex-based detection) - **Severity levels**: High, Medium, and Low vulnerabilities - **Categories**: SQL injection, command injection, hardcoded secrets, etc. Example output: ```json { - "workflow": "static_analysis_scan", + "workflow": "security_assessment", "status": "completed", "total_findings": 18, "scan_status": { - "opengrep": "success", - "bandit": "success" + "file_scanner": "success", + "security_analyzer": "success" }, "severity_counts": { "high": 6, diff --git a/docs/index.md b/docs/index.md index a5a3a24..dc0a13e 100644 --- a/docs/index.md +++ b/docs/index.md @@ -8,16 +8,13 @@ Welcome to FuzzForge, a comprehensive security analysis platform built on Tempor Perfect for newcomers who want to learn FuzzForge step by step. - [**Getting Started**](tutorials/getting-started.md) - Complete setup from installation to first workflow -- [**First Workflow**](tutorials/first-workflow.md) - Run your first security workflow -- [**Building Custom Workflows**](tutorials/building-custom-workflow.md) - Create and deploy custom workflows ### πŸ› οΈ **How-To Guides** - *Problem-focused solutions* Step-by-step guides for specific tasks and common problems. -- [**Installation**](how-to/installation.md) - Install FuzzForge with proper Docker setup -- [**Docker Setup**](how-to/docker-setup.md) - Configure Docker with insecure registry (required) -- [**Running Workflows**](how-to/running-workflows.md) - Execute different workflow types -- [**CLI Usage**](how-to/cli-usage.md) - Command-line interface patterns +- [**Docker Setup**](how-to/docker-setup.md) - Docker requirements and worker profiles +- [**Create Workflow**](how-to/create-workflow.md) - Build custom security workflows +- [**Create Module**](how-to/create-module.md) - Develop security analysis modules - [**API Integration**](how-to/api-integration.md) - REST API usage and integration - [**MCP Integration**](how-to/mcp-integration.md) - AI assistant integration setup - [**Troubleshooting**](how-to/troubleshooting.md) - Common issues and solutions @@ -59,16 +56,19 @@ Technical reference materials and specifications. ## 🎯 FuzzForge at a Glance -**6 Production Workflows:** -- Static Analysis (Semgrep, Bandit, CodeQL) -- Secret Detection (TruffleHog, Gitleaks, detect-secrets) -- Infrastructure Scan (Checkov, Hadolint, Kubesec) -- Penetration Testing (Nuclei, Nmap, SQLMap, Nikto) -- Language Fuzzing (AFL++, libFuzzer, Cargo Fuzz) -- Security Assessment (Comprehensive multi-tool analysis) +**Production-Ready Workflows:** +- Security Assessment - Regex-based analysis for secrets, SQL injection, dangerous functions +- Gitleaks Detection - Pattern-based secret scanning +- TruffleHog Detection - Pattern-based secret scanning +- LLM Secret Detection - AI-powered secret detection (requires API key) + +**Development Workflows:** +- Atheris Fuzzing - Python fuzzing (early development) +- Cargo Fuzzing - Rust fuzzing (early development) +- OSS-Fuzz Campaign - OSS-Fuzz integration (heavy development) **Multiple Interfaces:** -- πŸ’» **CLI**: `fuzzforge runs submit static_analysis_scan /path/to/code` +- πŸ’» **CLI**: `fuzzforge workflow run security_assessment /path/to/code` - 🐍 **Python SDK**: Programmatic workflow integration - 🌐 **REST API**: HTTP-based workflow management - πŸ€– **MCP**: AI assistant integration (Claude, ChatGPT) From c652340db63f8133b0b80d88f6aed9d757194b18 Mon Sep 17 00:00:00 2001 From: tduhamel42 Date: Fri, 17 Oct 2025 10:57:48 +0200 Subject: [PATCH 5/5] docs: fix broken link in docker-setup --- docs/docs/how-to/docker-setup.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/docs/how-to/docker-setup.md b/docs/docs/how-to/docker-setup.md index 993aa12..39c0de9 100644 --- a/docs/docs/how-to/docker-setup.md +++ b/docs/docs/how-to/docker-setup.md @@ -234,7 +234,6 @@ docker compose pull ## Next Steps - [Getting Started Guide](../tutorial/getting-started.md): Complete setup walkthrough -- [Running Workflows](running-workflows.md): Execute security workflows - [Troubleshooting](troubleshooting.md): Fix common issues ---