mirror of
https://github.com/FuzzingLabs/fuzzforge_ai.git
synced 2026-08-21 01:17:24 +02:00
feat: Complete migration from Prefect to Temporal
BREAKING CHANGE: Replaces Prefect workflow orchestration with Temporal ## Major Changes - Replace Prefect with Temporal for workflow orchestration - Implement vertical worker architecture (rust, android) - Replace Docker registry with MinIO for unified storage - Refactor activities to be co-located with workflows - Update all API endpoints for Temporal compatibility ## Infrastructure - New: docker-compose.temporal.yaml (Temporal + MinIO + workers) - New: workers/ directory with rust and android vertical workers - New: backend/src/temporal/ (manager, discovery) - New: backend/src/storage/ (S3-cached storage with MinIO) - New: backend/toolbox/common/ (shared storage activities) - Deleted: docker-compose.yaml (old Prefect setup) - Deleted: backend/src/core/prefect_manager.py - Deleted: backend/src/services/prefect_stats_monitor.py - Deleted: Docker registry and insecure-registries requirement ## Workflows - Migrated: security_assessment workflow to Temporal - New: rust_test workflow (example/test workflow) - Deleted: secret_detection_scan (Prefect-based, to be reimplemented) - Activities now co-located with workflows for independent testing ## API Changes - Updated: backend/src/api/workflows.py (Temporal submission) - Updated: backend/src/api/runs.py (Temporal status/results) - Updated: backend/src/main.py (727 lines, TemporalManager integration) - Updated: All 16 MCP tools to use TemporalManager ## Testing - ✅ All services healthy (Temporal, PostgreSQL, MinIO, workers, backend) - ✅ All API endpoints functional - ✅ End-to-end workflow test passed (72 findings from vulnerable_app) - ✅ MinIO storage integration working (target upload/download, results) - ✅ Worker activity discovery working (6 activities registered) - ✅ Tarball extraction working - ✅ SARIF report generation working ## Documentation - ARCHITECTURE.md: Complete Temporal architecture documentation - QUICKSTART_TEMPORAL.md: Getting started guide - MIGRATION_DECISION.md: Why we chose Temporal over Prefect - IMPLEMENTATION_STATUS.md: Migration progress tracking - workers/README.md: Worker development guide ## Dependencies - Added: temporalio>=1.6.0 - Added: boto3>=1.34.0 (MinIO S3 client) - Removed: prefect>=3.4.18
This commit is contained in:
Tanguy Duhamel
@@ -0,0 +1,303 @@
|
||||
# FuzzForge Vertical Workers
|
||||
|
||||
This directory contains vertical-specific worker implementations for the Temporal architecture.
|
||||
|
||||
## Architecture
|
||||
|
||||
Each vertical worker is a long-lived container pre-built with domain-specific security toolchains:
|
||||
|
||||
```
|
||||
workers/
|
||||
├── rust/ # Rust/Native security (AFL++, cargo-fuzz, gdb, valgrind)
|
||||
├── android/ # Android security (apktool, Frida, jadx, MobSF)
|
||||
├── web/ # Web security (OWASP ZAP, semgrep, eslint)
|
||||
├── ios/ # iOS security (class-dump, Clutch, Frida)
|
||||
├── blockchain/ # Smart contract security (mythril, slither, echidna)
|
||||
└── go/ # Go security (go-fuzz, staticcheck, gosec)
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Worker Startup**: Worker discovers workflows from `/app/toolbox/workflows`
|
||||
2. **Filtering**: Only loads workflows where `metadata.yaml` has `vertical: <name>`
|
||||
3. **Dynamic Import**: Dynamically imports workflow Python modules
|
||||
4. **Registration**: Registers discovered workflows with Temporal
|
||||
5. **Processing**: Polls Temporal task queue for work
|
||||
|
||||
## Adding a New Vertical
|
||||
|
||||
### Step 1: Create Worker Directory
|
||||
|
||||
```bash
|
||||
mkdir -p workers/my_vertical
|
||||
cd workers/my_vertical
|
||||
```
|
||||
|
||||
### Step 2: Create Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# workers/my_vertical/Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
# Install your vertical-specific tools
|
||||
RUN apt-get update && apt-get install -y \
|
||||
tool1 \
|
||||
tool2 \
|
||||
tool3 \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Python dependencies
|
||||
COPY requirements.txt /tmp/
|
||||
RUN pip install --no-cache-dir -r /tmp/requirements.txt
|
||||
|
||||
# Copy worker files
|
||||
COPY worker.py /app/worker.py
|
||||
COPY activities.py /app/activities.py
|
||||
|
||||
WORKDIR /app
|
||||
ENV PYTHONPATH="/app:/app/toolbox:${PYTHONPATH}"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
CMD ["python", "worker.py"]
|
||||
```
|
||||
|
||||
### Step 3: Copy Worker Files
|
||||
|
||||
```bash
|
||||
# Copy from rust worker as template
|
||||
cp workers/rust/worker.py workers/my_vertical/
|
||||
cp workers/rust/activities.py workers/my_vertical/
|
||||
cp workers/rust/requirements.txt workers/my_vertical/
|
||||
```
|
||||
|
||||
**Note**: The worker.py and activities.py are generic and work for all verticals. You only need to customize the Dockerfile with your tools.
|
||||
|
||||
### Step 4: Add to docker-compose.temporal.yaml
|
||||
|
||||
```yaml
|
||||
worker-my-vertical:
|
||||
build:
|
||||
context: ./workers/my_vertical
|
||||
dockerfile: Dockerfile
|
||||
container_name: fuzzforge-worker-my-vertical
|
||||
depends_on:
|
||||
temporal:
|
||||
condition: service_healthy
|
||||
minio:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
TEMPORAL_ADDRESS: temporal:7233
|
||||
WORKER_VERTICAL: my_vertical # ← Important: matches metadata.yaml
|
||||
WORKER_TASK_QUEUE: my-vertical-queue
|
||||
MAX_CONCURRENT_ACTIVITIES: 5
|
||||
# MinIO configuration (same for all workers)
|
||||
STORAGE_BACKEND: s3
|
||||
S3_ENDPOINT: http://minio:9000
|
||||
S3_ACCESS_KEY: fuzzforge
|
||||
S3_SECRET_KEY: fuzzforge123
|
||||
S3_BUCKET: targets
|
||||
CACHE_DIR: /cache
|
||||
volumes:
|
||||
- ./backend/toolbox:/app/toolbox:ro
|
||||
- worker_my_vertical_cache:/cache
|
||||
networks:
|
||||
- fuzzforge-network
|
||||
restart: unless-stopped
|
||||
```
|
||||
|
||||
### Step 5: Add Volume
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
worker_my_vertical_cache:
|
||||
name: fuzzforge_worker_my_vertical_cache
|
||||
```
|
||||
|
||||
### Step 6: Create Workflows for Your Vertical
|
||||
|
||||
```bash
|
||||
mkdir -p backend/toolbox/workflows/my_workflow
|
||||
```
|
||||
|
||||
**metadata.yaml:**
|
||||
```yaml
|
||||
name: my_workflow
|
||||
version: 1.0.0
|
||||
vertical: my_vertical # ← Must match WORKER_VERTICAL
|
||||
```
|
||||
|
||||
**workflow.py:**
|
||||
```python
|
||||
from temporalio import workflow
|
||||
from datetime import timedelta
|
||||
|
||||
@workflow.defn
|
||||
class MyWorkflow:
|
||||
@workflow.run
|
||||
async def run(self, target_id: str) -> dict:
|
||||
# Download target
|
||||
target_path = await workflow.execute_activity(
|
||||
"get_target",
|
||||
target_id,
|
||||
start_to_close_timeout=timedelta(minutes=5)
|
||||
)
|
||||
|
||||
# Your analysis logic here
|
||||
results = {"status": "success"}
|
||||
|
||||
# Cleanup
|
||||
await workflow.execute_activity(
|
||||
"cleanup_cache",
|
||||
target_path,
|
||||
start_to_close_timeout=timedelta(minutes=1)
|
||||
)
|
||||
|
||||
return results
|
||||
```
|
||||
|
||||
### Step 7: Test
|
||||
|
||||
```bash
|
||||
# Start services
|
||||
docker-compose -f docker-compose.temporal.yaml up -d
|
||||
|
||||
# Check worker logs
|
||||
docker logs -f fuzzforge-worker-my-vertical
|
||||
|
||||
# You should see:
|
||||
# "Discovered workflow: MyWorkflow from my_workflow (vertical: my_vertical)"
|
||||
```
|
||||
|
||||
## Worker Components
|
||||
|
||||
### worker.py
|
||||
|
||||
Generic worker entrypoint. Handles:
|
||||
- Workflow discovery from mounted `/app/toolbox`
|
||||
- Dynamic import of workflow modules
|
||||
- Connection to Temporal
|
||||
- Task queue polling
|
||||
|
||||
**No customization needed** - works for all verticals.
|
||||
|
||||
### activities.py
|
||||
|
||||
Common activities available to all workflows:
|
||||
|
||||
- `get_target(target_id: str) -> str`: Download target from MinIO
|
||||
- `cleanup_cache(target_path: str) -> None`: Remove cached target
|
||||
- `upload_results(workflow_id, results, format) -> str`: Upload results to MinIO
|
||||
|
||||
**Can be extended** with vertical-specific activities:
|
||||
|
||||
```python
|
||||
# workers/my_vertical/activities.py
|
||||
|
||||
from temporalio import activity
|
||||
|
||||
@activity.defn(name="my_custom_activity")
|
||||
async def my_custom_activity(input_data: str) -> str:
|
||||
# Your vertical-specific logic
|
||||
return "result"
|
||||
|
||||
# Add to worker.py activities list:
|
||||
# activities=[..., my_custom_activity]
|
||||
```
|
||||
|
||||
### Dockerfile
|
||||
|
||||
**Only component that needs customization** for each vertical. Install your tools here.
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
All workers support these environment variables:
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `TEMPORAL_ADDRESS` | `localhost:7233` | Temporal server address |
|
||||
| `TEMPORAL_NAMESPACE` | `default` | Temporal namespace |
|
||||
| `WORKER_VERTICAL` | `rust` | Vertical name (must match metadata.yaml) |
|
||||
| `WORKER_TASK_QUEUE` | `{vertical}-queue` | Task queue name |
|
||||
| `MAX_CONCURRENT_ACTIVITIES` | `5` | Max concurrent activities per worker |
|
||||
| `S3_ENDPOINT` | `http://minio:9000` | MinIO/S3 endpoint |
|
||||
| `S3_ACCESS_KEY` | `fuzzforge` | S3 access key |
|
||||
| `S3_SECRET_KEY` | `fuzzforge123` | S3 secret key |
|
||||
| `S3_BUCKET` | `targets` | Bucket for uploaded targets |
|
||||
| `CACHE_DIR` | `/cache` | Local cache directory |
|
||||
| `CACHE_MAX_SIZE` | `10GB` | Max cache size (not enforced yet) |
|
||||
| `LOG_LEVEL` | `INFO` | Logging level |
|
||||
|
||||
## Scaling
|
||||
|
||||
### Vertical Scaling (More Work Per Worker)
|
||||
|
||||
Increase concurrent activities:
|
||||
|
||||
```yaml
|
||||
environment:
|
||||
MAX_CONCURRENT_ACTIVITIES: 10 # Handle 10 tasks at once
|
||||
```
|
||||
|
||||
### Horizontal Scaling (More Workers)
|
||||
|
||||
```bash
|
||||
# Scale to 3 workers for rust vertical
|
||||
docker-compose -f docker-compose.temporal.yaml up -d --scale worker-rust=3
|
||||
|
||||
# Each worker polls the same task queue
|
||||
# Temporal automatically load balances
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Worker Not Discovering Workflows
|
||||
|
||||
Check:
|
||||
1. Volume mount is correct: `./backend/toolbox:/app/toolbox:ro`
|
||||
2. Workflow has `metadata.yaml` with correct `vertical:` field
|
||||
3. Workflow has `workflow.py` with `@workflow.defn` decorated class
|
||||
4. Worker logs show discovery attempt
|
||||
|
||||
### Cannot Connect to Temporal
|
||||
|
||||
Check:
|
||||
1. Temporal container is healthy: `docker ps`
|
||||
2. Network connectivity: `docker exec worker-rust ping temporal`
|
||||
3. `TEMPORAL_ADDRESS` environment variable is correct
|
||||
|
||||
### Cannot Download from MinIO
|
||||
|
||||
Check:
|
||||
1. MinIO is healthy: `docker ps`
|
||||
2. Buckets exist: `docker exec fuzzforge-minio mc ls fuzzforge/targets`
|
||||
3. S3 credentials are correct
|
||||
4. Target was uploaded: Check MinIO console at http://localhost:9001
|
||||
|
||||
### Activity Timeouts
|
||||
|
||||
Increase timeout in workflow:
|
||||
|
||||
```python
|
||||
await workflow.execute_activity(
|
||||
"my_activity",
|
||||
args,
|
||||
start_to_close_timeout=timedelta(hours=2) # Increase from default
|
||||
)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Keep Dockerfiles lean**: Only install necessary tools
|
||||
2. **Use multi-stage builds**: Reduce final image size
|
||||
3. **Pin tool versions**: Ensure reproducibility
|
||||
4. **Log liberally**: Helps debugging workflow issues
|
||||
5. **Handle errors gracefully**: Don't fail workflow for non-critical issues
|
||||
6. **Test locally first**: Use docker-compose before deploying
|
||||
|
||||
## Examples
|
||||
|
||||
See existing verticals for examples:
|
||||
- `workers/rust/` - Complete working example
|
||||
- `backend/toolbox/workflows/rust_test/` - Simple test workflow
|
||||
@@ -0,0 +1,94 @@
|
||||
# FuzzForge Vertical Worker: Android Security
|
||||
#
|
||||
# Pre-installed tools for Android security analysis:
|
||||
# - Android SDK (adb, aapt)
|
||||
# - apktool (APK decompilation)
|
||||
# - jadx (Dex to Java decompiler)
|
||||
# - Frida (dynamic instrumentation)
|
||||
# - androguard (Python APK analysis)
|
||||
# - MobSF dependencies
|
||||
|
||||
FROM python:3.11-slim-bookworm
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
# Build essentials
|
||||
build-essential \
|
||||
git \
|
||||
curl \
|
||||
wget \
|
||||
unzip \
|
||||
# Java (required for Android tools)
|
||||
openjdk-17-jdk \
|
||||
# Android tools dependencies
|
||||
lib32stdc++6 \
|
||||
lib32z1 \
|
||||
# Frida dependencies
|
||||
libc6-dev \
|
||||
# XML/Binary analysis
|
||||
libxml2-dev \
|
||||
libxslt-dev \
|
||||
# Network tools
|
||||
netcat-openbsd \
|
||||
tcpdump \
|
||||
# Cleanup
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install Android SDK Command Line Tools
|
||||
ENV ANDROID_HOME=/opt/android-sdk
|
||||
ENV PATH="${ANDROID_HOME}/cmdline-tools/latest/bin:${ANDROID_HOME}/platform-tools:${PATH}"
|
||||
|
||||
RUN mkdir -p ${ANDROID_HOME}/cmdline-tools && \
|
||||
cd ${ANDROID_HOME}/cmdline-tools && \
|
||||
wget -q https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip && \
|
||||
unzip -q commandlinetools-linux-9477386_latest.zip && \
|
||||
mv cmdline-tools latest && \
|
||||
rm commandlinetools-linux-9477386_latest.zip && \
|
||||
# Accept licenses
|
||||
yes | ${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager --licenses && \
|
||||
# Install platform tools (adb, fastboot)
|
||||
${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager "platform-tools" "build-tools;33.0.0"
|
||||
|
||||
# Install apktool
|
||||
RUN wget -q https://raw.githubusercontent.com/iBotPeaches/Apktool/master/scripts/linux/apktool -O /usr/local/bin/apktool && \
|
||||
wget -q https://bitbucket.org/iBotPeaches/apktool/downloads/apktool_2.9.3.jar -O /usr/local/bin/apktool.jar && \
|
||||
chmod +x /usr/local/bin/apktool
|
||||
|
||||
# Install jadx (Dex to Java decompiler)
|
||||
RUN wget -q https://github.com/skylot/jadx/releases/download/v1.4.7/jadx-1.4.7.zip -O /tmp/jadx.zip && \
|
||||
unzip -q /tmp/jadx.zip -d /opt/jadx && \
|
||||
ln -s /opt/jadx/bin/jadx /usr/local/bin/jadx && \
|
||||
ln -s /opt/jadx/bin/jadx-gui /usr/local/bin/jadx-gui && \
|
||||
rm /tmp/jadx.zip
|
||||
|
||||
# Install Python dependencies for Android security tools
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt && \
|
||||
rm /tmp/requirements.txt
|
||||
|
||||
# Install androguard (Python APK analysis framework)
|
||||
RUN pip3 install --no-cache-dir androguard pyaxmlparser
|
||||
|
||||
# Install Frida
|
||||
RUN pip3 install --no-cache-dir frida-tools frida
|
||||
|
||||
# Create cache directory
|
||||
RUN mkdir -p /cache && chmod 755 /cache
|
||||
|
||||
# Copy worker entrypoint (generic, works for all verticals)
|
||||
COPY worker.py /app/worker.py
|
||||
|
||||
# Add toolbox to Python path (mounted at runtime)
|
||||
ENV PYTHONPATH="/app:/app/toolbox:${PYTHONPATH}"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
|
||||
|
||||
# Healthcheck
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD python3 -c "import sys; sys.exit(0)"
|
||||
|
||||
# Run worker
|
||||
CMD ["python3", "/app/worker.py"]
|
||||
@@ -0,0 +1,19 @@
|
||||
# Temporal Python SDK
|
||||
temporalio>=1.5.0
|
||||
|
||||
# S3/MinIO client
|
||||
boto3>=1.34.0
|
||||
botocore>=1.34.0
|
||||
|
||||
# Data validation
|
||||
pydantic>=2.5.0
|
||||
|
||||
# YAML parsing
|
||||
PyYAML>=6.0.1
|
||||
|
||||
# Utilities
|
||||
python-dotenv>=1.0.0
|
||||
aiofiles>=23.2.1
|
||||
|
||||
# Logging
|
||||
structlog>=24.1.0
|
||||
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
FuzzForge Vertical Worker: Rust/Native Security
|
||||
|
||||
This worker:
|
||||
1. Discovers workflows for the 'rust' vertical from mounted toolbox
|
||||
2. Dynamically imports and registers workflow classes
|
||||
3. Connects to Temporal and processes tasks
|
||||
4. Handles activities for target download/upload from MinIO
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import importlib
|
||||
import inspect
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List, Any
|
||||
|
||||
import yaml
|
||||
from temporalio.client import Client
|
||||
from temporalio.worker import Worker
|
||||
|
||||
# Add toolbox to path for workflow and activity imports
|
||||
sys.path.insert(0, '/app/toolbox')
|
||||
|
||||
# Import common storage activities
|
||||
from toolbox.common.storage_activities import (
|
||||
get_target_activity,
|
||||
cleanup_cache_activity,
|
||||
upload_results_activity
|
||||
)
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=os.getenv('LOG_LEVEL', 'INFO'),
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def discover_workflows(vertical: str) -> List[Any]:
|
||||
"""
|
||||
Discover workflows for this vertical from mounted toolbox.
|
||||
|
||||
Args:
|
||||
vertical: The vertical name (e.g., 'rust', 'android', 'web')
|
||||
|
||||
Returns:
|
||||
List of workflow classes decorated with @workflow.defn
|
||||
"""
|
||||
workflows = []
|
||||
toolbox_path = Path("/app/toolbox/workflows")
|
||||
|
||||
if not toolbox_path.exists():
|
||||
logger.warning(f"Toolbox path does not exist: {toolbox_path}")
|
||||
return workflows
|
||||
|
||||
logger.info(f"Scanning for workflows in: {toolbox_path}")
|
||||
|
||||
for workflow_dir in toolbox_path.iterdir():
|
||||
if not workflow_dir.is_dir():
|
||||
continue
|
||||
|
||||
# Skip special directories
|
||||
if workflow_dir.name.startswith('.') or workflow_dir.name == '__pycache__':
|
||||
continue
|
||||
|
||||
metadata_file = workflow_dir / "metadata.yaml"
|
||||
if not metadata_file.exists():
|
||||
logger.debug(f"No metadata.yaml in {workflow_dir.name}, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Parse metadata
|
||||
with open(metadata_file) as f:
|
||||
metadata = yaml.safe_load(f)
|
||||
|
||||
# Check if workflow is for this vertical
|
||||
workflow_vertical = metadata.get("vertical")
|
||||
if workflow_vertical != vertical:
|
||||
logger.debug(
|
||||
f"Workflow {workflow_dir.name} is for vertical '{workflow_vertical}', "
|
||||
f"not '{vertical}', skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
# Check if workflow.py exists
|
||||
workflow_file = workflow_dir / "workflow.py"
|
||||
if not workflow_file.exists():
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has metadata but no workflow.py, skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
# Dynamically import workflow module
|
||||
module_name = f"toolbox.workflows.{workflow_dir.name}.workflow"
|
||||
logger.info(f"Importing workflow module: {module_name}")
|
||||
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import workflow module {module_name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
# Find @workflow.defn decorated classes
|
||||
found_workflows = False
|
||||
for name, obj in inspect.getmembers(module, inspect.isclass):
|
||||
# Check if class has Temporal workflow definition
|
||||
if hasattr(obj, '__temporal_workflow_definition'):
|
||||
workflows.append(obj)
|
||||
found_workflows = True
|
||||
logger.info(
|
||||
f"✓ Discovered workflow: {name} from {workflow_dir.name} "
|
||||
f"(vertical: {vertical})"
|
||||
)
|
||||
|
||||
if not found_workflows:
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has no @workflow.defn decorated classes"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error processing workflow {workflow_dir.name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
logger.info(f"Discovered {len(workflows)} workflows for vertical '{vertical}'")
|
||||
return workflows
|
||||
|
||||
|
||||
async def discover_activities(workflows_dir: Path) -> List[Any]:
|
||||
"""
|
||||
Discover activities from workflow directories.
|
||||
|
||||
Looks for activities.py files alongside workflow.py in each workflow directory.
|
||||
|
||||
Args:
|
||||
workflows_dir: Path to workflows directory
|
||||
|
||||
Returns:
|
||||
List of activity functions decorated with @activity.defn
|
||||
"""
|
||||
activities = []
|
||||
|
||||
if not workflows_dir.exists():
|
||||
logger.warning(f"Workflows directory does not exist: {workflows_dir}")
|
||||
return activities
|
||||
|
||||
logger.info(f"Scanning for workflow activities in: {workflows_dir}")
|
||||
|
||||
for workflow_dir in workflows_dir.iterdir():
|
||||
if not workflow_dir.is_dir():
|
||||
continue
|
||||
|
||||
# Skip special directories
|
||||
if workflow_dir.name.startswith('.') or workflow_dir.name == '__pycache__':
|
||||
continue
|
||||
|
||||
# Check if activities.py exists
|
||||
activities_file = workflow_dir / "activities.py"
|
||||
if not activities_file.exists():
|
||||
logger.debug(f"No activities.py in {workflow_dir.name}, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Dynamically import activities module
|
||||
module_name = f"toolbox.workflows.{workflow_dir.name}.activities"
|
||||
logger.info(f"Importing activities module: {module_name}")
|
||||
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import activities module {module_name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
# Find @activity.defn decorated functions
|
||||
found_activities = False
|
||||
for name, obj in inspect.getmembers(module, inspect.isfunction):
|
||||
# Check if function has Temporal activity definition
|
||||
if hasattr(obj, '__temporal_activity_definition'):
|
||||
activities.append(obj)
|
||||
found_activities = True
|
||||
logger.info(
|
||||
f"✓ Discovered activity: {name} from {workflow_dir.name}"
|
||||
)
|
||||
|
||||
if not found_activities:
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has activities.py but no @activity.defn decorated functions"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error processing activities from {workflow_dir.name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
logger.info(f"Discovered {len(activities)} workflow-specific activities")
|
||||
return activities
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main worker entry point"""
|
||||
# Get configuration from environment
|
||||
vertical = os.getenv("WORKER_VERTICAL", "rust")
|
||||
temporal_address = os.getenv("TEMPORAL_ADDRESS", "localhost:7233")
|
||||
temporal_namespace = os.getenv("TEMPORAL_NAMESPACE", "default")
|
||||
task_queue = os.getenv("WORKER_TASK_QUEUE", f"{vertical}-queue")
|
||||
max_concurrent_activities = int(os.getenv("MAX_CONCURRENT_ACTIVITIES", "5"))
|
||||
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"FuzzForge Vertical Worker: {vertical}")
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"Temporal Address: {temporal_address}")
|
||||
logger.info(f"Temporal Namespace: {temporal_namespace}")
|
||||
logger.info(f"Task Queue: {task_queue}")
|
||||
logger.info(f"Max Concurrent Activities: {max_concurrent_activities}")
|
||||
logger.info("=" * 60)
|
||||
|
||||
# Discover workflows for this vertical
|
||||
logger.info(f"Discovering workflows for vertical: {vertical}")
|
||||
workflows = await discover_workflows(vertical)
|
||||
|
||||
if not workflows:
|
||||
logger.error(f"No workflows found for vertical: {vertical}")
|
||||
logger.error("Worker cannot start without workflows. Exiting...")
|
||||
sys.exit(1)
|
||||
|
||||
# Discover activities from workflow directories
|
||||
logger.info("Discovering workflow-specific activities...")
|
||||
workflows_dir = Path("/app/toolbox/workflows")
|
||||
workflow_activities = await discover_activities(workflows_dir)
|
||||
|
||||
# Combine common storage activities with workflow-specific activities
|
||||
activities = [
|
||||
get_target_activity,
|
||||
cleanup_cache_activity,
|
||||
upload_results_activity
|
||||
] + workflow_activities
|
||||
|
||||
logger.info(
|
||||
f"Total activities registered: {len(activities)} "
|
||||
f"(3 common + {len(workflow_activities)} workflow-specific)"
|
||||
)
|
||||
|
||||
# Connect to Temporal
|
||||
logger.info(f"Connecting to Temporal at {temporal_address}...")
|
||||
try:
|
||||
client = await Client.connect(
|
||||
temporal_address,
|
||||
namespace=temporal_namespace
|
||||
)
|
||||
logger.info("✓ Connected to Temporal successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to connect to Temporal: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
# Create worker with discovered workflows and activities
|
||||
logger.info(f"Creating worker on task queue: {task_queue}")
|
||||
|
||||
try:
|
||||
worker = Worker(
|
||||
client,
|
||||
task_queue=task_queue,
|
||||
workflows=workflows,
|
||||
activities=activities,
|
||||
max_concurrent_activities=max_concurrent_activities
|
||||
)
|
||||
logger.info("✓ Worker created successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create worker: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
# Start worker
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"🚀 Worker started for vertical '{vertical}'")
|
||||
logger.info(f"📦 Registered {len(workflows)} workflows")
|
||||
logger.info(f"⚙️ Registered {len(activities)} activities")
|
||||
logger.info(f"📨 Listening on task queue: {task_queue}")
|
||||
logger.info("=" * 60)
|
||||
logger.info("Worker is ready to process tasks...")
|
||||
|
||||
try:
|
||||
await worker.run()
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Shutting down worker (keyboard interrupt)...")
|
||||
except Exception as e:
|
||||
logger.error(f"Worker error: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Worker stopped")
|
||||
except Exception as e:
|
||||
logger.error(f"Fatal error: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,85 @@
|
||||
# FuzzForge Vertical Worker: Rust/Native Security
|
||||
#
|
||||
# Pre-installed tools for Rust and native binary security analysis:
|
||||
# - Rust toolchain (rustc, cargo)
|
||||
# - AFL++ (fuzzing)
|
||||
# - cargo-fuzz (Rust fuzzing)
|
||||
# - gdb (debugging)
|
||||
# - valgrind (memory analysis)
|
||||
# - AddressSanitizer/MemorySanitizer support
|
||||
# - Common reverse engineering tools
|
||||
|
||||
FROM rust:1.83-slim-bookworm
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
# Build essentials
|
||||
build-essential \
|
||||
cmake \
|
||||
git \
|
||||
curl \
|
||||
wget \
|
||||
pkg-config \
|
||||
libssl-dev \
|
||||
# AFL++ dependencies
|
||||
clang \
|
||||
llvm \
|
||||
# Debugging and analysis tools
|
||||
gdb \
|
||||
valgrind \
|
||||
strace \
|
||||
# Binary analysis (binutils includes objdump, readelf, etc.)
|
||||
binutils \
|
||||
# Network tools
|
||||
netcat-openbsd \
|
||||
tcpdump \
|
||||
# Python for Temporal worker
|
||||
python3 \
|
||||
python3-pip \
|
||||
python3-venv \
|
||||
# Cleanup
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install AFL++
|
||||
RUN git clone https://github.com/AFLplusplus/AFLplusplus /tmp/aflplusplus && \
|
||||
cd /tmp/aflplusplus && \
|
||||
make all && \
|
||||
make install && \
|
||||
cd / && \
|
||||
rm -rf /tmp/aflplusplus
|
||||
|
||||
# Install Rust toolchain components
|
||||
RUN rustup component add rustfmt clippy && \
|
||||
rustup target add x86_64-unknown-linux-musl
|
||||
|
||||
# Install cargo-fuzz and other Rust security tools
|
||||
RUN cargo install --locked \
|
||||
cargo-fuzz \
|
||||
cargo-audit \
|
||||
cargo-outdated \
|
||||
cargo-tree
|
||||
|
||||
# Install Python dependencies for Temporal worker
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
RUN pip3 install --break-system-packages --no-cache-dir -r /tmp/requirements.txt && \
|
||||
rm /tmp/requirements.txt
|
||||
|
||||
# Create cache directory for downloaded targets
|
||||
RUN mkdir -p /cache && chmod 755 /cache
|
||||
|
||||
# Copy worker entrypoint
|
||||
COPY worker.py /app/worker.py
|
||||
|
||||
# Add toolbox to Python path (mounted at runtime)
|
||||
ENV PYTHONPATH="/app:/app/toolbox:${PYTHONPATH}"
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
|
||||
# Healthcheck
|
||||
HEALTHCHECK --interval=30s --timeout=10s --start-period=40s --retries=3 \
|
||||
CMD python3 -c "import sys; sys.exit(0)"
|
||||
|
||||
# Run worker
|
||||
CMD ["python3", "/app/worker.py"]
|
||||
@@ -0,0 +1,19 @@
|
||||
# Temporal Python SDK
|
||||
temporalio>=1.5.0
|
||||
|
||||
# S3/MinIO client
|
||||
boto3>=1.34.0
|
||||
botocore>=1.34.0
|
||||
|
||||
# Data validation
|
||||
pydantic>=2.5.0
|
||||
|
||||
# YAML parsing
|
||||
PyYAML>=6.0.1
|
||||
|
||||
# Utilities
|
||||
python-dotenv>=1.0.0
|
||||
aiofiles>=23.2.1
|
||||
|
||||
# Logging
|
||||
structlog>=24.1.0
|
||||
@@ -0,0 +1,309 @@
|
||||
"""
|
||||
FuzzForge Vertical Worker: Rust/Native Security
|
||||
|
||||
This worker:
|
||||
1. Discovers workflows for the 'rust' vertical from mounted toolbox
|
||||
2. Dynamically imports and registers workflow classes
|
||||
3. Connects to Temporal and processes tasks
|
||||
4. Handles activities for target download/upload from MinIO
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import importlib
|
||||
import inspect
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import List, Any
|
||||
|
||||
import yaml
|
||||
from temporalio.client import Client
|
||||
from temporalio.worker import Worker
|
||||
|
||||
# Add toolbox to path for workflow and activity imports
|
||||
sys.path.insert(0, '/app/toolbox')
|
||||
|
||||
# Import common storage activities
|
||||
from toolbox.common.storage_activities import (
|
||||
get_target_activity,
|
||||
cleanup_cache_activity,
|
||||
upload_results_activity
|
||||
)
|
||||
|
||||
# Configure logging
|
||||
logging.basicConfig(
|
||||
level=os.getenv('LOG_LEVEL', 'INFO'),
|
||||
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
|
||||
)
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
async def discover_workflows(vertical: str) -> List[Any]:
|
||||
"""
|
||||
Discover workflows for this vertical from mounted toolbox.
|
||||
|
||||
Args:
|
||||
vertical: The vertical name (e.g., 'rust', 'android', 'web')
|
||||
|
||||
Returns:
|
||||
List of workflow classes decorated with @workflow.defn
|
||||
"""
|
||||
workflows = []
|
||||
toolbox_path = Path("/app/toolbox/workflows")
|
||||
|
||||
if not toolbox_path.exists():
|
||||
logger.warning(f"Toolbox path does not exist: {toolbox_path}")
|
||||
return workflows
|
||||
|
||||
logger.info(f"Scanning for workflows in: {toolbox_path}")
|
||||
|
||||
for workflow_dir in toolbox_path.iterdir():
|
||||
if not workflow_dir.is_dir():
|
||||
continue
|
||||
|
||||
# Skip special directories
|
||||
if workflow_dir.name.startswith('.') or workflow_dir.name == '__pycache__':
|
||||
continue
|
||||
|
||||
metadata_file = workflow_dir / "metadata.yaml"
|
||||
if not metadata_file.exists():
|
||||
logger.debug(f"No metadata.yaml in {workflow_dir.name}, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Parse metadata
|
||||
with open(metadata_file) as f:
|
||||
metadata = yaml.safe_load(f)
|
||||
|
||||
# Check if workflow is for this vertical
|
||||
workflow_vertical = metadata.get("vertical")
|
||||
if workflow_vertical != vertical:
|
||||
logger.debug(
|
||||
f"Workflow {workflow_dir.name} is for vertical '{workflow_vertical}', "
|
||||
f"not '{vertical}', skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
# Check if workflow.py exists
|
||||
workflow_file = workflow_dir / "workflow.py"
|
||||
if not workflow_file.exists():
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has metadata but no workflow.py, skipping"
|
||||
)
|
||||
continue
|
||||
|
||||
# Dynamically import workflow module
|
||||
module_name = f"toolbox.workflows.{workflow_dir.name}.workflow"
|
||||
logger.info(f"Importing workflow module: {module_name}")
|
||||
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import workflow module {module_name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
# Find @workflow.defn decorated classes
|
||||
found_workflows = False
|
||||
for name, obj in inspect.getmembers(module, inspect.isclass):
|
||||
# Check if class has Temporal workflow definition
|
||||
if hasattr(obj, '__temporal_workflow_definition'):
|
||||
workflows.append(obj)
|
||||
found_workflows = True
|
||||
logger.info(
|
||||
f"✓ Discovered workflow: {name} from {workflow_dir.name} "
|
||||
f"(vertical: {vertical})"
|
||||
)
|
||||
|
||||
if not found_workflows:
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has no @workflow.defn decorated classes"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error processing workflow {workflow_dir.name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
logger.info(f"Discovered {len(workflows)} workflows for vertical '{vertical}'")
|
||||
return workflows
|
||||
|
||||
|
||||
async def discover_activities(workflows_dir: Path) -> List[Any]:
|
||||
"""
|
||||
Discover activities from workflow directories.
|
||||
|
||||
Looks for activities.py files alongside workflow.py in each workflow directory.
|
||||
|
||||
Args:
|
||||
workflows_dir: Path to workflows directory
|
||||
|
||||
Returns:
|
||||
List of activity functions decorated with @activity.defn
|
||||
"""
|
||||
activities = []
|
||||
|
||||
if not workflows_dir.exists():
|
||||
logger.warning(f"Workflows directory does not exist: {workflows_dir}")
|
||||
return activities
|
||||
|
||||
logger.info(f"Scanning for workflow activities in: {workflows_dir}")
|
||||
|
||||
for workflow_dir in workflows_dir.iterdir():
|
||||
if not workflow_dir.is_dir():
|
||||
continue
|
||||
|
||||
# Skip special directories
|
||||
if workflow_dir.name.startswith('.') or workflow_dir.name == '__pycache__':
|
||||
continue
|
||||
|
||||
# Check if activities.py exists
|
||||
activities_file = workflow_dir / "activities.py"
|
||||
if not activities_file.exists():
|
||||
logger.debug(f"No activities.py in {workflow_dir.name}, skipping")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Dynamically import activities module
|
||||
module_name = f"toolbox.workflows.{workflow_dir.name}.activities"
|
||||
logger.info(f"Importing activities module: {module_name}")
|
||||
|
||||
try:
|
||||
module = importlib.import_module(module_name)
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import activities module {module_name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
# Find @activity.defn decorated functions
|
||||
found_activities = False
|
||||
for name, obj in inspect.getmembers(module, inspect.isfunction):
|
||||
# Check if function has Temporal activity definition
|
||||
if hasattr(obj, '__temporal_activity_definition'):
|
||||
activities.append(obj)
|
||||
found_activities = True
|
||||
logger.info(
|
||||
f"✓ Discovered activity: {name} from {workflow_dir.name}"
|
||||
)
|
||||
|
||||
if not found_activities:
|
||||
logger.warning(
|
||||
f"Workflow {workflow_dir.name} has activities.py but no @activity.defn decorated functions"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error processing activities from {workflow_dir.name}: {e}",
|
||||
exc_info=True
|
||||
)
|
||||
continue
|
||||
|
||||
logger.info(f"Discovered {len(activities)} workflow-specific activities")
|
||||
return activities
|
||||
|
||||
|
||||
async def main():
|
||||
"""Main worker entry point"""
|
||||
# Get configuration from environment
|
||||
vertical = os.getenv("WORKER_VERTICAL", "rust")
|
||||
temporal_address = os.getenv("TEMPORAL_ADDRESS", "localhost:7233")
|
||||
temporal_namespace = os.getenv("TEMPORAL_NAMESPACE", "default")
|
||||
task_queue = os.getenv("WORKER_TASK_QUEUE", f"{vertical}-queue")
|
||||
max_concurrent_activities = int(os.getenv("MAX_CONCURRENT_ACTIVITIES", "5"))
|
||||
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"FuzzForge Vertical Worker: {vertical}")
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"Temporal Address: {temporal_address}")
|
||||
logger.info(f"Temporal Namespace: {temporal_namespace}")
|
||||
logger.info(f"Task Queue: {task_queue}")
|
||||
logger.info(f"Max Concurrent Activities: {max_concurrent_activities}")
|
||||
logger.info("=" * 60)
|
||||
|
||||
# Discover workflows for this vertical
|
||||
logger.info(f"Discovering workflows for vertical: {vertical}")
|
||||
workflows = await discover_workflows(vertical)
|
||||
|
||||
if not workflows:
|
||||
logger.error(f"No workflows found for vertical: {vertical}")
|
||||
logger.error("Worker cannot start without workflows. Exiting...")
|
||||
sys.exit(1)
|
||||
|
||||
# Discover activities from workflow directories
|
||||
logger.info("Discovering workflow-specific activities...")
|
||||
workflows_dir = Path("/app/toolbox/workflows")
|
||||
workflow_activities = await discover_activities(workflows_dir)
|
||||
|
||||
# Combine common storage activities with workflow-specific activities
|
||||
activities = [
|
||||
get_target_activity,
|
||||
cleanup_cache_activity,
|
||||
upload_results_activity
|
||||
] + workflow_activities
|
||||
|
||||
logger.info(
|
||||
f"Total activities registered: {len(activities)} "
|
||||
f"(3 common + {len(workflow_activities)} workflow-specific)"
|
||||
)
|
||||
|
||||
# Connect to Temporal
|
||||
logger.info(f"Connecting to Temporal at {temporal_address}...")
|
||||
try:
|
||||
client = await Client.connect(
|
||||
temporal_address,
|
||||
namespace=temporal_namespace
|
||||
)
|
||||
logger.info("✓ Connected to Temporal successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to connect to Temporal: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
# Create worker with discovered workflows and activities
|
||||
logger.info(f"Creating worker on task queue: {task_queue}")
|
||||
|
||||
try:
|
||||
worker = Worker(
|
||||
client,
|
||||
task_queue=task_queue,
|
||||
workflows=workflows,
|
||||
activities=activities,
|
||||
max_concurrent_activities=max_concurrent_activities
|
||||
)
|
||||
logger.info("✓ Worker created successfully")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create worker: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
|
||||
# Start worker
|
||||
logger.info("=" * 60)
|
||||
logger.info(f"🚀 Worker started for vertical '{vertical}'")
|
||||
logger.info(f"📦 Registered {len(workflows)} workflows")
|
||||
logger.info(f"⚙️ Registered {len(activities)} activities")
|
||||
logger.info(f"📨 Listening on task queue: {task_queue}")
|
||||
logger.info("=" * 60)
|
||||
logger.info("Worker is ready to process tasks...")
|
||||
|
||||
try:
|
||||
await worker.run()
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Shutting down worker (keyboard interrupt)...")
|
||||
except Exception as e:
|
||||
logger.error(f"Worker error: {e}", exc_info=True)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
asyncio.run(main())
|
||||
except KeyboardInterrupt:
|
||||
logger.info("Worker stopped")
|
||||
except Exception as e:
|
||||
logger.error(f"Fatal error: {e}", exc_info=True)
|
||||
sys.exit(1)
|
||||
Reference in New Issue
Block a user