CalvinBackup/fuzzforge_ai
1
0
Fork 0
mirror of https://github.com/FuzzingLabs/fuzzforge_ai.git synced 2026-02-12 23:52:47 +00:00
Code Issues Packages Projects Releases Wiki Activity

feat: add platform-aware worker architecture with ARM64 support

Browse Source 
Implement platform-specific Dockerfile selection and graceful tool degradation to support both x86_64 and ARM64 (Apple Silicon) platforms.

**Backend Changes:**
- Add system info API endpoint (/system/info) exposing host filesystem paths
- Add FUZZFORGE_HOST_ROOT environment variable to backend service
- Add graceful degradation in MobSF activity for ARM64 platforms

**CLI Changes:**
- Implement multi-strategy path resolution (backend API, .fuzzforge marker, env var)
- Add platform detection (linux/amd64 vs linux/arm64)
- Add worker metadata.yaml reading for platform capabilities
- Auto-select appropriate Dockerfile based on detected platform
- Pass platform-specific env vars to docker-compose

**Worker Changes:**
- Create workers/android/metadata.yaml defining platform capabilities
- Rename Dockerfile -> Dockerfile.amd64 (full toolchain with MobSF)
- Create Dockerfile.arm64 (excludes MobSF due to Rosetta 2 incompatibility)
- Update docker-compose.yml to use ${ANDROID_DOCKERFILE} variable

**Workflow Changes:**
- Handle MobSF "skipped" status gracefully in workflow
- Log clear warnings when tools are unavailable on platform

**Key Features:**
- Automatic platform detection and Dockerfile selection
- Graceful degradation when tools unavailable (MobSF on ARM64)
- Works from any directory (backend API provides paths)
- Manual override via environment variables
- Clear user feedback about platform and selected Dockerfile

**Benefits:**
- Android workflow now works on Apple Silicon Macs
- No code changes needed for other workflows
- Convention established for future platform-specific workers

Closes: MobSF Rosetta 2 incompatibility issue
Implements: Platform-aware worker architecture (Option B)
This commit is contained in:
tduhamel42
2025-10-23 16:43:17 +02:00
parent 1d3e033bcc
commit 0801ca3d78
9 changed files with 424 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
backend/src/api/system.py Normal file
View File

@@ -0,0 +1,47 @@
# Copyright (c) 2025 FuzzingLabs
#
# Licensed under the Business Source License 1.1 (BSL). See the LICENSE file
# at the root of this repository for details.
#
# After the Change Date (four years from publication), this version of the
# Licensed Work will be made available under the Apache License, Version 2.0.
# See the LICENSE-APACHE file or http://www.apache.org/licenses/LICENSE-2.0
#
# Additional attribution and requirements are provided in the NOTICE file.
"""
System information endpoints for FuzzForge API.
Provides system configuration and filesystem paths to CLI for worker management.
"""
import os
from typing import Dict
from fastapi import APIRouter
router = APIRouter(prefix="/system", tags=["system"])
@router.get("/info")
async def get_system_info() -> Dict[str, str]:
"""
Get system information including host filesystem paths.
This endpoint exposes paths needed by the CLI to manage workers via docker-compose.
The FUZZFORGE_HOST_ROOT environment variable is set by docker-compose and points
to the FuzzForge installation directory on the host machine.
Returns:
Dictionary containing:
- host_root: Absolute path to FuzzForge root on host
- docker_compose_path: Path to docker-compose.yml on host
- workers_dir: Path to workers directory on host
"""
host_root = os.getenv("FUZZFORGE_HOST_ROOT", "")
return {
"host_root": host_root,
"docker_compose_path": f"{host_root}/docker-compose.yml" if host_root else "",
"workers_dir": f"{host_root}/workers" if host_root else "",
}

3
backend/src/main.py
View File

@@ -24,7 +24,7 @@ from fastmcp.server.http import create_sse_app
from src.temporal.manager import TemporalManager
from src.core.setup import setup_result_storage, validate_infrastructure
from src.api import workflows, runs, fuzzing
from src.api import workflows, runs, fuzzing, system
from fastmcp import FastMCP
@@ -76,6 +76,7 @@ app = FastAPI(
app.include_router(workflows.router)
app.include_router(runs.router)
app.include_router(fuzzing.router)
app.include_router(system.router)
def get_temporal_status() -> Dict[str, Any]:

15
backend/toolbox/workflows/android_static_analysis/activities.py
View File

@@ -112,10 +112,23 @@ async def scan_with_mobsf_activity(workspace_path: str, config: dict) -> dict:
config: MobSFScanner configuration
Returns:
Scan results dictionary
Scan results dictionary (or skipped status if MobSF unavailable)
"""
logger.info(f"Activity: scan_with_mobsf (workspace={workspace_path})")
# Check if MobSF is installed (graceful degradation for ARM64 platform)
mobsf_path = Path("/app/mobsf")
if not mobsf_path.exists():
logger.warning("MobSF not installed on this platform (ARM64/Rosetta limitation)")
return {
"status": "skipped",
"findings": [],
"summary": {
"total_findings": 0,
"skip_reason": "MobSF unavailable on ARM64 platform (Rosetta 2 incompatibility)"
}
}
try:
from modules.android import MobSFScanner

13
backend/toolbox/workflows/android_static_analysis/workflow.py
View File

@@ -196,9 +196,16 @@ class AndroidStaticAnalysisWorkflow:
maximum_attempts=2 # MobSF can be flaky, limit retries
),
)
workflow.logger.info(
f"✓ MobSF completed: {mobsf_result.get('summary', {}).get('total_findings', 0)} findings"
)
# Handle skipped or completed status
if mobsf_result.get("status") == "skipped":
workflow.logger.warning(
f"⚠️ MobSF skipped: {mobsf_result.get('summary', {}).get('skip_reason', 'Unknown reason')}"
)
else:
workflow.logger.info(
f"✓ MobSF completed: {mobsf_result.get('summary', {}).get('total_findings', 0)} findings"
)
except Exception as e:
workflow.logger.warning(f"MobSF scan failed (continuing without it): {e}")
mobsf_result = None

211
cli/src/fuzzforge_cli/worker_manager.py
View File

@@ -15,11 +15,15 @@ Manages on-demand startup and shutdown of Temporal workers using Docker Compose.
# Additional attribution and requirements are provided in the NOTICE file.
import logging
import os
import platform
import subprocess
import time
from pathlib import Path
from typing import Optional, Dict, Any
import requests
import yaml
from rich.console import Console
logger = logging.getLogger(__name__)
@@ -57,27 +61,181 @@ class WorkerManager:
def _find_compose_file(self) -> Path:
"""
Auto-detect docker-compose.yml location.
Auto-detect docker-compose.yml location using multiple strategies.
Searches upward from current directory to find the compose file.
Strategies (in order):
1. Query backend API for host path
2. Search upward for .fuzzforge marker directory
3. Use FUZZFORGE_ROOT environment variable
4. Fallback to current directory
Returns:
Path to docker-compose.yml
Raises:
FileNotFoundError: If docker-compose.yml cannot be located
"""
current = Path.cwd()
# Strategy 1: Ask backend for location
try:
backend_url = os.getenv("FUZZFORGE_API_URL", "http://localhost:8000")
response = requests.get(f"{backend_url}/system/info", timeout=2)
if response.ok:
info = response.json()
if compose_path_str := info.get("docker_compose_path"):
compose_path = Path(compose_path_str)
if compose_path.exists():
logger.debug(f"Found docker-compose.yml via backend API: {compose_path}")
return compose_path
except Exception as e:
logger.debug(f"Backend API not reachable for path lookup: {e}")
# Try current directory and parents
# Strategy 2: Search upward for .fuzzforge marker directory
current = Path.cwd()
for parent in [current] + list(current.parents):
compose_path = parent / "docker-compose.yml"
if (parent / ".fuzzforge").exists():
compose_path = parent / "docker-compose.yml"
if compose_path.exists():
logger.debug(f"Found docker-compose.yml via .fuzzforge marker: {compose_path}")
return compose_path
# Strategy 3: Environment variable
if fuzzforge_root := os.getenv("FUZZFORGE_ROOT"):
compose_path = Path(fuzzforge_root) / "docker-compose.yml"
if compose_path.exists():
logger.debug(f"Found docker-compose.yml via FUZZFORGE_ROOT: {compose_path}")
return compose_path
# Fallback to default location
return Path("docker-compose.yml")
# Strategy 4: Fallback to current directory
compose_path = Path("docker-compose.yml")
if compose_path.exists():
return compose_path
def _run_docker_compose(self, *args: str) -> subprocess.CompletedProcess:
raise FileNotFoundError(
"Cannot find docker-compose.yml. Ensure backend is running, "
"run from FuzzForge directory, or set FUZZFORGE_ROOT environment variable."
)
def _get_workers_dir(self) -> Path:
"""
Run docker-compose command.
Get the workers directory path.
Uses same strategy as _find_compose_file():
1. Query backend API
2. Derive from compose_file location
3. Use FUZZFORGE_ROOT
Returns:
Path to workers directory
"""
# Strategy 1: Ask backend
try:
backend_url = os.getenv("FUZZFORGE_API_URL", "http://localhost:8000")
response = requests.get(f"{backend_url}/system/info", timeout=2)
if response.ok:
info = response.json()
if workers_dir_str := info.get("workers_dir"):
workers_dir = Path(workers_dir_str)
if workers_dir.exists():
return workers_dir
except Exception:
pass
# Strategy 2: Derive from compose file location
if self.compose_file.exists():
workers_dir = self.compose_file.parent / "workers"
if workers_dir.exists():
return workers_dir
# Strategy 3: Use environment variable
if fuzzforge_root := os.getenv("FUZZFORGE_ROOT"):
workers_dir = Path(fuzzforge_root) / "workers"
if workers_dir.exists():
return workers_dir
# Fallback
return Path("workers")
def _detect_platform(self) -> str:
"""
Detect the current platform.
Returns:
Platform string: "linux/amd64" or "linux/arm64"
"""
machine = platform.machine().lower()
if machine in ["x86_64", "amd64"]:
return "linux/amd64"
elif machine in ["arm64", "aarch64"]:
return "linux/arm64"
return "unknown"
def _read_worker_metadata(self, vertical: str) -> dict:
"""
Read worker metadata.yaml for a vertical.
Args:
vertical: Worker vertical name (e.g., "android", "python")
Returns:
Dictionary containing metadata, or empty dict if not found
"""
try:
workers_dir = self._get_workers_dir()
metadata_file = workers_dir / vertical / "metadata.yaml"
if not metadata_file.exists():
logger.debug(f"No metadata.yaml found for {vertical}")
return {}
with open(metadata_file, 'r') as f:
return yaml.safe_load(f) or {}
except Exception as e:
logger.debug(f"Failed to read metadata for {vertical}: {e}")
return {}
def _select_dockerfile(self, vertical: str) -> str:
"""
Select the appropriate Dockerfile for the current platform.
Args:
vertical: Worker vertical name
Returns:
Dockerfile name (e.g., "Dockerfile.amd64", "Dockerfile.arm64")
"""
detected_platform = self._detect_platform()
metadata = self._read_worker_metadata(vertical)
if not metadata:
# No metadata: use default Dockerfile
logger.debug(f"No metadata for {vertical}, using Dockerfile")
return "Dockerfile"
platforms = metadata.get("platforms", {})
# Try detected platform first
if detected_platform in platforms:
dockerfile = platforms[detected_platform].get("dockerfile", "Dockerfile")
logger.debug(f"Selected {dockerfile} for {vertical} on {detected_platform}")
return dockerfile
# Fallback to default platform
default_platform = metadata.get("default_platform", "linux/amd64")
if default_platform in platforms:
dockerfile = platforms[default_platform].get("dockerfile", "Dockerfile.amd64")
logger.debug(f"Using default platform {default_platform}: {dockerfile}")
return dockerfile
# Last resort
return "Dockerfile"
def _run_docker_compose(self, *args: str, env: Optional[Dict[str, str]] = None) -> subprocess.CompletedProcess:
"""
Run docker-compose command with optional environment variables.
Args:
*args: Arguments to pass to docker-compose
env: Optional environment variables to set
Returns:
CompletedProcess with result
@@ -88,11 +246,18 @@ class WorkerManager:
cmd = ["docker-compose", "-f", str(self.compose_file)] + list(args)
logger.debug(f"Running: {' '.join(cmd)}")
# Merge with current environment
full_env = os.environ.copy()
if env:
full_env.update(env)
logger.debug(f"Environment overrides: {env}")
return subprocess.run(
cmd,
capture_output=True,
text=True,
check=True
check=True,
env=full_env
)
def _service_to_container_name(self, service_name: str) -> str:
@@ -135,21 +300,35 @@ class WorkerManager:
def start_worker(self, service_name: str) -> bool:
"""
Start a worker service using docker-compose.
Start a worker service using docker-compose with platform-specific Dockerfile.
Args:
service_name: Name of the Docker Compose service to start (e.g., "worker-python")
service_name: Name of the Docker Compose service to start (e.g., "worker-android")
Returns:
True if started successfully, False otherwise
"""
try:
console.print(f"🚀 Starting worker: {service_name}")
# Extract vertical name from service name
vertical = service_name.replace("worker-", "")
# Use docker-compose up to create and start the service
result = self._run_docker_compose("up", "-d", service_name)
# Detect platform and select appropriate Dockerfile
detected_platform = self._detect_platform()
dockerfile = self._select_dockerfile(vertical)
logger.info(f"Worker {service_name} started")
# Set environment variable for docker-compose
env_var_name = f"{vertical.upper()}_DOCKERFILE"
env = {env_var_name: dockerfile}
console.print(
f"🚀 Starting worker: {service_name} "
f"(platform: {detected_platform}, using {dockerfile})"
)
# Use docker-compose up with --build to ensure correct Dockerfile is used
result = self._run_docker_compose("up", "-d", "--build", service_name, env=env)
logger.info(f"Worker {service_name} started with {dockerfile}")
return True
except subprocess.CalledProcessError as e:

5
docker-compose.yml
View File

@@ -342,7 +342,7 @@ services:
worker-android:
build:
context: ./workers/android
dockerfile: Dockerfile
dockerfile: ${ANDROID_DOCKERFILE:-Dockerfile.amd64}
container_name: fuzzforge-worker-android
profiles:
- workers
@@ -430,6 +430,9 @@ services:
PYTHONPATH: /app
PYTHONUNBUFFERED: 1
# Host filesystem paths (for CLI worker management)
FUZZFORGE_HOST_ROOT: ${PWD}
# Logging
LOG_LEVEL: INFO
ports:

0
workers/android/Dockerfile → workers/android/Dockerfile.amd64
View File

110
workers/android/Dockerfile.arm64 Normal file
View File

@@ -0,0 +1,110 @@
# FuzzForge Vertical Worker: Android Security (ARM64)
#
# Pre-installed tools for Android security analysis:
# - Android SDK (adb, aapt)
# - apktool (APK decompilation)
# - jadx (Dex to Java decompiler)
# - Frida (dynamic instrumentation)
# - androguard (Python APK analysis)
#
# Note: MobSF is excluded due to Rosetta 2 syscall incompatibility
# Note: Uses amd64 platform for compatibility with Android 32-bit tools
FROM --platform=linux/amd64 python:3.11-slim-bookworm
# Set working directory
WORKDIR /app
# Install system dependencies
RUN apt-get update && apt-get install -y \
# Build essentials
build-essential \
git \
curl \
wget \
unzip \
# Java (required for Android tools)
openjdk-17-jdk \
# Android tools dependencies (32-bit libraries for emulated amd64)
lib32stdc++6 \
lib32z1 \
# Frida dependencies
libc6-dev \
# XML/Binary analysis
libxml2-dev \
libxslt-dev \
# Network tools
netcat-openbsd \
tcpdump \
# Cleanup
&& rm -rf /var/lib/apt/lists/*
# Install Android SDK Command Line Tools
ENV ANDROID_HOME=/opt/android-sdk
ENV PATH="${ANDROID_HOME}/cmdline-tools/latest/bin:${ANDROID_HOME}/platform-tools:${PATH}"
RUN mkdir -p ${ANDROID_HOME}/cmdline-tools && \
cd ${ANDROID_HOME}/cmdline-tools && \
wget -q https://dl.google.com/android/repository/commandlinetools-linux-9477386_latest.zip && \
unzip -q commandlinetools-linux-9477386_latest.zip && \
mv cmdline-tools latest && \
rm commandlinetools-linux-9477386_latest.zip && \
# Accept licenses
yes | ${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager --licenses && \
# Install platform tools (adb, fastboot)
${ANDROID_HOME}/cmdline-tools/latest/bin/sdkmanager "platform-tools" "build-tools;33.0.0"
# Install apktool
RUN wget -q https://raw.githubusercontent.com/iBotPeaches/Apktool/master/scripts/linux/apktool -O /usr/local/bin/apktool && \
wget -q https://bitbucket.org/iBotPeaches/apktool/downloads/apktool_2.9.3.jar -O /usr/local/bin/apktool.jar && \
chmod +x /usr/local/bin/apktool
# Install jadx (Dex to Java decompiler)
RUN wget -q https://github.com/skylot/jadx/releases/download/v1.4.7/jadx-1.4.7.zip -O /tmp/jadx.zip && \
unzip -q /tmp/jadx.zip -d /opt/jadx && \
ln -s /opt/jadx/bin/jadx /usr/local/bin/jadx && \
ln -s /opt/jadx/bin/jadx-gui /usr/local/bin/jadx-gui && \
rm /tmp/jadx.zip
# Install Python dependencies for Android security tools
COPY requirements.txt /tmp/requirements.txt
RUN pip3 install --no-cache-dir -r /tmp/requirements.txt && \
rm /tmp/requirements.txt
# Install androguard (Python APK analysis framework)
RUN pip3 install --no-cache-dir androguard pyaxmlparser
# Install Frida
RUN pip3 install --no-cache-dir frida-tools frida
# Install OpenGrep/Semgrep (expose as opengrep command)
RUN pip3 install --no-cache-dir semgrep==1.45.0 && \
ln -sf /usr/local/bin/semgrep /usr/local/bin/opengrep
# NOTE: MobSF is NOT installed on ARM64 platform due to Rosetta 2 incompatibility
# The workflow will gracefully skip MobSF analysis on this platform
# Create cache directory
RUN mkdir -p /cache && chmod 755 /cache
# Copy worker entrypoint (generic, works for all verticals)
COPY worker.py /app/worker.py
# Create simplified startup script (no MobSF)
RUN echo '#!/bin/bash\n\
# ARM64 worker - MobSF disabled due to Rosetta 2 limitations\n\
echo "Starting Temporal worker (ARM64 platform - MobSF disabled)..."\n\
exec python3 /app/worker.py\n\
' > /app/start.sh && chmod +x /app/start.sh
# Add toolbox to Python path (mounted at runtime)
ENV PYTHONPATH="/app:/app/toolbox:${PYTHONPATH}"
ENV PYTHONUNBUFFERED=1
ENV JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
# Healthcheck
HEALTHCHECK --interval=30s --timeout=10s --start-period=60s --retries=5 \
CMD python3 -c "import sys; sys.exit(0)"
# Run startup script
CMD ["/app/start.sh"]

42
workers/android/metadata.yaml Normal file
View File

@@ -0,0 +1,42 @@
# Android Worker Metadata
#
# Platform-specific configuration for Android security analysis worker.
# This file defines which Dockerfile to use for each platform and what tools
# are available on that platform.
name: android
version: "1.0.0"
description: "Android application security testing worker with Jadx, OpenGrep, and MobSF"
# Default platform when auto-detection fails or metadata is not platform-aware
default_platform: linux/amd64
# Platform-specific configurations
platforms:
# x86_64 / Intel / AMD platform (full toolchain including MobSF)
linux/amd64:
dockerfile: Dockerfile.amd64
description: "Full Android toolchain with MobSF support"
supported_tools:
- jadx # APK decompiler
- opengrep # Static analysis with custom Android rules
- mobsf # Mobile Security Framework
- frida # Dynamic instrumentation
- androguard # Python APK analysis
# ARM64 / Apple Silicon platform (MobSF excluded due to Rosetta limitations)
linux/arm64:
dockerfile: Dockerfile.arm64
description: "Android toolchain without MobSF (ARM64/Apple Silicon compatible)"
supported_tools:
- jadx # APK decompiler
- opengrep # Static analysis with custom Android rules
- frida # Dynamic instrumentation
- androguard # Python APK analysis
disabled_tools:
mobsf: "Incompatible with Rosetta 2 emulation (requires syscall 284: copy_file_range)"
notes: |
MobSF cannot run under Rosetta 2 on Apple Silicon Macs due to missing
syscall implementations. The workflow will gracefully skip MobSF analysis
on this platform while still providing comprehensive security testing via
Jadx decompilation and OpenGrep static analysis.