Files
fuzzforge_ai/cli/src/fuzzforge_cli/exceptions.py
T
tduhamel42 ec812461d6 CI/CD Integration with Ephemeral Deployment Model (#14)
* feat: Complete migration from Prefect to Temporal

BREAKING CHANGE: Replaces Prefect workflow orchestration with Temporal

## Major Changes
- Replace Prefect with Temporal for workflow orchestration
- Implement vertical worker architecture (rust, android)
- Replace Docker registry with MinIO for unified storage
- Refactor activities to be co-located with workflows
- Update all API endpoints for Temporal compatibility

## Infrastructure
- New: docker-compose.temporal.yaml (Temporal + MinIO + workers)
- New: workers/ directory with rust and android vertical workers
- New: backend/src/temporal/ (manager, discovery)
- New: backend/src/storage/ (S3-cached storage with MinIO)
- New: backend/toolbox/common/ (shared storage activities)
- Deleted: docker-compose.yaml (old Prefect setup)
- Deleted: backend/src/core/prefect_manager.py
- Deleted: backend/src/services/prefect_stats_monitor.py
- Deleted: Docker registry and insecure-registries requirement

## Workflows
- Migrated: security_assessment workflow to Temporal
- New: rust_test workflow (example/test workflow)
- Deleted: secret_detection_scan (Prefect-based, to be reimplemented)
- Activities now co-located with workflows for independent testing

## API Changes
- Updated: backend/src/api/workflows.py (Temporal submission)
- Updated: backend/src/api/runs.py (Temporal status/results)
- Updated: backend/src/main.py (727 lines, TemporalManager integration)
- Updated: All 16 MCP tools to use TemporalManager

## Testing
-  All services healthy (Temporal, PostgreSQL, MinIO, workers, backend)
-  All API endpoints functional
-  End-to-end workflow test passed (72 findings from vulnerable_app)
-  MinIO storage integration working (target upload/download, results)
-  Worker activity discovery working (6 activities registered)
-  Tarball extraction working
-  SARIF report generation working

## Documentation
- ARCHITECTURE.md: Complete Temporal architecture documentation
- QUICKSTART_TEMPORAL.md: Getting started guide
- MIGRATION_DECISION.md: Why we chose Temporal over Prefect
- IMPLEMENTATION_STATUS.md: Migration progress tracking
- workers/README.md: Worker development guide

## Dependencies
- Added: temporalio>=1.6.0
- Added: boto3>=1.34.0 (MinIO S3 client)
- Removed: prefect>=3.4.18

* feat: Add Python fuzzing vertical with Atheris integration

This commit implements a complete Python fuzzing workflow using Atheris:

## Python Worker (workers/python/)
- Dockerfile with Python 3.11, Atheris, and build tools
- Generic worker.py for dynamic workflow discovery
- requirements.txt with temporalio, boto3, atheris dependencies
- Added to docker-compose.temporal.yaml with dedicated cache volume

## AtherisFuzzer Module (backend/toolbox/modules/fuzzer/)
- Reusable module extending BaseModule
- Auto-discovers fuzz targets (fuzz_*.py, *_fuzz.py, fuzz_target.py)
- Recursive search to find targets in nested directories
- Dynamically loads TestOneInput() function
- Configurable max_iterations and timeout
- Real-time stats callback support for live monitoring
- Returns findings as ModuleFinding objects

## Atheris Fuzzing Workflow (backend/toolbox/workflows/atheris_fuzzing/)
- Temporal workflow for orchestrating fuzzing
- Downloads user code from MinIO
- Executes AtherisFuzzer module
- Uploads results to MinIO
- Cleans up cache after execution
- metadata.yaml with vertical: python for routing

## Test Project (test_projects/python_fuzz_waterfall/)
- Demonstrates stateful waterfall vulnerability
- main.py with check_secret() that leaks progress
- fuzz_target.py with Atheris TestOneInput() harness
- Complete README with usage instructions

## Backend Fixes
- Fixed parameter merging in REST API endpoints (workflows.py)
- Changed workflow parameter passing from positional args to kwargs (manager.py)
- Default parameters now properly merged with user parameters

## Testing
 Worker discovered AtherisFuzzingWorkflow
 Workflow executed end-to-end successfully
 Fuzz target auto-discovered in nested directories
 Atheris ran 100,000 iterations
 Results uploaded and cache cleaned

* chore: Complete Temporal migration with updated CLI/SDK/docs

This commit includes all remaining Temporal migration changes:

## CLI Updates (cli/)
- Updated workflow execution commands for Temporal
- Enhanced error handling and exceptions
- Updated dependencies in uv.lock

## SDK Updates (sdk/)
- Client methods updated for Temporal workflows
- Updated models for new workflow execution
- Updated dependencies in uv.lock

## Documentation Updates (docs/)
- Architecture documentation for Temporal
- Workflow concept documentation
- Resource management documentation (new)
- Debugging guide (new)
- Updated tutorials and how-to guides
- Troubleshooting updates

## README Updates
- Main README with Temporal instructions
- Backend README
- CLI README
- SDK README

## Other
- Updated IMPLEMENTATION_STATUS.md
- Removed old vulnerable_app.tar.gz

These changes complete the Temporal migration and ensure the
CLI/SDK work correctly with the new backend.

* fix: Use positional args instead of kwargs for Temporal workflows

The Temporal Python SDK's start_workflow() method doesn't accept
a 'kwargs' parameter. Workflows must receive parameters as positional
arguments via the 'args' parameter.

Changed from:
  args=workflow_args  # Positional arguments

This fixes the error:
  TypeError: Client.start_workflow() got an unexpected keyword argument 'kwargs'

Workflows now correctly receive parameters in order:
- security_assessment: [target_id, scanner_config, analyzer_config, reporter_config]
- atheris_fuzzing: [target_id, target_file, max_iterations, timeout_seconds]
- rust_test: [target_id, test_message]

* fix: Filter metadata-only parameters from workflow arguments

SecurityAssessmentWorkflow was receiving 7 arguments instead of 2-5.
The issue was that target_path and volume_mode from default_parameters
were being passed to the workflow, when they should only be used by
the system for configuration.

Now filters out metadata-only parameters (target_path, volume_mode)
before passing arguments to workflow execution.

* refactor: Remove Prefect leftovers and volume mounting legacy

Complete cleanup of Prefect migration artifacts:

Backend:
- Delete registry.py and workflow_discovery.py (Prefect-specific files)
- Remove Docker validation from setup.py (no longer needed)
- Remove ResourceLimits and VolumeMount models
- Remove target_path and volume_mode from WorkflowSubmission
- Remove supported_volume_modes from API and discovery
- Clean up metadata.yaml files (remove volume/path fields)
- Simplify parameter filtering in manager.py

SDK:
- Remove volume_mode parameter from client methods
- Remove ResourceLimits and VolumeMount models
- Remove Prefect error patterns from docker_logs.py
- Clean up WorkflowSubmission and WorkflowMetadata models

CLI:
- Remove Volume Modes display from workflow info

All removed features are Prefect-specific or Docker volume mounting
artifacts. Temporal workflows use MinIO storage exclusively.

* feat: Add comprehensive test suite and benchmark infrastructure

- Add 68 unit tests for fuzzer, scanner, and analyzer modules
- Implement pytest-based test infrastructure with fixtures
- Add 6 performance benchmarks with category-specific thresholds
- Configure GitHub Actions for automated testing and benchmarking
- Add test and benchmark documentation

Test coverage:
- AtherisFuzzer: 8 tests
- CargoFuzzer: 14 tests
- FileScanner: 22 tests
- SecurityAnalyzer: 24 tests

All tests passing (68/68)
All benchmarks passing (6/6)

* fix: Resolve all ruff linting violations across codebase

Fixed 27 ruff violations in 12 files:
- Removed unused imports (Depends, Dict, Any, Optional, etc.)
- Fixed undefined workflow_info variable in workflows.py
- Removed dead code with undefined variables in atheris_fuzzer.py
- Changed f-string to regular string where no placeholders used

All files now pass ruff checks for CI/CD compliance.

* fix: Configure CI for unit tests only

- Renamed docker-compose.temporal.yaml → docker-compose.yml for CI compatibility
- Commented out integration-tests job (no integration tests yet)
- Updated test-summary to only depend on lint and unit-tests

CI will now run successfully with 68 unit tests. Integration tests can be added later.

* feat: Add CI/CD integration with ephemeral deployment model

Implements comprehensive CI/CD support for FuzzForge with on-demand worker management:

**Worker Management (v0.7.0)**
- Add WorkerManager for automatic worker lifecycle control
- Auto-start workers from stopped state when workflows execute
- Auto-stop workers after workflow completion
- Health checks and startup timeout handling (90s default)

**CI/CD Features**
- `--fail-on` flag: Fail builds based on SARIF severity levels (error/warning/note/info)
- `--export-sarif` flag: Export findings in SARIF 2.1.0 format
- `--auto-start`/`--auto-stop` flags: Control worker lifecycle
- Exit code propagation: Returns 1 on blocking findings, 0 on success

**Exit Code Fix**
- Add `except typer.Exit: raise` handlers at 3 critical locations
- Move worker cleanup to finally block for guaranteed execution
- Exit codes now propagate correctly even when build fails

**CI Scripts & Examples**
- ci-start.sh: Start FuzzForge services with health checks
- ci-stop.sh: Clean shutdown with volume preservation option
- GitHub Actions workflow example (security-scan.yml)
- GitLab CI pipeline example (.gitlab-ci.example.yml)
- docker-compose.ci.yml: CI-optimized compose file with profiles

**OSS-Fuzz Integration**
- New ossfuzz_campaign workflow for running OSS-Fuzz projects
- OSS-Fuzz worker with Docker-in-Docker support
- Configurable campaign duration and project selection

**Documentation**
- Comprehensive CI/CD integration guide (docs/how-to/cicd-integration.md)
- Updated architecture docs with worker lifecycle details
- Updated workspace isolation documentation
- CLI README with worker management examples

**SDK Enhancements**
- Add get_workflow_worker_info() endpoint
- Worker vertical metadata in workflow responses

**Testing**
- All workflows tested: security_assessment, atheris_fuzzing, secret_detection, cargo_fuzzing
- All monitoring commands tested: stats, crashes, status, finding
- Full CI pipeline simulation verified
- Exit codes verified for success/failure scenarios

Ephemeral CI/CD model: ~3-4GB RAM, ~60-90s startup, runs entirely in CI containers.

* fix: Resolve ruff linting violations in CI/CD code

- Remove unused variables (run_id, defaults, result)
- Remove unused imports
- Fix f-string without placeholders

All CI/CD integration files now pass ruff checks.
2025-10-14 10:13:45 +02:00

478 lines
16 KiB
Python

"""
Enhanced exception handling and error utilities for FuzzForge CLI with rich context display.
"""
# Copyright (c) 2025 FuzzingLabs
#
# Licensed under the Business Source License 1.1 (BSL). See the LICENSE file
# at the root of this repository for details.
#
# After the Change Date (four years from publication), this version of the
# Licensed Work will be made available under the Apache License, Version 2.0.
# See the LICENSE-APACHE file or http://www.apache.org/licenses/LICENSE-2.0
#
# Additional attribution and requirements are provided in the NOTICE file.
import time
import functools
from typing import Any, Callable, Optional, Union, List
from pathlib import Path
import typer
import httpx
from rich.console import Console
from rich.panel import Panel
from rich.text import Text
from rich.table import Table
# Import SDK exceptions for rich handling
from fuzzforge_sdk.exceptions import (
FuzzForgeError as SDKFuzzForgeError
)
console = Console()
class FuzzForgeError(Exception):
"""Base exception for FuzzForge CLI errors (legacy CLI-specific errors)"""
def __init__(self, message: str, hint: Optional[str] = None, exit_code: int = 1):
self.message = message
self.hint = hint
self.exit_code = exit_code
super().__init__(message)
class ProjectNotFoundError(FuzzForgeError):
"""Raised when no FuzzForge project is found in current directory"""
def __init__(self):
super().__init__(
"No FuzzForge project found in current directory",
"Run 'ff init' to initialize a new project"
)
class APIConnectionError(FuzzForgeError):
"""Legacy API connection error for backward compatibility"""
def __init__(self, url: str, original_error: Exception):
self.url = url
self.original_error = original_error
if isinstance(original_error, httpx.ConnectTimeout):
message = f"Connection timeout to FuzzForge API at {url}"
hint = "Check if the API server is running and the URL is correct"
elif isinstance(original_error, httpx.ConnectError):
message = f"Failed to connect to FuzzForge API at {url}"
hint = "Verify the API URL is correct and the server is accessible"
elif isinstance(original_error, httpx.TimeoutException):
message = f"Request timeout to FuzzForge API at {url}"
hint = "The API server may be overloaded. Try again later"
else:
message = f"API connection error: {str(original_error)}"
hint = "Check your network connection and API configuration"
super().__init__(message, hint)
class DatabaseError(FuzzForgeError):
"""Raised when database operations fail"""
def __init__(self, operation: str, original_error: Exception):
self.operation = operation
self.original_error = original_error
message = f"Database error during {operation}: {str(original_error)}"
hint = "The database may be corrupted. Try 'ff init --force' to reset"
super().__init__(message, hint)
class ValidationError(FuzzForgeError):
"""Legacy validation error for CLI-specific validation"""
def __init__(self, field: str, value: Any, expected: str):
self.field = field
self.value = value
self.expected = expected
message = f"Invalid {field}: {value}"
hint = f"Expected {expected}"
super().__init__(message, hint)
class FileOperationError(FuzzForgeError):
"""Raised when file operations fail"""
def __init__(self, operation: str, path: Union[str, Path], original_error: Exception):
self.operation = operation
self.path = Path(path)
self.original_error = original_error
if isinstance(original_error, FileNotFoundError):
message = f"File not found: {path}"
hint = "Check the path exists and you have permission to access it"
elif isinstance(original_error, PermissionError):
message = f"Permission denied: {path}"
hint = "Check file permissions or run with appropriate privileges"
else:
message = f"File operation failed ({operation}): {str(original_error)}"
hint = "Check the file path and permissions"
super().__init__(message, hint)
def display_container_logs(diagnostics, title: str = "Container Logs"):
"""Display container logs in a rich format."""
if not diagnostics or not diagnostics.logs:
return
# Show last 20 lines of logs
recent_logs = diagnostics.logs[-20:] if len(diagnostics.logs) > 20 else diagnostics.logs
log_content = []
for log_entry in recent_logs:
timestamp = log_entry.timestamp.strftime("%H:%M:%S")
level_color = {
'ERROR': 'red',
'WARNING': 'yellow',
'INFO': 'blue',
'DEBUG': 'dim white'
}.get(log_entry.level, 'white')
log_line = f"[dim]{timestamp}[/dim] [{level_color}]{log_entry.level}[/{level_color}] {log_entry.message}"
log_content.append(log_line)
if log_content:
logs_panel = Panel(
"\n".join(log_content),
title=title,
title_align="left",
border_style="dim",
expand=False
)
console.print(logs_panel)
def display_container_diagnostics(diagnostics):
"""Display comprehensive container diagnostics."""
if not diagnostics:
return
# Container Status Table
status_table = Table(title="Container Status", show_header=False, box=None)
status_table.add_column("Property", style="bold")
status_table.add_column("Value")
status_color = {
'running': 'green',
'exited': 'red',
'failed': 'red',
'created': 'yellow',
'unknown': 'dim'
}.get(diagnostics.status.lower(), 'white')
status_table.add_row("Status", f"[{status_color}]{diagnostics.status}[/{status_color}]")
if diagnostics.exit_code is not None:
exit_color = 'green' if diagnostics.exit_code == 0 else 'red'
status_table.add_row("Exit Code", f"[{exit_color}]{diagnostics.exit_code}[/{exit_color}]")
if diagnostics.error:
status_table.add_row("Error", f"[red]{diagnostics.error}[/red]")
# Resource Usage
if diagnostics.resource_usage:
memory_limit = diagnostics.resource_usage.get('memory_limit', 0)
if memory_limit > 0:
memory_mb = memory_limit // (1024 * 1024)
status_table.add_row("Memory Limit", f"{memory_mb} MB")
console.print(status_table)
# Volume Mounts
if diagnostics.volume_mounts:
console.print("\n[bold]Volume Mounts:[/bold]")
for mount in diagnostics.volume_mounts:
mount_info = f" {mount['source']}{mount['destination']} ([dim]{mount['mode']}[/dim])"
console.print(mount_info)
def display_error_patterns(error_patterns):
"""Display detected error patterns."""
if not error_patterns:
return
console.print("\n[bold red]🔍 Detected Issues:[/bold red]")
for error_type, messages in error_patterns.items():
# Format error type name
formatted_type = error_type.replace('_', ' ').title()
console.print(f"\n[bold yellow]• {formatted_type}:[/bold yellow]")
for message in messages[:3]: # Show first 3 messages
console.print(f" [dim]▸[/dim] {message}")
if len(messages) > 3:
console.print(f" [dim]▸ ... and {len(messages) - 3} more similar messages[/dim]")
def display_suggestions(suggestions: List[str]):
"""Display actionable suggestions."""
if not suggestions:
return
console.print("\n[bold green]💡 Suggested Fixes:[/bold green]")
for i, suggestion in enumerate(suggestions[:6], 1): # Show max 6 suggestions
console.print(f" [bold green]{i}.[/bold green] {suggestion}")
def handle_error(error: Exception, context: str = "") -> None:
"""
Display comprehensive error messages with rich context and exit appropriately.
Args:
error: The exception that occurred
context: Additional context about where the error occurred
"""
# Handle SDK errors with rich context
if isinstance(error, SDKFuzzForgeError):
console.print() # Add some spacing
# Main error message
error_title = f"{error.__class__.__name__}"
if context:
error_title += f" during {context}"
console.print(Panel(
error.get_summary(),
title=error_title,
title_align="left",
border_style="red",
expand=False
))
# Show detailed context if available
if hasattr(error, 'context') and error.context:
ctx = error.context
# Container diagnostics
if ctx.container_diagnostics:
console.print("\n[bold]Container Diagnostics:[/bold]")
display_container_diagnostics(ctx.container_diagnostics)
display_container_logs(ctx.container_diagnostics)
# Error patterns
if ctx.error_patterns:
display_error_patterns(ctx.error_patterns)
# API context
if ctx.url:
console.print(f"\n[dim]Request URL: {ctx.url}[/dim]")
if ctx.response_data and isinstance(ctx.response_data, dict) and 'raw' not in ctx.response_data:
console.print(f"[dim]API Response: {ctx.response_data}[/dim]")
# Suggestions
if ctx.suggested_fixes:
display_suggestions(ctx.suggested_fixes)
console.print() # Add spacing before exit
raise typer.Exit(1)
# Handle legacy CLI errors
elif isinstance(error, FuzzForgeError):
error_text = Text()
error_text.append("", style="red")
error_text.append(error.message, style="red")
if context:
error_text.append(f" ({context})", style="dim red")
console.print(error_text)
if error.hint:
hint_text = Text()
hint_text.append("💡 ", style="yellow")
hint_text.append(error.hint, style="yellow")
console.print(hint_text)
raise typer.Exit(error.exit_code)
elif isinstance(error, KeyboardInterrupt):
console.print("\n⏹️ Operation cancelled by user", style="yellow")
raise typer.Exit(130) # Standard exit code for SIGINT
else:
# Unexpected errors - show minimal info to user, log details
console.print()
error_panel = Panel(
f"An unexpected error occurred: {str(error)}",
title="❌ Unexpected Error",
title_align="left",
border_style="red",
expand=False
)
if context:
error_panel.title += f" during {context}"
console.print(error_panel)
# Show error details for debugging
console.print(f"\n[dim yellow]Error type: {type(error).__name__}[/dim yellow]")
console.print("[dim yellow]Please report this issue if it persists[/dim yellow]")
console.print()
raise typer.Exit(1)
def retry_on_network_error(max_retries: int = 3, delay: float = 1.0, backoff_multiplier: float = 2.0):
"""
Decorator to retry network operations with exponential backoff.
Args:
max_retries: Maximum number of retry attempts
delay: Initial delay between retries in seconds
backoff_multiplier: Multiplier for exponential backoff
"""
def decorator(func: Callable) -> Callable:
@functools.wraps(func)
def wrapper(*args, **kwargs):
last_exception = None
current_delay = delay
for attempt in range(max_retries + 1):
try:
return func(*args, **kwargs)
except (httpx.ConnectError, httpx.TimeoutException, httpx.NetworkError) as e:
last_exception = e
if attempt < max_retries:
console.print(
f"🔄 Network error, retrying in {current_delay:.1f}s... "
f"(attempt {attempt + 1}/{max_retries})",
style="yellow"
)
time.sleep(current_delay)
current_delay *= backoff_multiplier
else:
# Convert to our custom error type
api_url = getattr(args[0], 'base_url', 'unknown') if args else 'unknown'
raise APIConnectionError(str(api_url), e)
# Should never reach here, but just in case
if last_exception:
raise last_exception
return wrapper
return decorator
def validate_path(path: Union[str, Path], must_exist: bool = True, must_be_file: bool = False,
must_be_dir: bool = False) -> Path:
"""
Validate file/directory paths with user-friendly error messages.
Args:
path: Path to validate
must_exist: Whether the path must exist
must_be_file: Whether the path must be a file
must_be_dir: Whether the path must be a directory
Returns:
Validated Path object
Raises:
ValidationError: If validation fails
"""
path_obj = Path(path)
if must_exist and not path_obj.exists():
raise ValidationError("path", str(path), "an existing path")
if must_be_file and path_obj.exists() and not path_obj.is_file():
raise ValidationError("path", str(path), "a file")
if must_be_dir and path_obj.exists() and not path_obj.is_dir():
raise ValidationError("path", str(path), "a directory")
return path_obj
def validate_run_id(run_id: str) -> str:
"""
Validate run ID format.
Args:
run_id: Run ID to validate
Returns:
Validated run ID
Raises:
ValidationError: If run ID format is invalid
"""
if not run_id or len(run_id) < 8:
raise ValidationError("run_id", run_id, "at least 8 characters")
# Allow alphanumeric characters, hyphens, and underscores
if not run_id.replace('-', '').replace('_', '').isalnum():
raise ValidationError("run_id", run_id, "alphanumeric characters, hyphens, and underscores only")
return run_id
def safe_json_load(file_path: Union[str, Path]) -> dict:
"""
Safely load JSON file with proper error handling.
Args:
file_path: Path to JSON file
Returns:
Parsed JSON data
Raises:
FileOperationError: If file operation fails
ValidationError: If JSON is invalid
"""
path_obj = Path(file_path)
try:
with open(path_obj, 'r', encoding='utf-8') as f:
import json
return json.load(f)
except FileNotFoundError as e:
raise FileOperationError("read", path_obj, e)
except PermissionError as e:
raise FileOperationError("read", path_obj, e)
except json.JSONDecodeError as e:
raise ValidationError("JSON file", str(path_obj), f"valid JSON format (error: {e})")
except Exception as e:
raise FileOperationError("read", path_obj, e)
def require_project() -> Path:
"""
Ensure we're in a FuzzForge project directory.
Returns:
Path to project root
Raises:
ProjectNotFoundError: If not in a project directory
"""
current = Path.cwd()
# Look for .fuzzforge directory in current or parent directories
for path in [current] + list(current.parents):
fuzzforge_dir = path / ".fuzzforge"
if fuzzforge_dir.is_dir():
return path
raise ProjectNotFoundError()