mirror of
https://github.com/FuzzingLabs/fuzzforge_ai.git
synced 2026-05-24 06:54:01 +02:00
CI/CD Integration with Ephemeral Deployment Model (#14)
* feat: Complete migration from Prefect to Temporal BREAKING CHANGE: Replaces Prefect workflow orchestration with Temporal ## Major Changes - Replace Prefect with Temporal for workflow orchestration - Implement vertical worker architecture (rust, android) - Replace Docker registry with MinIO for unified storage - Refactor activities to be co-located with workflows - Update all API endpoints for Temporal compatibility ## Infrastructure - New: docker-compose.temporal.yaml (Temporal + MinIO + workers) - New: workers/ directory with rust and android vertical workers - New: backend/src/temporal/ (manager, discovery) - New: backend/src/storage/ (S3-cached storage with MinIO) - New: backend/toolbox/common/ (shared storage activities) - Deleted: docker-compose.yaml (old Prefect setup) - Deleted: backend/src/core/prefect_manager.py - Deleted: backend/src/services/prefect_stats_monitor.py - Deleted: Docker registry and insecure-registries requirement ## Workflows - Migrated: security_assessment workflow to Temporal - New: rust_test workflow (example/test workflow) - Deleted: secret_detection_scan (Prefect-based, to be reimplemented) - Activities now co-located with workflows for independent testing ## API Changes - Updated: backend/src/api/workflows.py (Temporal submission) - Updated: backend/src/api/runs.py (Temporal status/results) - Updated: backend/src/main.py (727 lines, TemporalManager integration) - Updated: All 16 MCP tools to use TemporalManager ## Testing - ✅ All services healthy (Temporal, PostgreSQL, MinIO, workers, backend) - ✅ All API endpoints functional - ✅ End-to-end workflow test passed (72 findings from vulnerable_app) - ✅ MinIO storage integration working (target upload/download, results) - ✅ Worker activity discovery working (6 activities registered) - ✅ Tarball extraction working - ✅ SARIF report generation working ## Documentation - ARCHITECTURE.md: Complete Temporal architecture documentation - QUICKSTART_TEMPORAL.md: Getting started guide - MIGRATION_DECISION.md: Why we chose Temporal over Prefect - IMPLEMENTATION_STATUS.md: Migration progress tracking - workers/README.md: Worker development guide ## Dependencies - Added: temporalio>=1.6.0 - Added: boto3>=1.34.0 (MinIO S3 client) - Removed: prefect>=3.4.18 * feat: Add Python fuzzing vertical with Atheris integration This commit implements a complete Python fuzzing workflow using Atheris: ## Python Worker (workers/python/) - Dockerfile with Python 3.11, Atheris, and build tools - Generic worker.py for dynamic workflow discovery - requirements.txt with temporalio, boto3, atheris dependencies - Added to docker-compose.temporal.yaml with dedicated cache volume ## AtherisFuzzer Module (backend/toolbox/modules/fuzzer/) - Reusable module extending BaseModule - Auto-discovers fuzz targets (fuzz_*.py, *_fuzz.py, fuzz_target.py) - Recursive search to find targets in nested directories - Dynamically loads TestOneInput() function - Configurable max_iterations and timeout - Real-time stats callback support for live monitoring - Returns findings as ModuleFinding objects ## Atheris Fuzzing Workflow (backend/toolbox/workflows/atheris_fuzzing/) - Temporal workflow for orchestrating fuzzing - Downloads user code from MinIO - Executes AtherisFuzzer module - Uploads results to MinIO - Cleans up cache after execution - metadata.yaml with vertical: python for routing ## Test Project (test_projects/python_fuzz_waterfall/) - Demonstrates stateful waterfall vulnerability - main.py with check_secret() that leaks progress - fuzz_target.py with Atheris TestOneInput() harness - Complete README with usage instructions ## Backend Fixes - Fixed parameter merging in REST API endpoints (workflows.py) - Changed workflow parameter passing from positional args to kwargs (manager.py) - Default parameters now properly merged with user parameters ## Testing ✅ Worker discovered AtherisFuzzingWorkflow ✅ Workflow executed end-to-end successfully ✅ Fuzz target auto-discovered in nested directories ✅ Atheris ran 100,000 iterations ✅ Results uploaded and cache cleaned * chore: Complete Temporal migration with updated CLI/SDK/docs This commit includes all remaining Temporal migration changes: ## CLI Updates (cli/) - Updated workflow execution commands for Temporal - Enhanced error handling and exceptions - Updated dependencies in uv.lock ## SDK Updates (sdk/) - Client methods updated for Temporal workflows - Updated models for new workflow execution - Updated dependencies in uv.lock ## Documentation Updates (docs/) - Architecture documentation for Temporal - Workflow concept documentation - Resource management documentation (new) - Debugging guide (new) - Updated tutorials and how-to guides - Troubleshooting updates ## README Updates - Main README with Temporal instructions - Backend README - CLI README - SDK README ## Other - Updated IMPLEMENTATION_STATUS.md - Removed old vulnerable_app.tar.gz These changes complete the Temporal migration and ensure the CLI/SDK work correctly with the new backend. * fix: Use positional args instead of kwargs for Temporal workflows The Temporal Python SDK's start_workflow() method doesn't accept a 'kwargs' parameter. Workflows must receive parameters as positional arguments via the 'args' parameter. Changed from: args=workflow_args # Positional arguments This fixes the error: TypeError: Client.start_workflow() got an unexpected keyword argument 'kwargs' Workflows now correctly receive parameters in order: - security_assessment: [target_id, scanner_config, analyzer_config, reporter_config] - atheris_fuzzing: [target_id, target_file, max_iterations, timeout_seconds] - rust_test: [target_id, test_message] * fix: Filter metadata-only parameters from workflow arguments SecurityAssessmentWorkflow was receiving 7 arguments instead of 2-5. The issue was that target_path and volume_mode from default_parameters were being passed to the workflow, when they should only be used by the system for configuration. Now filters out metadata-only parameters (target_path, volume_mode) before passing arguments to workflow execution. * refactor: Remove Prefect leftovers and volume mounting legacy Complete cleanup of Prefect migration artifacts: Backend: - Delete registry.py and workflow_discovery.py (Prefect-specific files) - Remove Docker validation from setup.py (no longer needed) - Remove ResourceLimits and VolumeMount models - Remove target_path and volume_mode from WorkflowSubmission - Remove supported_volume_modes from API and discovery - Clean up metadata.yaml files (remove volume/path fields) - Simplify parameter filtering in manager.py SDK: - Remove volume_mode parameter from client methods - Remove ResourceLimits and VolumeMount models - Remove Prefect error patterns from docker_logs.py - Clean up WorkflowSubmission and WorkflowMetadata models CLI: - Remove Volume Modes display from workflow info All removed features are Prefect-specific or Docker volume mounting artifacts. Temporal workflows use MinIO storage exclusively. * feat: Add comprehensive test suite and benchmark infrastructure - Add 68 unit tests for fuzzer, scanner, and analyzer modules - Implement pytest-based test infrastructure with fixtures - Add 6 performance benchmarks with category-specific thresholds - Configure GitHub Actions for automated testing and benchmarking - Add test and benchmark documentation Test coverage: - AtherisFuzzer: 8 tests - CargoFuzzer: 14 tests - FileScanner: 22 tests - SecurityAnalyzer: 24 tests All tests passing (68/68) All benchmarks passing (6/6) * fix: Resolve all ruff linting violations across codebase Fixed 27 ruff violations in 12 files: - Removed unused imports (Depends, Dict, Any, Optional, etc.) - Fixed undefined workflow_info variable in workflows.py - Removed dead code with undefined variables in atheris_fuzzer.py - Changed f-string to regular string where no placeholders used All files now pass ruff checks for CI/CD compliance. * fix: Configure CI for unit tests only - Renamed docker-compose.temporal.yaml → docker-compose.yml for CI compatibility - Commented out integration-tests job (no integration tests yet) - Updated test-summary to only depend on lint and unit-tests CI will now run successfully with 68 unit tests. Integration tests can be added later. * feat: Add CI/CD integration with ephemeral deployment model Implements comprehensive CI/CD support for FuzzForge with on-demand worker management: **Worker Management (v0.7.0)** - Add WorkerManager for automatic worker lifecycle control - Auto-start workers from stopped state when workflows execute - Auto-stop workers after workflow completion - Health checks and startup timeout handling (90s default) **CI/CD Features** - `--fail-on` flag: Fail builds based on SARIF severity levels (error/warning/note/info) - `--export-sarif` flag: Export findings in SARIF 2.1.0 format - `--auto-start`/`--auto-stop` flags: Control worker lifecycle - Exit code propagation: Returns 1 on blocking findings, 0 on success **Exit Code Fix** - Add `except typer.Exit: raise` handlers at 3 critical locations - Move worker cleanup to finally block for guaranteed execution - Exit codes now propagate correctly even when build fails **CI Scripts & Examples** - ci-start.sh: Start FuzzForge services with health checks - ci-stop.sh: Clean shutdown with volume preservation option - GitHub Actions workflow example (security-scan.yml) - GitLab CI pipeline example (.gitlab-ci.example.yml) - docker-compose.ci.yml: CI-optimized compose file with profiles **OSS-Fuzz Integration** - New ossfuzz_campaign workflow for running OSS-Fuzz projects - OSS-Fuzz worker with Docker-in-Docker support - Configurable campaign duration and project selection **Documentation** - Comprehensive CI/CD integration guide (docs/how-to/cicd-integration.md) - Updated architecture docs with worker lifecycle details - Updated workspace isolation documentation - CLI README with worker management examples **SDK Enhancements** - Add get_workflow_worker_info() endpoint - Worker vertical metadata in workflow responses **Testing** - All workflows tested: security_assessment, atheris_fuzzing, secret_detection, cargo_fuzzing - All monitoring commands tested: stats, crashes, status, finding - Full CI pipeline simulation verified - Exit codes verified for success/failure scenarios Ephemeral CI/CD model: ~3-4GB RAM, ~60-90s startup, runs entirely in CI containers. * fix: Resolve ruff linting violations in CI/CD code - Remove unused variables (run_id, defaults, result) - Remove unused imports - Fix f-string without placeholders All CI/CD integration files now pass ruff checks.
This commit is contained in:
tduhamel42
@@ -0,0 +1,184 @@
|
||||
# FuzzForge Benchmark Suite
|
||||
|
||||
Performance benchmarking infrastructure organized by module category.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
benchmarks/
|
||||
├── conftest.py # Benchmark fixtures
|
||||
├── category_configs.py # Category-specific thresholds
|
||||
├── by_category/ # Benchmarks organized by category
|
||||
│ ├── fuzzer/
|
||||
│ │ ├── bench_cargo_fuzz.py
|
||||
│ │ └── bench_atheris.py
|
||||
│ ├── scanner/
|
||||
│ │ └── bench_file_scanner.py
|
||||
│ ├── secret_detection/
|
||||
│ │ ├── bench_gitleaks.py
|
||||
│ │ └── bench_trufflehog.py
|
||||
│ └── analyzer/
|
||||
│ └── bench_security_analyzer.py
|
||||
├── fixtures/ # Benchmark test data
|
||||
│ ├── small/ # ~1K LOC
|
||||
│ ├── medium/ # ~10K LOC
|
||||
│ └── large/ # ~100K LOC
|
||||
└── results/ # Benchmark results (JSON)
|
||||
```
|
||||
|
||||
## Module Categories
|
||||
|
||||
### Fuzzer
|
||||
**Expected Metrics**: execs/sec, coverage_rate, time_to_crash, memory_usage
|
||||
|
||||
**Performance Thresholds**:
|
||||
- Min 1000 execs/sec
|
||||
- Max 10s for small projects
|
||||
- Max 2GB memory
|
||||
|
||||
### Scanner
|
||||
**Expected Metrics**: files/sec, LOC/sec, findings_count
|
||||
|
||||
**Performance Thresholds**:
|
||||
- Min 100 files/sec
|
||||
- Min 10K LOC/sec
|
||||
- Max 512MB memory
|
||||
|
||||
### Secret Detection
|
||||
**Expected Metrics**: patterns/sec, precision, recall, F1
|
||||
|
||||
**Performance Thresholds**:
|
||||
- Min 90% precision
|
||||
- Min 95% recall
|
||||
- Max 5 false positives per 100 secrets
|
||||
|
||||
### Analyzer
|
||||
**Expected Metrics**: analysis_depth, files/sec, accuracy
|
||||
|
||||
**Performance Thresholds**:
|
||||
- Min 10 files/sec (deep analysis)
|
||||
- Min 85% accuracy
|
||||
- Max 2GB memory
|
||||
|
||||
## Running Benchmarks
|
||||
|
||||
### All Benchmarks
|
||||
```bash
|
||||
cd backend
|
||||
pytest benchmarks/ --benchmark-only -v
|
||||
```
|
||||
|
||||
### Specific Category
|
||||
```bash
|
||||
pytest benchmarks/by_category/fuzzer/ --benchmark-only -v
|
||||
```
|
||||
|
||||
### With Comparison
|
||||
```bash
|
||||
# Run and save baseline
|
||||
pytest benchmarks/ --benchmark-only --benchmark-save=baseline
|
||||
|
||||
# Compare against baseline
|
||||
pytest benchmarks/ --benchmark-only --benchmark-compare=baseline
|
||||
```
|
||||
|
||||
### Generate Histogram
|
||||
```bash
|
||||
pytest benchmarks/ --benchmark-only --benchmark-histogram=histogram
|
||||
```
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
Results are saved as JSON and include:
|
||||
- Mean execution time
|
||||
- Standard deviation
|
||||
- Min/Max values
|
||||
- Iterations per second
|
||||
- Memory usage
|
||||
|
||||
Example output:
|
||||
```
|
||||
------------------------ benchmark: fuzzer --------------------------
|
||||
Name Mean StdDev Ops/Sec
|
||||
bench_cargo_fuzz[discovery] 0.0012s 0.0001s 833.33
|
||||
bench_cargo_fuzz[execution] 0.1250s 0.0050s 8.00
|
||||
bench_cargo_fuzz[memory] 0.0100s 0.0005s 100.00
|
||||
---------------------------------------------------------------------
|
||||
```
|
||||
|
||||
## CI/CD Integration
|
||||
|
||||
Benchmarks run:
|
||||
- **Nightly**: Full benchmark suite, track trends
|
||||
- **On PR**: When benchmarks/ or modules/ changed
|
||||
- **Manual**: Via workflow_dispatch
|
||||
|
||||
### Regression Detection
|
||||
|
||||
Benchmarks automatically fail if:
|
||||
- Performance degrades >10%
|
||||
- Memory usage exceeds thresholds
|
||||
- Throughput drops below minimum
|
||||
|
||||
See `.github/workflows/benchmark.yml` for configuration.
|
||||
|
||||
## Adding New Benchmarks
|
||||
|
||||
### 1. Create benchmark file in category directory
|
||||
```python
|
||||
# benchmarks/by_category/fuzzer/bench_new_fuzzer.py
|
||||
|
||||
import pytest
|
||||
from benchmarks.category_configs import ModuleCategory, get_threshold
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_execution_performance(benchmark, new_fuzzer, test_workspace):
|
||||
"""Benchmark execution speed"""
|
||||
result = benchmark(new_fuzzer.execute, config, test_workspace)
|
||||
|
||||
# Validate against threshold
|
||||
threshold = get_threshold(ModuleCategory.FUZZER, "max_execution_time_small")
|
||||
assert result.execution_time < threshold
|
||||
```
|
||||
|
||||
### 2. Update category_configs.py if needed
|
||||
Add new thresholds or metrics for your module.
|
||||
|
||||
### 3. Run locally
|
||||
```bash
|
||||
pytest benchmarks/by_category/fuzzer/bench_new_fuzzer.py --benchmark-only -v
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use mocking** for external dependencies (network, disk I/O)
|
||||
2. **Fixed iterations** for consistent benchmarking
|
||||
3. **Warm-up runs** for JIT-compiled code
|
||||
4. **Category-specific metrics** aligned with module purpose
|
||||
5. **Realistic fixtures** that represent actual use cases
|
||||
6. **Memory profiling** using tracemalloc
|
||||
7. **Compare apples to apples** within the same category
|
||||
|
||||
## Interpreting Results
|
||||
|
||||
### Good Performance
|
||||
- ✅ Execution time below threshold
|
||||
- ✅ Memory usage within limits
|
||||
- ✅ Throughput meets minimum
|
||||
- ✅ <5% variance across runs
|
||||
|
||||
### Performance Issues
|
||||
- ⚠️ Execution time 10-20% over threshold
|
||||
- ❌ Execution time >20% over threshold
|
||||
- ❌ Memory leaks (increasing over iterations)
|
||||
- ❌ High variance (>10%) indicates instability
|
||||
|
||||
## Tracking Performance Over Time
|
||||
|
||||
Benchmark results are stored as artifacts with:
|
||||
- Commit SHA
|
||||
- Timestamp
|
||||
- Environment details (Python version, OS)
|
||||
- Full metrics
|
||||
|
||||
Use these to track long-term performance trends and detect gradual degradation.
|
||||
@@ -0,0 +1,221 @@
|
||||
"""
|
||||
Benchmarks for CargoFuzzer module
|
||||
|
||||
Tests performance characteristics of Rust fuzzing:
|
||||
- Execution throughput (execs/sec)
|
||||
- Coverage rate
|
||||
- Memory efficiency
|
||||
- Time to first crash
|
||||
"""
|
||||
|
||||
import pytest
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from unittest.mock import AsyncMock, patch
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[3] / "toolbox"))
|
||||
|
||||
from modules.fuzzer.cargo_fuzzer import CargoFuzzer
|
||||
from benchmarks.category_configs import ModuleCategory, get_threshold
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def cargo_fuzzer():
|
||||
"""Create CargoFuzzer instance for benchmarking"""
|
||||
return CargoFuzzer()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def benchmark_config():
|
||||
"""Benchmark-optimized configuration"""
|
||||
return {
|
||||
"target_name": None,
|
||||
"max_iterations": 10000, # Fixed iterations for consistent benchmarking
|
||||
"timeout_seconds": 30,
|
||||
"sanitizer": "address"
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_rust_workspace(tmp_path):
|
||||
"""Create a minimal Rust workspace for benchmarking"""
|
||||
workspace = tmp_path / "rust_project"
|
||||
workspace.mkdir()
|
||||
|
||||
# Cargo.toml
|
||||
(workspace / "Cargo.toml").write_text("""[package]
|
||||
name = "bench_project"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
""")
|
||||
|
||||
# src/lib.rs
|
||||
src = workspace / "src"
|
||||
src.mkdir()
|
||||
(src / "lib.rs").write_text("""
|
||||
pub fn benchmark_function(data: &[u8]) -> Vec<u8> {
|
||||
data.to_vec()
|
||||
}
|
||||
""")
|
||||
|
||||
# fuzz structure
|
||||
fuzz = workspace / "fuzz"
|
||||
fuzz.mkdir()
|
||||
(fuzz / "Cargo.toml").write_text("""[package]
|
||||
name = "bench_project-fuzz"
|
||||
version = "0.0.0"
|
||||
edition = "2021"
|
||||
|
||||
[dependencies]
|
||||
libfuzzer-sys = "0.4"
|
||||
|
||||
[dependencies.bench_project]
|
||||
path = ".."
|
||||
|
||||
[[bin]]
|
||||
name = "fuzz_target_1"
|
||||
path = "fuzz_targets/fuzz_target_1.rs"
|
||||
""")
|
||||
|
||||
targets = fuzz / "fuzz_targets"
|
||||
targets.mkdir()
|
||||
(targets / "fuzz_target_1.rs").write_text("""#![no_main]
|
||||
use libfuzzer_sys::fuzz_target;
|
||||
use bench_project::benchmark_function;
|
||||
|
||||
fuzz_target!(|data: &[u8]| {
|
||||
let _ = benchmark_function(data);
|
||||
});
|
||||
""")
|
||||
|
||||
return workspace
|
||||
|
||||
|
||||
class TestCargoFuzzerPerformance:
|
||||
"""Benchmark CargoFuzzer performance metrics"""
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_target_discovery_performance(self, benchmark, cargo_fuzzer, mock_rust_workspace):
|
||||
"""Benchmark fuzz target discovery speed"""
|
||||
def discover():
|
||||
return asyncio.run(cargo_fuzzer._discover_fuzz_targets(mock_rust_workspace))
|
||||
|
||||
result = benchmark(discover)
|
||||
assert len(result) > 0
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_config_validation_performance(self, benchmark, cargo_fuzzer, benchmark_config):
|
||||
"""Benchmark configuration validation speed"""
|
||||
result = benchmark(cargo_fuzzer.validate_config, benchmark_config)
|
||||
assert result is True
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_module_initialization_performance(self, benchmark):
|
||||
"""Benchmark module instantiation time"""
|
||||
def init_module():
|
||||
return CargoFuzzer()
|
||||
|
||||
module = benchmark(init_module)
|
||||
assert module is not None
|
||||
|
||||
|
||||
class TestCargoFuzzerThroughput:
|
||||
"""Benchmark execution throughput"""
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_execution_throughput(self, benchmark, cargo_fuzzer, mock_rust_workspace, benchmark_config):
|
||||
"""Benchmark fuzzing execution throughput"""
|
||||
|
||||
# Mock actual fuzzing to focus on orchestration overhead
|
||||
async def mock_run(workspace, target, config, callback):
|
||||
# Simulate 10K execs at 1000 execs/sec
|
||||
if callback:
|
||||
await callback({
|
||||
"total_execs": 10000,
|
||||
"execs_per_sec": 1000.0,
|
||||
"crashes": 0,
|
||||
"coverage": 50,
|
||||
"corpus_size": 10,
|
||||
"elapsed_time": 10
|
||||
})
|
||||
return [], {"total_executions": 10000, "execution_time": 10.0}
|
||||
|
||||
with patch.object(cargo_fuzzer, '_build_fuzz_target', new_callable=AsyncMock, return_value=True):
|
||||
with patch.object(cargo_fuzzer, '_run_fuzzing', side_effect=mock_run):
|
||||
with patch.object(cargo_fuzzer, '_parse_crash_artifacts', new_callable=AsyncMock, return_value=[]):
|
||||
def run_fuzzer():
|
||||
# Run in new event loop
|
||||
loop = asyncio.new_event_loop()
|
||||
try:
|
||||
return loop.run_until_complete(
|
||||
cargo_fuzzer.execute(benchmark_config, mock_rust_workspace)
|
||||
)
|
||||
finally:
|
||||
loop.close()
|
||||
|
||||
result = benchmark(run_fuzzer)
|
||||
assert result.status == "success"
|
||||
|
||||
# Verify performance threshold
|
||||
threshold = get_threshold(ModuleCategory.FUZZER, "max_execution_time_small")
|
||||
assert result.execution_time < threshold, \
|
||||
f"Execution time {result.execution_time}s exceeds threshold {threshold}s"
|
||||
|
||||
|
||||
class TestCargoFuzzerMemory:
|
||||
"""Benchmark memory efficiency"""
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_memory_overhead(self, benchmark, cargo_fuzzer, mock_rust_workspace, benchmark_config):
|
||||
"""Benchmark memory usage during execution"""
|
||||
import tracemalloc
|
||||
|
||||
def measure_memory():
|
||||
tracemalloc.start()
|
||||
|
||||
# Simulate operations
|
||||
cargo_fuzzer.validate_config(benchmark_config)
|
||||
asyncio.run(cargo_fuzzer._discover_fuzz_targets(mock_rust_workspace))
|
||||
|
||||
current, peak = tracemalloc.get_traced_memory()
|
||||
tracemalloc.stop()
|
||||
|
||||
return peak / 1024 / 1024 # Convert to MB
|
||||
|
||||
peak_mb = benchmark(measure_memory)
|
||||
|
||||
# Check against threshold
|
||||
max_memory = get_threshold(ModuleCategory.FUZZER, "max_memory_mb")
|
||||
assert peak_mb < max_memory, \
|
||||
f"Peak memory {peak_mb:.2f}MB exceeds threshold {max_memory}MB"
|
||||
|
||||
|
||||
class TestCargoFuzzerScalability:
|
||||
"""Benchmark scalability characteristics"""
|
||||
|
||||
@pytest.mark.benchmark(group="fuzzer")
|
||||
def test_multiple_target_discovery(self, benchmark, cargo_fuzzer, tmp_path):
|
||||
"""Benchmark discovery with multiple targets"""
|
||||
workspace = tmp_path / "multi_target"
|
||||
workspace.mkdir()
|
||||
|
||||
# Create workspace with 10 fuzz targets
|
||||
(workspace / "Cargo.toml").write_text("[package]\nname = \"test\"\nversion = \"0.1.0\"\nedition = \"2021\"")
|
||||
src = workspace / "src"
|
||||
src.mkdir()
|
||||
(src / "lib.rs").write_text("pub fn test() {}")
|
||||
|
||||
fuzz = workspace / "fuzz"
|
||||
fuzz.mkdir()
|
||||
targets = fuzz / "fuzz_targets"
|
||||
targets.mkdir()
|
||||
|
||||
for i in range(10):
|
||||
(targets / f"fuzz_target_{i}.rs").write_text("// Target")
|
||||
|
||||
def discover():
|
||||
return asyncio.run(cargo_fuzzer._discover_fuzz_targets(workspace))
|
||||
|
||||
result = benchmark(discover)
|
||||
assert len(result) == 10
|
||||
@@ -0,0 +1,151 @@
|
||||
"""
|
||||
Category-specific benchmark configurations
|
||||
|
||||
Defines expected metrics and performance thresholds for each module category.
|
||||
"""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import List, Dict
|
||||
from enum import Enum
|
||||
|
||||
|
||||
class ModuleCategory(str, Enum):
|
||||
"""Module categories for benchmarking"""
|
||||
FUZZER = "fuzzer"
|
||||
SCANNER = "scanner"
|
||||
ANALYZER = "analyzer"
|
||||
SECRET_DETECTION = "secret_detection"
|
||||
REPORTER = "reporter"
|
||||
|
||||
|
||||
@dataclass
|
||||
class CategoryBenchmarkConfig:
|
||||
"""Benchmark configuration for a module category"""
|
||||
category: ModuleCategory
|
||||
expected_metrics: List[str]
|
||||
performance_thresholds: Dict[str, float]
|
||||
description: str
|
||||
|
||||
|
||||
# Fuzzer category configuration
|
||||
FUZZER_CONFIG = CategoryBenchmarkConfig(
|
||||
category=ModuleCategory.FUZZER,
|
||||
expected_metrics=[
|
||||
"execs_per_sec",
|
||||
"coverage_rate",
|
||||
"time_to_first_crash",
|
||||
"corpus_efficiency",
|
||||
"execution_time",
|
||||
"peak_memory_mb"
|
||||
],
|
||||
performance_thresholds={
|
||||
"min_execs_per_sec": 1000, # Minimum executions per second
|
||||
"max_execution_time_small": 10.0, # Max time for small project (seconds)
|
||||
"max_execution_time_medium": 60.0, # Max time for medium project
|
||||
"max_memory_mb": 2048, # Maximum memory usage
|
||||
"min_coverage_rate": 1.0, # Minimum new coverage per second
|
||||
},
|
||||
description="Fuzzing modules: coverage-guided fuzz testing"
|
||||
)
|
||||
|
||||
# Scanner category configuration
|
||||
SCANNER_CONFIG = CategoryBenchmarkConfig(
|
||||
category=ModuleCategory.SCANNER,
|
||||
expected_metrics=[
|
||||
"files_per_sec",
|
||||
"loc_per_sec",
|
||||
"execution_time",
|
||||
"peak_memory_mb",
|
||||
"findings_count"
|
||||
],
|
||||
performance_thresholds={
|
||||
"min_files_per_sec": 100, # Minimum files scanned per second
|
||||
"min_loc_per_sec": 10000, # Minimum lines of code per second
|
||||
"max_execution_time_small": 1.0,
|
||||
"max_execution_time_medium": 10.0,
|
||||
"max_memory_mb": 512,
|
||||
},
|
||||
description="File scanning modules: fast pattern-based scanning"
|
||||
)
|
||||
|
||||
# Secret detection category configuration
|
||||
SECRET_DETECTION_CONFIG = CategoryBenchmarkConfig(
|
||||
category=ModuleCategory.SECRET_DETECTION,
|
||||
expected_metrics=[
|
||||
"patterns_per_sec",
|
||||
"precision",
|
||||
"recall",
|
||||
"f1_score",
|
||||
"false_positive_rate",
|
||||
"execution_time",
|
||||
"peak_memory_mb"
|
||||
],
|
||||
performance_thresholds={
|
||||
"min_patterns_per_sec": 1000,
|
||||
"min_precision": 0.90, # 90% precision target
|
||||
"min_recall": 0.95, # 95% recall target
|
||||
"max_false_positives": 5, # Max false positives per 100 secrets
|
||||
"max_execution_time_small": 2.0,
|
||||
"max_execution_time_medium": 20.0,
|
||||
"max_memory_mb": 1024,
|
||||
},
|
||||
description="Secret detection modules: high precision pattern matching"
|
||||
)
|
||||
|
||||
# Analyzer category configuration
|
||||
ANALYZER_CONFIG = CategoryBenchmarkConfig(
|
||||
category=ModuleCategory.ANALYZER,
|
||||
expected_metrics=[
|
||||
"analysis_depth",
|
||||
"files_analyzed_per_sec",
|
||||
"execution_time",
|
||||
"peak_memory_mb",
|
||||
"findings_count",
|
||||
"accuracy"
|
||||
],
|
||||
performance_thresholds={
|
||||
"min_files_per_sec": 10, # Slower than scanners due to deep analysis
|
||||
"max_execution_time_small": 5.0,
|
||||
"max_execution_time_medium": 60.0,
|
||||
"max_memory_mb": 2048,
|
||||
"min_accuracy": 0.85, # 85% accuracy target
|
||||
},
|
||||
description="Code analysis modules: deep semantic analysis"
|
||||
)
|
||||
|
||||
# Reporter category configuration
|
||||
REPORTER_CONFIG = CategoryBenchmarkConfig(
|
||||
category=ModuleCategory.REPORTER,
|
||||
expected_metrics=[
|
||||
"report_generation_time",
|
||||
"findings_per_sec",
|
||||
"peak_memory_mb"
|
||||
],
|
||||
performance_thresholds={
|
||||
"max_report_time_100_findings": 1.0, # Max 1 second for 100 findings
|
||||
"max_report_time_1000_findings": 10.0, # Max 10 seconds for 1000 findings
|
||||
"max_memory_mb": 256,
|
||||
},
|
||||
description="Reporting modules: fast report generation"
|
||||
)
|
||||
|
||||
|
||||
# Category configurations map
|
||||
CATEGORY_CONFIGS = {
|
||||
ModuleCategory.FUZZER: FUZZER_CONFIG,
|
||||
ModuleCategory.SCANNER: SCANNER_CONFIG,
|
||||
ModuleCategory.SECRET_DETECTION: SECRET_DETECTION_CONFIG,
|
||||
ModuleCategory.ANALYZER: ANALYZER_CONFIG,
|
||||
ModuleCategory.REPORTER: REPORTER_CONFIG,
|
||||
}
|
||||
|
||||
|
||||
def get_category_config(category: ModuleCategory) -> CategoryBenchmarkConfig:
|
||||
"""Get benchmark configuration for a category"""
|
||||
return CATEGORY_CONFIGS[category]
|
||||
|
||||
|
||||
def get_threshold(category: ModuleCategory, metric: str) -> float:
|
||||
"""Get performance threshold for a specific metric"""
|
||||
config = get_category_config(category)
|
||||
return config.performance_thresholds.get(metric, 0.0)
|
||||
@@ -0,0 +1,60 @@
|
||||
"""
|
||||
Benchmark fixtures and configuration
|
||||
"""
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
import pytest
|
||||
|
||||
# Add parent directories to path
|
||||
BACKEND_ROOT = Path(__file__).resolve().parents[1]
|
||||
TOOLBOX = BACKEND_ROOT / "toolbox"
|
||||
|
||||
if str(BACKEND_ROOT) not in sys.path:
|
||||
sys.path.insert(0, str(BACKEND_ROOT))
|
||||
if str(TOOLBOX) not in sys.path:
|
||||
sys.path.insert(0, str(TOOLBOX))
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# Benchmark Fixtures
|
||||
# ============================================================================
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def benchmark_fixtures_dir():
|
||||
"""Path to benchmark fixtures directory"""
|
||||
return Path(__file__).parent / "fixtures"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def small_project_fixture(benchmark_fixtures_dir):
|
||||
"""Small project fixture (~1K LOC)"""
|
||||
return benchmark_fixtures_dir / "small"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def medium_project_fixture(benchmark_fixtures_dir):
|
||||
"""Medium project fixture (~10K LOC)"""
|
||||
return benchmark_fixtures_dir / "medium"
|
||||
|
||||
|
||||
@pytest.fixture(scope="session")
|
||||
def large_project_fixture(benchmark_fixtures_dir):
|
||||
"""Large project fixture (~100K LOC)"""
|
||||
return benchmark_fixtures_dir / "large"
|
||||
|
||||
|
||||
# ============================================================================
|
||||
# pytest-benchmark Configuration
|
||||
# ============================================================================
|
||||
|
||||
def pytest_configure(config):
|
||||
"""Configure pytest-benchmark"""
|
||||
config.addinivalue_line(
|
||||
"markers", "benchmark: mark test as a benchmark"
|
||||
)
|
||||
|
||||
|
||||
def pytest_benchmark_group_stats(config, benchmarks, group_by):
|
||||
"""Group benchmark results by category"""
|
||||
return group_by
|
||||
Reference in New Issue
Block a user