CalvinBackup/fuzzforge_ai
1
0
Fork 0
mirror of https://github.com/FuzzingLabs/fuzzforge_ai.git synced 2026-07-05 20:17:52 +02:00
Code Issues Packages Projects Releases Wiki Activity

chore: Complete Temporal migration with updated CLI/SDK/docs

Browse Source 
This commit includes all remaining Temporal migration changes:

## CLI Updates (cli/)
- Updated workflow execution commands for Temporal
- Enhanced error handling and exceptions
- Updated dependencies in uv.lock

## SDK Updates (sdk/)
- Client methods updated for Temporal workflows
- Updated models for new workflow execution
- Updated dependencies in uv.lock

## Documentation Updates (docs/)
- Architecture documentation for Temporal
- Workflow concept documentation
- Resource management documentation (new)
- Debugging guide (new)
- Updated tutorials and how-to guides
- Troubleshooting updates

## README Updates
- Main README with Temporal instructions
- Backend README
- CLI README
- SDK README

## Other
- Updated IMPLEMENTATION_STATUS.md
- Removed old vulnerable_app.tar.gz

These changes complete the Temporal migration and ensure the
CLI/SDK work correctly with the new backend.
This commit is contained in:
Tanguy Duhamel
2025-10-02 11:26:32 +02:00
parent fe50d4ef72
commit 8e0e167ddd
21 changed files with 2159 additions and 459 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sdk/README.md
+213 -5
View File
@@ -5,6 +5,7 @@ A comprehensive Python SDK for the FuzzForge security testing workflow orchestra
## Features
- **Complete API Coverage**: All FuzzForge API endpoints supported
- **File Upload**: Automatic tarball creation and multipart upload for local files
- **Async & Sync**: Both synchronous and asynchronous client methods
- **Real-time Monitoring**: WebSocket and Server-Sent Events for live fuzzing updates
- **Type Safety**: Full Pydantic model validation for all data structures
@@ -27,9 +28,11 @@ pip install fuzzforge-sdk
## Quick Start
### Method 1: File Upload (Recommended)
```python
from fuzzforge_sdk import FuzzForgeClient
from fuzzforge_sdk.utils import create_workflow_submission
from pathlib import Path
# Initialize client
client = FuzzForgeClient(base_url="http://localhost:8000")
@@ -37,14 +40,20 @@ client = FuzzForgeClient(base_url="http://localhost:8000")
# List available workflows
workflows = client.list_workflows()
# Submit a workflow
submission = create_workflow_submission(
target_path="/path/to/your/project",
# Submit a workflow with automatic file upload
target_path = Path("/path/to/your/project")
response = client.submit_workflow_with_upload(
workflow_name="security_assessment",
target_path=target_path,
volume_mode="ro",
timeout=300
)
response = client.submit_workflow("static-analysis", submission)
# The SDK automatically:
# - Creates a tarball if target_path is a directory
# - Uploads the file to the backend via HTTP
# - Backend stores it in MinIO
# - Returns the workflow run_id
# Wait for completion and get results
final_status = client.wait_for_completion(response.run_id)
@@ -53,6 +62,27 @@ findings = client.get_run_findings(response.run_id)
client.close()
```
### Method 2: Path-Based Submission (Legacy)
```python
from fuzzforge_sdk import FuzzForgeClient
from fuzzforge_sdk.utils import create_workflow_submission
# Initialize client
client = FuzzForgeClient(base_url="http://localhost:8000")
# Submit a workflow with path (only works if backend can access the path)
submission = create_workflow_submission(
target_path="/path/on/backend/filesystem",
volume_mode="ro",
timeout=300
)
response = client.submit_workflow("security_assessment", submission)
client.close()
```
## Examples
The `examples/` directory contains complete working examples:
@@ -61,6 +91,184 @@ The `examples/` directory contains complete working examples:
- **`fuzzing_monitor.py`**: Real-time fuzzing monitoring with WebSocket/SSE
- **`batch_analysis.py`**: Batch analysis of multiple projects
## File Upload API Reference
### `submit_workflow_with_upload()`
Submit a workflow with automatic file upload from local filesystem.
```python
def submit_workflow_with_upload(
self,
workflow_name: str,
target_path: Union[str, Path],
parameters: Optional[Dict[str, Any]] = None,
volume_mode: str = "ro",
timeout: Optional[int] = None,
progress_callback: Optional[Callable[[int, int], None]] = None
) -> RunSubmissionResponse:
"""
Submit workflow with file upload.
Args:
workflow_name: Name of the workflow to execute
target_path: Path to file or directory to upload
parameters: Optional workflow parameters
volume_mode: Volume mount mode ('ro' or 'rw')
timeout: Optional execution timeout in seconds
progress_callback: Optional callback(bytes_sent, total_bytes)
Returns:
RunSubmissionResponse with run_id and status
Raises:
FileNotFoundError: If target_path doesn't exist
ValidationError: If parameters are invalid
FuzzForgeHTTPError: If upload fails
"""
```
**Example with progress tracking:**
```python
from fuzzforge_sdk import FuzzForgeClient
from pathlib import Path
def upload_progress(bytes_sent, total_bytes):
pct = (bytes_sent / total_bytes) * 100
print(f"Upload progress: {pct:.1f}% ({bytes_sent}/{total_bytes} bytes)")
client = FuzzForgeClient(base_url="http://localhost:8000")
response = client.submit_workflow_with_upload(
workflow_name="security_assessment",
target_path=Path("./my-project"),
parameters={"check_secrets": True},
volume_mode="ro",
progress_callback=upload_progress
)
print(f"Workflow started: {response.run_id}")
```
### `asubmit_workflow_with_upload()`
Async version of `submit_workflow_with_upload()`.
```python
import asyncio
from fuzzforge_sdk import FuzzForgeClient
async def main():
client = FuzzForgeClient(base_url="http://localhost:8000")
response = await client.asubmit_workflow_with_upload(
workflow_name="security_assessment",
target_path="/path/to/project",
parameters={"timeout": 3600}
)
print(f"Workflow started: {response.run_id}")
await client.aclose()
asyncio.run(main())
```
### Internal: `_create_tarball()`
Creates a compressed tarball from a file or directory.
```python
def _create_tarball(
self,
source_path: Path,
progress_callback: Optional[Callable[[int], None]] = None
) -> Path:
"""
Create compressed tarball (.tar.gz) from source.
Args:
source_path: Path to file or directory
progress_callback: Optional callback(files_added)
Returns:
Path to created tarball in temp directory
Note:
Caller is responsible for cleaning up the tarball
"""
```
**How it works:**
1. **Directory**: Creates tarball with all files, preserving structure
```python
# For directory: /path/to/project/
# Creates: /tmp/tmpXXXXXX.tar.gz containing:
# project/file1.py
# project/subdir/file2.py
```
2. **Single file**: Creates tarball with just that file
```python
# For file: /path/to/binary.elf
# Creates: /tmp/tmpXXXXXX.tar.gz containing:
# binary.elf
```
### Upload Flow Diagram
```
User Code
↓
submit_workflow_with_upload()
↓
_create_tarball() ───→ Compress files
↓
HTTP POST multipart/form-data
↓
Backend API (/workflows/{name}/upload-and-submit)
↓
MinIO Storage (S3) ───→ Store with target_id
↓
Temporal Workflow
↓
Worker downloads from MinIO
↓
Workflow execution
```
### Error Handling
The SDK provides detailed error context:
```python
from fuzzforge_sdk import FuzzForgeClient
from fuzzforge_sdk.exceptions import (
FuzzForgeHTTPError,
ValidationError,
ConnectionError
)
client = FuzzForgeClient(base_url="http://localhost:8000")
try:
response = client.submit_workflow_with_upload(
workflow_name="security_assessment",
target_path="./nonexistent",
)
except FileNotFoundError as e:
print(f"Target not found: {e}")
except ValidationError as e:
print(f"Invalid parameters: {e}")
except FuzzForgeHTTPError as e:
print(f"Upload failed (HTTP {e.status_code}): {e.message}")
if e.context.response_data:
print(f"Server response: {e.context.response_data}")
except ConnectionError as e:
print(f"Cannot connect to backend: {e}")
```
## Development
Install with development dependencies:
sdk/src/fuzzforge_sdk/client.py
+236 -1
View File
@@ -19,7 +19,10 @@ including real-time monitoring capabilities for fuzzing workflows.
import asyncio
import json
import logging
from typing import Dict, Any, List, Optional, AsyncIterator, Iterator, Union
import tarfile
import tempfile
from pathlib import Path
from typing import Dict, Any, List, Optional, AsyncIterator, Iterator, Union, Callable
from urllib.parse import urljoin, urlparse
import warnings
@@ -235,6 +238,238 @@ class FuzzForgeClient:
data = await self._ahandle_response(response)
return RunSubmissionResponse(**data)
def _create_tarball(
self,
source_path: Path,
progress_callback: Optional[Callable[[int], None]] = None
) -> Path:
"""
Create a compressed tarball from a file or directory.
Args:
source_path: Path to file or directory to archive
progress_callback: Optional callback(bytes_written) for progress tracking
Returns:
Path to the created tarball
Raises:
FileNotFoundError: If source_path doesn't exist
"""
if not source_path.exists():
raise FileNotFoundError(f"Source path not found: {source_path}")
# Create temp file for tarball
temp_fd, temp_path = tempfile.mkstemp(suffix=".tar.gz")
try:
logger.info(f"Creating tarball from {source_path}")
bytes_written = 0
with tarfile.open(temp_path, "w:gz") as tar:
if source_path.is_file():
# Add single file
tar.add(source_path, arcname=source_path.name)
bytes_written = source_path.stat().st_size
if progress_callback:
progress_callback(bytes_written)
else:
# Add directory recursively
for item in source_path.rglob("*"):
if item.is_file():
arcname = item.relative_to(source_path.parent)
tar.add(item, arcname=arcname)
bytes_written += item.stat().st_size
if progress_callback:
progress_callback(bytes_written)
tarball_path = Path(temp_path)
tarball_size = tarball_path.stat().st_size
logger.info(
f"Created tarball: {tarball_size / (1024**2):.2f} MB "
f"(compressed from {bytes_written / (1024**2):.2f} MB)"
)
return tarball_path
except Exception as e:
# Cleanup on error
if Path(temp_path).exists():
Path(temp_path).unlink()
raise
def submit_workflow_with_upload(
self,
workflow_name: str,
target_path: Union[str, Path],
parameters: Optional[Dict[str, Any]] = None,
volume_mode: str = "ro",
timeout: Optional[int] = None,
progress_callback: Optional[Callable[[int, int], None]] = None
) -> RunSubmissionResponse:
"""
Submit a workflow with file upload from local filesystem.
This method automatically creates a tarball if target_path is a directory,
uploads it to the backend, and submits the workflow for execution.
Args:
workflow_name: Name of the workflow to execute
target_path: Local path to file or directory to analyze
parameters: Workflow-specific parameters
volume_mode: Volume mount mode ("ro" or "rw")
timeout: Timeout in seconds
progress_callback: Optional callback(bytes_uploaded, total_bytes) for progress
Returns:
Run submission response with run_id
Raises:
FileNotFoundError: If target_path doesn't exist
FuzzForgeHTTPError: For API errors
"""
target_path = Path(target_path)
tarball_path = None
try:
# Create tarball if needed
if target_path.is_dir():
logger.info(f"Target is directory, creating tarball...")
tarball_path = self._create_tarball(target_path)
upload_file = tarball_path
filename = f"{target_path.name}.tar.gz"
else:
upload_file = target_path
filename = target_path.name
# Prepare multipart form data
url = urljoin(self.base_url, f"/workflows/{workflow_name}/upload-and-submit")
files = {
"file": (filename, open(upload_file, "rb"), "application/gzip")
}
data = {
"volume_mode": volume_mode
}
if parameters:
data["parameters"] = json.dumps(parameters)
if timeout:
data["timeout"] = str(timeout)
logger.info(f"Uploading {filename} to {workflow_name}...")
# Track upload progress
if progress_callback:
file_size = upload_file.stat().st_size
def track_progress(monitor):
progress_callback(monitor.bytes_read, file_size)
# Note: httpx doesn't have built-in progress tracking for uploads
# This is a placeholder - real implementation would need custom approach
pass
response = self._client.post(url, files=files, data=data)
# Close file handle
files["file"][1].close()
data = self._handle_response(response)
return RunSubmissionResponse(**data)
finally:
# Cleanup temporary tarball
if tarball_path and tarball_path.exists():
try:
tarball_path.unlink()
logger.debug(f"Cleaned up temporary tarball: {tarball_path}")
except Exception as e:
logger.warning(f"Failed to cleanup tarball {tarball_path}: {e}")
async def asubmit_workflow_with_upload(
self,
workflow_name: str,
target_path: Union[str, Path],
parameters: Optional[Dict[str, Any]] = None,
volume_mode: str = "ro",
timeout: Optional[int] = None,
progress_callback: Optional[Callable[[int, int], None]] = None
) -> RunSubmissionResponse:
"""
Submit a workflow with file upload from local filesystem (async).
This method automatically creates a tarball if target_path is a directory,
uploads it to the backend, and submits the workflow for execution.
Args:
workflow_name: Name of the workflow to execute
target_path: Local path to file or directory to analyze
parameters: Workflow-specific parameters
volume_mode: Volume mount mode ("ro" or "rw")
timeout: Timeout in seconds
progress_callback: Optional callback(bytes_uploaded, total_bytes) for progress
Returns:
Run submission response with run_id
Raises:
FileNotFoundError: If target_path doesn't exist
FuzzForgeHTTPError: For API errors
"""
target_path = Path(target_path)
tarball_path = None
try:
# Create tarball if needed
if target_path.is_dir():
logger.info(f"Target is directory, creating tarball...")
tarball_path = self._create_tarball(target_path)
upload_file = tarball_path
filename = f"{target_path.name}.tar.gz"
else:
upload_file = target_path
filename = target_path.name
# Prepare multipart form data
url = urljoin(self.base_url, f"/workflows/{workflow_name}/upload-and-submit")
files = {
"file": (filename, open(upload_file, "rb"), "application/gzip")
}
data = {
"volume_mode": volume_mode
}
if parameters:
data["parameters"] = json.dumps(parameters)
if timeout:
data["timeout"] = str(timeout)
logger.info(f"Uploading {filename} to {workflow_name}...")
response = await self._async_client.post(url, files=files, data=data)
# Close file handle
files["file"][1].close()
response_data = await self._ahandle_response(response)
return RunSubmissionResponse(**response_data)
finally:
# Cleanup temporary tarball
if tarball_path and tarball_path.exists():
try:
tarball_path.unlink()
logger.debug(f"Cleaned up temporary tarball: {tarball_path}")
except Exception as e:
logger.warning(f"Failed to cleanup tarball {tarball_path}: {e}")
# Run management methods
def get_run_status(self, run_id: str) -> WorkflowStatus:
sdk/src/fuzzforge_sdk/models.py
+3 -2
View File
@@ -124,9 +124,10 @@ class WorkflowMetadata(BaseModel):
class WorkflowParametersResponse(BaseModel):
"""Response for workflow parameters endpoint"""
workflow: str = Field(..., description="Workflow name")
parameters: Dict[str, Any] = Field(..., description="Parameters schema")
defaults: Dict[str, Any] = Field(default_factory=dict, description="Default values")
required: List[str] = Field(default_factory=list, description="Required parameter names")
default_parameters: Dict[str, Any] = Field(default_factory=dict, description="Default parameter values")
required_parameters: List[str] = Field(default_factory=list, description="Required parameter names")
class RunSubmissionResponse(BaseModel):
sdk/uv.lock
Generated
+1 -1
View File
@@ -85,7 +85,7 @@ wheels = [
[[package]]
name = "fuzzforge-sdk"
version = "0.1.0"
version = "0.6.0"
source = { editable = "." }
dependencies = [
{ name = "httpx" },