CalvinBackup/shannon
1
0
Fork 0
mirror of https://github.com/KeygraphHQ/shannon.git synced 2026-06-06 15:33:57 +02:00
Code Issues Packages Projects Releases Wiki Activity

docs: update README and CLAUDE.md for Temporal workflow usage

Browse Source 
- Replace Docker CLI instructions with Task-based commands
- Add monitoring/stopping sections and workflow examples
- Document Temporal orchestration layer and troubleshooting
- Simplify file structure to key files overview
This commit is contained in:
ajmallesh
2026-01-12 17:23:25 -08:00
parent 4de1508cb8
commit 1f303b02b8
2 changed files with 183 additions and 224 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+100 -113
View File
@@ -8,51 +8,67 @@ This is an AI-powered penetration testing agent designed for defensive security
## Commands
### Installation & Setup
### Prerequisites
- **Docker** - Container runtime
- **Task** - Task runner ([Install Task](https://taskfile.dev/installation/))
- **Anthropic API key** - Set in `.env` file
### Running the Penetration Testing Agent (Docker + Temporal)
```bash
npm install
# Configure credentials
cp .env.example .env
# Edit .env:
# ANTHROPIC_API_KEY=your-key
# CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 # Prevents token limits during long reports
# Start a pentest workflow
task start URL=<url> REPO=<path>
```
### Running the Penetration Testing Agent
Examples:
```bash
shannon <WEB_URL> <REPO_PATH> [--config <CONFIG_FILE>] [--output <OUTPUT_DIR>]
task start URL=https://example.com REPO=/path/to/repo
task start URL=https://example.com REPO=/path/to/repo CONFIG=./configs/my-config.yaml
task start URL=https://example.com REPO=/path/to/repo OUTPUT=./my-reports
```
Example:
### Monitoring Progress
```bash
shannon "https://example.com" "/path/to/local/repo"
shannon "https://juice-shop.herokuapp.com" "/home/user/juice-shop" --config juice-shop-config.yaml
shannon "https://example.com" "/path/to/repo" --output /path/to/reports
task logs # View real-time worker logs
task query ID=<workflow-id> # Query specific workflow progress
# Temporal Web UI available at http://localhost:8233
```
### Alternative Execution
### Stopping Shannon
```bash
npm start <WEB_URL> <REPO_PATH> --config <CONFIG_FILE>
task stop # Stop containers (preserves workflow data)
task stop CLEAN=true # Full cleanup including volumes
```
### Options
```bash
--config <file> YAML configuration file for authentication and testing parameters
--output <path> Custom output directory for session folder (default: ./audit-logs/)
--pipeline-testing Use minimal prompts for fast pipeline testing (creates minimal deliverables)
--disable-loader Disable the animated progress loader (useful when logs interfere with spinner)
--help Show help message
CONFIG=<file> YAML configuration file for authentication and testing parameters
OUTPUT=<path> Custom output directory for session folder (default: ./audit-logs/)
PIPELINE_TESTING=true Use minimal prompts for fast pipeline testing
```
### Configuration Validation
### Direct CLI (Local Development)
For development without Docker/Temporal:
```bash
# Configuration validation is built into the main script
shannon --help # Shows usage and validates config on execution
npm install
shannon <WEB_URL> <REPO_PATH> [--config <CONFIG_FILE>] [--output <OUTPUT_DIR>]
```
### Generate TOTP for Authentication
TOTP generation is now handled automatically via the `generate_totp` MCP tool during authentication flows.
TOTP generation is handled automatically via the `generate_totp` MCP tool during authentication flows.
### Development Commands
```bash
# No linting or testing commands available in this project
# Development is done by running the agent in pipeline-testing mode
shannon <WEB_URL> <REPO_PATH> --pipeline-testing
# Build TypeScript
npm run build
# Run with pipeline testing mode (fast, minimal deliverables)
task start URL=<url> REPO=<path> PIPELINE_TESTING=true
```
## Architecture & Components
@@ -67,6 +83,21 @@ shannon <WEB_URL> <REPO_PATH> --pipeline-testing
- `src/session-manager.ts` - Agent definitions, execution order, and parallel groups
- `src/queue-validation.ts` - Validates deliverables and agent prerequisites
### Temporal Orchestration Layer
Shannon uses Temporal for durable workflow orchestration:
- `src/temporal/shared.ts` - Types, interfaces, query definitions
- `src/temporal/workflows.ts` - Main workflow (pentestPipelineWorkflow)
- `src/temporal/activities.ts` - Activity implementations with heartbeats
- `src/temporal/worker.ts` - Worker process entry point
- `src/temporal/client.ts` - CLI client for starting workflows
- `src/temporal/query.ts` - Query tool for progress inspection
Key features:
- **Crash recovery** - Workflows resume automatically after worker restart
- **Queryable progress** - Real-time status via `task query` or Temporal Web UI
- **Intelligent retry** - Distinguishes transient vs permanent errors
- **Parallel execution** - 5 concurrent agents in vulnerability/exploitation phases
### Five-Phase Testing Workflow
1. **Pre-Reconnaissance** (`pre-recon`) - External tool scans (nmap, subfinder, whatweb) + source code analysis
@@ -163,7 +194,6 @@ The agent implements a crash-safe audit system with the following features:
- Phase-level and agent-level timing/cost aggregations
- Validation results integrated with metrics
For detailed design, see `docs/unified-audit-system-design.md`.
## Development Notes
@@ -183,18 +213,25 @@ A working POC exists at `/Users/arjunmalleswaran/Code/shannon-pocs` that demonst
- "What retry configuration does the POC use for long-running agent activities?"
- "How does the POC integrate Claude Agent SDK calls within Temporal activities?"
**Current reference implementations:**
**Reference implementation:**
- **Temporal + Claude Agent SDK**: `/Users/arjunmalleswaran/Code/shannon-pocs` - working implementation demonstrating workflows, activities, worker setup, and SDK integration
- **Implementation plan**: `docs/temporal-implementation-plan.md` - detailed plan for Shannon integration
### Adding a New Agent
1. Define the agent in `src/session-manager.ts` (add to `AGENT_QUEUE` and appropriate parallel group)
2. Create prompt template in `prompts/` (e.g., `vuln-newtype.txt` or `exploit-newtype.txt`)
3. Add activity function in `src/temporal/activities.ts`
4. Register activity in `src/temporal/workflows.ts` within the appropriate phase
### Modifying Prompts
- Prompt templates use variable substitution: `{{TARGET_URL}}`, `{{CONFIG_CONTEXT}}`, `{{LOGIN_INSTRUCTIONS}}`
- Shared partials in `prompts/shared/` are included via `prompt-manager.ts`
- Test changes with `PIPELINE_TESTING=true` for faster iteration
### Key Design Patterns
- **Configuration-Driven Architecture**: YAML configs with JSON Schema validation
- **Modular Error Handling**: Categorized error types with retry logic
- **Pure Functions**: Most functionality is implemented as pure functions for testability
- **SDK-First Approach**: Heavy reliance on Claude Agent SDK for autonomous AI operations
- **Progressive Analysis**: Each phase builds on previous phase results
- **Local Repository Setup**: Target applications are accessed directly from user-provided local directories
- **Fire-and-Forget Execution**: Single entry point, runs all phases to completion
### Error Handling Strategy
The application uses a comprehensive error handling system with:
@@ -206,7 +243,7 @@ The application uses a comprehensive error handling system with:
### Testing Mode
The agent includes a testing mode that skips external tool execution for faster development cycles:
```bash
shannon <WEB_URL> <REPO_PATH> --pipeline-testing
task start URL=<url> REPO=<path> PIPELINE_TESTING=true
```
### Security Focus
@@ -218,89 +255,29 @@ This is explicitly designed as a **defensive security tool** for:
The tool should only be used on systems you own or have explicit permission to test.
## File Structure
## Key Files & Directories
```
src/ # TypeScript source files
├── shannon.ts # Main orchestration script (entry point)
├── constants.ts # Shared constants
├── config-parser.ts # Configuration handling
├── error-handling.ts # Error management
├── tool-checker.ts # Tool validation
├── session-manager.ts # Agent definitions, order, and parallel groups
├── queue-validation.ts # Deliverable validation
├── splash-screen.ts # ASCII art splash screen
├── progress-indicator.ts # Progress display utilities
├── types/ # TypeScript type definitions
│ ├── index.ts # Barrel exports
│ ├── agents.ts # Agent type definitions
│ ├── config.ts # Configuration interfaces
│ ├── errors.ts # Error type definitions
│ └── session.ts # Session type definitions
├── audit/ # Audit system
│ ├── index.ts # Public API
│ ├── audit-session.ts # Main facade (logger + metrics + mutex)
│ ├── logger.ts # Append-only crash-safe logging
│ ├── metrics-tracker.ts # Timing, cost, attempt tracking
│ └── utils.ts # Path generation, atomic writes
├── ai/
│ └── claude-executor.ts # Claude Agent SDK integration
├── phases/
│ ├── pre-recon.ts # Pre-reconnaissance phase
│ └── reporting.ts # Final report assembly
├── prompts/
│ └── prompt-manager.ts # Prompt loading and variable substitution
├── setup/
│ └── environment.ts # Local repository setup
├── cli/
│ ├── ui.ts # Help text display
│ └── input-validator.ts # URL and path validation
└── utils/
├── git-manager.ts # Git operations
├── metrics.ts # Timing utilities
├── output-formatter.ts # Output formatting utilities
└── concurrency.ts # SessionMutex for parallel execution
dist/ # Compiled JavaScript output
├── shannon.js # Compiled entry point
└── ... # Other compiled files
package.json # Node.js dependencies
.shannon-store.json # Session lock file
audit-logs/ # Centralized audit data (default, or use --output)
└── {hostname}_{sessionId}/
├── session.json # Comprehensive metrics
├── prompts/ # Prompt snapshots
│ └── {agent}.md
├── agents/ # Agent execution logs
│ └── {timestamp}_{agent}_attempt-{N}.log
└── deliverables/ # Security reports and findings
└── ...
configs/ # Configuration files
├── config-schema.json # JSON Schema validation
├── example-config.yaml # Template configuration
├── juice-shop-config.yaml # Juice Shop example
├── keygraph-config.yaml # Keygraph configuration
├── chatwoot-config.yaml # Chatwoot configuration
├── metabase-config.yaml # Metabase configuration
└── cal-com-config.yaml # Cal.com configuration
prompts/ # AI prompt templates
├── shared/ # Shared content for all prompts
│ ├── _target.txt # Target URL template
│ ├── _rules.txt # Rules template
│ ├── _vuln-scope.txt # Vulnerability scope template
│ ├── _exploit-scope.txt # Exploitation scope template
│ └── login-instructions.txt # Login flow template
├── pre-recon-code.txt # Code analysis
├── recon.txt # Reconnaissance
├── vuln-*.txt # Vulnerability assessment
├── exploit-*.txt # Exploitation
└── report-executive.txt # Executive reporting
scripts/ # Utility scripts
└── export-metrics.js # Export metrics to CSV
deliverables/ # Output directory (in target repo)
docs/ # Documentation
├── unified-audit-system-design.md
└── migration-guide.md
```
**Entry Points:**
- `src/shannon.ts` - Main orchestration (direct CLI)
- `src/temporal/workflows.ts` - Temporal workflow definition
- `src/temporal/activities.ts` - Activity implementations with heartbeats
- `src/temporal/worker.ts` - Worker process entry point
- `src/temporal/client.ts` - CLI client for starting workflows
**Core Logic:**
- `src/session-manager.ts` - Agent definitions, execution order, parallel groups
- `src/ai/claude-executor.ts` - Claude Agent SDK integration
- `src/config-parser.ts` - YAML config parsing with JSON Schema validation
- `src/audit/` - Crash-safe logging and metrics system
**Configuration:**
- `Taskfile.yml` - Task runner commands
- `docker-compose.yml` - Temporal server + worker containers
- `configs/` - YAML configs with `config-schema.json` for validation
- `prompts/` - AI prompt templates (`vuln-*.txt`, `exploit-*.txt`, etc.)
**Output:**
- `audit-logs/{hostname}_{sessionId}/` - Session metrics, agent logs, deliverables
## Troubleshooting
@@ -309,8 +286,15 @@ docs/ # Documentation
- **"Repository not found"**: Ensure target local directory exists and is accessible
- **Concurrent runs blocked**: Only one session can run at a time per target
### Temporal & Docker Issues
- **"Temporal not ready"**: Wait for health check or run `docker compose logs temporal`
- **Worker not processing**: Ensure worker container is running with `docker compose ps`
- **Reset workflow state**: `task stop CLEAN=true` removes all Temporal data and volumes
- **Local apps unreachable**: Use `host.docker.internal` instead of `localhost` for URLs
- **Container permissions**: On Linux, may need `sudo` for docker commands
### External Tool Dependencies
Missing tools can be skipped using `--pipeline-testing` mode during development:
Missing tools can be skipped using `PIPELINE_TESTING=true` mode during development:
- `nmap` - Network scanning
- `subfinder` - Subdomain discovery
- `whatweb` - Web technology detection
@@ -319,6 +303,9 @@ Missing tools can be skipped using `--pipeline-testing` mode during development:
```bash
# Export metrics to CSV
./scripts/export-metrics.js --session-id <id> --output metrics.csv
# View Temporal workflow history
open http://localhost:8233
```
Note: For recovery from corrupted state, simply delete `.shannon-store.json` or edit JSON files directly.
Note: For recovery from corrupted state, simply delete `.shannon-store.json` or run `task stop CLEAN=true`.
README.md
+83 -111
View File
@@ -79,10 +79,11 @@ Shannon is available in two editions:
- [Product Line](#-product-line)
- [Setup & Usage Instructions](#-setup--usage-instructions)
- [Prerequisites](#prerequisites)
- [Authentication Setup](#authentication-setup)
- [Quick Start with Docker](#quick-start-with-docker)
- [Quick Start](#quick-start)
- [Monitoring Progress](#monitoring-progress)
- [Stopping Shannon](#stopping-shannon)
- [Usage Examples](#usage-examples)
- [Configuration (Optional)](#configuration-optional)
- [Usage Patterns](#usage-patterns)
- [Output and Results](#output-and-results)
- [Sample Reports & Benchmarks](#-sample-reports--benchmarks)
- [Architecture](#-architecture)
@@ -98,36 +99,72 @@ Shannon is available in two editions:
### Prerequisites
- **Claude Console account with credits** - Required for AI-powered analysis
- **Docker installed** - Primary deployment method
- **Docker** - Container runtime ([Install Docker](https://docs.docker.com/get-docker/))
- **Task** - Task runner for simplified commands ([Install Task](https://taskfile.dev/installation/))
- **Anthropic API key or Claude Code OAuth token** - Get from [Anthropic Console](https://console.anthropic.com)
### Authentication Setup
You need either a **Claude Code OAuth token** or an **Anthropic API key** to run Shannon. Get your token from the [Anthropic Console](https://console.anthropic.com) and pass it to Docker via the `-e` flag.
### Environment Configuration (Recommended)
To prevent Claude Code from hitting token limits during long report generation, set the max output tokens environment variable:
**For local runs:**
```bash
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000
```
**For Docker runs:**
```bash
-e CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000
```
### Quick Start with Docker
#### Build the Container
### Quick Start
```bash
docker build -t shannon:latest .
# 1. Clone Shannon
git clone https://github.com/KeygraphHQ/shannon.git
cd shannon
# 2. Configure credentials (choose one method)
# Option A: Export environment variables
export ANTHROPIC_API_KEY="your-api-key" # or CLAUDE_CODE_OAUTH_TOKEN
export CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 # recommended
# Option B: Create a .env file
cat > .env << 'EOF'
ANTHROPIC_API_KEY=your-api-key
CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000
EOF
# 3. Run a pentest
task start URL=https://your-app.com REPO=/path/to/your/repo
```
#### Prepare Your Repository
Shannon will build the containers, start the workflow, and return a workflow ID. The pentest runs in the background.
### Monitoring Progress
```bash
# View real-time worker logs
task logs
# Query a specific workflow's progress
task query ID=shannon-1234567890
# Open the Temporal Web UI for detailed monitoring
open http://localhost:8233
```
### Stopping Shannon
```bash
# Stop all containers (preserves workflow data)
task stop
# Full cleanup (removes all data)
task stop CLEAN=true
```
### Usage Examples
```bash
# Basic pentest
task start URL=https://example.com REPO=/path/to/repo
# With a configuration file
task start URL=https://example.com REPO=/path/to/repo CONFIG=./configs/my-config.yaml
# Custom output directory
task start URL=https://example.com REPO=/path/to/repo OUTPUT=./my-reports
```
### Prepare Your Repository
Shannon is designed for **web application security testing** and expects all application code to be available in a single directory structure. This works well for:
@@ -137,105 +174,35 @@ Shannon is designed for **web application security testing** and expects all app
**For monorepos:**
```bash
git clone https://github.com/your-org/your-monorepo.git repos/your-app
git clone https://github.com/your-org/your-monorepo.git /path/to/your-app
```
**For multi-repository applications** (e.g., separate frontend/backend):
```bash
mkdir repos/your-app
cd repos/your-app
mkdir /path/to/your-app
cd /path/to/your-app
git clone https://github.com/your-org/frontend.git
git clone https://github.com/your-org/backend.git
git clone https://github.com/your-org/api.git
```
**For existing local repositories:**
```bash
cp -r /path/to/your-existing-repo repos/your-app
```
#### Run Your First Pentest
**With Claude Console OAuth Token:**
```bash
docker run --rm -it \
--network host \
--cap-add=NET_RAW \
--cap-add=NET_ADMIN \
-e CLAUDE_CODE_OAUTH_TOKEN="$CLAUDE_CODE_OAUTH_TOKEN" \
-e CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 \
-v "$(pwd)/repos:/app/repos" \
-v "$(pwd)/configs:/app/configs" \
# Comment below line if using custom output directory
-v "$(pwd)/audit-logs:/app/audit-logs" \
shannon:latest \
"https://your-app.com/" \
"/app/repos/your-app" \
--config /app/configs/example-config.yaml
# Optional: uncomment below for custom output directory
# -v "$(pwd)/reports:/app/reports" \
# --output /app/reports
```
**With Anthropic API Key:**
```bash
docker run --rm -it \
--network host \
--cap-add=NET_RAW \
--cap-add=NET_ADMIN \
-e ANTHROPIC_API_KEY="$ANTHROPIC_API_KEY" \
-e CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 \
-v "$(pwd)/repos:/app/repos" \
-v "$(pwd)/configs:/app/configs" \
# Comment below line if using custom output directory
-v "$(pwd)/audit-logs:/app/audit-logs" \
shannon:latest \
"https://your-app.com/" \
"/app/repos/your-app" \
--config /app/configs/example-config.yaml
# Optional: uncomment below for custom output directory
# -v "$(pwd)/reports:/app/reports" \
# --output /app/reports
```
#### Platform-Specific Instructions
### Platform-Specific Instructions
**For Linux (Native Docker):**
Add the `--user $(id -u):$(id -g)` flag to the Docker commands above to avoid permission issues with volume mounts. Docker Desktop on macOS and Windows handles this automatically, but native Linux Docker requires explicit user mapping.
You may need to run Task commands with `sudo` depending on your Docker setup. If you encounter permission issues with output files, ensure your user has access to the Docker socket.
**Network Capabilities:**
**For macOS:**
- `--cap-add=NET_RAW` - Enables advanced port scanning with nmap
- `--cap-add=NET_ADMIN` - Allows network administration for security tools
- `--network host` - Provides access to target network interfaces
Works out of the box with Docker Desktop installed.
**Testing Local Applications:**
Docker containers cannot reach `localhost` on your host machine. Use `host.docker.internal` in place of `localhost`:
```bash
docker run --rm -it \
--add-host=host.docker.internal:host-gateway \
--cap-add=NET_RAW \
--cap-add=NET_ADMIN \
-e CLAUDE_CODE_OAUTH_TOKEN="$CLAUDE_CODE_OAUTH_TOKEN" \
-e CLAUDE_CODE_MAX_OUTPUT_TOKENS=64000 \
-v "$(pwd)/repos:/app/repos" \
-v "$(pwd)/configs:/app/configs" \
# Comment below line if using custom output directory
-v "$(pwd)/audit-logs:/app/audit-logs" \
shannon:latest \
"http://host.docker.internal:3000" \
"/app/repos/your-app" \
--config /app/configs/example-config.yaml
# Optional: uncomment below for custom output directory
# -v "$(pwd)/reports:/app/reports" \
# --output /app/reports
task start URL=http://host.docker.internal:3000 REPO=/path/to/repo
```
### Configuration (Optional)
@@ -288,12 +255,17 @@ If your application uses two-factor authentication, simply add the TOTP secret t
### Output and Results
All results are saved to `./audit-logs/` by default. Use `--output <path>` to specify a custom directory. If using `--output`, ensure that path is mounted to an accessible host directory (e.g., `-v "$(pwd)/custom-directory:/app/reports"`).
All results are saved to `./audit-logs/{hostname}_{sessionId}/` by default. Use `--output <path>` to specify a custom directory.
- **Pre-reconnaissance reports** - External scan results
- **Vulnerability assessments** - Potential vulnerabilities from thorough code analysis and network mapping
- **Exploitation results** - Proof-of-concept attempts
- **Executive reports** - Business-focused security summaries
Output structure:
```
audit-logs/{hostname}_{sessionId}/
├── session.json # Metrics and session data
├── agents/ # Per-agent execution logs
├── prompts/ # Prompt snapshots for reproducibility
└── deliverables/
└── executive-report.md # Final comprehensive security report
```
---