refactor: remove direct CLI and .shannon-store.json in favor of Temporal

- Delete src/shannon.ts direct CLI entry point (Temporal is now the only mode)
- Remove .shannon-store.json session lock (Temporal handles workflow deduplication)
- Remove broken scripts/export-metrics.js (imported non-existent function)
- Update package.json to remove main, start script, and bin entry
- Clean up CLAUDE.md and debug.md to remove obsolete references
This commit is contained in:
ajmallesh
2026-01-12 18:06:44 -08:00
parent e521e98a8f
commit 50629a24ab
6 changed files with 8 additions and 1034 deletions
+8 -17
View File
@@ -31,26 +31,18 @@ ls -lt audit-logs/<session>/agents/
cat audit-logs/<session>/agents/<latest>.log
```
**Check for lock file issues:**
```bash
# Session lock file (prevents concurrent runs)
cat .shannon-store.json
# Remove if stale (no active session)
rm .shannon-store.json
```
## Step 3: Trace the Call Path
For Shannon, trace through these layers:
1. **CLI Entry**`src/shannon.ts` - Argument parsing, session setup
2. **Config**`src/config-parser.ts` - YAML loading, schema validation
3. **Session**`src/session-manager.ts` - Agent definitions, execution order
4. **Audit**`src/audit/audit-session.ts` - Logging facade, metrics tracking
5. **Executor**`src/ai/claude-executor.ts` - SDK calls, MCP setup, retry logic
6. **Phases**`src/phases/pre-recon.ts`, `reporting.ts` - Phase-specific logic
7. **Validation**`src/queue-validation.ts` - Deliverable checks
1. **Temporal Client**`src/temporal/client.ts` - Workflow initiation
2. **Workflow**`src/temporal/workflows.ts` - Pipeline orchestration
3. **Activities**`src/temporal/activities.ts` - Agent execution with heartbeats
4. **Config**`src/config-parser.ts` - YAML loading, schema validation
5. **Session**`src/session-manager.ts` - Agent definitions, execution order
6. **Audit**`src/audit/audit-session.ts` - Logging facade, metrics tracking
7. **Executor**`src/ai/claude-executor.ts` - SDK calls, MCP setup, retry logic
8. **Validation**`src/queue-validation.ts` - Deliverable checks
## Step 4: Identify Root Cause
@@ -58,7 +50,6 @@ For Shannon, trace through these layers:
| Symptom | Likely Cause | Fix |
|---------|--------------|-----|
| "A session is already running" | Stale `.shannon-store.json` | Delete the lock file |
| Agent hangs indefinitely | MCP server crashed, Playwright timeout | Check Playwright logs in `/tmp/playwright-*` |
| "Validation failed: Missing deliverable" | Agent didn't create expected file | Check `deliverables/` dir, review prompt |
| Git checkpoint fails | Uncommitted changes, git lock | Run `git status`, remove `.git/index.lock` |
-1
View File
@@ -1,5 +1,4 @@
node_modules/
.shannon-store.json
.env
audit-logs/
dist/
-20
View File
@@ -51,13 +51,6 @@ OUTPUT=<path> Custom output directory for session folder (default: ./audi
PIPELINE_TESTING=true Use minimal prompts for fast pipeline testing
```
### Direct CLI (Local Development)
For development without Docker/Temporal:
```bash
npm install
shannon <WEB_URL> <REPO_PATH> [--config <CONFIG_FILE>] [--output <OUTPUT_DIR>]
```
### Generate TOTP for Authentication
TOTP generation is handled automatically via the `generate_totp` MCP tool during authentication flows.
@@ -72,9 +65,6 @@ npm run build
## Architecture & Components
### Main Entry Point
- `src/shannon.ts` - Main orchestration script that coordinates the entire penetration testing workflow (compiles to `dist/shannon.js`)
### Core Modules
- `src/config-parser.ts` - Handles YAML configuration parsing, validation, and distribution to agents
- `src/error-handling.ts` - Comprehensive error handling with retry logic and categorized error types
@@ -177,7 +167,6 @@ The agent implements a crash-safe audit system with the following features:
- `{hostname}_{sessionId}/prompts/` - Exact prompts used for reproducibility
- `{hostname}_{sessionId}/agents/` - Turn-by-turn execution logs
- `{hostname}_{sessionId}/deliverables/` - Security reports and findings
- **.shannon-store.json**: Minimal session lock file (prevents concurrent runs)
**Crash Safety:**
- Append-only logging with immediate flush (survives kill -9)
@@ -189,7 +178,6 @@ The agent implements a crash-safe audit system with the following features:
- 5x faster execution with parallel vulnerability and exploitation phases
**Metrics & Reporting:**
- Export metrics to CSV with `./scripts/export-metrics.js`
- Phase-level and agent-level timing/cost aggregations
- Validation results integrated with metrics
@@ -257,7 +245,6 @@ The tool should only be used on systems you own or have explicit permission to t
## Key Files & Directories
**Entry Points:**
- `src/shannon.ts` - Main orchestration (direct CLI)
- `src/temporal/workflows.ts` - Temporal workflow definition
- `src/temporal/activities.ts` - Activity implementations with heartbeats
- `src/temporal/worker.ts` - Worker process entry point
@@ -281,9 +268,7 @@ The tool should only be used on systems you own or have explicit permission to t
## Troubleshooting
### Common Issues
- **"A session is already running"**: Wait for the current session to complete, or delete `.shannon-store.json`
- **"Repository not found"**: Ensure target local directory exists and is accessible
- **Concurrent runs blocked**: Only one session can run at a time per target
### Temporal & Docker Issues
- **"Temporal not ready"**: Wait for health check or run `docker compose logs temporal`
@@ -300,11 +285,6 @@ Missing tools can be skipped using `PIPELINE_TESTING=true` mode during developme
### Diagnostic & Utility Scripts
```bash
# Export metrics to CSV
./scripts/export-metrics.js --session-id <id> --output metrics.csv
# View Temporal workflow history
open http://localhost:8233
```
Note: For recovery from corrupted state, simply delete `.shannon-store.json` or run `./shannon stop CLEAN=true`.
-5
View File
@@ -2,10 +2,8 @@
"name": "shannon",
"version": "1.0.0",
"type": "module",
"main": "./dist/shannon.js",
"scripts": {
"build": "tsc",
"start": "node ./dist/shannon.js",
"temporal:server": "docker compose -f docker/docker-compose.temporal.yml up temporal -d",
"temporal:server:stop": "docker compose -f docker/docker-compose.temporal.yml down",
"temporal:worker": "node dist/temporal/worker.js",
@@ -29,9 +27,6 @@
"zod": "^3.22.4",
"zx": "^8.0.0"
},
"bin": {
"shannon": "./dist/shannon.js"
},
"devDependencies": {
"@types/js-yaml": "^4.0.9",
"@types/node": "^25.0.3",
-181
View File
@@ -1,181 +0,0 @@
// Copyright (C) 2025 Keygraph, Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License version 3
// as published by the Free Software Foundation.
#!/usr/bin/env node
/**
* Export Metrics to CSV
*
* Converts session.json from audit-logs into CSV format for spreadsheet analysis.
*
* DATA SOURCE:
* - Reads from: audit-logs/{hostname}_{sessionId}/session.json
* - Source of truth for all metrics, timing, and cost data
* - Automatically created by Shannon during agent execution
*
* CSV OUTPUT:
* - One row per agent with: agent, phase, status, attempts, duration_ms, cost_usd
* - Perfect for importing into Excel/Google Sheets for analysis
*
* USE CASES:
* - Compare performance across multiple sessions
* - Track costs and optimize budget
* - Identify slow agents for optimization
* - Generate charts and visualizations
* - Export data for external reporting tools
*
* EXAMPLES:
* ```bash
* # Export to stdout
* ./scripts/export-metrics.js --session-id abc123
*
* # Export to file
* ./scripts/export-metrics.js --session-id abc123 --output metrics.csv
*
* # Find session ID from Shannon store
* cat .shannon-store.json | jq '.sessions | keys'
* ```
*
* NOTE: For raw metrics, just read audit-logs/.../session.json directly.
* This script only exists to provide a spreadsheet-friendly CSV format.
*/
import chalk from 'chalk';
import { fs, path } from 'zx';
import { getSession } from '../src/session-manager.js';
import { AuditSession } from '../src/audit/index.js';
// Parse command-line arguments
function parseArgs() {
const args = {
sessionId: null,
output: null
};
for (let i = 2; i < process.argv.length; i++) {
const arg = process.argv[i];
if (arg === '--session-id' && process.argv[i + 1]) {
args.sessionId = process.argv[i + 1];
i++;
} else if (arg === '--output' && process.argv[i + 1]) {
args.output = process.argv[i + 1];
i++;
} else if (arg === '--help' || arg === '-h') {
printUsage();
process.exit(0);
} else {
console.log(chalk.red(`❌ Unknown argument: ${arg}`));
printUsage();
process.exit(1);
}
}
return args;
}
function printUsage() {
console.log(chalk.cyan('\n📊 Export Metrics to CSV'));
console.log(chalk.gray('\nUsage: ./scripts/export-metrics.js [options]\n'));
console.log(chalk.white('Options:'));
console.log(chalk.gray(' --session-id <id> Session ID to export (required)'));
console.log(chalk.gray(' --output <file> Output CSV file path (default: stdout)'));
console.log(chalk.gray(' --help, -h Show this help\n'));
console.log(chalk.white('Examples:'));
console.log(chalk.gray(' # Export to stdout'));
console.log(chalk.gray(' ./scripts/export-metrics.js --session-id abc123\n'));
console.log(chalk.gray(' # Export to file'));
console.log(chalk.gray(' ./scripts/export-metrics.js --session-id abc123 --output metrics.csv\n'));
}
// Export metrics for a session
async function exportMetrics(sessionId) {
const session = await getSession(sessionId);
if (!session) {
throw new Error(`Session ${sessionId} not found`);
}
const auditSession = new AuditSession(session);
await auditSession.initialize();
const metrics = await auditSession.getMetrics();
return exportAsCSV(session, metrics);
}
// Export as CSV
function exportAsCSV(session, metrics) {
const lines = [];
// Header
lines.push('agent,phase,status,attempts,duration_ms,cost_usd');
// Phase mapping
const phaseMap = {
'pre-recon': 'pre-recon',
'recon': 'recon',
'injection-vuln': 'vulnerability-analysis',
'xss-vuln': 'vulnerability-analysis',
'auth-vuln': 'vulnerability-analysis',
'authz-vuln': 'vulnerability-analysis',
'ssrf-vuln': 'vulnerability-analysis',
'injection-exploit': 'exploitation',
'xss-exploit': 'exploitation',
'auth-exploit': 'exploitation',
'authz-exploit': 'exploitation',
'ssrf-exploit': 'exploitation',
'report': 'reporting'
};
// Agent rows
for (const [agentName, agentData] of Object.entries(metrics.metrics.agents)) {
const phase = phaseMap[agentName] || 'unknown';
lines.push([
agentName,
phase,
agentData.status,
agentData.attempts.length,
agentData.final_duration_ms,
agentData.total_cost_usd.toFixed(4)
].join(','));
}
return lines.join('\n');
}
// Main execution
async function main() {
const args = parseArgs();
if (!args.sessionId) {
console.log(chalk.red('❌ Must specify --session-id'));
printUsage();
process.exit(1);
}
console.log(chalk.cyan.bold('\n📊 Exporting Metrics to CSV\n'));
console.log(chalk.gray(`Session ID: ${args.sessionId}\n`));
const output = await exportMetrics(args.sessionId);
if (args.output) {
await fs.writeFile(args.output, output);
console.log(chalk.green(`✅ Exported to: ${args.output}`));
} else {
console.log(chalk.cyan('CSV Output:\n'));
console.log(output);
}
console.log();
}
main().catch(error => {
console.log(chalk.red.bold(`\n🚨 Fatal error: ${error.message}`));
if (process.env.DEBUG) {
console.log(chalk.gray(error.stack));
}
process.exit(1);
});
-810
View File
@@ -1,810 +0,0 @@
#!/usr/bin/env node
// Copyright (C) 2025 Keygraph, Inc.
//
// This program is free software: you can redistribute it and/or modify
// it under the terms of the GNU Affero General Public License version 3
// as published by the Free Software Foundation.
import { path, fs, $ } from 'zx';
import chalk, { type ChalkInstance } from 'chalk';
import dotenv from 'dotenv';
dotenv.config();
// Config and Tools
import { parseConfig, distributeConfig } from './config-parser.js';
import { checkToolAvailability, handleMissingTools } from './tool-checker.js';
// Session
import { AGENTS, getParallelGroups } from './session-manager.js';
import { getPromptNameForAgent } from './types/agents.js';
import type { AgentName, PromptName } from './types/index.js';
// Setup and Deliverables
import { setupLocalRepo } from './setup/environment.js';
// AI and Prompts
import { runClaudePromptWithRetry } from './ai/claude-executor.js';
import { loadPrompt } from './prompts/prompt-manager.js';
// Phases
import { executePreReconPhase } from './phases/pre-recon.js';
import { assembleFinalReport } from './phases/reporting.js';
// Utils
import { timingResults, displayTimingSummary, Timer } from './utils/metrics.js';
import { formatDuration } from './utils/formatting.js';
import { generateAuditPath } from './audit/utils.js';
import type { SessionMetadata } from './audit/utils.js';
import { AuditSession } from './audit/audit-session.js';
// CLI
import { showHelp, displaySplashScreen } from './cli/ui.js';
import { validateWebUrl, validateRepoPath } from './cli/input-validator.js';
// Error Handling
import { PentestError, logError } from './error-handling.js';
import type { DistributedConfig } from './types/config.js';
import type { ToolAvailability } from './tool-checker.js';
import { safeValidateQueueAndDeliverable } from './queue-validation.js';
// Extend global namespace for SHANNON_DISABLE_LOADER
declare global {
var SHANNON_DISABLE_LOADER: boolean | undefined;
}
// Session Lock File Management
const STORE_PATH = path.join(process.cwd(), '.shannon-store.json');
interface Session {
id: string;
webUrl: string;
repoPath: string;
status: 'in-progress' | 'completed' | 'failed';
startedAt: string;
}
interface SessionStore {
sessions: Session[];
}
function generateSessionId(): string {
return crypto.randomUUID();
}
async function loadSessions(): Promise<SessionStore> {
try {
if (await fs.pathExists(STORE_PATH)) {
return await fs.readJson(STORE_PATH) as SessionStore;
}
} catch {
// Corrupted file, start fresh
}
return { sessions: [] };
}
async function saveSessions(store: SessionStore): Promise<void> {
await fs.writeJson(STORE_PATH, store, { spaces: 2 });
}
// Session prevents concurrent runs on same repo - different repos can run in parallel
async function createSession(webUrl: string, repoPath: string): Promise<Session> {
const store = await loadSessions();
// Check for existing in-progress session
const existing = store.sessions.find(
s => s.repoPath === repoPath && s.status === 'in-progress'
);
if (existing) {
throw new PentestError(
`Session already in progress for ${repoPath}`,
'validation',
false,
{ sessionId: existing.id }
);
}
const session: Session = {
id: generateSessionId(),
webUrl,
repoPath,
status: 'in-progress',
startedAt: new Date().toISOString()
};
store.sessions.push(session);
await saveSessions(store);
return session;
}
async function updateSessionStatus(
sessionId: string,
status: 'in-progress' | 'completed' | 'failed'
): Promise<void> {
const store = await loadSessions();
const session = store.sessions.find(s => s.id === sessionId);
if (session) {
session.status = status;
await saveSessions(store);
}
}
interface PromptVariables {
webUrl: string;
repoPath: string;
sourceDir: string;
}
interface MainResult {
reportPath: string;
auditLogsPath: string;
}
interface AgentResult {
success: boolean;
duration: number;
cost?: number;
error?: string;
retryable?: boolean;
}
interface ParallelAgentResult {
agentName: AgentName;
success: boolean;
timing?: number | undefined;
cost?: number | undefined;
attempts: number;
error?: string | undefined;
}
type VulnType = 'injection' | 'xss' | 'auth' | 'ssrf' | 'authz';
interface ParallelAgentConfig {
phaseType: 'vuln' | 'exploit';
headerText: string;
specialistLabel: string;
}
interface AgentExecutionContext {
sourceDir: string;
variables: PromptVariables;
distributedConfig: DistributedConfig | null;
pipelineTestingMode: boolean;
sessionMetadata: SessionMetadata;
}
// Configure zx to disable timeouts (let tools run as long as needed)
$.timeout = 0;
function getAgentColor(agentName: AgentName): ChalkInstance {
const colorMap: Partial<Record<AgentName, ChalkInstance>> = {
'injection-vuln': chalk.red,
'injection-exploit': chalk.red,
'xss-vuln': chalk.yellow,
'xss-exploit': chalk.yellow,
'auth-vuln': chalk.blue,
'auth-exploit': chalk.blue,
'ssrf-vuln': chalk.magenta,
'ssrf-exploit': chalk.magenta,
'authz-vuln': chalk.green,
'authz-exploit': chalk.green
};
return colorMap[agentName] || chalk.cyan;
}
// Non-fatal copy - failure logs warning but doesn't halt pipeline
async function consolidateOutputs(sourceDir: string, sessionPath: string): Promise<void> {
const srcDeliverables = path.join(sourceDir, 'deliverables');
const destDeliverables = path.join(sessionPath, 'deliverables');
try {
if (await fs.pathExists(srcDeliverables)) {
await fs.copy(srcDeliverables, destDeliverables, { overwrite: true });
console.log(chalk.gray(`📄 Deliverables copied to session folder`));
} else {
console.log(chalk.yellow(`⚠️ No deliverables directory found at ${srcDeliverables}`));
}
} catch (error) {
const err = error as Error;
console.log(chalk.yellow(`⚠️ Failed to consolidate deliverables: ${err.message}`));
}
}
/**
* Run a single agent
*/
async function runAgent(
agentName: AgentName,
sourceDir: string,
variables: PromptVariables,
distributedConfig: DistributedConfig | null,
pipelineTestingMode: boolean,
sessionMetadata: SessionMetadata
): Promise<AgentResult> {
const agent = AGENTS[agentName];
const promptName = getPromptNameForAgent(agentName);
const prompt = await loadPrompt(promptName, variables, distributedConfig, pipelineTestingMode);
return await runClaudePromptWithRetry(
prompt,
sourceDir,
'*',
'',
agent.displayName,
agentName,
getAgentColor(agentName),
sessionMetadata
);
}
/**
* Execute a single agent with retry logic
*/
async function executeAgentWithRetry(
agentName: AgentName,
context: AgentExecutionContext,
onSuccess?: (agentName: AgentName) => Promise<void>
): Promise<ParallelAgentResult> {
const { sourceDir, variables, distributedConfig, pipelineTestingMode, sessionMetadata } = context;
const maxAttempts = 3;
let lastError: Error | undefined;
let attempts = 0;
while (attempts < maxAttempts) {
attempts++;
try {
const result = await runAgent(
agentName,
sourceDir,
variables,
distributedConfig,
pipelineTestingMode,
sessionMetadata
);
if (onSuccess) {
await onSuccess(agentName);
}
return {
agentName,
success: result.success,
timing: result.duration,
cost: result.cost,
attempts
};
} catch (error) {
lastError = error as Error;
if (attempts < maxAttempts) {
console.log(chalk.yellow(`Warning: ${agentName} failed attempt ${attempts}/${maxAttempts}, retrying...`));
await new Promise(resolve => setTimeout(resolve, 5000));
}
}
}
return {
agentName,
success: false,
attempts,
error: lastError?.message || 'Unknown error'
};
}
/**
* Display results table for parallel agent execution
*/
function displayParallelResults(
results: PromiseSettledResult<ParallelAgentResult>[],
agents: AgentName[],
headerText: string,
totalDuration: number
): ParallelAgentResult[] {
console.log(chalk.cyan(`\n${headerText}`));
console.log(chalk.gray('-'.repeat(80)));
console.log(chalk.bold('Agent Status Attempt Duration Cost'));
console.log(chalk.gray('-'.repeat(80)));
const processedResults: ParallelAgentResult[] = [];
results.forEach((result, index) => {
const agentName = agents[index]!;
const agentDisplay = agentName.padEnd(22);
if (result.status === 'fulfilled') {
const data = result.value;
processedResults.push(data);
if (data.success) {
const duration = formatDuration(data.timing || 0);
const cost = `$${(data.cost || 0).toFixed(4)}`;
console.log(
`${chalk.green(agentDisplay)} ${chalk.green('Success')} ` +
`${data.attempts}/3 ${duration.padEnd(11)} ${cost}`
);
} else {
console.log(
`${chalk.red(agentDisplay)} ${chalk.red('Failed ')} ` +
`${data.attempts}/3 - -`
);
if (data.error) {
console.log(chalk.gray(` Error: ${data.error.substring(0, 60)}...`));
}
}
} else {
processedResults.push({
agentName,
success: false,
attempts: 3,
error: String(result.reason)
});
console.log(
`${chalk.red(agentDisplay)} ${chalk.red('Failed ')} ` +
`3/3 - -`
);
}
});
console.log(chalk.gray('-'.repeat(80)));
const successCount = processedResults.filter(r => r.success).length;
console.log(chalk.cyan(`Summary: ${successCount}/${agents.length} succeeded in ${formatDuration(totalDuration)}`));
return processedResults;
}
/**
* Run agents in parallel with retry logic and result display
*/
async function runParallelAgents(
context: AgentExecutionContext,
config: ParallelAgentConfig
): Promise<ParallelAgentResult[]> {
const { sourceDir } = context;
const { phaseType, headerText, specialistLabel } = config;
const parallelGroups = getParallelGroups();
const allAgents = parallelGroups[phaseType];
// For exploit phase, filter to only eligible agents
let agents: AgentName[];
if (phaseType === 'exploit') {
const eligibilityChecks = await Promise.all(
allAgents.map(async (agentName) => {
const vulnAgentName = agentName.replace('-exploit', '-vuln') as AgentName;
const vulnType = vulnAgentName.replace('-vuln', '') as VulnType;
const validation = await safeValidateQueueAndDeliverable(vulnType, sourceDir);
if (!validation.success || !validation.data?.shouldExploit) {
console.log(chalk.gray(`Skipping ${agentName} (no vulnerabilities found in ${vulnAgentName})`));
return { agentName, eligible: false };
}
console.log(chalk.blue(`${agentName} eligible (${validation.data.vulnerabilityCount} vulnerabilities from ${vulnAgentName})`));
return { agentName, eligible: true };
})
);
agents = eligibilityChecks
.filter(check => check.eligible)
.map(check => check.agentName);
if (agents.length === 0) {
console.log(chalk.gray('No exploitation agents eligible (no vulnerabilities found)'));
return [];
}
} else {
agents = allAgents;
}
console.log(chalk.cyan(`\nStarting ${agents.length} ${specialistLabel} in parallel...`));
console.log(chalk.gray(' Specialists: ' + agents.join(', ')));
console.log();
const startTime = Date.now();
// Build onSuccess callback for vuln phase (validation logging)
const onSuccess = phaseType === 'vuln'
? async (agentName: AgentName): Promise<void> => {
const vulnType = agentName.replace('-vuln', '') as VulnType;
try {
const validation = await safeValidateQueueAndDeliverable(vulnType, sourceDir);
if (validation.success && validation.data) {
const message = validation.data.shouldExploit
? `Ready for exploitation (${validation.data.vulnerabilityCount} vulnerabilities)`
: 'No vulnerabilities found';
console.log(chalk.blue(`${agentName}: ${message}`));
}
} catch {
// Validation failure is non-critical
}
}
: undefined;
const results = await Promise.allSettled(
agents.map(async (agentName, index) => {
// Add 2-second stagger to prevent API overwhelm
await new Promise(resolve => setTimeout(resolve, index * 2000));
return executeAgentWithRetry(agentName, context, onSuccess);
})
);
const totalDuration = Date.now() - startTime;
return displayParallelResults(results, agents, headerText, totalDuration);
}
// Setup graceful cleanup on process signals
process.on('SIGINT', async () => {
console.log(chalk.yellow('\n⚠️ Received SIGINT, cleaning up...'));
process.exit(0);
});
process.on('SIGTERM', async () => {
console.log(chalk.yellow('\n⚠️ Received SIGTERM, cleaning up...'));
process.exit(0);
});
// Main orchestration function
async function main(
webUrl: string,
repoPath: string,
configPath: string | null = null,
pipelineTestingMode: boolean = false,
disableLoader: boolean = false,
outputPath: string | null = null
): Promise<MainResult> {
// Set global flag for loader control
global.SHANNON_DISABLE_LOADER = disableLoader;
const totalTimer = new Timer('total-execution');
timingResults.total = totalTimer;
// Display splash screen
await displaySplashScreen();
console.log(chalk.cyan.bold('🚀 AI PENETRATION TESTING AGENT'));
console.log(chalk.cyan(`🎯 Target: ${webUrl}`));
console.log(chalk.cyan(`📁 Source: ${repoPath}`));
if (configPath) {
console.log(chalk.cyan(`⚙️ Config: ${configPath}`));
}
if (outputPath) {
console.log(chalk.cyan(`📂 Output: ${outputPath}`));
}
console.log(chalk.gray('─'.repeat(60)));
// Parse configuration if provided
let distributedConfig: DistributedConfig | null = null;
if (configPath) {
try {
// Resolve config path - check configs folder if relative path
let resolvedConfigPath = configPath;
if (!path.isAbsolute(configPath)) {
const configsDir = path.join(process.cwd(), 'configs');
const configInConfigsDir = path.join(configsDir, configPath);
// Check if file exists in configs directory, otherwise use original path
if (await fs.pathExists(configInConfigsDir)) {
resolvedConfigPath = configInConfigsDir;
}
}
const config = await parseConfig(resolvedConfigPath);
distributedConfig = distributeConfig(config);
console.log(chalk.green(`✅ Configuration loaded successfully`));
} catch (error) {
await logError(error as Error, `Configuration loading from ${configPath}`);
throw error; // Let the main error boundary handle it
}
}
// Check tool availability
const toolAvailability: ToolAvailability = await checkToolAvailability();
handleMissingTools(toolAvailability);
// Setup local repository
console.log(chalk.blue('📁 Setting up local repository...'));
let sourceDir: string;
try {
sourceDir = await setupLocalRepo(repoPath);
console.log(chalk.green('✅ Local repository setup successfully'));
} catch (error) {
const err = error as Error;
console.log(chalk.red(`❌ Failed to setup local repository: ${err.message}`));
console.log(chalk.gray('This could be due to:'));
console.log(chalk.gray(' - Insufficient permissions'));
console.log(chalk.gray(' - Repository path not accessible'));
console.log(chalk.gray(' - Git initialization issues'));
console.log(chalk.gray(' - Insufficient disk space'));
process.exit(1);
}
const variables: PromptVariables = { webUrl, repoPath, sourceDir };
// Create session (acts as lock file)
const session: Session = await createSession(webUrl, repoPath);
console.log(chalk.blue(`Session created: ${session.id.substring(0, 8)}...`));
// Session metadata for audit logging
const sessionMetadata: SessionMetadata = {
id: session.id,
webUrl,
repoPath: sourceDir,
...(outputPath && { outputPath })
};
// Create outputs directory in source directory
try {
const outputsDir = path.join(sourceDir, 'outputs');
await fs.ensureDir(outputsDir);
await fs.ensureDir(path.join(outputsDir, 'schemas'));
await fs.ensureDir(path.join(outputsDir, 'scans'));
} catch (error) {
const err = error as Error;
throw new PentestError(
`Failed to create output directories: ${err.message}`,
'filesystem',
false,
{ sourceDir, originalError: err.message }
);
}
try {
// PHASE 1: PRE-RECONNAISSANCE
const { duration: preReconDuration } = await executePreReconPhase(
webUrl,
sourceDir,
variables,
distributedConfig,
toolAvailability,
pipelineTestingMode,
session.id,
outputPath
);
console.log(chalk.green(`Pre-reconnaissance complete in ${formatDuration(preReconDuration)}`));
// PHASE 2: RECONNAISSANCE
console.log(chalk.magenta.bold('\n🔎 PHASE 2: RECONNAISSANCE'));
console.log(chalk.magenta('Analyzing initial findings...'));
const reconTimer = new Timer('phase-2-recon');
await runAgent(
'recon',
sourceDir,
variables,
distributedConfig,
pipelineTestingMode,
sessionMetadata
);
const reconDuration = reconTimer.stop();
console.log(chalk.green(`✅ Reconnaissance complete in ${formatDuration(reconDuration)}`));
// PHASE 3: VULNERABILITY ANALYSIS
const vulnTimer = new Timer('phase-3-vulnerability-analysis');
console.log(chalk.red.bold('\n🚨 PHASE 3: VULNERABILITY ANALYSIS'));
const executionContext: AgentExecutionContext = {
sourceDir,
variables,
distributedConfig,
pipelineTestingMode,
sessionMetadata
};
const vulnResults = await runParallelAgents(executionContext, {
phaseType: 'vuln',
headerText: 'Vulnerability Analysis Results',
specialistLabel: 'vulnerability analysis specialists'
});
const vulnDuration = vulnTimer.stop();
console.log(chalk.green(`✅ Vulnerability analysis phase complete in ${formatDuration(vulnDuration)}`));
// PHASE 4: EXPLOITATION
const exploitTimer = new Timer('phase-4-exploitation');
console.log(chalk.red.bold('\n💥 PHASE 4: EXPLOITATION'));
const exploitResults = await runParallelAgents(executionContext, {
phaseType: 'exploit',
headerText: 'Exploitation Results',
specialistLabel: 'exploitation specialists'
});
const exploitDuration = exploitTimer.stop();
console.log(chalk.green(`✅ Exploitation phase complete in ${formatDuration(exploitDuration)}`));
// PHASE 5: REPORTING
console.log(chalk.greenBright.bold('\n📊 PHASE 5: REPORTING'));
console.log(chalk.greenBright('Generating executive summary and assembling final report...'));
const reportTimer = new Timer('phase-5-reporting');
// Assemble all deliverables into a single concatenated report
console.log(chalk.blue('📝 Assembling deliverables from specialist agents...'));
try {
await assembleFinalReport(sourceDir);
} catch (error) {
const err = error as Error;
console.log(chalk.red(`❌ Error assembling final report: ${err.message}`));
}
// Run reporter agent to create executive summary
console.log(chalk.blue('Generating executive summary and cleaning up report...'));
await runAgent(
'report',
sourceDir,
variables,
distributedConfig,
pipelineTestingMode,
sessionMetadata
);
const reportDuration = reportTimer.stop();
console.log(chalk.green(`✅ Final report generated in ${formatDuration(reportDuration)}`));
// Calculate final timing
timingResults.total.stop();
// Mark session as completed in both stores
await updateSessionStatus(session.id, 'completed');
// Update audit system's session.json status
const auditSession = new AuditSession(sessionMetadata);
await auditSession.updateSessionStatus('completed');
// Display comprehensive timing summary
displayTimingSummary();
console.log(chalk.cyan.bold('\n🎉 PENETRATION TESTING COMPLETE!'));
console.log(chalk.gray('─'.repeat(60)));
// Calculate audit logs path
const auditLogsPath = generateAuditPath(sessionMetadata);
// Consolidate deliverables into the session folder
await consolidateOutputs(sourceDir, auditLogsPath);
console.log(chalk.green(`\n📂 All outputs consolidated: ${auditLogsPath}`));
return {
reportPath: path.join(sourceDir, 'deliverables', 'comprehensive_security_assessment_report.md'),
auditLogsPath
};
} catch (error) {
// Mark session as failed in both stores
await updateSessionStatus(session.id, 'failed');
// Update audit system's session.json status
const auditSession = new AuditSession(sessionMetadata);
await auditSession.updateSessionStatus('failed');
throw error;
}
}
// Entry point - handle both direct node execution and shebang execution
let args = process.argv.slice(2);
// If first arg is the script name (from shebang), remove it
if (args[0] && args[0].includes('shannon')) {
args = args.slice(1);
}
// Parse flags and arguments
let configPath: string | null = null;
let outputPath: string | null = null;
let pipelineTestingMode = false;
let disableLoader = false;
const nonFlagArgs: string[] = [];
for (let i = 0; i < args.length; i++) {
if (args[i] === '--config') {
if (i + 1 < args.length) {
configPath = args[i + 1]!;
i++; // Skip the next argument
} else {
console.log(chalk.red('❌ --config flag requires a file path'));
process.exit(1);
}
} else if (args[i] === '--output') {
if (i + 1 < args.length) {
outputPath = path.resolve(args[i + 1]!);
i++; // Skip the next argument
} else {
console.log(chalk.red('❌ --output flag requires a directory path'));
process.exit(1);
}
} else if (args[i] === '--pipeline-testing') {
pipelineTestingMode = true;
} else if (args[i] === '--disable-loader') {
disableLoader = true;
} else if (!args[i]!.startsWith('-')) {
nonFlagArgs.push(args[i]!);
}
}
// Handle help flag
if (args.includes('--help') || args.includes('-h') || args.includes('help')) {
showHelp();
process.exit(0);
}
// Handle no arguments - show help
if (nonFlagArgs.length === 0) {
console.log(chalk.red.bold('❌ Error: No arguments provided\n'));
showHelp();
process.exit(1);
}
// Handle insufficient arguments
if (nonFlagArgs.length < 2) {
console.log(chalk.red('❌ Both WEB_URL and REPO_PATH are required'));
console.log(chalk.gray('Usage: shannon <WEB_URL> <REPO_PATH> [--config config.yaml]'));
console.log(chalk.gray('Help: shannon --help'));
process.exit(1);
}
const [webUrl, repoPath] = nonFlagArgs;
// Validate web URL
const webUrlValidation = validateWebUrl(webUrl!);
if (!webUrlValidation.valid) {
console.log(chalk.red(`❌ Invalid web URL: ${webUrlValidation.error}`));
console.log(chalk.gray(`Expected format: https://example.com`));
process.exit(1);
}
// Validate repository path
const repoPathValidation = await validateRepoPath(repoPath!);
if (!repoPathValidation.valid) {
console.log(chalk.red(`❌ Invalid repository path: ${repoPathValidation.error}`));
console.log(chalk.gray(`Expected: Accessible local directory path`));
process.exit(1);
}
// Success - show validated inputs
console.log(chalk.green('✅ Input validation passed:'));
console.log(chalk.gray(` Target Web URL: ${webUrl}`));
console.log(chalk.gray(` Target Repository: ${repoPathValidation.path}\n`));
console.log(chalk.gray(` Config Path: ${configPath}\n`));
if (outputPath) {
console.log(chalk.gray(` Output Path: ${outputPath}\n`));
}
if (pipelineTestingMode) {
console.log(chalk.yellow('⚡ PIPELINE TESTING MODE ENABLED - Using minimal test prompts for fast pipeline validation\n'));
}
if (disableLoader) {
console.log(chalk.yellow('⚙️ LOADER DISABLED - Progress indicator will not be shown\n'));
}
try {
const result = await main(webUrl!, repoPathValidation.path!, configPath, pipelineTestingMode, disableLoader, outputPath);
console.log(chalk.green.bold('\n📄 FINAL REPORT AVAILABLE:'));
console.log(chalk.cyan(result.reportPath));
console.log(chalk.green.bold('\n📂 AUDIT LOGS AVAILABLE:'));
console.log(chalk.cyan(result.auditLogsPath));
} catch (error) {
// Enhanced error boundary with proper logging
if (error instanceof PentestError) {
await logError(error, 'Main execution failed');
console.log(chalk.red.bold('\n🚨 PENTEST EXECUTION FAILED'));
console.log(chalk.red(` Type: ${error.type}`));
console.log(chalk.red(` Retryable: ${error.retryable ? 'Yes' : 'No'}`));
if (error.retryable) {
console.log(chalk.yellow(' Consider running the command again or checking network connectivity.'));
}
} else {
const err = error as Error;
console.log(chalk.red.bold('\n🚨 UNEXPECTED ERROR OCCURRED'));
console.log(chalk.red(` Error: ${err?.message || err?.toString() || 'Unknown error'}`));
if (process.env.DEBUG) {
console.log(chalk.gray(` Stack: ${err?.stack || 'No stack trace available'}`));
}
}
process.exit(1);
}