feat: add model tracking and reporting across pipeline

- Track actual model name from router through audit logs, session.json, and query output
- Add router-utils.ts to resolve model names from ROUTER_DEFAULT env var
- Inject model info into final report's Executive Summary section
- Update documentation with supported providers, pricing, and config examples
- Update router-config.json with latest model versions (GPT-5.2, Gemini 2.5, etc.)
This commit is contained in:
ajmallesh
2026-01-15 18:30:19 -08:00
parent 429b74cff9
commit f4e2b49493
16 changed files with 312 additions and 56 deletions

View File

@@ -84,6 +84,7 @@ Shannon is available in two editions:
- [Stopping Shannon](#stopping-shannon)
- [Usage Examples](#usage-examples)
- [Configuration (Optional)](#configuration-optional)
- [Router Mode (Alternative Providers)](#router-mode-alternative-providers)
- [Output and Results](#output-and-results)
- [Sample Reports & Benchmarks](#-sample-reports--benchmarks)
- [Architecture](#-architecture)
@@ -100,7 +101,9 @@ Shannon is available in two editions:
### Prerequisites
- **Docker** - Container runtime ([Install Docker](https://docs.docker.com/get-docker/))
- **Anthropic API key or Claude Code OAuth token** - Get from [Anthropic Console](https://console.anthropic.com)
- **AI Provider Credentials** (choose one):
- **Anthropic API key or Claude Code OAuth token** (recommended) - Get from [Anthropic Console](https://console.anthropic.com)
- **Alternative providers via Router Mode** - OpenAI, Google Gemini, DeepSeek, or OpenRouter (see [Router Mode](#router-mode-alternative-providers))
### Quick Start
@@ -252,6 +255,49 @@ rules:
If your application uses two-factor authentication, simply add the TOTP secret to your config file. The AI will automatically generate the required codes during testing.
### Router Mode (Alternative Providers)
Shannon can route requests through alternative AI providers instead of Anthropic. This is useful for:
- **Cost optimization** - DeepSeek is significantly cheaper (~$0.14/M input tokens vs $3/M for Claude)
- **Model experimentation** - Test with GPT-5.2, Gemini 3, or open-source models
- **API availability** - Use OpenRouter if Anthropic API is unavailable in your region
#### Quick Setup
1. Add your provider API key to `.env`:
```bash
# Choose one provider:
OPENAI_API_KEY=sk-...
# OR
GEMINI_API_KEY=...
# OR
DEEPSEEK_API_KEY=...
# OR
OPENROUTER_API_KEY=sk-or-...
# Set default model:
ROUTER_DEFAULT=openai,gpt-5.2 # provider,model format
```
2. Run with `ROUTER=true`:
```bash
./shannon start URL=https://example.com REPO=/path/to/repo ROUTER=true
```
#### Supported Providers
| Provider | Models | Approx. Cost | Notes |
|----------|--------|--------------|-------|
| **Anthropic** (default) | Claude Sonnet 4 | $3/$15 per M tokens | Best quality, recommended |
| **OpenAI** | gpt-5.2, gpt-5-mini | $2.50/$10 per M tokens | Good alternative |
| **Google Gemini** | gemini-2.5-pro | ~$1.25/$5 per M tokens | Long context (1M+), strong reasoning |
| **DeepSeek** | DeepSeek-V3 | ~$0.14/$0.28 per M tokens | Cheapest option |
| **OpenRouter** | 100+ models | Varies | Single API for many models |
#### Disclaimer
> **Output quality depends on model choice.** Shannon is optimized for and tested with Anthropic's Claude models. Alternative providers may produce varying results depending on the model's reasoning capabilities.
### Output and Results
All results are saved to `./audit-logs/{hostname}_{sessionId}/` by default. Use `--output <path>` to specify a custom directory.
@@ -430,7 +476,7 @@ Shannon is designed for legitimate security auditing purposes only.
#### **5. Cost & Performance**
- **Time**: As of the current version, a full test run typically takes **1 to 1.5 hours** to complete.
- **Cost**: Running the full test using Anthropic's Claude 4.5 Sonnet model may incur costs of approximately **$50 USD**. Please note that costs are subject to change based on model pricing and the complexity of the target application.
- **Cost**: Running the full test using Anthropic's Claude 4.5 Sonnet model may incur costs of approximately **$50 USD**. Using Router Mode with DeepSeek can reduce this to **~$5 USD** (see [Router Mode](#router-mode-alternative-providers)). Costs vary based on model pricing and application complexity.
#### **6. Windows Antivirus False Positives**