* feat(worker): add agentic static analysis Add the ten-stage Agentic SAST pipeline, confined repository tools, model runtime, prompt templates, and SARIF export. Make retries, repair sessions, reduced coverage, usage accounting, and model-output drift durable across Temporal replay and resume. Keep retry diagnostics in their actionable closed vocabulary. Package the Mantis-derived license material with the prompts that require it. * feat(worker): deduplicate static and runtime findings before exploitation Parse Agentic SAST SARIF into typed observations, enrich and route those observations, and reconcile them with pentest findings before exploitation. Publish deterministic exploitation queues with stable lineage, exact-path Git commits, retry-safe manifests, named drop reasons, and confined task formation. Reject duplicate producer IDs before commit and adopt either legal provenance shape after a lost acknowledgement. * feat(config)!: replace vuln_classes with agentic_sast Wire Agentic SAST and reconciliation into the main pipeline, persist their durable state, and add the Miscellaneous finding and exploitation lane. Make scan completion, cancellation, partial outcomes, resume identity, and report recovery use the integrated final workflow contract. Introduce the atomic finalization, ordering, renumbering, compaction, and output services that workflow calls. Keep completed Miscellaneous work and report drafts idempotent across resume, preserve public main's default-on exploit SARIF behavior, and describe stage-fallback candidates without claiming they were exported. BREAKING CHANGE: `vuln_classes` has been removed. Configs containing it now fail validation, and all five core pentest classes run on every scan. Workspaces created by Shannon 2.x cannot be resumed. Finish or discard in-flight scans before upgrading, then start a new workspace name. * perf: overlap static analysis and the Miscellaneous lane with the pentest Run Agentic SAST alongside vulnerability analysis and run Miscellaneous exploitation alongside the specialist exploitation lanes. Keep reconciliation dependent on the completed static-analysis result while preserving parallel work everywhere that has no data dependency. * feat(cli)!: default the scan target and add a JSON error contract List local scans, resolve the active or most recent workspace automatically, and make logs, status, and stop use one canonical scan identity. Add stable machine-readable failures, richer status output, explicit help errors, and seven-day Temporal retention. Treat absent Temporal pending-activity failures as absent whether the decoder represents them as `null` or missing. BREAKING CHANGE: `status --json` now returns a fixed `failureMessage`. Read `partialReasons`, `agenticSast`, and `workflow.log` for diagnostic detail. * feat(logging): trace tool calls and write a log per agent Record complete tool-call arguments in the workflow log and project each agent's events into its own durable log. Add agent listing and agent-specific log tailing while preserving byte-exact output and draining log handles before activities return. * feat(worker): standardize severity and reporting guidance in exploit prompts Give every exploit agent the same status, confidence, severity-reasoning, report-writing, credential-handling, and scope contract. Apply the same task-formation and SAST-enrichment procedure to the Miscellaneous lane. * feat(worker): disclose scan coverage and make reporting auditable Build on the retry-safe finalization foundation to preserve correct identities, source locations, scan dates, partial-coverage limitations, and consistent report JSON, Markdown, SARIF, and PDF output. Report Agentic SAST, reconciliation wall-clock time, stage usage, retry spend, and background work without duplicate or hardcoded totals. Keep report findings canonical, drop cross-class restatements, name enrichment losses, and render the executive-summary narrative in the PDF. * chore(license): attribute Mantis and Pi and refresh the docs Add the final Mantis and Pi notices, license copies, acknowledgements, and residual copyright updates. Update the README, maintained documentation, contributor guidance, and hand-maintained mirrors to describe Agentic SAST, reconciliation, the Miscellaneous lane, current CLI behavior, and the final release contract. Correct stale workspace and container guidance and annotate long-standing internals for maintainers. * fix(logging): treat a slash as a word separator in agent labels * feat(cli)!: rebuild scan status around model work - show Capella stages beneath the concurrent Agentic SAST phase - attach reconciliation time to the class row it feeds - hide completed bookkeeping and the duplicate miscellaneous wrapper - carry validated child-workflow progress into durable parent state - derive the terminal tree and status JSON from the same phase shape BREAKING CHANGE: `status --json` replaces phase `parallel` with `children` and `meta`, adds phase summaries and notes plus agent attachment fields, and removes the `analysis-engines` and `operational-work` phases. * fix(report): drop the empty Critical Findings section from the PDF summary * fix(sast): align Capella export with the submit-time code-path contract The export gate required every code_paths entry to be file:line, but submit only requires the primary sink to be file:line and accepts bare trace steps. A single malformed trace step therefore dropped an otherwise-valid finding at export. - add isValidPrimaryCodePath as the one shared primary-sink contract - validate only the primary at export; buildResult already drops unusable steps - route the submit-time validator through the same helper so the two cannot drift * feat(sast): tolerate hygiene-only Capella reductions instead of going partial A reduction only makes a run partial when it loses real coverage or a whole finding. Malformed model output, salvaged turn-limit work, and rejected duplicate verdicts are recorded as evidence but no longer flip the run to partial. - add reductionIsTolerable: partial only when genuine-loss counts are nonzero - drive runCapella's partial reasons and display coverage off non-tolerable ones - keep every reduction in agenticSast.reductions so nothing is lost as evidence * feat(logging): record the provider reason for a failed agent turn A failed provider turn collapsed to AGENT_EXECUTION_FAILED/unknown with the underlying reason discarded, so a model-side rejection or safeguard was indistinguishable from a transport fault in the error log. - add safeProviderTurnDetails: write bounded, non-sensitive fields (provider, model, responseId, stop reason, tool-in-flight, category, retryable) to error.log - gate a sanitized errorMessage snippet behind SHANNON_DEBUG_PROVIDER_ERRORS, off by default - forward SHANNON_DEBUG_PROVIDER_ERRORS from the CLI into the worker container * fix(cli): keep shannon logs tailing through a Temporal blip - End the interactive tail on the log's own terminal marker or Ctrl-C, so a transient Temporal outage no longer aborts the command with exit 1. - Rebuild the memoized Temporal client after a failed poll: a wedged gRPC channel was cached forever, so "retrying…" could never reconnect. - Keep start --follow (CI) bounded — a genuinely dead Temporal still fails the run instead of hanging. * fix(worker): correct PDF finding reporting - Render OWASP category, authentication state, and remediation - Omit the redundant per-finding exploited status - Preserve canonical category and field ordering across report modes - Continue Proof of Impact numbering across embedded code blocks - Wrap long PDF code lines without changing canonical report content * fix: attribute a reconciliation failure to exploitation only - Stop marking a class's vulnerability-analysis agent failed when that agent succeeded and only reconciliation failed; the status tree now renders the analysis row completed and the exploitation row failed - Consume the worker's failedReconciliations signal in the CLI, which the mirrored PipelineState already declared but never read - Correct the class_reconciliation_failed message, which claimed the class's analysis results were still in the report when the class is excluded from it * fix(pi): give each task sub-session its own resource loader to prevent stale extension ctx * fix(prompts): scope exploit agents to in-band proof, mark OOB-only findings blocked * fix(cli): reject a shell credential that shadows a gateway config.toml key * fix(cli): make scan shutdown verifiable - preselect and persist workflow identity before worker launch - cancel first, then verify bounded Temporal termination - reconcile Docker workers with Temporal open workflows - fail closed on stale images and unavailable lifecycle state - mark cancellation only after confirmed shutdown * feat(cli): prompt for setup on a bare npx invocation with no credentials * fix(cli): don't blame anthropic when no credentials are configured at all * chore(release): bump beta base version to 3.0.0 * feat(cli): show a 'start your first scan' box in help on a TTY * docs: refresh README and platform overview for Shannon 3.0 - lead with the 3.0 launch note and rewrite key capabilities around security code analysis, the rebuilt terminal experience, native CI/CD, and PDF/SARIF - recast the editions table as Shannon Open Source against the Keygraph Enterprise Platform, stating open source is not a trial edition - rewrite the platform overview around exhaustive agentic SAST, canonical findings, automated remediation, targeted verification, and governance - add five product screenshots under assets/keygraph-platform/, referenced relative to docs/ * docs: add the Shannon naming section and swap in the 3.0 demo GIF - explain the Claude Shannon information-theory origin under "What is Shannon?" - point "Shannon in Action" at the 3.0 recording in assets/Shannon3GIF.gif Both taken from the README half of #438. * docs: document CI/CD integrations and the reconciled analysis pipeline - add a CI/CD Integrations section covering the official GitHub Action and GitLab component, pipeline artifacts, and exploit-only severity gates - redraw the architecture section as a Mermaid flow: agentic code analysis and recon feed finding reconciliation, then exploitation and reporting - describe open-source code analysis as a multi-stage agentic workflow and reserve parsed-code CPGs and exhaustive verification for Enterprise - sharpen the privacy wording: results stay local, but model requests carry source context to whichever endpoint you configure - drop the "not recommended" framing on local models and add a section on why Shannon complements rather than replaces human pentesters - regenerate llms-full.txt from the updated README and docs * docs: add the Photoview benchmark across three models - Add a "Shannon in Action" table for Photoview 2.4.0 runs on DeepSeek v4 Flash, Grok 4.6, and Claude Opus 5, each linking its PDF report and SARIF output - Store the per-model reports under benchmark/ - Link the (forthcoming) benchmark writeup from the section intro * docs: add the Shannon vs XBOW/Aikido Photoview benchmark writeup - Add docs/shannon-xbow-aikido-benchmark.md with methodology, per-model cost/coverage tables, and links to each model's report and SARIF - Link the writeup from the README "Shannon in Action" section * docs: link the benchmark announcement discussion from the README * fix(readme): restore theme-aware banner, badge, and buttons * feat!: trigger the Shannon 3.0 major release --------- Co-authored-by: ezl-keygraph <ezhil@keygraph.io>
32 KiB
CLAUDE.md
AI-powered penetration testing agent for defensive security analysis. Automates vulnerability assessment by combining reconnaissance tools with AI-powered code analysis.
Commands
Prerequisites: Docker, AI provider credentials (.env for local, npx @keygraph/shannon setup or env vars for npx)
Dual CLI
Shannon supports two CLI modes, auto-detected based on the current working directory:
npx (npx @keygraph/shannon) |
Local (./shannon) |
|
|---|---|---|
| Install | Zero-install via npm | Clone the repo |
| Image | Pulled from Docker Hub (keygraph/shannon:latest) |
Built locally (shannon-worker) |
| State | ~/.shannon/ |
Project directory |
| Credentials | ~/.shannon/config.toml (via npx @keygraph/shannon setup) or env vars |
./.env |
| Config | ~/.shannon/config.toml (via npx @keygraph/shannon setup) |
N/A |
| Prompts | Bundled in Docker image | Mounted from ./apps/worker/prompts/ (live-editable) |
Mode auto-detection: local mode activates when env var SHANNON_LOCAL=1 is set by the ./shannon entry point (apps/cli/src/mode.ts). Otherwise npx mode.
npx Quick Start
# Configure credentials (interactive wizard)
npx @keygraph/shannon setup
# Or export env vars directly (non-interactive / CI)
export ANTHROPIC_API_KEY=your-key
# Run
npx @keygraph/shannon start -u <url> -r /path/to/repo
Local (Development) Quick Start
# Setup
echo "ANTHROPIC_API_KEY=your-key" > .env
# Build (auto-runs if image missing)
./shannon build
# Run
./shannon start -u <url> -r ./my-repo
./shannon start -u <url> -r ./my-repo -c ./apps/worker/configs/my-config.yaml
./shannon start -u <url> -r /any/path/to/repo
Common Commands
# Setup (npx mode only — one-time credential configuration)
npx @keygraph/shannon setup
# Workspaces & Resume
./shannon start -u <url> -r ./my-repo -w my-audit # New named workspace
./shannon start -u <url> -r ./my-repo -w my-audit # Resume (same command)
# Monitor
./shannon scans # List running and completed scans, with each report's path
./shannon logs [<workspace>] # Show a scan's live log (default: the single running scan, else the most recent)
./shannon logs [<workspace>] --agent <name> # Tail one agent's own log (from .shannon/agents/)
./shannon logs [<workspace>] --list-agents # List the agents that have their own log
./shannon status [<workspace>] # Live phase/agent progress of one scan, read from Temporal (redraws, then exits; same default target)
# Dashboard: http://localhost:8233
# Stop
./shannon stop [<workspace>] # Stop one scan (default: the single running scan; confirms first; --yes/-y to skip)
./shannon stop --all # Stop all running scans (Temporal stays up; confirms first)
./shannon reset # Stop everything and wipe all Temporal data + volumes (type 'confirm' to proceed; cannot be skipped)
# Version
./shannon version # npx: package version; local: git SHA
# Image management
./shannon build [--no-cache] # Local mode: build worker image
# Build TypeScript (development)
pnpm run build # Build all packages via Turborepo
pnpm run check # Type-check all packages
pnpm biome # Biome lint + format + import sorting check
pnpm biome:fix # Auto-fix lint, format, and import sorting
Monorepo tooling: pnpm workspaces, Turborepo for task orchestration, Biome for linting/formatting. TypeScript compiler options shared via tsconfig.base.json at the root. All packages extend it, overriding only rootDir and outDir. Shared devDependencies (typescript, @types/node, turbo, @biomejs/biome) are hoisted to the root workspace.
Options: -c <file> (YAML config), -o <path> (output directory), -w <name> (named workspace; auto-resumes if exists), --pipeline-testing (minimal prompts, 10s retries), --keep-container (preserve worker container after exit for log inspection), --yes/-y (skip the confirmation prompt on stop; required for non-interactive use; reset requires a typed confirm and cannot be skipped)
Architecture
Monorepo Layout
apps/cli/ — @keygraph/shannon (published to npm, bundled with tsdown)
apps/worker/ — @shannon/worker (private, Temporal worker + pipeline logic)
CLI Package (apps/cli/)
Published as @keygraph/shannon on npm. Contains Docker orchestration and a direct @temporalio/client integration for read-only status plus bounded workflow lifecycle operations; no worker/pipeline business logic or prompts. Bundled with tsdown for single-file ESM output (deps stay external).
apps/cli/src/index.ts— CLI dispatcher (setup,start,stop,reset,logs,status,scans,build,version)apps/cli/src/temporal-client.ts—@temporalio/clientintegration: connects to the frontend on127.0.0.1:7233(published by compose), providesdescribeScan(status +pendingActivities→ running agents),queryProgress(livegetProgressquery →PipelineState),getTerminalOutcome(workflowresult()), and bounded lifecycle RPCs forstop.stoprequests cancellation first, waits up to 10 seconds, then requests termination only when necessary and verifies closure within a bounded window. No worker of its own; scans are visible within Temporal's retention window, whichensureInfra(apps/cli/src/docker.ts) converges to168h(7 days) on every successfulshannon start; override withSHANNON_TEMPORAL_RETENTION(a positive whole-hour value like72h)apps/cli/src/scan/—statusrendering:pipeline.ts(static phase/agent plan +run*Agentactivity-type→agent map + mirroredPipelineState/AgentMetricstypes; keep in sync with the worker),derive.ts(pure phase/agent state derivation shared by the tree and--json),render.ts(one renderer for both the live query state and the terminal result). The tree shows model work only: every row is an agent, an Agentic SAST stage, or a report step that is currently running or failed. Reconciliation is model work owned by a class, so its wall time renders as a trailing+ durationon that class's exploitation row (its analysis row whenexploit: false) rather than as a row of its own; deterministic bookkeeping stages (report:*renumber/assemble/finalize/surface) never appear once they complete.DerivedPhase.children(renders sub-rows) andDerivedPhase.meta(durationvs ak/N donetally) are independent — Agentic SAST lists stages under a duration, exploitation lists classes under a tallyapps/cli/src/mode.ts— Auto-detection: local mode ifSHANNON_LOCAL=1env var is setapps/cli/src/docker.ts— Compose lifecycle, image pull/build, and ephemeraldocker runworker spawning. Each worker carries workspace, task-queue, and preselected workflow-ID labels so stop can correlate the local worker with its Temporal execution beforesession.jsonexists. Beforedocker run, start fsyncs that exact candidate under the workspace's hidden internals and clears it only whensession.jsonregisters the same ID; stop reconciles any candidate left by an interrupted launch. Start also checks the image's workflow-ID protocol label and refuses a stale worker that would ignore the preselected IDapps/cli/src/home.ts— State directory management (~/.shannon/for npx,./for local)apps/cli/src/env.ts—.envloading, TOML fallback (npx only) viaapps/cli/src/config/resolver.ts, credential validation, provider-scoped env flag buildingapps/cli/src/model-spec.ts—SHANNON_AI_MODEL(<provider>:<model-id>) parsing; mirrorsapps/worker/src/ai/models.tsapps/cli/src/config/resolver.ts— Cascading config (npx only): env vars →~/.shannon/config.toml(parsed withsmol-toml)apps/cli/src/config/writer.ts— TOML serialization and secure file persistence (0o600)apps/cli/src/commands/setup.ts— Interactive TUI wizard (@clack/prompts) for provider credential setup (npx only)apps/cli/src/paths.ts— Repo/config path resolution (any absolute or relative path)apps/cli/src/version.ts— Version reporting (npx:package.jsonversion; local:git-<sha>)apps/cli/src/tty.ts— Terminal capability detection:requireInteractiveguard (fails fast off-TTY instead of hanging on a prompt),supportsColorcolor gating (NO_COLOR/FORCE_COLOR), andstdoutIsTerminalfor spinner/cursor outputapps/cli/src/commands/— Command handlersapps/cli/infra/compose.yml— Bundled Temporal compose file for npx modeapps/cli/tsdown.config.ts— tsdown bundler configshannon— Node.js entry point (#!/usr/bin/env node) that delegates toapps/cli/dist/index.mjs
Docker Architecture
Infra (Temporal) runs via docker-compose.yml. Workers are ephemeral docker run --rm containers, one per scan, each with a unique task queue, preselected workflow ID, matching identity labels, and isolated volume mounts. shannon stop --all takes the union of labeled running workers and Temporal-running workflows, so an orphaned workflow is still stopped after its worker has disappeared.
docker-compose.yml— Infra only:shannon-temporal(port 7233/8233). Network:shannon-netDockerfile— 2-stage build (builder + Chainguard Wolfi runtime). Uses pnpm. Entrypoint:CMD ["node", "apps/worker/dist/temporal/worker.js"]- No
docker-compose.docker.yml— host gateway handled via--add-hostflag in CLI /etc/hostsforwarding — at worker spawn,forwardEtcHostsFlagsinapps/cli/src/docker.tsreads the host's/etc/hostsand emits one--add-hostflag per valid user-added entry. Loopback IPs (127.x,::1) are rewritten tohost-gateway; IPv6 addresses are bracketed. Disable per-scan viaSHANNON_FORWARD_HOSTS=false. No-op on Windows native (WSL2 reads its own/etc/hostsvia the Linux path).
Worker Package (apps/worker/)
apps/worker/src/paths.ts— Centralized path constants (PROMPTS_DIR,CONFIGS_DIR,WORKSPACES_DIR)apps/worker/src/session-manager.ts— Agent definitions (AGENTSrecord). Agent types inapps/worker/src/types/agents.tsapps/worker/src/config-parser.ts— YAML config parsing with JSON Schema validationapps/worker/src/ai/pi/pi-executor.ts— pi harness integration (agent-level retry disabled so Temporal owns restarts; provider-level retry on, seeapps/worker/src/ai/pi/retry-settings.ts)apps/worker/src/services/— Business logic layer (Temporal-agnostic). Activities delegate here. Key:agent-execution.ts,error-handling.ts,container.tsapps/worker/src/types/— Consolidated types:Result<T,E>,ErrorCode,AgentName,ActivityLogger, etc.apps/worker/src/utils/— Shared utilities (file I/O, formatting, concurrency)
Temporal Orchestration
Durable workflow orchestration with crash recovery, queryable progress, intelligent retry, and parallel execution (5 concurrent agents in vuln/exploit phases).
apps/worker/src/temporal/workflows.ts— Main workflow (pentestPipelineWorkflow)apps/worker/src/temporal/activities.ts— Thin wrappers — heartbeat loop, error classification, container lifecycle. Business logic delegated toapps/worker/src/services/apps/worker/src/temporal/activity-logger.ts—TemporalActivityLoggerimplementation ofActivityLoggerinterfaceapps/worker/src/temporal/summary-mapper.ts— MapsPipelineSummarytoWorkflowSummaryapps/worker/src/temporal/worker.ts— Combined worker + client entry point (per-invocation task queue, submits workflow, waits for result)apps/worker/src/temporal/shared.ts— Types, interfaces, query definitions
Five-Phase Pipeline
- Pre-Recon (
pre-recon) — Source code analysis to build the architectural baseline - Recon (
recon) — Attack surface mapping from initial findings - Vulnerability Analysis (5 parallel agents) — injection, xss, auth, authz, ssrf
- Exploitation (5 parallel agents, conditional) — Exploits confirmed vulnerabilities
- Reporting (
report) — Executive-level security report
Around those phases:
- Optional agentic static analysis runs before the pentest when
agentic_sast.enabledis"true", as a child workflow. - After each class's analysis, reconciliation groups its findings into exploitation tasks.
- Findings outside the five classes form an internal
miscellaneousclass with its own exploitation agent (miscellaneous-exploit). - A scan can finish
completed,partial,failed, orcancelled;partialcarries an ordered set of reasons.
Supporting Systems
- Configuration — YAML configs in
apps/worker/configs/use the closed JSON Schema inconfig-schema.json. Every fresh scan runs the fixed five analysis classes; there is no public class selector.agentic_sast.enabledis the only public agentic-SAST setting. Finding reconciliation runs on every scan and has no public setting of its own. Config also supports authentication (MFA/TOTP), URL/code rule scoping (rules.avoid/rules.focus),exploit, free-formrules_of_engagement, and post-hocreportoptions (min_severity,min_confidence,guidance, and exploit-onlysarifoutput viaapps/worker/src/services/sarif-renderer.ts, on by default for exploit runs and opt out withreport.sarif: "false").code_pathavoid rules are enforced via the@gotgenes/pi-permission-systemextension:apps/worker/src/temporal/activities.ts:syncCodePathDenyRuleswrites a globalpathdeny config once per workflow (apps/worker/src/ai/pi/permission-system.ts:syncPermissionSystemConfig), and the executor loads the extension when that config is present (apps/worker/src/ai/pi/pi-executor.ts), so denies fire across every tool and childtasksession. Credential resolution — local mode: env vars →./.env; npx mode: env vars →~/.shannon/config.toml(vianpx @keygraph/shannon setup) - Agentic SAST progress — Capella runs as a child workflow, so its activities are absent from the parent's
pendingActivitiesand invisible to the CLI. The child signals each stage boundary up viacapellaStageProgress(apps/worker/src/temporal/shared.ts); the parent's handler validates the payload and writes the child-suppliedstartedAtanddurationMsdirectly tooperationalStages['agentic-sast:<stage>'], so both the livegetProgressquery and the terminal result carry per-stage rows. Signalling is best-effort and every failure is swallowed — a closed or unreachable parent must never fail a SAST run.CAPELLA_STAGE_LABELSinapps/worker/src/ai/sast/types.tsis the one label table, shared by the scan log and the status tree;CAPELLA_PROGRESS_STAGESomitsexport, which runs no model and so never becomes a row. Scans predating the signal keep the aggregateagentic-sastspan and render as a bare phase line - Prompts — Per-phase templates in
apps/worker/prompts/with variable substitution ({{TARGET_URL}},{{CONFIG_CONTEXT}}). Shared partials inapps/worker/prompts/shared/viaapps/worker/src/services/prompt-manager.ts, including_code-path-rules.txt(focus/avoid[FILE]/[GLOB]routing) and_rules-of-engagement.txt(free-text engagement rules). Whenexploit: false,apps/worker/src/services/findings-renderer.tsdeterministically converts each*_exploitation_queue.jsoninto a*_findings.mdfor report assembly — no LLM in the loop - Agent Harness (pi) — Uses the pi harness (
@earendil-works/pi-coding-agent, requires Node ≥ 22.19) viaapps/worker/src/ai/pi/pi-executor.ts(runPiPrompt→createAgentSession). Retry is split inapps/worker/src/ai/pi/retry-settings.ts: pi's agent-level loop is off so Temporal owns agent restarts, whileprovider.maxRetriesstays on — pi reads theproviderblock independently of theenabledflag — so transport faults are absorbed in-session rather than costing a full agent re-run.maxRetryDelayMsis left at pi's 60s default. One model runs every phase, named bySHANNON_AI_MODEL=<provider>:<model-id>(defaultanthropic:claude-sonnet-4-6).apps/worker/src/ai/models.tsparses the spec — splitting on the first colon only, so Bedrock IDs keep theirs — and resolves it through pi'sModelRuntime. pi ships theCredentialStoreinterface but no in-memory implementation (its own readsauth.jsonfrom disk), soRuntimeCredentialStorein that file supplies one: credentials arrive as env vars in an ephemeral container and must never touch disk.createModelRuntime(providerId, apiKey)builds the runtime;allowModelNetworkstays at its defaultfalseso a scan never blocks on a catalog refresh.resolveModelSelection()is async becauseModelRuntime.create()is. Any pi-ai provider id is accepted —parseModelSpecno longer rejects against a hardcoded list, so pi's registry is the authority (an unknown provider/model surfaces as a clear "not found in pi registry" error at preflight, which points to the browsable catalogue atpi.dev/models—PI_CATALOG_URLinapps/worker/src/ai/models.ts, appended to the not-found errors and shown in the setup wizard's "Other provider" hint). Four providers are curated (CURATED_PROVIDERS:anthropic,openai,xai,amazon-bedrock) with their own credential variables, config sections, and setup flows; each provider's API key env var is declared once inPROVIDER_API_KEY_ENV— Shannon uses each vendor's own variable name (OPENAI_API_KEY,XAI_API_KEY, …), never an invented one; Bedrock's entry isAWS_BEARER_TOKEN_BEDROCK, paired withAWS_REGION, which preflight requires separately as provider config rather than a credential. Any other provider uses the generic credential path:SHANNON_AI_API_KEY(GENERIC_API_KEY_ENV) supplies the key for any provider whose credential is a plain API key. Curated providers' own variables take precedence over it, and it also works as a fallback for them — Bedrock is the sole exception (it authenticates through its AWS_ variables, so the generic key never stands in for it). The CLI forwardsSHANNON_AI_API_KEYinCOMMON_FORWARD_VARS(it is provider-neutral, binding to whateverSHANNON_AI_MODELnames, so the "only one provider configured" guard counts only named credentials), and stores it under a generic[provider]config.toml section (provider.api_key).npx @keygraph/shannon setupexposes this as the "Other provider" option: free-text provider id + model id + key (a curated provider id is rejected there, since it has its own option).SHANNON_AI_BASE_URLoverrides the endpoint for any provider (proxies/gateways); the credential is unchanged.pointAtGateway(apps/worker/src/ai/models.ts) applies the one dialect change: behind a base URL,openaifollowsSHANNON_AI_OPENAI_FORMAT(chat-completionsdefault, orresponses). Onchat-completionsit switches the API toopenai-completionsand drops the catalogue's Responses-shapedcompatblock so pi'sdetectCompatderives completions settings; onresponsesthe descriptor is unchanged but for the endpoint.resolveGatewayFormatrejects the variable when the provider is notopenaior no base URL is set, since it cannot take effect there. All other providers keep their API. The CLI mirrors the accepted values inapps/cli/src/model-spec.ts, forwards the variable inCOMMON_FORWARD_VARS, and maps it toopenai.formatin config.toml.buildEnvFlagsforwards only the selected provider's credential into the worker container. The CLI mirrors the parse rule and the provider/credential tables inapps/cli/src/model-spec.ts(it cannot import from the worker package); the two must stay in sync. pi ships no JSON-schema output orTask/TodoWritebuilt-ins, so structured queues are captured via asubmit_exploitation_queuecustom tool (apps/worker/src/ai/queue-schemas.ts), andtask(child sessions scoped toread,grep,find,ls,write, andbash— no nestedtaskor collector tools;CHILD_TOOLSinapps/worker/src/ai/pi/task-tool.ts) +todo_write(apps/worker/src/ai/pi/session-tools.ts) are provided as custom tools; the per-phase collectors are pi custom tools (TypeBoxdefineToolinapps/worker/src/collectors/). Shannon sets no thinking configuration at all — nothinkingLevelis passed to anycreateAgentSessioncall, so pi's own default applies. There is no adaptive-thinking support and noCLAUDE_ADAPTIVE_THINKING/core.adaptive_thinkingsetting. Browser automation viaplaywright-cliwith session isolation (-s=<session>). TOTP generation viagenerate-totpCLI tool. Login flow template atapps/worker/prompts/shared/login-instructions.txtsupports form, SSO, API, and basic auth. On authenticated whitebox scans, thevalidate-authenticationpreflight performs the single real login and saves the browser session toauth-state.jsonin the per-session audit directory (path fromauthStateFile()inapps/worker/src/audit/utils.ts, derived fromgenerateAuditPath()). The validation activity (apps/worker/src/services/validate-authentication.ts) removes any stale file from a prior run before the agent runs and verifies the file parses and contains cookies or storage before the preflight is marked complete;logWorkflowCompletedeletes it when the workflow ends so authenticated cookies don't sit on disk between scans. Agent prompts opt in to session reuse by@include(shared/_shared-session.txt)before their<login_instructions>block — the partial restores the session and falls through to the full login flow if verification fails.vuln-auth/exploit-authomit the include and own their own login - Pi Credential Reuse —
SHANNON_USE_PI_AUTH=1opts into reusing the host's Pi login, including anopenai-codexChatGPT Plus/Pro subscription (SHANNON_AI_MODEL=openai-codex:<model-id>) or anxaiGrok subscription (SHANNON_AI_MODEL=xai:<model-id>); the mechanism is provider-agnostic and works for any Pi login.apps/cli/src/env.tsrequires~/.pi/agent/auth.json;start.tspasses its path tospawnWorker, which mounts only that file read-write at/tmp/.pi/agent/auth.json. The flag itself is not forwarded: the worker detects the file withpiAuthPresent()and passes its path toModelRuntime.create. CLI and worker API-key presence checks are skipped on this path, but the normal preflight model probe still validates the credential. The image and UID-remapping entrypoint keep/tmp/.pi/agentowned bypentestso adjacent Pi/Shannon configuration remains writable. Refreshed OAuth state is persisted to the host for subsequent scans. - Audit System — Crash-safe append-only logging in
workspaces/{hostname}_{sessionId}/. The run directory's top level holds the human-facing report in both formats (Security-Assessment-Report.pdfandSecurity-Assessment-Report.md,FINAL_REPORT_PDF_FILENAME/FINAL_REPORT_MD_FILENAMEinapps/worker/src/paths.ts); everything else — deliverables, per-agent logs, prompts,session.json,workflow.log, and browser artifacts — is nested under a hidden.shannon/internals dir (INTERNAL_DIR) so a customer sees only the report. Audit path helpers route throughgenerateInternalPath(apps/worker/src/audit/utils.ts); the CLI nests the overlay backing dirs under the same.shannon/(apps/cli/src/docker.ts,start.ts).session.json/workflow.logreads use dual-read resolvers (resolveSessionJsonPath,resolveRunFile) that prefer.shannon/and fall back to the legacy run-root layout, so pre-restructure workspaces stay listable (scans/logs) without migration. A pre-restructure workspace cannot be resumed:classifyWorkspaceLaunch(apps/cli/src/commands/start.ts) requires.shannon/launch.json, and its absence fails the launch as "created by an earlier version of Shannon" before anything on disk is touched. There is no in-place migration — the workspace's files and report are left untouched, and the operator starts a new scan under a different-wname. The report agent writes structured findings toreport.json, from whichreport-renderer.tsrenders the assembled markdown andreport-json-adapter.tsproduces the Typst-shaped JSON thatpdf-renderer.tscompiles intocomprehensive_security_assessment_report.pdfusing the bundledapps/worker/templates/typst/report.typtemplate (thetypstbinary is installed in the worker image).copyReportToRunRoot(apps/worker/src/services/reporting.ts) surfaces both the PDF and the markdown to the run root asSecurity-Assessment-Report.pdfandSecurity-Assessment-Report.md; the deliverables-dir copies remain as the git-checkpointed sources. PDF compilation is best-effort — a failure is logged and the run still completes. WorkflowLogger (apps/worker/src/audit/workflow-logger.ts) provides unified human-readable per-workflow logs, backed by LogStream (apps/worker/src/audit/log-stream.ts) shared stream primitive. Every combined-log line is also projected into a per-agent file under.shannon/agents/<slug>.log(one per pipeline agent, one per Capella stage; subagents fold into the parent's file, and a stage's concurrent sessions share its file with an inline session label). The projection boundary isapps/worker/src/audit/actor-projection.ts(projectActormaps aTraceActorto its combined prefix and owning file slug — slugs come only from closed fields); fan-out is best-effort and never blocks the canonical combined log. A lifecycle owner holds aLogStreamlease per agent file (the pipeline agent'slogAgentspan, or a Capella stage activity'stry/finally) so per-line writes ride the reference count;CapellaStageTrace.drain()flushes a stage's trace queue before its activity returns. The CLI tails one file withshannon logs --agent <name>(--list-agentsto enumerate); the defaultshannon logspath is unchanged - Deliverables — Saved to
.shannon/deliverables/in the target repo via thesave-deliverableCLI script (apps/worker/src/scripts/save-deliverable.ts) - Workspaces & Resume — Named workspaces via
-w <name>or auto-named from URL+timestamp. Resume detects completed agents viasession.json.loadResumeState()inapps/worker/src/temporal/activities.tsvalidates deliverable existence, restores git checkpoints, and cleans up incomplete deliverables
Development Notes
Adding a New Agent
- Define agent in
apps/worker/src/session-manager.ts(add toAGENTSrecord).ALL_AGENTS/AgentNametypes live inapps/worker/src/types/agents.ts - Create prompt template in
apps/worker/prompts/(e.g.,vuln-newtype.txt) - Two-layer pattern: add a thin activity wrapper in
apps/worker/src/temporal/activities.ts(heartbeat + error classification).AgentExecutionServiceinapps/worker/src/services/agent-execution.tshandles the agent lifecycle automatically via theAGENTSregistry - Register activity in
apps/worker/src/temporal/workflows.tswithin the appropriate phase
Modifying Prompts
- Variable substitution:
{{TARGET_URL}},{{CONFIG_CONTEXT}},{{LOGIN_INSTRUCTIONS}} - Shared partials in
apps/worker/prompts/shared/included viaapps/worker/src/services/prompt-manager.ts - Test with
--pipeline-testingfor fast iteration
Key Design Patterns
- Configuration-Driven — YAML configs with JSON Schema validation
- Progressive Analysis — Each phase builds on previous results
- Harness-First — the pi harness (
@earendil-works/pi-coding-agent) handles autonomous analysis - Modular Error Handling —
ErrorCodeenum,Result<T,E>for explicit error propagation, automatic retry (3 attempts per agent) - Services Boundary — Activities are thin Temporal wrappers;
apps/worker/src/services/owns business logic, acceptsActivityLogger, returnsResult<T,E>. No Temporal imports in services - DI Container — Per-workflow in
apps/worker/src/services/container.ts.AuditSessionexcluded (parallel safety) - Ephemeral Workers — Each scan runs in its own
docker run --rmcontainer with a per-invocation task queue. Temporal routes activities by queue name, so per-scan queues ensure activities never land on a worker with the wrong repo mounted
Security
Defensive security tool only. Use only on systems you own or have explicit permission to test.
Code Style Guidelines
Formatting
Biome handles formatting and linting. Run pnpm biome:fix to auto-fix. Config in biome.json: single quotes, semicolons, trailing commas, 2-space indent, 120 char line width.
Clarity Over Brevity
- Optimize for readability, not line count — three clear lines beat one dense expression
- Use descriptive names that convey intent
- Prefer explicit logic over clever one-liners
Structure
- Keep functions focused on a single responsibility
- Use early returns and guard clauses instead of deep nesting
- Never use nested ternary operators — use if/else or switch
- Extract complex conditions into well-named boolean variables
TypeScript Conventions
- Use
functionkeyword for top-level functions (not arrow functions) - Explicit return type annotations on exported/top-level functions
- Prefer
readonlyfor data that shouldn't be mutated exactOptionalPropertyTypesis enabled — use spread for optional props, not directundefinedassignment
Avoid
- Combining multiple concerns into a single function to "save lines"
- Dense callback chains when sequential logic is clearer
- Sacrificing readability for DRY — some repetition is fine if clearer
- Abstractions for one-time operations
- Backwards-compatibility shims, deprecated wrappers, or re-exports for removed code — delete the old code, don't preserve it
Comments
Comments must be timeless — no references to this conversation, refactoring history, or the AI.
Patterns used in this codebase:
/** JSDoc */— file headers (after license) and exported functions/interfaces// N. Description— numbered sequential steps inside function bodies. Use when a function has 3+ distinct phases where at least one isn't immediately obvious from the code. Each step marks the start of a logical phase. Reference:AgentExecutionService.execute(steps 1-9) andinjectModelIntoReport(steps 1-5)// === Section ===— high-level dividers between groups of functions in long files, or to label major branching/classification blocks (e.g.,// === SPENDING CAP SAFEGUARD ===). Not for sequential steps inside function bodies — use numbered steps for that// NOTE:/// WARNING:/// IMPORTANT:— gotchas and constraints
Never: obvious comments, conversation references ("as discussed"), history ("moved from X")
Key Files
CLI: shannon (entry point), apps/cli/src/index.ts (dispatcher), apps/cli/src/docker.ts (orchestration), apps/cli/src/mode.ts (auto-detection)
Entry Points: apps/worker/src/temporal/workflows.ts, apps/worker/src/temporal/activities.ts, apps/worker/src/temporal/worker.ts
Core Logic: apps/worker/src/session-manager.ts, apps/worker/src/ai/pi/pi-executor.ts, apps/worker/src/ai/pi/permission-system.ts (writes code_path deny rules to the @gotgenes/pi-permission-system global config), apps/worker/src/config-parser.ts, apps/worker/src/services/ (incl. preflight.ts, findings-renderer.ts, reporting.ts), apps/worker/src/audit/
Config: docker-compose.yml, apps/cli/infra/compose.yml, apps/worker/configs/, apps/worker/prompts/, tsconfig.base.json (shared compiler options), turbo.json, biome.json
CI/CD: .github/workflows/release.yml (Docker Hub push + npm publish + GitHub release, manual dispatch)
Package Installation
Package managers are configured with a minimum release age (7 days). Requires pnpm >= 10.16.0. If pnpm install fails due to a package being too new, do not attempt to bypass it — report the blocked package to the user and stop.
Troubleshooting
- "Repository not found" — Pass a path to the target repo (
-r /path/to/repoor-r ./my-repo) - "Temporal not ready" — Wait for health check or
docker compose logs temporal - Worker not processing — Check
docker ps --filter "name=shannon-worker-" - Reset state —
./shannon reset - Local apps unreachable — Use
host.docker.internalinstead oflocalhost - Container permissions — On Linux, may need
sudofor docker commands