mirror of
https://github.com/KeygraphHQ/shannon.git
synced 2026-09-14 14:09:04 +02:00
v3.1.0
4
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Arjun Malleswaranandezl-keygraph
|
9767ebe633 |
feat: Shannon 3.0 Agentic SAST (#433)
* feat(worker): add agentic static analysis Add the ten-stage Agentic SAST pipeline, confined repository tools, model runtime, prompt templates, and SARIF export. Make retries, repair sessions, reduced coverage, usage accounting, and model-output drift durable across Temporal replay and resume. Keep retry diagnostics in their actionable closed vocabulary. Package the Mantis-derived license material with the prompts that require it. * feat(worker): deduplicate static and runtime findings before exploitation Parse Agentic SAST SARIF into typed observations, enrich and route those observations, and reconcile them with pentest findings before exploitation. Publish deterministic exploitation queues with stable lineage, exact-path Git commits, retry-safe manifests, named drop reasons, and confined task formation. Reject duplicate producer IDs before commit and adopt either legal provenance shape after a lost acknowledgement. * feat(config)!: replace vuln_classes with agentic_sast Wire Agentic SAST and reconciliation into the main pipeline, persist their durable state, and add the Miscellaneous finding and exploitation lane. Make scan completion, cancellation, partial outcomes, resume identity, and report recovery use the integrated final workflow contract. Introduce the atomic finalization, ordering, renumbering, compaction, and output services that workflow calls. Keep completed Miscellaneous work and report drafts idempotent across resume, preserve public main's default-on exploit SARIF behavior, and describe stage-fallback candidates without claiming they were exported. BREAKING CHANGE: `vuln_classes` has been removed. Configs containing it now fail validation, and all five core pentest classes run on every scan. Workspaces created by Shannon 2.x cannot be resumed. Finish or discard in-flight scans before upgrading, then start a new workspace name. * perf: overlap static analysis and the Miscellaneous lane with the pentest Run Agentic SAST alongside vulnerability analysis and run Miscellaneous exploitation alongside the specialist exploitation lanes. Keep reconciliation dependent on the completed static-analysis result while preserving parallel work everywhere that has no data dependency. * feat(cli)!: default the scan target and add a JSON error contract List local scans, resolve the active or most recent workspace automatically, and make logs, status, and stop use one canonical scan identity. Add stable machine-readable failures, richer status output, explicit help errors, and seven-day Temporal retention. Treat absent Temporal pending-activity failures as absent whether the decoder represents them as `null` or missing. BREAKING CHANGE: `status --json` now returns a fixed `failureMessage`. Read `partialReasons`, `agenticSast`, and `workflow.log` for diagnostic detail. * feat(logging): trace tool calls and write a log per agent Record complete tool-call arguments in the workflow log and project each agent's events into its own durable log. Add agent listing and agent-specific log tailing while preserving byte-exact output and draining log handles before activities return. * feat(worker): standardize severity and reporting guidance in exploit prompts Give every exploit agent the same status, confidence, severity-reasoning, report-writing, credential-handling, and scope contract. Apply the same task-formation and SAST-enrichment procedure to the Miscellaneous lane. * feat(worker): disclose scan coverage and make reporting auditable Build on the retry-safe finalization foundation to preserve correct identities, source locations, scan dates, partial-coverage limitations, and consistent report JSON, Markdown, SARIF, and PDF output. Report Agentic SAST, reconciliation wall-clock time, stage usage, retry spend, and background work without duplicate or hardcoded totals. Keep report findings canonical, drop cross-class restatements, name enrichment losses, and render the executive-summary narrative in the PDF. * chore(license): attribute Mantis and Pi and refresh the docs Add the final Mantis and Pi notices, license copies, acknowledgements, and residual copyright updates. Update the README, maintained documentation, contributor guidance, and hand-maintained mirrors to describe Agentic SAST, reconciliation, the Miscellaneous lane, current CLI behavior, and the final release contract. Correct stale workspace and container guidance and annotate long-standing internals for maintainers. * fix(logging): treat a slash as a word separator in agent labels * feat(cli)!: rebuild scan status around model work - show Capella stages beneath the concurrent Agentic SAST phase - attach reconciliation time to the class row it feeds - hide completed bookkeeping and the duplicate miscellaneous wrapper - carry validated child-workflow progress into durable parent state - derive the terminal tree and status JSON from the same phase shape BREAKING CHANGE: `status --json` replaces phase `parallel` with `children` and `meta`, adds phase summaries and notes plus agent attachment fields, and removes the `analysis-engines` and `operational-work` phases. * fix(report): drop the empty Critical Findings section from the PDF summary * fix(sast): align Capella export with the submit-time code-path contract The export gate required every code_paths entry to be file:line, but submit only requires the primary sink to be file:line and accepts bare trace steps. A single malformed trace step therefore dropped an otherwise-valid finding at export. - add isValidPrimaryCodePath as the one shared primary-sink contract - validate only the primary at export; buildResult already drops unusable steps - route the submit-time validator through the same helper so the two cannot drift * feat(sast): tolerate hygiene-only Capella reductions instead of going partial A reduction only makes a run partial when it loses real coverage or a whole finding. Malformed model output, salvaged turn-limit work, and rejected duplicate verdicts are recorded as evidence but no longer flip the run to partial. - add reductionIsTolerable: partial only when genuine-loss counts are nonzero - drive runCapella's partial reasons and display coverage off non-tolerable ones - keep every reduction in agenticSast.reductions so nothing is lost as evidence * feat(logging): record the provider reason for a failed agent turn A failed provider turn collapsed to AGENT_EXECUTION_FAILED/unknown with the underlying reason discarded, so a model-side rejection or safeguard was indistinguishable from a transport fault in the error log. - add safeProviderTurnDetails: write bounded, non-sensitive fields (provider, model, responseId, stop reason, tool-in-flight, category, retryable) to error.log - gate a sanitized errorMessage snippet behind SHANNON_DEBUG_PROVIDER_ERRORS, off by default - forward SHANNON_DEBUG_PROVIDER_ERRORS from the CLI into the worker container * fix(cli): keep shannon logs tailing through a Temporal blip - End the interactive tail on the log's own terminal marker or Ctrl-C, so a transient Temporal outage no longer aborts the command with exit 1. - Rebuild the memoized Temporal client after a failed poll: a wedged gRPC channel was cached forever, so "retrying…" could never reconnect. - Keep start --follow (CI) bounded — a genuinely dead Temporal still fails the run instead of hanging. * fix(worker): correct PDF finding reporting - Render OWASP category, authentication state, and remediation - Omit the redundant per-finding exploited status - Preserve canonical category and field ordering across report modes - Continue Proof of Impact numbering across embedded code blocks - Wrap long PDF code lines without changing canonical report content * fix: attribute a reconciliation failure to exploitation only - Stop marking a class's vulnerability-analysis agent failed when that agent succeeded and only reconciliation failed; the status tree now renders the analysis row completed and the exploitation row failed - Consume the worker's failedReconciliations signal in the CLI, which the mirrored PipelineState already declared but never read - Correct the class_reconciliation_failed message, which claimed the class's analysis results were still in the report when the class is excluded from it * fix(pi): give each task sub-session its own resource loader to prevent stale extension ctx * fix(prompts): scope exploit agents to in-band proof, mark OOB-only findings blocked * fix(cli): reject a shell credential that shadows a gateway config.toml key * fix(cli): make scan shutdown verifiable - preselect and persist workflow identity before worker launch - cancel first, then verify bounded Temporal termination - reconcile Docker workers with Temporal open workflows - fail closed on stale images and unavailable lifecycle state - mark cancellation only after confirmed shutdown * feat(cli): prompt for setup on a bare npx invocation with no credentials * fix(cli): don't blame anthropic when no credentials are configured at all * chore(release): bump beta base version to 3.0.0 * feat(cli): show a 'start your first scan' box in help on a TTY * docs: refresh README and platform overview for Shannon 3.0 - lead with the 3.0 launch note and rewrite key capabilities around security code analysis, the rebuilt terminal experience, native CI/CD, and PDF/SARIF - recast the editions table as Shannon Open Source against the Keygraph Enterprise Platform, stating open source is not a trial edition - rewrite the platform overview around exhaustive agentic SAST, canonical findings, automated remediation, targeted verification, and governance - add five product screenshots under assets/keygraph-platform/, referenced relative to docs/ * docs: add the Shannon naming section and swap in the 3.0 demo GIF - explain the Claude Shannon information-theory origin under "What is Shannon?" - point "Shannon in Action" at the 3.0 recording in assets/Shannon3GIF.gif Both taken from the README half of #438. * docs: document CI/CD integrations and the reconciled analysis pipeline - add a CI/CD Integrations section covering the official GitHub Action and GitLab component, pipeline artifacts, and exploit-only severity gates - redraw the architecture section as a Mermaid flow: agentic code analysis and recon feed finding reconciliation, then exploitation and reporting - describe open-source code analysis as a multi-stage agentic workflow and reserve parsed-code CPGs and exhaustive verification for Enterprise - sharpen the privacy wording: results stay local, but model requests carry source context to whichever endpoint you configure - drop the "not recommended" framing on local models and add a section on why Shannon complements rather than replaces human pentesters - regenerate llms-full.txt from the updated README and docs * docs: add the Photoview benchmark across three models - Add a "Shannon in Action" table for Photoview 2.4.0 runs on DeepSeek v4 Flash, Grok 4.6, and Claude Opus 5, each linking its PDF report and SARIF output - Store the per-model reports under benchmark/ - Link the (forthcoming) benchmark writeup from the section intro * docs: add the Shannon vs XBOW/Aikido Photoview benchmark writeup - Add docs/shannon-xbow-aikido-benchmark.md with methodology, per-model cost/coverage tables, and links to each model's report and SARIF - Link the writeup from the README "Shannon in Action" section * docs: link the benchmark announcement discussion from the README * fix(readme): restore theme-aware banner, badge, and buttons * feat!: trigger the Shannon 3.0 major release --------- Co-authored-by: ezl-keygraph <ezhil@keygraph.io> |
||
|
ezl-keygraph
|
dc2a4fe4e8 |
feat: brand the npm page, CLI output, and reports (#432)
* docs: rebuild the npm package README on the main README's identity * docs: point the README banner fallback at an asset that exists * docs: declare the npm package author, homepage, and issue tracker * docs(cli): retire "Framework" and settle on the canonical product line * docs: describe the banner image in alt text instead of repeating the lockup * feat(cli): print a plain-text banner when stdout is not a terminal * feat(report): attribute the markdown report from a shared brand constant * feat(cli): frame the plain-text banner with rules and split the version line * docs: drop the URL from the npm author field |
||
|
ezl-keygraph
|
d41ae9c20d |
feat(cli): overhaul commands and add live scan status (#424)
* refactor(cli): list workspaces natively instead of via the worker image * feat(cli): preflight that Docker is installed and running * feat(cli): stop scans by workspace or --all, terminating their Temporal workflows * fix(worker): abort the running agent on cancellation so Temporal cancel takes effect * refactor(cli): split destructive teardown out of stop into a reset command * refactor(cli): centralise flag parsing and confirmation across commands * fix(cli): pass provider credentials to docker by name to keep secrets out of argv * feat(cli): add per-command help via <command> --help/-h and help <command> * feat(cli): replace raw docker output with clack spinners for infra and scan teardown * fix(cli): verify scan stop by re-querying container and workflow state instead of assuming success * fix(cli): resolve running state before prompting on stop and report no-op stops honestly * refactor(cli): show splash first and drive start with one spinner resolving to a clean line * fix(cli): validate --url up front so a bad value fails cleanly instead of a late crash * refactor(cli): centralize error reporting with fail() for expected errors and a crash handler that logs the stack and links the issue tracker * feat(cli): add --json/--plain machine-readable output to workspaces and status * refactor(cli): remove the workspaces command * refactor(cli): remove the status command * feat(cli): add 'progress <workspace>' — live scan progress from Temporal * fix(cli): mark metric-less agents as skipped in progress, not done * feat(cli): animate running agents in progress with a clack-style spinner * feat(cli): rename progress->status, reveal agents as they run, show live per-agent elapsed * fix(cli): mark passed-over phases as skipped live, not pending * style(cli): rename status footer 'Wall-clock' to 'Time Taken', drop the parenthetical * style(cli): drop '(sum of agents)' from status total cost line * style(cli): green filled circle for completed, Shannon gold for running * style(cli): use Shannon gold in place of green in status * feat(cli): suggest closest command or flag on typo * refactor(cli): single-source start help and drop ./repos bare-name shortcut * feat(cli): name providers and fix in multi-provider credential error * feat(cli): support --flag=value syntax and expand leading ~ in paths * refactor(cli): centralize ANSI color codes in colors.ts * feat(cli): add scans command listing completed scans with cost and duration * fix(cli): keep stdout clean off-TTY for logs and start * feat(cli): add repo link to top-level help * feat(worker): record auth-validation metrics and register resume attempts early * refactor(cli): share resume-aware workflow-id resolution and surface root-cause failures * feat(cli): add status --json, auth phase, dashboard link, and stable live redraw * refactor(cli): drop cost from status and scans output * feat(worker): surface both PDF and markdown report at run root * refactor(cli): normalize error/warning prefixing through fail and warn * feat(cli): add version --json for machine-readable output * refactor(cli): rename start --debug to --keep-container * refactor(cli): point start's progress hint at status instead of the Temporal dashboard * refactor(cli): centralize the mode-aware command prefix * refactor(cli): trim start and logs output to durable facts off-TTY * feat(cli): require typed confirmation for reset instead of --yes reset permanently wipes all Temporal data and volumes — a severe, irreversible action. Replace its default y/N confirm (bypassable with --yes) with a typed-word confirmation that has no bypass, so the wipe can only be triggered by a deliberate interactive answer. * feat(cli): surface logs and status hints after start on a TTY * feat(cli): exit 2 on usage errors, distinct from operational failures * feat(cli): add start --follow to stream logs and exit on scan outcome * refactor(cli): redesign splash with sunset-gradient wordmark and truecolor * refactor(cli): remove the uninstall command * docs: sync CLI docs with removed uninstall/workspaces, new scans and --follow * docs: fix reset confirmation — typed confirm, not --yes/-y * style(cli): restructure status footer with divider, aligned Logs/Temporal rows * feat(cli): show splash in the status command * fix(worker): validate auth-state shape, not entry count * docs: correct reset confirmation and add markdown report to run-root docs |
||
|
ezl-keygraph
|
bc8fd203ed |
feat: add npx CLI with monorepo, CI/CD, and ephemeral worker architecture (#256)
* feat: integrate npx CLI, CI/CD, and ephemeral worker architecture
Bring in changes from shannon-npx: npx-distributable CLI package (cli/),
semantic-release CI/CD workflows, ephemeral per-scan worker containers,
TOML config support, setup wizard, and workspace management.
Preserves all shannon-only changes: security hardening (localhost-bound
ports, MCP env allowlist, path traversal guard), updated benchmarks
(XBEN 19/31/35/44), README assets, and prompt injection disclaimer.
Applies security hardening to cli/infra/compose.yml as well.
* refactor: migrate to Turborepo + pnpm + Biome monorepo
Restructure into apps/worker, apps/cli, packages/mcp-server with
Turborepo task orchestration, pnpm workspaces, Biome linting/formatting,
and tsdown CLI bundling.
Key changes:
- src/ -> apps/worker/src/, cli/ -> apps/cli/, mcp-server/ -> packages/mcp-server/
- prompts/ and configs/ moved into apps/worker/
- npm replaced with pnpm, package-lock.json replaced with pnpm-lock.yaml
- Dockerfile updated for pnpm-based builds
- CLI logs command rewritten with chokidar for cross-platform reliability
- Router health checking added for auto-detected router mode
- Centralized path resolution via apps/worker/src/paths.ts
* fix: resolve all biome warnings and formatting issues
- Remove unnecessary non-null assertions where values are guaranteed
- Replace array index access with .at() for safer element retrieval
- Use local variables to avoid repeated process.env lookups
- Replace any types with unknown in functional utilities
- Use nullish coalescing for TOTP hash byte access
- Auto-format security patches to match biome config
* fix: pin pnpm to 10.12.1 in Dockerfile for catalog support
* fix: handle Esc cancellation in Bedrock setup flow
Replace p.group() with individual prompts and per-field cancel checks,
matching the pattern used by all other provider setup flows.
* feat: add optional model customization to Anthropic setup
* fix: resolve Docker bind mount permission errors on Linux
Use entrypoint-based UID remapping instead of --user flag so the
container's pentest user matches the host UID/GID, keeping bind-mounted
volumes writable. Git config moved to --system level to survive remapping.
* fix: show resumed workflow ID in splash screen URL
When resuming a workflow, the Temporal Web UI link pointed to the old
(terminated) workflow ID. Now extracts "New Workflow ID" from the resume
header in workflow.log, falling back to the original ID for fresh scans.
* style: fix biome formatting in docker.ts
* fix: align TypeScript config types with JSON Schema
- SuccessCondition.type: use schema values (url_contains,
element_present, url_equals_exactly, text_contains) instead of
stale values (url, cookie, element, redirect)
- Authentication.login_flow: mark optional to match schema which
does not require it
* feat: mark GitHub release as latest during rollback
* fix: use native ARM64 runners for Docker multi-platform builds
Replace QEMU emulation with parallel native builds using a matrix
strategy (ubuntu-latest for amd64, ubuntu-24.04-arm for arm64).
Each platform pushes by digest, then a merge job creates the
multi-arch manifest list before signing with cosign.
* fix: resolve SessionMutex race condition with 3+ concurrent waiters
* fix: skip POSIX permission check on Windows
writeFileSync mode option is ignored on Windows, so config.toml
gets 0o666 and the guard rejects it.
* fix: resolve unsubstituted placeholders in report prompt
Remove unused {{GITHUB_URL}} placeholder and wire up {{AUTH_CONTEXT}}
with structured auth context (login type, username, URL, MFA status).
* fix: remove duplicate environment gate from merge-docker job
Move DOCKERHUB_USERNAME from vars to secrets so merge-docker can access
credentials without its own environment scope. This eliminates the
redundant double approval since build-docker already gates on
release-publish.
* fix: replace POSIX sleep binary with cross-platform async sleep
execFileSync('sleep') is unavailable on Windows. Use node:timers/promises
setTimeout instead, making ensureInfra async.
* fix: use session.json for workflow ID on resume instead of parsing workflow.log
On resume, workflow.log already exists with stale headers from the
previous run. The CLI poll found '====' immediately and extracted the
old workflow ID, producing a wrong Temporal Web UI URL.
Read the workflow ID from session.json instead — the worker writes
resume attempts there atomically. For fresh runs, poll until
originalWorkflowId appears. For resumes, poll until a new
resumeAttempts entry is appended.
* feat: add custom base URL support for Anthropic-compatible proxies
Support ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN to route SDK requests
through LiteLLM or any Anthropic-compatible proxy. Adds TUI wizard
option, TOML config mapping, credential validation, and preflight
endpoint reachability check via SDK query.
* fix: remove environment gates and add NPM_TOKEN to publish step
* feat: add beta release and rollback workflows with cosign signing
* fix: remove redundant checkout and pnpm steps from beta release workflow
* docs: normalize README commands to mode-neutral shorthand
Add a substitution note after Quick Start sections so all subsequent
examples use bare `shannon` instead of mixing `./shannon` and
`npx @keygraph/shannon`. Mode-specific commands (build, update,
uninstall) get inline annotations. Also fixes a broken command in the
Custom Base URL section.
* fix: remove redundant `update` command
Image is already auto-pulled by `ensureImage()` during `start` when the
pinned version tag is missing locally. Manual `update` was unnecessary.
* docs: add CLI package README stub
* docs: update README setup instructions for dual CLI modes
* docs: update announcement banner to npx availability
* feat: migrate from MCP tools to CLI based tools (#252)
* feat: migrate from MCP tools to CLI tools
* fix: restore browser action emoji formatters for CLI output
Adapt formatBrowserAction for playwright-cli commands, replacing the old
mcp__playwright__browser_* tool name matching removed during migration.
* fix: mount credential file to fixed container path for Vertex AI
GOOGLE_APPLICATION_CREDENTIALS was forwarded as-is to the container,
causing the relative host path to resolve against the repo mount
instead of the credentials mount. Now both local and npx modes mount
the resolved file to /app/credentials/google-sa-key.json and rewrite
the env var to match.
* feat: add git awareness and optional description field to config
* fix: drop redundant --ipc host flag from worker container
* fix: align announcement banner URL with main branch
* feat: add target URL reachability preflight check (#254)
* Moving asset benchmark graph image to this folder
* Move benchmark results to benchmark repo
Windows Defender flags exploit code in the pentest reports as false positives, forcing every Windows user to add a Defender exclusion just to clone Shannon.
* Updated README
* fix: case-insensitive grep for semantic-release version probe
* fix: harden supply chain security (#255)
* fix: patch smol-toml and tsdown vulnerabilities
Update smol-toml 1.6.0→1.6.1 (DoS via recursive comment parsing) and
tsdown 0.21.2→0.21.5 (picomatch ReDoS + method injection).
* fix: pin all unpinned dependency versions in Dockerfile
Pins subfinder v2.13.0, WhatWeb v0.6.3 (switched from git clone to
release tarball), schemathesis 4.13.0, addressable 2.8.9,
claude-code 2.1.84, and playwright-cli 0.1.1 for reproducible builds.
* fix: pin GitHub Actions to commit SHAs for supply chain security
* fix: pin GitHub Actions to commit SHAs in beta and rollback workflows
|