mirror of
https://github.com/KeygraphHQ/shannon.git
synced 2026-08-28 22:10:45 +02:00
fix(report): emit SARIF by default for exploit runs (#431)
* fix(report): emit SARIF by default for exploit runs, opt out with report.sarif: false * docs: describe SARIF as on-by-default for exploit runs
This commit is contained in:
ezl-keygraph
@@ -111,7 +111,7 @@ For source builds, authenticated scans, provider-specific setup, and platform no
|
||||
- **Authenticated testing**: configuration files can describe login flows, test credentials, TOTP, email-based login flows, focus areas, and rules of engagement.
|
||||
- **OWASP-focused coverage**: Shannon targets exploitable Injection, XSS, SSRF, Broken Authentication, and Broken Authorization issues.
|
||||
- **Resumable workspaces**: Shannon can resume interrupted runs without re-running completed agents.
|
||||
- **Machine-readable output**: Shannon emits findings as structured JSON, and as SARIF 2.1.0 when you enable it in configuration. SARIF is the OASIS standard for static analysis results, so findings flow into any code scanning service, vulnerability management platform, security dashboard, or CI/CD pipeline that reads it.
|
||||
- **Machine-readable output**: Shannon emits findings as structured JSON, and as SARIF 2.1.0 by default on exploit-mode scans (opt out with `report.sarif: "false"`). SARIF is the OASIS standard for static analysis results, so findings flow into any code scanning service, vulnerability management platform, security dashboard, or CI/CD pipeline that reads it.
|
||||
- **Bring your own key, provider-agnostic**: Shannon runs on Anthropic, OpenAI, xAI, AWS Bedrock, and any endpoint speaking the Anthropic Messages API or the OpenAI Chat Completions or Responses API, including self-hosted models served through Ollama, vLLM, or LM Studio and gateways such as OpenRouter and LiteLLM. You supply the credentials, so source code and model traffic stay inside your infrastructure. Local and self-hosted models are technically supported but not recommended: they may not follow Shannon's instructions or tool-use constraints as reliably as frontier models, so take that path only if you know how your chosen model behaves.
|
||||
|
||||
## Editions
|
||||
@@ -267,7 +267,7 @@ Yes, always. You provide the LLM credentials Shannon uses to run a pentest, in e
|
||||
|
||||
### Does Shannon output SARIF?
|
||||
|
||||
Yes. Shannon emits SARIF 2.1.0, the OASIS standard format for static analysis results, alongside structured JSON. Any SARIF consumer reads it: code scanning services, vulnerability management platforms, security dashboards, and CI/CD pipelines. Set `report.sarif` to `"true"` in your configuration file to enable the SARIF log.
|
||||
Yes. Shannon emits SARIF 2.1.0, the OASIS standard format for static analysis results, alongside structured JSON. Any SARIF consumer reads it: code scanning services, vulnerability management platforms, security dashboards, and CI/CD pipelines. It is written by default on exploit-mode scans; set `report.sarif` to `"false"` in your configuration file to opt out.
|
||||
|
||||
### Which AI providers does Shannon support?
|
||||
|
||||
|
||||
@@ -164,7 +164,7 @@
|
||||
"sarif": {
|
||||
"type": "string",
|
||||
"enum": ["true", "false"],
|
||||
"description": "Emit a SARIF 2.1.0 log (report.sarif) beside the report. Requires exploit=true; ignored otherwise."
|
||||
"description": "Emit a SARIF 2.1.0 log (report.sarif) beside the report. On by default for exploit runs; set \"false\" to opt out. Ignored when exploit=false."
|
||||
}
|
||||
},
|
||||
"additionalProperties": false
|
||||
|
||||
@@ -96,8 +96,9 @@ rules:
|
||||
# Report filters applied by the report agent when assembling the final report (optional).
|
||||
# Example below is illustrative; edit, remove, or add sections as needed.
|
||||
# report:
|
||||
# # Emit a SARIF 2.1.0 log (report.sarif) beside the report. Requires exploit: "true".
|
||||
# sarif: "true"
|
||||
# # SARIF 2.1.0 log (report.sarif) beside the report. On by default for exploit runs;
|
||||
# # set "false" to opt out. Ignored when exploit is "false".
|
||||
# sarif: "false"
|
||||
# min_severity: low
|
||||
# min_confidence: low
|
||||
# guidance: |
|
||||
|
||||
@@ -679,7 +679,8 @@ export const distributeConfig = (config: Config | null): DistributedConfig => {
|
||||
const exploit = config?.exploit !== undefined ? config.exploit === 'true' : true;
|
||||
|
||||
const report = {
|
||||
sarif: config?.report?.sarif === 'true',
|
||||
// Default on; only an explicit "false" opts out.
|
||||
sarif: config?.report?.sarif !== 'false',
|
||||
...(config?.report?.min_severity && { min_severity: config.report.min_severity }),
|
||||
...(config?.report?.min_confidence && { min_confidence: config.report.min_confidence }),
|
||||
...(config?.report?.guidance && { guidance: config.report.guidance.trim() }),
|
||||
|
||||
@@ -43,7 +43,7 @@ export const FINAL_REPORT_MD_FILENAME = 'Security-Assessment-Report.md';
|
||||
/** Structured findings the report agent emits; the markdown report is rendered from it. */
|
||||
export const REPORT_JSON_FILENAME = 'report.json';
|
||||
|
||||
/** SARIF 2.1.0 log, written only for exploit=true runs when report.sarif is enabled. */
|
||||
/** SARIF 2.1.0 log, written for exploit=true runs unless report.sarif is set to false. */
|
||||
export const SARIF_FILENAME = 'report.sarif';
|
||||
|
||||
/**
|
||||
|
||||
@@ -29,7 +29,7 @@ export function getAgentGitPaths(agentName: AgentName): string[] {
|
||||
paths.push(queueFilename);
|
||||
}
|
||||
// The report agent also emits the structured findings the markdown is rendered from, and the
|
||||
// SARIF log when enabled. Listing the log unconditionally is harmless when it was not written,
|
||||
// SARIF log when produced. Listing the log unconditionally is harmless when it was not written,
|
||||
// and keeps a stale one from surviving the rollback of a failed attempt.
|
||||
if (agentName === 'report') {
|
||||
paths.push(REPORT_JSON_FILENAME);
|
||||
|
||||
@@ -181,7 +181,7 @@ export async function injectModelIntoReport(
|
||||
*
|
||||
* The SARIF log is surfaced beside it when present, since a CI step consuming it needs a stable
|
||||
* path and cannot be expected to reach into the internals directory. It is absent whenever the
|
||||
* run was analysis-only or `report.sarif` was not enabled.
|
||||
* run was analysis-only or `report.sarif` was set to false.
|
||||
*/
|
||||
export async function copyReportToRunRoot(
|
||||
repoPath: string,
|
||||
|
||||
@@ -450,13 +450,14 @@ export async function runAuthzExploitAgent(input: ActivityInput): Promise<AgentM
|
||||
}
|
||||
|
||||
/**
|
||||
* Write report.sarif when the run is exploitative and the operator asked for it.
|
||||
* Write report.sarif for exploitative runs unless the operator opted out with report.sarif: false.
|
||||
*
|
||||
* Skipped entirely for analysis-only runs. The original reason was that those findings carried
|
||||
* no severity, so every `result.level` would have been invented; since severity is recorded in
|
||||
* both modes an analysis run could now populate `level`, but it would report an assessed
|
||||
* severity as a measured one, so the gate stays. Failures are logged and swallowed — the SARIF
|
||||
* log is a secondary artifact and must not fail a run whose report is already written.
|
||||
* On by default so a CI step consuming the log always finds one. Skipped for analysis-only runs.
|
||||
* The original reason was that those findings carried no severity, so every `result.level` would
|
||||
* have been invented; since severity is recorded in both modes an analysis run could now populate
|
||||
* `level`, but it would report an assessed severity as a measured one, so the gate stays. Failures
|
||||
* are logged and swallowed — the SARIF log is a secondary artifact and must not fail a run whose
|
||||
* report is already written.
|
||||
*/
|
||||
async function writeSarifIfEnabled(
|
||||
input: ActivityInput,
|
||||
@@ -469,7 +470,8 @@ async function writeSarifIfEnabled(
|
||||
|
||||
const container = getOrCreateContainer(input.workflowId, buildSessionMetadata(input), buildContainerConfig(input));
|
||||
const configResult = await container.configLoader.loadOptional(input.configPath, undefined, input.configYAML);
|
||||
if (isErr(configResult) || configResult.value?.report?.sarif !== true) return;
|
||||
// Only an explicit false opts out; a missing config keeps the default on.
|
||||
if (isErr(configResult) || configResult.value?.report?.sarif === false) return;
|
||||
|
||||
try {
|
||||
const { renderSarif } = await import('../services/sarif-renderer.js');
|
||||
|
||||
@@ -32,7 +32,10 @@ export interface ReportConfig {
|
||||
min_severity?: Severity;
|
||||
min_confidence?: Confidence;
|
||||
guidance?: string;
|
||||
/** Emit report.sarif alongside the markdown report. Ignored when exploit is false. */
|
||||
/**
|
||||
* Emit report.sarif alongside the markdown report. On by default for exploit runs; set 'false'
|
||||
* to opt out. Ignored when exploit is false.
|
||||
*/
|
||||
sarif?: 'true' | 'false';
|
||||
}
|
||||
|
||||
|
||||
@@ -99,7 +99,7 @@ rules:
|
||||
# min_confidence: low
|
||||
# guidance: |
|
||||
# Drop findings about missing security headers and rate-limit gaps.
|
||||
# sarif: "true"
|
||||
# sarif: "false"
|
||||
```
|
||||
|
||||
## Report Options
|
||||
@@ -109,18 +109,17 @@ rules:
|
||||
| `min_severity` | Drops findings rated below this severity. Applies in both exploitative and analysis-only runs. |
|
||||
| `min_confidence` | Drops findings rated below this confidence. Applies only when `exploit` is `"false"`. |
|
||||
| `guidance` | Free-text instruction to the report agent, such as which topics to exclude. |
|
||||
| `sarif` | Emits a SARIF 2.1.0 log alongside the Markdown report. Requires `exploit: "true"`. |
|
||||
| `sarif` | SARIF 2.1.0 log alongside the Markdown report. On by default for exploit runs; set `"false"` to opt out. Ignored when `exploit` is `"false"`. |
|
||||
|
||||
Every finding carries a severity, but it does not mean the same thing in each mode: an exploitative run measures severity from what the exploit demonstrated, while an analysis-only run assesses it from the class of flaw and the impact it would have. An analysis-only finding carries a confidence rating alongside its severity, since nothing was proven. Setting `min_confidence` on an exploitative run is ignored, and Shannon logs a warning naming the threshold to use instead.
|
||||
|
||||
### SARIF Output
|
||||
|
||||
Set `sarif: "true"` to write `report.sarif` next to `Security-Assessment-Report.pdf` at the workspace root, for upload to GitHub code scanning or any other SARIF consumer.
|
||||
On exploit-mode runs Shannon writes `report.sarif` next to `Security-Assessment-Report.pdf` at the workspace root by default, for upload to GitHub code scanning or any other SARIF consumer. No configuration is needed; set `sarif: "false"` to opt out.
|
||||
|
||||
```yaml
|
||||
exploit: "true"
|
||||
report:
|
||||
sarif: "true"
|
||||
sarif: "false"
|
||||
```
|
||||
|
||||
Each finding becomes one SARIF result, filed under a rule per vulnerability class (`shannon/injection`, `shannon/xss`, `shannon/auth`, `shannon/authz`, `shannon/ssrf`) and tagged with its OWASP Top Ten 2025 category. Results are anchored to the code location the analysis phase recorded, falling back to the HTTP entry point when the finding names no file. Severity maps onto SARIF's three levels: `critical` and `high` become `error`, `medium` becomes `warning`, everything else becomes `note`.
|
||||
|
||||
+6
-7
@@ -120,7 +120,7 @@ For source builds, authenticated scans, provider-specific setup, and platform no
|
||||
- **Authenticated testing**: configuration files can describe login flows, test credentials, TOTP, email-based login flows, focus areas, and rules of engagement.
|
||||
- **OWASP-focused coverage**: Shannon targets exploitable Injection, XSS, SSRF, Broken Authentication, and Broken Authorization issues.
|
||||
- **Resumable workspaces**: Shannon can resume interrupted runs without re-running completed agents.
|
||||
- **Machine-readable output**: Shannon emits findings as structured JSON, and as SARIF 2.1.0 when you enable it in configuration. SARIF is the OASIS standard for static analysis results, so findings flow into any code scanning service, vulnerability management platform, security dashboard, or CI/CD pipeline that reads it.
|
||||
- **Machine-readable output**: Shannon emits findings as structured JSON, and as SARIF 2.1.0 by default on exploit-mode scans (opt out with `report.sarif: "false"`). SARIF is the OASIS standard for static analysis results, so findings flow into any code scanning service, vulnerability management platform, security dashboard, or CI/CD pipeline that reads it.
|
||||
- **Bring your own key, provider-agnostic**: Shannon runs on Anthropic, OpenAI, xAI, AWS Bedrock, and any endpoint speaking the Anthropic Messages API or the OpenAI Chat Completions or Responses API, including self-hosted models served through Ollama, vLLM, or LM Studio and gateways such as OpenRouter and LiteLLM. You supply the credentials, so source code and model traffic stay inside your infrastructure. Local and self-hosted models are technically supported but not recommended: they may not follow Shannon's instructions or tool-use constraints as reliably as frontier models, so take that path only if you know how your chosen model behaves.
|
||||
|
||||
## Editions
|
||||
@@ -280,7 +280,7 @@ Yes, always. You provide the LLM credentials Shannon uses to run a pentest, in e
|
||||
|
||||
### Does Shannon output SARIF?
|
||||
|
||||
Yes. Shannon emits SARIF 2.1.0, the OASIS standard format for static analysis results, alongside structured JSON. Any SARIF consumer reads it: code scanning services, vulnerability management platforms, security dashboards, and CI/CD pipelines. Set `report.sarif` to `"true"` in your configuration file to enable the SARIF log.
|
||||
Yes. Shannon emits SARIF 2.1.0, the OASIS standard format for static analysis results, alongside structured JSON. Any SARIF consumer reads it: code scanning services, vulnerability management platforms, security dashboards, and CI/CD pipelines. It is written by default on exploit-mode scans; set `report.sarif` to `"false"` in your configuration file to opt out.
|
||||
|
||||
### Which AI providers does Shannon support?
|
||||
|
||||
@@ -567,7 +567,7 @@ rules:
|
||||
# min_confidence: low
|
||||
# guidance: |
|
||||
# Drop findings about missing security headers and rate-limit gaps.
|
||||
# sarif: "true"
|
||||
# sarif: "false"
|
||||
```
|
||||
|
||||
## Report Options
|
||||
@@ -577,18 +577,17 @@ rules:
|
||||
| `min_severity` | Drops findings rated below this severity. Applies in both exploitative and analysis-only runs. |
|
||||
| `min_confidence` | Drops findings rated below this confidence. Applies only when `exploit` is `"false"`. |
|
||||
| `guidance` | Free-text instruction to the report agent, such as which topics to exclude. |
|
||||
| `sarif` | Emits a SARIF 2.1.0 log alongside the Markdown report. Requires `exploit: "true"`. |
|
||||
| `sarif` | SARIF 2.1.0 log alongside the Markdown report. On by default for exploit runs; set `"false"` to opt out. Ignored when `exploit` is `"false"`. |
|
||||
|
||||
Every finding carries a severity, but it does not mean the same thing in each mode: an exploitative run measures severity from what the exploit demonstrated, while an analysis-only run assesses it from the class of flaw and the impact it would have. An analysis-only finding carries a confidence rating alongside its severity, since nothing was proven. Setting `min_confidence` on an exploitative run is ignored, and Shannon logs a warning naming the threshold to use instead.
|
||||
|
||||
### SARIF Output
|
||||
|
||||
Set `sarif: "true"` to write `report.sarif` next to `Security-Assessment-Report.pdf` at the workspace root, for upload to GitHub code scanning or any other SARIF consumer.
|
||||
On exploit-mode runs Shannon writes `report.sarif` next to `Security-Assessment-Report.pdf` at the workspace root by default, for upload to GitHub code scanning or any other SARIF consumer. No configuration is needed; set `sarif: "false"` to opt out.
|
||||
|
||||
```yaml
|
||||
exploit: "true"
|
||||
report:
|
||||
sarif: "true"
|
||||
sarif: "false"
|
||||
```
|
||||
|
||||
Each finding becomes one SARIF result, filed under a rule per vulnerability class (`shannon/injection`, `shannon/xss`, `shannon/auth`, `shannon/authz`, `shannon/ssrf`) and tagged with its OWASP Top Ten 2025 category. Results are anchored to the code location the analysis phase recorded, falling back to the HTTP entry point when the finding names no file. Severity maps onto SARIF's three levels: `critical` and `high` become `error`, `medium` becomes `warning`, everything else becomes `note`.
|
||||
|
||||
Reference in New Issue
Block a user