feat(worker): record severity in analysis mode and fix prompt substitutions (#413)

* feat(worker): record severity in analysis mode alongside confidence

* fix(worker): align add_finding severity with the exploit collector's four levels

* refactor(worker): drop the dead REPORT_VULN_HEADING substitution

No prompt in the tree uses the placeholder, so the replacement was a no-op
on every render.

* fix(worker): strip all whitespace from TOTP secrets, not just the ends

* fix(worker): render rule type and value in the agent prompt

* refactor(worker): drop the dead vuln-summary subsection substitution

apps/worker/configs/config-schema.json

@@ -206,7 +206,6 @@
       "properties": {
         "description": {
           "type": "string",
-          "minLength": 1,
           "maxLength": 200,
           "description": "Human-readable description of the rule"
         },
@@ -222,7 +221,7 @@
           "description": "Value to match"
         }
       },
-      "required": ["description", "type", "value"],
+      "required": ["type", "value"],
       "additionalProperties": false
     }
   }

apps/worker/prompts/report-executive.txt

@@ -89,7 +89,7 @@ Fields:
 - `executive_summary`: 2-3 sentences summarizing the security posture for technical leadership (CTOs, CISOs, Engineering VPs). Must include the target URL and assessment date. Provide a high-level characterization based on the findings — severity distribution, most critical issues, and overall risk demonstrated by exploitation. If no vulnerabilities were confirmed in the assessed classes, state that scope clearly. A clean report is valid only when no <not_assessed_classes> block is present. If that block is present, explicitly say the listed classes were not assessed and do not assert they are free of vulnerabilities.
 </exploit_mode_summary>
 <analysis_mode_summary>
-- `executive_summary`: 2-3 sentences summarizing the security posture for technical leadership (CTOs, CISOs, Engineering VPs). Must include the target URL and assessment date. Provide a high-level characterization based on the findings — confidence distribution, the most serious weaknesses identified, and overall risk. State plainly that this was an analysis-only assessment and that no finding was confirmed by exploitation; do not describe risk as demonstrated or proven. Findings carry no severity rating in this mode, so do not assert one. If no vulnerabilities were identified in the assessed classes, state that scope clearly. A clean report is valid only when no <not_assessed_classes> block is present. If that block is present, explicitly say the listed classes were not assessed and do not assert they are free of vulnerabilities.
+- `executive_summary`: 2-3 sentences summarizing the security posture for technical leadership (CTOs, CISOs, Engineering VPs). Must include the target URL and assessment date. Provide a high-level characterization based on the findings — severity and confidence distribution, the most serious weaknesses identified, and overall risk. State plainly that this was an analysis-only assessment and that no finding was confirmed by exploitation; do not describe risk as demonstrated or proven, and present severity as assessed rather than measured. If no vulnerabilities were identified in the assessed classes, state that scope clearly. A clean report is valid only when no <not_assessed_classes> block is present. If that block is present, explicitly say the listed classes were not assessed and do not assert they are free of vulnerabilities.
 </analysis_mode_summary>
 </record_report_meta>
 
@@ -112,6 +112,7 @@ Map the finding's content from the per-class deliverable sections to `add_findin
 </exploit_mode_fields>
 <analysis_mode_fields>
 - `confidence`: From the finding's "Confidence" field. Use as-is; do not reassess.
+- `severity`: The analysis deliverables carry no severity field — no exploit ran to measure impact. Assess it from the vulnerability class and the impact you describe. It is an assessed rating, not a measured one.
 </analysis_mode_fields>
 - `owasp_category`: Map to the appropriate OWASP Top 10 (2025) category:
   - `"A01:2025 — Broken Access Control"`
@@ -139,7 +140,7 @@ Map the finding's content from the per-class deliverable sections to `add_findin
 <analysis_mode_fields>
 - `impact`: What an attacker could achieve if this vulnerability were exploited. Derive it from the finding's "Impact" and "Overview" fields. Write it as assessed, never as achieved.
 
-This run had no exploitation phase. Nothing was executed against the target, nothing was demonstrated, and no exploit evidence exists. Accordingly `severity`, `auth_state`, `prerequisites`, `exploitation_steps`, `proof_of_impact` and `status` are **not** part of your tool schema — the deliverables contain no source for any of them. `confidence` is the only rating this run produces; take it straight from the deliverable. Do not compensate for the missing fields by describing attack execution in `overview`, `impact` or `notes`. Report the weakness and how to fix it; that is the whole deliverable for this run.
+This run had no exploitation phase. Nothing was executed against the target, nothing was demonstrated, and no exploit evidence exists. Accordingly `auth_state`, `prerequisites`, `exploitation_steps`, `proof_of_impact` and `status` are **not** part of your tool schema — the deliverables contain no source for any of them. `confidence` is the deliverable's own rating and carries over verbatim; `severity` is yours to assess, since nothing measured it. Do not compensate for the missing fields by describing attack execution in `overview`, `impact` or `notes`. Report the weakness and how to fix it; that is the whole deliverable for this run.
 </analysis_mode_fields>
 
 **Optional fields:**
@@ -160,6 +161,7 @@ If no valid findings exist after filtering, do not call `add_finding` at all. Th
 - **No Fabrications:** Every piece of data must come from the deliverable files. If a finding has incomplete data, include it but note the gap in `overview`.
 - **Nothing Was Demonstrated:** No exploit ran. Do not write that a vulnerability was confirmed, proven, exploited, or verified against the running target, and do not describe payloads, requests, or responses as having been sent.
 - **No Confidence Changes:** Use the confidence from the deliverable as-is. Do not raise or lower it.
+- **Severity Is Assessed:** Rate severity from the vulnerability class and the impact you describe. Never present it as measured or demonstrated.
 </analysis_mode_constraints>
 - **No Speculation:** Only record findings that appear in the deliverables with valid vulnerability IDs. Do not add your own assessments.
 - **OWASP 2025:** Map all findings to OWASP Top 10 (2025) categories.
@@ -187,6 +189,7 @@ Before finalizing, verify:
 </exploit_mode_checks>
 <analysis_mode_checks>
 - [ ] Does every finding have `confidence` carried over unchanged from the deliverable?
+- [ ] Is every `severity` assessed from the impact I described, with no claim that it was measured?
 - [ ] Is every `impact` phrased as assessed rather than demonstrated, with no claim that anything was executed?
 </analysis_mode_checks>
 - [ ] Are remediation recommendations specific and actionable (not generic)?

apps/worker/src/collectors/finding-collector.ts

@@ -37,7 +37,7 @@ const OWASP_CATEGORY_VALUES = [
   'A10:2025 — Mishandling of Exceptional Conditions',
 ] as const;
 
-const SEVERITY_VALUES = ['critical', 'high', 'medium', 'low', 'informational'] as const;
+const SEVERITY_VALUES = ['critical', 'high', 'medium', 'low'] as const;
 const STATUS_VALUES = ['exploited', 'out_of_scope', 'blocked_by_constraints', 'false_positive'] as const;
 const CONFIDENCE_VALUES = ['high', 'medium', 'low'] as const;
 
@@ -117,8 +117,18 @@ const AdditionalSectionSchema = Type.Object({
   }),
 });
 
-function identityFields() {
+/**
+ * `severity` is recorded in both modes, but it does not mean the same thing in each: an exploit
+ * run measures it from what the exploit demonstrated, an analysis run assesses it from the class
+ * of flaw. The description says which, so the agent never presents an assessment as a measurement.
+ */
+function identityFields(exploit: boolean) {
+  const severityDescription = exploit
+    ? 'Severity of the finding, based on the impact the exploit demonstrated.'
+    : 'Severity of the finding, assessed from the vulnerability class and the impact it would have.';
+
   return {
+    severity: stringEnum(SEVERITY_VALUES, { description: severityDescription }),
     finding_id: Type.String({
       minLength: 1,
       description: 'Finding identifier (e.g., "AUTH-VULN-07", "INJ-VULN-03"). Must be unique per report.',
@@ -178,9 +188,6 @@ function narrativeFields(exploit: boolean) {
 /** Fields that only mean something once an exploit has run. Absent from the analysis schema. */
 function exploitOnlyFields() {
   return {
-    severity: stringEnum(SEVERITY_VALUES, {
-      description: 'Severity of the finding, based on the impact the exploit demonstrated.',
-    }),
     auth_state: Type.String({
       minLength: 1,
       description: 'Authentication state during testing (e.g., "Unauthenticated", "Any authenticated user").',
@@ -205,7 +212,7 @@ function exploitOnlyFields() {
   };
 }
 
-/** Replaces `severity` when nothing was exploited. */
+/** Accompanies `severity` when nothing was exploited — the rating the analysis deliverable itself carries. */
 function analysisOnlyFields() {
   return {
     confidence: stringEnum(CONFIDENCE_VALUES, {
@@ -233,7 +240,7 @@ function sharedOptionalFields() {
 
 export function buildAddFindingSchema(exploit: boolean) {
   return Type.Object({
-    ...identityFields(),
+    ...identityFields(exploit),
     ...(exploit ? exploitOnlyFields() : analysisOnlyFields()),
     ...locationFields(),
     ...narrativeFields(exploit),
@@ -243,12 +250,12 @@ export function buildAddFindingSchema(exploit: boolean) {
 
 /**
  * Superset of both modes, for typing only. Consumers must check presence rather than assume:
- * `report.json` from an analysis run has no `severity` or `exploitation_steps` key at all.
+ * `report.json` from an analysis run has no `exploitation_steps` key at all. `severity` is the
+ * exception — both modes record it, so it is required here too.
  */
 const AddFindingSupersetSchema = Type.Object({
-  ...identityFields(),
+  ...identityFields(true),
   code_locations: Type.Optional(Type.Array(CodeLocationSchema)),
-  severity: Type.Optional(stringEnum(SEVERITY_VALUES)),
   auth_state: Type.Optional(Type.String()),
   prerequisites: Type.Optional(Type.String()),
   exploitation_steps: Type.Optional(Type.Array(StructuredStepSchema)),

apps/worker/src/config-parser.ts

@@ -514,7 +514,7 @@ const validateRulesSecurity = (rules: Rule[] | undefined, ruleType: string): voi
           ErrorCode.CONFIG_VALIDATION_FAILED,
         );
       }
-      if (pattern.test(rule.description)) {
+      if (rule.description !== undefined && pattern.test(rule.description)) {
         throw new PentestError(
           `rules.${ruleType}[${index}].description contains potentially dangerous pattern: ${pattern.source}`,
           'config',
@@ -656,11 +656,15 @@ const checkForConflicts = (avoidRules: Rule[] = [], focusRules: Rule[] = []): vo
 };
 
 const sanitizeRule = (rule: Rule): Rule => {
-  return {
-    description: rule.description.trim(),
+  const sanitized: Rule = {
     type: rule.type.toLowerCase().trim() as Rule['type'],
     value: rule.value.trim(),
   };
+  const description = rule.description?.trim();
+  if (description) {
+    sanitized.description = description;
+  }
+  return sanitized;
 };
 
 export const distributeConfig = (config: Config | null): DistributedConfig => {
@@ -702,13 +706,15 @@ const sanitizeAuthentication = (auth: Authentication): Authentication => {
     credentials: {
       username: auth.credentials.username.trim(),
       ...(auth.credentials.password && { password: auth.credentials.password }),
-      ...(auth.credentials.totp_secret && { totp_secret: auth.credentials.totp_secret.trim() }),
+      ...(auth.credentials.totp_secret && {
+        totp_secret: auth.credentials.totp_secret.replace(/\s/g, ''),
+      }),
       ...(auth.credentials.email_login && {
         email_login: {
           address: auth.credentials.email_login.address.trim(),
           password: auth.credentials.email_login.password,
           ...(auth.credentials.email_login.totp_secret && {
-            totp_secret: auth.credentials.email_login.totp_secret.trim(),
+            totp_secret: auth.credentials.email_login.totp_secret.replace(/\s/g, ''),
           }),
         },
       }),

apps/worker/src/services/preflight.ts

@@ -179,7 +179,7 @@ type RuleKind = 'avoid' | 'focus';
 interface MissingCodePath {
   kind: RuleKind;
   value: string;
-  description: string;
+  description?: string;
 }
 
 async function validateCodePathsExist(
@@ -202,12 +202,16 @@ async function validateCodePathsExist(
   const missing: MissingCodePath[] = [];
   for (const { kind, rule } of tagged) {
     if (!(await patternMatchesAny(repoPath, rule.value))) {
-      missing.push({ kind, value: rule.value, description: rule.description });
+      const entry: MissingCodePath = { kind, value: rule.value };
+      if (rule.description) {
+        entry.description = rule.description;
+      }
+      missing.push(entry);
     }
   }
 
   if (missing.length > 0) {
-    const lines = missing.map((m) => `[${m.kind}] '${m.value}' — ${m.description}`);
+    const lines = missing.map((m) => `[${m.kind}] '${m.value}'${m.description ? ` - ${m.description}` : ''}`);
     return err(
       new PentestError(
         `code_path rules don't match any file or directory in the repo:\n  - ${lines.join('\n  - ')}\n` +

apps/worker/src/services/prompt-manager.ts

@@ -12,61 +12,32 @@ import type { Authentication, DistributedConfig, DistributedReportConfig, Rule,
 import { isGlobPattern } from '../utils/glob.js';
 import { handlePromptError, PentestError } from './error-handling.js';
 
+function renderRuleLine(tag: string, value: string, description?: string): string {
+  const base = `- ${tag} ${value}`;
+  return description ? `${base} - ${description}` : base;
+}
+
+function renderUrlRules(rules: Rule[]): string {
+  if (rules.length === 0) return 'None';
+  return rules.map((r) => renderRuleLine(`[${r.type.toUpperCase()}]`, r.value, r.description)).join('\n');
+}
+
 function renderCodePathRules(rules: Rule[]): string {
   const filtered = rules.filter((r) => r.type === 'code_path');
   if (filtered.length === 0) return 'None';
   return filtered
-    .map((r) => {
-      const kind = isGlobPattern(r.value) ? '[GLOB]' : '[FILE]';
-      return `- ${r.value} ${kind} — ${r.description}`;
-    })
+    .map((r) => renderRuleLine(isGlobPattern(r.value) ? '[GLOB]' : '[FILE]', r.value, r.description))
     .join('\n');
 }
 
-interface VulnSummarySpec {
-  readonly heading: string;
-  readonly evidenceSection: string;
-  readonly noneFoundLabel: string;
-}
-
-const VULN_SUMMARY_SPECS: Record<VulnClass, VulnSummarySpec> = {
-  auth: {
-    heading: 'Authentication Vulnerabilities',
-    evidenceSection: 'Authentication Exploitation Evidence',
-    noneFoundLabel: 'authentication',
-  },
-  authz: {
-    heading: 'Authorization Vulnerabilities',
-    evidenceSection: 'Authorization Exploitation Evidence',
-    noneFoundLabel: 'authorization',
-  },
-  xss: {
-    heading: 'Cross-Site Scripting (XSS) Vulnerabilities',
-    evidenceSection: 'XSS Exploitation Evidence',
-    noneFoundLabel: 'XSS',
-  },
-  injection: {
-    heading: 'SQL/Command Injection Vulnerabilities',
-    evidenceSection: 'Injection Exploitation Evidence',
-    noneFoundLabel: 'SQL or command injection',
-  },
-  ssrf: {
-    heading: 'Server-Side Request Forgery (SSRF) Vulnerabilities',
-    evidenceSection: 'SSRF Exploitation Evidence',
-    noneFoundLabel: 'SSRF',
-  },
+const VULN_CLASS_HEADINGS: Record<VulnClass, string> = {
+  auth: 'Authentication Vulnerabilities',
+  authz: 'Authorization Vulnerabilities',
+  xss: 'Cross-Site Scripting (XSS) Vulnerabilities',
+  injection: 'SQL/Command Injection Vulnerabilities',
+  ssrf: 'Server-Side Request Forgery (SSRF) Vulnerabilities',
 };
 
-function renderVulnSummarySubsections(selected: readonly VulnClass[]): string {
-  const classes = selected.length > 0 ? selected : (Object.keys(VULN_SUMMARY_SPECS) as VulnClass[]);
-  return classes
-    .map((cls) => {
-      const spec = VULN_SUMMARY_SPECS[cls];
-      return `**${spec.heading}:**\n{Check for "${spec.evidenceSection}" section. Include actually exploited vulnerabilities and those blocked by security controls. Exclude theoretical vulnerabilities requiring internal network access. If vulnerabilities exist, summarize their impact and severity. If section is missing or empty, state: "No ${spec.noneFoundLabel} vulnerabilities were found."}`;
-    })
-    .join('\n\n');
-}
-
 /**
  * Renders the <not_assessed_classes> block. Empty when every class completed.
  *
@@ -86,9 +57,8 @@ function renderNotAssessedClassesBlock(failed: readonly VulnClass[] = []): strin
   ];
 
   for (const cls of classes) {
-    const spec = VULN_SUMMARY_SPECS[cls];
     lines.push(
-      `- ${spec.heading}: analysis did not complete; this class was NOT assessed. Absence of findings here does not indicate the class is clean.`,
+      `- ${VULN_CLASS_HEADINGS[cls]}: analysis did not complete; this class was NOT assessed. Absence of findings here does not indicate the class is clean.`,
     );
   }
 
@@ -103,13 +73,13 @@ function renderNotAssessedClassesBlock(failed: readonly VulnClass[] = []): strin
 /**
  * Which configured filters this run can actually enforce.
  *
- * The two ratings are mode-exclusive (see ../collectors/finding-collector.ts): an exploited
- * finding carries `severity`, an analysed one carries `confidence`. Handing the agent a
- * threshold for the rating its findings do not have is a directive it cannot honor.
+ * Every finding carries `severity` (see ../collectors/finding-collector.ts), so a severity
+ * threshold always applies. `confidence` exists only on an analysed finding — handing an
+ * exploit run a confidence threshold is a directive it cannot honor.
  */
 function applicableFilters(report: DistributedReportConfig | undefined, exploitEnabled: boolean) {
   return {
-    severity: Boolean(report?.min_severity) && exploitEnabled,
+    severity: Boolean(report?.min_severity),
     confidence: Boolean(report?.min_confidence) && !exploitEnabled,
     guidance: Boolean(report?.guidance?.trim()),
   };
@@ -375,8 +345,8 @@ async function interpolateVariables(
     if (avoidUrlRules.length === 0 && focusUrlRules.length === 0) {
       result = result.replace(/<rules>[\s\S]*?<\/rules>\s*/g, '');
     } else {
-      const avoidStr = avoidUrlRules.length > 0 ? avoidUrlRules.map((r) => `- ${r.description}`).join('\n') : 'None';
-      const focusStr = focusUrlRules.length > 0 ? focusUrlRules.map((r) => `- ${r.description}`).join('\n') : 'None';
+      const avoidStr = renderUrlRules(avoidUrlRules);
+      const focusStr = renderUrlRules(focusUrlRules);
       result = replaceLiteral(result, /{{RULES_AVOID}}/g, avoidStr);
       result = replaceLiteral(result, /{{RULES_FOCUS}}/g, focusStr);
     }
@@ -416,7 +386,6 @@ async function interpolateVariables(
       /{{VULN_CLASSES_TESTED}}/g,
       vulnClasses.length > 0 ? vulnClasses.join(', ') : 'injection, xss, auth, authz, ssrf',
     );
-    result = replaceLiteral(result, /{{VULN_SUMMARY_SUBSECTIONS}}/g, renderVulnSummarySubsections(vulnClasses));
     result = replaceLiteral(
       result,
       /{{NOT_ASSESSED_CLASSES}}/g,
@@ -432,19 +401,12 @@ async function interpolateVariables(
     result = result.replace(/<\/?(?:exploit|analysis)_mode_[a-z_]+>\n?/g, '');
 
     result = replaceLiteral(result, /{{EXPLOITATION}}/g, exploitEnabled ? 'enabled' : 'disabled');
-    result = replaceLiteral(result, /{{REPORT_VULN_HEADING}}/g, exploitEnabled ? 'Exploitation Evidence' : 'Findings');
     result = replaceLiteral(
       result,
       /{{REPORT_VULN_SUBHEADING}}/g,
       exploitEnabled ? 'Successfully Exploited Vulnerabilities' : 'Identified Vulnerabilities',
     );
 
-    if (config?.report?.min_severity && !exploitEnabled) {
-      logger.warn(
-        `report.min_severity="${config.report.min_severity}" is ignored when exploit=false: an ` +
-          'analysis-only run rates findings by confidence, not severity. Use report.min_confidence.',
-      );
-    }
     if (config?.report?.min_confidence && exploitEnabled) {
       logger.warn(
         `report.min_confidence="${config.report.min_confidence}" is ignored when exploit=true: an ` +

apps/worker/src/services/report-renderer.ts

@@ -263,7 +263,16 @@ export function renderReport(data: ReportData): string {
     sections.push(`### ${cat}`);
     sections.push('');
     for (const f of catFindings) {
-      const suffix = f.severity ? ` (${titleCase(f.severity)})` : '';
+      // Both ratings when the mode produced both. Confidence is labelled so it is never
+      // read as a severity in the position where a severity usually sits.
+      const ratings: string[] = [];
+      if (f.severity) {
+        ratings.push(titleCase(f.severity));
+      }
+      if (f.confidence) {
+        ratings.push(`${titleCase(f.confidence)} confidence`);
+      }
+      const suffix = ratings.length > 0 ? ` (${ratings.join(', ')})` : '';
       sections.push(`- **${f.finding_id}:** ${f.title}${suffix}`);
     }
     sections.push('');

apps/worker/src/types/config.ts

@@ -11,7 +11,7 @@
 export type RuleType = 'url_path' | 'subdomain' | 'domain' | 'method' | 'header' | 'parameter' | 'code_path';
 
 export interface Rule {
-  description: string;
+  description?: string;
   type: RuleType;
   value: string;
 }