NeuroSploit/benchmarks/typesafe-2026-09-20/README.md

NeuroSploit × TypeSafe — benchmark (2026-09-20)

Two identical NeuroSploit engagements against the same vulnerable target — one plain, one with TypeSafe System One (Jev) as a calibrated confirmation layer. Same model, same focus, same 13 seeded vulnerabilities. Only the --typesafe flag differs.

Open report.html for the full visual write-up.

Setup

Harness	NeuroSploit v4.0.0
Model	claude-opus-4-8 (subscription)
Target	NimbusCart / BenchMarkBurpAT · http://localhost:3000
Mode	black-box, --recon 2, --vote-n 1, --max-agents 15
Ground truth	13 seeded scenarios (IDOR/BOLA, SQLi ×5, XSS ×4, open redirect, CRLF)
Solver	none — the LLM discovered and confirmed everything live

Run commands (the only difference is --typesafe):

# A — no TypeSafe
NEUROSPLOIT_TYPESAFE=off neurosploit run http://localhost:3000 \
  --subscription --model anthropic:claude-opus-4-8 \
  --typesafe off --recon 2 --max-agents 15 --vote-n 1 --focus "<13 endpoints>" -v

# B — with TypeSafe (TYPESAFE_API_KEY set in env, never committed)
NEUROSPLOIT_TYPESAFE=on  neurosploit run http://localhost:3000 \
  --subscription --model anthropic:claude-opus-4-8 \
  --typesafe on  --recon 2 --max-agents 15 --vote-n 1 --focus "<13 endpoints>" -v

Result

Metric	A — no TypeSafe	B — TypeSafe
Targets hit	10 / 13	9 / 13
Findings	16	18
Wall-clock	32m 12s	26m 53s
Criticals	5	2 (recalibrated)
Belief-gate holds (POMDP)	3	—
Assurance P1–P5	all present	all present
Model cost	$0 (subscription)	$0 + TypeSafe ≪ $5

Union coverage (both runs): 11 / 13. Neither reached web_sqli_second_order or web_crlf_header_go.

Reading it honestly

• Recall is a tie — 10 vs 9 is within run-to-run variance at vote-n 1. TypeSafe is a judgment layer, not a recall multiplier.
• B surfaced 2 real net-new findings the plain run missed (config.json API-key exposure CWE-200, no-lockout brute force CWE-307) and caught web_idor_invoice.
• TypeSafe recalibrated severity — 5 class-inflated Criticals → 2 evidence- backed ones. On this target it under-rated one genuine critical (the BOLA credential dump: A = Critical 9.1, B = Low). Calibration is a dial toward defensibility, not a correctness oracle.
• Harness gap found & fixed: an earlier B collapsed to 0 findings when the subscription hit a session limit mid-run — NeuroSploit treated the limit message as a normal (exit-0) response and burned every agent. Now the session-limit sentinel parks the run (fix(models)).

Confounders

Single samples, not averages. vote-n 1 = no cross-model agreement in either arm. Recall scored by class + endpoint-keyword match (coverage, not graded proof). One target. Treat as one honest data point, not a leaderboard.

Files

report.html          the visual write-up
score.py             the scorer (class + endpoint keyword match vs the 13 targets)
scores.txt           scorer output for both runs
run_a_no_typesafe/   findings.json · assurance.json · meta.json · report.html · run.log
run_b_typesafe/      findings.json · assurance.json · meta.json · report.html · run.log

The TypeSafe API key and any subscription tokens are not in these files (env-only during the runs; verified clean before commit).