From 391e4c1e7c419998309123a6199577158cde9a6d Mon Sep 17 00:00:00 2001 From: Victor Kuznetsov Date: Mon, 10 Aug 2026 00:38:14 -0700 Subject: [PATCH] Record closed-corpus SynthID detector ablations --- docs/synthid-detector-removal-plan.md | 39 +++++++++++++++++++++++++++ docs/synthid.md | 11 ++++++++ 2 files changed, 50 insertions(+) diff --git a/docs/synthid-detector-removal-plan.md b/docs/synthid-detector-removal-plan.md index 2a0ba44..c81359e 100644 --- a/docs/synthid-detector-removal-plan.md +++ b/docs/synthid-detector-removal-plan.md @@ -985,6 +985,45 @@ late-fusion ablation for a larger paired corpus; complex spectral analysis remains a residual-discovery diagnostic unless a future held-out test reverses this result. +### 2026-08-09: closed-corpus architecture and score ablations + +A temporally split closed three-class corpus then tested whether the open-proxy +architecture result transferred to real external-source images. These labels +establish source classes, not watermark presence; they can measure +generalization and false positives but cannot substitute for counterfactual +clean/marked labels. + +The frozen normalized-patch baseline was repeated over three training seeds. +Individual original-image AUCs ranged from 0.864 to 0.879. Averaging the three +models increased AUC to 0.884, but a threshold above every validation negative +still produced one false positive among 259 test negatives. At that operating +point the two positive-source TPRs were only 2.45% and 8.97%. Calibrating near +1% validation FPR yielded 1.54% test FPR and TPRs of 6.68% and 18.16%. +Accepted positives had 98.8-100% vendor accuracy, so vendor attribution is +useful only after an independently reliable presence decision. + +A naive full-field spatial model reached only 0.599 AUC. Adding a separately +encoded stationary-wavelet branch and late fusion reduced AUC to 0.597. Both +had effectively zero strict TPR. The result does not contradict the causal +open proxy: it shows that global average pooling over source-labeled images +does not isolate the transferable signal and that SWT does not repair the +source shift. + +Post-hoc image aggregation also failed to create an operating point. Averaging +presence logits raised AUC from 0.884 to 0.890, but increased strict test FPR to +0.77% while reducing TPR. Median, lower-quartile, upper-quartile, and top-tail +aggregation either preserved the overlap or suppressed almost every positive. +A separately trained binary presence head plus vendor head reached 0.880 +ensemble AUC, 0.77% strict test FPR, and TPRs of 1.11% and 7.26%. Decoupling the +heads is architecturally faithful but insufficient without faithful labels. + +These ablations close spectral feature engineering, naive full-field pooling, +and score aggregation as the next detector step. The next valid training epoch +requires same-content counterfactual pairs or independent watermark oracle +labels, matched transformations, and an image-level detection loss. External +generator corpora, including difficult non-target providers, remain hard +negative and FPR-challenge sets only. + ## Decision record The program has four possible honest outcomes per provider: diff --git a/docs/synthid.md b/docs/synthid.md index 0b4a34d..5ab7b83 100644 --- a/docs/synthid.md +++ b/docs/synthid.md @@ -268,6 +268,17 @@ chance, and fixed phase scoring collapsed under crop. Wavelets therefore remain a gated late-fusion ablation, while complex spectral analysis remains a residual-discovery tool rather than a presence score. +The same conclusion held on a larger temporally split closed corpus. A +three-seed normalized-patch ensemble reached 0.884 AUC, but its threshold above +all validation negatives produced a test false positive and detected only +2.45% and 8.97% of the two positive-source classes. Naive full-field pooling, +late-fused SWT, robust patch aggregation, and a separate binary detection head +did not improve that low-FPR operating point. Vendor classification was highly +accurate among accepted positives, but it cannot rescue an unreliable presence +decision. Source labels therefore remain suitable for vendor triage and hard +negative evaluation, not for establishing a SynthID detector without +counterfactual or oracle watermark labels. + The protocol, exact limitations, and next experiments are recorded in the [`detector and removal research plan`](synthid-detector-removal-plan.md).