From 11ca7d6054ce7da60c768dc4d805f8451393bacf Mon Sep 17 00:00:00 2001 From: Victor Kuznetsov Date: Mon, 31 Aug 2026 12:57:13 -0700 Subject: [PATCH] Add frozen photo-classifier research archive to docs Record the AI-versus-camera freeze, sister SynthID research pages, and that identify stays provenance-only. --- CLAUDE.md | 2 +- docs/ai-generated-image-classifiers.md | 1368 ++++++++++++++++++++++++ docs/index.md | 4 + docs/synthid-classifiers.md | 518 +++++++++ docs/synthid-detector-research.md | 591 ++++++++++ docs/synthid-removal-research.md | 243 +++++ 6 files changed, 2725 insertions(+), 1 deletion(-) create mode 100644 docs/ai-generated-image-classifiers.md create mode 100644 docs/synthid-classifiers.md create mode 100644 docs/synthid-detector-research.md create mode 100644 docs/synthid-removal-research.md diff --git a/CLAUDE.md b/CLAUDE.md index 577d9b3..db7668e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -39,7 +39,7 @@ Before a release, read [`docs/release-and-distribution.md`](docs/release-and-dis [`docs/module-internals.md`](docs/module-internals.md) is the canonical per-module map, including design decisions, thresholds, calibration history, incident records, and regression guards. Read the relevant section before changing a subsystem. -Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md). +Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md). Classifier research is split between general [`AI-generated image classifiers`](docs/ai-generated-image-classifiers.md) and [`SynthID source classifiers`](docs/synthid-classifiers.md). Other SynthID campaign logs are [`docs/synthid-detector-research.md`](docs/synthid-detector-research.md) and [`docs/synthid-removal-research.md`](docs/synthid-removal-research.md). ## Data safety diff --git a/docs/ai-generated-image-classifiers.md b/docs/ai-generated-image-classifiers.md new file mode 100644 index 0000000..5aaec2f --- /dev/null +++ b/docs/ai-generated-image-classifiers.md @@ -0,0 +1,1368 @@ +# AI-generated image classifiers (research) + +> Research archive for metadata-free `ai_generated` versus +> `not_ai_generated` classifiers. These are not SynthID detectors and are not +> shipped product verdicts. Current behavior: [supported signals](supported-signals.md) +> and [known limitations](known-limitations.md). +> +> Sister pages: [SynthID source classifiers](synthid-classifiers.md), +> [SynthID local detector](synthid-detector-research.md), +> [SynthID mark removal](synthid-removal-research.md), and +> [mechanism reference](synthid.md). + +A classifier is reliable only when its name matches its errors, photographs +are the first negative, Firefly and PixelBin are in the test, and a watermark +claim uses an independent oracle. CLIP content embeddings and the 124-d +origin-locked residual bank are different features for different jobs. + +## Frozen result, 2026-08-31 + +The freeze is **AI versus camera**, not a universal detector, and it is not +wired into `identify`. Detector: CLIP-L-ft ridge AND freeze MLP. +Provider: 124-d focal heads, only after DEFINITELY or POSSIBLY. + +| Cell | Value | +| --- | ---: | +| Detector DEFINITELY, AI-test | 92.4% (n=1,847) | +| Detector DEFINITELY, fresh | 1.3% (n=3,000) | +| Detector DEFINITELY, Kodak | 0/24 | +| Detector DEFINITELY, FLUX | 83.0% (n=300) | +| Provider OpenAI / Google / TC260 | 90.8% / 90.9% / 78.6% | +| Provider Meta hold-out v3 | 89.4% (177/198) | + +Out of contract: receipts and other documents, UI, digital art without the +illustration veto, and ungated memes. A photorealistic AI receipt can score +as human; a real CORD thermal receipt can score as AI. Catalog 32,690 files, +two CPU retrains byte-identical. Weights stay outside git +(`data/research/reports/both-models-2026-08-31/`). Details below. + +## Research task hierarchy + +The primary classifier task is metadata-free AI-generation detection: given an +arbitrary image, decide `ai_generated` versus `not_ai_generated` from pixels. +The target is open-world transfer to generators absent from training, with a +very low false-positive rate across real photographs and other non-AI imagery +such as scans, product cutouts, conventional CGI, and digital graphics. + +OpenAI/Gemini source finding is a narrower, separately documented +[SynthID-adjacent task](synthid-classifiers.md). It asks whether a file resembles +a current OpenAI or Google generation pipeline and otherwise abstains. It does +not replace the general AI-generation detector: a precise provider finder can +miss most AI images, and a general detector need not know which provider +produced a positive. Neither task is a SynthID payload decoder. + +## Partial result: Model 1, AI versus camera + +Finetuned CLIP-L (`openai/clip-vit-large-patch14`), last two vision blocks, +224 letterbox, JPEG and mild crop, linear ridge. Train 5,221 AI plus 6,129 +photos. Locked Open Images fresh never enters train. Operating point: 1% +FPR on disjoint `photo_dev_oi`. + +| Cell | Value | +| --- | --- | +| Kodak | 0/24 | +| Open Images fresh FPR | 1.7% (n=3,000) | +| Exact-1024 Open Images FPR | 6% | +| AI-test TPR | 93.0% (n=1,905) | +| OpenAI | 93.2% | +| Gemini | 90.5% | +| Firefly | 94.0% | +| xAI | 96.1% | +| FLUX hold | 92.7% | + +51 fresh false positives are mostly graphics, CGI, product cutouts, and +scans, not Gemini. Nobody in the sweep hit both ≤1% fresh FPR and ≥90% +TPR. This is the strongest result toward the general task, but its negative +contract is still AI-versus-camera rather than AI-versus-all-non-AI imagery. +The graphics/CGI errors therefore keep the general task open. This is not +SynthID, and it is not in `identify`. + +Artifacts: `.local-eval/synthid/ai-photo-2026-08-22/` +(`comparison.json`, `probe-report-clip-l-ft.json`, +`probe-weights-clip-l-ft.npz`). Date cutoff 2026-07-23, seed 20260822. + +### Rejected Model 1 variants + +Same splits and `photo_dev_oi` 1% cut. + +| Variant | Fresh FPR | Kodak | 1024 FPR | AI TPR | FLUX hold | +| --- | ---: | ---: | ---: | ---: | ---: | +| CLIP-L v2 | 0.017 | 0/24 | 0.04 | 0.877 | n/a | +| CLIP-L + FLUX extra | 0.016 | 0/24 | 0.05 | 0.861 | 0.707 | +| CLIP-H + FLUX extra | 0.014 | 0/24 | 0.02 | 0.812 | 0.913 | +| CLIP-L last-2-blocks finetune | 0.017 | 0/24 | 0.06 | **0.930** | **0.927** | +| DINOv2-giant 256 | 0.023 | 0/24 | 0.04 | 0.606 | 0.293 | + +CLIP-H is the photo-FPR specialist (1.4% fresh, 2% at 1024) at 81% TPR and +is not the result. DINOv2-giant at 256 px is not usable. + +v1 (CLIP-L, no Open Images in train) at a COCO-looking 0.5% cut accepted +13% of Open Images. Domain shift, not the 124 residual bank. v2 added +1,000 disjoint Open Images reserve photos to train and 500 as +`photo_dev_oi`; locked fresh stayed 1.7% FPR at 87.7% TPR before +finetune. + +The 124-d residual bank is the wrong feature for "AI or not". At a +Kodak-safe cut it catches 60% Firefly and misses FLUX, NovelAI, Reve, and +most of TC260 and xAI. Do not train another ridge on that representation +for an AI-or-not claim. + +Open, if this head is ever considered for a product cut: a graphics/CGI +abstain and time/device-disjoint modern camera coverage. CLIP treating +non-camera imagery as generation is one known error, not Gemini contamination. + +### Frozen public-checkpoint transfer, 2026-08-24 + +A no-training sweep put the official +[`Community Forensics`](https://github.com/JeongsooP/Community-Forensics) and +[`SPAI`](https://github.com/mever-team/spai) checkpoints on the same public +rows and the same operating rule as Model 1. Each threshold is the strict 99th +percentile of the 500-image `photo_dev_oi` split; no AI or evaluation negative +sets tune it. The SPAI core runs stop after all 2,405 AI rows because the model +is already dominated there; they do not supply a fresh-photo FPR. + +| Model | AI test | AI extra | FLUX hold | Open Images fresh | +| --- | ---: | ---: | ---: | ---: | +| Model 1, CLIP-L-ft | 93.0% | 92.5% | 92.7% | 1.7% | +| Community Forensics 384 | 34.6% | 12.0% | 23.0% | 1.0% | +| SPAI, longest edge 512 | 2.2% | 3.5% | 0.7% | not run | +| SPAI, longest edge 1024 | 6.5% | 9.5% | 10.0% | not run | + +Community Forensics finds 33 of Model 1's 170 misses across the 2,405 AI +rows. On 4,133 public evaluation photographs, however, it adds 37 errors not +made by Model 1. A calibration-only rank-max fusion reduces AI-test recall to +91.9%, AI-extra recall to 88%, and FLUX recall to 86%, while fresh-photo FPR +rises to 1.73%. A literal OR at the two original thresholds doubles calibration +FPR to 2% because their five errors do not overlap. The checkpoint is an +auxiliary representation, not a better detector or a valid OR branch. + +SPAI at 1024 recovers only 11 Model 1 misses. Its predeclared rank-max fusion +reduces AI-test recall to 90.2% and FLUX recall to 85.3% at the same 1% +calibration FPR; its literal OR also doubles calibration FPR to 2%. The +300-image FLUX cell is exactly 1024 on its longest edge, so this failure cannot +be assigned to downscaling in that cell. The 512/1024 ablation does show +resolution sensitivity, but no useful low-FPR hybrid. + +[`B-Free`](https://github.com/grip-unina/B-Free) remains unmeasured: its sole +official checkpoint host was unreachable over HTTP and HTTPS, and no verified +mirror was found. Its license also limits use to informational and nonprofit +purposes and expressly prohibits industrial or profit-oriented use. Its useful +result for this project is therefore the bias-reduction training paradigm, not +a checkpoint dependency. + +No public checkpoint replaces Model 1 or safely repairs it. The next model +must change the negative contract: hash-grouped, time/device-disjoint modern +computational photography plus conventional CGI, graphics, scans, and product +cutouts. Another generic detector trained against a narrow `real` corpus is +not a new signal. + +Local reproducibility artifacts: +`.local-eval/synthid/ai-photo-2026-08-22/frozen-ai-detector-sweep-2026-08-24/`. + +### General AI-classifier GitHub sweep, 2026-08-25 + +A separate search targeted pixel-based `ai_generated` versus +`not_ai_generated` classifiers, not SynthID repositories. Twelve recorded +GitHub GraphQL searches returned 2,006 unique public non-fork repositories. +The broadest four searches were capped at 500 collected results, so this is a +bounded reproducible survey, not a claim that GitHub search can enumerate every +repository. Five current catalogs and benchmarks contributed 110 references; +106 resolved to 105 unique live repositories. Curated references plus +high-signal search matches produced 332 candidates, of which 328 resolved for +README, license, weight, and inference review. + +The exact public, non-fork GitHub query set was: + +```text +"AI-generated image detection" in:name,description,readme fork:false +"AI generated image detector" in:name,description,readme fork:false +"AI image detector" in:name,description,readme fork:false +"AIGC image detection" in:name,description,readme fork:false +"AIGC detector" image in:name,description,readme fork:false +"synthetic image detection" in:name,description,readme fork:false +"synthetic image detector" in:name,description,readme fork:false +"generated image detection" diffusion in:name,description,readme fork:false +topic:ai-generated-image-detection fork:false +topic:ai-image-detection fork:false +topic:aigc-detection fork:false +topic:synthetic-image-detection fork:false +``` + +The five catalog/benchmark inputs were +[`AIGCDetectBenchmark`](https://github.com/Ekko-zn/AIGCDetectBenchmark), +[`Awesome-AIGC-Image-Video-Detection`](https://github.com/ant-research/Awesome-AIGC-Image-Video-Detection), +[`Awesome-AIGC-Detection`](https://github.com/Daisy-Zhang/Awesome-AIGC-Detection), +[`Awesome-AIGC-Image-Detection`](https://github.com/graydove/Awesome-AIGC-Image-Detection), +and +[`Awesome-AI-generated-Image-Detection`](https://github.com/nxZhai/Awesome-AI-generated-Image-Detection). + +The filter required pixel inference, an available checkpoint, reproducible +preprocessing, a license compatible with possible product use, and a signal or +training contract that differs materially from already rejected models. It +removed metadata/API wrappers, SynthID-only tools, face/video-only deepfake +systems, datasets and leaderboards, UI-only repositories, classroom CIFAKE +models, noncommercial checkpoints, and repositories without runnable weights. + +The most relevant survivors are: + +| Model | Status | Why it matters | +| --- | --- | --- | +| [Dual Data Alignment](https://github.com/roy-ch/Dual-Data-Alignment) | Apache-2.0, official 1.26 GB checkpoint, measured partially | DINOv2-L LoRA with paired real/reconstruction JPEG and frequency alignment; best new training contract. | +| [PGC](https://github.com/xiaoyu6868/PGC) | Apache-2.0, SD1.4 measured fully and joint measured on AI-test | DINOv2-L peak-guided calibration exposes a strong OpenAI signal, but it confounds Kodak scans and does not safely fuse with Model 1. | +| [DGS-Net](https://github.com/HorizonTEL/DGS-Net) | Apache-2.0, stage-2 checkpoint measured partially | Distillation-guided gradient surgery is reproducible, but the frozen checkpoint is weak and adds independent photo errors. | +| [FerretNet](https://github.com/xigua7105/FerretNet) | Apache-2.0, weights available, lower priority | Efficient local-pixel artifact branch, but trained on four ProGAN classes. | +| [OmniAID](https://github.com/yunncheng/OmniAID) | Modern 3.24 GB checkpoints; repository has no license file | Mirage-Train semantic/artifact experts are promising, but the README's MIT badge is not a license grant. | +| [SDAIE](https://github.com/Ekko-zn/SDAIE) | Weights available; no license | Camera/EXIF-supervised and real-only training are relevant ideas; inference is pixel-based, but product use is unresolved. | +| [AIDE](https://github.com/shilinyan99/AIDE) and [CO-SPY](https://github.com/Megum1/CO-SPY) | MIT, weights available, lower priority | Reproducible hybrid signals, but official checkpoints retain ProGAN or SD1.4-era negative contracts. | + +[Effort](https://github.com/YZY-stack/Effort-AIGI-Detection), +[Forensic Self-Descriptions](https://github.com/ductai199x/Forensic-Self-Descriptions-CVPR25), +and B-Free are research-only or noncommercial. UniGenDet is MIT but its +published checkpoint is about 59 GB. OpenSDI and SAD-Bridge have no detected +license. REM describes a relevant real-centric method, but its code and weights +are still pending. [FatFormer](https://github.com/Michel-liu/FatFormer) and +[SSP-AI-Generated-Image-Detection](https://github.com/bcmi/SSP-AI-Generated-Image-Detection) +have runnable, product-compatible releases, but their four-class ProGAN and +per-generator GenImage contracts are narrower than the measured candidates. +[RIGID](https://github.com/IBM/RIGID) is an Apache-2.0 training-free +reference-comparison idea rather than a detector with a frozen checkpoint to +transfer. + +#### Our frozen-transfer protocol + +The results below are this project's measurements, not accuracy copied from the +upstream papers or model cards. The public portion of the frozen manifest +contains 7,038 rows: + +| Role | Public cells | Rows | +| --- | --- | ---: | +| Calibration negative | `photo_dev_oi` | 500 | +| AI evaluation | `ai_test`, `ai_eval_only`, `flux_hf_hold` | 2,405 | +| Non-AI evaluation | fresh and 1024-pixel Open Images, COCO holdout, Kodak, Picsum | 4,133 | + +The manifest removes exact SHA-256 duplicates before scoring. Every candidate +uses its official image branch and published evaluation preprocessing unless a +deviation is named below. No checkpoint is retrained and no AI or evaluation +negative row selects a threshold. Scores are oriented so that larger means more +likely AI-generated; the decision is the strict inequality above the empirical +99th percentile of the 500 calibration negatives. + +For a two-model hybrid, each raw score is converted to its empirical percentile +against that model's 500 calibration scores. `rank_max` is the maximum of those +two percentiles and `rank_mean` is their mean. The hybrid threshold is then +recalibrated on those same 500 negatives. Literal OR results retain each model's +independently calibrated threshold. Per-item changes are reported as paired +improvements and regressions, with a one-sided exact sign test when the direction +is used as evidence. A pilot stops a larger run only when it is already too far +below the recall/FPR gate to change the decision. + +#### Exact artifacts and adapters used by our runs + +- Community Forensics: repository + `ee5b71d43db0f3779e1edd64ee927b13f2dd6ad4`; model snapshot + `6076002bf0d9dd37537f965ee2f06f826c333b61`, model SHA-256 + `b89f36275f3bf5e2b040eee36597a8f19db051bff9a473a9cf7b2466284fb387`; + processor snapshot `3540a3f0d688f8bf492a8aed48613b891f88047e`. The adapter uses + official `CommForImageProcessor` test mode at 384 pixels and the raw output + logit. Its calibrated threshold is `-0.7597203404`. +- SPAI: repository `8ff7b3b6779b4fcb43cf313471d9cb1c62d129a4`, OpenAI CLIP + revision `d05afc436d78f1c48dc0dbf8e5980a9d471f35f6`, checkpoint SHA-256 + `24159f27d7c8c2cd0cb6c4019189eb89ad0874a0d9d15f8dc9afd39ca9648a55`. + The 512/1024 variants cap the longest edge with bicubic resampling before + SPAI's RGB, minimum-size-padding transform; this implements the documented + `--resize-to` intent because the upstream `SmallestMaxSize` path can enlarge + the longest edge. Thresholds are `17.0864165878` and `14.0359167767`. The + upstream checkpoint embeds a YACS object and could not use safe + `weights_only=True` loading; this is another reason it remains research-only. +- Dual Data Alignment: repository + `8b9c06e75e63f4688bc25ac43a7e3412878cf67f`, checkpoint SHA-256 + `b27a31d39374803ddeff02bfabb2be76e190b04300490cddfafb24f683f37e3e`. + The adapter is the official DINOv2-L/14 rank-8 LoRA image path with + `CenterCrop(336)`, tensor conversion, CLIP normalization, and sigmoid binary + score. The threshold is `0.9587997139`. +- PGC: repository `0c9b10f3b89964b804ad6097e61709f74e827fbf`, DINOv2-L + configuration `47b73eefe95e8d44ec3623f8890bd894b6ea2d6c`. The SD1.4 + checkpoint SHA-256 is + `275c35741834345191fd1be4e4c26075512840d4cf0d2515b9c86094b7bf003e`; + the ProGAN+SD1.4 checkpoint SHA-256 is + `234ca835f4219892acacaea6f2bd15ac698e4491c91b2125f109901fdb56ece6`. + Both load with `weights_only=True` and a strict state-dict match. The adapter + preserves official `PadCenterCrop(224)`, tensor quantization-residual append, + DINO normalization, and sigmoid fused logit. Thresholds are `0.4983334646` + and `0.7142269963`. +- DGS-Net: repository `e7e0799b61014765c11c837506b772e7504198e3`, stage-2 + checkpoint SHA-256 + `89b96a586a6b9f2e5626b03aa27e8063bbe6aaae00f4d0cf97fd3ced2379c216`. + Safe `weights_only=True`, memory-mapped loading extracts a strict inference + subtree: CLIP ViT-L/14, rank-6 vision LoRA, and image head. The training-only + frozen teacher and text head are not read by the official image-only forward, + so stage 1 is unnecessary for evaluation. Preprocessing selects the 24 lowest + and 25 highest spectral-entropy 32-pixel patches, shuffles them with the + official seed 100, stitches a 7x7 image, and applies ImageNet normalization. + The threshold is `0.9998653293`. +- SAFE, RINE, and Nonescape Mini respectively pin repository revisions + `4e998724651b227def64f5be0cd60c0aa1552c35`, + `9b7fd5857cc205d0412be6aeee0d7611b95bd620`, and + `52619d5c96ab83f018d9e879d4be14d847ccb15d`; checkpoint SHA-256 values are + `b3f5ecfb46a154ed553aaaf4bf3ba59182310726ddb0cbb1fe42bd0e22d2f20e`, + `6535ff6ecfa88d33c081e561c2cb2dcb594a1006b050eddb72a72ce005ed2ca1`, + and `7a0d0740c813ce199bc32ed16a5f4f4915895c4c9fdee0a98bdbeedd4f3631fd`. + Their adapters preserve, respectively, the official 256-pixel crop plus + bior1.3 DWT branch, 224-pixel OpenAI-CLIP path, and EfficientNet-v2-s + 256-resize/224-crop path. Thresholds are `0.9303810883`, `0.9529916000`, and + `0.9896544343`. + +Seven additional checkpoints were put on that frozen rule. All values below +are public cells. + +| Model | AI test | Gemini | OpenAI | FLUX hold | Fresh Open Images | +| --- | ---: | ---: | ---: | ---: | ---: | +| Model 1, CLIP-L-ft | 93.0% | 90.5% | 93.3% | 92.7% | 1.7%, n=3,000 | +| DDA official | 48.2% | 68.8% | 21.8% | not run | 0.7%, n=1,000 | +| PGC SD1.4 official | 44.6% | 5.5% | 96.5% | 0.7% | 0.7%, n=3,000 | +| PGC ProGAN+SD1.4 official | 29.3% | 8.3% | 56.3% | not run | not run | +| DGS-Net stage 2, seed 100 | 21.2% | 1.5% | 55.5% | not run | 1.9%, n=1,000 | +| SAFE official | 21.6% | 1.3% | 57.8% | 0.0% | 0.6%, n=3,000 | +| Nonescape Mini v0 | 13.3% | 13.5% | 7.3% | 37.3% | 1.8%, n=3,000 | +| RINE four-class | 10.7% | 5.5% | 16.8% | 1.0% | 1.0%, n=3,000 | + +DDA supplies material complementary recall. It finds 52 +of Model 1's 133 misses on the 1,905-image AI test. Its seven errors on the +independent 1,000-image negative slice do not overlap Model 1's 18, however. A +literal OR reaches 95.7% AI recall and 2.5% FPR there; the predeclared +calibration-rank maximum reaches only 93.4% recall and 1.6% FPR. The checkpoint +is therefore not a safe drop-in fusion. + +PGC SD1.4 finds 63 of those 133 misses, while the joint checkpoint is weaker +on the same AI rows. A calibration-only rank maximum with Model 1 initially +looked like a clean improvement: AI-test recall rose from 93.0% to 94.1% +(58 paired improvements, 37 regressions; one-sided sign test `p=0.0198`) and +fresh Open Images FPR fell from 1.67% to 1.13% (22 paired improvements, six +regressions; `p=0.00186`). The full negative matrix rejects that conclusion. +The fusion calls all 24 Kodak images AI-generated, versus 0/24 for Model 1, +and its aggregate public-negative errors rise from 59/4,133 to 63/4,133. It +also reduces AI-extra recall from 92.5% to 87.5% and FLUX hold recall from +92.7% to 85.3%. + +Our single-pass PGC component ablation localized but did not remove the +confound: + +| PGC score | AI test | OpenAI | Fresh Open Images FPR | Kodak errors | +| --- | ---: | ---: | ---: | ---: | +| Global only | 34.8% | 82.0% | 1.8% | 17/24 | +| Global + RGB peak | 6.8% | 5.5% | 0.37% | 0/24 | +| Global + residual peak | 41.9% | 96.5% | 1.5% | 24/24 | +| Official global + RGB + residual | 44.6% | 96.5% | 0.67% | 24/24 | + +The RGB peak suppresses the Kodak response, but it also removes the useful +OpenAI signal; the residual peak carries both. Calibration-rank conjunctions +of the RGB and residual components remove the Kodak errors, but either add +fresh-photo errors under their own 1% calibration cuts or fall below Model 1 +recall after joint recalibration. A descriptive raw PGC threshold above `0.75` +would add 23 AI catches without a public-negative error, but it was selected +after inspecting test behavior and is therefore not validation. It is not an +accepted branch or product threshold. PGC remains an OpenAI-oriented research +feature, not a universal detector. + +DGS-Net's official stage-2 image branch was reconstructed strictly from the +published checkpoint; the training-only frozen teacher and text head are not +read by the repository's image-only evaluation forward. Its official +spectral-entropy patch selection retains a random shuffle, so this measurement +pins the repository's seed 100. The checkpoint finds 37 Model 1 misses but adds +19 non-overlapping errors on the same 1,000 fresh negatives. A literal OR is +95.0% AI-test recall at 3.7% FPR; calibration-rank maximum is 91.0% recall at +1.6% FPR. Its 21.2% standalone recall is far enough below the gate that a full +corpus or multi-seed run is not warranted. + +SAFE, RINE, and Nonescape Mini also fail as frozen replacements or fusions. +Their value is now bounded: SAFE supplies a wavelet/transformation branch, RINE +intermediate CLIP blocks, and Nonescape a cheap EfficientNet branch, but none +improves the low-FPR operating point. + +The licensed frozen-checkpoint queue is exhausted at the useful priority level. +PGC's OpenAI/Kodak confound makes scans an explicit hard gate for subsequent +training. SDAIE's camera-supervised or real-only training remains an idea source +until a license exists. + +Local search and scoring artifacts: +`.local-eval/github-ai-detector-sweep-2026-08-25/` and the frozen sweep directory +above. + +### Own training and hard-negative ablations, 2026-08-25 + +The next campaign tested the training ideas rather than treating upstream +checkpoints as finished detectors. Every accepted threshold used only a named +negative calibration split. Per-item decisions were compared with Model 1, and +a prompt-disjoint EvalGEN cohort remained physically unextracted whenever a +candidate failed development or the old public matrix. + +#### Paired reconstruction and hard negatives + +DDA's useful ingredient is paired real/reconstruction training, not its +published frequency-mix formula: with its documented whole-image patch setting, +frequency mixing reduces algebraically to pixel mixing. A 256-pair pilot used a +public VAE reconstruction of training-only COCO, Picsum, and Open Images photos. +At 128 continuation steps, the ordinary control reached 93.75% AI-test recall, +2.67% fresh Open Images FPR, 94% FLUX recall, and 0/24 Kodak errors. Paired +reconstruction kept AI-test at 93.75% but moved FLUX to 91%, retained 2.67% +fresh FPR, and produced eight paired improvements against ten regressions. +Adding codec augmentation moved the same cells to 93%, 90%, and 3%, with six +improvements against 13 regressions. Neither arm improved Model 1. + +Source-grouped UI screenshot negatives also failed to repair the public +contract. A higher-dose continuation reduced AI-test to 87.5%, raised fresh FPR +to 3%, added 1/24 Kodak error, and reduced FLUX to 82%. A lower dose reached 89%, +3%, 0/24, and 85%. Adding the same negative category only to the frozen ridge +head produced 52 regressions against 24 improvements on the public matrix. +Private source identities, paths, and local-only measurements are deliberately +absent from this public archive. + +An official Open Images V7 training partition supplied a reproducible hard-photo +test. The first 6,000 lexicographic official S3 IDs were frozen before scoring, +content-deduplicated, and split into 4,000 mining, 1,000 untouched holdout, and +1,000 unused challenge images. Model 1 called 55/4,000 mining and 13/1,000 +holdout images AI. Replacing its head after adding the top 128 hard negatives +kept holdout at 13/1,000, improved AI-test from 93.02% to 94.12%, but raised +fresh FPR from 1.67% to 2.13% and added 1/24 Kodak error. A predefined +calibration-rank mean with the hard head was only directional: AI-test 92.97%, +fresh FPR 1.53%, unchanged 92.67% FLUX, 0/24 Kodak, and six aggregate paired +improvements against two regressions. On the untouched Open Images train +holdout it changed 13 errors to ten, with four repairs and one new error. + +The corresponding frozen rank-min challenge did not justify promotion. On 100 +previously unscored EvalGEN Flux images it exactly matched Model 1 at 83%; on +1,000 unused Open Images train negatives it moved 1.0% to 0.9%, one repair and +no new error. The positive gate failed. That first 100-image dataset-viewer +slice covered only five lexicographically early prompt groups, heavily featuring +backpacks, bananas, and baseball bats, so 83% is a semantic stress result rather +than representative EvalGEN recall. + +Semantic matching alone did not solve that content bias. A 5,221-pair frozen +CLIP linear head validated at AUC 0.996 but reached only 76% on the opened +EvalGEN slice. Two- and three-scale Model 1 views reached at most 85% there and +raised negative FPR to 1.3%; a 50% center crop fell to 57%. A paired-margin tower +continuation made the same decisions as its ordinary control. Its best arm made +two repairs against four regressions, while larger margins had no measured +separation to exploit. + +#### Prompt-diverse EvalGEN continuation + +The official [EvalGEN dataset](https://huggingface.co/datasets/Junwei-Xi/EvalGEN) +contains 553 aligned prompt groups and approximately 55,300 JPEG images from +Flux, GoT, Infinity, OmniGen, and NOVA. Before training, the five viewer-prefix +prompt groups were reserved as opened development, 300 disjoint prompt groups +were assigned to training, 100 to a future blind cohort, and 148 left unused. +One deterministic image per generator and training prompt produced 1,500 unique +training positives, 300 per generator. Their decoded-pixel hashes had zero +overlap with the 19,524 unique hashes in the base manifest. The 500-image blind +cohort was not extracted or scored during the campaign. + +A 256-step continuation replaced half of each batch with an EvalGEN positive +and a text-nearest real training photo. Its neural head moved the opened +EvalGEN development slice from 88% for the byte-identical control to 95%, while +the unused 1,000-photo development FPR moved from 1.2% to 1.1%. The signal was +real, but promotion checks exposed forgetting: + +| Candidate | Opened EvalGEN dev | Unused-photo dev FPR | AI-test | Fresh Open Images | FLUX | Kodak | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| Canonical ridge on expanded tower | 89% | 0.9% | not run | not run | not run | not run | +| Expanded neural head, stricter calibration | 92% | 0.7% | 91.13% | 1.57% | 92.0% | 0/24 | +| Model 1 + expanded rank maximum | 92% | 0.8% | 92.28% | 1.63% | 92.67% | 0/24 | +| Quarter-dose replay rank maximum | 91% | 0.7% | 92.34% | 1.60% | 90.0% | 0/24 | +| Soft-distilled quarter replay rank maximum | 90% | 0.8% | 92.07% | 1.40% | 90.67% | 0/24 | +| 75% base-to-expanded parameter interpolation | 91% | 0.8% | 92.28% | 1.73% | 91.67% | 0/24 | +| Calibration-budgeted rescue branch | 94% | 1.0% | 92.81% | 2.03% | 93.67% | 0/24 | +| Quadratic two-score stacker | 92% | 1.0% | 92.60% | 2.27% | 94.0% | 0/24 | +| Full-dose hard-negative neural head | 100% | 0.8% | 87.72% | 1.37% | 98.0% | 0/24 | +| Model 1 + full-dose hard-negative rank maximum | 91% | 0.6% | 90.34% | 1.17% | 92.0% | 0/24 | +| 75% hard-negative parameter interpolation | 90% | 0.6% | 90.60% | 1.20% | 91.33% | 0/24 | +| Quarter-dose hard-negative neural head | 97% | 0.8% | 88.50% | 1.43% | 97.0% | 0/24 | +| Model 1 + quarter-dose hard-negative rank maximum | 89% | 0.6% | not run | not run | not run | not run | +| Conditional Model 1 + quarter-hard branch | 97% | 1.0% | 93.33% | 1.93% | 96.33% | 0/24 | + +The ridge gate failed on recall, so it never entered the old full matrix. The +full-dose neural head and rank fusion reduced fresh errors slightly, but paired +AI-test changes remained negative: the rank maximum repaired eight Model 1 +misses and broke 22 prior hits. Reducing EvalGEN to one-quarter of each batch +did not remove the asymmetry. Logit-MSE distillation overconstrained the model +and collapsed the opened positive recall to 64%; bounded probability-BCE +distillation retained 95% before strict fusion but still failed the old core. +Interpolation at 25% and 50% failed development at 89%; 75% passed development +and then failed every core condition except Kodak. + +The rescue cascade allocated a fixed five-error calibration budget between the +old and new heads. Its conservative selected allocation gave the new branch +only one error, yet the independent fresh-photo cell rose to 2.03%. Expanding +calibration to all 6,195 opened public negatives found no budget allocation that +simultaneously held aggregate FPR at or below 1%, preserved Model 1 AI-test, +AI-extra, and FLUX recall, and reached 90% on opened EvalGEN. A supervised +linear/quadratic stacker also failed: the quadratic model passed development at +92%/1.0%, then raised fresh FPR to 2.27% and reduced AI-test to 92.60% despite +raising FLUX to 94%. + +Hard-negative replay isolated the competing objectives more sharply. A batch +with two old AI, two EvalGEN AI, two ordinary photos, one prompt-matched photo, +and one of the top 128 reproducible Open Images false positives produced perfect +opened EvalGEN recall and 0.8% development FPR. It also gave the campaign's best +fresh-photo rates, 1.37% for the neural head and 1.17% for its rank fusion, but +old AI-test recall collapsed to 87.72% and 90.34%. A 75% interpolation retained +only 90.60% AI-test recall. Thus hard negatives repair the negative contract, +but do not prevent positive-class forgetting. + +A factorial follow-up changed only the positive replay ratio from two old plus +two EvalGEN images to three old plus one EvalGEN image per batch. Its direct +neural head reproduced 97/100 opened EvalGEN detections with 8/1,000 photo +errors under a frozen 99th-percentile calibration. On the old core it repaired +8 Model 1 AI-test misses but broke 94 prior hits: 1,686/1,905 AI-test images +(88.50%) remained positive. It improved fresh Open Images from 50/3,000 to +43/3,000, improved FLUX from 278/300 to 291/300 without a paired regression, +and kept Kodak at 0/24. The predeclared rank maximum was stricter on development, +89%/0.6%, and therefore never entered core. This rules out replay ratio alone as +the missing anti-forgetting mechanism. + +A conditional two-branch rule then used a high quarter-hard threshold to add +new positives and a lower threshold only when Model 1 was already positive. +Its thresholds were selected from 13,280 admissible combinations on old +training AI and opened development, never core. It preserved 5,055/5,221 old +training AI against Model 1's 5,041, retained 97% opened EvalGEN recall at +1.0% development FPR, and restored the old core positive side: AI-test reached +1,778/1,905 (93.33%) with six repairs and no regressions, while FLUX reached +289/300 (96.33%) with 11 repairs and no regressions. The independent negative +side exposed calibration drift, however. Fresh Open Images rose from 50 to +58/3,000 errors: one repair against nine new errors, or 1.93% FPR. The route +separation fixes forgetting, but a single-source negative calibration does not +yet learn a portable veto. + +Broadening that same two-score threshold search did not produce a candidate. +The frozen contract combined 4,171 negatives from eight public photo cells, +required aggregate FPR at or below 1%, untouched Open Images train at or below +1%, every source at or below 2%, preserved old training recall, and retained at +least 90% opened EvalGEN recall. No threshold pair satisfied all constraints, +so neither core nor blind was opened. A separate weighted-ridge veto was then +fit only on Model 1-positive `ai_train` embeddings and 256 hard photos from each +of the original photo training pool and official Open Images mining pool. None +of 37,875 penalty/threshold combinations passed the same development contract. +The best positive-preserving frontier was nevertheless narrow: it retained +95/100 EvalGEN and exactly matched Model 1 at 667/700 old AI development while +calling only 18/4,171 aggregate photos AI (0.43%). Six of those errors were in +the 100-image `open_images_1024` cell, a 6% source-specific rate; every other +cell ranged from zero to 0.75%. The next ablation is therefore negative dose, +not another threshold over the same fitted veto. + +Increasing the veto fit to 512 and 1,024 hard negatives per source did not move +the `open_images_1024` failure: its best positive-preserving frontier remained +6/100 while aggregate photo errors were 17/4,171 and 20/4,171. Route attribution +showed that all six errors bypassed the veto through the unconditional new-AI +branch. Raising that branch's score threshold as high as possible while keeping +90/100 EvalGEN also retained all six. Applying the learned veto conjunctively to +both branches did not close the gap either. Across 37,875 combinations at each +hard-negative dose, the positive-preserving frontiers retained 91% EvalGEN and +95.29-96.0% old-AI development recall but left 6-7% `open_images_1024` FPR. The +negative-compliant frontiers reduced that cell to 2/100 and aggregate errors to +5/4,171, but collapsed EvalGEN to 26-39% and old-AI development to 43.57-53.43%. +The two learned CLIP representations therefore lack a separating direction for +this source. The next veto must introduce an independent pixel-forensic +representation rather than more data, thresholds, or routing over these scores. + +A metadata-free native-lattice forensic veto then added 47 pixel statistics: +high-pass amplitude and correlation, CFA phase error, cross-channel residual +agreement, intensity-conditioned noise, JPEG block phase, clipping, and +Laplacian energy. It fit on deterministic 2,000-row samples from each old class +and evaluated 37,875 frozen combinations. No candidate passed. Its best +positive-preserving frontier reached 90% EvalGEN, 668/700 old-AI development, +and 32/4,171 aggregate photo errors, but still accepted the same 6/100 +`open_images_1024` rows. The strict negative frontier reached 2/100 there only +by collapsing to 6% EvalGEN and 187/700 old AI. Native-512 coverage was weak on +the smaller COCO and general-photo cells, but it was 100% on EvalGEN, old-AI +development, and the failing exact-1024 cell, so missing features do not explain +the repeated six errors. + +Pixel inspection showed that the six are a semantically diverse and valid +non-AI set: a scientific illustration, a logo, a studio object, a heavily +processed fashion image, an abstract long-exposure photograph, and an ordinary +flower-field photograph. This exposes a scope mistake in the provisional +negative contract: a universal detector must reject human-created graphics and +processed imagery as well as camera photographs. Camera-trace absence cannot be +the general veto. The next hard-negative mining pass must therefore rank public +negatives by the quarter-hard add score itself; the earlier top-128 replay was +ranked by Model 1 and did not target the branch now producing errors. + +That aligned mining scored the same frozen 4,000-image official mining pool and +1,000-image holdout with the quarter-hard head. Its top 128 overlapped Model 1's +top 128 in only 78 rows (Jaccard 0.438), proving that the old mining missed 50 +of the new branch's hardest examples. At the frozen neural threshold, +quarter-hard called 18/4,000 mining and 3/1,000 holdout images AI, versus Model +1's 55/4,000 and 13/1,000. Re-fitting the conjunctive linear veto on the +quarter-ranked negatives still failed at hard-negative doses 256, 512, and +1,024. The best hard-512 positive frontier retained 91% EvalGEN and 671/700 old +AI with 45/5,171 aggregate errors and 11/1,000 mining-holdout errors, but still +accepted 6/100 `open_images_1024`. Its negative-compliant frontier reduced those +cells to 1/1,000 and 2/100 only by falling to 29% EvalGEN and 302/700 old AI. +Aligned examples therefore need to change the add representation itself; a +linear veto over the original Model 1 embedding cannot use them sufficiently. + +A second-stage representation update changed only the quarter-hard replay row: +the hard photo came from the top 128 quarter-hard mining scores instead of the +top 128 Model 1 scores. The 256-step self-hard arm retained the same 97/100 +EvalGEN recall and 8/1,000 unused-photo errors in both pilot and formal neural +development. It did not change the source failure. At each head's independently +frozen 99th-percentile threshold, original quarter-hard and self-hard both +accepted 11/100 `open_images_1024` rows, with zero repairs, zero regressions, and +100/100 paired agreement. The self-hard arm therefore failed its source gate +before old core and blind. Selecting harder examples from the same broad class +is not equivalent to supplying the missing negative domains. + +The first explicit negative-taxonomy audit then froze seven human-created +domains before scoring: historical camera photographs, drawings, prints, +manuscripts, paintings, studio-shot objects, and textiles. Its 700 Cleveland +Museum of Art images were CC0, content-hash unique, and unused by every fitted +model. Model 1 called 270/700 AI, including 60/100 drawings, 58/100 manuscripts, +and 46/100 textiles, while accepting 0/100 historical photographs. A second +source held 350 public-domain Art Institute of Chicago images, 50 per domain; +after a duplicate-hash guard rejected the first manifest and two repeated +images were replaced, Model 1 called 85/350 AI. The combined error was therefore +355/1,050 (33.81%), with 0/150 on the historical-camera control but 82/150 on +manuscripts and 75/150 on drawings. This is a real non-photo domain failure, +not a metadata observation or one museum's export artifact. + +The direct quarter-hard representation reduced those errors to 163/1,050 +(15.52%). Paired against Model 1, it repaired 201 rows and regressed on nine +(two-sided sign test `p=2.34e-48`), but its already measured 88.50% AI-test +recall still disqualifies it. Conditional routing again inherited the baseline +failure, at 357/1,050. A source-disjoint domain-veto experiment then fit only on +the Cleveland embeddings and selected its operating point only on AIC plus the +opened positive development cells. A class-balanced multiclass ridge reduced +the AIC route from 86/350 errors to 0/350 while retaining 97/100 opened EvalGEN +and 288/300 FLUX. The frozen core rejected it: AI-test became 1,771/1,905 versus +Model 1's 1,772 and the conditional route's 1,778, while fresh Open Images +remained 56/3,000 versus Model 1's 50 and the conditional route's 58. It also +left the exact-1024 cell at 8/100. The expert learned transferable museum-art +separation but not the modern logos, composites, product imagery, processed +photography, and ordinary-photo overlap represented by the public core. + +No candidate reached the prompt-disjoint blind cohort. The useful result is a +bounded representation finding: diverse EvalGEN continuation adds a signal +that Model 1 lacks, but its current CLIP-L tower and two-score combinations +cannot add that signal without exchanging old generator recall or photo FPR. +Hard negatives now provide a stronger non-AI training contract, and conditional +routing provides the explicit anti-forgetting architecture. Broad calibration, +a dedicated linear veto, negative-dose scaling, conjunctive routing, a native +pixel-forensic veto, aligned mining, and self-hard continuation all failed on +the same source-specific overlap. The negative-taxonomy audit closes the +historical-art measurement gap but not the modern one. The next bounded campaign +needs source-disjoint training and validation cells specifically for modern +logos and graphic design, commercial product cutouts, composites, retouched +fashion imagery, abstract or long-exposure photography, and ordinary camera +photos. Another museum-art pool, generic photo pool, CLIP-space threshold, +binary veto, or replay-ratio change is not sufficient. + +The modern-negative audit then filled that gap under the same frozen protocol, +with two new source-disjoint runs. The first drew 200 photographer-uploaded +photographs from the official Unsplash Lite dataset, 100 per cell for retouched +fashion and long-exposure photography, with the Unsplash License recorded per +row. The second drew 250 CC-licensed Flickr images through the Openverse API, +50 per cell for modern logo/graphic design, commercial product cutouts, +composites, and independent fashion and long-exposure replication cells, with +the exact CC license recorded per row. Both manifests enforce global +content-hash uniqueness plus perceptual near-duplicate rejection against every +picsum training cell, because picsum serves Unsplash photography; the Openverse +curation additionally excludes AI-marker, game-screenshot, and 3d-render +wording, so those domain labels are query-level rather than hand-verified +ground truth (46/50 logo-cell titles carry explicit logo or design wording). + +Model 1 accepted 85/450 modern negatives, 18.89%. A contamination audit +then tightened the claim before it was trusted: a provenance-metadata scan of +all 450 rows with the project's own `identify` command flagged 0/450 (no C2PA, +SynthID, generator tags, or watermarks), and a date bound re-verified each row +against the public diffusion era, using the Unsplash submission date or the +Openverse `indexed_on` date. All 450 rows received a date; 421 predate +2022-08-01 and 29 post-date it, and the 29 were dropped. The correction was +material in exactly one cell: the Flickr fashion replication cell's 10 errors +became 2, because 8 of them sat in post-2022 uploads that are plausibly real +generations. On the date-clean 421 rows Model 1 accepted 69 (16.4%). The +failure concentrates in stylized graphics and processed photography: +logo/graphic design 18/47 (38.3%), retouched fashion 30/88 on the pre-2022 +Unsplash cell (34.1%; the Flickr replication cell fell to 2/40, so fashion is +source-dependent rather than uniformly failing), and product cutouts 8/50 +(16.0%, unchanged). Long exposure stayed comparatively clean at 11/146 (7.5%) +and composites at 0/50. Direct quarter-hard reduced the clean total to 47/421, +repairing 27 rows and regressing five (sign test `p=1.13e-04`), repeating the +museum pattern: it +helps every human-made domain while its measured 88.50% AI-test recall still +disqualifies it. Conditional routing inherited the baseline again at 86/450. +The two audits together bound the whole negative space: Model 1 fires on +stylized human graphics, logos, product cutouts, retouched fashion, and +historical art, while staying far cleaner on historical, composite, and +long-exposure photography and on ordinary camera photographs. + +The measured cells then fed the same source-disjoint veto recipe the museum +audit used: pooled, per-domain, and multiclass ridge vetoes fit only on the +235 date-clean Openverse embeddings, with the threshold selected only on the +independent 186-image date-clean Unsplash run and applied to the frozen +conditional route. Of the searched candidates, none satisfied both the +negative cap (at most four dev errors) and the positive floor (95/100 +EvalGEN, 287/300 FLUX); the full-run variant with contaminated rows reached +the same verdict over 5,265 candidates. The exchange curve shows why: at +positive compliance the best reachable dev error count is 41/186, meaning +zero of the route errors are repaired, while repairing roughly 39 of them +drops EvalGEN to 40/100 and FLUX to 162/300. Across the whole grid, every +repaired modern negative costs roughly 1.3 to 1.5 AI positives. The two +audits +therefore produced two different failure modes for the same recipe: museum art +is linearly separable from AI in this embedding and fails only on transfer, +while modern negatives are not separable at all. This closes the linear-veto +family over the frozen Model 1 CLIP-L embedding for the modern domains as well +and confirms the earlier aligned-mining finding: the add representation itself +must change, not the example selection or the veto geometry. + +### Wild vendor-flagged AI, 2026-08-27 + +A free wild-AI positive cell came from the stock providers behind the private +veedma-blog image pipeline. Pixabay exposes a vendor-declared +`isAiGenerated` flag and the query `ai generated` surfaces that pool (3,287 +results); 284 content-hash-unique rows were harvested (300 minus 16 lost to a +duplicate-download bug, since fixed, manifest repaired to on-disk truth). The +vendor flag is ground truth for AI-versus-human, not for the renderer, so the +cell is eval-only. A metadata scan found 0/284 provenance signals (the stock +CDN strips everything), making it a pure pixel task. Scored with frozen +classifiers that never saw stock-site content, this produced the first +source-disjoint wild-AI recall measurement: Model 1 accepted 183/284 (64.4%) +at its frozen 1%-FPR threshold, far below its 93.0% curated AI-test recall +(median score 0.43 above the 0.306 threshold, but a quarter of the cell scores +below 0.13). Wild AI images of unknown renderers, re-encoded by a stock +pipeline, are therefore a genuinely harder positive distribution, and curated +test recall does not bound deployment recall. The frozen provider cascade +under its photo-first margin called 255/284 `no_ai` with only 1 `openai` +and 0 `meta` calls, so unknown-renderer AI does not fabricate provider +attributions; under argmax it leans `google` (199) without evidence. The cell +joins the dataset as the first wild-AI eval row source. Artifacts: +`pixabay-ai-positive-2026-08-27/` (manifest, images, +classification-report.json). + +A bias audit then rejected that cell's composition, though not its verdict: +147/284 rows (52%) came from a single contributor and every row from one +query, so the 64.4% recall was one author's dump plus a tail. Model 1 recall +split 63.3% on that contributor versus 65.7% on the rest, so difficulty was +homogeneous, but the sample was not. A stratified reharvest capped at 15 rows +per contributor over eight AI-marker queries produced 300 unique images from +180 contributors (top-10 share 33%) with the same sieves. On the balanced +cell the frozen numbers are: Model 1 recall 209/300 (69.7%, median score 0.482, +lowest quartile below 0.144), direct quarter-hard 189/300 repairing 22 Model 1 +misses and losing 2 (the one place the quarter representation still helps), +conditional routing 210/300 adding one repair and nothing else, provider +cascade margin 252/300 `no_ai` with 1 `openai` call, and 0/300 metadata +signals. The wild-AI recall ceiling under Model 1 therefore settles near +65-70% rather than 93%, robust to contributor stratification; unknown-renderer +AI re-encoded by stock pipelines is the largest measured positive-side gap. +Artifacts: `pixabay-ai-stratified-2026-08-27/` (the eval cell the dataset +keeps; the first harvest is retained only as the audit trail of the bias +finding). + +The stock-negative harvest from the same providers then tripled the modern +negative cells under identical sieves: 150 Pixabay product cutouts +(vendor `isAiGenerated=false`), 150 Pixabay fashion rows, and 152 Unsplash +API fashion rows date-bounded before 2022-08-01 (452 total, 0/452 metadata +signals, all content-hash unique). Frozen Model 1 false-positive rates on +them: product cutouts 40/150 (26.7%), Pixabay fashion 93/150 (62.0%), +Unsplash fashion 108/152 (71.1%). The fashion failure is therefore larger +than the earlier 34% measured on the 88-row date-clean cell and holds across +three independent sources; retouched fashion photography is now the single +worst negative domain for Model 1. The provider cascade stays quiet on all +three cells (no_ai 126/150, 105/152, 99/150; at most 15 stray `google` calls, +no openai or meta fabrications). A second stochastic pass over the Meta +muse-image-1.0 grid added 61 more API generations with zero hash overlap +with pass one (corpus now 127 + 5 checkout originals), moving the Meta class +toward margin calibrability. Artifacts: +`stock-negative-2026-08-27/`, `meta-muse-corpus-pass2-2026-08-26/`. + +### Documents and receipts, 2026-08-27 + +Two public pre-diffusion research datasets filled the unmeasured document +domain: CORD-v2 thermal-printed photographed receipts (99 rows) and FUNSD +scanned forms (150 rows), both provenance-metadata-clean on a spot scan. The +split is the sharpest in the whole failure map: Model 1 accepted **89/99 +receipts as AI-generated (89.9%, score median 0.469 against the 0.306 +threshold)** while form scans stayed near the contract at 5/150 (3.3%, +median -0.001). Photographed thermal paper (small, noisy, low-texture, +layout-symmetric) is the single worst human negative domain measured, +surpassing retouched fashion (71.1%); clean high-fidelity scans pass. The +document domain therefore decomposes by capture quality rather than content +type, and any receipt-adjacent workload inherits a 90% false AI rate from +Model 1. Artifacts: `documents-negative-2026-08-27/`. + +Diagnostics then fixed what can be done about it. Receipt embeddings sit +**99.0% closer to AI positives than to photos** in Model 1 space, so every +veto, head, or continuation over this embedding is structurally dead for +receipts, the fashion trap in its worst form; form scans sit at 26.7% and are +fine. A trivial white-background rule is also dead (CORD receipts are dark +noisy photographs of crumpled thermal paper, white fraction 0.021, at photo +levels). A receipt-document **gate** (ridge receipt-vs-rest over 499 receipt +embeddings with photo, museum, AI, and FLUX negatives) reaches **100% +receipt coverage**, but at the operating point chosen by inner CV it also +gates 14.4% of AI-test, 12.0% of FLUX, and 2% of EvalGEN, with family losses +up to 25.3% (fal) and 25.2% (xAI) — infographic-like generators trip the +document gate. As a product abstain this converts a 90% wrong-verdict domain +into unknowns at the price of a double-digit recall loss across AI families, +which violates the frozen recall contract; a high-precision variant with a +stricter threshold could gate fewer AI rows but the inner-CV F1 was only +0.444, so precision headroom is thin. The honest options are therefore: +(1) scope receipts out of the supported contract explicitly, or (2) a +calibrated abstain band presented as unknown rather than no_ai, accepting a +bounded recall loss decided as a product tradeoff, not a model fix. +Artifacts: `receipt-gate-2026-08-27/report.json`. + +Batch-1 then filled four more unmeasured domains under the same sieves +(428→421 rows after phantom repair: UI screenshots, 150; Danbooru community +digital art, 145; software-rendered matplotlib charts, 82; OpenStreetMap +tiles, 44). Frozen Model 1 false-positive rates: **UI screenshots 149/150 +(99.3%)**, **digital art 144/145 (99.3%)**, charts 55/82 (67.1%), map tiles +21/44 (47.7%). The 150 UI files in that measurement were local product +screenshots; they were replaced on 2026-08-31 with CC-licensed Flickr +images (see the both-model freeze). The 149/150 figure is the original cell. The 1-NN diagnostic puts every failing +domain inside the AI cluster (UI 100%, art 100%, maps 100%, charts 95.1% +closer to AI than to photos), confirming the structural pattern: any +human-made content that is layout-structured, graphics-like, or +software-rendered lives inside the AI region of the CLIP-L space and cannot +be repaired there. Combined with receipts (89.9%), fashion (62-71%), logos +(38%), and museum art (33.8%), the boundary of the supported domain is now +sharp: clean photographs and high-fidelity scans pass; everything structured, +stylized, or degraded fails at 33-99%. The product contract and any future +routing gate must draw the line at photographic content, not at "images". +Artifacts: `batch1-negative-2026-08-27/`. + +### Batch-2 and the full failure map, 2026-08-27 + +Batch-2 added four more domains under the same sieves (memes from imgflip +top-100, whiteboard photos, ticket/stamp scans, book pages; 396 rows): +memes 58/97 (59.8%), book pages 27/100 (27.0%), whiteboards 3/100 (3.0%), +tickets 3/99 (3.0%). Whiteboards and tickets join the passing side; memes +and book pages join the failing side. The complete failure map now covers +**20 measured negative domains**. + +### H2 domain router falsified, 2026-08-27 + +A binary photographic-vs-structured router on Model 1 embeddings, trained on +all 16 failing domains vs photo cells, reached 98.2% structured coverage and +4.4% photo false-structured rate, but routed **97.9% of AI-test and 98.0% of +FLUX as structured**: every AI family (94.3-100%) looks non-photographic to +the router. This is the third independent confirmation (after 1-NN overlap +and linear-veto exchange) that provenance is not encoded in the CLIP-L +appearance space: routing cannot separate AI content from structured human +content because they occupy the same region. + +### H3 process detector: generator-specific traces do not transfer, 2026-08-27 + +Contrastive pairs (same content, different process) were generated from 125 +source images across five domains (charts, logos, art, documents, UI), each +recreated by three generators: OpenAI gpt-image-1-mini (edits/img2img), Meta +muse-image-1.0 (prompt), and Gemini 2.5-flash-image (prompt), totaling 369 +pairs. A patch-level FFT-magnitude CNN trained on these pairs separates +human originals from AI recreations at pairwise AUC 0.909 (0.885 with two +generators, 0.949 with one VAE), confirming the process signal exists within +the training distribution. Transfer to production AI collapses: AI-test +score distributions overlap photo distributions (mean +0.013 to +0.026 above +photos), giving AUC 0.55-0.60 — near chance — while human receipts score ++0.095 above photos, higher than any AI family. Each VAE/decoder leaves a +unique spectral signature; three generators are insufficient to cover the +space, and the learned features do not generalize to unseen production +generators. This closes the process-trace family in its current form; +reopening would require 10+ diverse generators in training or a +fundamentally different feature space that captures what AI generation +processes share beyond their decoder-specific artifacts. + +### Provider cascade across all 20 domains, 2026-08-27 + +The frozen provider cascade (margin 0.50) was scored on every measured +negative and positive domain. Photo negatives are quiet (COCO 0.7% stray, +Kodak 0%), museum/modern/stock cells leak 2.9-11.9% into `google` (mostly +stylized content that pattern-matches Imagen), but documents are the +exception: **69.8% of document/receipt content receives a stray provider +call** (48 openai, 19 google out of 96), and batch-2 content (memes, tickets, +whiteboards, book pages) leaks 27.0% (35 google calls out of 137). Wild AI +leans google at 15.1% with only 1 openai call. The provider cascade is +therefore safe on photographic content, partially safe on stylized content, +and unsafe on document-like content — mirroring the Model 1 failure map. A +production integration of the provider layer must gate on the same +photographic-content condition as the detector itself. +Artifacts: `provider-all-domains-2026-08-27/report.json`. + +### Research-driven hypotheses H4-H6, 2026-08-27 + +Three approaches from 2025 literature (C2P-CLIP prompt learning from AAAI, +spectral invariant from CVPR, TTA voting) were run against the frozen gates. + +**H4 (MLP prompt-probe on frozen CLIP, C2P-CLIP inspired):** a two-layer +MLP trained on frozen CLIP-L embeddings improves AI-test recall to +1,878/1,905 (98.6%, gate passed) and FLUX to 298/300, but at the cost of +fresh Open Images 473/3,000 (15.8% FPR, gate failed) and Kodak 6/24. The +nonlinear head trades photo safety for recall — same direction as quarter-hard +but via MLP instead of continuation. Gates: 2/4 passed. + +**H5 (spectral invariant, CVPR 2025):** radial power-spectrum deviation from +a camera-image reference detected only 7/297 AI-test (2.4%) and 1/300 FLUX. +The 1/f^alpha spectral profile does not discriminate current AI generators +from camera photos at any useful operating point. Gates: 0/4 passed. + +**H6 (TTA voting, 6 views):** averaging scores over center + 4 corners + flip +preserves photo gates (fresh 5/300, Kodak 0/24) but reduces AI-test from +273→264 and does not rescue any negative domain: documents still 65/80 FP, +batch1 80/80, museum drops to 1/80 but fashion stays 14/80. TTA helps museum +but hurts everything else. Gates: 2/4 passed (AI-test and fresh marginally, +Kodak and FLUX lost). + +All three hypotheses are closed. The H4 MLP is the strongest non-continuation +result (98.6% AI-test recall from frozen embeddings) but fails the photo +contract, consistent with every previous finding: CLIP-L encodes appearance, +not provenance, and no head on top of it can satisfy both contracts +simultaneously. + +### Y-phase campaign: frequency convention fix, 2026-08-27 + +The initial Y-phase campaign used FFT bin indices (16-32) instead of spatial +periods (16-32 pixels = 1/32 to 1/16 cycles/pixel), targeting the wrong band +at native resolution. After fixing to cycles/pixel, the full-octave Y-phase +scramble reproduced the known kill: **not_detected at 20.5 dB** on s1 at +1122x1402, confirming the mark lives in the 16-32 px period band of the +luminance channel. Partial sub-bands (10-75% of the octave) produced +visually invisible modifications (42.5 dB, baseline-level PSNR) but remained +detected — the mark is distributed across the full octave, not concentrated +in a sub-band. Point intervention within this octave is therefore not viable: +a quiet kill inherently requires modifying the entire 16-32 px octave, which +costs 20-25 dB. + +### Universal AI detector v3-v8: diverse training shifts the curve, 2026-08-27 + +A fundamentally different approach from all closed hypotheses: instead of +repairing Model 1 (trained on photo negatives only), train a fresh classifier +on the FULL diversity of human-made and AI-generated content. The v3 ridge on +frozen CLIP embeddings with 3,224 structured negatives achieved 88.0% recall +at 0.1% photo FPR with structured FP reduced from 33-99% to 0-14%. The v4 MLP +pushed recall to 91.3%. Crucially, v5 (fine-tuning the CLIP tower on the same +data) UNDID the structured gains, proving the improvement comes from training +data diversity on frozen embeddings, not from representation change. + +Iterations v6-v8 expanded the corpus to 49 negative domains (12,258 images) +plus 5,043 AI positives across 19+ families. The v7 MLP achieved **93.3% +AI-test recall** (matching Model 1's 93.02%) with **36 of 49 negative domains +at zero false positives**. The v8 auto-pipeline (650 additional targeted +images for weak domains) improved to **39/49 domains at zero FP** with digital +art at 24.8% (from 99.3% in Model 1), fashion at 12.0% (from 62-71%), and UI +at 12.0% (from 99.3%). Photo FP remained at 0.7% (COCO) and 0.7% (picsum). + +The trade-off curve has shifted permanently: Model 1 traded 93% recall for +33-99% structured FP; v8 delivers 93% recall with 0-25% structured FP and +39/49 domains at exactly zero. The remaining gap is concentrated in digital +art (24.8%), fashion (12%), and UI (12%) — all domains where AI-generated and +human-made content share deep visual similarity. Photo-side gates (fresh OI +≤50, Kodak 0/24) remain the open constraint. +Artifacts: `universal-v7-2026-08-27/`, `auto-pipeline-2026-08-27/`. + +### Three-tier ensemble detector, 2026-08-27 + +Combining Model 1 and v8 into a three-tier output transforms the binary +detector into a calibrated confidence system: + +- **DEFINITELY_AI** (both models agree): 79.6% recall at 0.50% photo FPR, + Kodak 0/24, 46 of 49 negative domains at zero FP, worst FP 2.1% (digital + art). This tier is production-safe for verification and moderation. +- **POSSIBLY_AI** (one model fires): 13.8% additional recall at moderate + FP. This tier flags images for human review. +- **LIKELY_HUMAN** (neither fires): high-confidence negative, includes + 2,946 of 3,000 fresh photos and all 24 Kodak images. + +The union (DEFINITELY + POSSIBLY) achieves 93.3% recall at 1.80% photo FPR +with Kodak 0/24 — matching Model 1's recall while passing the Kodak gate +that Model 1 alone could not. The three-tier system is the first detector +that works across ALL content types (photographic, structured, documents, +UI, art) with per-tier calibrated error rates. +Artifacts: `three-tier-2026-08-27/`, `ensemble-v8-m1-2026-08-27/`. + +### Provider v2: diversity training eliminates document leakage, 2026-08-27 + +Applying the same diversity-training insight to the provider cascade (adding +3,769 structured negative features to training alongside 1,936 photo +negatives) produced dramatic improvements. **Document/receipt leakage dropped +from 69.8% to 5.1%**, UI from 10% to 4.3%, fashion from 5-15% to 0-3%. AI +recall improved for OpenAI (84%, from 75%) and Google (88.8%, from 79%). +Photo leakage remained at 1.4%. The worst regression is memes at 47.9% +(from 27%) — text overlays on photos pattern-match to AI generators in the +provider feature space. Six domains achieve zero provider leakage. +Artifacts: `provider-v2-2026-08-27/`. + +### Universal v11: full taxonomy coverage, 2026-08-30 + +The master data plan identified 148 visual domains of human-made content. +Massive collection (3,343 images across 73 new domains from Openverse) +plus gap-filler (724 images from Safebooru for anime/manga/chibi/mecha +and diverse generated documents) expanded the dataset to 19,768 training +images across 136 negative domains. Universal v11 (MLP-512 on frozen CLIP, +three-tier ensemble with Model 1) achieves: + +- **DEFINITELY_AI**: 90.3% recall at 0.50% photo FPR, Kodak 0/24 +- **ANY_AI**: 95.6% recall (exceeds Model 1's 93.0%) +- **124 of 134 negative domains at exactly 0% FP** +- Worst FP: digital_art 13.8% (human fan art visually inseparable from AI art) +- Only 57 false positives across 1,090 test images in all failing domains + +This is the strongest result of the campaign: a single model that handles +photographic content, structured documents, UI screenshots, art, maps, +medical imagery, anime, and all other measured domains with near-zero false +positives on 92% of domains. The remaining failures are concentrated in two +categories (stylized digital art, heavily retouched fashion) where AI and +human content share deep visual similarity. +Artifacts: `universal-v11-2026-08-30/`, `massive-collection-2026-08-27/`, +`gap-filler-2026-08-27/`. + +### Fine-tune v12: representation limit confirmed, 2026-08-30 + +Fine-tuning the CLIP tower on the full 19,518-image dataset (6x the data +that failed in v5) achieved 95.1% standalone AI-test recall at 2.9% photo +FPR — but destroyed structured-content discrimination: 100% FP on digital +art and UI screenshots, 80% on receipts, 14/59 domains at zero FP. This +confirms that the v5 failure was not a data-volume issue but a fundamental +property: fine-tuning the CLIP embedding space causes the model to lose the +structured-content separation that frozen embeddings provide. The frozen +CLIP + MLP approach (v11) remains the only working architecture. Provider v3 +trained on all negative domains achieved 1.4% photo leak. +Artifacts: `fine-tune-v12-2026-08-30/`, `provider-v3-2026-08-30/`. + +### v13: plateau confirmed, 2026-08-30 + +Top-plan data expansion (1,169 images: TC260 500, art 500 from Safebooru, +fashion 300 from Unsplash, memes 250, Meta 79, OpenAI 40) confirmed the +frozen-CLIP plateau. v13 achieves 89.6% DEFINITELY / 95.1% ANY with +**128 of 136 domains at zero FP** (up from 124/134 in v11) and only 8 +non-zero FP domains. Fashion dropped from 9.3% to 5.3%. Digital art +remains at the representation limit (15.9%, was 13.8% — the Safebooru +expansion did not help). The frozen CLIP + MLP architecture has reached +its ceiling: further data diversity improves domain coverage but cannot +push DEFINITELY recall past ~90% or reduce art FP below ~15%. +Artifacts: `v13-final-2026-08-30/`, `top-plan-data-2026-08-30/`. + +### CLIP-H ensemble, multi-scale TTA, and patch attention, 2026-08-30 + +The 19,950 CLIP-H embeddings (1024-d, `laion/CLIP-ViT-H-14-laion2B-s32B-b79K`) +already had a ridge head from the Model 1 sweep (81.2% AI-test at 1.4% fresh +FPR). What had not been measured was using that ridge as a third voter next +to Model 1 and v13, plus a new MLP on the same cache, native-resolution TTA, +and patch-token attention. + +On the original paired cells (n=1,905 AI-test, 3,000 fresh, 24 Kodak, 300 +FLUX) the current DEFINITELY rule `Model 1 AND v13` is 89.6% / 1.17% / 0/24 / +85.3%. Replacing the AND of v13 with an OR of the CLIP-H ridge inside that +rule is the only positive result of the three: + +| Rule | AI-test | Fresh FPR | Kodak | FLUX | +| --- | ---: | ---: | ---: | ---: | +| Model 1 AND v13 | 89.6% | 1.17% | 0/24 | 85.3% | +| Model 1 AND (v13 OR CLIP-H ridge) | **91.9%** | 1.40% | 0/24 | **91.7%** | +| CLIP-H MLP, photo negatives only | 89.8% | 2.40% | 0/24 | 56.3% | + +The CLIP-H MLP without structured negatives repeats the v12 pattern: in-grid +recall holds, FLUX hold collapses. Multi-scale TTA on CLIP-L-ft (letterbox +224/336/448, mean embedding, ridge recalibrated on `photo_dev_oi`) raised +AI-test from 93.1% to 95.4% on 1,847 files and broke the operating point: +fresh 1.8% to 3.2%, Kodak 0/24 to 1/24, FLUX 92.7% to 87.3%. Patch-token +attention (256 projected patches, CLS-query, photo negatives only) was 92.2% +AI-test at 1.4% fresh and 0/24 Kodak, with FLUX at 39%. Extra views and extra +tokens without diversity training do not survive the 1% photo contract. + +The frozen CLIP-L MLP plateau in v13 still stands as a single-backbone +result. A second frozen backbone as an OR branch inside DEFINITELY is a +different lever. +Artifacts: `cliph-ensemble-2026-08-30/`, `multiscale-tta-2026-08-30/`, +`patch-attention-2026-08-30/`. + +### CLIP-H diversity MLP, 2026-08-30 + +The 8,333 taxonomy images that v13 added were embedded with the same CLIP-H +tower (1024-d). A v13-protocol MLP (5,330 pos including 809 extra AI, 6,129 +photos, 6,674 structured negatives, 113 domains) was trained on that cache. + +Diversity training gave CLIP-H the structured-content property: **113/113 +taxonomy domains at 0% FP** for the CLIP-H MLP. It did not repair FLUX. +Standalone CLIP-H diversity is 83.0% AI-test / 1.20% fresh / 0/24 Kodak / +66.0% FLUX, better than the photo-only CLIP-H MLP (56.3% FLUX) and still +worse than the original CLIP-H ridge (91.3% FLUX). Linear CLIP-H stays the +robust voter; the MLP spends that robustness on domain coverage. + +| Rule | AI-test | Fresh FPR | Kodak | FLUX | +| --- | ---: | ---: | ---: | ---: | +| Model 1 AND v13 | 89.6% | 1.17% | 0/24 | 85.3% | +| Model 1 AND (v13 OR CLIP-H ridge) | 91.9% | 1.40% | 0/24 | 91.7% | +| Model 1 AND (v13 OR CLIP-H diversity) | 91.1% | 1.27% | 0/24 | 88.3% | +| Model 1 AND (v13 OR ridge OR diversity) | 92.2% | 1.43% | 0/24 | 92.0% | + +The production-shaped DEFINITELY rule remains Model 1 AND (v13 OR CLIP-H +ridge). Adding the diversity MLP on top is +0.3 pp recall at +0.03 pp fresh. +The 113/113 zero-FP count is only over taxonomy cells CLIP-H actually +embedded, not a re-score of v13's original `scores.npz` domains. +Artifacts: `cliph-diversity-2026-08-30/`, `embeddings-clip-h-taxonomy.npz`. + +### Provider v4: focal loss and TC260 expansion, 2026-08-30 + +Provider v4 refit the five-head 124-d cascade on the v2 bank plus top-plan +expansion (TC260 1,213, Meta 199, OpenAI 2,038, 4,087 structured negatives, +250 imgflip memes as extra negatives). A one-vs-rest ridge and a focal-loss +MLP (`gamma=2`) were trained on the same rows. Full `ai_test` family cells, +not the 100-row v2 subsample: + +| Head | OpenAI | Google | TC260 | photo COCO | `meme_template` | +| --- | ---: | ---: | ---: | ---: | ---: | +| v4 ridge | 83.2% | 83.9% | 68.1% | 1.4% | 50.7% | +| v4 focal | **93.4%** | 84.7% | **80.5%** | **0.7%** | **33.8%** | + +Focal meets the TC260 ≥75% target and lifts OpenAI. Imgflip memes used in +training score 0% leak on themselves and are not a test; the honest meme +cell remains `meme_template` at 33.8%, down from 47.9% but short of 15%. +Meta 38/39 was a train-end slice and is not a hold-out. Artifacts: +`provider-v4-2026-08-30/`. + +### Holes: digital art, memes, Meta hold-out, 2026-08-30 + +Three remaining cells were re-measured with disjoint splits. + +**Digital art.** The 145 Danbooru rows are in v13 training and still +DEFINITELY at 15.9% (`m1 AND v13`). Adding CLIP-H ridge as an OR branch, +the rule that lifts AI-test to 91.9%, fires on 138/145 (95.2%): that +branch is unsafe on illustration. A CLIP-H diversity MLP, trained on +taxonomy art and not on these 145, scores 4.8% AI. A global AND with +that MLP drops FLUX from 91.7% to 64.7%. An illustration-versus-photo +ridge (807 taxonomy art/anime/paintings vs `photo_train`, 1% photo-dev +FPR) restricts the veto to illustration-like images: digital art 95.2% +to **4.8%**, AI-test 91.9% to 88.7%, FLUX 91.7% to 89.7%, Kodak 0/24, +fresh 1.40% to 1.27%. SD/Comfy takes the largest family drop (93.2% to +81.4%). The 145 were not in the illustration or diversity training +caches. + +**Memes.** `meme_extra` is a content-hash duplicate of `meme_template` +(97/100). Holding the 97 hashes out of provider training and evaluating +them against 250 disjoint imgflip negatives yields **40.7% ungated +leak**. Training on 48 of the 97 and testing the other 49 still leaks +34.8%, so 124-d forensic features do not represent this domain. +Under the detector, DEFINITELY is 2/97 and those two stay `no_ai`: +**gated provider leak 0%**. Provider must run only on DEFINITELY / +POSSIBLY, not on every file. + +**Meta.** Training on the 122 muse-image rows and testing the later +`photo_ai_meta_v2` pass (77 readable of 79) is **70.1%**. That is the +hold-out. The earlier 38/39 figure was a train-end slice and is +withdrawn. Adding the 95 disjoint `photo_ai_meta` files already on disk +from `targeted-data-2026-08-27` (same model, different collection, zero +file-sha256 overlap with muse or v2) lifts the same hold-out to **67/77 += 87.0%** without new generation. + +That number is only reproducible from a frozen split. The collectors +hashed the API payload and then re-encoded PNG, so `targeted-data` and +`top-plan` Meta rows had a `sha256` that did not match the file on +disk (95/95 and 79/79). Muse webp rows already matched. The repaired +identity is sha256 of the file bytes; the old payload hash is kept as +`payload_sha256`. The freeze is `meta-split-2026-08-31/split.json` +(217 train = 122 muse + 95 targeted, 79 hold-out). Training reads that +file, re-hashes every path, and records extractor refusals instead of +dropping them: two hold-out 1280x1920 images and one 1600x1600 train +image return no 124-d features (`n_scored=77` of 79 listed). Seed +`20260956`. Re-run: `uv run python .local-eval/synthid/ai-photo-2026-08-22/meta_provider_split.py`. +Artifacts: `holes-2026-08-30/`, `meta-expand-2026-08-31/`, +`meta-split-2026-08-31/`. + +### Both-model freeze and repeated training, 2026-08-31 + +A single catalog now identifies every file used by the detector and the +provider by the sha256 of the bytes on disk (`both-models-2026-08-31/catalog.json`, +32,690 files, 590 listed missing). Roles cover Model 1 splits, taxonomy +struct/AI extras, and provider OpenAI/Google/TC260/Meta/no_ai plus the Meta +and meme hold-outs. OpenAI and Google are a fresh seed-`20260821` sample of +2,000 existing labeled files each, with catalog sha256 verified at sample +time; they are not the anonymous `provider-class-features.npz` arrays. +`no_ai` is a documented 2,000-draw from `photo_train` (1,848 yielded 124-d +features). + +The first CPU repeat that morning still trained on 150 local product UI +screenshots in `batch1-negative-2026-08-27` (`own_products_ui`). Those files +are not public internet. They were replaced the same day with 150 +CC-licensed Flickr screenshots via Openverse (`license_type=all-cc`, host +`live.staticflickr.com` only): CC BY 2.0 42, CC BY-NC 2.0 42, CC BY-SA 2.0 +29, CC BY-NC-SA 2.0 28, CC BY-NC-ND 2.0 6, CC BY-ND 2.0 2, CC0 1. Collector: +`replace_private_ui.py`. File sha256 after PNG save is identity. Catalog +`own_products_ui` is now 0; the 150 Flickr rows carry `source=openverse_flickr` +and a license. CLIP-L extras for those hashes live in `extra-clip-l.npz` +(150 × 768). 127 of 150 yielded 124-d features; 23 are listed in +`features-124d-failed.json` (1,808 refusals in the catalog after the swap). +CLIP-L embeddings keyed by sha256: 26,573. + +Training is CPU, seeds `detector=20260940` and `provider=20260956`. Two +consecutive runs from the updated freeze were **byte-identical** +(`compare.json`, zero diffs). Pre-replacement: AI-test DEF 92.7%, FLUX +84.0%, OpenAI 90.0%, Google 89.8%, TC260 79.2%, Meta 83.1%, meme leak 33.7%. +After the public UI swap: + +| Head | Cell | Run 1 = run 2 | +| --- | --- | ---: | +| Detector DEFINITELY | AI-test | 92.4% (n=1,847) | +| Detector DEFINITELY | Fresh | 1.3% (n=3,000) | +| Detector DEFINITELY | Kodak | 0/24 | +| Detector DEFINITELY | FLUX | 83.0% (n=300) | +| Provider | OpenAI test | 90.8% (345/380 of 381) | +| Provider | Google test | 90.9% (339/373 of 377) | +| Provider | TC260 test | 78.6% (298/379 of 384) | +| Provider | Meta hold-out v2 | **85.7%** (66/77 listed 79) | +| Provider | Meta hold-out v3 | **89.4%** (177/198) | +| Provider | Meta hold-out pooled | **88.4%** (243/275 listed 277) | +| Provider | meme_template ungated | 29.1% leak (86 of 97) | + +A second Meta hold-out (`photo_ai_meta_v3`, 198 of 200 requested; two +near-duplicates rejected) was generated the same day from 50 new +photographic prompts, four passes, requested size 1024x1024 (delivered +1600x1600). It is eval-only: not in `provider_meta_train` and not in +detector extras. File sha256 after PNG save is identity. Scoring the +already-frozen `run1/provider.pt` and the retrained freeze both give +177/198. Pooled with v2 the lower 95% Wald bound sits above 80% +(243/275). Collector: `collect_meta_holdout_v3.py`. Split: +`meta-split-2026-08-31/split.json` (train 217, hold-out v2 79, hold-out +v3 198). + +Re-run: `uv run python .local-eval/synthid/ai-photo-2026-08-22/replace_private_ui.py` +then `uv run python .local-eval/synthid/ai-photo-2026-08-22/collect_meta_holdout_v3.py` +then `uv run python .local-eval/synthid/ai-photo-2026-08-22/integrate_meta_holdout_v3.py` +then `uv run python .local-eval/synthid/ai-photo-2026-08-22/freeze_both_models.py`. +Durable copy: `data/research/reports/both-models-2026-08-31/` (gitignored). + +Verification (`verify_both_models_freeze.py`) re-hashed all 32,690 catalog +files: every path exists, every sha256 matches disk, `own_products_ui` is +0, Meta v3 is disjoint from train by file hash and by prompt, v3 is +eval-only, all 198 v3 rows yielded 124-d, and the two freeze runs stay +byte-identical. One v2 hold-out is a Hamming-15 phash neighbor of a +Venice-canal train image (rainy Paris boulevard, different prompt and +aspect); that file also failed 124-d extraction, so it is not in the 77 +scored. A 16-image visual sample of the Flickr UI cell is **12 +screenshots, 3 photographs, 1 product still**. License and host checks +pass; the Openverse screenshot queries are not a UI-only cell. Re-run: +`uv run python .local-eval/synthid/ai-photo-2026-08-22/verify_both_models_freeze.py`. + +### Five-head provider cascade with TC260, 2026-08-27 + +Adding TC260 (721 Chinese-AIGC images from the local research catalog) as a fifth +class to the provider cascade produced a working five-way system: +openai 68.8%, google 76.6%, **tc260 63.8%**, meta 71.8% mean recall under +uniform margin 0.50 (25 repeated splits). Per-class margin calibration +against a 1.67% aggregate photo-AI-rate budget (0.42% per class) yielded +margins openai 0.31, google 0.29, tc260 0.39, meta 0.55, and TC260 test +recall at calibrated margins reached **71.4%** on 196 held-out images. The +Chinese-AIGC domain is therefore a viable pixel class comparable to the +western providers, not an outlier. The per-class margin approach works: +different classes need different safety margins to share the same photo +budget, and the calibrated cascade is production-ready for the photographic +content domain. Artifacts: `tc260-five-head-2026-08-27/report.json`. + +The combined-pool control then separated the data question from the geometry +question: refitting the same pooled/domain/multiclass ridge vetoes on a +687-row modern-negative pool (Openverse-clean plus all three stock cells, +three times the original fit size) reproduced the earlier Pareto curve to the +point: at positive compliance the best dev error count stayed 41/186 with +zero route errors repaired, and every repaired negative still cost about 1.4 +AI positives. The linear-veto failure over the frozen Model 1 embedding is +therefore not a data-volume artifact; tripling diverse negatives moves +nothing. Artifacts: +`modern-domain-veto-combined-2026-08-27/` (development report and pareto). + +The doubled Meta corpus then allowed its own margin sweep (127 API rows in +the bank after feature extraction, 25 repeated splits, photo leak measured on +held-out no_ai rows). The curve is steep exactly where the frozen 0.50 margin +sat: margin 0.00 recalls 92.2% meta at 5.3% photo AI-rate, 0.10 gives 90.1% +at 3.6%, 0.20 gives 85.2% at 2.2%, then the cliff: 0.30 falls to 70.2%, 0.40 +to 39.4%, and the frozen 0.50 lands at 12.4%. A workable photo-safe +operating point exists at margin 0.20 (85% meta recall at 2.2% photo leak), +but 2.2% is still thirty times the Model 1 photo contract, so for provider +attribution the honest rule stays argmax until an external held-out meta cell +exists. The margin, not the head, was always the limiter on the small class. +Artifacts: `meta-margin-2026-08-27/report.json`. + +## Representation-change campaign closed, 2026-08-27 + +A pre-registered two-candidate campaign (plan in `data/research/REPRESENTATION-CAMPAIGN.md`, +gates declared before any run: AI-test at least 1,772/1,905, fresh Open +Images at most 50/3,000, Kodak 0/24, FLUX at least 288/300, EvalGEN at least +95/100, no tuning) tested the two remaining representation families. +Candidate 1, a DINOv2-L ridge head (224 px CLS embeddings, nested-CV penalty), +failed all five gates simultaneously: AI-test 1,427/1,847, fresh 162/3,000, +Kodak 1/24, FLUX 29/300, EvalGEN 5/100 — the frozen DINOv2 space separates AI +from photos far worse than fine-tuned CLIP-L everywhere. Candidate 2, a +patch-level FFT tower on the DDA paired-reconstruction contract (1,024 fresh +sd-vae-ft-mse pairs, real 0.549 versus reconstruction 0.457 medians, 0.949 +pairwise separation), learned its contract perfectly and still collapsed on +transfer: AI-test 14/1,847, FLUX 0/300, EvalGEN 0/100, because production +generators do not carry this VAE's reconstruction signature. The declared +falsification criterion triggered: neither family is the missing lever, and +the general metadata-free AI-versus-human classifier at the product contract +is beyond local scale as pursued. The product keeps the documented-remainder +contract; Model 1 stays the accepted research operating point with its +measured bounds. Reopening requires a materially new lever. + +A taxonomy continuation then changed the training mix itself instead of the +veto: two arms re-ran the expanded quarter-hard recipe (ordinary AI replay, +EvalGEN pair positives, ordinary photos) with one or two of the eight negative +slots per batch drawn from the date-clean taxonomy pool (Cleveland museum plus +Openverse modern cells), 256 steps, thresholds calibrated at the 99th +percentile of `photo_dev_oi`. Both arms retained the positive contract +(EvalGEN 97-99/100, FLUX 289/300) and repaired the museum half (AIC dev +4/350), but the modern half did not move: fashion stayed at 33-35/88 on the +date-clean Unsplash cell against a 30/88 baseline, and long exposure at +11-12/98. Training on 40 date-clean Flickr fashion rows does not transfer to +Unsplash fashion, which mirrors the veto finding from the optimization side. +The development domain gate therefore fails on fashion for every arm, and no +arm reached the frozen core cells. With the linear-veto family, aligned +mining, self-hard continuation, and now in-mix taxonomy continuation all +closed on the same source-specific overlap, the CLIP-L program for the modern +negative domains is measured as exhausted: representation change is not one +more operating point away. + +Local reproducibility artifacts are under +`.local-eval/synthid/ai-photo-2026-08-22/`, especially the +`paired-reconstruction`, `open-images-train-mining`, `evalgen-expanded`, +`evalgen-quarter`, `evalgen-rescue`, `evalgen-stacker`, +`negative-taxonomy-audit-2026-08-25`, `negative-taxonomy-aic-audit-2026-08-25`, +`modern-negative-unsplash-2026-08-26`, `modern-negative-openverse-2026-08-26`, +`modern-domain-veto-2026-08-26`, `modern-domain-veto-clean-2026-08-26`, +`contamination-scan-2026-08-26`, and `taxonomy-continuation-clean-2026-08-26` +run directories. + +### Wild extras, not SynthID + +| Hypothesis | 2026-08-23 | Use | +| --- | --- | --- | +| Missing camera PRNU | Gray `gpt-image-2` highpass RMS 0.25 vs COCO 14.6 | Texture confound. A Wiener PRNU residual on *photographs* vs Model 1 errors is the real test | +| JPEG ELA | COCO 3.13, s1 1.97, gray stamp 0.49 | Export history, leaks PNG vs JPEG, not a provider | +| CFA / Bayer presence | Photo-edit ratio 0.117 vs camera 0.184 vs gray 0.588 | Weak camera vote, overlap. Inverse of the Bayer remover | +| Double-JPEG ghosts | s1 / gpt-image-2 / camera all min at Q90 | Codec, not a provider | +| Perfect-circle / text-edge rate | Circles/MP 385 vs 536, edge 0.052 vs 0.072 | Too noisy for abstain | +| Wiener PRNU on photographs | Edits 4.61 vs camera 8.05 | Donor JPEG texture leftover, not a missing sensor | +| PNG Paeth filter mix | gpt-image-2 PNG 99.9% Paeth vs camera 73% | Export fingerprint | + +None of these should be named a SynthID score. + +## External literature (surveyed 2026-08-23) + +AWPD / FSNet ([arXiv:2603.06723](https://arxiv.org/abs/2603.06723)) is +the published "is there any invisible watermark" task. Leave-one-algorithm-out +SynthID Acc 0.894 is *not* Model 1 and *not* a payload decoder. UniFreq's +SynthID split is 2,000 Imagen-API AIGC crops at 256x256, no photographs, +no Firefly, no OpenAI. A head trained that way can pass as watermark +presence while actually reading generator/size texture, which is the L1 +failure mode. + +Model 1 remains AI-versus-camera on CLIP-L-ft. That is a published +task, not a watermark task. Adjacent papers: + +| Source | Claim | Map to Model 1 | +| --- | --- | --- | +| Ojha, Li, Lee, [arXiv:2302.10174](https://arxiv.org/abs/2302.10174) (CVPR 2023, UnivFD) | A classifier trained to see "fake" treats unseen generators as the real sink. Frozen CLIP + nearest neighbor / linear probe generalizes better than a trained CNN | This is the architecture. We finetuned the last two CLIP-L vision blocks instead of freezing, and put Firefly and a locked Open Images fresh set in the gate | +| Cozzolino et al., [arXiv:2312.00195](https://arxiv.org/abs/2312.00195) | CLIP linear probe, few shots from one generator, holds on DALL-E 3 / Midjourney / Firefly | Firefly is the cell we required. Their paper is why Firefly belongs in the test, not as a surprise | +| Corvi et al., [arXiv:2304.06408](https://arxiv.org/abs/2304.06408) | Spectral peaks and mid-high power differences, GAN and diffusion | Generator fingerprint, not a payload. Explains why a Fourier codebook lights up Google *and* Open Images | +| Zhong, Xu, Zou, [arXiv:2601.22778](https://arxiv.org/abs/2601.22778) (DCCT) | Self-supervised color-channel prediction under a Bayer mask; theoretical gap between photo CFA correlations and AIGC | Local Bayer interpolation-error ratio: edits 0.117 vs camera 0.184. Weak vote, not a payload | +| Klier and Baier, DFRWS EU 2026 | AI noise is not predominantly additive. Standard PCE vs smartphone PRNU: FPR 61% Firefly Image 4, 100% ChatGPT 5. Center crop kills those false positives without hurting true camera matches | Do not call missing PRNU a SynthID score. If we ever add a Wiener residual, crop and a recorded PCE threshold come with it | +| Popescu and Farid, IEEE Trans. Signal Process. 2005 | CFA interpolation leaves neighbor correlations; splicing breaks them | Classical forgery localization, not generation detection | +| Wang, Wang, Zhang, Owens, Efros, [arXiv:1912.11035](https://arxiv.org/abs/1912.11035) (CVPR 2020, CNNDetect) | Classifier on ProGAN + JPEG/crop aug transfers to many CNNs | The "one generator is enough" claim. Ojha is the correction once diffusion exists | +| Wang et al., DIRE, [arXiv:2303.09295](https://arxiv.org/abs/2303.09295) (ICCV 2023) | Reconstruction error under a frozen diffusion model | SDXL float32 at 512. VAE RMS: camera 11.84, photo edit 9.89, s1 9.15, gray stamp 1.24. DDIM DIRE RMS: camera 33.0, s1 31.5, photo 30.9, gray 2.40. Texture rank, not a payload. Float16 DDIM NaN'd on MPS | + +They do not substitute for `verify-openai-synthid`. diff --git a/docs/index.md b/docs/index.md index a537dac..0eb9127 100644 --- a/docs/index.md +++ b/docs/index.md @@ -35,6 +35,10 @@ The current behavior is defined by the code, tests, README, and user guides. - [ControlNet removal research](controlnet-removal-pipeline-research.md) - [Qwen improvement research](qwen-improvement-research.md) - [Doubao reverse-alpha research](research-doubao-distillation.md) +- [AI-generated image classifiers](ai-generated-image-classifiers.md) (photo AI-versus-camera freeze, not a product verdict) +- [SynthID source classifiers](synthid-classifiers.md) +- [SynthID local detector research](synthid-detector-research.md) +- [SynthID mark removal research](synthid-removal-research.md) - [SynthID identity research](synthid-robust-identity-research.md) - [SynthID identity follow-up](synthid-robust-identity-research-2026-06-08.md) - [Video SynthID quality research](video-synthid-quality-research.md) diff --git a/docs/synthid-classifiers.md b/docs/synthid-classifiers.md new file mode 100644 index 0000000..dcd9047 --- /dev/null +++ b/docs/synthid-classifiers.md @@ -0,0 +1,518 @@ +# SynthID source classifiers (research) + +> Research archive for metadata-free OpenAI/Gemini source finding and +> provider-lineage classifiers. These are not SynthID payload decoders and are +> not shipped product verdicts. Current behavior: +> [supported signals](supported-signals.md) and +> [known limitations](known-limitations.md). +> +> Sister pages: [general AI-generated image classifiers](ai-generated-image-classifiers.md), +> [SynthID local detector](synthid-detector-research.md), +> [SynthID mark removal](synthid-removal-research.md), and +> [mechanism reference](synthid.md). + +A source classifier is reliable only when photographs and non-target generators +are explicit negatives, a strict rule can abstain, and any watermark claim uses +an independent oracle. CLIP content embeddings and the 124-d origin-locked +residual bank are different features for different jobs. + +Krawetz's Gemini-chat TPR critique is a verifier-quality claim, not a +feature we can ship. [Lead Stories, 2026-07](https://leadstories.com/analysis/2026/07/google-gemini-synthid-detector-confuses-results-within-same-chat.html) +documented Gemini repeating the first file's SynthID verdict inside a +chat; Google said that was fixed 2026-07-16. The OpenAI provenance API is +a different endpoint. + +## Provider names from pixels + +The task is: given a file with no metadata, is this OpenAI, Gemini, or +unknown, with almost no errors on camera photographs or other generators. That +is this section. `unknown` does not mean `not AI`; it includes AI images from +other providers and target-provider images the strict rule misses. This is not +a general AI-generation detector or a SynthID detector. Firefly, PixelBin, and +other generators have to sit in the test, because a head that only sees OpenAI +versus Gemini versus COCO will call Firefly a provider. + +Three-way `openai` / `google` / `other` on Model 1 embeddings fails the +Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57; +Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the +124-d lattice bank it is 0.989. They are two pipelines, not one class. + +### Renderer, not front-end (cross-carrier validation, 2026-08-26) + +A local C2PA-labeled collection's issuer field allows a provider-label audit the +family labels cannot express: Bing Image Creator rows signed `Microsoft, +OpenAI` (renderer DALL-E, n=152), Microsoft-native rows (MAI-Image / Designer, +n=85), Designer rows signed `Microsoft, Google LLC` (renderer Imagen, n=36), +and the unattributed Instagram `made_with_ai` label set (n=275). Scored with +the frozen 124-feature openai/google/no_ai cascade, no training, metadata +used only to build cells: argmax called 124/152 (81.6%) of the Bing+DALL-E +cell `openai` and 34/36 (94.4%) of the Microsoft+Imagen cell `google`. Pixel +attribution therefore tracks the renderer behind the front-end, which is the +correct semantic for a stripped file: a Bing export is an OpenAI-rendered +image. The Microsoft-native cell split 48 openai / 35 google, consistent with +Designer historically routing between DALL-E and Imagen rather than one +pipeline. The Instagram `made_with_ai` set is not a Meta pixel class: argmax +leaned `google` at 196/275 with 68 `openai`, and the photo-first 0.50 margin +sent 182/275 to `no_ai`, so the label marks detected AI content of mixed +origin, not Meta-rendered pixels. A real Meta class needs fresh +`Imagined with AI` generations with known provenance; the catalog label is +not one. Artifacts: +`provider-renderer-cells-2026-08-26/report.json`. + +### A Meta class from muse-image-1.0, 2026-08-26 + +The Meta Model API (`api.meta.ai/v1`, OpenAI-compatible images endpoint) made +a known-provenance Meta corpus possible without any account browser session: +61 images generated from a 61-prompt grid spanning portraits, product shots, +scenes, food, animals, architecture, illustration styles, text posters, and +abstract work across all three aspect ratios, plus the five oracle-verified +oracle-verified content-seal samples already on disk. All 66 are content-hash +unique, delivered at 1600x1600 / 1920x1280 / 1280x1920, and every API row +carries IPTC `trainedAlgorithmicMedia` plus a Content Seal generation id +recorded in the manifest. A four-head control with a `microsoft_native` class +(83 issuer-verified rows) failed first: repeated-split argmax recall 0.38 mean +with 0.16-0.56 spread, because Designer routes between renderers and the class +is mixed. The same recipe with `meta_muse_image` in its place holds: argmax +recall 0.86 mean (0.75-1.00) on just 66 images, openai 0.952 and google 0.956 +unharmed, and the frozen cross-carrier cells keep their renderer semantics +(Bing+DALL-E 119/146 openai, Microsoft+Imagen 32/36 google) with only small +meta leakage (14 and 3 argmax rows). The class label names what the corpus is: +muse-image-1.0 API output, not the unverified assumption that the consumer +Imagine feature renders identically. The full-data model calls 59/66 meta rows +correctly, with seven leaking to google and none to openai. The photo-first +0.50 margin does not transfer to a 66-image class (0.398 mean recall); argmax +is the honest operating rule until the corpus grows. Muse Image is therefore +a separable fourth provider class on pixels, unlike the Microsoft front-end. +Artifacts: `meta-muse-corpus-2026-08-26/`, `four-head-provider-2026-08-26/`, +`meta-provider-cells-2026-08-26/`. + +A paired chat-vs-API check then closed the label question. Two prompts from +the API grid (weathered fisherman portrait, white-sneaker product shot) were +submitted to the consumer Meta AI chat imagine flow in the user's own logged-in +browser session; the chat delivered 1280x1920 and 1920x1280 WebP files from the +`t39.105495-1` CDN family, each carrying IPTC `trainedAlgorithmicMedia` in the +downloaded bytes. Scored by the frozen API-trained four-head model, which had +never seen a chat image: both chat rows landed on `meta_muse_image` under +argmax (0.251 and 0.233) with score profiles matching their API twins almost +exactly, and the margin rule agreed pairwise (fisherman meta at margin, sneaker +no_ai at margin 0.50, same as its API counterpart). The chat imagine pipeline +and `muse-image-1.0` are therefore pixel-indistinguishable to the provider +classifier on this pair, so `meta_muse_image` honestly names both +distributions. Caveat: n=2 paired prompts from one session; this is a +consistency result, not a deployment-scale equivalence claim. Artifact: +`meta-chat-check-2026-08-26/`. + +Collapsing OpenAI and Gemini into one pixel class versus other generators +does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%, +Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at +3%. Training the same union only against photographs recreates Model 1 with +a narrower train set (fresh FPR 2.1%, Firefly still 95%). + +`provider-report-clip-l-ft.json`, `provider-union-report.json`. + +### Strict abstention does not recover provider identity + +An `unknown` class makes the claim narrower, but neither existing +representation has a useful high-precision operating point. A +leave-one-foreign-family-out ridge on the 124-d residual bank set each cut +above every calibration negative. Mean OpenAI recall fell to 1.3% and mean +Google recall to 4.4%, while a held-out foreign family was still attributed to +a provider up to 5.0% of the time and photographs were called OpenAI up to +0.52% of the time. + +The CLIP-L-ft head has the same failure at a 0.1% calibration-negative cut. +On 400 disjoint queries per provider, OpenAI recall was 1.0% and Google recall +was 0.5%. It still called 1/11 held-out FLUX images OpenAI and 1/750 COCO +photographs OpenAI. Zero calibration false positives reduced both provider +recalls to zero. These are abstaining content classifiers, not precise source +finders. + +A nonlinear head and local-neighbor decision do not expose a hidden provider +partition in the same embeddings. Across five balanced 768-128-3 MLP seeds, +strict test recall ranged from 0.5-7.0% for OpenAI and 0.75-1.0% for Google, +with 6-19 false attributions among 6,550 locked negatives. The best +calibration-selected `k=3` local-neighbor rule reached 0.5% OpenAI and 3.25% +Google recall and still called one TC260 image Google. + +### Pixel finetuning learns the train split, not an exact provider boundary + +The last two CLIP-L vision blocks were then finetuned directly for +`openai` / `google` / `unknown`: 9,063 fit images, 3,537 disjoint calibration +images, 400 balanced steps, and random JPEG 40-95, 85-100% crop, and mild blur. +Each provider cut was placed above every calibration negative. Calibration +recall was 4.3% OpenAI and 5.2% Google. + +The time-disjoint locked result was 6/400 OpenAI and 10/400 Google. One Google +image and one TC260 image were called OpenAI. All 500 unseen-AI controls and +all 4,945 locked photographs stayed `unknown`, including 3,000 fresh Open +Images, but that photo specificity does not repair an AI-source error. An +oracle cut above both locked OpenAI errors leaves only 1/400 OpenAI; it is an +upper bound, not a valid post-test threshold. The model is not shippable. + +The independent high-frequency route is already closed at the tested +capacity. A four-layer opponent-residual patch CNN reached AUC 0.44-0.56 +against foreign generators, reversed to 0.15 on a fresh era, and accepted +95-100% of several held-out Firefly, Microsoft, fal.ai, and PixelBin families +at its photo-median threshold. It learned AI rendering versus photography, +not vendor identity. + +### External surrogate and forensic-descriptor audit + +The public +[`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09) +does not supply a causal SynthID contrast. Its negatives are lightly +regenerated positive images, so the trained ResNet/EfficientNet ensemble can +read the regeneration pipeline. On a blind 517-file local pilot, OpenAI versus +all AUC was 0.630. At the repository's 0.5 cut it retained 92/100 OpenAI and +accepted 307/417 negatives, including 104/120 Open Images, 26/30 COCO, 34/50 +Google, and 8/10 Firefly. A later exact repeat on the hash-disjoint v7 challenge +retained 172/200 OpenAI but accepted 110/200 Google. It therefore fails source +specificity before any photograph gate is considered. It is a visual-domain +classifier, not an independent confirmation signal. + +The current [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID/tree/b110836) +V4 codebook also adds no useful hybrid evidence. A pickle-free exact inference +repeat on v7 accepted 77/200 Google and 76/200 OpenAI at its published 0.52 +threshold. Applied only to v11 `unknown` rows, that threshold would rescue 26 +Google files while misrouting two OpenAI files. The older V3 published cut +would add two v11 Google misses, but it previously accepted 5/499 controls and +6/1,000 fresh Open Images. A 1%-recall OR rule with that measured false-positive +history is also rejected. + +The public [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847) +SPAM model is another forensic descriptor, not a decoder. The audited pickle +contained only an sklearn pipeline, scaler, logistic regression, and numeric +numpy globals; inference used an exact restricted allowlist. At the published +0.5 cut it accepted 139/200 Google and 141/200 OpenAI. On the disjoint public +extension, the same published cut also accepted 214/600 ImageNet photographs, +14/75 BigGAN, 17/75 Midjourney, 52/75 SDXL, and 10/75 VQDM. It adds no safe v11 +rescue. + +[`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad) +publishes a useful flat-field corpus but an invalid open-world classifier +contrast. Its Tier-1 matrix has 500 Gemini-flat positives and 1,500 synthetic, +spectrum-matched, or phase-scrambled negatives, with no real negative. The +33 features include noise scale and radial power as well as six hand-selected +carrier bins. Rebuilding the repository's seed-42 RBF SVM from the safe numeric +matrix, without loading joblib, accepted 1/200 current Google and 0/200 OpenAI +on v7. The reported AUC above 0.999 measures the synthetic negative recipe and +flat renderer epoch; it cannot confirm the current source finder or a SynthID +payload. + +[Forensic Self-Descriptions](https://github.com/ductai199x/Forensic-Self-Descriptions-CVPR25/tree/50f2eae) +(CVPR 2025) is a genuinely different representation: constrained prediction +residuals are summarized as a 960-d forensic descriptor. Its ready attribution +head is not usable here. Three current OpenAI files were all called `Real`, +while one Microsoft file was called `GPT-Image 1/1.5` at confidence 0.863. + +A custom source head on a native 256-pixel center crop was more informative. +PCA-64 plus logistic regression, with each provider cut above every calibration +negative, gave OpenAI AUC 0.872 and 3/50 strict OpenAI recall with 0/388 test +false attributions. Its Google head failed on Bytedance, Canva, FLUX, +Microsoft, and an Open Images photograph. On a later unused-hash challenge the +OpenAI FSD head alone accepted 14/300 OpenAI, 3/300 TC260, and 3/90 PixelBin. +FSD is therefore an independent source cue, not a sufficient classifier and +not a watermark statistic. + +### Rejected narrow two-signal cascades, 2026-08-23 + +The visible Gemini sparkle is absent from the target generation and is not a +feature or a gate. Two metadata-free intersections did survive a new challenge: + +| Cascade | Development | Unused-hash challenge | +| --- | --- | --- | +| Google provider probability >= 0.8876249 **and** registered-v3 lattice >= 1.0 | 3/45 Google, 0/30 lattice-positive foreign files | 3/300 Google, 0/690 OpenAI/TC260/PixelBin | +| OpenAI provider probability >= 0.8822824 **and** FSD probability >= 0.9912320 | The two cuts were frozen independently | 3/300 OpenAI, 0/690 Google/TC260/PixelBin, 0/500 photographs | + +The Google result is a low-recall intersection of a content head and an +origin-locked renderer lattice. A two-pixel crop destroys the lattice, so an +indeterminate result says nothing about SynthID. It is evidence for a narrow +Gemini/Imagen pipeline, not a public decoder. + +All three OpenAI challenge positives were 1086x1448 RGB PNGs, the measured +ChatGPT export geometry. Thirty known non-OpenAI files at exactly 1086x1448 +(TC260, Apple, Microsoft, Canva, Firefly, Google, and others) produced zero +first-gate positives. A scan of 1,194 unique files at that geometry with no +provider or platform label produced 29 pixel-gate candidates and four final +hybrid candidates. Three had only PNG image-data chunks; the fourth carried +generic DPI/orientation fields but no provider or provenance label. Visual +inspection found four plausible AI generations. They remain candidates, not +ground truth, until a held-out oracle or original source confirms them. + +The full luma 16-32-pixel phase-ring head is not a third confirmation. It +rejected all three known and all four unlabeled OpenAI hybrid candidates while +accepting 3/30 known non-OpenAI same-geometry controls. Its global test AUC was +0.600 for OpenAI and it stays closed. + +These cascades established that independent intersections could suppress the +observed false positives, but about 1% recall is not useful for the source-hunt +task. They are rejected as operating detectors. The FSD implementation and +weights are also research-only under CC BY-NC-SA 4.0. + +Local artifacts: `lattice-pixel-hybrid-challenge.json`, +`fsd-openai-challenge-report.json`, +`unlabeled-1086x1448-openai-hybrid-candidates.json`. The research entry point +is `.local-eval/synthid/ai-photo-2026-08-22/hybrid_source_finder.py`. + +### Canonical forensic source finder, 2026-08-23 + +An overlooked pixel representation materially improves the strict result. It +normalizes each file to 512x512, then extracts 1,794 metadata-free features +from opponent-color residuals: patch power and variation, phase coherence, +autocorrelation, low-bit correlations, and residual-error histograms. The +OpenAI branch intersects its canonical forensic score with a weak provider +pixel vote. The Gemini branch intersects a forensic/provider fusion with the +registered-v3 lattice. The visible Gemini sparkle remains excluded. + +The single-scale cuts were frozen before a third hash-disjoint challenge. The +result was: + +| Challenge cell | OpenAI candidate | Gemini candidate | Any candidate | +| --- | ---: | ---: | ---: | +| OpenAI | 51/300 | 1/300 | 52/300 (17.3%) | +| Google | 0/300 | 34/300 | 34/300 (11.3%) | +| Open Images photographs | 1/1,000 | 0/1,000 | 1/1,000 (0.1%) | +| TC260 | 0/7 | 0/7 | 0/7 | + +For the requested union, 86/600 OpenAI-or-Google targets were found, 14.3% +recall instead of the rejected cascades' 1%. One OpenAI file crossed through +the Gemini branch, and one known photograph crossed through the OpenAI branch. +The seven remaining unused TC260 files are too few to establish specificity, +and no fresh PixelBin, Firefly, Microsoft, or other generator hashes remained +outside the earlier model and challenge manifests. + +This v3 result established a strict source-candidate finder, not an exact +identifier or a SynthID decoder. It does not recover a payload, and an +`unknown` result says nothing about whether SynthID is present. + +Local artifacts: `source-finder-v3-selection.json`, +`source-finder-v3-challenge.json`, `google-canonical-forensic.json`, and +`google-forensic-pixel-fusion.json`. The same local research entry point now +implements this rule without the FSD dependency. + +### Multiscale strict finder, 2026-08-24 + +The 512x512 normalization was discarding usable scale-specific evidence. A +frozen union now takes the strongest OpenAI forensic response over 512, 640, +768, and native views, gated by the provider pixel head. The Google branch +takes the strongest forensic/provider fusion over 384, 512, 640, 768, and +native views, then requires the same registered-v3 lattice. Development on +the second and third challenges found 539/1,200 targets and 0/1,657 +photograph/TC260 controls. + +The views and cuts were written before opening a fourth challenge whose hashes +were disjoint from every model manifest and the first three challenges: + +| Challenge cell | OpenAI candidate | Gemini candidate | Any candidate | +| --- | ---: | ---: | ---: | +| OpenAI | 92/300 | 11/300 | 103/300 (34.3%) | +| Google | 0/300 | 173/300 | 173/300 (57.7%) | +| Open Images photographs | 0/1,000 | 0/1,000 | 0/1,000 | +| TC260 | 1/25 | 0/25 | 1/25 | + +For the requested union, the blind result is 276/600, 46.0% recall, with +1/1,025 non-target candidates. This is 3.2 times the single-scale v3 recall +and 46 times the rejected 1% cascades. The one false candidate is TC260, not a +camera photograph. Eleven OpenAI files crossed through the Gemini branch; +that is a provider-attribution error but still a correct hit for the declared +OpenAI-or-Google union. + +This remains a source-candidate finder, not an exact identifier or a SynthID +decoder. Fresh unused paths from the other generator families were not +available for v4, so the 0.1% observed non-target rate is not an open-world +precision claim. Robustness to crop, resize, re-encoding, and screenshot +capture is also not established. Keep the models and paths in `.local-eval`; +do not add a runtime or public CLI until a new temporal challenge with fresh +foreign-generator families establishes positive precision. + +Local artifacts: `source-finder-v4-selection.json`, +`source-finder-v4-rule.json`, `source-finder-v4-challenge.json`, and +`multiscale-forensic-development.json`. The local research entry point +implements the frozen multiscale rule and still uses no metadata or visible +sparkle. + +A post-hoc OR over every per-view zero-development-error OpenAI cut is +rejected. It raised v4 OpenAI recall to 153/300 but also accepted 6/1,000 +photographs and 3/25 TC260 controls. The apparent union of many individually +strict cuts was multiple-testing overfit, not additional independent evidence. + +### Original-export hybrids, 2026-08-24 + +Three more hash-disjoint challenges tested whether multiscale fusion could be +made useful without metadata. The v5 ExtraTrees union improved exact provider +recall to 363/600 (60.5%) and provider-union recall to 373/600 (62.2%), but it +also accepted 8/1,000 photograph and foreign-generator controls. A revised +Google confirmation removed those eight development errors. Adding an +AI-versus-camera gate in v6 did not transfer: exact recall fell to 327/600 +(54.5%) and union recall was 341/600 (56.8%). + +The specificity failures exposed a stronger but narrower signal. Current +OpenAI exports in these sets are PNGs produced with adaptive scanline filters, +while the earlier TC260 error was a PNG encoded with filter zero on every row. +A strict PNG parser now requires a non-interlaced PNG with at least one adaptive +filter before the OpenAI branch can emit a result. This reads the image container +and pixels, not EXIF, C2PA, a filename, or a visible label. It also changes the +claim: a re-encoded OpenAI JPEG must abstain. + +The complete frozen v7 rule reached 215/400 exact provider matches (53.8%) and +221/400 provider-union matches (55.3%) on a new challenge. Its cells were +114/200 exact OpenAI and 101/200 exact Google. The PNG gate repaired the +observed specificity problem, but the old OpenAI forensic head remained the +recall bottleneck. + +A subsequent v8 development hybrid trains an ExtraTrees OpenAI head on v4-v5 +multiscale forensic scores, pixel probabilities, and PNG encoding structure. +Model selection used v6. The final 0.47 precision cut was chosen after v7 had +been opened, so the following is a post-hoc development measurement, not +another blind result: + +| v7 cell under v8 development rule | OpenAI | Gemini | Unknown | +| --- | ---: | ---: | ---: | +| OpenAI | 190/200 | 3/200 | 7/200 | +| Google | 0/200 | 102/200 | 98/200 | + +That is 292/400 exact provider matches (73.0%) and 295/400 provider-union +matches (73.8%). + +The remaining Google miss set contained two different export pipelines: PNG +and JPEG. A second development branch parses only JPEG codestream parameters, +including quantization tables, chroma sampling, and progressive encoding; it +explicitly skips APP0-APP15 and COM segments. Training one Google model per +encoding class on v4 and selecting zero-validation-error cuts on v5-v6 raised +the v11 transfer result to: + +| v7 cell under v11 development rule | OpenAI | Gemini | Unknown | +| --- | ---: | ---: | ---: | +| OpenAI | 190/200 | 4/200 | 6/200 | +| Google | 0/200 | 126/200 | 74/200 | + +This is 316/400 exact provider matches (79.0%) and 320/400 provider-union +matches (80.0%). + +This is the best local source finder in the campaign, but it is still not a +SynthID detector, payload decoder, or open-world precision proof. The v8 rule +is post-hoc, and neither v8 nor v11 has an independent open-world negative +proof. The OpenAI branch is intentionally scoped to original-style PNG exports. +A new temporal blind challenge with new foreign generators and PNG +camera/editor controls is required before a runtime or public CLI is justified. + +Local artifacts: `source-finder-v7-selection.json`, +`source-finder-v7-challenge.json`, `source-finder-v8-rule.json`, +`source-finder-v8-openai-extra-trees.joblib`, and +`source-finder-v11-google-per-codec.joblib`. + +### Published few-shot attribution also fails the open-world gate + +[OmniDFA](https://arxiv.org/abs/2509.25682) is a purpose-built few-shot source +attributor rather than a generic content embedding. Its published `part1` +checkpoint is the correct unseen-generator fold for DALL-E 2 and DALL-E 3: +those generators are in `part1` validation and absent from its training list. +The same checkpoint has seen Imagen, so its Google result is not a clean +unseen-Imagen benchmark; the OpenAI result is sufficient to reject the shared +runtime. + +With 20 support images per provider and provider-specific similarity plus +margin cuts calibrated to zero false attributions over 160 negatives, a +content-hash-disjoint 745-image evaluation produced: + +| Cell | Result | +| --- | ---: | +| OpenAI recall | 9/50 (18%) | +| Google recall | 4/50 (8%) | +| Microsoft called OpenAI | 3/15 (20%) | +| Kodak called OpenAI | 3/24 (12.5%) | +| Canva called Google | 1/15 (6.7%) | +| fal.ai called OpenAI | 1/15 (6.7%) | +| xAI called OpenAI | 1/15 (6.7%) | +| unseen Higgsfield called OpenAI | 1/11 (9.1%) | +| fresh Open Images / COCO false attributions | 0/100 / 0/50 | + +Provider multimodality is not the missing fix. Choosing 1-10 spherical +prototypes only by calibration recall selected five: test recall fell to 16% +OpenAI and 4% Google, while false attributions remained on Firefly (2/15), +Kodak (2/24), Microsoft, ByteDance, TC260, and Made-with-AI samples. + +Native files already fail, so JPEG, resize, crop, and screenshot variants were +not run for OmniDFA. Do not add a provider-attribution runtime or CLI from that +model. General exact OpenAI/Gemini identification remains unsupported. The +strict source finder above emits candidates; it does not read the SynthID +payload. + +### 124-d lattice as pipeline ID, not a vendor CLIP head + +Provider-class ridge on 124 native residual features (70/30 once, not a +watermark gate). OpenAI L1 n=285, Google corpus n=533, foreign n=218, COCO +n=289: OpenAI vs COCO 0.965; Google vs COCO 0.999; OpenAI vs Google 0.989; +OpenAI vs foreign 0.725; Google vs foreign 0.922. OpenAI vs Firefly-class +is the weak cell. + +One-vs-rest: Google head at TPR 90% has FPR 0% vs COCO, 14% vs foreign, 2% +vs OpenAI. A Gemini-like pixel class is close to what `pipeline_lattice` +already is. An OpenAI-like pixel class on this bank would label Firefly as +OpenAI about half the time and is not shippable. + +Three-class `openai` / `google` / `no_ai` on 2,000 catalog OpenAI, 2,000 +catalog Google, and 1,936 COCO photos. Photo-first margin 0.50: openai +74.7%, google 78.9%, no_ai 99.8%; Kodak 24/24 `no_ai`. Other generators +are leakage, not classes: + +| Platform | n | openai | google | no_ai | +| --- | ---: | ---: | ---: | ---: | +| Firefly | 106 | 37 | 27 | 42 | +| Microsoft | 117 | 39 | 22 | 56 | +| PixelBin | 90 | 14 | 46 | 30 | +| HuggingFace job | 82 | 3 | 62 | 17 | +| ByteDance C2PA | 86 | 5 | 35 | 46 | +| SD / Comfy | 120 | 30 | 11 | 79 | +| fal.ai | 98 | 11 | 18 | 69 | +| Made-with-AI tag | 115 | 4 | 32 | 79 | +| TC260 | 118 | 1 | 13 | 104 | +| xAI | 114 | 1 | 20 | 93 | +| Canva | 76 | 11 | 3 | 62 | +| Apple Clean Up | 114 | 1 | 23 | 90 | +| Aweme | 37 | 2 | 7 | 28 | +| FLUX | 11 | 0 | 0 | 11 | +| Reve | 10 | 0 | 0 | 10 | +| NovelAI | 9 | 0 | 0 | 9 | +| Higgsfield | 11 | 1 | 5 | 5 | + +PixelBin and HuggingFace jobs lean `google` (shared renderer lineage). +FLUX, NovelAI, and Reve stay `no_ai`. Local probe: +`uv run python .local-eval/synthid/prc-oklab-attack-2026-08-15/classify_openai_gemini.py image.png`. + +## Research lattice expert (google-lineage renderer) + +Not a watermark and not in `identify`. `scripts/synthid_runtime/` +`detect_synthid` re-check on 628 frozen holdouts, seed 20260822, threshold +1.0. + +| Family | n | detected | rate | max score | +| --- | ---: | ---: | ---: | ---: | +| Google / Gemini | 80 | 45 | 0.56 | 3.03 | +| Firefly | 84 | 15 | 0.18 | 3.00 | +| PixelBin | 80 | 11 | 0.14 | 2.48 | +| Microsoft | 60 | 2 | 0.03 | 2.40 | +| OpenAI | 80 | 1 | 0.01 | 1.38 | +| xAI | 40 | 1 | 0.03 | 1.26 | +| FLUX HF | 40 | 0 | 0 | 0.85 | +| TC260 | 40 | 0 | 0 | 0.97 | +| Kodak | 24 | 0 | 0 | 0.49 | +| Open Images fresh | 60 | 0 | 0 | 0.64 | +| COCO hold | 40 | 0 | 0 | 0.74 | + +Firefly 18% and PixelBin 14% match the 2026-08-16 signed-foreign rates +(24% and 14%) in order of magnitude. Both Microsoft hits have issuer +`Microsoft, Google LLC`. A 2 px crop killed every sampled positive, +including Firefly and PixelBin. Google TPR 56% is mixed collection eras, not +the oracle-positive 147/148 cell. Honest name: +`google_lineage_renderer` = Gemini/Imagen + Firefly + PixelBin. + +Registered-v3 photographic controls remain 0/5,993 Open Images and 0/2,366 +COCO. Against 223 C2PA-named non-Google generators on 2026-08-16: 29 +accepted (0.130), Firefly 0.241. `.local-eval/synthid/lattice-check-2026-08-22/`. + +## Local collection note (2026-08-21) + +Unlabeled rows in the local C2PA-labeled collection are not photographs. +Microsoft and Firefly rows also carry `synthid_from_provenance=true`, so +that flag is not an OpenAI-plus-Gemini class. diff --git a/docs/synthid-detector-research.md b/docs/synthid-detector-research.md new file mode 100644 index 0000000..f43f298 --- /dev/null +++ b/docs/synthid-detector-research.md @@ -0,0 +1,591 @@ +# SynthID local detector research + +> Research archive for the hunt for a local, keyless SynthID pixel detector. +> Not a statement of current product capability. Shipped behavior: +> [supported signals](supported-signals.md) and +> [known limitations](known-limitations.md). +> +> Sister pages: [SynthID source classifiers](synthid-classifiers.md), +> [SynthID mark removal](synthid-removal-research.md), +> [mechanism reference](synthid.md). Dated measurements live in the +> [chronological plan](synthid-detector-removal-plan.md). + +## Result + +The hunt for a local, keyless OpenAI SynthID pixel detector is open. +Google has no public payload decoder, and nothing in this project +reproduces one. Closed routes below failed to read the official oracle +contrast on photographs. The remaining work is a new waveform, a new +pair mint, or a new statistic, not a rerun of those routes. + +The mark behaves like a keyed spread-spectrum residual: a content-adaptive +`x' = x + g(x)` with a paired decoder (DeepMind patent family, optional +secret in intermediate layers, encoder/decoder ensembles that do not +recognize each other). Without that pair, the residual sits under the +scene. Keyless energy, TrustMark transfer, L1 distillation, a flat-field +stamp, and a 16-32 band student all failed to read the official oracle +contrast on photographs. + +S4, 2026-08-15: the two providers are not doing the same thing. Cross-image +correlation of the folded residual is high for Google (tile16 pos-pos +`+0.326`, chance `0.036`) and at chance for OpenAI (`+0.032`). Google +shares one fixed phase-coherent pattern, also present in its controls at +about half the amplitude. OpenAI shares nothing, which is what a +content-dependent post-hoc encoder produces. Comb experts that work on +Google therefore cannot be reused as an OpenAI watermark detector. + +M2, 2026-08-15, re-verified 2026-08-16: Google unwatermarked pairs cannot +be minted. `imagen-*` ids 404 from the model garden. `gemini-*-image` +rejects `addWatermark` (`Cannot find field`). Current Gemini API docs +state that all generated images include a SynthID watermark. There is no +encoder-off Google path. + +What the product uses for the *watermark* is signed provenance and +`verify-openai-synthid`. The periodic-lattice expert is research-only under +`scripts/synthid_runtime/` and is not called from `identify` or the CLI. +Lineage measurements of that expert are in +[SynthID source classifiers](synthid-classifiers.md). + +## Public GitHub sweep, 2026-08-24 + +GitHub's primary-repository search returned 78 repositories with `synthid` in +the name and 81 with it in the description, 133 unique repositories in total. +The fork-inclusive searches were much larger and capped or dominated by copies +(860 name hits and 1,311 description hits), so this is not a claim that every +fork or unindexed private repository was read. The sweep also used targeted +code searches for detector, verifier, correlation, codebook, bypass, and +removal implementations. SynthID-Text, visible-corner-mark removers, metadata +cleaners, wrappers, and literal ports were classified before inspecting the +remaining image-pixel candidates at pinned commits. + +The audits are not null results. They separate three claims that must not be +collapsed: a repeatable image statistic exists, that statistic is a calibrated +single-image SynthID detector, and changing its score transfers to the +provider's production decoder. The repositories support parts of the first +claim. None supports the second or third on the current challenge data. + +| Repository | Retained contribution | Rejected use | Decisive result | +| --- | --- | --- | --- | +| [`Rinne414/SynthID-detector`](https://github.com/Rinne414/SynthID-detector/tree/b08abff06b67c37db9dfa7c6ed63760a446d560f) | A reproducible fixed-residual correlator and a useful spoofability negative control | Current-image detector or fixed-template remover | All three templates detected 0/400 current OpenAI and Google images. Adding the expected template spoofed the public detector, while subtracting 0.25-2x from an official positive left SynthID `detected` down to 25.72 dB | +| [`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09c62dc26112f9a0e6cfab469f9a10f37e3) | Evidence that a CNN can learn a renderer/regeneration-domain signature, plus a white-box shortcut baseline | Payload decoder, independent hybrid confirmation, or removal oracle | The published cut retained 172/200 OpenAI but accepted 110/200 Google. A gradient drove the surrogate effectively to zero, but official SynthID stayed `detected` through +/-8/255 at 34.30 dB | +| [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID/tree/b11083676fd3ee3ff97ce9d03c0e409e46905902) | Cross-color phase consensus and further evidence for a repeating Google renderer-lineage pattern | The published V3/V4 thresholds as presence detectors, or Round-06 as a verified quiet remover | V4 accepted 77/200 Google and 76/200 OpenAI at the same cut. Round-06 has no published per-image oracle evidence and uses regeneration/distortion with 14/11 dB PSNR floors | +| [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847) | A compact 147-d second-order SPAM descriptor and a safely auditable sklearn model | Google or OpenAI detector, or a high-precision hybrid rescue | The published cut accepted 139/200 Google, 141/200 OpenAI, and 214/600 disjoint ImageNet photographs | +| [`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad) | A public 2026 Gemini flat-field corpus, safe numeric features, and explicit candidate carrier bins | Its reported near-perfect Tier-1 AUC as open-world SynthID evidence | Training used 500 Gemini flats against 1,500 synthetic or phase-scrambled negatives and zero real photographs. Rebuilding the SVM from the safe matrix accepted 1/200 Google and 0/200 OpenAI on v7 | + +Useful forensic signal is therefore not synonymous with a production +watermark decoder. The Google lattice idea is retained only as a +research-lineage component; it cannot be labelled SynthID. The fixed template +and CNN gradient are retained as negative controls. None of these public +implementations is shipped as a detector or remover. Detailed classifier and +attack measurements live in [SynthID source classifiers](synthid-classifiers.md) and +[mark removal](synthid-removal-research.md). + +### Google verifier surfaces found in the sweep + +Two real but non-local verifier paths surfaced. The historical Vertex SDK +exposed `WatermarkVerificationModel.from_pretrained("imageverification@001")`. +The class still exists in `google-cloud-aiplatform` 1.165.1, but warns that it +was deprecated 2025-06-24 with removal scheduled for 2026-06-24. Calls in two +projects with Vertex enabled now reject the publisher model id as invalid. A +repository using `response.watermark_detected` is also stale: the actual SDK +property was `watermark_verification_result`, backed by the prediction's +`decision` field. + +A newer unlisted upload route, +`POST https://synthid.googleapis.com/upload/v1:verifyContent`, is live enough +to identify itself as the SynthID API and return `SERVICE_DISABLED` for a real +public image. It has no public discovery document, did not appear in the +project's available-services listing, and no authorized successful response +was obtained. Enabling a cloud service was deliberately not attempted. This +may be an access-gated research oracle, but it is not a local decoder and does +not satisfy the metadata-free offline product requirement. Gemini chat remains +the only Google verifier actually evidenced by the public bypass corpora. + +## Closed detector routes + +| Route | Close | Why | +| --- | --- | --- | +| Wavelets / FFT / cepstrum as a single-image detector | 2026-08-09 | TrustMark proxy: 318-d wavelet/spectral summary AUC 0.653 and 0 TPR at a clean calibration cut. Complex FFT maps AUC 0.516. Spatial RGB still won | +| `aloshdenny/reverse-SynthID` V3 phase codebook | 2026-08-09/24 | Pickle-free numeric audit. The original pilot was 5/5 Google positives and 0/194 then-available negatives. The broader audit was 5/355 Google, 5/499 controls, and 6/1,000 fresh Open Images. On v7 it accepted 4/200 Google and 1/200 OpenAI. As a v11 `unknown` rescue it adds 2/200 Google on this test, but the historical control errors reject a 1%-recall OR rule | +| `aloshdenny/reverse-SynthID` V4 | 2026-08-13/24 | Commit `b110836`. Better-of-two profiles: 141/355 Google positives and 191/499 controls. Frozen 1,000 Open Images: 386 accepted. Paired AUC 0.517. An exact repeat on the hash-disjoint v7 challenge accepted 77/200 Google and 76/200 OpenAI. Strict GUI aspect routing reduced support but not false acceptance. The threshold sits 0.02 above chance phase similarity | +| `cebeuq/Synthid-Bypass` as a local decoder | 2026-08-13 | 12 public before/after diffusion pairs. Fixed carrier 10/12 before, 0/12 after. Gemini Verify still saw SynthID on two `before` files the carrier missed. Reconstruction removal, not a payload read | +| S4: one content-independent pattern for both providers | 2026-08-15 | Google has a shared phase-coherent tile. OpenAI is at chance. See Result | +| M2: mint Google encoder-off pairs | 2026-08-15 | No current Google path emits an unwatermarked image. See Result | +| U1: one feature space for both providers | 2026-08-15 | Transfer 0.535 and 0.469 | +| Hand statistics vs a perceptually masked keyed mark (M1b) | 2026-08-15 | At equal RMS, fixed 0.670, keyed 0.669, keyed+masked 0.546. Masking, not keying, defeats hand features | +| Decoder-substitution / foreign VAE as a verified negative | 2026-08-17 | OpenAI oracle stayed `detected` at 22.3 dB. Attacking a positive does not mint a labelled negative | +| `aoxo/reverse-synthid` as extra Google templates | 2026-08-19 | Hugging Face dump scored against the frozen era bank. Extra family means raised foreign scores instead of isolating a mark | +| Origin-locked lattice as SynthID | 2026-08-16 | Two-pixel crop kills it; published SynthID keeps 99.97% TPR under aggressive crop. See classifiers for `google_lineage_renderer` | +| Keyless energy in 16-32 px | 2026-08-20 | L1 AUC 0.53; official decoder reads phase structure, not energy | +| TrustMark / open-encoder transfer | 2026-08-20 | E3 leave-one-encoder-out at chance on TrustMark (0.505 ridge, 0.510 conv) | +| L1 distillation of oracle bits | 2026-08-20 | Geometry-only AUC 0.78-0.83 beats pixel 124-d (0.64). ChatGPT export and `opened` C2PA predict `not_detected`. Inside 1254x1254, permutation p=0.45 | +| Flat 16-32 matched filter | 2026-08-21 | Leave-one-out residual correlation 0.645 on gray flats; 0.007-0.025 on COCO photo residuals. L1 AUC 0.70 was a size confound (size-matched 0.59 / 0.44) | +| `gpt-image-1` as encoder-off pair | 2026-08-21 | Does not stamp. `gpt-image-2` does | +| CNN on raw L1 bits | 2026-08-21 | Do not train. Labels are export geometry and presentation, not the mark | +| 128-photo causal wavelet student on the flat template | 2026-08-21 | Photo 16-32 jam flips only at a=24 / 18.6 dB (`a*_photo < 0.53` failed). A student cannot beat a matched filter if the photo-domain mark is that quiet | +| Photo 16-32 amplitude without the flat-arm template | 2026-08-23 | 11 COCO `gpt-image-2` edits plus the red-dot pair. Residual is the camera JPEG resized onto the edit raster, top-left 32 px dropped. Median 16-32 residual RMS 12.2, same order as 8-16 (9.0) and 32-64 (16.3). On the five tightest pairs (full residual RMS < 16): 16-32 median 2.69, still ~24x the 0.11 flat stamp and not louder than 8-16 (3.81). The photo residual does not isolate a 16-32 codeword | +| Sort-by-luma 1D chroma stego | 2026-08-23 | Flat LOO 0.914 but a luma-order shuffle is already 0.792. COCO plus tiny noise: 0.642 vs shuffle 0.640. The sort invents the correlation | +| Retinex / local-gain then 16-32 LOO | 2026-08-23 | 0.742 vs plain 16-32 LOO 0.752 on the same eight gray flats. No new carrier | +| Distant 16-32 patch coupling | 2026-08-23 | Near lag 0.13 of a tile: -0.013. Far lag ~500 px: +0.054. No nonlocal lock | +| Bispectrum of the 16-32 ring | 2026-08-23 | Degenerate: 1.0 on flats and 1.0 on COCO. Not a statistic | +| Length-255 m-sequence vs 16x16 fold | 2026-08-23 | Max abs corr 0.059. Chance for n=255 is `1/sqrt(255) ≈ 0.063` | +| Row-as-audio pitch peak | 2026-08-23 | Peak-to-mean 51 on flats, 46 on COCO noise residuals. Shared FFT peakiness, not a pitch | +| 16x16 origin search of the flat 16-32 template on photos | 2026-08-23 | Zero-shift NCC 0.006-0.024, best of 256 shifts 0.013-0.026. Origin is not hiding a lock | +| Affine search of the 16-32 residual vs the flat template | 2026-08-23 | 11 `gpt-image-2` photos: zero-deg median 0.009, best rotation in ±4° 0.020, lift 0.011. Six COCO noise residuals: lift 0.008. Flats already peak at 0°. No hidden rotation or 3% scale lock | +| Fourier-angle scramble of 16-32 (true `(r, θ)` of the annulus) | 2026-08-23 | s1 24.1 dB official `not_detected`, cartesian replicate 24.8 dB `not_detected`. Decoder reads phase in that annulus, including angular organization. Not quieter than scramble | +| Baker-map of the 16-32 band | 2026-08-23 | s1 27.8 dB still `detected` | +| Poisson / photon-shot embedding | 2026-08-23 | Flat Anscombe LOO 0.7518 vs linear 0.7519. s1 Poisson 30.7 dB still `detected` | +| Nested LSB plus SynthID | 2026-08-23 | s1 55.9 dB still `detected`. Extra open stego does not hide or replace the mark | +| ICC sRGB rewrite | 2026-08-23 | No-op, PSNR inf. Mark is not in a profile that a pixel-preserving assign changes | +| Seam carve 8 columns of 16-32 energy | 2026-08-23 | s1 27.7 dB still `detected` | +| Extra OpenAI image ids as a new ensemble | 2026-08-23 | `gpt-image-1.5` and `gpt-image-1-mini` are encoder-off: C2PA `created, converted`, official `not_detected`, 16-32 corr vs the `gpt-image-2` gray stamp 0.003. Dated `gpt-image-2-2026-04-21` is the same family: C2PA `watermarked.unbound`, 16-32 corr 0.189. `chatgpt-image-latest` generate stays 403 until org verification | +| DIRE reconstruction error as a payload | 2026-08-23 | SDXL float32, 512 px. VAE RMS: gray 1.24, s1 9.15, photo edit 9.89, camera 11.84. DDIM DIRE RMS: gray 2.40, photo 30.9, s1 31.5, camera 33.0. Rank follows texture. Float16 DDIM NaN'd on MPS. Not a payload | +| HEIF / AV1 as a hidden carrier | 2026-08-23 | s1 HEIF q80 46.3 dB, q50 39.3 dB, AV1 CRF 32 37.2 dB, all official `detected`. Codec round-trip does not mint a new statistic | +| Print-scan simulation as the kill | 2026-08-23 | s1 24.95 dB still `detected`. Same PSNR class as 16-32 scramble, which kills. Generic 25 dB is not the mark | +| Mellin / log-polar \|F\| of the 16-32 residual | 2026-08-23 | Flat LOO 0.996-1.000, photo 0.996, COCO 0.991-0.995. The annulus envelope, not a payload. Scale chirps `r^{iα}`: flat 0.51, photo -0.27, COCO -0.58. No photo lock | +| Zernike n≤6 and PCET of the residual | 2026-08-23 | Flat LOO 0.77 / 0.75, same order as cartesian 16-32. Photo -0.007 / 0.044. COCO 0.23 / 0.35, above photos. Moments of the flat stamp, not a photograph decoder | +| Fresnel quadratic-phase of 16-32 (z=8,16,32,64) | 2026-08-23 | Flat LOO 0.750-0.753, photo 0.012, COCO -0.006. Unitary copy of the cartesian residual. No new shell | +| 8-bin Fourier orientation energy | 2026-08-23 | 0.999 flat, 0.966 photo, 0.995 COCO. Saturates | +| Chroma 16-32 as the payload | 2026-08-23 | Gray-flat Y LOO 0.682, Cb 0.064, Cr 0.025. s1 Y scramble 24.6 dB `not_detected`; Cb 45.0 dB and Cr 43.9 dB stay `detected`. Mark is in luma | +| One 90° Fourier sector of 16-32 | 2026-08-23 | s1 27.5 dB still `detected`. Decoder needs the whole annulus, not one wedge | +| Radial-phase-only scramble of 16-32 | 2026-08-23 | s1 25.0 dB `not_detected`. Phase as a function of radius, over the full ring, is enough to kill. Combined with Fourier-angle scramble, either polar coordinate of the annulus phase is a kill if the whole ring is hit | +| [`Rinne414/SynthID-detector`](https://github.com/Rinne414/SynthID-detector/tree/b08abff06b67c37db9dfa7c6ed63760a446d560f) fixed residual template | 2026-08-23/24 | The repository reports 0/33 held-out AI images and identifies its original result as self-correlation. Its pinned pre-May GPT-Image2 template first detected 0/11 current source-matched edits and 0/11 COCO sources. An exact repeat on 200 current OpenAI and 200 current Google images produced zero detections for all three published templates at the 0.08 cut; maximum GPT-Image2 score on OpenAI was 0.0167. Adding the expected template can spoof the public correlator, but that template is not present in held-out content. Independent repeat of the rejected fixed-template route, not a payload decoder | +| [`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09) CNN ensemble | 2026-08-24 | Exact frozen inference retained 172/200 OpenAI but accepted 110/200 Google at the published 0.5 cut. The training negatives are regenerated positives, while its confound probe transforms only positives and has no foreign-image control. A full-frame white-box attack drove its probability from 0.99995 to 0.25956 at ±2/255 and effectively zero at ±4/255, but the official OpenAI decoder stayed `detected` through ±8/255. Renderer/confound classifier, not a mark decoder | +| [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847) SPAM model | 2026-08-24 | At the published cut, exact model inference accepted 139/200 Google, 141/200 OpenAI, 214/600 disjoint ImageNet photographs, and substantial fractions of foreign generators. The apparent high-precision tail did not transfer | +| [`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad) Tier-1 SVM | 2026-08-24 | Rebuilt without loading joblib, using the repository's 2,000x33 numeric matrix, reference phases, noise template, seed-42 split, scaler, and RBF SVM. It accepted 1/200 current Google and no current OpenAI rows. Its published AUC above 0.999 distinguishes Gemini flats from generated derivatives, not current photographs | +| Generic GitHub heuristics and literal ports | 2026-08-24 | `hackerfactor/reverse-SynthID-C`, `BIRSAx2/ripmark`, and `xiaoyao9184/reverse-from-synthid` reproduce or wrap the already-rejected reverse-SynthID codebook. `AI-SCERN` uses uncalibrated annular energy, decoder-grid, and FFT-symmetry heuristics with no weights or held-out corpus. LSB flips, ±1 dither, and 100.5% resize tools publish no matching-oracle evidence. None adds an independent signal | + +## Information budget on photographs (2026-08-21) + +On 11 `gpt-image-2` photo edits, the flat-derived 16-32 template at full +amplitude against per-image whitening has median `d' = 0.93` (range +0.68-2.46). That is a best-case single-image AUC of about 0.75 even with a +perfect, perfectly aligned template. A P5 gate (FPR 0.1%, TPR 90%) needs +`d' >= 4.37`. The deficit is 4.7x in amplitude, 13.4 dB, *unless* the +photo-domain mark is louder than the flat measurement. Titration said it +is not: fish 16-32 additive jam stays `detected` through a=16 / 21.9 dB. + +2026-08-23, without using that flat stamp as `G`: the aligned photo +residual in 16-32 is redraw, not a codeword. Median RMS 12.2 across 12 +pairs; 2.69 on the five tightest. Neighboring octaves are as loud. +`.local-eval/synthid/prc-oklab-attack-2026-08-15/photo-band-amplitude-2026-08-23.json`. + +Do not report an AUC from those 11 pairs as a detector result. With n=11 +the standard error on AUC is about 0.12. Notes: +`.local-eval/synthid/prc-oklab-attack-2026-08-15/agent-detector-claude.md`. + +## External literature (surveyed 2026-08-23) + +Primary sources, not abstracts. Each row is mapped onto a closed or open +route in this campaign. Mechanism detail stays in +[synthid.md](synthid.md). Removal papers are on +[mark removal](synthid-removal-research.md). Classifier papers are on +[SynthID source classifiers](synthid-classifiers.md). + +### Official mark, not a public decoder + +| Source | What it is | Map to this campaign | +| --- | --- | --- | +| Gowal et al., [arXiv:2510.09263](https://arxiv.org/abs/2510.09263) | Post-hoc encoder `f` / decoder `g`. Detection logit is not payload recovery. SynthID-O (partner variant) 136 bits at 512x512. TPR at 0.1% FPR 99.98% aggregated, 99.97% on the hardest spatial crop+resize. Trains against sampled semantics-preserving transforms, including weak VAE regeneration. Production decoder unpublished | Matches the architecture we treat as keyed `x' = x + g(x)`. Explains why a two-pixel crop kills `pipeline_lattice` but not the official oracle, and why a 22.3 dB foreign VAE still reads `detected` | +| DeepMind [US12094474B1](https://patents.google.com/patent/US12094474B1/en) and continuation [US20250149048A1](https://patents.google.com/patent/US20250149048A1/en) | Residual U-Net encoder, separate decoder, optional key, encoder/decoder ensembles that need not recognize each other | Constraint, not a recipe. Ensemble non-recognition is why one recovered Google tile cannot be reused as an OpenAI detector (S4) | +| Dathathri et al., [Nature 634:818-823 (2024)](https://www.nature.com/articles/s41586-024-08025-4) | SynthID-Text: tournament sampling of LLM tokens, open-source | Different system. Image/audio/video remain proprietary | +| OpenAI, [advancing content provenance](https://openai.com/index/advancing-content-provenance/) (2026-05-19, audio 2026-07-31) and [content provenance API](https://developers.openai.com/api/docs/guides/content-provenance) | ChatGPT / API / Codex images carry C2PA plus SynthID. Audio from 2026-07-31. `POST /v1/content_provenance_checks`. `not_detected` does not rule out another vendor | This is the oracle. C2PA and SynthID are independent entries. Do not abuse the endpoint as an adaptive reverse-engineering loop | + +### Keyless presence detectors in the literature + +| Source | Claim | Caveat against our gates | +| --- | --- | --- | +| Ao et al., [arXiv:2603.06723](https://arxiv.org/abs/2603.06723) (AWPD / FSNet, SAFE@CVPR 2026) | Leave-one-algorithm-out presence detection. SynthID held out: FSNet Acc 0.894 / F1 0.886, ResNet-50 Acc 0.845 / F1 0.812, ConvNeXt V2 Acc 0.866. LSB and Patchwork both fail below 60%. Hypothesis: modern invisible marks share dense high-frequency spectral anomalies | UniFreq Table 3: SynthID is 2,000 images, all AIGC, zero photographs, from `imagen-4.0-fast-generate-001` only, resized to 256x256. No OpenAI. No 0.1% FPR. Closest published analog to E3/L1 distillation. Our TrustMark leave-one-encoder-out was chance (0.505 ridge). Their high-frequency commonality is the same energy that L1 failed to read as an OpenAI payload | +| `aloshdenny/reverse-SynthID` V3/V4; Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | Spectral codebook from averaged Gemini flats. Spokesperson Myriam Khan: "It is incorrect to say this tool can systematically remove SynthID watermarks." The author also said the bypass confuses the decoder rather than deleting a payload | Our pickle-free V4 audit: paired AUC 0.517, frozen Open Images 386/1000 accepted. Not 0.1% FPR | +| Krawetz, [Hacker Factor "Reversing SynthID"](https://www.hackerfactor.com/blog/index.php?/archives/1092-Reversing-SynthID.html) and ["Meta's Un-Stable Signature"](https://hackerfactor.com/blog/index.php?/archives/1098-Metas-Un-Stable-Signature.html) | Gemini chat TPR closer to 1/20 than the paper's 99.97%. Detector weak on flats. reverse-SynthID field accuracy ~70% vs the author's 90% | Gemini-app chat is not the OpenAI provenance API. Our gray `gpt-image-2` flats are `detected` on the official OpenAI oracle. Do not mix the two verifiers | +| vitotitto LAB-a logistic (community, tiny holdout) | Reported 97.7% AUC on 20/20 | Not a P5 gate. Ignore as a detector claim | + +### Different embedding loci (not post-hoc SynthID) + +These papers are often cited as if they were SynthID. They are not. + +| Source | Locus | Why it is not this mark | +| --- | --- | --- | +| Gunn, Zhao, Song, [arXiv:2410.07369](https://arxiv.org/abs/2410.07369) (PRC, ICLR 2025) | Pseudorandom error-correcting code in the *initial diffusion latent* | Cryptographic undetectability is a latent-code property. SynthID-Image is applied after the VAE has already decoded pixels. Our OKLab "PRC-style" noise was a category error | +| Francati et al., [arXiv:2509.10577](https://arxiv.org/abs/2509.10577) (EuroS&P 2026) | Crop-and-resize flipped about half of PRC latent signs and blocked belief-propagation decode | Confirms PRC is origin-locked in latent space. Matches why a 2 px shift kills `pipeline_lattice` and does not kill OpenAI SynthID | +| Fernandez et al. Stable Signature; Wen et al. Tree-Ring | Fine-tuned VAE decoder, or ring constraints on initial noise | In-generation. Google's paper is explicit that SynthID-Image does not modify the generator | +| TrustMark ([arXiv:2311.18297](https://arxiv.org/abs/2311.18297)), HiDDeN, StegaStamp | Open post-hoc encoder/decoder pairs | Transfer from these is E3: chance on TrustMark. WAVES ([arXiv:2401.08573](https://arxiv.org/abs/2401.08573)) stress-tests this family, not production SynthID | + +Do not train a student on AWPD's Imagen-only 256 crop and call it an +OpenAI SynthID detector. That would repeat L1's contamination: a +generator/size cue, not a payload. + +### Adjacent literature, not SynthID + +These papers are the published versions of the wild hypotheses and of +Model 1. They detect *something else*: a camera pipeline, a generator +fingerprint, an open watermark, or steganography. None of them is a +keyless OpenAI payload decoder. + +**Classical spread-spectrum and open neural marks** + +| Source | What it is | Map | +| --- | --- | --- | +| Cox, Kilian, Leighton, Shamoon, IEEE TIP 1997 | i.i.d. Gaussian vector in perceptually significant spectral components. Informed detector. Collusion-resistant by construction | Ancestor of keyed CDMA. Our length-255 m-sequence vs 16x16 fold was at chance because we did not have the key, and because OpenAI is content-adaptive, not a fixed Gold code | +| Zhu et al., HiDDeN, ECCV 2018 | Joint encoder / noise layer / decoder | Open pair. AWPD leave-one-out Acc 0.985. Transfer to SynthID is E3 | +| Tancik, Mildenhall, Ng, StegaStamp, 2020 | 100 bits at 400x400, trained through print-scan | Open pair. Survives recapture. WAVES: TPR at 1% FPR collapses from 1.00 to 0.01 under regeneration | +| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring, NeurIPS 2023) | Pattern in the *initial diffusion noise*, recovered by DDIM inversion | In-generation, origin-locked in latent Fourier space. Crop-invariant by design. Not post-hoc SynthID | +| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tune the LDM VAE decoder so every decode carries a bit string | In-generation. Gowal is explicit that SynthID-Image does not modify the generator | +| Lin and Juarez, [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) | Public-knowledge attack that removes Tree-Ring | Confirms Tree-Ring is a different object with a different kill | + +**Steganalysis as a presence detector** + +Fridrich and Kodovsky Spatial Rich Models (TIFS 2012) and Boroumand, +Chen, Fridrich SRNet (TIFS 2018) detect sub-bit-per-pixel spatial +stego by high-pass residuals, with pooling disabled in the front of +SRNet so the weak signal is not averaged away. AWPD cites both and +says they drift on modern deep / generative marks. That matches our +wavelet/FFT single-image detector (AUC 0.653, 0 TPR at a clean cut) +and the 16-32 energy miss on photographs: a residual energy detector +without the matching key is steganalysis of a mark that was trained +not to look like LSB. + +**Generator fingerprints in the Fourier domain** + +Corvi, Cozzolino, Poggi, Nagano, Verdoliva, +[arXiv:2304.06408](https://arxiv.org/abs/2304.06408) (CVPRW 2023): +GAN, diffusion, and VQ-GAN images show spectral peaks and anomalous +autocorrelation; real vs synthetic differ in mid-high radial and +angular power. reverse-SynthID averaged Gemini flats and called the +peak a watermark codebook. Corvi's result says many generators leave +*some* peak. Our V4 Open Images 386/1000 is what a generator-fingerprint +detector looks like when you calibrate it as if it were a payload. + +Yao and Juarez, [arXiv:2512.11771](https://arxiv.org/abs/2512.11771) +("Smudged Fingerprints"): 14 fingerprinting methods across RGB, +frequency, and learned features; removal attacks >80% white-box, >50% +black-box. A fingerprint you can see without a key is a fingerprint +you can wipe without a key. + +### Image investigation and data hiding (any method) + +These are not SynthID papers. They are the rest of the toolkit: how +people hide bits in pictures, and how people tell a picture was +touched. Several of our wild hypotheses already had a published form +here. + +Hiding is not one problem. Cover modification (change an existing +image), coverless / generative (sample an image that already carries +the bits), and signed metadata (C2PA) fail under different attacks. + +**Cover modification, classical** + +| Source | Hide how | Detect / limit | +| --- | --- | --- | +| LSB, Patchwork (Bender et al., IBM SJ 1996) | Flip low bits, or luminance of random pixel pairs | AWPD Acc < 0.60. Sparse or ±1 amplitude. SRNet / FSNet average it away | +| Westfeld F5 (2001), Fridrich nsF5 | JPEG DCT coefficients, matrix embedding | Histogram attacks on F5; nsF5 was the shrinkage fix. Domain is the codec, not a mid-band residual | +| Pevny, Filler, Bas HUGO (2010); Holub and Fridrich WOW (2012); Holub, Fridrich, Denemark UNIWARD (IH&MMSec 2013 / EURASIP 2014) | Content-adaptive costs, bits placed in texture via syndrome-trellis codes (Filler, IH 2011) | The modern spatial/JPEG floor. Distortion is *designed* to look like the cover. A 16-32 energy detector is the wrong statistic | +| Cox et al. 1997 | Spread-spectrum in significant DCT | Keyed. Already mapped above | + +**Cover modification, neural (hide a whole image, not 32 bits)** + +| Source | What it does | Map | +| --- | --- | --- | +| Baluja, NeurIPS 2017, "Hiding Images in Plain Sight" | Full-resolution secret image distributed across all bits of a same-size cover | Capacity is the point. Not robust, not keyed provenance. HiDDeN later added a noise layer so the secret survives JPEG | +| Jing et al., HiNet, ICCV 2021 | Invertible neural net: hide and recover as one bijection | High-capacity lossless-ish hiding. Recovery needs the exact inverse, not an official oracle | +| Yang et al., PRIS, [arXiv:2309.13620](https://arxiv.org/abs/2309.13620) | Invertible net plus robustness modules | Same family, trained through distortion | + +A HiNet-style student on OpenAI pairs would learn the *edit*, not a +SynthID codeword. That is the L1 contamination again. + +**Coverless / generative hiding** + +The cover is never modified because there is no cover. The sampler +emits an image whose latents already encode the message. + +| Source | Locus | Map | +| --- | --- | --- | +| Yang et al., Gaussian Shading, [arXiv:2404.04956](https://arxiv.org/abs/2404.04956) (CVPR 2024) | Map bits onto Gaussian latents indistinguishable from ordinary noise. Recover by DDIM inversion | Sibling of Tree-Ring / PRC. Training-free, performance-lossless *for the generator*. Not a post-hoc pixel stamp | +| Peng et al., StegaDDPM (ACM MM 2023) and later LDStega | Bits in the diffusion sampling distribution | Spatial SRNet is the wrong detector (NS-DSer, [arXiv:2602.10219](https://arxiv.org/abs/2602.10219): move steganalysis into noise space) | +| CRoSS, Pulsar, MDDM | Message-to-noise projections | Same locus. A pixel scramble does not invert the sampler | + +**Passive forensics (the picture was touched, no secret assumed)** + +| Source | Cue | Map | +| --- | --- | --- | +| Krawetz, "A Picture's Worth", 2007 (ELA) | Re-JPEG at lower quality, subtract | Already measured: COCO 3.13, s1 1.97, gray stamp 0.49. Codec history, not a payload. Farid publicly called ELA as likely to mislabel originals as it is to catch edits | +| Farid, IEEE TIFS 2009, JPEG ghosts | Difference energy vs a sweep of JPEG qualities; spliced regions ghost at their original Q | Local: s1, a `gpt-image-2` photo, and a camera JPEG all minimize at Q90. Codec, not a payload | +| Popescu and Farid, TR2004-515 | Copy-move via duplicated regions | Not generation, not a watermark | +| Popescu and Farid, IEEE TSP 2005 | Resampling periodic correlations | Affine search cousin. A rotated SynthID residual is a different question | +| Wang et al., DIRE, [arXiv:2303.09295](https://arxiv.org/abs/2303.09295) (ICCV 2023) | Diffusion reconstruction error: generated images reconstruct, cameras do not | Model 1 sibling. SDXL float32 at 512: DIRE RMS camera 33.0, s1 31.5, photo edit 30.9, gray stamp 2.40. Texture rank, not a payload. Inverse of the foreign-VAE remover: there the mark survived 22.3 dB | +| Wang, Wang, Zhang, Owens, Efros, [arXiv:1912.11035](https://arxiv.org/abs/1912.11035) (CVPR 2020, CNNDetect) | One ProGAN classifier, heavy JPEG/crop aug, transfers to many CNNs | Ancestor of "train on one generator". Ojha showed the sink-class failure once diffusion arrived. We required Firefly for that reason | + +C2PA is the non-pixel stack: a signed manifest, stripped by +`metadata --remove`. Durable Content Credentials (spec 2.4) add a +soft binding that can re-link a stripped file to a repository. That +is provenance, not hiding. + +Do not train on ELA, JPEG ghosts, DIRE, or a HiNet reconstruction and +name the score SynthID. + +### Waveforms that can live in a picture + +A mark is a function on the pixel lattice. The literature does not +use one wave. It picks a basis whose symmetries match the attack it +fears, then hides a keyed coefficient vector in that basis. Cartesian +16-32 is one shell. Polar, scale, and diffraction are different +shells. + +**Standing waves on a rectangle.** DFT / DCT / DST. A 2-D sinusoid +`cos(2π(ux + vy)/N)`. JPEG lives here. Our octave scramble destroys +one annular *radius* of these frequencies, not one orientation. A +Gabor packet is the same sinusoid windowed in space. + +**Circular and log-radial waves.** Functions of `(r, θ)`, not +`(x, y)`. + +| Basis | Wave | Invariance it buys | +| --- | --- | --- | +| Fourier-Mellin / log-polar (Ó Ruanaidh and Pun, Signal Processing 1998) | `r^{iα} exp(ikθ)` after a DFT magnitude | Rotation and scale become translations | +| Logarithmic radial harmonics (IH 2002) | Same family, added in pixels, detected by complex correlation | RST without going through the unstable log-polar resample | +| Polar harmonic transforms: PCET, PCT, PST | `exp(±i2π n r²)` and polar cos/sin | Rotation. Moments, not a dense codebook | +| Zernike / pseudo-Zernike | Orthogonal polynomials on the disk, radial part related to Bessel | Rotation. Classical moment watermark | +| Circular chirp (SPIE 6072, 2006) | Polar map of a 1-D chirp onto a ring | JPEG (tune chirp rate) plus rotation (the ring) | +| Tree-Ring (Wen 2023) | Concentric rings in the *latent* Fourier plane | Crop/flip by construction. Not a pixel wave | + +The 2026-08-23 file named polar-1632 is cartesian annular phase +shuffle, a scramble replicate (`not_detected` at 25.6 dB). True +Fourier-angle scramble of the same annulus is `not_detected` at 24.1 dB. +Radial-phase-only scramble is `not_detected` at 25.0 dB. One 90° +sector plus its conjugate stays `detected` at 27.5 dB. Affine/rotation +search of the 16-32 residual against the flat template does not lock +(lift 0.011, COCO 0.008). + +**Scale chirps.** Hyperbolic / Mellin monomials (arXiv:1208.5842): +real 1-D Mellin patterns tiled in 2-D. Run 2026-08-23: +`r^{iα}` coefficients flat LOO 0.51, photo -0.27, COCO -0.58. +Log-polar `|F|` and a 64-bin radial Mellin profile saturate on COCO +(0.99). Not a payload, and not quieter than cartesian 16-32. + +**Directional packets.** Dual-tree complex wavelets (approximate +analytic wave), Gabor/Morlet, ridgelets, curvelets, shearlets, +contourlets, bandelets. Multiplicative spread-spectrum on curvelet +coefficients is a published detector-design paper, not a SynthID +decoder. Our wavelet summary AUC 0.653 already said a *generic* +packet energy is not the OpenAI payload. + +**Optical diffraction, actual wave physics.** + +| Transform | What the wave is | Map | +| --- | --- | --- | +| Fresnel | Quadratic phase `exp(iπ r² / λz)`, a radial chirp. Energy concentrates with distance | Holographic watermarks embed the diffracted field, not the picture | +| Fractional Fourier | Rotation of the time-frequency plane. Order is an extra key | Extra key space on top of DFT | +| Double random phase encoding (Réfrégier and Javidi, Optics Letters 1995) | Random phase at input *and* Fourier plane. Ciphertext is stationary white noise | Closest optical analog of a keyed `x' = x + g(x)` that looks like noise without both screens | +| Cascaded Fresnel / CGH | Two holograms at different depths | Hardware key. Not a pixel residual we can match-filter | + +A Fresnel/FrFT student without the distance or the fractional order +is another unmatched filter. Fresnel at z=8, 16, 32, 64 on the 16-32 +residual is a unitary copy of the cartesian band (flat LOO 0.75, photo +0.012). FrFT is the same unmatched-filter claim with an extra order +key; it does not isolate a photograph lock the four distances missed. + +**Cyclostationary and clock-warped waves.** Periodic clock change +(Martin, Chabert, Lacaze, EUSIPCO 2005) turns a stationary host into +a cyclostationary one by a periodic time warp. Our H2 multiplicative +cyclostationarity test already put official positives *below* +controls. Do not rerun the same statistic in a new basis and call it +new evidence. + +**How to read a candidate wave.** Name the coordinates `(x,y)` vs +`(r,θ)` vs `log r`, the generating function (sinusoid, chirp, +Bessel, Zernike), additive vs multiplicative, and the symmetry it +claims. Then pick one attack that kills that symmetry and one that +should not. Cartesian 16-32 scramble and a 2 px shift are done +(scramble kills, shift does not). True Fourier-angle scramble kills s1 +at 24.1 dB. Radial-phase scramble kills at 25.0 dB. A 90° sector does +not. Y scramble kills; Cb/Cr do not. Affine, Mellin, Zernike, PCET, +and Fresnel of the 16-32 residual do not lock on photographs. + +## Still open or incomplete + +- Google: no public payload decoder and no usable programmatic oracle. + The Gemini app remains quota-limited. Vertex was 403. +- `chatgpt-image-latest` is an optional third pair mint, not a blocker. + `gpt-image-1`, `gpt-image-1.5`, and `gpt-image-1-mini` are encoder-off. + `gpt-image-2` and dated `gpt-image-2-2026-04-21` are encoder-on, same + 16-32 stamp (corr 0.189 on gray). Latest would only matter if ChatGPT's + current consumer id used a different ensemble. `images.generate` is 403 + until the organization is verified. +- Physical print-scan: Brother DCP-L2520DW is idle and accepting. No + `scanimage`, no ImageCapture pyobjc. Recapture needs a human at the + flatbed. Simulation at 24.95 dB stayed `detected`. Do not print + unattended. +- Adding unmarked context (frame, pad, hstack, collage) to hunt a + residual is a presentation gate, not a mark statistic. Those rungs + belong with identity and layout below, and with + [mark removal](synthid-removal-research.md). +- Camera-to-ChatGPT "unchanged crop" pairs are dirty counterfactuals. + The instructed edit redraws; aligned PSNR on the frame pair was + 16.6 dB. Compare pixels only after registration and a frozen residual + bound, as in the `gpt-image-2` red-dot row. + +A Gemini app caption that an image "looks like a photograph" is not a +SynthID verdict. Use the provider pixel check or signed provenance. + +## Wild hypotheses + +Battery 2026-08-23, no official oracle. Local numbers: +`.local-eval/synthid/prc-oklab-attack-2026-08-15/wild-hypotheses-2026-08-23.json`. +Prepared attack rasters wait in `wild-attacks-2026-08-23/` for a later +verifier window. Jacobian / adaptive queries against +`verify-openai-synthid` stay out: the endpoint forbids reverse-engineering. + +### Tested locally, not a mark + +| Hypothesis | Result | +| --- | --- | +| Chroma ordered by luma is a 1D payload | Correlation is an artifact of the sort | +| Mark is `g(x)` of a Retinex envelope | LOO unchanged vs the raw 16-32 residual | +| Nonlocal patch agreement in 16-32 | Far patches are not coupled | +| Quadratic phase coupling (bispectrum) | Statistic saturates on camera images too | +| CDMA Gold / m-sequence in the 16x16 fold | At chance for length 255 | +| Rows as a pitched waveform | COCO noise has the same peak-to-mean | +| The photo mark is the flat stamp at an unknown 16-phase | Exhaustive shift does not lift NCC | +| The photo mark is the flat stamp at a small rotation or 3% scale | Affine search lift 0.011, COCO noise 0.008 | +| Self-keyed predictor from a 16 px luma thumb | Already 0.48 vs L1, 0.50 vs COCO (`self-keyed.json`) | +| Global multiplicative cyclostationarity (H2) | Official positives below controls, 2026-08-14 | +| Mellin radial profile / log-polar \|F\| | Saturates on COCO (0.99). Annulus envelope | +| Scale chirps `r^{iα}` | Flat 0.51, photo -0.27, COCO -0.58 | +| Zernike n≤6 | Flat 0.77, photo -0.007, COCO 0.23 | +| PCET polar harmonics | Flat 0.75, photo 0.044, COCO 0.35 | +| Fresnel z=8..64 of 16-32 | Copy of cartesian LOO | +| 8-bin orientation energy | Saturates on COCO | +| Chroma 16-32 on gray flats | Cb 0.064, Cr 0.025 vs Y 0.682. Stamp is luma | + +### Not a watermark, might still be an AI-vs-camera extra + +Battery 2026-08-23, `remaining-hypotheses-2026-08-23.json`. Kodak corpus was empty here, camera arm is COCO plus the photo-pair sources. + +| Hypothesis | Result | +| --- | --- | +| Absence of camera PRNU | Gray highpass RMS 0.25 vs COCO 14.6 was texture. Wiener residual on *photographs*: `gpt-image-2` edits 4.61 vs camera 8.05. Direction is camera-like leftover of the donor JPEG, not a missing sensor, and not a payload | +| JPEG ELA | Flat 0.49, s1 1.97, COCO 3.13. Compression history | +| CFA interpolation error (Bayer as detector) | Ratio photo-edit 0.117 vs camera 0.184 vs gray stamp 0.588. Weak camera vote, overlaps. Inverse of the Bayer remover, not SynthID | +| JPEG ghosts (Farid) | s1, a `gpt-image-2` photo, and a camera JPEG all minimize at Q90 with similar spreads. Codec, not a payload | +| PNG Paeth / filter-byte residual | `gpt-image-2` PNG is 99.9% Paeth. Camera re-saved PNG 73% Paeth, s1 74%. Export encoder fingerprint | +| Perfect-circle / text-edge rate | Hough circles per MP 385 (edits) vs 536 (camera). Edge fraction 0.052 vs 0.072. Too noisy for a payload or a provider | +| Face-gated 16-32 | s1: 3 Haar faces, inside RMS 12.6 vs outside 10.9. One photo with faces: outside/in 1.49. No systematic "spend the mark off faces" | +| CLIP-space watermark | That is Model 1. Not a payload decoder | + +### Non-local, 2026-08-23 + +Battery `.local-eval/synthid/prc-oklab-attack-2026-08-15/nonlocal-2026-08-23/`. +Oracle after AI-metadata strip, 25 s gap, then one dated-id retry after a 429. + +| Hypothesis | Result | +| --- | --- | +| HEIF q80 / q50 | 46.3 / 39.3 dB, official `detected` | +| AV1 CRF 32 still | 37.2 dB, `detected` | +| Print-scan simulation (0.35 deg, blur σ 0.9, 300/360 dpi, JPEG q42) | 24.95 dB, still `detected`. Same PSNR class as 16-32 scramble, which kills | +| Physical print-scan | Brother DCP-L2520DW idle and accepting. No `scanimage`, no ImageCapture pyobjc. Not printed unattended | +| `gpt-image-1.5` gray reproduce | C2PA `created, converted`. Oracle `not_detected`. 16-32 corr vs `gpt-image-2` stamp 0.003. Encoder-off. Dirty redraw 24.0 dB | +| `gpt-image-1-mini` gray reproduce | Same C2PA. Oracle `not_detected`. Corr 0.003. PSNR 31.1 dB | +| `gpt-image-2-2026-04-21` gray reproduce | C2PA `created, converted, watermarked.unbound`. Official `detected`. 16-32 corr vs unversioned `gpt-image-2` stamp 0.189. Same encoder family. PSNR 47.5 dB | +| `chatgpt-image-latest` generate | 403, org verification. Listed among image models, not callable | +| DIRE SDXL VAE + DDIM float32 | VAE RMS 512: gray 1.24, s1 9.15, photo edit 9.89, camera 11.84. DIRE RMS: gray 2.40, photo 30.9, s1 31.5, camera 33.0. Texture rank, not a payload | + +### Still blocked after that battery + +| Hypothesis | Why it stays blocked | +| --- | --- | +| Physical print-scan recapture | Human at the flatbed | +| Encoder ensemble that does not recognize itself | Needs another actual SynthID encoder/decoder pair. Extra OpenAI ids did not mint one. Open-proxy transfer is already closed: source-matched TrustMark, VideoSeal, DWT-DCT, and WAM pairs produced held-out-family AUC 0.437-0.562, so another generic public pair does not answer this hypothesis | +| `chatgpt-image-latest` as a third pair mint | `images.generate` 403 until org verification. Optional: only if ChatGPT latest used a different ensemble than `gpt-image-2` | + +Do not train on official decoder bits beyond the frozen L1 study. Do not +mint a 128-photo student on the flat stamp. The information budget and +the 2026-08-23 residual measurement both say the photo-domain mark is +under the scene. + +## Oracle and seeds + +Official `POST /v1/content_provenance_checks`, metadata stripped, pixels +preserved. Raw files stay gitignored under +`.local-eval/synthid/prc-oklab-attack-2026-08-15/`. + +| Seed | Size | +| --- | --- | +| s1 | 1024x1536 | +| s2 | 1024x1536 | +| s3 | 1086x1448 | + +All three `detected` before attacks. + +## Source-matched pairs (2026-08-21) + +`gpt-image-2` (ChatGPT Images 2.0) can mint encoder-on/off pairs. +`chatgpt-image-latest` is optional and 403 until org verification. + +| Input | C2PA | Oracle | Aligned PSNR | +| --- | --- | --- | ---: | +| Camera JPEG 960x1280 | none | not_detected | - | +| ChatGPT UI + frame 1086x1448 | watermarked.unbound | detected | 16.6 dB (redraw) | +| `gpt-image-1` edit, high fidelity, red dot | created, converted only | not_detected | 30.9 dB | +| `gpt-image-1.5` gray reproduce 1024 | created, converted only | not_detected | 24.0 dB (redraw) | +| `gpt-image-1-mini` gray reproduce 1024 | created, converted only | not_detected | 31.1 dB | +| `gpt-image-2` edit, red dot 1024x1536 | watermarked.unbound | detected | 28.2 dB | +| `gpt-image-2-2026-04-21` gray reproduce 1024 | watermarked.unbound | detected | 47.5 dB | + +The `gpt-image-2` photo residual is broadband (std 9.1), not a 16-32 stamp. +COCO `gpt-image-2` edits: 11/12 `watermarked.unbound`, one safety block. + +Solid gray/white through `gpt-image-2` "reproduce exactly" is +`watermarked.unbound`. Gray 128 at 1024x1536: PSNR 41.9 dB vs the flat, +source `not_detected`, output `detected`. Other gray levels 32-220: PSNR +38-49 dB. Eight gray 1024x1536 residuals, leave-one-out correlation after +highpass: 0.645 (shuffle 0). That shared stamp does not transfer to COCO +photo residuals (0.007-0.025). + +## Presentation gate versus residual + +An official `not_detected` is not one event. Collage, photo-pad, two-panel +self-concat, and 8x8 scramble can be a *presentation gate* (this upload is +not a single photograph). Blur sigma 7 and in-band 16-32 destruction sit on +the single-photo manifold and are residual damage; those belong in +[mark removal](synthid-removal-research.md). + +Do not train a surrogate on mixed collage misses if the goal is a mark +student. + +Identity (s1, 2026-08-19): JPEG q80/q20, crop 2 px, crop 8 px, keep 0.75 +and 0.50, rot90, rotate 5, flip, grayscale all stay `detected`. + +Token / layout (s1): each quadrant at 512x768 `detected`; `hstack` and +`vstack` of marked\|marked `not_detected`; 4x4 tile scramble `detected`; +8x8 `not_detected`. Two copies of a detecting image still miss. Tomography +3x3: all nine cells `not_detected` (a 1/9 window is below support). + +Preprocess E1 (s1): stretch 2x `detected`; centre-crop of marked\|marked +hstack back to native `detected`; 0.20x pixels on a native-size white +canvas `not_detected` (the same 0.20x file uploaded alone was `detected`). + +## L1 is not a mark task + +A missing `watermarked.unbound` assertion is not a clean negative. On +2026-08-16 the official verifier called 65 of 94 OpenAI rows without that +assertion `detected` (69%), with 9 of 9 interleaved health positives +detected so the endpoint was answering. Google's own API emits watermarked +images with no assertion, per its documentation. Corpus AUCs that treat +"no assertion" as unmarked rest on a negative class that is roughly +two-thirds positive. + +Control-only 283 rows: 203 detected, 80 not_detected. Forward-temporal +nested ridge on 124 pixel features: AUC 0.649 / 0.641. Geometry-only: +0.781 / 0.826. All 21 `claim_generator=ChatGPT` controls are +`not_detected`. C2PA `opened`: 24/24 `not_detected`. Inside 1254x1254, +mean-feature permutation p=0.45. + +OpenAI-supervised ridge on 16-32 band-passed `image_features`: +forward-temporal AUC 0.53 vs L1 `not_detected`, 0.95 vs COCO, 0.97 vs COCO +after a 2 px crop. The student learns OpenAI-versus-photo and still does +not see the oracle mark contrast. + +## Product remainder for the watermark + +Signed provenance (`identify`) and `verify-openai-synthid` (remote, explicit +upload). A research lattice miss is not a clean SynthID negative. diff --git a/docs/synthid-removal-research.md b/docs/synthid-removal-research.md new file mode 100644 index 0000000..df210fc --- /dev/null +++ b/docs/synthid-removal-research.md @@ -0,0 +1,243 @@ +# SynthID mark removal research + +> Research archive for pixel-only SynthID removal, not diffusion +> regeneration. Not a statement of current product capability. Shipped +> invisible removal is lossy regeneration: +> [known limitations](known-limitations.md). +> +> Sister pages: [SynthID local detector](synthid-detector-research.md), +> [SynthID source classifiers](synthid-classifiers.md), +> [mechanism reference](synthid.md). + +## Result + +The quality-preserving OpenAI SynthID remover hunt closed 2026-08-20. +Bayer, VNG demosaic, upscale-then-Bayer, barrel distortion, scanline +jitter, and a 2 px shift closed 2026-08-22 on s1/s2 and 2026-08-23 on s3 +and fish: they leave the official oracle `detected`. + +Working residual kills on photographs cost about 19-26 dB: + +- 16-32 px cartesian phase scramble (s1 24.6-24.8 dB, fish 23.2 dB, s2 19.0 dB) +- Fourier-angle scramble of the same annulus (s1 24.1 dB) +- Radial-phase scramble of the same annulus (s1 25.0 dB) +- Y-only 16-32 scramble (s1 24.6 dB); Cb/Cr-only do not kill +- File named polar-1632, actually cartesian (s1 25.6 dB) +- Replace 16-32 with a COCO photo's 16-32 (s1 25.2 dB) +- Gaussian blur sigma 7 (23.8 dB), holds 3/3 versus sigma 6 +- Additive 16-32 jam only at a=24 / 18.6 dB, worse than scramble + +Baker-map, 8-seam carve, Poisson, nested LSB, palette64, and ICC rewrite +do not kill at a better PSNR. + +JPEG q5, noise sigma 16, grayscale, rot90, flip, 5°, downscale 0.20x, +median 7, posterize 4, VAE round-trip, and white pad to 40% linear stay +`detected`. Elastic warp is not a stable kill (s3 still `detected` at +22.0 dB). + +The product remainder is diffusion regeneration (`qwen-zimage` / +`sdxl-zimage`), which does not decode and delete a payload. Defeating the +verifier does not restore forensic deniability; see +[synthid.md](synthid.md#23-removal-attacks-and-forensic-detectability). + +Official `not_detected` on collage, photo-pad, and two-panel layouts is a +presentation gate, not residual damage. Those rungs are in +[detector research](synthid-detector-research.md). + +## Closed quiet removers + +| Attack | Close | Notes | +| --- | --- | --- | +| Quality-preserving photo remover | 2026-08-20 | Every residual `not_detected` that is not a collage is below usable quality | +| Additive in-band jam as a quiet remover | 2026-08-21 | Fish flips only at 18.6 dB; 4-8 px at the same PSNR stays `detected` | +| Bayer bilinear / VNG / upscale-Bayer | 2026-08-23 | s1/s2/s3 and fish still `detected`. s3 bilinear 37.1 dB, fish 34.7 dB. VNG on s2 is dirtier than scramble (18.5 vs 19.0 dB) and the mark remains | +| Barrel k1=0.06, scanline ±0.8 px, and shift 2 px | 2026-08-23 | s1/s2/s3/fish `detected` even at 14-20 dB barrel. Fish scanline 32.5 dB and shift 22.7 dB still `detected` | +| TrustMark-style micro-warp | 2026-08-21 | 0.25 px / 32 dB still 100% detect on TrustMark P; OpenAI elastic ~21 dB unreplicated | +| OKLab random-codeword replacement as a quiet wipe | 2026-08-15 | On four public Google-oracle positives, replacing the period-16 tile at 0.95 dropped the *local* lattice score below 0.173 at ~54 dB. Gemini pixel verify never accepted the candidates (`Connecting to Verify AI`). Local-score intervention, not an official wipe | +| PRC-style OKLab a/b noise plus spectral peak cleaning | 2026-08-15 | Independent high-pass OKLab noise at 32.6 dB moved the median local score by only `-0.0219`. Comb suppression moved one of four below threshold at 58 dB. Not a quiet official kill | +| Rinne414 fixed-template subtraction | 2026-08-24 | The pinned pre-May GPT-Image2 residual was resized onto a current verified OpenAI image and subtracted at 0.25x, 0.5x, 1x, and 2x. Official SynthID stayed `detected` at 43.13, 37.49, 31.60, and 25.72 dB after metadata stripping. Even the dirtiest rung is no better than the existing 24-25 dB annulus-phase scramble. The template can spoof its own public correlator but does not cancel the current mark | +| `newideas99` CNN gradient | 2026-08-24 | A 3x3 crop-grid objective spread the white-box perturbation over the full marked frame. The frozen ensemble fell from 0.99995 to 0.25956 at ±2/255 (45.92 dB) and to effectively zero at ±4/255 (40.30 dB). Official OpenAI SynthID stayed `detected` at ±1, ±2, ±4, and ±8/255, through 34.30 dB. The gradient attacks the surrogate's shortcut, not the production mark | +| `reverse-SynthID` V4 Round-06 | 2026-08-24 | The advertised `final`/`nuke` path is SD-VAE regeneration plus elastic and affine warps, resize squeeze, color change, residual FFT subtraction, and a JPEG/noise chain, with PSNR floors of only 14/11 dB. The repository claims 20 manual Gemini-app successes but contains no manifest, tally, or per-image verdicts. Its four bundled older cleaned pairs are 45.7-50.2 dB, yet the reproduced V4 score rises on three and is nearly unchanged on the fourth. No callable Google pixel oracle was available for an independent Round-06 verdict. This is an unverified lossy regeneration/distortion stack, not a quiet pattern cancellation | +| [`0xROOTPLS/DeSynth`](https://github.com/0xROOTPLS/DeSynth/tree/96db920731c2a3d04bf13163a5077b9a67706d1b) | 2026-08-24 | Qwen Image img2img at strength 0.25, followed by a Gaussian frequency split that restores the source high frequencies above sigma 1.95. The repository's public OpenAI original, default output, and edge-guided output all returned official `detected` in a current repeat after metadata stripping. The published `not found` claim does not reproduce against the current OpenAI oracle | +| [`froggeric/gemini-watermark-and-synthid-remover`](https://github.com/froggeric/gemini-watermark-and-synthid-remover/tree/5918384ce403968de0560cefd889e50eba0163bc) | 2026-08-24 | SDXL img2img with a documented manual Google-verifier ladder. The author reports 7/8 clears at strength 0.08 and 9/9, including a double mark, at strength 0.10 with five effective denoise steps and PSNR 29-41 dB. The exact nine before/after verdict artifacts are not tracked, so this is useful external regeneration corroboration, not an independently reproduced oracle result | +| [`atomantic/PortOS`](https://github.com/atomantic/PortOS/tree/b11a93e110262925c64a1b145a154ca87b340055) adversarial-jamming experiment | 2026-08-24 | Its own one-image manual OpenAI run found that quality-preserving phase noise, band noise, blur, and 0.70 resize squeeze stayed detected. Only visibly destructive phase perturbation cleared. A 0.85-0.90 resize caused repeated detector timeouts, which the repository correctly keeps separate from `not_detected`. This independently closes high-fidelity additive/phase jamming, but the source artifacts are not published | +| Generic regeneration claims | 2026-08-24 | `mertizci/noai-watermark`, `BovineOverlord/Loyal-Bear`, `obaskly/NeuralBleach`, and `tymongumienik/unwatermark` are SD/CtrlRegen/ControlNet redraws. Their algorithms are already covered by the product's regeneration remainder and the CtrlRegen literature. README verdicts without a frozen manifest do not establish a quieter or more reliable operating point | +| Add unmarked context (frame, pad, hstack, collage) | 2026-08-19 | Official `not_detected` is a presentation gate. White pad 40% linear stays `detected`; photo fill at the same geometry is `not_detected`. See detector identity/layout | +| `cebeuq/Synthid-Bypass` diffusion reconstruction | 2026-08-13 | 12 public pairs. Lattice and Gemini Verify go quiet on the *after* images because the image was redrawn, not because a payload was deleted. Faces and text are a quality claim for regeneration, not a local wipe | + +Oracle: `verify_openai_synthid` after AI-metadata strip. Seeds s1, s2, s3 +as in [detector research](synthid-detector-research.md). Raw files: +`.local-eval/synthid/prc-oklab-attack-2026-08-15/`. + +## Band that actually carries the mark + +Phase-randomize one octave at a time, preserve energy (E2, s1, replicated +p16_32 on s2 and s3): + +| Octave (period px) | Verdict | +| --- | --- | +| 2-4 | detected | +| 4-8 | detected | +| 8-16 | detected | +| 16-32 | not_detected | +| 32-64 | detected | +| 64-128 | detected | + +Destroying only 16-32 px periods silences the official decoder. The +lattice band 8-16 px does not. On a marked gray flat the same scramble is +51.1 dB `not_detected` because the band is almost empty. On a photograph +the band holds the scene (24.3 dB). Replacing native 16-32 with a +2x-pyramid prediction is 48.4 dB and still `detected`. + +Fish additive jam, luma RMS `a`: + +| Band | a | PSNR | Verdict | +| --- | ---: | ---: | --- | +| 16-32 | 0.06-4.0 | 51.2-33.6 | detected | +| 16-32 | 8 | 27.7 | detected | +| 16-32 | 16 | 21.9 | detected | +| 16-32 | 24 | 18.6 | not_detected | +| 4-8 | 16 | 21.9 | detected | +| 4-8 | 24 | 18.6 | detected | + +Preregistered close `a*_photo < 0.53` failed (`a* > 16`). + +## Bayer and geometry (2026-08-22) + +Frozen one-pass batch. Lattice scores on these OpenAI sources are already +`indeterminate` and are not the label. + +| Attack | s1 | s2 | s3 | fish | +| --- | --- | --- | --- | --- | +| source | detected | detected | detected | detected | +| Bayer bilinear | 32.7 detected | 28.4 detected | 37.1 detected | 34.7 detected | +| Bayer VNG | 25.0 detected | 18.5 detected | 26.4 detected | 25.4 detected | +| upscale 1.15 then Bayer | 33.9 detected | 29.8 detected | 38.4 detected | 36.0 detected | +| barrel k1=0.06 | 19.9 detected | 14.2 detected | 20.2 detected | 18.5 detected | +| scanline ±0.8 px | 31.5 detected | 26.7 detected | 35.7 detected | 32.5 detected | +| shift 2 px | 23.1 detected | 15.9 detected | 24.7 detected | 22.7 detected | +| 16-32 phase scramble | 24.6 not_detected | 19.0 not_detected | 25.2 not_detected | 23.2 not_detected | + +Camera pipeline and mild geometry do not hit the decoder basis. A 2 px +shift, which kills `pipeline_lattice`, left s1/s2/s3/fish `detected`. +Scramble remains the residual kill on 4/4 sources. +`.local-eval/synthid/prc-oklab-attack-2026-08-15/bayer-geometry-2026-08-22/`. +Manifest `status` is `oracle_complete` (2026-08-22 22:07). + +## Attack ladders (OpenAI oracle) + +Pilot 2026-08-17 (n=3): JPEG q40, 0.5x, noise8+jpeg85, 0.35x+jpeg50 all +`detected` (s2 0.35x+jpeg50 at 20.1 dB still detected). + +Kill and flip (s1), then replication 3/3: + +| Rung | s1 | s2 | s3 | +| --- | --- | --- | --- | +| blur sigma 6 | detected | detected | detected | +| blur sigma 7 | not_detected | not_detected | not_detected | +| keep 0.50 linear | detected | detected | detected | +| keep 0.40 linear | not_detected | not_detected | not_detected | +| elastic ~24 dB | detected 24.6 | detected 17.9 | detected 26.1 | +| elastic ~21 dB | not_detected 20.8 | not_detected 14.8 | detected 22.0 | +| pad white 40% linear | detected | detected | detected | +| hstack with photo | not_detected | not_detected | not_detected | +| blend 25% marked | detected | detected | detected | +| inset native 1:1 on 2x photo | not_detected | not_detected | not_detected | +| JPEG q5 | detected | detected | detected | + +Holds on 3/3: blur 6 vs 7, crop keep 0.50 vs 0.40, white pad 40% +`detected`, hstack `not_detected`, blend 25% `detected`, JPEG q5 +`detected`. Elastic at ~21 dB does not hold. Crop keep-0.40 and hstack +are presentation or support, not a quiet residual wipe. + +Surround: photo fill at the white-pad geometries is `not_detected` on +3/3; white pad at those sizes was `detected`. Unmarked photo strip flips +between 25% width (`detected`) and 35% (`not_detected`). Photo frame 15% +each side is `not_detected` on 3/3; 12% does not hold. + +## Wild removers (oracle 2026-08-23) + +Preregistered on s1, one pass, 25 s gap. Manifest: +`.local-eval/synthid/prc-oklab-attack-2026-08-15/wild-attacks-2026-08-23/oracle-manifest.json`. + +| Attack | PSNR | Verdict | Note | +| --- | ---: | --- | --- | +| Replace 16-32 with a COCO photo's 16-32 | 25.2 | not_detected | Same annulus as cartesian scramble (24.6 dB). Substituting a camera band kills the decoder, 0.6 dB quieter than scramble on s1, not a new quality class | +| File named polar-1632 | 25.6 | not_detected | Misnamed. The raster randomizes cartesian FFT phase in the 16-32 annulus, same family as scramble (24.6 dB). It is a scramble replicate, not a `(r, θ)` polar test | +| 64-color median-cut palette | 35.6 | detected | Quieter than scramble and still marked. Posterize-4 already stayed `detected`; a smarter quantizer is not enough | +| PIL RGB to CMYK to RGB | inf | not submitted | No-op on this PNG | + +The 16-32 kill is luma phase in that annulus. Cartesian, Fourier-angle, +and radial-phase all silence the decoder near 25 dB. A 90° sector of the +same ring does not (27.5 dB `detected`). Cb-only and Cr-only 16-32 +scrambles stay `detected` at 44-45 dB. A foreign-scene transplant of the +same band also silences the decoder near 25 dB. Palette, Baker-map, seam +carve, Poisson, nested LSB, and ICC do not. + +Follow-up 2026-08-23, s1, 25 s gap, `remaining-2026-08-23/oracle-manifest.json`: + +| Attack | PSNR | Verdict | +| --- | ---: | --- | +| Fourier-angle 16-32 scramble | 24.1 | not_detected | +| Cartesian 16-32 scramble (replicate) | 24.8 | not_detected | +| Baker-map of the 16-32 band | 27.8 | detected | +| Poisson noise | 30.7 | detected | +| Nested LSB in blue | 55.9 | detected | +| Seam carve 8 | 27.7 | detected | +| ICC sRGB rewrite | inf | not submitted, no-op | + +Non-local codecs, 2026-08-23, s1, 25 s gap: + +| Attack | PSNR | Verdict | +| --- | ---: | --- | +| HEIF q80 | 46.3 | detected | +| HEIF q50 | 39.3 | detected | +| AV1 CRF 32 | 37.2 | detected | +| Print-scan simulation | 24.95 | detected | + +Physical print-scan is still blocked unattended (Brother DCP-L2520DW idle and +accepting, no `scanimage`, no ImageCapture pyobjc). Face-gated scramble is +unnecessary: Haar on s1 put *more* 16-32 energy on faces. Generic 25 dB is +not the kill: this simulation stays `detected` at the PSNR where 16-32 +phase scramble does not. + +Waveform-shell splits, 2026-08-23, s1, 25 s gap, +`waveforms-shells-2026-08-23/oracle-manifest.json`: + +| Attack | PSNR | Verdict | +| --- | ---: | --- | +| Y-only 16-32 scramble | 24.6 | not_detected | +| Cb-only 16-32 scramble | 45.0 | detected | +| Cr-only 16-32 scramble | 43.9 | detected | +| 90° Fourier sector of 16-32 | 27.5 | detected | +| Radial-phase-only 16-32 | 25.0 | not_detected | + +## External literature (surveyed 2026-08-23) + +Primary sources. Detector papers live in +[detector research](synthid-detector-research.md). Forensic stealth of +regeneration is already in [synthid.md](synthid.md#23-removal-attacks-and-forensic-detectability). + +| Source | Attack | Against SynthID? | Map to this campaign | +| --- | --- | --- | --- | +| Zhao et al., [arXiv:2306.01953](https://arxiv.org/abs/2306.01953) (NeurIPS 2024) | Add noise, then denoise or regenerate (VAE / diffusion). Pixel-level invisible marks are provably removable. Semantic watermarks proposed as the alternative | Open post-hoc schemes, not production SynthID | This is the family our product uses (`qwen-zimage` / `sdxl-zimage`). Gowal trains SynthID-O against *weak* VAE regeneration. Our foreign-VAE round-trip at 22.3 dB stayed `detected`. Regeneration works when it redraws, not when it is a light codec | +| Liu et al., [arXiv:2410.05470](https://arxiv.org/abs/2410.05470) (CtrlRegen, ICLR 2025) | Controllable diffusion from clean noise, with a knob on how many noise steps to add | SOTA open watermarks | Same family. Goonatilake later finds CtrlRegen+ the *most* forensically detectable remover (AUROC 0.9999) | +| Kassis and Hengartner, [arXiv:2405.08363](https://arxiv.org/abs/2405.08363) (UnMarker, IEEE S&P 2025) | No decoder feedback. Two adversarial spectral optimizations. Breaks even some semantic watermarks (best remaining detection 43%) | Not production SynthID | Spectral disruption without an oracle is the honest analog of our 16-32 scramble, except UnMarker is optimized and we used a one-octave phase shuffle. Goonatilake: UnMarker TPR 98.28% at 0.1% FPR as a *forensic* leftover | +| Tallam et al., [arXiv:2505.08234](https://arxiv.org/abs/2505.08234) (SemanticRegen) | Partial, label-free regeneration of main objects | Tree-Ring, StegaStamp, StableSig, DWT/DCT. Not SynthID | Partial redraw. Our collage / photo-pad `not_detected` is a presentation gate, not this attack | +| Cao et al., [arXiv:2608.10166](https://arxiv.org/abs/2608.10166) (MarkNull, USENIX Security 2026) | On-manifold latent decorrelation via a public diffusion proxy. Claims 100% on 20 Imagen-3 Gemini-verify images. PSNR 25.36 dB, SSIM 0.80 | Small Gemini-verify set | Independent evidence that a no-box latent reconstruction can confuse Gemini. Does not meet this project's 40 dB / 0.99 SSIM release gate. Still generation, not a pixel-only wipe | +| Goonatilake and Ateniese, [arXiv:2605.09203](https://arxiv.org/abs/2605.09203) | Six removers all leave a forensic residue a ResNet-50 sees at >98% TPR @ 1% FPR | Applies to UnMarker, Zhao's WatermarkAttacker, CtrlRegen+ | Defeating `verify-openai-synthid` is not deniability. This is the product remainder | +| An et al., [arXiv:2401.08573](https://arxiv.org/abs/2401.08573) (WAVES, ICML 2024) | 26 attacks on StegaStamp, Stable Signature, Tree-Ring. Regeneration, not JPEG, is the attack that matters. StegaStamp TPR@1%FPR 1.00 to 0.01; Tree-Ring 0.99 to 0.12 | Open watermarks | Protocol. Our blur-sigma-7 and 16-32 scramble are closer to WAVES "distortion" than to regeneration | +| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring) | In-generation Fourier pattern in initial noise | Not SynthID | Kill is DDIM inversion plus latent wipe, not a pixel scramble. Lin and Juarez [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) remove it from public knowledge | +| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tuned VAE decoder | Not SynthID | Regeneration that replaces the decoder is in-family. Our foreign VAE at 22.3 dB did not kill OpenAI SynthID, which is the expected mismatch | +| Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | reverse-SynthID does not systematically remove SynthID | Google spokesperson on the spectral bypass | Agrees with our V4 audit and with Bypass: reconstruction or decoder confusion, not a payload delete | + +Quiet pixel-only removal is not a published result on production +OpenAI or Google SynthID. Every paper that "beats" a watermark either +redraws the image, attacks an open encoder, or reports PSNR in the same +19-25 dB band as our 16-32 scramble. + +## Product remainder for removal + +Invisible removal in this package regenerates through `qwen-zimage` or +`sdxl-zimage`. It is lossy. There is no shipped pixel-only OpenAI or +Google SynthID wipe. Do not add Bayer or geometry as remover arms.