diff --git a/data/README.md b/data/README.md index a76aadd..de5d994 100644 --- a/data/README.md +++ b/data/README.md @@ -22,6 +22,12 @@ data/ ## Storage rules +Large local-only research data (the 211 GB Spaces corpus and the frozen +checkpoints, research corpora, and campaign reports) lives under gitignored +paths documented in `data/research/README.md` (ignored via `.git/info/exclude`, +never committed). Worktree `.local-eval/` directories are scratch copies; the +durable store is `data/research/` in the main checkout. + 1. Store each binary image once. Evaluation manifests and documentation point to its canonical location. 2. Put executable test fixtures in `fixtures/`. diff --git a/docs/ai-generated-image-classifiers.md b/docs/ai-generated-image-classifiers.md index 67f6c70..8155b5e 100644 --- a/docs/ai-generated-image-classifiers.md +++ b/docs/ai-generated-image-classifiers.md @@ -694,6 +694,89 @@ family over the frozen Model 1 CLIP-L embedding for the modern domains as well and confirms the earlier aligned-mining finding: the add representation itself must change, not the example selection or the veto geometry. +### Wild vendor-flagged AI, 2026-08-27 + +A free wild-AI positive cell came from the stock providers behind the private +veedma-blog image pipeline. Pixabay exposes a vendor-declared +`isAiGenerated` flag and the query `ai generated` surfaces that pool (3,287 +results); 284 content-hash-unique rows were harvested (300 minus 16 lost to a +duplicate-download bug, since fixed, manifest repaired to on-disk truth). The +vendor flag is ground truth for AI-versus-human, not for the renderer, so the +cell is eval-only. A metadata scan found 0/284 provenance signals (the stock +CDN strips everything), making it a pure pixel task. Scored with frozen +classifiers that never saw stock-site content, this produced the first +source-disjoint wild-AI recall measurement: Model 1 accepted 183/284 (64.4%) +at its frozen 1%-FPR threshold, far below its 93.0% curated AI-test recall +(median score 0.43 above the 0.306 threshold, but a quarter of the cell scores +below 0.13). Wild AI images of unknown renderers, re-encoded by a stock +pipeline, are therefore a genuinely harder positive distribution, and curated +test recall does not bound deployment recall. The frozen provider cascade +under its photo-first margin called 255/284 `no_ai` with only 1 `openai` +and 0 `meta` calls, so unknown-renderer AI does not fabricate provider +attributions; under argmax it leans `google` (199) without evidence. The cell +joins the dataset as the first wild-AI eval row source. Artifacts: +`pixabay-ai-positive-2026-08-27/` (manifest, images, +classification-report.json). + +A bias audit then rejected that cell's composition, though not its verdict: +147/284 rows (52%) came from a single contributor and every row from one +query, so the 64.4% recall was one author's dump plus a tail. Model 1 recall +split 63.3% on that contributor versus 65.7% on the rest, so difficulty was +homogeneous, but the sample was not. A stratified reharvest capped at 15 rows +per contributor over eight AI-marker queries produced 300 unique images from +180 contributors (top-10 share 33%) with the same sieves. On the balanced +cell the frozen numbers are: Model 1 recall 209/300 (69.7%, median score 0.482, +lowest quartile below 0.144), direct quarter-hard 189/300 repairing 22 Model 1 +misses and losing 2 (the one place the quarter representation still helps), +conditional routing 210/300 adding one repair and nothing else, provider +cascade margin 252/300 `no_ai` with 1 `openai` call, and 0/300 metadata +signals. The wild-AI recall ceiling under Model 1 therefore settles near +65-70% rather than 93%, robust to contributor stratification; unknown-renderer +AI re-encoded by stock pipelines is the largest measured positive-side gap. +Artifacts: `pixabay-ai-stratified-2026-08-27/` (the eval cell the dataset +keeps; the first harvest is retained only as the audit trail of the bias +finding). + +The stock-negative harvest from the same providers then tripled the modern +negative cells under identical sieves: 150 Pixabay product cutouts +(vendor `isAiGenerated=false`), 150 Pixabay fashion rows, and 152 Unsplash +API fashion rows date-bounded before 2022-08-01 (452 total, 0/452 metadata +signals, all content-hash unique). Frozen Model 1 false-positive rates on +them: product cutouts 40/150 (26.7%), Pixabay fashion 93/150 (62.0%), +Unsplash fashion 108/152 (71.1%). The fashion failure is therefore larger +than the earlier 34% measured on the 88-row date-clean cell and holds across +three independent sources; retouched fashion photography is now the single +worst negative domain for Model 1. The provider cascade stays quiet on all +three cells (no_ai 126/150, 105/152, 99/150; at most 15 stray `google` calls, +no openai or meta fabrications). A second stochastic pass over the Meta +muse-image-1.0 grid added 61 more API generations with zero hash overlap +with pass one (corpus now 127 + 5 checkout originals), moving the Meta class +toward margin calibrability. Artifacts: +`stock-negative-2026-08-27/`, `meta-muse-corpus-pass2-2026-08-26/`. + +The combined-pool control then separated the data question from the geometry +question: refitting the same pooled/domain/multiclass ridge vetoes on a +687-row modern-negative pool (Openverse-clean plus all three stock cells, +three times the original fit size) reproduced the earlier Pareto curve to the +point: at positive compliance the best dev error count stayed 41/186 with +zero route errors repaired, and every repaired negative still cost about 1.4 +AI positives. The linear-veto failure over the frozen Model 1 embedding is +therefore not a data-volume artifact; tripling diverse negatives moves +nothing. Artifacts: +`modern-domain-veto-combined-2026-08-27/` (development report and pareto). + +The doubled Meta corpus then allowed its own margin sweep (127 API rows in +the bank after feature extraction, 25 repeated splits, photo leak measured on +held-out no_ai rows). The curve is steep exactly where the frozen 0.50 margin +sat: margin 0.00 recalls 92.2% meta at 5.3% photo AI-rate, 0.10 gives 90.1% +at 3.6%, 0.20 gives 85.2% at 2.2%, then the cliff: 0.30 falls to 70.2%, 0.40 +to 39.4%, and the frozen 0.50 lands at 12.4%. A workable photo-safe +operating point exists at margin 0.20 (85% meta recall at 2.2% photo leak), +but 2.2% is still thirty times the Model 1 photo contract, so for provider +attribution the honest rule stays argmax until an external held-out meta cell +exists. The margin, not the head, was always the limiter on the small class. +Artifacts: `meta-margin-2026-08-27/report.json`. + A taxonomy continuation then changed the training mix itself instead of the veto: two arms re-ran the expanded quarter-hard recipe (ordinary AI replay, EvalGEN pair positives, ordinary photos) with one or two of the eight negative diff --git a/docs/synthid-classifiers.md b/docs/synthid-classifiers.md index 342ddcc..6df312f 100644 --- a/docs/synthid-classifiers.md +++ b/docs/synthid-classifiers.md @@ -37,6 +37,71 @@ Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57; Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the 124-d lattice bank it is 0.989. They are two pipelines, not one class. +### Renderer, not front-end (cross-carrier validation, 2026-08-26) + +The Spaces catalog's C2PA issuer field allows a provider-label audit the +family labels cannot express: Bing Image Creator rows signed `Microsoft, +OpenAI` (renderer DALL-E, n=152), Microsoft-native rows (MAI-Image / Designer, +n=85), Designer rows signed `Microsoft, Google LLC` (renderer Imagen, n=36), +and the unattributed Instagram `made_with_ai` label set (n=275). Scored with +the frozen 124-feature openai/google/no_ai cascade, no training, metadata +used only to build cells: argmax called 124/152 (81.6%) of the Bing+DALL-E +cell `openai` and 34/36 (94.4%) of the Microsoft+Imagen cell `google`. Pixel +attribution therefore tracks the renderer behind the front-end, which is the +correct semantic for a stripped file: a Bing export is an OpenAI-rendered +image. The Microsoft-native cell split 48 openai / 35 google, consistent with +Designer historically routing between DALL-E and Imagen rather than one +pipeline. The Instagram `made_with_ai` set is not a Meta pixel class: argmax +leaned `google` at 196/275 with 68 `openai`, and the photo-first 0.50 margin +sent 182/275 to `no_ai`, so the label marks detected AI content of mixed +origin, not Meta-rendered pixels. A real Meta class needs fresh +`Imagined with AI` generations with known provenance; the catalog label is +not one. Artifacts: +`provider-renderer-cells-2026-08-26/report.json`. + +### A Meta class from muse-image-1.0, 2026-08-26 + +The Meta Model API (`api.meta.ai/v1`, OpenAI-compatible images endpoint) made +a known-provenance Meta corpus possible without any account browser session: +61 images generated from a 61-prompt grid spanning portraits, product shots, +scenes, food, animals, architecture, illustration styles, text posters, and +abstract work across all three aspect ratios, plus the five oracle-verified +samples already in `data/contentseal/originals/`. All 66 are content-hash +unique, delivered at 1600x1600 / 1920x1280 / 1280x1920, and every API row +carries IPTC `trainedAlgorithmicMedia` plus a Content Seal generation id +recorded in the manifest. A four-head control with a `microsoft_native` class +(83 issuer-verified rows) failed first: repeated-split argmax recall 0.38 mean +with 0.16-0.56 spread, because Designer routes between renderers and the class +is mixed. The same recipe with `meta_muse_image` in its place holds: argmax +recall 0.86 mean (0.75-1.00) on just 66 images, openai 0.952 and google 0.956 +unharmed, and the frozen cross-carrier cells keep their renderer semantics +(Bing+DALL-E 119/146 openai, Microsoft+Imagen 32/36 google) with only small +meta leakage (14 and 3 argmax rows). The class label names what the corpus is: +muse-image-1.0 API output, not the unverified assumption that the consumer +Imagine feature renders identically. The full-data model calls 59/66 meta rows +correctly, with seven leaking to google and none to openai. The photo-first +0.50 margin does not transfer to a 66-image class (0.398 mean recall); argmax +is the honest operating rule until the corpus grows. Muse Image is therefore +a separable fourth provider class on pixels, unlike the Microsoft front-end. +Artifacts: `meta-muse-corpus-2026-08-26/`, `four-head-provider-2026-08-26/`, +`meta-provider-cells-2026-08-26/`. + +A paired chat-vs-API check then closed the label question. Two prompts from +the API grid (weathered fisherman portrait, white-sneaker product shot) were +submitted to the consumer Meta AI chat imagine flow in the user's own logged-in +browser session; the chat delivered 1280x1920 and 1920x1280 WebP files from the +`t39.105495-1` CDN family, each carrying IPTC `trainedAlgorithmicMedia` in the +downloaded bytes. Scored by the frozen API-trained four-head model, which had +never seen a chat image: both chat rows landed on `meta_muse_image` under +argmax (0.251 and 0.233) with score profiles matching their API twins almost +exactly, and the margin rule agreed pairwise (fisherman meta at margin, sneaker +no_ai at margin 0.50, same as its API counterpart). The chat imagine pipeline +and `muse-image-1.0` are therefore pixel-indistinguishable to the provider +classifier on this pair, so `meta_muse_image` honestly names both +distributions. Caveat: n=2 paired prompts from one session; this is a +consistency result, not a deployment-scale equivalence claim. Artifact: +`meta-chat-check-2026-08-26/`. + Collapsing OpenAI and Gemini into one pixel class versus other generators does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%, Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at diff --git a/tests/test_synthid_detector.py b/tests/test_synthid_detector.py index f547215..f8e4e3d 100644 --- a/tests/test_synthid_detector.py +++ b/tests/test_synthid_detector.py @@ -689,12 +689,22 @@ def test_non_divisible_fold_matches_modulo_cell_means() -> None: rng = np.random.default_rng(44041) pixels = rng.integers(0, 256, size=(53, 71, 3), dtype=np.uint8) source = pixels.astype(np.float32) - residual = source - cv2.GaussianBlur( - source, - (0, 0), - sigmaX=1.25, - sigmaY=1.25, - borderType=cv2.BORDER_REFLECT_101, + # Mirror the module's documented per-channel blur: OpenCV's multi-channel + # GaussianBlur is not bit-identical to per-channel calls (observed 3e-5 on + # cv2 4.10.0), so a three-channel reference cannot satisfy atol=0. + residual = np.stack( + [ + source[:, :, channel] + - cv2.GaussianBlur( + source[:, :, channel].copy(), + (0, 0), + sigmaX=1.25, + sigmaY=1.25, + borderType=cv2.BORDER_REFLECT_101, + ) + for channel in range(3) + ], + axis=2, ) expected = np.empty((16, 16, 3), dtype=np.float64) for tile_y in range(16):