Victor Kuznetsov
99146cf2de
Keep Model 1 as a research AI-versus-camera result outside the library. Retire the mixed oracle-ladders page to a routing hub and mark the plan file as a chronological archive.
7.3 KiB
Classifier models (research)
Research archive for pixel classifiers that are not SynthID detectors. Not a statement of current product capability. Shipped behavior: supported signals and known limitations.
Sister pages: SynthID local detector, SynthID mark removal, mechanism reference.
A classifier is reliable only when its name matches its errors, photographs are the first negative, Firefly and PixelBin are in the test, and a watermark claim uses an independent oracle. CLIP content embeddings and the 124-d origin-locked residual bank are different features for different jobs.
Result: Model 1, AI versus camera
Finetuned CLIP-L (openai/clip-vit-large-patch14), last two vision blocks,
224 letterbox, JPEG and mild crop, linear ridge. Train 5,221 AI plus 6,129
photos. Locked Open Images fresh never enters train. Operating point: 1%
FPR on disjoint photo_dev_oi.
| Cell | Value |
|---|---|
| Kodak | 0/24 |
| Open Images fresh FPR | 1.7% (n=3,000) |
| Exact-1024 Open Images FPR | 6% |
| AI-test TPR | 93.0% (n=1,905) |
| OpenAI | 93.2% |
| Gemini | 90.5% |
| Firefly | 94.0% |
| xAI | 96.1% |
| FLUX hold | 92.7% |
51 fresh false positives are mostly graphics, CGI, product cutouts, and
scans, not Gemini. Nobody in the sweep hit both ≤1% fresh FPR and ≥90%
TPR. This is AI-versus-camera, not SynthID, and it is not in identify.
Artifacts: .local-eval/synthid/ai-photo-2026-08-22/
(comparison.json, probe-report-clip-l-ft.json,
probe-weights-clip-l-ft.npz). Date cutoff 2026-07-23, seed 20260822.
Rejected Model 1 variants
Same splits and photo_dev_oi 1% cut.
| Variant | Fresh FPR | Kodak | 1024 FPR | AI TPR | FLUX hold |
|---|---|---|---|---|---|
| CLIP-L v2 | 0.017 | 0/24 | 0.04 | 0.877 | n/a |
| CLIP-L + FLUX extra | 0.016 | 0/24 | 0.05 | 0.861 | 0.707 |
| CLIP-H + FLUX extra | 0.014 | 0/24 | 0.02 | 0.812 | 0.913 |
| CLIP-L last-2-blocks finetune | 0.017 | 0/24 | 0.06 | 0.930 | 0.927 |
| DINOv2-giant 256 | 0.023 | 0/24 | 0.04 | 0.606 | 0.293 |
CLIP-H is the photo-FPR specialist (1.4% fresh, 2% at 1024) at 81% TPR and is not the result. DINOv2-giant at 256 px is not usable.
v1 (CLIP-L, no Open Images in train) at a COCO-looking 0.5% cut accepted
13% of Open Images. Domain shift, not the 124 residual bank. v2 added
1,000 disjoint Open Images reserve photos to train and 500 as
photo_dev_oi; locked fresh stayed 1.7% FPR at 87.7% TPR before
finetune.
The 124-d residual bank is the wrong feature for "AI or not". At a Kodak-safe cut it catches 60% Firefly and misses FLUX, NovelAI, Reve, and most of TC260 and xAI. Do not train another ridge on that representation for an AI-or-not claim.
Open, if this head is ever considered for a product cut: a graphics/CGI abstain. CLIP treats non-camera imagery as generation; that is the remaining error, not Gemini contamination.
Closed: provider names from pixels
Three-way openai / google / other on Model 1 embeddings fails the
Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57;
Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the
124-d lattice bank it is 0.989. They are two pipelines, not one class.
Collapsing OpenAI and Gemini into one pixel class versus other generators does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%, Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at 3%. Training the same union only against photographs recreates Model 1 with a narrower train set (fresh FPR 2.1%, Firefly still 95%).
provider-report-clip-l-ft.json, provider-union-report.json.
124-d lattice as pipeline ID, not a vendor CLIP head
Provider-class ridge on 124 native residual features (70/30 once, not a watermark gate). OpenAI L1 n=285, Google corpus n=533, foreign n=218, COCO n=289: OpenAI vs COCO 0.965; Google vs COCO 0.999; OpenAI vs Google 0.989; OpenAI vs foreign 0.725; Google vs foreign 0.922. OpenAI vs Firefly-class is the weak cell.
One-vs-rest: Google head at TPR 90% has FPR 0% vs COCO, 14% vs foreign, 2%
vs OpenAI. A Gemini-like pixel class is close to what pipeline_lattice
already is. An OpenAI-like pixel class on this bank would label Firefly as
OpenAI about half the time and is not shippable.
Three-class openai / google / no_ai on 2,000 catalog OpenAI, 2,000
catalog Google, and 1,936 COCO photos. Photo-first margin 0.50: openai
74.7%, google 78.9%, no_ai 99.8%; Kodak 24/24 no_ai. Other generators
are leakage, not classes:
| Platform | n | openai | no_ai | |
|---|---|---|---|---|
| Firefly | 106 | 37 | 27 | 42 |
| Microsoft | 117 | 39 | 22 | 56 |
| PixelBin | 90 | 14 | 46 | 30 |
| HuggingFace job | 82 | 3 | 62 | 17 |
| ByteDance C2PA | 86 | 5 | 35 | 46 |
| SD / Comfy | 120 | 30 | 11 | 79 |
| fal.ai | 98 | 11 | 18 | 69 |
| Made-with-AI tag | 115 | 4 | 32 | 79 |
| TC260 | 118 | 1 | 13 | 104 |
| xAI | 114 | 1 | 20 | 93 |
| Canva | 76 | 11 | 3 | 62 |
| Apple Clean Up | 114 | 1 | 23 | 90 |
| Aweme | 37 | 2 | 7 | 28 |
| FLUX | 11 | 0 | 0 | 11 |
| Reve | 10 | 0 | 0 | 10 |
| NovelAI | 9 | 0 | 0 | 9 |
| Higgsfield | 11 | 1 | 5 | 5 |
PixelBin and HuggingFace jobs lean google (shared renderer lineage).
FLUX, NovelAI, and Reve stay no_ai. Local probe:
uv run python .local-eval/synthid/prc-oklab-attack-2026-08-15/classify_openai_gemini.py image.png.
Production pipeline_lattice (google-lineage renderer)
Experimental signal in identify, never a watermark. Production
detect_synthid re-check on 628 frozen holdouts, seed 20260822, threshold
1.0.
| Family | n | detected | rate | max score |
|---|---|---|---|---|
| Google / Gemini | 80 | 45 | 0.56 | 3.03 |
| Firefly | 84 | 15 | 0.18 | 3.00 |
| PixelBin | 80 | 11 | 0.14 | 2.48 |
| Microsoft | 60 | 2 | 0.03 | 2.40 |
| OpenAI | 80 | 1 | 0.01 | 1.38 |
| xAI | 40 | 1 | 0.03 | 1.26 |
| FLUX HF | 40 | 0 | 0 | 0.85 |
| TC260 | 40 | 0 | 0 | 0.97 |
| Kodak | 24 | 0 | 0 | 0.49 |
| Open Images fresh | 60 | 0 | 0 | 0.64 |
| COCO hold | 40 | 0 | 0 | 0.74 |
Firefly 18% and PixelBin 14% match the 2026-08-16 signed-foreign rates
(24% and 14%) in order of magnitude. Both Microsoft hits have issuer
Microsoft, Google LLC. A 2 px crop killed every sampled positive,
including Firefly and PixelBin. Google TPR 56% is mixed Spaces eras, not
the oracle-positive 147/148 cell. Honest name:
google_lineage_renderer = Gemini/Imagen + Firefly + PixelBin.
Registered-v3 photographic controls remain 0/5,993 Open Images and 0/2,366
COCO. Against 223 C2PA-named non-Google generators on 2026-08-16: 29
accepted (0.130), Firefly 0.241. .local-eval/synthid/lattice-check-2026-08-22/.
Spaces catalog sizes (2026-08-21)
49,082 unique sha256. Unlabeled 24,832 rows are not photographs. Microsoft
127/279 and Firefly 64/210 also carry synthid_from_provenance=true, so
that flag is not an OpenAI-plus-Gemini class.
| Platform | n |
|---|---|
| none / unlabeled | 24,832 |
| OpenAI | 11,722 (11,347 SynthID-from-provenance) |
| Google / Gemini | 6,875 (plus 94 Google C2PA without a named generator) |
| China AIGC TC260 (not a brand) | 3,610 |
| Microsoft | 279 |
| Meta-style Made-with-AI tag | 275 |
| Adobe Firefly | 210 |
| xAI | 179 |
| local SD / Comfy | 178 |
| ByteDance platform | 88 |
| fal.ai | 98 |
| Canva | 79 |
| ByteDance Aweme tag | 40 |
| Dreamina tag | 4 |