Document general AI classifier sweep

docs/index.md

@@ -36,7 +36,7 @@ The current behavior is defined by the code, tests, README, and user guides.
 - [Qwen improvement research](qwen-improvement-research.md)
 - [Doubao reverse-alpha research](research-doubao-distillation.md)
 - [SynthID local detector research](synthid-detector-research.md)
-- [Classifier models](synthid-classifiers.md) (Model 1 result)
+- [Classifier models](synthid-classifiers.md) (primary metadata-free AI-generation task, Model 1 partial result, and secondary provider source finding)
 - [SynthID mark removal research](synthid-removal-research.md)
 - [SynthID identity research](synthid-robust-identity-research.md)
 - [SynthID identity follow-up](synthid-robust-identity-research-2026-06-08.md)

docs/known-limitations.md

@@ -349,9 +349,25 @@ The `trustmark` extra adds Adobe TrustMark decoding. The implementation retains
 an additional JPEG re-encode gate because isolated decoder hits can otherwise
 be content noise.
 
-External AI versus real image classifiers are out of scope. The project
-identifies concrete local provenance signals instead of shipping a generic
-statistical classifier.
+A generic metadata-free AI-generated-image classifier is not shipped. It is a
+separate open research task from provenance detection: the current Model 1
+result is AI-versus-camera and still confuses some conventional graphics, CGI,
+product cutouts, and scans with generation. The OpenAI/Gemini source finder is
+narrower again and cannot substitute for the general classifier. The shipped
+product identifies concrete local provenance signals instead of presenting
+either research classifier as a supported verdict.
+
+Frozen transfers of Community Forensics, SPAI, SAFE, RINE, Nonescape Mini,
+Dual Data Alignment, PGC, and DGS-Net did not close this gap at the required low
+false-positive rate. DDA supplies a complementary representation but its
+independent errors make simple hybrids worse. PGC SD1.4 is strong on OpenAI,
+but its published output and the useful global/residual ablations misclassify
+most or all of the Kodak scan set; removing that branch removes the OpenAI
+gain. DGS-Net is weak at the same operating point and also adds independent
+photo errors. Model 1 also lacks a time/device-disjoint negative contract for
+modern computational photography. The measured public protocol, GitHub survey, and
+rejected fusions are recorded in
+[classifier research](synthid-classifiers.md#general-ai-classifier-github-sweep-2026-08-25).
 
 ## Output and traceability
 

docs/synthid-classifiers.md

@@ -14,7 +14,21 @@ are the first negative, Firefly and PixelBin are in the test, and a watermark
 claim uses an independent oracle. CLIP content embeddings and the 124-d
 origin-locked residual bank are different features for different jobs.
 
-## Result: Model 1, AI versus camera
+## Research task hierarchy
+
+The primary classifier task is metadata-free AI-generation detection: given an
+arbitrary image, decide `ai_generated` versus `not_ai_generated` from pixels.
+The target is open-world transfer to generators absent from training, with a
+very low false-positive rate across real photographs and other non-AI imagery
+such as scans, product cutouts, conventional CGI, and digital graphics.
+
+OpenAI/Gemini source finding is a narrower secondary task. It asks whether a
+file resembles a current OpenAI or Google generation pipeline and otherwise
+abstains. It does not replace the general AI-generation detector: a precise
+provider finder can miss most AI images, and a general detector need not know
+which provider produced a positive. Neither task is a SynthID payload decoder.
+
+## Partial result: Model 1, AI versus camera
 
 Finetuned CLIP-L (`openai/clip-vit-large-patch14`), last two vision blocks,
 224 letterbox, JPEG and mild crop, linear ridge. Train 5,221 AI plus 6,129
@@ -35,7 +49,10 @@ FPR on disjoint `photo_dev_oi`.
 
 51 fresh false positives are mostly graphics, CGI, product cutouts, and
 scans, not Gemini. Nobody in the sweep hit both ≤1% fresh FPR and ≥90%
-TPR. This is AI-versus-camera, not SynthID, and it is not in `identify`.
+TPR. This is the strongest result toward the general task, but its negative
+contract is still AI-versus-camera rather than AI-versus-all-non-AI imagery.
+The graphics/CGI errors therefore keep the general task open. This is not
+SynthID, and it is not in `identify`.
 
 Artifacts: `.local-eval/synthid/ai-photo-2026-08-22/`
 (`comparison.json`, `probe-report-clip-l-ft.json`,
@@ -68,8 +85,164 @@ most of TC260 and xAI. Do not train another ridge on that representation
 for an AI-or-not claim.
 
 Open, if this head is ever considered for a product cut: a graphics/CGI
-abstain. CLIP treats non-camera imagery as generation; that is the remaining
-error, not Gemini contamination.
+abstain and time/device-disjoint modern camera coverage. CLIP treating
+non-camera imagery as generation is one known error, not Gemini contamination.
+
+### Frozen public-checkpoint transfer, 2026-08-24
+
+A no-training sweep put the official
+[`Community Forensics`](https://github.com/JeongsooP/Community-Forensics) and
+[`SPAI`](https://github.com/mever-team/spai) checkpoints on the same public
+rows and the same operating rule as Model 1. Each threshold is the strict 99th
+percentile of the 500-image `photo_dev_oi` split; no AI or evaluation negative
+sets tune it. The SPAI core runs stop after all 2,405 AI rows because the model
+is already dominated there; they do not supply a fresh-photo FPR.
+
+| Model | AI test | AI extra | FLUX hold | Open Images fresh |
+| --- | ---: | ---: | ---: | ---: |
+| Model 1, CLIP-L-ft | 93.0% | 92.5% | 92.7% | 1.7% |
+| Community Forensics 384 | 34.6% | 12.0% | 23.0% | 1.0% |
+| SPAI, longest edge 512 | 2.2% | 3.5% | 0.7% | not run |
+| SPAI, longest edge 1024 | 6.5% | 9.5% | 10.0% | not run |
+
+Community Forensics finds 33 of Model 1's 170 misses across the 2,405 AI
+rows. On 4,133 public evaluation photographs, however, it adds 37 errors not
+made by Model 1. A calibration-only rank-max fusion reduces AI-test recall to
+91.9%, AI-extra recall to 88%, and FLUX recall to 86%, while fresh-photo FPR
+rises to 1.73%. A literal OR at the two original thresholds doubles calibration
+FPR to 2% because their five errors do not overlap. The checkpoint is an
+auxiliary representation, not a better detector or a valid OR branch.
+
+SPAI at 1024 recovers only 11 Model 1 misses. Its predeclared rank-max fusion
+reduces AI-test recall to 90.2% and FLUX recall to 85.3% at the same 1%
+calibration FPR; its literal OR also doubles calibration FPR to 2%. The
+300-image FLUX cell is exactly 1024 on its longest edge, so this failure cannot
+be assigned to downscaling in that cell. The 512/1024 ablation does show
+resolution sensitivity, but no useful low-FPR hybrid.
+
+[`B-Free`](https://github.com/grip-unina/B-Free) remains unmeasured: its sole
+official checkpoint host was unreachable over HTTP and HTTPS, and no verified
+mirror was found. Its license also limits use to informational and nonprofit
+purposes and expressly prohibits industrial or profit-oriented use. Its useful
+result for this project is therefore the bias-reduction training paradigm, not
+a checkpoint dependency.
+
+No public checkpoint replaces Model 1 or safely repairs it. The next model
+must change the negative contract: hash-grouped, time/device-disjoint modern
+computational photography plus conventional CGI, graphics, scans, and product
+cutouts. Another generic detector trained against a narrow `real` corpus is
+not a new signal.
+
+Local reproducibility artifacts:
+`.local-eval/synthid/ai-photo-2026-08-22/frozen-ai-detector-sweep-2026-08-24/`.
+
+### General AI-classifier GitHub sweep, 2026-08-25
+
+A separate search targeted pixel-based `ai_generated` versus
+`not_ai_generated` classifiers, not SynthID repositories. Twelve recorded
+GitHub GraphQL searches returned 2,006 unique public non-fork repositories.
+The broadest four searches were capped at 500 collected results, so this is a
+bounded reproducible survey, not a claim that GitHub search can enumerate every
+repository. Five current catalogs and benchmarks contributed 110 references;
+106 resolved to 105 unique live repositories. Curated references plus
+high-signal search matches produced 332 candidates, of which 328 resolved for
+README, license, weight, and inference review.
+
+The filter required pixel inference, an available checkpoint, reproducible
+preprocessing, a license compatible with possible product use, and a signal or
+training contract that differs materially from already rejected models. It
+removed metadata/API wrappers, SynthID-only tools, face/video-only deepfake
+systems, datasets and leaderboards, UI-only repositories, classroom CIFAKE
+models, noncommercial checkpoints, and repositories without runnable weights.
+
+The most relevant survivors are:
+
+| Model | Status | Why it matters |
+| --- | --- | --- |
+| [Dual Data Alignment](https://github.com/roy-ch/Dual-Data-Alignment) | Apache-2.0, official 1.26 GB checkpoint, measured partially | DINOv2-L LoRA with paired real/reconstruction JPEG and frequency alignment; best new training contract. |
+| [PGC](https://github.com/xiaoyu6868/PGC) | Apache-2.0, SD1.4 measured fully and joint measured on AI-test | DINOv2-L peak-guided calibration exposes a strong OpenAI signal, but it confounds Kodak scans and does not safely fuse with Model 1. |
+| [DGS-Net](https://github.com/HorizonTEL/DGS-Net) | Apache-2.0, stage-2 checkpoint measured partially | Distillation-guided gradient surgery is reproducible, but the frozen checkpoint is weak and adds independent photo errors. |
+| [FerretNet](https://github.com/xigua7105/FerretNet) | Apache-2.0, weights available, lower priority | Efficient local-pixel artifact branch, but trained on four ProGAN classes. |
+| [OmniAID](https://github.com/yunncheng/OmniAID) | Modern 3.24 GB checkpoints; repository has no license file | Mirage-Train semantic/artifact experts are promising, but the README's MIT badge is not a license grant. |
+| [SDAIE](https://github.com/Ekko-zn/SDAIE) | Weights available; no license | Camera/EXIF-supervised and real-only training are relevant ideas; inference is pixel-based, but product use is unresolved. |
+| [AIDE](https://github.com/shilinyan99/AIDE) and [CO-SPY](https://github.com/Megum1/CO-SPY) | MIT, weights available, lower priority | Reproducible hybrid signals, but official checkpoints retain ProGAN or SD1.4-era negative contracts. |
+
+[Effort](https://github.com/YZY-stack/Effort-AIGI-Detection),
+[Forensic Self-Descriptions](https://github.com/ductai199x/Forensic-Self-Descriptions-CVPR25),
+and B-Free are research-only or noncommercial. UniGenDet is MIT but its
+published checkpoint is about 59 GB. OpenSDI and SAD-Bridge have no detected
+license. REM describes a relevant real-centric method, but its code and weights
+are still pending.
+
+Seven additional checkpoints were put on the same frozen rule: the strict 99th
+percentile of `photo_dev_oi`, with no AI row used for calibration. All values
+below are public cells.
+
+| Model | AI test | Gemini | OpenAI | FLUX hold | Fresh Open Images |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Model 1, CLIP-L-ft | 93.0% | 90.5% | 93.3% | 92.7% | 1.7%, n=3,000 |
+| DDA official | 48.2% | 68.8% | 21.8% | not run | 0.7%, n=1,000 |
+| PGC SD1.4 official | 44.6% | 5.5% | 96.5% | 0.7% | 0.7%, n=3,000 |
+| PGC ProGAN+SD1.4 official | 29.3% | 8.3% | 56.3% | not run | not run |
+| DGS-Net stage 2, seed 100 | 21.2% | 1.5% | 55.5% | not run | 1.9%, n=1,000 |
+| SAFE official | 21.6% | 1.3% | 57.8% | 0.0% | 0.6%, n=3,000 |
+| Nonescape Mini v0 | 13.3% | 13.5% | 7.3% | 37.3% | 1.8%, n=3,000 |
+| RINE four-class | 10.7% | 5.5% | 16.8% | 1.0% | 1.0%, n=3,000 |
+
+DDA supplies material complementary recall. It finds 52
+of Model 1's 133 misses on the 1,905-image AI test. Its seven errors on the
+independent 1,000-image negative slice do not overlap Model 1's 18, however. A
+literal OR reaches 95.7% AI recall and 2.5% FPR there; the predeclared
+calibration-rank maximum reaches only 93.4% recall and 1.6% FPR. The checkpoint
+is therefore not a safe drop-in fusion.
+
+PGC SD1.4 finds 63 of those 133 misses, while the joint checkpoint is weaker
+on the same AI rows. A calibration-only rank maximum with Model 1 initially
+looked like a clean improvement: AI-test recall rose from 93.0% to 94.1%
+(58 paired improvements, 37 regressions; one-sided sign test `p=0.0198`) and
+fresh Open Images FPR fell from 1.67% to 1.13% (22 paired improvements, six
+regressions; `p=0.00186`). The full negative matrix rejects that conclusion.
+The fusion calls all 24 Kodak images AI-generated, versus 0/24 for Model 1,
+and its aggregate public-negative errors rise from 59/4,133 to 63/4,133. It
+also reduces AI-extra recall from 92.5% to 87.5% and FLUX hold recall from
+92.7% to 85.3%.
+
+A full-corpus checkpoint ablation localized but did not remove the confound.
+The global head alone still rejects 17/24 Kodak images while retaining 82.0%
+OpenAI recall. Global plus residual peak retains 96.5% OpenAI and rejects
+24/24 Kodak; global plus RGB peak accepts every Kodak image but retains only
+5.5% OpenAI and 6.8% overall AI-test recall. Calibration-rank conjunctions of
+the RGB and residual components remove the Kodak errors, but either add fresh
+photo errors under their own 1% calibration cuts or fall below Model 1 recall
+after joint recalibration. A post-test high-confidence PGC threshold is not an
+independent result and is not accepted. PGC is an OpenAI-oriented research
+feature, not a universal detector branch.
+
+DGS-Net's official stage-2 image branch was reconstructed strictly from the
+published checkpoint; the training-only frozen teacher and text head are not
+read by the repository's image-only evaluation forward. Its official
+spectral-entropy patch selection retains a random shuffle, so this measurement
+pins the repository's seed 100. The checkpoint finds 37 Model 1 misses but adds
+19 non-overlapping errors on the same 1,000 fresh negatives. A literal OR is
+95.0% AI-test recall at 3.7% FPR; calibration-rank maximum is 91.0% recall at
+1.6% FPR. Its 21.2% standalone recall is far enough below the gate that a full
+corpus or multi-seed run is not warranted.
+
+SAFE, RINE, and Nonescape Mini also fail as frozen replacements or fusions.
+Their value is now bounded: SAFE supplies a wavelet/transformation branch, RINE
+intermediate CLIP blocks, and Nonescape a cheap EfficientNet branch, but none
+improves the low-FPR operating point.
+
+The licensed frozen-checkpoint queue is exhausted at the useful priority level.
+The higher-value path is now a training ablation that imports DDA's paired
+codec/frequency alignment into the project's own time/device-disjoint camera
+and non-photo negative contract. PGC's OpenAI/Kodak confound makes scans an
+explicit hard gate for that work. SDAIE's camera-supervised or real-only
+training remains an idea source until a license exists.
+
+Local search and scoring artifacts:
+`.local-eval/github-ai-detector-sweep-2026-08-25/` and the frozen sweep directory
+above.
 
 ### Wild extras, not SynthID
 
@@ -117,14 +290,15 @@ documented Gemini repeating the first file's SynthID verdict inside a
 chat; Google said that was fixed 2026-07-16. The OpenAI provenance API is
 a different endpoint.
 
-## Provider names from pixels
+## Secondary task: provider names from pixels
 
-With the keyless mark hunt still open, the adjacent ask was: given a file
-with no metadata, is this OpenAI, Gemini, or not AI, with almost no
-errors on camera photographs. That is this section. It is not a SynthID
-detector. Firefly, PixelBin, and other generators have to sit in the
-test, because a head that only sees OpenAI versus Gemini versus COCO
-will call Firefly a provider.
+The narrower ask is: given a file with no metadata, is this OpenAI, Gemini, or
+unknown, with almost no errors on camera photographs or other generators. That
+is this section. `unknown` does not mean `not AI`; it includes AI images from
+other providers and target-provider images the strict rule misses. This is not
+a general AI-generation detector or a SynthID detector. Firefly, PixelBin, and
+other generators have to sit in the test, because a head that only sees OpenAI
+versus Gemini versus COCO will call Firefly a provider.
 
 Three-way `openai` / `google` / `other` on Model 1 embeddings fails the
 Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57;

docs/synthid-oracle-ladders.md

@@ -11,7 +11,7 @@
 | Page | Use it for |
 | --- | --- |
 | [SynthID local detector research](synthid-detector-research.md) | Hunt for a keyless local mark detector. Open. Closed routes on that page. |
-| [Classifier models](synthid-classifiers.md) | Model 1 AI-versus-camera result, rejected provider CLIP, `pipeline_lattice` as google-lineage. |
+| [Classifier models](synthid-classifiers.md) | Primary metadata-free AI-generation task and Model 1 partial result; secondary OpenAI/Gemini source finding; `pipeline_lattice` as google-lineage. |
 | [SynthID mark removal research](synthid-removal-research.md) | Quiet-remover hunt. Closed except ~19-24 dB 16-32 scramble and blur sigma 7. |
 | [Mechanism reference](synthid.md) | How SynthID works, provenance, robustness, regeneration. |
 | [Chronological plan archive](synthid-detector-removal-plan.md) | Dated H-gates, corpora, and session notes in original order. |
@@ -49,3 +49,5 @@
 | Google verifier surfaces 2026-08-24 (retired Vertex `imageverification@001`; live but unlisted and inaccessible `synthid.googleapis.com/upload/v1:verifyContent`) | [detector](synthid-detector-research.md) |
 | Metadata-free source hunt 2026-08-24 (1% cascades rejected; frozen multiscale fusion found 276/600 OpenAI-or-Google targets, 0/1,000 photographs and 1/25 TC260 on blind v4) | [classifiers](synthid-classifiers.md) |
 | Original-export source hybrids 2026-08-24 (frozen v7 215/400 exact; post-hoc v8 292/400 and per-codec v11 316/400 exact, 0/564 independent camera errors; local precision scan 131 OpenAI plus 17 Gemini candidates) | [classifiers](synthid-classifiers.md) |
+| Frozen general-detector transfer 2026-08-24 (Community Forensics and SPAI do not replace or safely fuse with Model 1 at the 1% calibration FPR; B-Free weights unavailable and license noncommercial) | [classifiers](synthid-classifiers.md#frozen-public-checkpoint-transfer-2026-08-24) |
+| General-classifier GitHub sweep 2026-08-25 (2,006 unique repositories found; DDA and PGC expose complementary frozen signals, but DDA raises independent FPR and PGC's OpenAI gain confounds 24/24 Kodak scans; DGS-Net, SAFE, RINE, and Nonescape Mini rejected) | [classifiers](synthid-classifiers.md#general-ai-classifier-github-sweep-2026-08-25) |