diff --git a/docs/ai-generated-image-classifiers.md b/docs/ai-generated-image-classifiers.md index eea6999..c449852 100644 --- a/docs/ai-generated-image-classifiers.md +++ b/docs/ai-generated-image-classifiers.md @@ -907,6 +907,143 @@ CLIP vector (asset `receipt-gate-2026-09-02.npz`, operating point updated); train corpus `cord-train-2026-09-02/`, artifacts `receipt-gate-shipped-2026-09-02/report.json`. +The retrain tail closed 2026-09-07, and the gate moved MODEL-SIDE. The +59 field receipts entered train (owner-approved; the eval cell is spent, +out-of-sample expectation stays the LOO record), phone positives +duplicated 3x against the CORD-dominated pool (an exact unweighted +logistic on the expanded set, shipped update rule; at 1x the harder miss +stays 0.056 below the certified floor even as a train row, at 2x its +leave-one-out score misses by 0.002). Threshold re-certified on CORD +validation + synthetic holdout, no phone pixels: 2.0109 (synthetic floor +binds). Frozen eval: **59/59 field receipts gated, false `ai` 19/59 -> +0/59, CORD 99/99, synth holdout 99/100, ai_test DEFINITELY cost 5/1,847 +(0.27%, better than shipped)**; full leave-one-out over all 59 phone +rows at the certified threshold: **59/59**. The artifact ships under the +STABLE name `receipt-gate.npz` with the threshold inside, the lib loads +head and threshold from one artifact (legacy dated spelling and package +fallback stay readable), so gate updates ride the model version, not a +lib release. Artifacts: +`receipt-gate-retrain-2026-09-07/report.json`, `weight-sweep.json`. + +The tc260 producer split was then measured directly, 2026-09-07, and is +rejected at current mass. The premise that blocked it was false in one +half and true in the other: ContentProducer IS beside the pixels, not +only in the app database. All 1,114 tc260 catalog rows sit in the local +freeze mirror with matching sha256, and `aigc_label` parses a producer +for 1,107 of them (doubao 815, jimeng 144, qwen 24, yuanbao 13, +runninghub 13, kling 5, baidu 5, liblib 4, unmapped USCC tail, 6 +dual-payload Tencent rows whose producer sits in a second AIGC block, +1 escaped-attribute row). The fit harness reproduced `run1/provider.pt` +byte-identically and all four card cells exactly, then refit two split +designs with the public recipe verbatim. A flat split into per-producer +one-vs-rest heads (doubao/jimeng/rest, with and without qwen) collapses +its own cells: doubao test recall 235/265 to 162-168, jimeng 38/58 to +4-7, while google pays 339/373 to 325-304 and meta v3 rises 177/198 to +186-194, the same shared-negative-pool trade the drift campaign +measured, in the opposite direction. Mapped-China recall barely moves +(298 to 300-302/379). A hierarchical second stage that leaves the five +frozen heads untouched and discriminates producer only inside +collective-tc260 predictions cannot regress siblings by construction, +but attributes correctly only 215/298 (72.1%): doubao 75.7%, jimeng +39.5%, rest 88.0%. A company-level class fares no better: replacing the +tc260 bank with a `bytedance` bank (doubao+jimeng hold 628 of 730 train +rows; both products are ByteDance) keeps doubao at 86-87% but drops +jimeng to 55-59%, fires a false company attribution on 41 of 56 +non-ByteDance China test rows (bytedance 16, openai 14, google 4, +meta 4; the cross-fire survives using those rows as train negatives), +recognizes only 17-21% of other ByteDance-family hold-out content, and +a margin sweep trades cross-fire against recall with no viable point +(margin 2.0: cross-fire 1/56 but doubao 30.6%, jimeng 10.3%). At +current mass there is no honest provider class for the China segment in +the 124-d space: `tc260` must stop being a provider VALUE, and the +collective head survives only as a family signal (rows it names report +`provider=None` plus a China-AIGC family hint, weights byte-identical; +every other row's behavior is unchanged by construction). Named China +classes return only with measured train mass. The 124-d residual space +does not separate China producers at current train mass, so "do not +treat a tc260 score as a named manufacturer" is a measured property, +not only contract hygiene. The split precondition is China train mass +(the producer join is banked as +`tc260-producer-split-2026-09-07/producer-join.json`), not metadata +access. Artifacts: `tc260-producer-split-2026-09-07/report.json`, +`stage2-report.json`. + +Product decision, 2026-09-07: `tc260` stops being a provider VALUE. It +was never one class of anything: it covers China's generator ecosystem, +producers (Doubao, Jimeng, Qwen, Kling, Yuanbao, ...) that are peers of +openai/google/meta, and the 2026-09-07 split campaign measured that no +mixed head can honestly name the group at current mass (flat split: +collapsing own cells and google -14/-35; company-level bytedance bank: +cross-fire on 41/56 non-ByteDance China rows, no margin with a viable +point; hierarchical second stage: 72.1% attribution). Shipping the +removal needed one measurement first: deleting the head from the argmax +falsely names 207/379 China test rows openai/google/muse-image (153 of +them muse-image). So the head stays in the checkpoint and in the argmax +as a GROUP VETO: when it wins, `provider=None` is published. Every other +cell is byte-identical (openai 345/380, google 339/373, meta v3 177/198); +China content publishes 350/379 None (the 29 residual misnamings are the +pre-existing ones, unchanged from shipped behavior). Public values are +`openai` / `google` / `muse-image` / `None`. Named China provider classes +return when their train mass holds their own cells; the producer join is +banked as `tc260-producer-split-2026-09-07/producer-join.json`. + +The microsoft class was measured the same day and rejected for a +structural reason, not a mass one. Owner decision had been to add +microsoft as a fourth named class; the fit (family-microsoft bank, 114 +train rows with features freshly extracted from the mirror, 9 sub-256px +rows refused; microsoft appended last so the five original heads keep +seed alignment) recognizes its own test cell at **10/125 (8%)**: 71 of +those rows name `openai`, 9 `google`, and the genuinely fresh Copilot +generation of 2026-09-03 names `openai` too. The 124-d space encodes the +RENDERER, and Microsoft's products render with other vendors' models +(Bing with DALL-E, Designer with Imagen, Copilot with gpt-image), so a +microsoft pixel class cannot exist at any mass; Microsoft attribution is +a metadata job (C2PA signer) that `identify` already surfaces. Side +effect measured and accepted as rejection grounds: google pays 17 test +rows to the reshuffled negative pools. "Provider names the renderer, not +the product UI" is now a measured property. Artifacts: +`microsoft-provider-2026-09-07/report.json`. + +Generator-taxonomy pivot, 2026-09-07. The provider semantics is now +explicitly GENERATOR-based: the class names the model that rendered the +pixels; brands stay metadata. Ground truth exists per file: C2PA +`c2pa.created` `softwareAgent` names the creating software, read by raw +byte scan (uncompressed manifests; compressed-CBOR rows count as +unreadable). Findings: (1) 144 of the 253 microsoft-bank rows name an +OpenAI model as their generator (98 `Azure OpenAI ImageGen`, 46 +`gpt-image` 2.0) and none names a Microsoft first-party renderer; +(2) the entire ChatGPT drift corpus, May 30 through Aug 9, is `gpt-image` +-- the "renderer drift" is WITHIN-generator drift, not a model switch; +(3) 2,498 new unique China rows already sit on the local disk from +historical uploads the freeze never sampled (doubao 1,791, jimeng 310, +qwen 60, yuanbao 46, runninghub 48, kling 15), exceeding the measured +per-producer preconditions roughly twofold, so the generator split for +China is fittable now from local data. Artifacts: +`generator-taxonomy-2026-09-07/generator-labels.json`, +`tc260-producer-split-2026-09-07/historical-harvest.json`. + +The generator split then fitted, 2026-09-07, and shipped as a SURGICAL +candidate. Mass alone did not fix per-producer heads: with the harvest +(jimeng 85 to 351 train rows, 3.5x) separate doubao/jimeng heads still +collapse (jimeng 30/95; the confusion is mutual: jimeng->doubao 32, +doubao->jimeng 22) because Doubao and Jimeng render with ONE ByteDance +model family. The union class fixes it: a `bytedance` bank (doubao+jimeng, +2,394 train rows with the harvest) holds its cell, and the shipped +variant retrains ONLY the bytedance and tc260-fallback heads while +openai/google/meta/no_ai stay byte-identical to run1, so google cannot +regress (a full refit trades google -19 through the shared negative +pools). Numbers, frozen test cells: bytedance 271/323 = 83.9% (extended +529/633 = 83.6% with the harvest; qwen folded into the fallback at 64 +train rows / 27%), google 339/373 unchanged, openai 345->344, meta +177->176 (one row each lost to the new argmax candidate). Gated through +the DEFINITELY path: bytedance names 266/317, openai 318/351 (-1), +google 310/337 identical. Deterministic across refits. The loader +tolerates snapshots predating bytedance (five heads load, their rows +fall to the veto). Public value approved by the owner: `bytedance`. +Artifacts: `provider-bytedance-surgical.pt`, `generator-split-report.json`, +`bytedance-union-report.json`, `bytedance-surgical-report.json`, +`gated-eval.json`, feature cache `china-harvest-124d.npz`. + Open tails from the same day, all measured. (1) The two field misses sit at gate scores -0.15 and -0.09 against the certified 2.043 threshold; covering them would need a threshold at -0.153, raising the ai_test diff --git a/docs/cli.md b/docs/cli.md index 97882f5..592ea47 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -88,15 +88,18 @@ remove-ai-watermarks classify image.png --json The command runs the frozen photo detector (CLIP-L-ft ridge AND freeze MLP). Only a DEFINITELY result is reported as `ai`. On that result only, the receipt-document gate abstains receipt photographs to `unknown` before the -124-d provider head, which may return `openai`, `google`, `muse-image`, or -`tc260`. POSSIBLY is `unknown`. Camera-like photographs are `human`. The call +124-d provider head, which may return `openai`, `google`, `bytedance`, or +`muse-image`. +POSSIBLY is `unknown`. Camera-like photographs are `human`. The call does not run cleanup and does not set `is_ai_generated`. The contract is AI versus camera. Receipts abstain through the gate but are not a supported verdict; UI and digital art are out of scope. Microsoft is not a pixel class: a DALL-E Bing image scores as `openai`, -an Imagen Designer image as `google`. `tc260` is the China AIGC label -standard, not one producer. Weights download on first use from +an Imagen Designer image as `google`. `bytedance` names the shared +Doubao+Jimeng generator lineage. The rest of China's generators (Qwen, +Kling, ...) are peers of the named ones but have no honest class yet; +their residual head abstains, so that content publishes no provider. Weights download on first use from [`wiltodelta/raiw-photo-classify`](https://huggingface.co/wiltodelta/raiw-photo-classify), or from `RAIW_CLASSIFY_WEIGHTS` if that directory already holds the freeze files. diff --git a/docs/module-internals.md b/docs/module-internals.md index 98a796d..b6e3147 100644 --- a/docs/module-internals.md +++ b/docs/module-internals.md @@ -1015,13 +1015,19 @@ photo freeze: CLIP-L-ft ridge AND freeze MLP, then 124-d focal heads only on DEFINITELY. Since 2026-09-02 a linear receipt-document gate runs first on that DEFINITELY path, on the same CLIP vector: a hit publishes `unknown` (detector stays `definitely`, provider is not read, 124-d extraction is -skipped). The head ships with the model: `receipt-gate-2026-09-02.npz` in -the Hub snapshot or `RAIW_CLASSIFY_WEIGHTS` directory, with the package -asset `src/remove_ai_watermarks/assets/receipt-gate-2026-09-02.npz` as +skipped). The head ships with the model: since 2026-09-07 under the +STABLE name `receipt-gate.npz` in the Hub snapshot or +`RAIW_CLASSIFY_WEIGHTS` directory, with the legacy dated spelling +`receipt-gate-2026-09-02.npz` still readable and the package asset +`src/remove_ai_watermarks/assets/receipt-gate-2026-09-02.npz` as the fallback for weights directories frozen before the gate existed (the head is fitted on the freeze CLIP-L-ft embedding space, so it versions -with the model, not with the code); its threshold is pinned in -`classify.py` and in the operating-point sidecar. +with the model, not with the code). Head and threshold are read from ONE +artifact: the threshold travels inside the npz and +`classify_from_scores` resolves it from the loaded gate, so a model-side +gate update needs no lib release. `RECEIPT_GATE_THRESHOLD` in +`classify.py` is only the legacy pinned value for the package fallback +and its pinned tests. Training data and certification live in `receipt-gate-shipped-2026-09-02/report.json` in the research tree; positives are CORD-v2 train (800, CC BY 4.0, disjoint from the CORD test split the @@ -1029,10 +1035,16 @@ eval corpus uses) plus regenerable synthetic receipts, negatives are ai_train rows. `tests/test_classify.py` pins the asset threshold, the downgrade, and that the gate skips forensics. The public label is `ai` / `human` / `unknown`. POSSIBLY is `unknown`. The named class is `openai` / `google` / -`muse-image` / `tc260` / `None`. `tc260` is the China AIGC label standard, -not one producer. The freeze file keys the Muse Image head -`meta_muse_image`. Tests in `tests/test_classify.py` pin the gate without -downloads and pin that `identify` does not import this module. +`muse-image` / `bytedance` / `None`. `bytedance` is the shared ByteDance +generator lineage (Doubao and Jimeng render with one model family; +measured 83.9% on the frozen test cell). `tc260` covers the REST of +China's generator ecosystem, providers that are peers of +openai/google/meta; no mixed head can honestly name the group, so its +argmax win abstains to `None` (measured 2026-09-07: without the veto +207/379 China test rows would be falsely named openai/google/muse-image). The freeze file keys the Muse +Image head `meta_muse_image`. Tests in `tests/test_classify.py` pin the +gate without downloads and pin that `identify` does not import this +module. Weights stay out of git. The Hub snapshot is `wiltodelta/raiw-photo-classify`. `RAIW_CLASSIFY_WEIGHTS` overrides it. The extra is `classify`. The runtime diff --git a/docs/photo-classify-hf/README.md b/docs/photo-classify-hf/README.md index 41ffd55..97953fd 100644 --- a/docs/photo-classify-hf/README.md +++ b/docs/photo-classify-hf/README.md @@ -44,8 +44,10 @@ The training catalog is not in this Hub repository and is not on GitHub. Use it when you have a photographic still and you want a pixel opinion after metadata is gone: -- `label=ai` plus `provider=openai|google|muse-image|tc260` on a stripped ChatGPT, - Gemini, Muse Image, or TC260 photograph +- `label=ai` plus `provider=openai|google|bytedance|muse-image` on a stripped ChatGPT, + Gemini, ByteDance (Doubao/Jimeng), or Muse Image photograph; a stripped + photograph from the rest of China's generators (Qwen, Kling, ...) + publishes `provider=None` (the residual head abstains) - `label=human` on camera photographs at a low false-positive rate - `label=unknown` when the detector is only POSSIBLY AI, or when 124-d features cannot be extracted @@ -66,9 +68,12 @@ ridge trained on that embedding at a 1% Open Images false-positive cut is the strongest Model 1 we measured. A small MLP on the same vectors, ANDed with the ridge, is the freeze DEFINITELY gate. -The receipt gate (`receipt-gate-2026-09-02.npz`) ships in this repository -alongside the freeze files, so it versions with the embedding space it was -fitted on; the Python package carries a fallback copy for weights +The receipt gate ships in this repository alongside the freeze files, so +it versions with the embedding space it was fitted on. Since 2026-09-07 +the artifact carries the STABLE name `receipt-gate.npz` with the +threshold inside, so head and operating point move together and a gate +update needs no library release; the legacy dated spelling stays +readable, and the Python package carries a fallback copy for weights directories frozen before the gate existed. The 124-d bank is a different feature: patch-local residual ratios (FFT band @@ -88,7 +93,8 @@ flowchart TD m1 -->|definitely| rg[Receipt gate on the same CLIP vector] rg -->|receipt document| rgu[label unknown, detector definitely] rg -->|not a receipt| m2[Model 2: 124-d one-vs-rest focal] - m2 -->|openai google muse-image tc260| named[label ai plus provider] + m2 -->|openai google muse-image bytedance| named[label ai plus provider] + m2 -->|china group wins| none2[label ai, provider none] m2 -->|no_ai or extract fail| aiOnly[label ai, provider none] ``` @@ -103,10 +109,16 @@ it is not a receipt-versus-rest specialist and adds no provider class. Provider attribution tracks the **renderer**, not the front-end. Bing Image Creator rows signed Microsoft, OpenAI score as `openai`. Designer rows signed Microsoft, Google LLC score as `google`. Native Designer is mixed. There is -no Microsoft pixel class. `openai` and `google` are provider classes. +no Microsoft pixel class -- measured 2026-09-07: a head trained on +Microsoft-brand rows recognizes 8% of its own test cell, and Microsoft +attribution stays a metadata (C2PA) job. `openai` and `google` are provider classes. `muse-image` is Muse Image output, not a general Meta class. Instagram -`made_with_ai` is not that class. `tc260` is the China AIGC label standard, -not one producer: Doubao, Jimeng, Qwen, Kling, and others share that class. +`made_with_ai` is not that class. `bytedance` names the shared +ByteDance generator lineage: Doubao and Jimeng render with one model +family (separate heads cross-fire even at 3.5x train mass; the union +holds 83.9% on the frozen test cell). `tc260` covers the REST of China's +generator ecosystem -- producers that are peers of openai/google/meta; +no mixed head can honestly name that group, so its win abstains. ## Architecture @@ -132,8 +144,10 @@ not one producer: Doubao, Jimeng, Qwen, Kling, and others share that class. - Decision: a class wins only if it beats `no_ai` by margin 0.30, then argmax among those that passed - Public name `muse-image` is the Muse Image class. The freeze file still - keys that head `meta_muse_image`. `tc260` is the China AIGC label - standard (mixed producers), not a company. `no_ai` is not reported as + keys that head `meta_muse_image`. `bytedance` is the shared + Doubao+Jimeng generator lineage. `tc260` covers the rest of China's + generator ecosystem; no mixed head can honestly name the group, so its + argmax win publishes `provider=None`. `no_ai` is not reported as `human`; it becomes `provider=None` on an `ai` label (for example FLUX) ## Training @@ -176,7 +190,7 @@ DEFINITELY is the shipped operating point. | Detector DEFINITELY | FLUX hold | 83.0% (n=300) | | Class | OpenAI test | 90.8% (345/380 of 381) | | Class | Google test | 90.9% (339/373 of 377) | -| Class | TC260 test | 78.6% (298/379 of 384) | +| Class | Bytedance test (Doubao+Jimeng) | 83.9% (271/323; 83.6% extended with the historical harvest) | | Class | Muse Image hold-out v2 | 85.7% (66/77 listed 79) | | Class | Muse Image hold-out v3 | 89.4% (177/198) | | Class | Muse Image hold-out pooled | 88.4% (243/275 listed 277) | @@ -209,9 +223,12 @@ DEF) at a small recall cost. - **Open-world generators.** FLUX is a hold-out at 83% DEF, not 100%. Unknown generators may land on `ai` with `provider=None`, or miss DEFINITELY. -- **`tc260` is mixed producers.** Doubao, Jimeng, Qwen, Kling, and others - share one label-standard head. A later retrain should split by - `ContentProducer`. Do not read `tc260` as a named manufacturer. +- **`bytedance` is one lineage, not two products.** Doubao and Jimeng + render with one ByteDance model family, so the class names the lineage. + The residual China head (`tc260`: Qwen, Yuanbao, Kling, ...) never + publishes a value; per-producer classes return by `ContentProducer` + when their train mass holds its own cells (qwen at 64 train rows: + 27%, stays in the fallback). ## How to use diff --git a/docs/photo-classify-hf/operating-point.json b/docs/photo-classify-hf/operating-point.json index 2ac8605..91f5f4c 100644 --- a/docs/photo-classify-hf/operating-point.json +++ b/docs/photo-classify-hf/operating-point.json @@ -21,17 +21,22 @@ "model2": { "feature_width": 124, "margin": 0.3, - "classes": ["openai", "google", "tc260", "muse-image", "no_ai"], + "classes": ["openai", "google", "bytedance", "tc260", "muse-image", "no_ai"], "class_kind": { "openai": "provider", "google": "provider", - "tc260": "label-standard", + "bytedance": "provider", + "tc260": "abstain-veto", "muse-image": "model", "no_ai": "negative" }, + "public_values": ["openai", "google", "bytedance", "muse-image", null], + "bytedance_note": "Doubao and Jimeng render with one ByteDance model family; the class names that shared generator lineage. Measured 2026-09-07 on the frozen test cell: 83.9% (271/323), extended with the historical harvest 83.6% (529/633).", + "tc260_note": "the head covers the REST of China's generator ecosystem -- producers (Qwen, Yuanbao, Kling, ...) that are peers of openai/google/meta, not one vendor. No mixed head can honestly name the group, so its argmax win abstains (provider=None). Measured 2026-09-07: without the veto 207/379 China test rows would be falsely named openai/google/muse-image. Per-producer classes return when their train mass holds their own cells (qwen measured at 64 train rows: 27% -- stays in the fallback).", "checkpoint_keys": { "muse-image": "meta_muse_image" }, + "snapshot_tolerance": "provider.pt snapshots predating bytedance load with five heads; absent heads are skipped and their rows fall to the tc260 veto", "runs_only_after": "definitely" }, "receipt_gate": { diff --git a/docs/photo-classify-training.md b/docs/photo-classify-training.md index 524cfe4..3c7c462 100644 --- a/docs/photo-classify-training.md +++ b/docs/photo-classify-training.md @@ -89,10 +89,18 @@ inference and embedding dependency, not a retrain-pack input. ## Next retrain -Keep the current `tc260` head until then. The next provider-head fit should -split that catalog by `ContentProducer` (Doubao, Jimeng, Qwen, Kling, and -the rest) instead of one label-standard class. Do not treat a `tc260` score -as a named manufacturer. +The taxonomy is GENERATOR-based: a class names the model that rendered the +pixels, never the brand. `bytedance` (the shared Doubao+Jimeng lineage) +shipped 2026-09-07 as a surgical head refit at 83.9% on the frozen test +cell; the rest of China (Qwen, Yuanbao, Kling, ...) abstains behind the +tc260 veto until per-producer train mass holds its own cells (qwen +measured at 64 train rows: 27%). The producer metadata is available +locally (all catalog tc260 rows parse their producer from the +sha256-verified mirror; the earlier "app database" premise was wrong), +and the historical harvest keeps adding mass +(`tc260-producer-split-2026-09-07/historical-harvest.json`). Microsoft +has no pixel class and cannot have one: its products render with other +vendors' models, named by their own C2PA softwareAgent. The 2026-09-02 drift campaign set two preconditions for adding renderer drift rows to that fit. First, scale: 14 openai drift rows recovered exactly diff --git a/docs/photo-classify.md b/docs/photo-classify.md index 864b032..608c468 100644 --- a/docs/photo-classify.md +++ b/docs/photo-classify.md @@ -64,7 +64,8 @@ flowchart TD m1 -->|definitely| rg[Receipt gate on the same CLIP vector] rg -->|receipt document| rgu["label unknown, detector definitely, provider none"] rg -->|not a receipt| m2[Model 2 124-d focal heads] - m2 -->|openai google muse-image tc260| named["label ai plus provider"] + m2 -->|openai google muse-image bytedance| named["label ai plus provider"] + m2 -->|china rest wins| none["label ai, provider none"] m2 -->|no_ai or extract fail| aiOnly["label ai, provider none"] ``` @@ -73,7 +74,7 @@ flowchart TD | `label` | `ai`, `human`, `unknown` | Public verdict. `ai` only on DEFINITELY | | `domain` | `photo` | This freeze is photographic only | | `detector` | `definitely`, `possibly`, `likely_human` | Raw Model 1 gate | -| `provider` | `openai`, `google`, `muse-image`, `tc260`, or `None` | Model 2, only if `label` is `ai` | +| `provider` | `openai`, `google`, `bytedance`, `muse-image`, or `None` | Model 2, only if `label` is `ai` | `unknown` with `detector=definitely` is the receipt-document gate abstaining: the file looked AI-generated to Model 1 and like a receipt photograph to the @@ -98,10 +99,21 @@ wrong job on that bank, which is why Model 2 is gated. Provider names the renderer, not the product UI. Bing Image Creator signed Microsoft, OpenAI scores `openai`. Designer signed Microsoft, Google LLC -scores `google`. There is no Microsoft pixel class. `openai` and `google` are +scores `google`. There is no Microsoft pixel class, and that is measured, +not assumed (2026-09-07: a head trained on 114 Microsoft-brand rows +recognizes 8% of its own test cell and names a fresh Copilot generation +`openai`): Microsoft's products render with other vendors' models, so +Microsoft attribution is metadata, not pixels. `openai` and `google` are provider classes. `muse-image` is Muse Image output, not a general Meta -class. `tc260` is the China AIGC label standard, not one producer: Doubao, -Jimeng, Qwen, Kling, and others share that residual class. +class. `tc260` covers the REST of China's generator ecosystem, producers +that are peers of openai/google/meta, not one producer: Qwen, Yuanbao, +Kling, and others sit behind that residual class. No mixed head can +honestly name that group, so its win abstains: China-generated content +from those producers publishes `provider=None` until per-producer classes +exist. `bytedance` is the shared ByteDance generator lineage: Doubao and +Jimeng render with one model family (measured 2026-09-07: separate heads +cross-fire even at 3.5x train mass, the union holds its cell), so the +class names the lineage, not the app that signed. ## Evaluation @@ -115,7 +127,7 @@ Two CPU retrains were byte-identical. DEFINITELY is the shipped cut. | Detector DEFINITELY | FLUX hold | 83.0% (n=300) | | Class | OpenAI | 90.8% (345/380 of 381) | | Class | Google | 90.9% (339/373 of 377) | -| Class | TC260 | 78.6% (298/379 of 384) | +| Class | Bytedance (Doubao+Jimeng lineage) | 83.9% (271/323 of 323; 83.6% extended 529/633 with the historical harvest) | | Class | Muse Image hold-out v3 | 89.4% (177/198) | | Class | Muse Image hold-out pooled | 88.4% (243/275 listed 277) | | Class | meme templates, ungated | 29.1% leak | @@ -138,10 +150,12 @@ publishes `unknown` and skips the 124-d provider pass. | CORD photographs (99) | **0/99** false `ai` | | ai_test DEFINITELY cost | 7/1,847 (0.38%) | -Artifacts: `receipt-gate-shipped-2026-09-02/report.json` (private research -tree). The head ships with the model (`receipt-gate-2026-09-02.npz` in the -Hub snapshot, package asset as the fallback for pre-gate freezes); the -operating point records it in +Artifacts: `receipt-gate-shipped-2026-09-02/report.json` and +`receipt-gate-retrain-2026-09-07/report.json` (private research tree). +The head ships with the model under the STABLE name `receipt-gate.npz` +in the Hub snapshot with the threshold inside the artifact (legacy dated +spelling still readable, package asset as the fallback for pre-gate +freezes); the operating point records it in [photo-classify-hf/operating-point.json](photo-classify-hf/operating-point.json). ## Limits @@ -152,8 +166,10 @@ operating point records it in - Not SynthID, not C2PA, not `is_ai_generated`. - Images under 256 px cannot yield 124-d features, so provider abstains. - FLUX is a hold-out at 83% DEF, not a named provider class. -- `tc260` is mixed producers under one label standard. A later retrain - should split it by manufacturer. +- China's generator ecosystem splits by generator: `bytedance` (the + shared Doubao+Jimeng lineage) is named; the rest (Qwen, Yuanbao, + Kling, ...) abstains until per-producer train mass holds its own cells + (qwen measured at 64 train rows: 27%, stays in the fallback). `identify`, `has_invisible_target`, `all`, and `invisible` do not import this module. A no-signal provenance result stays unknown until you call `classify` diff --git a/docs/python-api.md b/docs/python-api.md index 9627991..4fa853c 100644 --- a/docs/python-api.md +++ b/docs/python-api.md @@ -218,9 +218,11 @@ print(result.label, result.detector, result.provider) `label` is `ai` only on a DEFINITELY detector result (ridge AND freeze MLP). POSSIBLY is `unknown`. Camera-like photographs are `human`. `provider` is -`openai`, `google`, `muse-image`, or `tc260` only when `label` is `ai` and the 124-d -head beats `no_ai` by the freeze margin. `tc260` is the China AIGC label -standard (mixed producers), not a company. Otherwise it is `None`, including +`openai`, `google`, `bytedance`, or `muse-image` only when `label` is `ai` +and the 124-d head beats `no_ai` by the freeze margin. `bytedance` names +the shared Doubao+Jimeng generator lineage; the rest of China's +generators (Qwen, Kling, ...) have no honest class yet and their residual +head abstains to `None`. Otherwise it is `None`, including when 124-d extraction refuses the file. `device` is a library parameter: `None` / `"auto"` detect, `"cpu"` or `"cuda"` diff --git a/scripts/publish_photo_classify_hf.py b/scripts/publish_photo_classify_hf.py index e342c57..1f6f323 100644 --- a/scripts/publish_photo_classify_hf.py +++ b/scripts/publish_photo_classify_hf.py @@ -33,6 +33,7 @@ DETECTOR_FILE = "detector.pt" PROVIDER_FILE = "provider.pt" WEIGHT_FILES = (CLIP_FILE, PROBE_FILE, DETECTOR_FILE, PROVIDER_FILE) GATE_FILE = "receipt-gate-2026-09-02.npz" +GATE_STABLE_FILE = "receipt-gate.npz" REPO_ROOT = Path(__file__).resolve().parents[1] GATE_SOURCE = REPO_ROOT / "src" / "remove_ai_watermarks" / "assets" / GATE_FILE CARD_DIR = REPO_ROOT / "docs" / "photo-classify-hf" @@ -83,6 +84,10 @@ def stage_weights(dest: Path, src: Path) -> None: for name, path in weight_paths(src).items(): _place(path, dest / name) log.info("stage %s (%s bytes)", name, path.stat().st_size) + stable_gate = src / GATE_STABLE_FILE + if stable_gate.is_file(): + _place(stable_gate, dest / GATE_STABLE_FILE) + log.info("stage %s (%s bytes)", GATE_STABLE_FILE, stable_gate.stat().st_size) def publish(stage: Path, *, token: str, message: str) -> str: @@ -95,7 +100,7 @@ def publish(stage: Path, *, token: str, message: str) -> str: repo_id=HUB_REPO, repo_type="model", commit_message=message, - allow_patterns=["README.md", "operating-point.json", *WEIGHT_FILES, GATE_FILE], + allow_patterns=["README.md", "operating-point.json", *WEIGHT_FILES, GATE_FILE, GATE_STABLE_FILE], ) return getattr(commit, "oid", "") or "" diff --git a/src/remove_ai_watermarks/classify.py b/src/remove_ai_watermarks/classify.py index e1b5b8a..6a14ef0 100644 --- a/src/remove_ai_watermarks/classify.py +++ b/src/remove_ai_watermarks/classify.py @@ -36,6 +36,14 @@ _WEIGHT_FILES = (CLIP_FILE, PROBE_FILE, DETECTOR_FILE, PROVIDER_FILE) # holdout; no field-receipt pixels were used for training or the threshold. # The head versions with the model (Hub snapshot or RAIW_CLASSIFY_WEIGHTS # directory); the package asset is the fallback for pre-gate freezes. +# 2026-09-07: the gate is MODEL-SIDE. The weights directory carries the +# artifact under the STABLE name ``receipt-gate.npz`` with the threshold +# inside; the dated name below stays readable as the legacy spelling, and +# the bundled package asset is the last-resort fallback. Head and +# threshold always come from ONE artifact, so a model-side gate update +# needs no lib release; ``RECEIPT_GATE_THRESHOLD`` is only the legacy +# pinned value kept for the package-asset fallback and its pinned tests. +RECEIPT_GATE_STABLE_FILE = "receipt-gate.npz" RECEIPT_GATE_FILE = "receipt-gate-2026-09-02.npz" RECEIPT_GATE_THRESHOLD = 2.0432573877459697 @@ -48,6 +56,7 @@ RECEIPT_GATE_THRESHOLD = 2.0432573877459697 WEIGHTS_ALLOW_PATTERNS: tuple[str, ...] = ( *_WEIGHT_FILES, OPERATING_POINT_FILE, + RECEIPT_GATE_STABLE_FILE, RECEIPT_GATE_FILE, ) @@ -57,19 +66,32 @@ MLP_THRESHOLD = 5.9586493237495395 RIDGE_THRESHOLD = 0.3056212276800537 PROVIDER_MARGIN = 0.30 CLIP_WIDTH = 768 -# Checkpoint keys in provider.pt. openai/google are provider classes; -# meta_muse_image is Muse Image; tc260 is the China AIGC label standard -# (mixed producers). Public names follow that split. -PROVIDER_LABELS = ("openai", "google", "tc260", "meta_muse_image", "no_ai") +# Checkpoint keys in provider.pt, the SUPERSET across snapshot generations. +# openai/google are provider classes; meta_muse_image is Muse Image; +# bytedance is the shared ByteDance generator lineage (Doubao and Jimeng +# render with the same model family -- measured 2026-09-07: separate heads +# cross-fire 32/22 ways even at 3.5x train mass, the union holds 83.9% +# on the frozen test cell). tc260 is NOT one class of anything: it covers +# the REST of China's generator ecosystem (Qwen, Yuanbao, Kling, ..., +# producers that are peers of openai/google/meta), whose heads do not +# exist yet at honest mass, so its argmax win abstains (provider=None). +# The loader tolerates older snapshots that predate bytedance: a head +# absent from provider.pt is skipped, and those rows fall to the veto. +# Measured on the shipped weights: deleting the veto from the argmax +# would falsely name 207/379 China test rows openai/google/muse-image +# (153 of them muse-image); the veto leaves every other cell +# byte-identical (openai 345/380, google 339/373, meta v3 177/198). +PROVIDER_LABELS = ("openai", "google", "bytedance", "tc260", "meta_muse_image", "no_ai") DetectorLevel = Literal["definitely", "possibly", "likely_human"] PixelLabel = Literal["ai", "human", "unknown"] PixelDomain = Literal["photo"] -PixelProvider = Literal["openai", "google", "muse-image", "tc260"] -PUBLIC_PROVIDER: dict[str, PixelProvider] = { +PixelProvider = Literal["openai", "google", "muse-image", "bytedance"] +PUBLIC_PROVIDER: dict[str, PixelProvider | None] = { "openai": "openai", "google": "google", - "tc260": "tc260", + "bytedance": "bytedance", + "tc260": None, "meta_muse_image": "muse-image", } @@ -81,18 +103,21 @@ _receipt_gate_cache: dict[str, dict[str, Any]] = {} def _load_receipt_gate(folder: Path | None = None) -> dict[str, Any]: """Load the receipt-gate head, preferring the weights directory. - Model-side first (the Hub snapshot and ``RAIW_CLASSIFY_WEIGHTS`` carry - ``receipt-gate-2026-09-02.npz``), with the bundled package asset as the - fallback for weights directories frozen before the gate existed. The - head is fitted on the freeze CLIP-L-ft embedding space, which is why it - versions with the model, not with the code. + Model-side first: the stable ``receipt-gate.npz`` in the Hub snapshot + or ``RAIW_CLASSIFY_WEIGHTS`` directory wins, then the legacy dated + spelling, with the bundled package asset as the fallback for weights + directories frozen before the gate existed. The head is fitted on the + freeze CLIP-L-ft embedding space, which is why it versions with the + model, not with the code; the threshold travels inside the artifact. """ import numpy as np key = str(folder) if folder is not None else "" if key in _receipt_gate_cache: return _receipt_gate_cache[key] - source: Path | None = _find_weight(folder, RECEIPT_GATE_FILE) if folder is not None else None + source: Path | None = None + if folder is not None: + source = _find_weight(folder, RECEIPT_GATE_STABLE_FILE) or _find_weight(folder, RECEIPT_GATE_FILE) if source is None: source = Path(__file__).parent / "assets" / RECEIPT_GATE_FILE payload = np.load(source) @@ -107,11 +132,17 @@ def _load_receipt_gate(folder: Path | None = None) -> dict[str, Any]: return gate -def receipt_gate_score(clip_vector: Any) -> float: - """Receipt-document score of a CLIP-L-ft vector; higher is more receipt-like.""" +def receipt_gate_score(clip_vector: Any, gate: dict[str, Any] | None = None) -> float: + """Receipt-document score of a CLIP-L-ft vector; higher is more receipt-like. + + ``gate`` must be the SAME artifact whose threshold decides the hit + (``classify_pixels`` passes the weights-directory gate, so head and + threshold can never come from different files); the default loads the + package-fallback artifact for standalone probing. + """ import numpy as np - g = _load_receipt_gate() + g = _load_receipt_gate() if gate is None else gate v = np.asarray(clip_vector, dtype=np.float64).ravel() v = v / (np.linalg.norm(v) + 1e-12) return float((((v - g["mu"]) / g["sd"]) @ g["w"]) + g["b"]) @@ -169,9 +200,18 @@ def label_for(level: DetectorLevel) -> PixelLabel: def provider_from_scores(scores: dict[str, float], *, margin: float = PROVIDER_MARGIN) -> PixelProvider | None: - """Argmax among openai/google/tc260/muse-image that beat ``no_ai`` by ``margin``.""" + """Argmax among the named classes that beat ``no_ai`` by ``margin``. + + Candidates are the heads the LOADED snapshot actually carries, so + older provider.pt files without a generation's newest class keep + working (their rows fall to the tc260 veto and abstain). The tc260 + head joins the argmax only as a group veto: it covers the rest of + China's generator ecosystem, producers that are peers of + openai/google/meta, so until per-producer classes exist its win + abstains and ``None`` is published instead of a false company claim. + """ no_ai = scores["no_ai"] - ai_names = [name for name in PROVIDER_LABELS if name != "no_ai"] + ai_names = [name for name in scores if name != "no_ai"] passed = [name for name in ai_names if scores[name] > no_ai + margin] if not passed: return None @@ -188,14 +228,18 @@ def classify_from_scores( mlp_threshold: float = MLP_THRESHOLD, provider_margin: float = PROVIDER_MARGIN, receipt_score: float | None = None, - receipt_threshold: float = RECEIPT_GATE_THRESHOLD, + receipt_threshold: float | None = None, ) -> PixelClassification: """Pure gate: detector AND provider, no file I/O. A DEFINITELY verdict whose ``receipt_score`` meets ``receipt_threshold`` downgrades the public label to ``unknown`` (document-domain abstain): the raw detector level stays ``definitely`` and no provider is read. + ``receipt_threshold=None`` (the default) resolves from the loaded gate + artifact, so the operating point versions with the model. """ + if receipt_threshold is None: + receipt_threshold = float(_load_receipt_gate()["threshold"]) level = detector_level( ridge_score, mlp_score, @@ -234,7 +278,7 @@ def classify_pixels(path: Path, *, device: str | None = None) -> PixelClassifica mlp_threshold=runtime.mlp_threshold, ) gate = _load_receipt_gate(runtime.weights_dir) - r_score = receipt_gate_score(clip_vector) if level == "definitely" else None + r_score = receipt_gate_score(clip_vector, gate) if level == "definitely" else None provider_scores = None if r_score is not None and r_score < gate["threshold"]: forensic = _forensic(rgb) @@ -388,6 +432,9 @@ def _load_runtime(device: Any) -> _Runtime: packed = torch.load(_weight_path(folder, PROVIDER_FILE), map_location="cpu", weights_only=True) heads = {} for name in PROVIDER_LABELS: + if name not in packed: + log.info("provider head %s absent from this snapshot; rows it would name abstain", name) + continue head = _build_provider_head() head.load_state_dict(packed[name]) heads[name] = head.to(device).eval() diff --git a/tests/test_classify.py b/tests/test_classify.py index 772af40..218e0d1 100644 --- a/tests/test_classify.py +++ b/tests/test_classify.py @@ -38,14 +38,18 @@ def _scores( tc260: float = 0.0, meta: float = 0.0, no_ai: float = 0.0, + bytedance: float | None = None, ) -> dict[str, float]: - return { + scores = { "openai": openai, "google": google, "tc260": tc260, "meta_muse_image": meta, "no_ai": no_ai, } + if bytedance is not None: + scores["bytedance"] = bytedance + return scores def test_hub_snapshot_is_the_freeze_revision() -> None: @@ -87,11 +91,13 @@ def test_shipped_operating_point_matches_the_runtime_defaults() -> None: assert payload["model1"]["ridge_threshold"] == RIDGE_THRESHOLD assert payload["model2"]["margin"] == PROVIDER_MARGIN assert payload["model2"]["runs_only_after"] == "definitely" - assert payload["model2"]["classes"] == ["openai", "google", "tc260", "muse-image", "no_ai"] + assert payload["model2"]["classes"] == ["openai", "google", "bytedance", "tc260", "muse-image", "no_ai"] assert payload["model2"]["class_kind"]["openai"] == "provider" assert payload["model2"]["class_kind"]["google"] == "provider" - assert payload["model2"]["class_kind"]["tc260"] == "label-standard" + assert payload["model2"]["class_kind"]["bytedance"] == "provider" + assert payload["model2"]["class_kind"]["tc260"] == "abstain-veto" assert payload["model2"]["class_kind"]["muse-image"] == "model" + assert payload["model2"]["public_values"] == ["openai", "google", "bytedance", "muse-image", None] assert payload["model2"]["checkpoint_keys"]["muse-image"] == "meta_muse_image" assert payload["receipt_gate"]["threshold"] == RECEIPT_GATE_THRESHOLD assert payload["receipt_gate"]["asset"] == RECEIPT_GATE_FILE @@ -117,7 +123,34 @@ def test_provider_requires_margin_over_no_ai() -> None: assert provider_from_scores(_scores(openai=0.5, no_ai=0.4)) is None assert provider_from_scores(_scores(google=2.0, openai=1.5, no_ai=0.0)) == "google" assert provider_from_scores(_scores(meta=1.0, no_ai=0.0)) == "muse-image" - assert provider_from_scores(_scores(tc260=1.0, no_ai=0.0)) == "tc260" + + +def test_bytedance_names_the_shared_bytedance_renderer_lineage() -> None: + """Doubao and Jimeng render with one ByteDance model family, so the + generator class is the union; the value names the lineage.""" + assert provider_from_scores(_scores(bytedance=1.0, no_ai=0.0)) == "bytedance" + assert provider_from_scores(_scores(bytedance=3.0, openai=1.0, no_ai=0.0)) == "bytedance" + assert provider_from_scores(_scores(bytedance=1.0, openai=2.0, no_ai=0.0)) == "openai" + + +def test_older_snapshots_without_bytedance_keep_working() -> None: + """provider.pt predating bytedance has five heads; the candidate set + comes from the LOADED scores, so such snapshots run unchanged and + their would-be bytedance rows fall to the tc260 veto.""" + legacy = _scores(tc260=2.0, openai=1.0, no_ai=0.0) + assert "bytedance" not in legacy + assert provider_from_scores(legacy) is None + assert provider_from_scores(_scores(openai=3.0, no_ai=0.0)) == "openai" + + +def test_tc260_wins_abstain_instead_of_naming_a_provider() -> None: + """tc260 is not one class of anything: it covers China's generator + ecosystem, providers that are peers of openai/google/meta. No mixed + head can honestly name that group, so its argmax win publishes None; a + row where a real provider outscores it keeps that provider.""" + assert provider_from_scores(_scores(tc260=1.0, no_ai=0.0)) is None + assert provider_from_scores(_scores(tc260=3.0, openai=1.0, no_ai=0.0)) is None + assert provider_from_scores(_scores(tc260=1.0, openai=2.0, no_ai=0.0)) == "openai" def test_definitely_plus_openai_is_ai_openai() -> None: @@ -223,6 +256,59 @@ def test_receipt_gate_prefers_the_weights_directory(tmp_path: Path) -> None: _receipt_gate_cache.clear() +def test_receipt_gate_stable_name_beats_the_legacy_spelling(tmp_path: Path) -> None: + """The model-side artifact (stable name) owns the gate: same directory, + stable spelling wins, and head plus threshold come from that ONE file. + """ + from remove_ai_watermarks.classify import ( + RECEIPT_GATE_STABLE_FILE, + _load_receipt_gate, + _receipt_gate_cache, + receipt_gate_score, + ) + + source = np.load(Path("src/remove_ai_watermarks/assets") / RECEIPT_GATE_FILE) + weights = tmp_path / "weights" + weights.mkdir() + np.savez(weights / RECEIPT_GATE_FILE, **{k: source[k] for k in ("w", "b", "mu", "sd")}, threshold=99.0) + np.savez( + weights / RECEIPT_GATE_STABLE_FILE, + w=np.full(768, 0.5), + b=0.0, + mu=np.zeros(768), + sd=np.ones(768), + threshold=-99.0, + ) + _receipt_gate_cache.clear() + try: + gate = _load_receipt_gate(weights) + assert gate["threshold"] == -99.0 + probe = np.full(768, 1.0 / np.sqrt(768), dtype=np.float64) + # The score must come from the SAME stable-name artifact: with its + # uniform weights the score is exactly 0.5*sqrt(768), not the legacy + # package head's value. + assert receipt_gate_score(probe, gate) == pytest.approx(0.5 * np.sqrt(768)) + finally: + _receipt_gate_cache.clear() + + +def test_receipt_threshold_defaults_to_the_artifact_value() -> None: + """classify_from_scores without an explicit threshold resolves the + operating point from the loaded gate artifact (legacy package fallback + here), not from a code constant.""" + from remove_ai_watermarks.classify import _load_receipt_gate, _receipt_gate_cache + + _receipt_gate_cache.clear() + try: + artifact_threshold = _load_receipt_gate(None)["threshold"] + just_above = classify_from_scores(1.0, 10.0, None, receipt_score=artifact_threshold + 1e-9) + just_below = classify_from_scores(1.0, 10.0, None, receipt_score=artifact_threshold - 1e-9) + assert just_above.label == "unknown" + assert just_below.label == "ai" + finally: + _receipt_gate_cache.clear() + + def test_receipt_gate_falls_back_to_the_package_asset() -> None: from remove_ai_watermarks.classify import _load_receipt_gate, _receipt_gate_cache