Compare commits

..
8 Commits
Author SHA1 Message Date
test-userandClaude Opus 4.7 d24d8a4b14 Extract _target_size helper + regression-test native resolution (v0.5.4)
The native-vs-downscale decision in InvisibleEngine.remove_watermark (the
issue #10/#15 fix: max_resolution=0 must not pre-downscale, since any
downscale both loses quality and lets SynthID survive) had no test. Extract
it into a pure helper invisible_engine._target_size(w, h, max_resolution)
and cover it with tests/test_invisible_engine.py::TestTargetSize so a
re-introduced forced downscale fails CI instead of silently regressing #15.

Also:
- Clamp the short side to >=1 in _target_size: extreme aspect ratios (e.g.
  5000x3 with --max-resolution 1024) truncated it to 0 and crashed
  image.resize(). Pre-existing in the inline math; fixed now that it is a
  named, tested function.
- Consolidate the two duplicated temp-file save blocks into one
  unconditional save (behavior unchanged: the EXIF-transposed image is
  still always persisted before WatermarkRemover reloads it by path), and
  drop the now-redundant `_tmp_path is not None` guard in finally.
- Bump version 0.5.3 -> 0.5.4 (pyproject, __init__, uv.lock); document the
  helper as the regression guard in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 14:09:33 -07:00
test-userandClaude Opus 4.7 28fe13db8f Document native-res MPS OOM -> CPU-fallback behavior in limitations
Concrete data point from the 2026-05-25 gpt-image SDXL run: native
1254x1254 fp32 OOMs at the UNet step (not just VAE) on a 20 GB MPS
ceiling, and img2img_runner auto-falls back to CPU and completes
(slow, weight-identical, still defeats SynthID). enable_vae_tiling()
alone does not prevent it. Fast Mac workarounds: fp16 on MPS or
--max-resolution; neither is the default.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:57:13 -07:00
test-userandClaude Opus 4.7 59d72c5db7 Record verified gpt-image-2 SynthID-cleaned chain in corpus
Add manifest row for the 4ef377bd -> f6dd47a5 chain: a gpt-image-2 sample
(openai.com/verify: SynthID + C2PA detected) cleaned via v0.5.3 `all` at
native 1254x1254 (prod-equivalent SDXL base, strength 0.05, 50 steps).
openai.com/verify reports SynthID NOT detected after the run, re-confirming
that the #10 native-resolution default defeats OpenAI SynthID and resolving
the #15 root cause (older SD-1.5/768px downscale default did not).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:55:10 -07:00
test-userandClaude Opus 4.7 e27f24f520 test(samples): commit real Doubao fixture + AIGC real-sample test
data/samples/doubao-1.png is the real #13 sample: carries the China TC260
<TC260:AIGC> XMP label and a visible '豆包AI生成' text mark (bottom-right).
Grounds the AIGC detection on a real file (alongside the synthetic tests)
and serves as the fixture for visible-watermark removal work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:37:15 -07:00
test-userandClaude Opus 4.7 1afc1e60ef test(samples): add real Doubao TC260 AIGC reference sample
2048x2048 PNG carrying China's TC260 <TC260:AIGC> label; identify reports
it as a China AIGC-labeled generator (TC260). Reference fixture for manual
re-verification of the TC260 detection path -- the automated tests use
synthetic blobs, so nothing depends on this file being present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:36:28 -07:00
test-userandClaude Opus 4.7 d45f0806a0 chore(release): v0.5.3 — detect China TC260 AIGC label (Doubao)
- feat(identify): detect the China TC260 <TC260:AIGC> XMP label (Doubao
  and other China-served generators); reports platform + ContentProducer.
  Removal already strips it via the existing metadata cleaner.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:30:40 -07:00
test-userandClaude Opus 4.7 c7f0d71f90 feat(identify): detect China TC260 AIGC label (Doubao et al.)
China-served generators embed an XMP <TC260:AIGC>{"Label":"1",...} block
(China's mandatory AI-content labeling, TC260 standard). Doubao (ByteDance)
uses it -- verified on the real #13 sample. It's none of C2PA / SynthID /
imwatermark / IPTC, so identify() previously returned unknown.

- metadata: AIGC_MARKERS + aigc_label() (json-decodes the HTML-entity-encoded
  block); has_ai_metadata + get_ai_metadata now surface it.
- identify: new 'aigc' signal -> is_ai True, platform 'China AIGC-labeled
  generator (TC260; e.g. Doubao)', carries the ContentProducer code.
- Container-agnostic raw-byte scan, so it covers the whole China-AIGC ecosystem
  (Jimeng/Kling/Qwen/Ernie share the standard).
- Tests: synthetic TC260 block (metadata + identify). Docs updated.

Addresses #13.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:29:51 -07:00
test-userandClaude Opus 4.7 768d997ef0 docs: scope SynthID provenance claims to source-verified facts
Threat model: replace the unverified deployment list (Gemini 3 Pro /
Nano Banana Pro / Imagen 4 / Veo) with the source-verified scope -- SynthID
across Imagen / Veo / Lyria plus Gemini app outputs (>10B items by Dec 2025),
and attribute the 136-bit payload to the paper's SynthID-O variant.

openai-images-2 sample: note the file predates the 19 May 2026 SynthID
rollout across ChatGPT / Codex / API, and that openai.com/verify is now the
public oracle (still no local decoder).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:18:13 -07:00
14 changed files with 241 additions and 38 deletions

No files matched your search

+4 -3
View File
@@ -14,7 +14,7 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
- `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto
- `maintain.sh` may not finish fully green (pre-existing, not per-change): strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.) To gate a change, run `uv run ruff check`, `uv run pyright <changed files>`, `uv run pytest` directly.
- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch.
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `mj-*` = Midjourney IPTC, `doubao-1.png` = ByteDance Doubao with the China TC260 `<TC260:AIGC>` XMP label **and** a visible "豆包AI生成" text mark bottom-right, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/` (`manifest.csv` tracked, `images/` gitignored); see its README for the collection protocol and verification oracles.
## Configuration
@@ -29,7 +29,7 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan.
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, IPTC "Made with AI", embedded SD/ComfyUI params, SynthID proxy, visible Gemini sparkle) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. Add platform mappings to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read.
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`.
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. The `detect` extra pulls **torch** transitively (invisible-watermark declares torch a hard dep, and `WatermarkDecoder` eagerly imports `rivaGan` -> `torch` at import time), so detection needs torch present even though dwtDct runs CPU-only on cv2/numpy/pywavelets — no GPU and no separate `gpu` extra required. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
- `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features
## Watermarking landscape (research 2026-05-24)
@@ -38,12 +38,13 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
- **Locally detectable (open decoder, no key/API):** Stable Diffusion / SDXL / FLUX via `imwatermark` DWT-DCT (now covered by `invisible_watermark.py`). FLUX uses the same library (`black-forest-labs/flux2` `src/flux2/watermark.py`, 48-bit `0b001010101111111010000111100111001111010100101110`); SDXL is the diffusers `WATERMARK_MESSAGE` (`0b101100111110110010010000011110111011000110011110`). Caveat: fragile to re-encoding.
- **C2PA / IPTC (covered by the issuer/marker scan):** OpenAI, Google, Adobe Firefly, Microsoft (Designer + **Bing Image Creator** — collected 2026-05-24; Bing now runs Microsoft's own **MAI-Image** model, signs C2PA as "Microsoft", NOT OpenAI/DALL-E), and **Stability AI** (collected from Brand Studio / DreamStudio successor; signs C2PA as "Stability AI Ltd", no SynthID, no imwatermark on its current Stable Image model — issuer added to `C2PA_ISSUERS`). Still unsampled: Canva (its downloads are re-encoded design *exports* that strip C2PA, so a Canva "positive" is inconclusive — skipped), Getty, Shutterstock. Midjourney embeds NO C2PA and no invisible watermark (our `mj-*` sample carried only the IPTC tag).
- **EXIF/XMP generator tag (caught by `exif_generator`):** **Ideogram** writes EXIF `Make="Ideogram AI"` (collected 2026-05-24 — no C2PA, no SynthID, no imwatermark; the Make tag is the only signal).
- **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `<TC260:AIGC>{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). **Doubao** (ByteDance) uses it (verified on the real #13 sample 2026-05-25; `ContentProducer` `001191110102MACQD9K64010000`, no C2PA/SynthID/imwatermark — the XMP block is the only signal; GitHub attachment upload did NOT strip it). The same standard is mandatory for Jimeng/Kling/Qwen/Ernie etc., so the one marker covers the whole China-AIGC-labeled ecosystem. `aigc_label` json-decodes the (HTML-entity-encoded) block; container-agnostic raw-byte scan.
- **No detectable signal on download (correctly reported `unknown`):** **Recraft** (PNG export is a re-encoded design export — strips everything), **Krea hosting FLUX 2** (no imwatermark despite FLUX — the host omits the encoder, same as Stability's hosted SDXL), and Midjourney (embeds nothing). Lesson: the imwatermark detector only fires on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL), not from re-hosts (Krea/Stability) or re-encoded exports (Recraft/Canva).
- **Invisible but NOT locally detectable (proprietary, API/oracle only — same wall as SynthID):** Amazon Titan Image Generator + Nova Canvas (Bedrock `DetectGeneratedContent` API), Kakao (new SynthID image adopter, May 2026), NVIDIA Cosmos (SynthID video). No local detector possible; treat like SynthID.
## Known limitations
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix.
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data point (verified 2026-05-25 on a 1254x1254 gpt-image SDXL run, fp32, 20 GB MPS ceiling):** native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. Adding `enable_vae_tiling()` alone does NOT prevent it (the peak is the UNet, not the VAE). The fast Mac workarounds are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files.
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are read for detection but not yet **scrubbed** on removal.
+3 -2
View File
@@ -38,12 +38,13 @@ Strips SynthID, C2PA Content Credentials, EXIF/XMP "Made with AI" labels, and vi
| **Microsoft Designer / Bing Image Creator** | — | ✅ SynthID via DALL-E backend (Designer) | ✅ C2PA (Bing runs MAI-Image, signed "Microsoft") | Metadata strip |
| **Midjourney** | — | — | ✅ EXIF + XMP (prompt, model, seed) | Metadata strip |
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
| **Doubao** (ByteDance) / China AIGC generators | — | — | ✅ TC260 `<TC260:AIGC>` XMP label (China's mandatory AI labeling) | Metadata strip |
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
| **TreeRing** | — | ✅ Latent space watermark | — | Diffusion regeneration |
> Visible watermarks (logo overlays) are currently used only by Google Gemini / Nano Banana. Other services rely on invisible watermarks and/or metadata. Our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, IPTC "Made with AI", embedded generation params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` extra) the open SD/SDXL/FLUX invisible watermark. The SynthID *pixel* watermark has no local decoder, so it is reported as a metadata proxy only.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, IPTC "Made with AI", the China TC260 AIGC label, embedded generation params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` extra) the open SD/SDXL/FLUX invisible watermark. The SynthID *pixel* watermark has no local decoder, so it is reported as a metadata proxy only.
## How it works
@@ -312,7 +313,7 @@ Watermarking and provenance for AI-generated content is now regulated in several
This tool defends already-distributed AI imagery against automatic detection systems (social-platform "Made with AI" labels, third-party classifiers, content-policy filters). It does **not** retroactively anonymise generation.
In particular, **SynthID-Image** (Google, deployed 2025 with Gemini 3 Pro / Nano Banana Pro / Imagen 4 / Veo) carries a **multi-bit payload** — the paper's SynthID-O variant encodes 136-bit payloads in 512x512 images ([arxiv 2510.09263](https://arxiv.org/abs/2510.09263)). The payload is believed to encode a user / session identifier. If the original watermarked file ever passed through a system controlled by the prompt originator (a saved Gemini account history, a screenshot uploaded to a Google product, a backup), Google retains the ability to link that original to the generating account. Stripping the watermark from a copy you possess does not erase Google's server-side record.
In particular, **SynthID** (Google DeepMind) is embedded across Google's generative media stack — Imagen (images), Veo (video), Lyria (audio) — and Gemini app image outputs (Nano Banana / Gemini 3 Pro, which we verified positive via the Gemini app's SynthID oracle); Google reported over 10 billion items watermarked by December 2025. It carries a **multi-bit payload** — the research paper's SynthID-O variant encodes 136-bit payloads in 512x512 images ([arxiv 2510.09263](https://arxiv.org/abs/2510.09263)). The payload is believed to encode a user / session identifier. If the original watermarked file ever passed through a system controlled by the prompt originator (a saved Gemini account history, a screenshot uploaded to a Google product, a backup), Google retains the ability to link that original to the generating account. Stripping the watermark from a copy you possess does not erase Google's server-side record.
Use cases where the threat model fits:
- You generated the image yourself, want to publish it as your own work, and accept the consequences if Google ever publishes their detector logs.
Binary file not shown.

After

Width:  |  Height:  |  Size: 5.1 MiB

+1 -1
View File
@@ -26,7 +26,7 @@ The `trainedAlgorithmicMedia` tag is what triggers "Made with AI" labels on Inst
### Invisible pixel-level watermark
OpenAI's system card for Images 2.0 states the model embeds an "imperceptible, robust, and content-specific" pixel-level watermark alongside C2PA. No public detector exists, so bypass cannot be verified empirically for this sample.
OpenAI's system card for Images 2.0 states the model embeds an "imperceptible, robust, and content-specific" pixel-level watermark alongside C2PA. This sample was downloaded 2026-04-22, before OpenAI's 19 May 2026 rollout of Google's SynthID watermark across ChatGPT / Codex / the API, so it likely predates SynthID and carries only the original content-specific watermark. Since that rollout, the openai.com/verify tool (in preview) is the public oracle for both signals; there is still no local decoder, so bypass cannot be verified empirically here without the oracle.
## Reproducing the removal
+1
View File
@@ -116,3 +116,4 @@ f9bb6b1039ee228221d2eead7daebd46422a36a4a8959a74fa0198e1aa19f9b8,f9bb6b10-a_red_
ca2162624607b8f030c63cb1371d6afdd3f0d7af3d9b17514f7ae64a2e8f378c,ca216262-0001_2_a-red-ceramic-coffee-mug-sits-on-weather_rs0Lt1beXV-Mwjs33vWKVw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
8a90f2ddf83ceedd7944531d27248191c79dc321ec76ea1f36df6cff9f6b8916,8a90f2dd-0001_3_a-red-ceramic-coffee-mug-sits-on-weather_Lm96lBwcXsmdeCfCpm52vw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
87fca198d5c436b27a16685972802a2ff0a310a941248c9ec8d81560d7f387a1,87fca198-0001_4_a-red-ceramic-coffee-mug-sits-on-weather_pGmlgcH4XqSaq41hc0a9AQ_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
f6dd47a5ffd319aea21bf10dcf9877097666420b02c2620080bac12b03976e7e,f6dd47a5-4ef377bd-gpt-image-2-cleaned.png,cleaned,"our pipeline (invisible/SDXL, native-res default)",stabilityai/stable-diffusion-xl-base-1.0,1254,1254,png,,,openai-verify,2026-05-25T20:50:38Z,"cleaned from 4ef377bd via v0.5.3 'all' at native 1254x1254 (prod-equivalent); openai.com/verify: SynthID NOT detected. Re-confirms #10 native-res default defeats OpenAI SynthID (closes #15 root cause). Note: native res OOMs on 20GB MPS, auto-fell back to CPU."
1 sha256 filename label source model width height format c2pa_issuer synthid_metadata verified_via added notes
116 ca2162624607b8f030c63cb1371d6afdd3f0d7af3d9b17514f7ae64a2e8f378c ca216262-0001_2_a-red-ceramic-coffee-mug-sits-on-weather_rs0Lt1beXV-Mwjs33vWKVw_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
117 8a90f2ddf83ceedd7944531d27248191c79dc321ec76ea1f36df6cff9f6b8916 8a90f2dd-0001_3_a-red-ceramic-coffee-mug-sits-on-weather_Lm96lBwcXsmdeCfCpm52vw_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
118 87fca198d5c436b27a16685972802a2ff0a310a941248c9ec8d81560d7f387a1 87fca198-0001_4_a-red-ceramic-coffee-mug-sits-on-weather_pGmlgcH4XqSaq41hc0a9AQ_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
119 f6dd47a5ffd319aea21bf10dcf9877097666420b02c2620080bac12b03976e7e f6dd47a5-4ef377bd-gpt-image-2-cleaned.png cleaned our pipeline (invisible/SDXL, native-res default) stabilityai/stable-diffusion-xl-base-1.0 1254 1254 png openai-verify 2026-05-25T20:50:38Z cleaned from 4ef377bd via v0.5.3 'all' at native 1254x1254 (prod-equivalent); openai.com/verify: SynthID NOT detected. Re-confirms #10 native-res default defeats OpenAI SynthID (closes #15 root cause). Note: native res OOMs on 20GB MPS, auto-fell back to CPU.
+6 -3
View File
@@ -1,6 +1,6 @@
[project]
name = "remove-ai-watermarks"
version = "0.5.2"
version = "0.5.4"
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
readme = "README.md"
requires-python = ">=3.10"
@@ -28,8 +28,11 @@ gpu = [
]
# Open invisible-watermark (imwatermark) decoder for detecting the DWT-DCT
# watermarks embedded by Stable Diffusion / SDXL / FLUX. Optional because it
# pulls non-headless opencv; identify() guards the import and skips the signal
# when absent.
# pulls non-headless opencv AND torch (invisible-watermark declares torch a hard
# dependency, and WatermarkDecoder eagerly imports rivaGan -> torch at import
# time, so the dwtDct-only detect path still needs torch present even though it
# never runs on GPU). So `detect` alone pulls torch -- no need to add `gpu` for
# detection. identify() guards the import and skips the signal when absent.
detect = [
"invisible-watermark>=0.2.0",
]
+1 -1
View File
@@ -1,3 +1,3 @@
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
__version__ = "0.5.2"
__version__ = "0.5.4"
+14 -1
View File
@@ -24,8 +24,10 @@ from typing import TYPE_CHECKING
from remove_ai_watermarks.metadata import (
AI_METADATA_KEYS,
AIGC_MARKERS,
C2PA_UUID,
IPTC_AI_MARKERS,
aigc_label,
exif_generator,
get_ai_metadata,
)
@@ -219,6 +221,15 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
if platform is None:
platform = "Made-with-AI tag (e.g. Meta AI); platform not specified"
# ── China TC260 AIGC label (Doubao and other China-served gens) ──
aigc = any(m in head for m in AIGC_MARKERS)
if aigc:
producer = (aigc_label(image_path) or {}).get("ContentProducer", "")
signals.append(Signal("aigc", f"TC260 AIGC label{f' (producer {producer})' if producer else ''}", "high"))
watermarks.append("China AIGC label (TC260 standard)")
if platform is None:
platform = "China AIGC-labeled generator (TC260; e.g. Doubao)"
# ── Local diffusion parameters (Stable Diffusion / ComfyUI) ──────
local_keys = sorted(k for k in meta if k.lower() in _LOCAL_GEN_KEYS)
if local_keys:
@@ -247,7 +258,9 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
# ── Verdict so far (metadata + embedded watermark) ──────────────
invisible_wm = any(s.name == "invisible_watermark" for s in signals)
exif_gen = any(s.name == "exif_generator" for s in signals)
ai_from_metadata = bool((has_c2pa and (c2pa_is_ai or synthid)) or iptc or local_keys or invisible_wm or exif_gen)
ai_from_metadata = bool(
(has_c2pa and (c2pa_is_ai or synthid)) or iptc or aigc or local_keys or invisible_wm or exif_gen
)
# ── Visible Gemini sparkle (fallback for stripped-metadata case) ─
if check_visible and (conf := _visible_sparkle(image_path)) is not None and conf >= _SPARKLE_THRESHOLD:
+33 -25
View File
@@ -42,6 +42,24 @@ def is_available() -> bool:
return False
def _target_size(width: int, height: int, max_resolution: int) -> tuple[int, int] | None:
"""Compute the downscaled (width, height) for a long-side cap, or None for native.
Returns None when no pre-downscale is needed: ``max_resolution <= 0`` (native
resolution, the default that matches the raiw.cc backend -- see issue #10) or
the long side already fits the cap. Otherwise scales the long side down to
``max_resolution`` preserving aspect ratio (integer-truncated, matching the
PIL ``resize`` call site). Pure function so the native-vs-downscale decision
is unit-testable without loading the diffusion model.
"""
if max_resolution > 0 and max(width, height) > max_resolution:
ratio = max_resolution / max(width, height)
# Clamp the short side to >=1: extreme aspect ratios (e.g. 5000x3 capped
# at 1024) would otherwise truncate it to 0 and crash image.resize().
return (max(1, int(width * ratio)), max(1, int(height * ratio)))
return None
class InvisibleEngine:
"""Remove invisible AI watermarks using diffusion model regeneration.
@@ -142,37 +160,26 @@ class InvisibleEngine:
image = Image.open(image_path)
image = ImageOps.exif_transpose(image)
orig_size = image.size # (width, height)
_tmp_path = None
if max_resolution > 0 and max(image.width, image.height) > max_resolution:
ratio = max_resolution / max(image.width, image.height)
new_size = (int(image.width * ratio), int(image.height * ratio))
# Optional long-side downscale; native resolution by default (issue #10).
target = _target_size(image.width, image.height, max_resolution)
if target is not None:
if self._progress_callback:
self._progress_callback(
f"Downscaling {image.width}x{image.height} "
f"to {new_size[0]}x{new_size[1]} "
f"to {target[0]}x{target[1]} "
f"(max-resolution cap {max_resolution}px)..."
)
image = image.resize(new_size, Image.Resampling.LANCZOS)
image = image.resize(target, Image.Resampling.LANCZOS)
# Save to a temp file instead of overwriting the original
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
import os as _os
_os.close(_tmp_fd)
image_path = _tmp_path
else:
# We must save the transposed image back to a tmp file if it was rotated
# otherwise WatermarkRemover will reload it without EXIF rotation!
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
import os as _os
_os.close(_tmp_fd)
image_path = _tmp_path
# Always persist to a temp file, even without downscaling: WatermarkRemover
# reloads by path, so the EXIF-transposed pixels must be saved or rotation
# is lost. Cleaned up in the finally block via _tmp_path.
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
os.close(_tmp_fd)
image_path = _tmp_path
try:
# Optional: Face protection (Phase 1 - Extraction)
@@ -253,7 +260,8 @@ class InvisibleEngine:
return out_path
finally:
if _tmp_path is not None and _tmp_path.exists():
# _tmp_path is always set above (we persist the image unconditionally).
if _tmp_path.exists():
_tmp_path.unlink()
def remove_watermark_batch(
+42
View File
@@ -73,6 +73,17 @@ IPTC_AI_MARKERS: tuple[bytes, ...] = (
b"compositeWithTrainedAlgorithmicMedia",
)
# China's mandatory AI-content labeling (TC260, the national cybersecurity
# standards committee). AI generators serving China embed an XMP block in the
# TC260 namespace -- ``<TC260:AIGC>{"Label":"1",...}``. Doubao (ByteDance) uses
# this; the same standard is mandatory for Jimeng, Kling, Qwen, Ernie, etc.,
# so the marker covers the whole China-AIGC-labeled ecosystem. Container-
# agnostic (XMP is text), so a raw-byte scan catches it in PNG/JPEG/etc.
AIGC_MARKERS: tuple[bytes, ...] = (
b"tc260.org.cn/ns/AIGC",
b"TC260:AIGC",
)
STANDARD_METADATA_KEYS: frozenset[str] = frozenset(
[
"Author",
@@ -139,9 +150,35 @@ def has_ai_metadata(image_path: Path) -> bool:
return True
if C2PA_UUID in data:
return True
if any(marker in data for marker in AIGC_MARKERS):
return True
return any(marker in data for marker in IPTC_AI_MARKERS)
def aigc_label(image_path: Path) -> dict[str, str] | None:
"""Parse a China TC260 ``<TC260:AIGC>`` AI-labeling block, if present.
Returns the decoded JSON (e.g. ``{"Label": "1", "ContentProducer": ...}``)
or None. The block is XMP text (HTML-entity encoded), so it is found by a
container-agnostic raw-byte scan and works for PNG/JPEG/WebP alike.
"""
import html
import json
import re
with open(image_path, "rb") as f:
data = f.read(1024 * 1024)
match = re.search(rb"<TC260:AIGC>(.*?)</TC260:AIGC>", data, re.DOTALL)
if not match:
return None
raw = html.unescape(match.group(1).decode("utf-8", "replace"))
try:
parsed = json.loads(raw)
except ValueError:
return None
return {str(k): str(v) for k, v in parsed.items()} if isinstance(parsed, dict) else None
def synthid_source(image_path: Path) -> str | None:
"""Return the vendor name(s) if the image carries a SynthID pixel watermark.
@@ -286,6 +323,11 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
# fall back to the format-agnostic source check for the SynthID verdict.
if "synthid_watermark" not in result and (vendor := synthid_source(image_path)):
result.setdefault("synthid_watermark", synthid_verdict(vendor))
# China TC260 AI-content label (Doubao and other China-served generators).
if aigc := aigc_label(image_path):
producer = aigc.get("ContentProducer", "")
result["aigc_label"] = f"China AIGC label (TC260){f'; producer {producer}' if producer else ''}"
return result
+32
View File
@@ -317,3 +317,35 @@ class TestIdentifyInvisibleWatermark:
def test_check_invisible_false_skips(self, tmp_path: Path):
r = identify(self._sdxl_watermarked(tmp_path), check_visible=False, check_invisible=False)
assert not any(s.name == "invisible_watermark" for s in r.signals)
class TestIdentifyAIGC:
"""China TC260 AIGC label is detected and attributed (e.g. Doubao)."""
def _aigc_png(self, tmp_path: Path) -> Path:
from PIL import Image
p = tmp_path / "doubao.png"
Image.new("RGB", (32, 32)).save(p)
xmp = (
'<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF '
'xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">'
'<rdf:Description xmlns:TC260="http://www.tc260.org.cn/ns/AIGC/1.0/">'
"<TC260:AIGC>{&quot;Label&quot;:&quot;1&quot;,&quot;ContentProducer&quot;:&quot;BYTEDANCE001&quot;}"
"</TC260:AIGC></rdf:Description></rdf:RDF></x:xmpmeta>"
)
with open(p, "ab") as f:
f.write(xmp.encode())
return p
def test_aigc_detected(self, tmp_path: Path):
r = identify(self._aigc_png(tmp_path), check_visible=False)
assert r.is_ai_generated is True
assert r.platform is not None
assert "AIGC" in r.platform or "TC260" in r.platform
assert any("AIGC" in w for w in r.watermarks)
def test_aigc_signal_carries_producer(self, tmp_path: Path):
r = identify(self._aigc_png(tmp_path), check_visible=False)
sig = next(s for s in r.signals if s.name == "aigc")
assert "BYTEDANCE001" in sig.detail
+43 -1
View File
@@ -2,7 +2,7 @@
from __future__ import annotations
from remove_ai_watermarks.invisible_engine import InvisibleEngine, is_available
from remove_ai_watermarks.invisible_engine import InvisibleEngine, _target_size, is_available
class TestIsAvailable:
@@ -26,3 +26,45 @@ class TestInvisibleEngineInit:
def test_ctrlregen_model_id(self):
assert InvisibleEngine.CTRLREGEN_MODEL_ID == "yepengliu/ctrlregen"
class TestTargetSize:
"""Regression guard for the native-resolution decision (issues #10 / #15).
max_resolution=0 must NOT downscale -- the forced downscale->upscale
round-trip was the quality loss in #10, and downscaling at all let SynthID
survive in #15 (the native SDXL pass at strength ~0.05 is what defeats it).
"""
def test_native_default_no_downscale(self):
# The default (0) means native resolution: no resize, regardless of size.
assert _target_size(4096, 4096, 0) is None
assert _target_size(123, 456, 0) is None
def test_negative_cap_treated_as_native(self):
assert _target_size(4096, 4096, -1) is None
def test_cap_below_long_side_downscales(self):
# 2000x1000, cap 1024 -> long side scaled to 1024, aspect preserved.
assert _target_size(2000, 1000, 1024) == (1024, 512)
def test_cap_uses_long_side_for_portrait(self):
# Portrait: height is the long side, so it drives the ratio.
assert _target_size(1000, 2000, 1024) == (512, 1024)
def test_cap_at_or_above_long_side_no_downscale(self):
# Already within the cap (and exactly equal) -> no resize.
assert _target_size(800, 600, 1024) is None
assert _target_size(1024, 768, 1024) is None
def test_integer_truncation_matches_pil_call_site(self):
# 1254x1254 (the gpt-image sample) capped at 1000: int(1254*1000/1254)=1000.
assert _target_size(1254, 1254, 1000) == (1000, 1000)
# Non-divisible ratio truncates toward zero like int() at the call site.
assert _target_size(1000, 333, 500) == (500, 166)
def test_extreme_aspect_ratio_clamps_short_side_to_one(self):
# 5000x3 capped at 1024: int(3 * 1024/5000) = 0 would crash resize();
# the short side must clamp to 1, never 0.
assert _target_size(5000, 3, 1024) == (1024, 1)
assert _target_size(3, 5000, 1024) == (1, 1024)
+60
View File
@@ -390,3 +390,63 @@ class TestExifGenerator:
def test_clean_image_is_none(self, tmp_clean_png: Path):
assert exif_generator(tmp_clean_png) is None
class TestAIGCLabel:
"""China TC260 AIGC labeling (Doubao and other China-served generators)."""
def _aigc_png(self, tmp_path: Path, label: str = "1", producer: str = "TESTPRODUCER001") -> Path:
from remove_ai_watermarks.metadata import aigc_label # noqa: F401 (import-time guard)
p = tmp_path / "doubao.png"
Image.new("RGB", (32, 32)).save(p)
# XMP is HTML-entity encoded in real files; aigc_label must unescape it.
xmp = (
'<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF '
'xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">'
'<rdf:Description rdf:about="" '
'xmlns:TC260="http://www.tc260.org.cn/ns/AIGC/1.0/"><TC260:AIGC>'
f"{{&quot;Label&quot;:&quot;{label}&quot;,&quot;ContentProducer&quot;:&quot;{producer}&quot;}}"
"</TC260:AIGC></rdf:Description></rdf:RDF></x:xmpmeta>"
)
with open(p, "ab") as f:
f.write(xmp.encode())
return p
def test_parses_label_and_producer(self, tmp_path: Path):
from remove_ai_watermarks.metadata import aigc_label
info = aigc_label(self._aigc_png(tmp_path))
assert info is not None
assert info["Label"] == "1"
assert info["ContentProducer"] == "TESTPRODUCER001"
def test_none_when_absent(self, tmp_clean_png):
from remove_ai_watermarks.metadata import aigc_label
assert aigc_label(tmp_clean_png) is None
def test_has_ai_metadata_detects_aigc(self, tmp_path: Path):
assert has_ai_metadata(self._aigc_png(tmp_path))
def test_get_ai_metadata_surfaces_aigc(self, tmp_path: Path):
meta = get_ai_metadata(self._aigc_png(tmp_path))
assert "aigc_label" in meta
assert "TC260" in meta["aigc_label"]
@pytest.mark.skipif(not (SAMPLES_DIR / "doubao-1.png").exists(), reason="doubao sample not present")
class TestAIGCRealSample:
"""Real Doubao (ByteDance) sample carries the China TC260 AIGC XMP label."""
def test_doubao_aigc_label(self):
from remove_ai_watermarks.metadata import aigc_label
info = aigc_label(SAMPLES_DIR / "doubao-1.png")
assert info is not None
assert info["Label"] == "1"
assert info["ContentProducer"] # ByteDance producer code present
def test_doubao_detected_as_ai(self):
assert has_ai_metadata(SAMPLES_DIR / "doubao-1.png")
assert "aigc_label" in get_ai_metadata(SAMPLES_DIR / "doubao-1.png")
Generated
+1 -1
View File
@@ -2150,7 +2150,7 @@ wheels = [
[[package]]
name = "remove-ai-watermarks"
version = "0.5.2"
version = "0.5.4"
source = { editable = "." }
dependencies = [
{ name = "click" },