mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-15 14:00:54 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
d24d8a4b14 | ||
|
|
28fe13db8f | ||
|
|
59d72c5db7 | ||
|
|
e27f24f520 | ||
|
|
1afc1e60ef | ||
|
|
d45f0806a0 | ||
|
|
c7f0d71f90 | ||
|
|
768d997ef0 | ||
|
|
37769453a9 | ||
|
|
f3ecea348a | ||
|
|
fb42295b3a | ||
|
|
18740969ae | ||
|
|
93c664f7fb | ||
|
|
20a754ef98 | ||
|
|
e60f183d29 | ||
|
|
d091b9f822 | ||
|
|
e8d698814a | ||
|
|
663c8c64ca | ||
|
|
b45e2a5731 |
No files matched your search
@@ -12,9 +12,9 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
## Test and lint
|
||||
|
||||
- `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto
|
||||
- `maintain.sh` does not currently finish green (pre-existing, not per-change): `uv-secure` aborts on a fixable transitive `idna` vuln, and strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). To gate a change, run `uv run ruff check`, `uv run pyright <changed files>`, `uv run pytest` directly.
|
||||
- `maintain.sh` may not finish fully green (pre-existing, not per-change): strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.) To gate a change, run `uv run ruff check`, `uv run pyright <changed files>`, `uv run pytest` directly.
|
||||
- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch.
|
||||
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
|
||||
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `mj-*` = Midjourney IPTC, `doubao-1.png` = ByteDance Doubao with the China TC260 `<TC260:AIGC>` XMP label **and** a visible "豆包AI生成" text mark bottom-right, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
|
||||
- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/` (`manifest.csv` tracked, `images/` gitignored); see its README for the collection protocol and verification oracles.
|
||||
|
||||
## Configuration
|
||||
@@ -29,7 +29,7 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan.
|
||||
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, IPTC "Made with AI", embedded SD/ComfyUI params, SynthID proxy, visible Gemini sparkle) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. Add platform mappings to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read.
|
||||
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`.
|
||||
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
|
||||
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. The `detect` extra pulls **torch** transitively (invisible-watermark declares torch a hard dep, and `WatermarkDecoder` eagerly imports `rivaGan` -> `torch` at import time), so detection needs torch present even though dwtDct runs CPU-only on cv2/numpy/pywavelets — no GPU and no separate `gpu` extra required. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
|
||||
- `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features
|
||||
|
||||
## Watermarking landscape (research 2026-05-24)
|
||||
@@ -38,15 +38,16 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
|
||||
- **Locally detectable (open decoder, no key/API):** Stable Diffusion / SDXL / FLUX via `imwatermark` DWT-DCT (now covered by `invisible_watermark.py`). FLUX uses the same library (`black-forest-labs/flux2` `src/flux2/watermark.py`, 48-bit `0b001010101111111010000111100111001111010100101110`); SDXL is the diffusers `WATERMARK_MESSAGE` (`0b101100111110110010010000011110111011000110011110`). Caveat: fragile to re-encoding.
|
||||
- **C2PA / IPTC (covered by the issuer/marker scan):** OpenAI, Google, Adobe Firefly, Microsoft (Designer + **Bing Image Creator** — collected 2026-05-24; Bing now runs Microsoft's own **MAI-Image** model, signs C2PA as "Microsoft", NOT OpenAI/DALL-E), and **Stability AI** (collected from Brand Studio / DreamStudio successor; signs C2PA as "Stability AI Ltd", no SynthID, no imwatermark on its current Stable Image model — issuer added to `C2PA_ISSUERS`). Still unsampled: Canva (its downloads are re-encoded design *exports* that strip C2PA, so a Canva "positive" is inconclusive — skipped), Getty, Shutterstock. Midjourney embeds NO C2PA and no invisible watermark (our `mj-*` sample carried only the IPTC tag).
|
||||
- **EXIF/XMP generator tag (caught by `exif_generator`):** **Ideogram** writes EXIF `Make="Ideogram AI"` (collected 2026-05-24 — no C2PA, no SynthID, no imwatermark; the Make tag is the only signal).
|
||||
- **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `<TC260:AIGC>{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). **Doubao** (ByteDance) uses it (verified on the real #13 sample 2026-05-25; `ContentProducer` `001191110102MACQD9K64010000`, no C2PA/SynthID/imwatermark — the XMP block is the only signal; GitHub attachment upload did NOT strip it). The same standard is mandatory for Jimeng/Kling/Qwen/Ernie etc., so the one marker covers the whole China-AIGC-labeled ecosystem. `aigc_label` json-decodes the (HTML-entity-encoded) block; container-agnostic raw-byte scan.
|
||||
- **No detectable signal on download (correctly reported `unknown`):** **Recraft** (PNG export is a re-encoded design export — strips everything), **Krea hosting FLUX 2** (no imwatermark despite FLUX — the host omits the encoder, same as Stability's hosted SDXL), and Midjourney (embeds nothing). Lesson: the imwatermark detector only fires on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL), not from re-hosts (Krea/Stability) or re-encoded exports (Recraft/Canva).
|
||||
- **Invisible but NOT locally detectable (proprietary, API/oracle only — same wall as SynthID):** Amazon Titan Image Generator + Nova Canvas (Bedrock `DetectGeneratedContent` API), Kakao (new SynthID image adopter, May 2026), NVIDIA Cosmos (SynthID video). No local detector possible; treat like SynthID.
|
||||
|
||||
## Known limitations
|
||||
|
||||
- `invisible` pipeline downscales to model-native resolution (1024 px for SDXL) before diffusion. Degrades fine text in infographics. Tracked; fix is tile-based diffusion.
|
||||
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data point (verified 2026-05-25 on a 1254x1254 gpt-image SDXL run, fp32, 20 GB MPS ceiling):** native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. Adding `enable_vae_tiling()` alone does NOT prevent it (the peak is the UNet, not the VAE). The fast Mac workarounds are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
|
||||
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files.
|
||||
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
|
||||
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are read for detection but not yet **scrubbed** on removal.
|
||||
- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. **Surface-dependent blind spot (verified 2026-05-24):** the same Google model emits different metadata per surface -- the Gemini *app* wraps outputs in Google C2PA, but the *API/playground* (AI Studio, Nano Banana / gemini-2.5-flash-image) emits the SynthID *pixel* watermark (confirmed via the Gemini-app oracle) + the visible sparkle but **no C2PA/IPTC at all**, so `synthid_source` returns None despite SynthID being present. Only the pixel oracle or the visible-sparkle detector catches those. (Meta AI is another surface mismatch: it writes the IPTC `digitalSourceType=trainedAlgorithmicMedia` marker, not C2PA and not SynthID.) Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. A from-scratch gpt-image pilot (2026-05-24) confirmed this independently: 5 independent solid-black gpt-image outputs share a near-identical fixed signature (pairwise residual correlation **0.92**, avg-template retains 97% energy), so the watermark/carrier IS strongly present and consistent on flat content — but the carrier frequencies extracted from it do NOT discriminate real content (carrier-to-random ratio: cleaned 1.86 > watermarked 1.53; a non-gpt-image image scored highest at 3.67). The signature drowns in content texture. Net: a perfectly consistent solid-color signature still yields no real-content pixel detector with magnitude/carrier methods. A corpus discrimination test (2026-05-24, `scripts/synthid_pixel_probe.py`, raw zero-mean residual NCC) independently re-confirms this: at matched resolution, SynthID positives do NOT cluster apart from negatives (within-Gemini 0.07; at 1024 px pos-vs-neg >= pos-vs-pos). The only high correlations were near-duplicate *content* (5 ChatGPT renders of one prompt at ~0.92, while a distinct ChatGPT image scored ~0 against them) — content, not a carrier. The probe is solid-fills-only and EXPERIMENTAL/DIAGNOSTIC; do not use it on real content. **Correction (deeper re-examination 2026-05-25):** the carrier IS real on solid fills — the earlier "no carrier" was a *method* artifact of using spatial / FFT-magnitude NCC, which can't see it. The carrier is a fixed *phase* at specific low frequencies, so the right metric is **per-bin phase coherence**. On 8 white `gemini-2.5-flash-image` fills (generated via the reverse-SynthID trick: identity-edit prompt "Recreate this image exactly as it is" on a synthetic pure-white PNG — this bypasses the recitation block that rejects text prompts for pure colors), phase coherence at the white carriers `(0,±7..±12,±20..±23)` = **0.86** vs **0.31** random; single-image leave-one-out phase-match **+0.83** vs real photos **-0.24**. (Black `2.5-flash` fills clip to std≈0 — SynthID can't push values below 0, so no carrier in black; the repo's dark carriers come from nano-banana-pro.) **But it does not generalize:** (a) carriers are model-version + resolution + color specific — the repo's v4 codebook (built for `gemini-3.1-flash-image-preview` + `nano-banana-pro-preview`) scores ~0.527 on my 2.5-flash white fills, indistinguishable from negatives (~0.50), i.e. carriers shift across model versions and need a per-model codebook; (b) on real content (30 `2.5-flash` images) the carrier collapses — set phase coherence at carriers 0.37 ≈ random 0.42, and the repo's v4 detector gives content 0.518 ≈ negatives 0.504 (no separation; a faint +0.24 single-image lean is likely a brightness confound). Net: the spectral/phase approach is a real *controlled-fill* characterizer, NOT an arbitrary-real-content detector, and is brittle to model version. Metadata proxy + visible sparkle + online oracles remain the ceiling for real content.
|
||||
- **External AI-vs-real classifier models are out of scope (decided 2026-05-24).** Generic HuggingFace detectors (`Organika/sdxl-detector` Swin Transformer, `umm-maybe/AI-image-detector`, and fine-tunes) exist and report ~0.98 on their *own* SDXL-vs-real validation sets, but they are per-generator and the model cards themselves note degraded accuracy off-distribution; they are untested on gpt-image / Gemini Nano Banana (the metadata-stripped surfaces we care about), and our own light SDXL pass would likely defeat them the same way it defeats SynthID. Detection here stays local + signal-based (metadata + visible sparkle); do not add a bundled classifier dependency.
|
||||
- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". Also confirmed against **OpenAI's** SynthID (2026-05-23): a fresh ChatGPT/gpt-image output read "SynthID detected" on openai.com/verify before the local SDXL run and "SynthID not detected" after (corpus regression chain: pos `4ef377bd` -> cleaned `47188e88`). The same configuration is used in raiw-app production (`fal-ai/fast-sdxl` at native ~1024 px, strength 0.05, steps 50). SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain.
|
||||
- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". Also confirmed against **OpenAI's** SynthID (2026-05-23): a fresh ChatGPT/gpt-image output read "SynthID detected" on openai.com/verify before the local SDXL run and "SynthID not detected" after (corpus regression chain: pos `4ef377bd` -> cleaned `47188e88`). The same configuration is used in raiw-app production (`fal-ai/fast-sdxl/image-to-image`, strength 0.05, steps 50, guidance 7.5, no pre-downscale). fal's own `llms.txt` for `fast-sdxl` names the base checkpoint as `stabilityai/stable-diffusion-xl-base-1.0` (verified 2026-05-25) -- the exact checkpoint the local CLI defaults to (`DEFAULT_MODEL_ID`). So the local `invisible` default is weight-for-weight identical to prod; "fast-sdxl" is fal's optimized serving, not different weights. After the native-resolution fix the local pipeline matches prod on weights + strength + steps + guidance + resolution. SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain.
|
||||
@@ -14,6 +14,7 @@ Strips SynthID, C2PA Content Credentials, EXIF/XMP "Made with AI" labels, and vi
|
||||
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
|
||||
- **Batch processing** — process entire directories
|
||||
- **Detection** — three-stage NCC watermark detection with confidence scoring
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, IPTC "Made with AI", embedded SD/ComfyUI params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and the open SD/SDXL/FLUX invisible watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
|
||||
> **Try it online** — don't want to install anything? Use [raiw.cc](https://raiw.cc), a free web service powered by this library.
|
||||
|
||||
@@ -27,17 +28,24 @@ Strips SynthID, C2PA Content Credentials, EXIF/XMP "Made with AI" labels, and vi
|
||||
|
||||
| AI model | Visible watermark | Invisible watermark | Metadata | Our approach |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Google Gemini / Nano Banana / Gemini 3 Pro** | ✅ Sparkle logo | ✅ SynthID v1 + v2 (default SDXL pipeline at native ~1024 px) | ✅ C2PA + EXIF | Alpha reversal + diffusion + metadata strip |
|
||||
| **Google Gemini / Nano Banana / Gemini 3 Pro** | ✅ Sparkle logo | ✅ SynthID v1 + v2 (default SDXL pipeline, native resolution) | ✅ C2PA + EXIF | Alpha reversal + diffusion + metadata strip |
|
||||
| **OpenAI DALL-E 3 / ChatGPT** | — | — | ✅ C2PA manifest | Metadata strip |
|
||||
| **OpenAI ChatGPT Images 2.0** (gpt-image-2) | — | ⚠️ imperceptible pixel watermark (no public detector yet) | ✅ C2PA manifest (verified) | Diffusion regeneration + metadata strip |
|
||||
| **Stable Diffusion (AUTOMATIC1111, ComfyUI)** | — | ✅ DWT / steganographic | ✅ PNG text chunks | Diffusion regeneration + metadata strip |
|
||||
| **OpenAI ChatGPT Images 2.0** (gpt-image-2) | — | ✅ SynthID + content-specific pixel watermark (since May 2026; no local decoder, openai.com/verify oracle) | ✅ C2PA manifest (verified) | Diffusion regeneration + metadata strip |
|
||||
| **Stable Diffusion / SDXL (AUTOMATIC1111, ComfyUI)** | — | ✅ DWT-DCT (imwatermark — locally detectable) | ✅ PNG text chunks | Diffusion regeneration + metadata strip |
|
||||
| **Black Forest Labs FLUX** | — | ✅ DWT-DCT (imwatermark — locally detectable) | ✅ C2PA (FLUX.2 Pro) | Diffusion regeneration + metadata strip |
|
||||
| **Adobe Firefly** | — | — | ✅ Content Credentials (C2PA) | Metadata strip |
|
||||
| **Stability AI** (DreamStudio / Stable Image) | — | — | ✅ C2PA ("Stability AI Ltd") | Metadata strip |
|
||||
| **Microsoft Designer / Bing Image Creator** | — | ✅ SynthID via DALL-E backend (Designer) | ✅ C2PA (Bing runs MAI-Image, signed "Microsoft") | Metadata strip |
|
||||
| **Midjourney** | — | — | ✅ EXIF + XMP (prompt, model, seed) | Metadata strip |
|
||||
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
|
||||
| **Doubao** (ByteDance) / China AIGC generators | — | — | ✅ TC260 `<TC260:AIGC>` XMP label (China's mandatory AI labeling) | Metadata strip |
|
||||
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
|
||||
| **TreeRing** | — | ✅ Latent space watermark | — | Diffusion regeneration |
|
||||
|
||||
> Visible watermarks (logo overlays) are currently used only by Google Gemini / Nano Banana. Other services rely on invisible watermarks and/or metadata. Our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain.
|
||||
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, IPTC "Made with AI", the China TC260 AIGC label, embedded generation params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` extra) the open SD/SDXL/FLUX invisible watermark. The SynthID *pixel* watermark has no local decoder, so it is reported as a metadata proxy only.
|
||||
|
||||
## How it works
|
||||
|
||||
### Removing the Gemini / Nano Banana sparkle watermark
|
||||
@@ -65,12 +73,14 @@ Google embeds **SynthID** into every image generated by Gemini / Nano Banana. Ot
|
||||
The removal pipeline (default profile, SDXL):
|
||||
|
||||
```text
|
||||
image → resize to ~1024px (SDXL native) → encode to latent space (VAE)
|
||||
image → encode to latent space (VAE) at native resolution
|
||||
→ add controlled noise (forward diffusion)
|
||||
→ denoise (reverse diffusion, ~50 steps at strength 0.05)
|
||||
→ decode back to pixels (VAE) → upscale to original resolution
|
||||
→ decode back to pixels (VAE)
|
||||
```
|
||||
|
||||
By default the image is processed at its **native resolution** with no pre-downscale, matching the hosted raiw.cc backend (fal `fast-sdxl`, which is `stabilityai/stable-diffusion-xl-base-1.0` — the same checkpoint the CLI defaults to). At strength ~0.05 SDXL img2img does not need the input shrunk, and the old forced downscale-to-1024 then upscale-back round-trip was the main quality loss. Pass `--max-resolution N` to cap the long side only when a very large image runs out of GPU/MPS memory (it reintroduces that lossy round-trip).
|
||||
|
||||
SDXL is the default since May 2026: empirically defeats SynthID v2 on Gemini 3 Pro outputs, where the older SD-1.5 pipeline at 768 px did not. The SD-1.5 path was removed once it was verified not to handle v2.
|
||||
|
||||
**Face Protection**: before diffusion, YOLO detects people in the image and extracts them. After diffusion, the original faces are blended back with a soft elliptical mask to prevent AI distortion of facial features.
|
||||
@@ -201,6 +211,8 @@ remove-ai-watermarks visible image.png -o clean.png
|
||||
|
||||
# Invisible watermark only (SynthID etc.) — requires GPU
|
||||
remove-ai-watermarks invisible image.png -o clean.png --humanize 4.0
|
||||
# Runs at native resolution by default. On a very large image that OOMs the
|
||||
# GPU/MPS, cap the long side: --max-resolution 2048
|
||||
|
||||
# Check / strip AI metadata (C2PA, EXIF, "Made with AI" labels)
|
||||
# --check also flags SynthID-bearing sources: a C2PA manifest signed by
|
||||
@@ -277,7 +289,7 @@ Tracked but not yet implemented:
|
||||
- **Local SynthID *pixel* detector**. Not feasible today: Google's decoder is proprietary, and magnitude/carrier spectral methods do not separate real content (confirmed by three independent evaluations, including a from-scratch gpt-image pilot; see CLAUDE.md). Blocked on either (a) a programmatic generation path (OpenAI / Gemini API) to build a per-(model, resolution) labeled corpus at scale, or (b) a raw watermarked-output dataset. If data arrives, the next approach to try is a learned classifier on diverse content rather than a fixed carrier codebook.
|
||||
- **Grow the SynthID reference corpus** (`data/synthid_corpus/`) with oracle-labeled samples per model and resolution (Gemini app for Google, openai.com/verify for OpenAI). Prerequisite for any pixel-detector attempt and for an automated removal-regression set.
|
||||
- **Real non-PNG C2PA fixtures**. SynthID-source detection for JPEG / WebP / AVIF is currently covered only by synthetic byte blobs; replace with real vendor-emitted files to ground the binary-scan path.
|
||||
- **Maintenance debt**. Bump transitive `idna` to 3.15 (blocks `uv-secure`); clear strict-pyright debt in `remove_ai_metadata` / `cli.py` (untyped piexif / PIL / click / rich) so `maintain.sh` can finish green.
|
||||
- **Maintenance debt**. Clear strict-pyright debt in `remove_ai_metadata` / `cli.py` (untyped piexif / PIL / click / rich) so `maintain.sh` can finish green. (`uv-secure` is already clean since `idna` was bumped to 3.16.)
|
||||
- **AVIF / HEIF / JPEG-XL detection limits**. Removal strips top-level C2PA `uuid` and JUMBF `jumb` boxes. EXIF/XMP boxes inside these containers are not yet scrubbed (PNG and JPEG are fully covered).
|
||||
- **Video pipeline (`noai-video`)**: per-frame inpainting and tracking for Sora 2 dynamic logo, Veo 3.1 badge, Kling, Runway. Separate package, not folded into this repo.
|
||||
|
||||
@@ -291,17 +303,17 @@ Watermarking and provenance for AI-generated content is now regulated in several
|
||||
|
||||
| Jurisdiction | Instrument | Status (May 2026) | Relevance |
|
||||
| --- | --- | --- | --- |
|
||||
| EU | AI Act, Article 50(2) | Marking obligations postponed to **2 December 2026** under the December 2025 omnibus agreement. Code of Practice finalising May/June 2026. | Removing mandated provenance markers with intent to deceive may be sanctioned under national implementations. |
|
||||
| US (federal) | COPIED Act | Enacted 2025. | Criminalises removal of provenance information with intent to deceive about content origin. The tool itself is lawful; usage may not be. |
|
||||
| US (state) | CA AB 2655, TX SB 751, similar | In force. | Content-specific (election deepfakes, sexual deepfakes). Not tool-specific. |
|
||||
| China | Deep Synthesis Regulation, 2025 updates | In force. | Mandatory visible label for AI content. Removal is an administrative offence. |
|
||||
| UK | Online Safety Act, 2025 transparency extension | In force. | Platform obligations, not user obligations. |
|
||||
| EU | AI Act, Article 50 | Transparency duties apply from **2 August 2026**. Legacy generative systems (placed on the market before that date) get a grandfathering period to **2 December 2026** for the Article 50(2) marking duty, under the Digital Omnibus (Commission proposal Nov 2025; co-legislator political agreement 7 May 2026). Article 50 guidelines and a marking Code of Practice are being finalised through 2026. | Removing mandated provenance markers with intent to deceive may be sanctioned under national implementations. |
|
||||
| US (federal) | COPIED Act | **Reintroduced April 2025; not enacted** (pending in the Senate). | If passed, would set NIST provenance standards and prohibit tampering with / removing provenance information. The tool itself is lawful; usage may not be. |
|
||||
| US (state) | CA AB 2655, TX SB 751, similar | TX SB 751 in force; **CA AB 2655 struck down** by a federal court (Aug 2025, Section 230 / First Amendment). | Content-specific (election deepfakes, sexual deepfakes). Not tool-specific. |
|
||||
| China | Measures for Labeling AI-Generated Content (+ GB 45438-2025) | In force since **1 September 2025**. | Mandatory explicit (visible) + implicit (metadata) labels for AI content; tampering with or removing labels is prohibited. |
|
||||
| UK | Online Safety Act 2023 / Ofcom guidance | In force, but **no statutory AI-provenance or watermarking obligation**. | Ofcom encourages watermarking / provenance metadata as voluntary "attribution measures"; platform duties, not user obligations. |
|
||||
|
||||
## Threat model
|
||||
|
||||
This tool defends already-distributed AI imagery against automatic detection systems (social-platform "Made with AI" labels, third-party classifiers, content-policy filters). It does **not** retroactively anonymise generation.
|
||||
|
||||
In particular, **SynthID-Image v2** (Google, deployed October 2025 with Gemini 3 Pro / Nano Banana Pro / Imagen 4 / Veo) embeds a **136-bit payload** ([arxiv 2510.09263](https://arxiv.org/abs/2510.09263)). The payload is believed to encode a user / session identifier. If the original watermarked file ever passed through a system controlled by the prompt originator (a saved Gemini account history, a screenshot uploaded to a Google product, a backup), Google retains the ability to link that original to the generating account. Stripping the watermark from a copy you possess does not erase Google's server-side record.
|
||||
In particular, **SynthID** (Google DeepMind) is embedded across Google's generative media stack — Imagen (images), Veo (video), Lyria (audio) — and Gemini app image outputs (Nano Banana / Gemini 3 Pro, which we verified positive via the Gemini app's SynthID oracle); Google reported over 10 billion items watermarked by December 2025. It carries a **multi-bit payload** — the research paper's SynthID-O variant encodes 136-bit payloads in 512x512 images ([arxiv 2510.09263](https://arxiv.org/abs/2510.09263)). The payload is believed to encode a user / session identifier. If the original watermarked file ever passed through a system controlled by the prompt originator (a saved Gemini account history, a screenshot uploaded to a Google product, a backup), Google retains the ability to link that original to the generating account. Stripping the watermark from a copy you possess does not erase Google's server-side record.
|
||||
|
||||
Use cases where the threat model fits:
|
||||
- You generated the image yourself, want to publish it as your own work, and accept the consequences if Google ever publishes their detector logs.
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 5.1 MiB |
@@ -26,7 +26,7 @@ The `trainedAlgorithmicMedia` tag is what triggers "Made with AI" labels on Inst
|
||||
|
||||
### Invisible pixel-level watermark
|
||||
|
||||
OpenAI's system card for Images 2.0 states the model embeds an "imperceptible, robust, and content-specific" pixel-level watermark alongside C2PA. No public detector exists, so bypass cannot be verified empirically for this sample.
|
||||
OpenAI's system card for Images 2.0 states the model embeds an "imperceptible, robust, and content-specific" pixel-level watermark alongside C2PA. This sample was downloaded 2026-04-22, before OpenAI's 19 May 2026 rollout of Google's SynthID watermark across ChatGPT / Codex / the API, so it likely predates SynthID and carries only the original content-specific watermark. Since that rollout, the openai.com/verify tool (in preview) is the public oracle for both signals; there is still no local decoder, so bypass cannot be verified empirically here without the oracle.
|
||||
|
||||
## Reproducing the removal
|
||||
|
||||
@@ -38,4 +38,4 @@ remove-ai-watermarks metadata amur-leopard.png --remove -o amur-leopard.clean.pn
|
||||
remove-ai-watermarks all amur-leopard.png -o amur-leopard.all.png --device mps
|
||||
```
|
||||
|
||||
Note: the diffusion step downscales to 768 px, which degrades fine text on text-heavy infographics like this one. Tracked as a known limitation; see project TODO.
|
||||
Note: the diffusion step runs at native resolution by default (no pre-downscale), so fine text on a text-heavy infographic like this one is preserved. Pass `--max-resolution N` only if a very large image OOMs the GPU/MPS (that reintroduces a lossy downscale then upscale round-trip).
|
||||
@@ -118,22 +118,29 @@ Download; Leonardo serves the original JPEG in-chat (download button matches).
|
||||
|
||||
What each platform actually embeds, verified by byte-scan (and Gemini-app oracle
|
||||
where noted). The detector's coverage is complementary: metadata catches C2PA /
|
||||
IPTC; the visible detector catches the Gemini-family sparkle; the SynthID pixel
|
||||
itself has no local detector (oracle only).
|
||||
IPTC; `exif_generator` catches EXIF `Make`/`Software` + XMP `CreatorTool`;
|
||||
`invisible_watermark.py` (imwatermark) catches the open SD/SDXL/FLUX DWT-DCT
|
||||
watermark on pristine files; the visible detector catches the Gemini-family
|
||||
sparkle; the SynthID *pixel* itself has no local detector (oracle only).
|
||||
|
||||
| Platform | C2PA issuer | SynthID pixel | IPTC "Made with AI" | Visible sparkle | Corpus label |
|
||||
|---|---|---|---|---|---|
|
||||
| Gemini app | Google | yes | - | yes | pos |
|
||||
| ChatGPT / gpt-image | OpenAI | yes | - | - | pos |
|
||||
| Microsoft Designer | OpenAI + Microsoft | yes (via OpenAI) | - | - | pos |
|
||||
| Google AI Studio (Nano Banana) | **none** | yes (oracle-confirmed) | - | yes | pos (metadata blind spot) |
|
||||
| Meta AI | none | no | **yes** | - | neg (for SynthID) |
|
||||
| Leonardo.ai | none | no | no | - | neg |
|
||||
| Grok (xAI) | none (non-adopter) | no | no | - | neg (captured: clean low-res preview) |
|
||||
| Bing Image Creator (DALL-E) | (expect OpenAI, like Designer) | - | - | - | not captured (UI uncooperative) |
|
||||
| Platform | C2PA issuer | SynthID pixel | IPTC "Made with AI" | Visible sparkle | imwatermark | Corpus label |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Gemini app | Google | yes | - | yes | - | pos |
|
||||
| ChatGPT / gpt-image | OpenAI | yes | - | - | - | pos |
|
||||
| Microsoft Designer | OpenAI + Microsoft | yes (via OpenAI) | - | - | - | pos |
|
||||
| Bing Image Creator | Microsoft (MAI-Image) | no | - | - | - | pos (C2PA "Microsoft", not OpenAI) |
|
||||
| Google AI Studio (Nano Banana) | **none** | yes (oracle-confirmed) | - | yes | - | pos (metadata blind spot) |
|
||||
| Stability AI (Brand Studio) | Stability AI Ltd | no | - | - | no | pos (C2PA only) |
|
||||
| Ideogram | none | no | - | - | no | pos (EXIF `Make="Ideogram AI"` only) |
|
||||
| Meta AI | none | no | **yes** | - | - | neg (for SynthID) |
|
||||
| Leonardo.ai | none | no | no | - | no | neg |
|
||||
| Recraft | none (export strips) | no | no | - | no | neg (re-encoded export, no signal) |
|
||||
| Krea (FLUX 2 host) | none | no | no | - | no | neg (host omits the imwatermark encoder) |
|
||||
| Grok (xAI) | none (non-adopter) | no | no | - | no | neg (captured: clean low-res preview) |
|
||||
|
||||
Key takeaways:
|
||||
- The same model differs by *surface*: Gemini app wraps C2PA, AI Studio (API/playground) emits none -- only the pixel + sparkle survive.
|
||||
- Microsoft Designer's DALL-E backend inherits OpenAI's C2PA+SynthID (issuer "OpenAI, Microsoft").
|
||||
- Microsoft Designer's DALL-E backend inherits OpenAI's C2PA+SynthID (issuer "OpenAI, Microsoft"); Bing now runs Microsoft's own **MAI-Image** and signs C2PA as "Microsoft" (not OpenAI/DALL-E).
|
||||
- Meta uses the IPTC `digitalSourceType` marker, not C2PA or SynthID.
|
||||
- The open imwatermark fires only on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL) -- not from re-hosts (Krea, Stability hosted SDXL) or re-encoded design exports (Recraft, Canva). Ideogram's only signal is the EXIF `Make` tag.
|
||||
- Bing and Grok web UIs are uncooperative for autonomous capture (no document_idle for screenshots; blob downloads intermittently no-op; low-res in-chat previews). Use their native download button manually if a full-res sample is needed.
|
||||
@@ -116,3 +116,4 @@ f9bb6b1039ee228221d2eead7daebd46422a36a4a8959a74fa0198e1aa19f9b8,f9bb6b10-a_red_
|
||||
ca2162624607b8f030c63cb1371d6afdd3f0d7af3d9b17514f7ae64a2e8f378c,ca216262-0001_2_a-red-ceramic-coffee-mug-sits-on-weather_rs0Lt1beXV-Mwjs33vWKVw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
|
||||
8a90f2ddf83ceedd7944531d27248191c79dc321ec76ea1f36df6cff9f6b8916,8a90f2dd-0001_3_a-red-ceramic-coffee-mug-sits-on-weather_Lm96lBwcXsmdeCfCpm52vw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
|
||||
87fca198d5c436b27a16685972802a2ff0a310a941248c9ec8d81560d7f387a1,87fca198-0001_4_a-red-ceramic-coffee-mug-sits-on-weather_pGmlgcH4XqSaq41hc0a9AQ_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
|
||||
f6dd47a5ffd319aea21bf10dcf9877097666420b02c2620080bac12b03976e7e,f6dd47a5-4ef377bd-gpt-image-2-cleaned.png,cleaned,"our pipeline (invisible/SDXL, native-res default)",stabilityai/stable-diffusion-xl-base-1.0,1254,1254,png,,,openai-verify,2026-05-25T20:50:38Z,"cleaned from 4ef377bd via v0.5.3 'all' at native 1254x1254 (prod-equivalent); openai.com/verify: SynthID NOT detected. Re-confirms #10 native-res default defeats OpenAI SynthID (closes #15 root cause). Note: native res OOMs on 20GB MPS, auto-fell back to CPU."
|
||||
|
+6
-3
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.5.0"
|
||||
version = "0.5.4"
|
||||
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
@@ -28,8 +28,11 @@ gpu = [
|
||||
]
|
||||
# Open invisible-watermark (imwatermark) decoder for detecting the DWT-DCT
|
||||
# watermarks embedded by Stable Diffusion / SDXL / FLUX. Optional because it
|
||||
# pulls non-headless opencv; identify() guards the import and skips the signal
|
||||
# when absent.
|
||||
# pulls non-headless opencv AND torch (invisible-watermark declares torch a hard
|
||||
# dependency, and WatermarkDecoder eagerly imports rivaGan -> torch at import
|
||||
# time, so the dwtDct-only detect path still needs torch present even though it
|
||||
# never runs on GPU). So `detect` alone pulls torch -- no need to add `gpu` for
|
||||
# detection. identify() guards the import and skips the signal when absent.
|
||||
detect = [
|
||||
"invisible-watermark>=0.2.0",
|
||||
]
|
||||
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
|
||||
|
||||
__version__ = "0.5.0"
|
||||
__version__ = "0.5.4"
|
||||
+138
-45
@@ -12,6 +12,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
@@ -21,6 +22,11 @@ from rich.table import Table
|
||||
|
||||
from remove_ai_watermarks import __version__
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy as np
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import DetectionResult
|
||||
|
||||
console = Console()
|
||||
|
||||
SUPPORTED_FORMATS = {".png", ".jpg", ".jpeg", ".webp"}
|
||||
@@ -56,6 +62,74 @@ def _validate_image(path: Path) -> Path:
|
||||
return path
|
||||
|
||||
|
||||
_ALPHA_FORMATS = {".png", ".webp"}
|
||||
|
||||
|
||||
def _watermark_region(det: DetectionResult, width: int, height: int) -> tuple[int, int, int, int]:
|
||||
"""Pick a watermark bbox: detector's region if confident, else the default config slot."""
|
||||
if det.confidence > 0.15:
|
||||
return det.region
|
||||
from remove_ai_watermarks.gemini_engine import get_watermark_config
|
||||
|
||||
config = get_watermark_config(width, height)
|
||||
px, py = config.get_position(width, height)
|
||||
return (px, py, config.logo_size, config.logo_size)
|
||||
|
||||
|
||||
def _read_bgr_and_alpha(path: Path) -> tuple[np.ndarray | None, np.ndarray | None]:
|
||||
"""Read an image preserving its alpha channel separately.
|
||||
|
||||
Returns ``(bgr, alpha)`` where ``alpha`` is a single-channel ndarray when the
|
||||
source has transparency, else ``None``. Greyscale inputs are promoted to BGR.
|
||||
Returns ``(None, None)`` if the image cannot be decoded.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
image = cv2.imread(str(path), cv2.IMREAD_UNCHANGED)
|
||||
if image is None:
|
||||
return None, None
|
||||
if image.ndim == 2:
|
||||
return cv2.cvtColor(image, cv2.COLOR_GRAY2BGR), None
|
||||
if image.shape[2] == 4:
|
||||
return image[:, :, :3].copy(), image[:, :, 3].copy()
|
||||
return image, None
|
||||
|
||||
|
||||
def _write_bgr_with_alpha(
|
||||
path: Path,
|
||||
bgr: np.ndarray,
|
||||
alpha: np.ndarray | None,
|
||||
clear_region: tuple[int, int, int, int] | None = None,
|
||||
pad: int = 6,
|
||||
) -> None:
|
||||
"""Write BGR (with optional alpha) to ``path``.
|
||||
|
||||
When ``alpha`` is provided and the output extension supports it, writes a
|
||||
4-channel image. If ``clear_region`` is given as ``(x, y, w, h)``, alpha is
|
||||
forced to 0 inside that bbox (expanded by ``pad`` px) so the watermark area
|
||||
becomes fully transparent in the saved file.
|
||||
"""
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
if alpha is None or path.suffix.lower() not in _ALPHA_FORMATS:
|
||||
cv2.imwrite(str(path), bgr)
|
||||
return
|
||||
|
||||
alpha_out = alpha
|
||||
if clear_region is not None:
|
||||
alpha_out = alpha.copy()
|
||||
x, y, w, h = clear_region
|
||||
height, width = alpha.shape[:2]
|
||||
x0, y0 = max(0, x - pad), max(0, y - pad)
|
||||
x1, y1 = min(width, x + w + pad), min(height, y + h + pad)
|
||||
if x1 > x0 and y1 > y0:
|
||||
alpha_out[y0:y1, x0:x1] = 0
|
||||
|
||||
bgra = np.dstack([bgr, alpha_out])
|
||||
cv2.imwrite(str(path), bgra)
|
||||
|
||||
|
||||
# ── Main group ───────────────────────────────────────────────────────
|
||||
|
||||
|
||||
@@ -110,8 +184,6 @@ def cmd_visible(
|
||||
|
||||
Uses reverse alpha blending — fast, deterministic, offline.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import GeminiEngine
|
||||
|
||||
_banner()
|
||||
@@ -122,8 +194,8 @@ def cmd_visible(
|
||||
|
||||
engine = GeminiEngine()
|
||||
|
||||
# Load image
|
||||
image = cv2.imread(str(source), cv2.IMREAD_COLOR)
|
||||
# Load image (preserving any alpha channel separately)
|
||||
image, alpha = _read_bgr_and_alpha(source)
|
||||
if image is None:
|
||||
console.print(f"[red]Error:[/] Failed to read image: {source}")
|
||||
raise SystemExit(1)
|
||||
@@ -151,18 +223,12 @@ def cmd_visible(
|
||||
|
||||
# Removal
|
||||
t0 = time.monotonic()
|
||||
region: tuple[int, int, int, int] | None = None
|
||||
with console.status("[cyan]Removing watermark…[/]"):
|
||||
result = engine.remove_watermark(image)
|
||||
|
||||
if inpaint:
|
||||
if det.confidence > 0.15:
|
||||
region = det.region
|
||||
else:
|
||||
from remove_ai_watermarks.gemini_engine import get_watermark_config
|
||||
|
||||
config = get_watermark_config(w, h)
|
||||
pos = config.get_position(w, h)
|
||||
region = (pos[0], pos[1], config.logo_size, config.logo_size)
|
||||
region = _watermark_region(det, w, h)
|
||||
result = engine.inpaint_residual(
|
||||
result,
|
||||
region,
|
||||
@@ -172,9 +238,9 @@ def cmd_visible(
|
||||
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
# Save
|
||||
# Save (preserves transparency by clearing alpha in the watermark region)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
cv2.imwrite(str(output), result)
|
||||
_write_bgr_with_alpha(output, result, alpha, clear_region=region)
|
||||
|
||||
# Strip metadata
|
||||
if strip_metadata:
|
||||
@@ -212,6 +278,12 @@ def cmd_visible(
|
||||
@click.option(
|
||||
"--humanize", type=float, default=0.0, help="Analog Humanizer film grain intensity (0 = off, typical: 2.0-6.0)."
|
||||
)
|
||||
@click.option(
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@click.pass_context
|
||||
def cmd_invisible(
|
||||
ctx: click.Context,
|
||||
@@ -224,6 +296,7 @@ def cmd_invisible(
|
||||
seed: int | None,
|
||||
hf_token: str | None,
|
||||
humanize: float,
|
||||
max_resolution: int,
|
||||
) -> None:
|
||||
"""Remove invisible AI watermarks (SynthID, StableSignature, TreeRing).
|
||||
|
||||
@@ -270,6 +343,7 @@ def cmd_invisible(
|
||||
guidance_scale=None,
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
max_resolution=max_resolution,
|
||||
)
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
@@ -410,6 +484,12 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
@click.option(
|
||||
"--humanize", type=float, default=0.0, help="Analog Humanizer film grain intensity (0 = off, typical: 2.0-6.0)."
|
||||
)
|
||||
@click.option(
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@click.pass_context
|
||||
def cmd_all(
|
||||
ctx: click.Context,
|
||||
@@ -425,6 +505,7 @@ def cmd_all(
|
||||
seed: int | None,
|
||||
hf_token: str | None,
|
||||
humanize: float,
|
||||
max_resolution: int,
|
||||
) -> None:
|
||||
"""Remove ALL watermarks: visible + invisible + metadata.
|
||||
|
||||
@@ -435,9 +516,7 @@ def cmd_all(
|
||||
|
||||
If invisible watermark deps are not installed, skips step 2 with a warning.
|
||||
"""
|
||||
import cv2
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import GeminiEngine, get_watermark_config
|
||||
from remove_ai_watermarks.gemini_engine import GeminiEngine
|
||||
|
||||
_banner()
|
||||
source = _validate_image(source)
|
||||
@@ -461,7 +540,7 @@ def cmd_all(
|
||||
# ── Step 1: Visible watermark ────────────────────────────────
|
||||
console.print("\n [bold cyan]① Visible watermark removal[/]")
|
||||
engine = GeminiEngine()
|
||||
image = cv2.imread(str(source), cv2.IMREAD_COLOR)
|
||||
image, alpha = _read_bgr_and_alpha(source)
|
||||
if image is None:
|
||||
console.print(f"[red]Error:[/] Failed to read image: {source}")
|
||||
raise SystemExit(1)
|
||||
@@ -469,25 +548,21 @@ def cmd_all(
|
||||
h, w = image.shape[:2]
|
||||
console.print(f" [dim]Input:[/] {source.name} ({w}x{h})")
|
||||
|
||||
region: tuple[int, int, int, int] | None = None
|
||||
with console.status("[cyan]Removing visible watermark…[/]"):
|
||||
det = engine.detect_watermark(image)
|
||||
if det.detected:
|
||||
result = engine.remove_watermark(image)
|
||||
if inpaint:
|
||||
if det.confidence > 0.15:
|
||||
region = det.region
|
||||
else:
|
||||
config = get_watermark_config(w, h)
|
||||
pos = config.get_position(w, h)
|
||||
region = (pos[0], pos[1], config.logo_size, config.logo_size)
|
||||
region = _watermark_region(det, w, h)
|
||||
result = engine.inpaint_residual(result, region, method=inpaint_method)
|
||||
console.print(" [green]✓[/] Visible watermark removed")
|
||||
else:
|
||||
result = image.copy()
|
||||
console.print(" [dim]Skipped (no visible watermark detected)[/]")
|
||||
|
||||
# Save to temp file for invisible engine input
|
||||
cv2.imwrite(str(tmp_path), result)
|
||||
# Save to temp file for invisible engine input (preserve alpha if present)
|
||||
_write_bgr_with_alpha(tmp_path, result, alpha, clear_region=region)
|
||||
|
||||
# ── Step 2: Invisible watermark ──────────────────────────────
|
||||
console.print("\n [bold cyan]② Invisible watermark removal[/]")
|
||||
@@ -522,6 +597,7 @@ def cmd_all(
|
||||
num_inference_steps=steps,
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
max_resolution=max_resolution,
|
||||
)
|
||||
console.print(" [green]✓[/] Invisible watermark removed")
|
||||
|
||||
@@ -536,10 +612,15 @@ def cmd_all(
|
||||
console.print(f" [yellow]⚠[/] Metadata strip failed: {e}")
|
||||
|
||||
# ── Write final result ────────────────────────────────────────
|
||||
import shutil
|
||||
|
||||
# The invisible step (and downstream cv2.IMREAD_COLOR paths) drops alpha,
|
||||
# so re-attach the original alpha (with the watermark region cleared)
|
||||
# when writing the final output for transparent formats.
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
shutil.move(str(tmp_path), str(output))
|
||||
final_bgr, _ = _read_bgr_and_alpha(tmp_path)
|
||||
if final_bgr is None:
|
||||
console.print(f"[red]Error:[/] Failed to read intermediate file: {tmp_path}")
|
||||
raise SystemExit(1)
|
||||
_write_bgr_with_alpha(output, final_bgr, alpha, clear_region=region)
|
||||
|
||||
finally:
|
||||
# Clean up temp file if it still exists
|
||||
@@ -568,6 +649,7 @@ def _process_batch_image(
|
||||
seed: int | None,
|
||||
hf_token: str | None,
|
||||
humanize: float,
|
||||
max_resolution: int = 0,
|
||||
) -> None:
|
||||
"""Process a single image for batch mode.
|
||||
|
||||
@@ -577,13 +659,11 @@ def _process_batch_image(
|
||||
Raises:
|
||||
ValueError: If the image cannot be opened.
|
||||
"""
|
||||
if mode in ("visible", "all"):
|
||||
import cv2
|
||||
saved_alpha: np.ndarray | None = None
|
||||
saved_region: tuple[int, int, int, int] | None = None
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import (
|
||||
GeminiEngine,
|
||||
get_watermark_config,
|
||||
)
|
||||
if mode in ("visible", "all"):
|
||||
from remove_ai_watermarks.gemini_engine import GeminiEngine
|
||||
|
||||
if "_vis_engine" not in ctx.obj:
|
||||
ctx.obj["_vis_engine"] = GeminiEngine()
|
||||
@@ -591,27 +671,24 @@ def _process_batch_image(
|
||||
read_path = img_path
|
||||
if mode == "all" and out_path.exists():
|
||||
read_path = out_path
|
||||
image = cv2.imread(str(read_path), cv2.IMREAD_COLOR)
|
||||
image, alpha = _read_bgr_and_alpha(read_path)
|
||||
if image is None:
|
||||
raise ValueError("Failed to read image")
|
||||
|
||||
region: tuple[int, int, int, int] | None = None
|
||||
det = engine.detect_watermark(image)
|
||||
if det.detected:
|
||||
result = engine.remove_watermark(image)
|
||||
if inpaint:
|
||||
if det.confidence > 0.15:
|
||||
region = det.region
|
||||
else:
|
||||
h, w = image.shape[:2]
|
||||
config = get_watermark_config(w, h)
|
||||
pos = config.get_position(w, h)
|
||||
region = (pos[0], pos[1], config.logo_size, config.logo_size)
|
||||
|
||||
h, w = image.shape[:2]
|
||||
region = _watermark_region(det, w, h)
|
||||
result = engine.inpaint_residual(result, region)
|
||||
else:
|
||||
result = image.copy()
|
||||
|
||||
cv2.imwrite(str(out_path), result)
|
||||
_write_bgr_with_alpha(out_path, result, alpha, clear_region=region)
|
||||
saved_alpha = alpha
|
||||
saved_region = region
|
||||
|
||||
if mode in ("invisible", "all"):
|
||||
from remove_ai_watermarks.invisible_engine import (
|
||||
@@ -635,6 +712,7 @@ def _process_batch_image(
|
||||
num_inference_steps=steps,
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
max_resolution=max_resolution,
|
||||
)
|
||||
|
||||
if mode in ("metadata", "all"):
|
||||
@@ -642,6 +720,13 @@ def _process_batch_image(
|
||||
|
||||
remove_ai_metadata(img_path if mode == "metadata" else out_path, out_path)
|
||||
|
||||
# In "all" mode, the invisible step (color-only OpenCV paths) drops alpha,
|
||||
# so re-attach the cached alpha when the input had transparency.
|
||||
if mode == "all" and saved_alpha is not None:
|
||||
final_bgr, _ = _read_bgr_and_alpha(out_path)
|
||||
if final_bgr is not None:
|
||||
_write_bgr_with_alpha(out_path, final_bgr, saved_alpha, clear_region=saved_region)
|
||||
|
||||
|
||||
@main.command("batch")
|
||||
@click.argument("directory", type=click.Path(exists=True, file_okay=False, path_type=Path))
|
||||
@@ -670,6 +755,12 @@ def _process_batch_image(
|
||||
@click.option("--device", type=click.Choice(["auto", "cpu", "mps", "cuda"]), default="auto", help="Inference device.")
|
||||
@click.option("--seed", type=int, default=None, help="Random seed for reproducibility.")
|
||||
@click.option("--hf-token", type=str, default=None, help="HuggingFace API token.")
|
||||
@click.option(
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@click.pass_context
|
||||
def cmd_batch(
|
||||
ctx: click.Context,
|
||||
@@ -684,6 +775,7 @@ def cmd_batch(
|
||||
hf_token: str | None,
|
||||
inpaint: bool,
|
||||
humanize: float,
|
||||
max_resolution: int,
|
||||
) -> None:
|
||||
"""Process all images in a directory."""
|
||||
_banner()
|
||||
@@ -733,6 +825,7 @@ def cmd_batch(
|
||||
seed=seed,
|
||||
hf_token=hf_token,
|
||||
humanize=humanize,
|
||||
max_resolution=max_resolution,
|
||||
)
|
||||
processed += 1
|
||||
|
||||
|
||||
@@ -24,8 +24,10 @@ from typing import TYPE_CHECKING
|
||||
|
||||
from remove_ai_watermarks.metadata import (
|
||||
AI_METADATA_KEYS,
|
||||
AIGC_MARKERS,
|
||||
C2PA_UUID,
|
||||
IPTC_AI_MARKERS,
|
||||
aigc_label,
|
||||
exif_generator,
|
||||
get_ai_metadata,
|
||||
)
|
||||
@@ -219,6 +221,15 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
if platform is None:
|
||||
platform = "Made-with-AI tag (e.g. Meta AI); platform not specified"
|
||||
|
||||
# ── China TC260 AIGC label (Doubao and other China-served gens) ──
|
||||
aigc = any(m in head for m in AIGC_MARKERS)
|
||||
if aigc:
|
||||
producer = (aigc_label(image_path) or {}).get("ContentProducer", "")
|
||||
signals.append(Signal("aigc", f"TC260 AIGC label{f' (producer {producer})' if producer else ''}", "high"))
|
||||
watermarks.append("China AIGC label (TC260 standard)")
|
||||
if platform is None:
|
||||
platform = "China AIGC-labeled generator (TC260; e.g. Doubao)"
|
||||
|
||||
# ── Local diffusion parameters (Stable Diffusion / ComfyUI) ──────
|
||||
local_keys = sorted(k for k in meta if k.lower() in _LOCAL_GEN_KEYS)
|
||||
if local_keys:
|
||||
@@ -247,7 +258,9 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
# ── Verdict so far (metadata + embedded watermark) ──────────────
|
||||
invisible_wm = any(s.name == "invisible_watermark" for s in signals)
|
||||
exif_gen = any(s.name == "exif_generator" for s in signals)
|
||||
ai_from_metadata = bool((has_c2pa and (c2pa_is_ai or synthid)) or iptc or local_keys or invisible_wm or exif_gen)
|
||||
ai_from_metadata = bool(
|
||||
(has_c2pa and (c2pa_is_ai or synthid)) or iptc or aigc or local_keys or invisible_wm or exif_gen
|
||||
)
|
||||
|
||||
# ── Visible Gemini sparkle (fallback for stripped-metadata case) ─
|
||||
if check_visible and (conf := _visible_sparkle(image_path)) is not None and conf >= _SPARKLE_THRESHOLD:
|
||||
|
||||
@@ -42,6 +42,24 @@ def is_available() -> bool:
|
||||
return False
|
||||
|
||||
|
||||
def _target_size(width: int, height: int, max_resolution: int) -> tuple[int, int] | None:
|
||||
"""Compute the downscaled (width, height) for a long-side cap, or None for native.
|
||||
|
||||
Returns None when no pre-downscale is needed: ``max_resolution <= 0`` (native
|
||||
resolution, the default that matches the raiw.cc backend -- see issue #10) or
|
||||
the long side already fits the cap. Otherwise scales the long side down to
|
||||
``max_resolution`` preserving aspect ratio (integer-truncated, matching the
|
||||
PIL ``resize`` call site). Pure function so the native-vs-downscale decision
|
||||
is unit-testable without loading the diffusion model.
|
||||
"""
|
||||
if max_resolution > 0 and max(width, height) > max_resolution:
|
||||
ratio = max_resolution / max(width, height)
|
||||
# Clamp the short side to >=1: extreme aspect ratios (e.g. 5000x3 capped
|
||||
# at 1024) would otherwise truncate it to 0 and crash image.resize().
|
||||
return (max(1, int(width * ratio)), max(1, int(height * ratio)))
|
||||
return None
|
||||
|
||||
|
||||
class InvisibleEngine:
|
||||
"""Remove invisible AI watermarks using diffusion model regeneration.
|
||||
|
||||
@@ -107,6 +125,7 @@ class InvisibleEngine:
|
||||
seed: int | None = None,
|
||||
humanize: float = 0.0,
|
||||
protect_faces: bool = True,
|
||||
max_resolution: int = 0,
|
||||
) -> Path:
|
||||
"""Remove invisible watermark from an image.
|
||||
|
||||
@@ -119,6 +138,11 @@ class InvisibleEngine:
|
||||
seed: Random seed for reproducibility.
|
||||
humanize: Intensity of Analog Humanizer film grain (0 = off).
|
||||
protect_faces: Boolean to extract and restore faces intact.
|
||||
max_resolution: Cap the long side (px) before diffusion. 0 (default)
|
||||
= native resolution, no pre-downscale -- matches the hosted
|
||||
raiw.cc backend. Set a positive value only to bound GPU/MPS
|
||||
memory on very large inputs (it reintroduces a lossy
|
||||
downscale->upscale round-trip).
|
||||
|
||||
Returns:
|
||||
Path to the cleaned image.
|
||||
@@ -127,43 +151,35 @@ class InvisibleEngine:
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
# SDXL is trained at 1024px and degrades both quality and watermark-removal
|
||||
# efficacy below that.
|
||||
max_dimension = 1024
|
||||
# Process at native resolution by default (max_resolution=0). The hosted
|
||||
# raiw.cc backend (fal fast-sdxl) does NO pre-downscale either, and at
|
||||
# strength ~0.05 SDXL img2img does not need the input shrunk to ~1024 --
|
||||
# the old forced downscale->upscale round-trip was the main quality loss
|
||||
# (see issue #10). A positive max_resolution caps the long side only to
|
||||
# bound GPU/MPS memory on very large inputs.
|
||||
image = Image.open(image_path)
|
||||
image = ImageOps.exif_transpose(image)
|
||||
orig_size = image.size # (width, height)
|
||||
_tmp_path = None
|
||||
|
||||
if max(image.width, image.height) > max_dimension:
|
||||
ratio = max_dimension / max(image.width, image.height)
|
||||
new_size = (int(image.width * ratio), int(image.height * ratio))
|
||||
# Optional long-side downscale; native resolution by default (issue #10).
|
||||
target = _target_size(image.width, image.height, max_resolution)
|
||||
if target is not None:
|
||||
if self._progress_callback:
|
||||
self._progress_callback(
|
||||
f"Downscaling {image.width}x{image.height} "
|
||||
f"to {new_size[0]}x{new_size[1]} "
|
||||
f"(model trained at {max_dimension}px)..."
|
||||
f"to {target[0]}x{target[1]} "
|
||||
f"(max-resolution cap {max_resolution}px)..."
|
||||
)
|
||||
image = image.resize(new_size, Image.Resampling.LANCZOS)
|
||||
image = image.resize(target, Image.Resampling.LANCZOS)
|
||||
|
||||
# Save to a temp file instead of overwriting the original
|
||||
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
|
||||
_tmp_path = Path(_tmp_str)
|
||||
image.save(_tmp_path)
|
||||
import os as _os
|
||||
|
||||
_os.close(_tmp_fd)
|
||||
image_path = _tmp_path
|
||||
else:
|
||||
# We must save the transposed image back to a tmp file if it was rotated
|
||||
# otherwise WatermarkRemover will reload it without EXIF rotation!
|
||||
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
|
||||
_tmp_path = Path(_tmp_str)
|
||||
image.save(_tmp_path)
|
||||
import os as _os
|
||||
|
||||
_os.close(_tmp_fd)
|
||||
image_path = _tmp_path
|
||||
# Always persist to a temp file, even without downscaling: WatermarkRemover
|
||||
# reloads by path, so the EXIF-transposed pixels must be saved or rotation
|
||||
# is lost. Cleaned up in the finally block via _tmp_path.
|
||||
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
|
||||
_tmp_path = Path(_tmp_str)
|
||||
image.save(_tmp_path)
|
||||
os.close(_tmp_fd)
|
||||
image_path = _tmp_path
|
||||
|
||||
try:
|
||||
# Optional: Face protection (Phase 1 - Extraction)
|
||||
@@ -244,7 +260,8 @@ class InvisibleEngine:
|
||||
|
||||
return out_path
|
||||
finally:
|
||||
if _tmp_path is not None and _tmp_path.exists():
|
||||
# _tmp_path is always set above (we persist the image unconditionally).
|
||||
if _tmp_path.exists():
|
||||
_tmp_path.unlink()
|
||||
|
||||
def remove_watermark_batch(
|
||||
|
||||
@@ -73,6 +73,17 @@ IPTC_AI_MARKERS: tuple[bytes, ...] = (
|
||||
b"compositeWithTrainedAlgorithmicMedia",
|
||||
)
|
||||
|
||||
# China's mandatory AI-content labeling (TC260, the national cybersecurity
|
||||
# standards committee). AI generators serving China embed an XMP block in the
|
||||
# TC260 namespace -- ``<TC260:AIGC>{"Label":"1",...}``. Doubao (ByteDance) uses
|
||||
# this; the same standard is mandatory for Jimeng, Kling, Qwen, Ernie, etc.,
|
||||
# so the marker covers the whole China-AIGC-labeled ecosystem. Container-
|
||||
# agnostic (XMP is text), so a raw-byte scan catches it in PNG/JPEG/etc.
|
||||
AIGC_MARKERS: tuple[bytes, ...] = (
|
||||
b"tc260.org.cn/ns/AIGC",
|
||||
b"TC260:AIGC",
|
||||
)
|
||||
|
||||
STANDARD_METADATA_KEYS: frozenset[str] = frozenset(
|
||||
[
|
||||
"Author",
|
||||
@@ -139,9 +150,35 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
return True
|
||||
if C2PA_UUID in data:
|
||||
return True
|
||||
if any(marker in data for marker in AIGC_MARKERS):
|
||||
return True
|
||||
return any(marker in data for marker in IPTC_AI_MARKERS)
|
||||
|
||||
|
||||
def aigc_label(image_path: Path) -> dict[str, str] | None:
|
||||
"""Parse a China TC260 ``<TC260:AIGC>`` AI-labeling block, if present.
|
||||
|
||||
Returns the decoded JSON (e.g. ``{"Label": "1", "ContentProducer": ...}``)
|
||||
or None. The block is XMP text (HTML-entity encoded), so it is found by a
|
||||
container-agnostic raw-byte scan and works for PNG/JPEG/WebP alike.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
|
||||
with open(image_path, "rb") as f:
|
||||
data = f.read(1024 * 1024)
|
||||
match = re.search(rb"<TC260:AIGC>(.*?)</TC260:AIGC>", data, re.DOTALL)
|
||||
if not match:
|
||||
return None
|
||||
raw = html.unescape(match.group(1).decode("utf-8", "replace"))
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except ValueError:
|
||||
return None
|
||||
return {str(k): str(v) for k, v in parsed.items()} if isinstance(parsed, dict) else None
|
||||
|
||||
|
||||
def synthid_source(image_path: Path) -> str | None:
|
||||
"""Return the vendor name(s) if the image carries a SynthID pixel watermark.
|
||||
|
||||
@@ -286,6 +323,11 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
|
||||
# fall back to the format-agnostic source check for the SynthID verdict.
|
||||
if "synthid_watermark" not in result and (vendor := synthid_source(image_path)):
|
||||
result.setdefault("synthid_watermark", synthid_verdict(vendor))
|
||||
|
||||
# China TC260 AI-content label (Doubao and other China-served generators).
|
||||
if aigc := aigc_label(image_path):
|
||||
producer = aigc.get("ContentProducer", "")
|
||||
result["aigc_label"] = f"China AIGC label (TC260){f'; producer {producer}' if producer else ''}"
|
||||
return result
|
||||
|
||||
|
||||
|
||||
@@ -36,9 +36,10 @@ _HAS_COLOR_MATCHER = False
|
||||
_HAS_DIFFUSERS = False
|
||||
|
||||
try:
|
||||
from ctrlregen.pipeline import CustomCtrlRegenPipeline
|
||||
from diffusers import AutoencoderKL, ControlNetModel, UniPCMultistepScheduler
|
||||
|
||||
from remove_ai_watermarks.noai.ctrlregen.pipeline import CustomCtrlRegenPipeline
|
||||
|
||||
_HAS_DIFFUSERS = True
|
||||
except ImportError:
|
||||
AutoencoderKL = None # type: ignore[assignment,misc]
|
||||
@@ -54,7 +55,7 @@ except ImportError:
|
||||
CannyDetector = None # type: ignore[assignment,misc]
|
||||
|
||||
try:
|
||||
from ctrlregen.color import color_match
|
||||
from remove_ai_watermarks.noai.ctrlregen.color import color_match
|
||||
|
||||
_HAS_COLOR_MATCHER = True
|
||||
except ImportError:
|
||||
@@ -239,7 +240,7 @@ class CtrlRegenEngine:
|
||||
needs_tiling = orig_w > TILE_SIZE or orig_h > TILE_SIZE
|
||||
|
||||
if needs_tiling:
|
||||
from ctrlregen.tiling import resize_center_crop, run_tiled
|
||||
from remove_ai_watermarks.noai.ctrlregen.tiling import resize_center_crop, run_tiled
|
||||
|
||||
aligned_w = orig_w // 8 * 8
|
||||
aligned_h = orig_h // 8 * 8
|
||||
@@ -264,7 +265,7 @@ class CtrlRegenEngine:
|
||||
ip_adapter_image=orig_image,
|
||||
)
|
||||
else:
|
||||
from ctrlregen.tiling import resize_center_crop
|
||||
from remove_ai_watermarks.noai.ctrlregen.tiling import resize_center_crop
|
||||
|
||||
proc_image = resize_center_crop(image, PROCESS_SIZE)
|
||||
self._set_progress(f"Preprocessed {orig_w}x{orig_h}px → {proc_image.size[0]}x{proc_image.size[1]}px")
|
||||
|
||||
@@ -176,8 +176,11 @@ def _reinstall_torch_cuda_and_restart() -> None:
|
||||
return
|
||||
|
||||
os.environ[_CUDA_FIX_ENV_KEY] = "1"
|
||||
restart_code = f"import sys; sys.argv = {sys.argv!r}; from remove_ai_watermarks.cli import main; sys.exit(main())"
|
||||
os.execl(sys.executable, sys.executable, "-c", restart_code)
|
||||
# Re-exec via ``-m`` rather than building a ``-c`` string from repr(sys.argv).
|
||||
# ``-m`` makes Python set argv[0] to the module path, so forward only the
|
||||
# actual args (sys.argv[1:]); passing the full argv would re-inject the
|
||||
# program name as a spurious first argument to Click.
|
||||
os.execv(sys.executable, [sys.executable, "-m", "remove_ai_watermarks.cli", *sys.argv[1:]])
|
||||
|
||||
|
||||
def _ensure_watermark_deps() -> None:
|
||||
|
||||
@@ -72,6 +72,27 @@ def _mock_invisible_engine():
|
||||
return mock_cls, mock_engine
|
||||
|
||||
|
||||
def _mock_invisible_engine_drops_alpha():
|
||||
"""Mock InvisibleEngine that mimics the real engine's BGR-only output path.
|
||||
|
||||
The real diffusion-based engine reads with cv2.IMREAD_COLOR and writes a
|
||||
3-channel result. This mock simulates that so we can regression-test alpha
|
||||
preservation across the ``all`` pipeline.
|
||||
"""
|
||||
|
||||
def _mock_remove_watermark(image_path, output_path=None, **kwargs):
|
||||
out = output_path or image_path.with_stem(image_path.stem + "_clean")
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
bgr = cv2.imread(str(image_path), cv2.IMREAD_COLOR)
|
||||
cv2.imwrite(str(out), bgr)
|
||||
return out
|
||||
|
||||
mock_engine = MagicMock()
|
||||
mock_engine.remove_watermark.side_effect = _mock_remove_watermark
|
||||
mock_cls = MagicMock(return_value=mock_engine)
|
||||
return mock_cls, mock_engine
|
||||
|
||||
|
||||
class TestMainGroup:
|
||||
"""Tests for the top-level CLI group."""
|
||||
|
||||
@@ -144,6 +165,73 @@ class TestVisibleCommand:
|
||||
result = runner.invoke(main, ["visible", "/nonexistent/file.png"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_visible_preserves_rgba_transparency(self, runner, tmp_path):
|
||||
"""Visible removal on an RGBA PNG must keep the alpha channel,
|
||||
not silently flatten the image onto an opaque background.
|
||||
"""
|
||||
rgba = np.zeros((200, 200, 4), dtype=np.uint8)
|
||||
rgba[:, :, :3] = 200 # light grey foreground
|
||||
rgba[50:150, 50:150, 3] = 255 # opaque square in the middle, rest transparent
|
||||
src = tmp_path / "rgba_in.png"
|
||||
cv2.imwrite(str(src), rgba)
|
||||
|
||||
output = tmp_path / "rgba_out.png"
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["visible", str(src), "-o", str(output), "--no-detect"],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
assert output.exists()
|
||||
|
||||
out = cv2.imread(str(output), cv2.IMREAD_UNCHANGED)
|
||||
assert out.ndim == 3, f"output is not 3D: shape={out.shape}"
|
||||
assert out.shape[2] == 4, f"output is not RGBA: shape={out.shape}"
|
||||
# The transparent corners must remain transparent.
|
||||
assert out[0, 0, 3] == 0
|
||||
assert out[199, 199, 3] == 0
|
||||
# The opaque centre remains opaque (the watermark region default is bottom-right,
|
||||
# which doesn't overlap the centre square at 200x200).
|
||||
assert out[100, 100, 3] == 255
|
||||
|
||||
def test_visible_clears_alpha_in_watermark_region(self, runner, tmp_path):
|
||||
"""When inpainting an RGBA image, the watermark region must be cleared
|
||||
in the alpha channel so the sparkle area becomes transparent, not opaque-black.
|
||||
"""
|
||||
rgba = np.full((200, 200, 4), 255, dtype=np.uint8) # fully opaque white
|
||||
src = tmp_path / "rgba_full.png"
|
||||
cv2.imwrite(str(src), rgba)
|
||||
|
||||
output = tmp_path / "rgba_cleared.png"
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["visible", str(src), "-o", str(output), "--no-detect"],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
out = cv2.imread(str(output), cv2.IMREAD_UNCHANGED)
|
||||
assert out.shape[2] == 4
|
||||
# Default sparkle position is in the bottom-right; alpha there must be 0.
|
||||
from remove_ai_watermarks.gemini_engine import get_watermark_config
|
||||
|
||||
cfg = get_watermark_config(200, 200)
|
||||
px, py = cfg.get_position(200, 200)
|
||||
size = cfg.logo_size
|
||||
assert out[py + size // 2, px + size // 2, 3] == 0, "alpha in the watermark region was not cleared"
|
||||
|
||||
def test_visible_rgb_input_stays_rgb(self, runner, sample_png, tmp_path):
|
||||
"""Regression: a plain RGB PNG must NOT gain a spurious alpha channel."""
|
||||
output = tmp_path / "rgb_out.png"
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["visible", str(sample_png), "-o", str(output), "--no-detect"],
|
||||
)
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
out = cv2.imread(str(output), cv2.IMREAD_UNCHANGED)
|
||||
assert out.ndim == 3, f"output is not 3D: shape={out.shape}"
|
||||
assert out.shape[2] == 3, f"RGB input produced non-RGB output: shape={out.shape}"
|
||||
|
||||
|
||||
class TestInvisibleCommand:
|
||||
"""Tests for the 'invisible' subcommand."""
|
||||
@@ -210,6 +298,33 @@ class TestAllCommand:
|
||||
result = runner.invoke(main, ["all", "/nonexistent/file.png"])
|
||||
assert result.exit_code != 0
|
||||
|
||||
def test_all_preserves_rgba_across_invisible_step(self, runner, tmp_path):
|
||||
"""Regression: ``all`` must keep transparency even when the invisible
|
||||
step writes a 3-channel result (as the real diffusion engine does).
|
||||
"""
|
||||
rgba = np.zeros((200, 200, 4), dtype=np.uint8)
|
||||
rgba[:, :, :3] = 200
|
||||
rgba[50:150, 50:150, 3] = 255 # opaque square; corners transparent
|
||||
src = tmp_path / "rgba_in.png"
|
||||
cv2.imwrite(str(src), rgba)
|
||||
|
||||
output = tmp_path / "rgba_out.png"
|
||||
mock_cls, _engine = _mock_invisible_engine_drops_alpha()
|
||||
with (
|
||||
patch("remove_ai_watermarks.cli.InvisibleEngine", mock_cls, create=True),
|
||||
patch("remove_ai_watermarks.invisible_engine.InvisibleEngine", mock_cls),
|
||||
patch("remove_ai_watermarks.cli.invisible_available", return_value=True, create=True),
|
||||
patch("remove_ai_watermarks.invisible_engine.is_available", return_value=True),
|
||||
):
|
||||
result = runner.invoke(main, ["all", str(src), "-o", str(output)])
|
||||
|
||||
assert result.exit_code == 0, result.output
|
||||
out = cv2.imread(str(output), cv2.IMREAD_UNCHANGED)
|
||||
assert out.ndim == 3, f"output not 3D: shape={out.shape}"
|
||||
assert out.shape[2] == 4, f"output is not RGBA: shape={out.shape}"
|
||||
assert out[0, 0, 3] == 0
|
||||
assert out[100, 100, 3] == 255
|
||||
|
||||
|
||||
class TestMetadataCommand:
|
||||
"""Tests for the 'metadata' subcommand."""
|
||||
@@ -350,6 +465,37 @@ class TestBatchCommand:
|
||||
assert result.exit_code == 0, result.output
|
||||
assert "3 processed" in result.output
|
||||
|
||||
def test_batch_all_mode_preserves_rgba(self, runner, tmp_path):
|
||||
"""Regression: batch ``all`` must keep transparency across the
|
||||
alpha-dropping invisible step (mirrors test_all_preserves_rgba_...).
|
||||
"""
|
||||
input_dir = tmp_path / "input"
|
||||
input_dir.mkdir()
|
||||
rgba = np.zeros((200, 200, 4), dtype=np.uint8)
|
||||
rgba[:, :, :3] = 200
|
||||
rgba[50:150, 50:150, 3] = 255
|
||||
cv2.imwrite(str(input_dir / "rgba.png"), rgba)
|
||||
|
||||
output_dir = tmp_path / "output"
|
||||
mock_cls, _engine = _mock_invisible_engine_drops_alpha()
|
||||
with (
|
||||
patch("remove_ai_watermarks.cli.InvisibleEngine", mock_cls, create=True),
|
||||
patch("remove_ai_watermarks.invisible_engine.InvisibleEngine", mock_cls),
|
||||
patch("remove_ai_watermarks.cli.invisible_available", return_value=True, create=True),
|
||||
patch("remove_ai_watermarks.invisible_engine.is_available", return_value=True),
|
||||
):
|
||||
result = runner.invoke(
|
||||
main,
|
||||
["batch", str(input_dir), "-o", str(output_dir), "--mode", "all"],
|
||||
)
|
||||
assert result.exit_code == 0, result.output
|
||||
|
||||
out = cv2.imread(str(output_dir / "rgba.png"), cv2.IMREAD_UNCHANGED)
|
||||
assert out.ndim == 3, f"output not 3D: shape={out.shape}"
|
||||
assert out.shape[2] == 4, f"output is not RGBA: shape={out.shape}"
|
||||
assert out[0, 0, 3] == 0
|
||||
assert out[100, 100, 3] == 255
|
||||
|
||||
def test_batch_default_output_dir(self, runner, tmp_path):
|
||||
input_dir = _make_batch_dir(tmp_path)
|
||||
result = runner.invoke(
|
||||
|
||||
@@ -317,3 +317,35 @@ class TestIdentifyInvisibleWatermark:
|
||||
def test_check_invisible_false_skips(self, tmp_path: Path):
|
||||
r = identify(self._sdxl_watermarked(tmp_path), check_visible=False, check_invisible=False)
|
||||
assert not any(s.name == "invisible_watermark" for s in r.signals)
|
||||
|
||||
|
||||
class TestIdentifyAIGC:
|
||||
"""China TC260 AIGC label is detected and attributed (e.g. Doubao)."""
|
||||
|
||||
def _aigc_png(self, tmp_path: Path) -> Path:
|
||||
from PIL import Image
|
||||
|
||||
p = tmp_path / "doubao.png"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
xmp = (
|
||||
'<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF '
|
||||
'xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">'
|
||||
'<rdf:Description xmlns:TC260="http://www.tc260.org.cn/ns/AIGC/1.0/">'
|
||||
"<TC260:AIGC>{"Label":"1","ContentProducer":"BYTEDANCE001"}"
|
||||
"</TC260:AIGC></rdf:Description></rdf:RDF></x:xmpmeta>"
|
||||
)
|
||||
with open(p, "ab") as f:
|
||||
f.write(xmp.encode())
|
||||
return p
|
||||
|
||||
def test_aigc_detected(self, tmp_path: Path):
|
||||
r = identify(self._aigc_png(tmp_path), check_visible=False)
|
||||
assert r.is_ai_generated is True
|
||||
assert r.platform is not None
|
||||
assert "AIGC" in r.platform or "TC260" in r.platform
|
||||
assert any("AIGC" in w for w in r.watermarks)
|
||||
|
||||
def test_aigc_signal_carries_producer(self, tmp_path: Path):
|
||||
r = identify(self._aigc_png(tmp_path), check_visible=False)
|
||||
sig = next(s for s in r.signals if s.name == "aigc")
|
||||
assert "BYTEDANCE001" in sig.detail
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from remove_ai_watermarks.invisible_engine import InvisibleEngine, is_available
|
||||
from remove_ai_watermarks.invisible_engine import InvisibleEngine, _target_size, is_available
|
||||
|
||||
|
||||
class TestIsAvailable:
|
||||
@@ -26,3 +26,45 @@ class TestInvisibleEngineInit:
|
||||
|
||||
def test_ctrlregen_model_id(self):
|
||||
assert InvisibleEngine.CTRLREGEN_MODEL_ID == "yepengliu/ctrlregen"
|
||||
|
||||
|
||||
class TestTargetSize:
|
||||
"""Regression guard for the native-resolution decision (issues #10 / #15).
|
||||
|
||||
max_resolution=0 must NOT downscale -- the forced downscale->upscale
|
||||
round-trip was the quality loss in #10, and downscaling at all let SynthID
|
||||
survive in #15 (the native SDXL pass at strength ~0.05 is what defeats it).
|
||||
"""
|
||||
|
||||
def test_native_default_no_downscale(self):
|
||||
# The default (0) means native resolution: no resize, regardless of size.
|
||||
assert _target_size(4096, 4096, 0) is None
|
||||
assert _target_size(123, 456, 0) is None
|
||||
|
||||
def test_negative_cap_treated_as_native(self):
|
||||
assert _target_size(4096, 4096, -1) is None
|
||||
|
||||
def test_cap_below_long_side_downscales(self):
|
||||
# 2000x1000, cap 1024 -> long side scaled to 1024, aspect preserved.
|
||||
assert _target_size(2000, 1000, 1024) == (1024, 512)
|
||||
|
||||
def test_cap_uses_long_side_for_portrait(self):
|
||||
# Portrait: height is the long side, so it drives the ratio.
|
||||
assert _target_size(1000, 2000, 1024) == (512, 1024)
|
||||
|
||||
def test_cap_at_or_above_long_side_no_downscale(self):
|
||||
# Already within the cap (and exactly equal) -> no resize.
|
||||
assert _target_size(800, 600, 1024) is None
|
||||
assert _target_size(1024, 768, 1024) is None
|
||||
|
||||
def test_integer_truncation_matches_pil_call_site(self):
|
||||
# 1254x1254 (the gpt-image sample) capped at 1000: int(1254*1000/1254)=1000.
|
||||
assert _target_size(1254, 1254, 1000) == (1000, 1000)
|
||||
# Non-divisible ratio truncates toward zero like int() at the call site.
|
||||
assert _target_size(1000, 333, 500) == (500, 166)
|
||||
|
||||
def test_extreme_aspect_ratio_clamps_short_side_to_one(self):
|
||||
# 5000x3 capped at 1024: int(3 * 1024/5000) = 0 would crash resize();
|
||||
# the short side must clamp to 1, never 0.
|
||||
assert _target_size(5000, 3, 1024) == (1024, 1)
|
||||
assert _target_size(3, 5000, 1024) == (1, 1024)
|
||||
@@ -390,3 +390,63 @@ class TestExifGenerator:
|
||||
|
||||
def test_clean_image_is_none(self, tmp_clean_png: Path):
|
||||
assert exif_generator(tmp_clean_png) is None
|
||||
|
||||
|
||||
class TestAIGCLabel:
|
||||
"""China TC260 AIGC labeling (Doubao and other China-served generators)."""
|
||||
|
||||
def _aigc_png(self, tmp_path: Path, label: str = "1", producer: str = "TESTPRODUCER001") -> Path:
|
||||
from remove_ai_watermarks.metadata import aigc_label # noqa: F401 (import-time guard)
|
||||
|
||||
p = tmp_path / "doubao.png"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
# XMP is HTML-entity encoded in real files; aigc_label must unescape it.
|
||||
xmp = (
|
||||
'<x:xmpmeta xmlns:x="adobe:ns:meta/"><rdf:RDF '
|
||||
'xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">'
|
||||
'<rdf:Description rdf:about="" '
|
||||
'xmlns:TC260="http://www.tc260.org.cn/ns/AIGC/1.0/"><TC260:AIGC>'
|
||||
f"{{"Label":"{label}","ContentProducer":"{producer}"}}"
|
||||
"</TC260:AIGC></rdf:Description></rdf:RDF></x:xmpmeta>"
|
||||
)
|
||||
with open(p, "ab") as f:
|
||||
f.write(xmp.encode())
|
||||
return p
|
||||
|
||||
def test_parses_label_and_producer(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
info = aigc_label(self._aigc_png(tmp_path))
|
||||
assert info is not None
|
||||
assert info["Label"] == "1"
|
||||
assert info["ContentProducer"] == "TESTPRODUCER001"
|
||||
|
||||
def test_none_when_absent(self, tmp_clean_png):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
assert aigc_label(tmp_clean_png) is None
|
||||
|
||||
def test_has_ai_metadata_detects_aigc(self, tmp_path: Path):
|
||||
assert has_ai_metadata(self._aigc_png(tmp_path))
|
||||
|
||||
def test_get_ai_metadata_surfaces_aigc(self, tmp_path: Path):
|
||||
meta = get_ai_metadata(self._aigc_png(tmp_path))
|
||||
assert "aigc_label" in meta
|
||||
assert "TC260" in meta["aigc_label"]
|
||||
|
||||
|
||||
@pytest.mark.skipif(not (SAMPLES_DIR / "doubao-1.png").exists(), reason="doubao sample not present")
|
||||
class TestAIGCRealSample:
|
||||
"""Real Doubao (ByteDance) sample carries the China TC260 AIGC XMP label."""
|
||||
|
||||
def test_doubao_aigc_label(self):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
info = aigc_label(SAMPLES_DIR / "doubao-1.png")
|
||||
assert info is not None
|
||||
assert info["Label"] == "1"
|
||||
assert info["ContentProducer"] # ByteDance producer code present
|
||||
|
||||
def test_doubao_detected_as_ai(self):
|
||||
assert has_ai_metadata(SAMPLES_DIR / "doubao-1.png")
|
||||
assert "aigc_label" in get_ai_metadata(SAMPLES_DIR / "doubao-1.png")
|
||||
@@ -815,11 +815,11 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "idna"
|
||||
version = "3.11"
|
||||
version = "3.16"
|
||||
source = { registry = "https://pypi.org/simple" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" }
|
||||
sdist = { url = "https://files.pythonhosted.org/packages/1a/88/bcf9709822fe69d02c2a6a77956c98ce6ea8ca8767a9aadcedc7eb6a2390/idna-3.16.tar.gz", hash = "sha256:d7a6da03db833450fca25d2358ac9ff06cd624577a4aea3a596d5c0f77b8e03d", size = 203770, upload-time = "2026-05-22T00:16:18.781Z" }
|
||||
wheels = [
|
||||
{ url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" },
|
||||
{ url = "https://files.pythonhosted.org/packages/94/16/70255075a9859a0e3adb789b68ceb0e210dec03934245fd98d248226572f/idna-3.16-py3-none-any.whl", hash = "sha256:cc246e3a3f89580c3a951b5ad298ca4638078b2cdd4f115654332b5c26daded5", size = 74165, upload-time = "2026-05-22T00:16:16.698Z" },
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2150,7 +2150,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.5.0"
|
||||
version = "0.5.4"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
|
||||
Reference in new issue
Block a user