Compare commits

..
7 Commits
Author SHA1 Message Date
test-userandClaude Opus 4.7 f9cf14c372 feat(metadata): strip container metadata from WebM/MP3/WAV/FLAC/OGG via ffmpeg (v0.6.4)
remove_ai_metadata now handles non-ISOBMFF audio/video (which the box walker
can't reach) by shelling out to ffmpeg with a lossless stream copy
(`-map_metadata -1 -map_chapters -1 -c copy`): codec data is untouched, only
container tags/chapters (ID3 / RIFF / Vorbis comments / EBML tags) are dropped.
Requires ffmpeg on PATH; raises a clear RuntimeError if absent or if ffmpeg
can't parse the input (instead of crashing in the image path).

Verified end-to-end: a real ffmpeg-made WAV/MP3 with a "Suno AI" title tag ->
tag gone, audio bytes preserved.

NOT built (evaluated, deliberate): Resemble PerTh audio *detection* --
`get_watermark()` returns a raw bit array with no presence/confidence flag, so
reliably telling watermarked from clean needs Resemble's fixed payload or a
confidence API (neither public; no real sample to calibrate). Same wall as the
SynthID pixel detector. AVIF/HEIF meta-box EXIF/XMP stripping also stays a gap
(needs exiftool, a non-installed binary). Both documented in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:39:42 -07:00
test-userandClaude Opus 4.7 bc3228d387 feat(visible): Doubao text-mark removal + universal region eraser
Add deterministic, CPU-only removal of the visible Doubao "豆包AI生成" mark and
a position-agnostic region eraser for any other visible watermark/logo.

- doubao_engine.py: locate (geometry, scales with width) + polarity-aware
  white-top-hat glyph mask + cv2 inpaint; coverage-gated detection and a
  dense-text safety guard. No GPU, ~30ms.
- region_eraser.py + `erase` command: inpaint arbitrary --region box(es).
  Default cv2 backend (no deps); optional big-LaMa via onnxruntime (`lama`
  extra, Carve/LaMa-ONNX, model downloaded on first use, never bundled).
- cli `visible --mark auto|gemini|doubao`: auto routes by detector confidence.
- tests for both engines; seed previously-unseeded CLI image fixtures to stop
  the Doubao detector flaking on random corners.
- .gitignore: doubao_capture/{seeds,captures} scratch (alpha-map calibration).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:31:51 -07:00
test-userandClaude Opus 4.7 9f93d9c0c5 feat(identify): add Sony C2PA device attribution, verified (v0.6.3)
Adds Sony to _DEVICE_C2PA_PLATFORM, matching Sony's own `sony.sig` / `sony.cert`
C2PA assertion namespace (NOT bare "Sony", which is a common EXIF Make). Verified
against a real Sony-signed file (Sony PXW-Z300, signer "Sony Corporation") found
in the Security4Media/c2pa-video-player repo. The sample is video (MP4) -- our
ISOBMFF C2PA path detects it; Sony Alpha stills likely share the namespace.

Verified device set is now Leica, Nikon, Google Pixel, Sony, Truepic. Canon /
Samsung / Bria still have no public direct-download C2PA sample to verify.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 21:13:49 -07:00
test-userandClaude Opus 4.7 64be9598f2 fix(identify): device-token-first C2PA attribution; add verified Pixel (v0.6.2)
Replaces the claim-generator-string match with a distinctive device-token scan
of the manifest bytes (_device_platform / _DEVICE_C2PA_PLATFORM), which is more
robust: it catches devices where the generator name lives under a non-standard
CBOR key (Pixel uses `claim_generator_info`, so it has no `claim_generator`).

- Adds Google Pixel, verified against a real Pixel 10 Pro C2PA file (attached to
  c2pa-rs issue #1609/#1554): cert CN "Pixel Camera", digitalSourceType
  `computationalCapture` -> capture authenticity, not AI (is_ai stays None).
- Token distinctiveness is load-bearing: bare "Truepic" matched the OpenAI
  chatgpt-1.png fixture (Truepic is a trust-chain signing authority), so the
  token is the specific "Truepic_Lens"; "Pixel Camera" (cert CN) not "Pixel".
- Verified Leica/Nikon/Truepic/Pixel attribute correctly and OpenAI/Adobe/MJ
  do not regress. Sony/Canon/Samsung/Bria stay unmapped: no public direct-
  download C2PA sample exists to verify their in-manifest string.
- Regression tests: device token beats incidental issuer mentions (Leica,
  Pixel-vs-Google).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:43:40 -07:00
test-userandClaude Opus 4.7 dda2ee7fbb fix(identify): attribute C2PA by claim_generator, not incidental issuer tokens (v0.6.1)
Verified on real signed files that the issuer byte-scan mis-attributes
multi-entity manifests: Leica read as "Truepic" (timestamp authority in the
chain), Nikon as "Adobe Firefly" (XMP-toolkit "Adobe" + the sample's
"Adobe_MAX" name), Truepic as "Google". Platform attribution now prefers the
claim generator (what produced the asset) and falls back to the issuer scan.

- New _CLAIM_GENERATOR_PLATFORM map + _platform_from_generator; claim generator
  read for non-PNG via the now-public c2pa.cbor_text_after.
- Device tokens listed only where verified against a real C2PA file (Leica
  lc_c2pa, Nikon, Truepic Lens); Pixel/Samsung/Sony/Canon/Bria deferred until a
  real sample confirms the in-manifest string. Camera C2PA marks capture
  authenticity, so these never set is_ai.
- cbor_text_after made public (was _cbor_text_after); call sites + tests updated.
- Regression test: claim_generator beats incidental Adobe/Google/Truepic tokens.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 20:10:07 -07:00
test-userandClaude Opus 4.7 2676325184 feat(c2pa): expand soft-binding vendor map with registry-verified algs
Adds Trufo, Overlai, MarkAny, Mentaport, LumaTrace, VerdaAI, ContentLens, ISCC
(io.iscc content code), and Adobe ICN fingerprint to C2PA_SOFT_BINDINGS, and
notes AIWatermark wraps Meta PixelSeal. All `alg` prefixes verified against the
official c2pa-org/softbinding-algorithm-list registry.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 18:00:16 -07:00
test-userandClaude Opus 4.7 c196a16900 feat: detect soft-binding vendors, IPTC 2025.1, video/audio C2PA, TrustMark (v0.6.0)
Broadens metadata provenance coverage at the detection and container-strip level.

Detection:
- C2PA soft-binding `alg` -> forensic-watermark vendor (Adobe TrustMark,
  Digimarc, Imatag, Steg.AI, Microsoft, ...) via C2PA_SOFT_BINDINGS +
  soft_binding_vendors_in(); names the watermark vendor even when the watermark
  itself can't be decoded.
- IPTC Photo Metadata 2025.1 AI-disclosure XMP fields (AISystemUsed etc.) via
  iptc_ai_system() + IPTC_AI_FIELD_MARKERS.
- Adobe TrustMark open keyless decoder (trustmark_detector.py, optional extra
  `trustmark`) -- the watermark behind Adobe Durable Content Credentials.
  Detects provenance, not AI origin, so it does not assert is_ai.

Removal / containers:
- isobmff.strip_c2pa_boxes now also drops a top-level XMP uuid box that carries
  an AI label (matched by AI-marker content, byte-order-robust; plain XMP kept).
- remove_ai_metadata routes MP4/MOV/M4V/M4A (and any ftyp-sniffed ISOBMFF)
  through the box stripper; raises a clear error for non-ISOBMFF audio/video
  (WebM/MP3/WAV) instead of crashing in the image path.

Tests: soft-binding scan, IPTC element/attribute/presence, MP4 + M4A detect/
strip, ISOBMFF XMP surgical strip, content-sniff, unsupported-container guard,
TrustMark absent-safety + identify integration. ruff clean; pyright clean on
all new modules.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-26 17:56:48 -07:00
23 changed files with 3369 additions and 954 deletions

No files matched your search

+7
View File
@@ -34,3 +34,10 @@ yolov8n.pt
# Claude Code local settings
.claude/settings.local.json
# Doubao watermark calibration (local only; ship only the derived alpha-map asset).
# Synthetic seeds + raw Doubao captures are regenerable and not committed.
# Non-ours reference artifacts go in any _refs/ dir (already ignored above): usable
# locally for bootstrap/validation, never redistributed in the repo.
data/doubao_capture/seeds/
data/doubao_capture/captures/
+18 -7
View File
@@ -5,6 +5,8 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
## How to run
- `uv run remove-ai-watermarks all <image.png> -o <output.png>`
- `uv run remove-ai-watermarks visible <image.png> -o <out.png>` — visible-mark removal, CPU, no GPU. `--mark auto` (default) routes between the Gemini sparkle and the Doubao "豆包AI生成" text strip by detector confidence; `--mark gemini` / `--mark doubao` force one.
- `uv run remove-ai-watermarks erase <image.png> --region x,y,w,h -o <out.png>` — universal region eraser (any logo/object, any position). `--backend cv2` (default, no deps) or `--backend lama` (big-LaMa via onnxruntime, extra `lama`); `--region` is repeatable.
- `uv run remove-ai-watermarks identify <image>` — provenance verdict (platform + watermark inventory + confidence); `--json` for machine output, `--no-visible` to skip the cv2 sparkle detector
- `uv run remove-ai-watermarks metadata <image.png> --check` — inspect AI metadata (C2PA, EXIF, PNG chunks)
- `uv run remove-ai-watermarks metadata <image.png> --remove -o <out.png>` — strip all AI metadata
@@ -20,18 +22,26 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
## Configuration
- GPU/ML modules (invisible_engine, ctrlregen, watermark_remover) are optional — guard imports with `is_available()` checks
- Optional detection extras: `detect` (imwatermark — open SD/SDXL/FLUX watermark) and `trustmark` (Adobe TrustMark decoder; pulls torch + downloads weights). Both are guarded by `is_available()` and skipped by `identify` when absent.
- Tests for the *model-running* paths are limited to availability checks (multi-GB downloads). But the **pure helpers inside ML-adjacent modules are unit-tested without any download** and must stay that way: `_target_size` (native-vs-downscale, `test_invisible_engine.py`), the MPS->CPU fallback control flow via mocked pipelines (`test_img2img_runner.py`, 100% cover), and the tiling math `tile_positions`/`make_blend_weight`/`resize_center_crop` (`test_tiling.py`; `pytest.importorskip("torch")` since `tiling.py` imports torch at module top). Don't skip these as "ML, needs a model" — only `run_tiled`/`remove_watermark`/the diffusion bodies do.
## Key modules
- `noai/c2pa.py` — PNG chunk parser; use `extract_c2pa_chunk(path)` to get raw caBX payload, `has_c2pa_metadata(path)` to detect. Do not reimplement chunk parsing. `extract_c2pa_info(path)` sets `synthid_watermark`/`synthid_vendors` when the manifest is signed by a SynthID-using vendor.
- `noai/constants.py` — PNG_SIGNATURE, C2PA_CHUNK_TYPE, C2PA_SIGNATURES, C2PA_ISSUERS, and `SYNTHID_C2PA_ISSUERS` (issuers that pair SynthID with C2PA: Google, OpenAI). Add a new issuer here, not inline.
- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan. `xai_signature(path)` detects xAI/Grok's EXIF-only scheme (`ImageDescription` = `Signature: <base64>` + UUID `Artist`); it feeds `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify`.
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, IPTC "Made with AI", embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, visible Gemini sparkle) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. Add platform mappings to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read.
- `noai/c2pa.py` — PNG chunk parser; use `extract_c2pa_chunk(path)` to get raw caBX payload, `has_c2pa_metadata(path)` to detect. Do not reimplement chunk parsing. `extract_c2pa_info(path)` sets `synthid_watermark`/`synthid_vendors` when the manifest is signed by a SynthID-using vendor, and `soft_binding`/`soft_binding_vendors` when a `c2pa.soft-binding` `alg` names a forensic-watermark vendor (`soft_binding_vendors_in(buffer)` is the shared byte-scan, used by both the PNG parser and the non-PNG binary path).
- `noai/constants.py` — PNG_SIGNATURE, C2PA_CHUNK_TYPE, C2PA_SIGNATURES, C2PA_ISSUERS, `SYNTHID_C2PA_ISSUERS` (issuers that pair SynthID with C2PA: Google, OpenAI), and `C2PA_SOFT_BINDINGS` (soft-binding `alg` prefix → forensic-watermark vendor: Adobe TrustMark, Digimarc, Imatag, Steg.AI, Microsoft, ...). Add a new issuer/binding here, not inline.
- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan. `xai_signature(path)` detects xAI/Grok's EXIF-only scheme (`ImageDescription` = `Signature: <base64>` + UUID `Artist`); it feeds `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify`. `iptc_ai_system(path)` detects the IPTC Photo Metadata 2025.1 AI-disclosure XMP properties (`IPTC_AI_FIELD_MARKERS` = `AISystemUsed`/`AISystemVersionUsed`/`AIPromptInformation`/`AIPromptWriterName`) and returns the `AISystemUsed` generator name (or `"fields present"`). `remove_ai_metadata` routes **ISOBMFF video** (`.mp4`/`.mov`/`.m4v`) through the same `isobmff.strip_c2pa_boxes` as AVIF/HEIF (MP4 is ISOBMFF), and `_scrub_ai_exif` removes the xAI signature + AI-generator EXIF tags on JPEG output.
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, C2PA soft-binding forensic-watermark vendor, IPTC "Made with AI" + IPTC 2025.1 `AISystemUsed`, embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, visible Gemini sparkle, open invisible watermark, Adobe TrustMark via `trustmark_detector`) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. **C2PA platform attribution is device-token-first, issuer-scan fallback** (`_device_platform` scans manifest bytes for `_DEVICE_C2PA_PLATFORM` tokens, then `_attribute_platform`/`_ISSUER_PLATFORM`). **Why, verified on real signed files 2026-05-26:** the old issuer-only byte-scan matched ANY issuer substring anywhere, so multi-entity manifests mis-attributed -- Leica→"Truepic" (a signing authority in the trust chain), Nikon→"Adobe Firefly" (XMP-toolkit "Adobe" + the sample's "Adobe_MAX" name), Pixel→"Google (Gemini)" ("Google LLC" cert org), Truepic→"Google". A distinctive device token wins instead. **Token distinctiveness is load-bearing:** bare `b"Truepic"` mis-fires (it appears in unrelated trust chains -- it mis-attributed the OpenAI `chatgpt-1.png` fixture), so the token is the specific `b"Truepic_Lens"` from the Lens SDK claim generator; likewise `b"Pixel Camera"` (cert CN) not bare `b"Pixel"`. `_DEVICE_C2PA_PLATFORM` lists ONLY tokens **verified against a real C2PA file**: Leica (`lc_c2pa`/`Leica Camera`), Nikon (`NIKON`), Pixel (`Pixel Camera` -- from a real Pixel 10 Pro file attached to c2pa-rs issue #1609/#1554), Sony (`sony.sig`/`sony.cert` -- Sony's own C2PA assertion namespace, verified on a real Sony PXW-Z300 file; NOT bare "Sony" which is a common EXIF Make), Truepic (`Truepic_Lens`). Canon/Samsung/Bria have **no public direct-download C2PA sample** (checked exhaustively: GitHub issue/PR attachments, contentcredentials gallery, HF datasets -- all upload-to-verify or token-gated; Canon's only public file was a self-signed hobbyist CR3, not factory), so they stay unmapped until a real file is captured (same fixture discipline as Grok/Doubao). The Sony sample is video (MP4) -- our ISOBMFF C2PA path detects it; Sony Alpha stills likely share the `sony.*` namespace but are not separately verified. Camera C2PA marks capture authenticity, not AI (Pixel carries `computationalCapture`, not `trainedAlgorithmicMedia`), so these never set `is_ai` -- that stays driven by digital-source-type. `c2pa.cbor_text_after` (now public) is best-effort for the `generator` detail string only and can be None when the manifest keys it `claim_generator_info` (Pixel). Add device tokens to `_DEVICE_C2PA_PLATFORM`, generator/issuer platforms to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read.
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`.
- `doubao_engine.py` — visible Doubao "豆包AI生成" remover/detector (cv2/numpy, no GPU). `DoubaoEngine.locate` anchors a bottom-right box by **geometry** (mark scales with image WIDTH, fractions in module constants; no bundled template), `extract_mask` pulls the light low-saturation glyphs with a **polarity-aware white top-hat** (brighter-than-blurred-local-bg, so white-paper documents are left untouched instead of smeared), `detect` thresholds glyph coverage (`DETECT_MIN_COVERAGE` 0.16 separates real marks ≥0.20 from corner noise, which stays ≤0.06 on large images but can spike to ~0.15 on tiny ones), `remove_watermark` inpaints (cv2 Telea/NS) and **bails when coverage > `MAX_INPAINT_COVERAGE` 0.50** (dense-text background → would smear). Wired into `visible --mark` via `cli._run_doubao_if_selected`. **Logo is near-white (~253), not the gray some third-party tools assume.** Best on photo/illustration backgrounds; high-contrast edges leave faint residue (cv2-inpaint limit). Clean per-pixel reverse-alpha (Gemini-style) is the future upgrade but needs a captured/distilled alpha map — see below.
- `region_eraser.py` — universal region eraser (`erase` CLI). `erase(image, boxes=|mask=, backend=)`: `boxes_to_mask` → `cv2.inpaint` (`cv2` backend, default, no deps) or big-LaMa via onnxruntime (`lama` backend, extra `lama`, `Carve/LaMa-ONNX` Apache-2.0 model downloaded on first use, never bundled). `erase_lama` crops a padded region around the mask, runs LaMa at its fixed 512² input, pastes only masked pixels back (untouched areas stay pixel-exact). Lazy `_get_lama_session` singleton; `lama_available()` guards the optional import. **LaMa-ONNX costs ~3.5-4 GB peak RAM and ~5-6 s/call on CPU** (FFC working set, not arena — `enable_cpu_mem_arena=False` does not help), so it does NOT fit a minimal droplet; the cv2 backend (tens of MB, ~30 ms) does. LaMa quality at low RAM = serverless/GPU, mirroring how raiw.cc offloads SDXL to fal.
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. The `detect` extra pulls **torch** transitively (invisible-watermark declares torch a hard dep, and `WatermarkDecoder` eagerly imports `rivaGan` -> `torch` at import time), so detection needs torch present even though dwtDct runs CPU-only on cv2/numpy/pywavelets — no GPU and no separate `gpu` extra required. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
- `trustmark_detector.py` — `detect_trustmark(path)` decodes the OPEN, keyless **Adobe TrustMark** watermark (the soft binding behind Adobe Durable Content Credentials, `alg` `com.adobe.trustmark.P`) via the optional `trustmark` package (extra `trustmark`; pulls torch, downloads model weights on first use). Mirrors `invisible_watermark.py` (lazy singleton, top-of-module pyright pragma, returns None when absent). It detects *provenance*, not AI origin as such (TrustMark also marks human-authored content), so `identify` lists it as a watermark without setting `is_ai_generated`. Other soft-binding vendors (Digimarc/Imatag/Steg.AI/...) have no public decoder — they are only *named* via the `C2PA_SOFT_BINDINGS` scan, not decoded.
- `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features
### Doubao clean-reverse-alpha distillation (researched 2026-05-26, NOT yet shipped)
Plan to get Gemini-style pixel-accurate reverse-alpha for Doubao at cv2 cost (so it runs on a minimal CPU droplet): use LaMa **offline** to reconstruct the clean background `O`, then per-pixel solve `I=(1-α)O+αL` for the alpha map + logo colour (this is Google's CVPR-2017 multi-image idea, with LaMa standing in for the multi-image optimisation). Validated in principle (glyph error drops as alignment improves) but **blocked on data**, confirmed empirically over 6 attempts: a clean canonical alpha map needs **several PRISTINE images at ONE resolution** with varied backgrounds. The findable public Doubao images (ayaya, Tools_CleanMark, ours) are all different resolutions AND lossy (jpg/webp), and Doubao's mark position drifts ~±50 px even at the same resolution, so cross-image registration (ECC / phase-correlation) fails (1/11 templates aligned) — error floor ~64/255 on glyph pixels, not shippable. To finish: ~5-8 pristine same-resolution Doubao originals → per-image α via LaMa → align the clean α templates → median → ship the tiny map + an NCC localiser (Doubao's mark drifts, so runtime needs the same locate-then-reverse-alpha the Gemini engine already does). Don't retry on lossy/mixed-resolution scraped images — proven insufficient.
## Watermarking landscape (research 2026-05-24)
Who embeds what, and whether it is locally detectable (so we know which gaps are fillable). See `identify.py` for what we read.
@@ -42,15 +52,16 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
- **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `<TC260:AIGC>{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). **Doubao** (ByteDance) uses it (verified on the real #13 sample 2026-05-25; `ContentProducer` `001191110102MACQD9K64010000`, no C2PA/SynthID/imwatermark — the XMP block is the only signal; GitHub attachment upload did NOT strip it). The same standard is mandatory for Jimeng/Kling/Qwen/Ernie etc., so the one marker covers the whole China-AIGC-labeled ecosystem. `aigc_label` json-decodes the (HTML-entity-encoded) block; container-agnostic raw-byte scan.
- **No detectable signal on download (correctly reported `unknown`):** **Recraft** (PNG export is a re-encoded design export — strips everything), **Krea hosting FLUX 2** (no imwatermark despite FLUX — the host omits the encoder, same as Stability's hosted SDXL), and Midjourney (embeds nothing). Lesson: the imwatermark detector only fires on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL), not from re-hosts (Krea/Stability) or re-encoded exports (Recraft/Canva).
- **Invisible but NOT locally detectable (proprietary, API/oracle only — same wall as SynthID):** Amazon Titan Image Generator + Nova Canvas (Bedrock `DetectGeneratedContent` API), Kakao (new SynthID image adopter, May 2026), NVIDIA Cosmos (SynthID video). No local detector possible; treat like SynthID.
- **C2PA 2.4 "Durable Content Credentials" (April 2026; verified against the spec) raise the bar for metadata stripping.** 2.4 defines soft bindings (an invisible watermark or a content fingerprint) plus a server-side manifest repository and a new `c2pa.repository-receipt` assertion. Per the spec: "if a C2PA manifest is removed from an asset, but a copy of that manifest remains in a provenance store elsewhere, the manifest and asset may be matched using available soft bindings." So our local `metadata --remove` deletes the *embedded* manifest, but a fingerprint/watermark soft binding can still re-link the image to its manifest in a repository server-side. Stripping the file is becoming necessary-but-not-sufficient against durable provenance. (Our parsers target the stable embedded-manifest format documented in C2PA 2.1 §11; that format is unchanged in 2.4 -- the new pieces are repository/soft-binding infra, not the on-file box layout, so no parser change is implied.) Spec: https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html
- **Regulatory driver (context, not a code change):** AI-content labeling mandates are expanding, which pushes more generators toward exactly the C2PA + watermark signals we read. The full per-jurisdiction table lives in README "## Legal" -- keep it there, not duplicated here. Newly added + primary-source verified 2026-05-26: **EU AI Act Article 50** machine-readable marking applicable **2026-08-02** (verified against the article text); **South Korea AI Framework Act Art. 31(3)** in force since Jan 2026 (verified via Kim & Chang; Enforcement Decree accepts an invisible-watermark label); **California AB 853** (amends the CA AI Transparency Act) latent-disclosure duty operative **2026-08-02**, requiring a disclosure "permanent or extraordinarily difficult to remove" (verified against the leginfo bill text -- this is the exact disclosure our tool strips). When extending the README table, verify each date/article against the statute/bill text before committing, not against search summaries.
- **C2PA 2.4 "Durable Content Credentials" (April 2026; verified against the spec) raise the bar for metadata stripping.** 2.4 defines soft bindings (an invisible watermark or a content fingerprint) plus a server-side manifest repository and a new `c2pa.repository-receipt` assertion. Per the spec: "if a C2PA manifest is removed from an asset, but a copy of that manifest remains in a provenance store elsewhere, the manifest and asset may be matched using available soft bindings." So our local `metadata --remove` deletes the *embedded* manifest, but a fingerprint/watermark soft binding can still re-link the image to its manifest in a repository server-side. Stripping the file is becoming necessary-but-not-sufficient against durable provenance. (Our parsers target the stable embedded-manifest format documented in C2PA 2.1 §11; that format is unchanged in 2.4 -- the new pieces are repository/soft-binding infra, not the on-file box layout, so no parser change is implied.) Spec: https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html We now READ the soft-binding `alg` (`C2PA_SOFT_BINDINGS` / `soft_binding_vendors_in`) to name the forensic-watermark vendor, and locally DECODE the one open scheme, Adobe TrustMark (`trustmark_detector`); the rest (Digimarc/Imatag/Steg.AI/...) stay name-only (proprietary decoders).
- **Built 2026-05-26 (this batch):** soft-binding `alg` vendor detection; IPTC Photo Metadata 2025.1 AI-disclosure fields (`AISystemUsed` etc.); **video C2PA metadata** detect + strip for MP4/MOV/M4V (free — `isobmff.py` is format-agnostic, MP4 is ISOBMFF); Adobe TrustMark open decoder. NOT done (out of cheap reach, per the feasibility review): visible video-logo removal (needs a video frame pipeline) and audio (SynthID/ElevenLabs/Resemble/Suno all oracle-only or unmarked). The soft-binding **box detection window**: non-PNG/video detection scans the first 1 MB, so a C2PA box placed after a large `mdat` in a streaming MP4 can be missed — front-placed manifests (the common case) are caught.
- **Regulatory driver (context, not a code change):** AI-content labeling mandates are expanding, which pushes more generators toward exactly the C2PA + watermark signals we read. The full per-jurisdiction table lives in README "## Legal" -- keep it there, not duplicated here. Newly added + primary-source verified 2026-05-26: **EU AI Act Article 50** machine-readable marking applicable **2026-08-02** (verified against the article text); **South Korea AI Framework Act Art. 31(3)** in force since Jan 2026 (verified via Kim & Chang; Enforcement Decree accepts an invisible-watermark label); **California AB 853** (amends the CA AI Transparency Act) latent-disclosure duty operative **2026-08-02**, requiring a disclosure "permanent or extraordinarily difficult to remove" (verified against the leginfo bill text -- this is the exact disclosure our tool strips); **India IT Amendment Rules 2026** in force **2026-02-20** (verified via Chambers), which prominently-label + permanent-provenance-id all synthetic media AND **expressly prohibit removing/suppressing the label or metadata** -- the first major all-content removal ban outside China. **Removal liability (README "## Legal" disclaimer):** the tool is lawful general-purpose software; liability sits with the remover and is intent-gated -- downstream acts (fraud/deception/IP), plus US DMCA 17 USC 1202 (removing copyright-management info to conceal infringement), plus the removal-as-such bans in China + India. When extending the README table, verify each date/article against the statute/bill text before committing, not against search summaries.
## Known limitations
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data point (verified 2026-05-25 on a 1254x1254 gpt-image SDXL run, fp32, 20 GB MPS ceiling):** native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. Adding `enable_vae_tiling()` alone does NOT prevent it (the peak is the UNet, not the VAE). The fast Mac workarounds are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files.
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are read for detection but not yet **scrubbed** on removal.
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding), which now also drops a top-level XMP ``uuid`` box that carries an AI label (matched by AI-marker content, not by the XMP UUID, so byte-order-robust) and covers MP4/MOV/M4V/M4A by content sniff. **Non-ISOBMFF audio/video removal is via ffmpeg** (`_FFMPEG_STRIP_EXTS` -> `_strip_with_ffmpeg`): WebM/Matroska (EBML), MP3 (ID3), WAV/FLAC/OGG (RIFF/Vorbis) are stripped losslessly with `ffmpeg -map_metadata -1 -map_chapters -1 -c copy` (codec data untouched). Requires ffmpeg on PATH; raises `RuntimeError` if absent or if ffmpeg can't parse the file. Verified end-to-end (a real ffmpeg-made WAV/MP3 with a `title=Suno AI` tag -> tag gone, audio bytes preserved). **Still NOT built (deliberate):** EXIF/XMP stored as *items inside the ``meta`` box* (typical for AVIF/HEIF images) needs meta-box surgery (iinf/iloc edit + mdat splice) with corruption risk -- exiftool would do it but is a non-installed binary dep, so it stays a documented gap. **Audio watermark DETECTION (Resemble PerTh) was evaluated and NOT built (2026-05-26):** `resemble-perth`'s `PerthImplicitWatermarker.get_watermark()` returns a raw bit-array with **no presence/confidence flag** (clean audio decodes to arbitrary bits too), so reliably distinguishing watermarked-from-clean needs either Resemble's fixed payload or a confidence API -- neither is public, and there's no real Resemble sample to calibrate against. Same wall-class as the SynthID pixel detector: the decode exists, reliable presence-detection does not. (perth's top-level `PerthImplicitWatermarker` is also gated to None unless `librosa` is importable.)
- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). Authoritative confirmation: Google DeepMind's own paper "SynthID-Image: Image watermarking at internet scale" (Gowal et al., arXiv:2510.09263) states the verification service is restricted to "trusted testers" and does not release detector weights or a reproducible algorithm — so a local pixel detector is infeasible by design, not just unbuilt. https://arxiv.org/abs/2510.09263 We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. **Surface-dependent blind spot (verified 2026-05-24):** the same Google model emits different metadata per surface -- the Gemini *app* wraps outputs in Google C2PA, but the *API/playground* (AI Studio, Nano Banana / gemini-2.5-flash-image) emits the SynthID *pixel* watermark (confirmed via the Gemini-app oracle) + the visible sparkle but **no C2PA/IPTC at all**, so `synthid_source` returns None despite SynthID being present. Only the pixel oracle or the visible-sparkle detector catches those. (Meta AI is another surface mismatch: it writes the IPTC `digitalSourceType=trainedAlgorithmicMedia` marker, not C2PA and not SynthID.) Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. A from-scratch gpt-image pilot (2026-05-24) confirmed this independently: 5 independent solid-black gpt-image outputs share a near-identical fixed signature (pairwise residual correlation **0.92**, avg-template retains 97% energy), so the watermark/carrier IS strongly present and consistent on flat content — but the carrier frequencies extracted from it do NOT discriminate real content (carrier-to-random ratio: cleaned 1.86 > watermarked 1.53; a non-gpt-image image scored highest at 3.67). The signature drowns in content texture. Net: a perfectly consistent solid-color signature still yields no real-content pixel detector with magnitude/carrier methods. A corpus discrimination test (2026-05-24, `scripts/synthid_pixel_probe.py`, raw zero-mean residual NCC) independently re-confirms this: at matched resolution, SynthID positives do NOT cluster apart from negatives (within-Gemini 0.07; at 1024 px pos-vs-neg >= pos-vs-pos). The only high correlations were near-duplicate *content* (5 ChatGPT renders of one prompt at ~0.92, while a distinct ChatGPT image scored ~0 against them) — content, not a carrier. The probe is solid-fills-only and EXPERIMENTAL/DIAGNOSTIC; do not use it on real content. **Correction (deeper re-examination 2026-05-25):** the carrier IS real on solid fills — the earlier "no carrier" was a *method* artifact of using spatial / FFT-magnitude NCC, which can't see it. The carrier is a fixed *phase* at specific low frequencies, so the right metric is **per-bin phase coherence**. On 8 white `gemini-2.5-flash-image` fills (generated via the reverse-SynthID trick: identity-edit prompt "Recreate this image exactly as it is" on a synthetic pure-white PNG — this bypasses the recitation block that rejects text prompts for pure colors), phase coherence at the white carriers `(0,±7..±12,±20..±23)` = **0.86** vs **0.31** random; single-image leave-one-out phase-match **+0.83** vs real photos **-0.24**. (Black `2.5-flash` fills clip to std≈0 — SynthID can't push values below 0, so no carrier in black; the repo's dark carriers come from nano-banana-pro.) **But it does not generalize:** (a) carriers are model-version + resolution + color specific — the repo's v4 codebook (built for `gemini-3.1-flash-image-preview` + `nano-banana-pro-preview`) scores ~0.527 on my 2.5-flash white fills, indistinguishable from negatives (~0.50), i.e. carriers shift across model versions and need a per-model codebook; (b) on real content (30 `2.5-flash` images) the carrier collapses — set phase coherence at carriers 0.37 ≈ random 0.42, and the repo's v4 deteLine truncated
- **External AI-vs-real classifier models are out of scope (decided 2026-05-24).** Generic HuggingFace detectors (`Organika/sdxl-detector` Swin Transformer, `umm-maybe/AI-image-detector`, and fine-tunes) exist and report ~0.98 on their *own* SDXL-vs-real validation sets, but they are per-generator and the model cards themselves note degraded accuracy off-distribution; they are untested on gpt-image / Gemini Nano Banana (the metadata-stripped surfaces we care about), and our own light SDXL pass would likely defeat them the same way it defeats SynthID. Detection here stays local + signal-based (metadata + visible sparkle); do not add a bundled classifier dependency.
- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". Also confirmed against **OpenAI's** SynthID (2026-05-23): a fresh ChatGPT/gpt-image output read "SynthID detected" on openai.com/verify before the local SDXL run and "SynthID not detected" after (corpus regression chain: pos `4ef377bd` -> cleaned `47188e88`). The same configuration is used in raiw-app production (`fal-ai/fast-sdxl/image-to-image`, strength 0.05, steps 50, guidance 7.5, no pre-downscale). fal's own `llms.txt` for `fast-sdxl` names the base checkpoint as `stabilityai/stable-diffusion-xl-base-1.0` (verified 2026-05-25) -- the exact checkpoint the local CLI defaults to (`DEFAULT_MODEL_ID`). So the local `invisible` default is weight-for-weight identical to prod; "fast-sdxl" is fal's optimized serving, not different weights. After the native-resolution fix the local pipeline matches prod on weights + strength + steps + guidance + resolution. SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain. **Scope of the claim: defeating the SynthID verifier is NOT the same as forensic invisibility.** "Removing the Watermark Is Not Enough: Forensic Stealth in Generative-AI Watermark Removal" (arXiv:2605.09203, 2026-05) shows that six removal attacks across four families (UnMarker, CtrlRegen+, WatermarkAttacker, etc.) all leave forensic traces: independent detectors flag *removal-processed* images vs genuinely-clean ones at **>98% TPR at 1% FPR**. So our SDXL pass makes the oracle read "SynthID not detected," but the output can still be classifiable as "an image that went through a removal pipeline." Do not over-claim "indistinguishable from a real photo." https://arxiv.org/abs/2605.09203
+40 -9
View File
@@ -14,15 +14,16 @@ If this tool saves you time, consider [sponsoring its development](https://githu
## Features
- **Visible watermark removal** — Gemini / Nano Banana sparkle logo via reverse alpha blending (fast, offline, deterministic)
- **Visible watermark removal** — Gemini / Nano Banana sparkle logo (reverse alpha blending) and the Doubao "豆包AI生成" text strip (locate + mask + inpaint); fast, offline, deterministic, no GPU. `visible --mark auto` picks the right one
- **Universal region eraser (`erase`)** — remove any logo / watermark / object inside boxes you specify, regardless of position or colour. Default cv2 inpainting (CPU, instant); optional big-LaMa via onnxruntime (`lama` extra) for higher quality
- **Invisible watermark removal** — SynthID, StableSignature, TreeRing via diffusion-based regeneration (needs a local GPU, or run it with no setup on [raiw.cc](https://raiw.cc))
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests (PNG / JPEG / AVIF / HEIF / JPEG-XL), XMP DigitalSourceType
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests (PNG / JPEG / AVIF / HEIF / JPEG-XL, and **MP4 / MOV / M4V video** at the container level), XMP DigitalSourceType
- **"Made with AI" label removal** — removes the metadata that triggers AI labels on Instagram, Facebook, X (Twitter)
- **Analog Humanizer** — film grain and chromatic aberration to bypass AI image classifiers
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
- **Batch processing** — process entire directories
- **Detection** — three-stage NCC watermark detection with confidence scoring
- **Provenance detection (`identify`)** — aggregate C2PA issuer, IPTC "Made with AI", embedded SD/ComfyUI params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and the open SD/SDXL/FLUX invisible watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
## Examples
@@ -45,13 +46,13 @@ If this tool saves you time, consider [sponsoring its development](https://githu
| **xAI Grok (Aurora)** | — | — | ✅ EXIF signature scheme (no C2PA): `Signature:` blob + UUID `Artist` | Detected (`identify`); metadata strip |
| **Midjourney** | — | — | ✅ EXIF + XMP (prompt, model, seed) | Metadata strip |
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
| **Doubao** (ByteDance) / China AIGC generators | — | — | ✅ TC260 `<TC260:AIGC>` XMP label (China's mandatory AI labeling) | Metadata strip |
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 `<TC260:AIGC>` XMP label (China's mandatory AI labeling) | Locate + mask + inpaint (cv2, CPU) + metadata strip |
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
| **TreeRing** | — | ✅ Latent space watermark | — | Diffusion regeneration |
> Visible watermarks (logo overlays) are currently used only by Google Gemini / Nano Banana. Other services rely on invisible watermarks and/or metadata. Our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain.
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo) and by Doubao / China AIGC generators (the mandated "...AI生成" corner text). Both are removed deterministically on CPU. Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any colour), use the universal `erase --region` command.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, IPTC "Made with AI", the China TC260 AIGC label, embedded generation params, EXIF/XMP generator tags, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` extra) the open SD/SDXL/FLUX invisible watermark. The SynthID *pixel* watermark has no local decoder, so it is reported as a metadata proxy only.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
## How it works
@@ -73,6 +74,16 @@ A three-stage NCC (Normalized Cross-Correlation) detector finds the watermark po
**Speed**: ~0.05s per image. No GPU needed.
### Removing the Doubao "豆包AI生成" text watermark
Doubao (ByteDance) stamps every output with a light, semi-transparent "豆包AI生成" text strip in the bottom-right corner — the visible AIGC label mandated by China's TC260 standard. Unlike the fixed-size Gemini sparkle, it is a text strip that scales with image width, so we anchor a generous bottom-right box by geometry, extract the light low-saturation glyph pixels with a polarity-aware white top-hat mask, and inpaint them (cv2 Telea/NS). The mask is background-relative, so it leaves white-paper documents untouched instead of smearing their text. On dense-text backgrounds where the mask would explode, removal is skipped rather than guessed.
**Speed**: ~0.03s per image. No GPU needed. Best on photo / illustration backgrounds; on high-contrast edges a faint residue can remain (use `erase --backend lama` for neural-quality fill).
### Universal region eraser
For any visible mark the dedicated engines do not cover — a logo anywhere, any colour — `erase --region x,y,w,h` inpaints the box you specify. The default `cv2` backend is instant and dependency-free; the optional `lama` backend (big-LaMa via onnxruntime, `lama` extra, ~200 MB model downloaded on first use) gives much cleaner fills on textured regions at the cost of ~3-4 GB RAM per call.
### Removing SynthID and other invisible watermarks
Google embeds **SynthID** into every image generated by Gemini / Nano Banana. Other AI services use StableSignature, TreeRing, and similar schemes. These imperceptible frequency-domain patterns survive cropping, resizing, and JPEG compression.
@@ -159,6 +170,14 @@ After installation the `remove-ai-watermarks` command is available system-wide.
> ```bash
> pip install -e ".[detect]" # or: uv pip install -e ".[detect]"
> ```
>
> To also decode the open **Adobe TrustMark** watermark (behind Adobe Durable
> Content Credentials), install the `trustmark` extra (pulls torch and downloads
> model weights on first use):
>
> ```bash
> pip install -e ".[trustmark]" # or: uv pip install -e ".[trustmark]"
> ```
#### Invisible watermark removal
@@ -213,9 +232,15 @@ remove-ai-watermarks batch ./images/ --mode all
# of a clean origin. Add --json for machine-readable output.
remove-ai-watermarks identify image.png
# Visible watermark only (Gemini / Nano Banana sparkle) — fast, offline
# Visible watermark only — fast, offline, CPU. --mark auto (default) picks
# between the Gemini sparkle and the Doubao "豆包AI生成" text strip; force one
# with --mark gemini / --mark doubao.
remove-ai-watermarks visible image.png -o clean.png
# Erase arbitrary region(s) — universal, any logo/watermark/object, any position.
# Default cv2 inpainting (CPU). --backend lama uses big-LaMa (extra 'lama').
remove-ai-watermarks erase image.png --region 1640,1930,400,100 -o clean.png
# Invisible watermark only (SynthID etc.) — requires GPU
remove-ai-watermarks invisible image.png -o clean.png --humanize 4.0
# Runs at native resolution by default. On a very large image that OOMs the
@@ -315,7 +340,8 @@ Watermarking and provenance for AI-generated content is now regulated in several
| US (state) | CA AB 2655, TX SB 751, similar | TX SB 751 in force; **CA AB 2655 struck down** by a federal court (Aug 2025, Section 230 / First Amendment). | Content-specific (election deepfakes, sexual deepfakes). Not tool-specific. |
| US (state) | CA AB 853 (amends the California AI Transparency Act) | Core provider duties operative **2 August 2026** (delayed from 1 January 2026); large platforms 1 January 2027; capture devices 1 January 2028. | Covered providers (1M+ monthly users) must embed a latent disclosure that is "permanent or extraordinarily difficult to remove" and offer a free detection tool. Removing that disclosure is what this tool does. |
| South Korea | AI Framework Act (Basic Act on AI), Article 31 | In force since **January 2026** (one-year transition after promulgation). | Art. 31(3): AI output "difficult to distinguish from reality" must be labeled so users "clearly recognize" it; the draft Enforcement Decree accepts a machine-readable (invisible-watermark) label. Artistic/creative works get a presentation exception. |
| China | Measures for Labeling AI-Generated Content (+ GB 45438-2025) | In force since **1 September 2025**. | Mandatory explicit (visible) + implicit (metadata) labels for AI content; tampering with or removing labels is prohibited. |
| China | Measures for Labeling AI-Generated Content (+ GB 45438-2025) | In force since **1 September 2025**. | Mandatory explicit (visible) + implicit (metadata) labels across image / audio / video; tampering with, forging, or removing labels is prohibited. |
| India | IT (Intermediary Guidelines and Digital Media Ethics Code) Amendment Rules, 2026 | In force since **20 February 2026** (notified 10 February 2026). | All "synthetically generated information" must be **prominently labelled** and carry **permanent metadata / a provenance identifier**; the rules expressly **prohibit modifying, suppressing, or removing** that label or metadata. Covers image, audio, and audio-visual content. |
| UK | Online Safety Act 2023 / Ofcom guidance | In force, but **no statutory AI-provenance or watermarking obligation**. | Ofcom encourages watermarking / provenance metadata as voluntary "attribution measures"; platform duties, not user obligations. |
## Threat model
@@ -340,7 +366,12 @@ This tool is intended for legitimate purposes such as:
- Removing false-positive "Made with AI" labels from human-edited photographs.
- Security research and watermark robustness study.
Removing AI provenance markers to misrepresent AI-generated content as human-created may violate the laws above, the DMCA, and platform terms of service. Users are solely responsible for ensuring their use complies with all applicable laws. The authors do not condone use of this tool for deception, fraud, or any activity that violates applicable laws or regulations.
**Who bears the liability.** This is general-purpose software and is itself lawful to publish and run; legal responsibility attaches to the person who removes a marker and to how the result is then used, and the hinge is intent. Removing AI provenance to pass AI-generated content off as human-made, to commit fraud, to produce non-consensual deepfakes, or to conceal copyright infringement can expose the remover to liability. Two kinds of exposure are worth knowing:
- **The downstream act.** Deception, fraud, defamation, IP infringement, or breaking a platform's terms — judged by intent and harm, not by the act of editing metadata itself. In the US, the **DMCA (17 U.S.C. § 1202)** specifically bars removing "copyright management information" *with intent to conceal or enable infringement*.
- **The removal itself.** Some jurisdictions penalise tampering with the label/metadata as such, regardless of downstream use — notably **China** (Labeling Measures) and **India** (IT Amendment Rules 2026), which expressly prohibit removing or suppressing the AI label and provenance metadata. The US **COPIED Act** would do the same if enacted.
Legitimate uses — publishing your own work, privacy (stripping metadata that leaks an account identifier), security / robustness research, or removing a false-positive "Made with AI" label from a human-edited photograph — are generally lawful. Users are solely responsible for ensuring their use complies with all applicable laws. The authors do not condone use of this tool for deception, fraud, or any activity that violates applicable laws or regulations. None of this is legal advice.
## License
+78
View File
@@ -0,0 +1,78 @@
# Doubao visible watermark capture
Goal: capture the Doubao "豆包AI生成" visible watermark over known flat backgrounds so we can
build a per-pixel alpha map and a reverse-alpha-blend remover, the same way the Gemini sparkle
engine works (`src/remove_ai_watermarks/gemini_engine.py`).
## What we already know (verified from prior art, 2026-05-26)
- Blend model: **alpha compositing with a white logo** `watermarked = a*logo + (1-a)*original`,
`logo = (255,255,255)`. Inversion: `original = (watermarked - a*logo) / (1-a)`.
Confirmed by two independent sources (an open-source remover's algorithm doc + aiwatermarkremover.dev,
both say "alpha map"). One commercial blog (pixelcleanai) claims "screen blend" instead; the gray
capture below settles it empirically.
- Position: **bottom-right corner**, small margins (right ~8-20px, bottom ~5px), scales with image size.
Confirmed by our sample `data/samples/doubao-1.png` (2048x2048) plus three sources.
- Size **scales with resolution**. Third-party numbers (~90x18 at <=1024, ~180x40 at >1024) are
approximate and calibrated for ~1024-1280 outputs; at 2048 the strip is much larger. A shipped
third-party alpha map is only 120x20, too small for our 2K/4K target -> capture fresh.
- In practice clean inversion leaves residue on textured backgrounds, so the remover pairs the alpha
map with inpainting (our Gemini engine already does gradient-masked inpainting for residual edges).
## Use doubao.com specifically
The "豆包AI生成" mark is Doubao's. Jimeng / Dreamina use a different mark. Generate on doubao.com so
the captured template matches our target.
## How to capture (image-edit path, most reliable)
For each seed in `seeds/`:
1. Open Doubao image generation, use the image-edit / reference mode, upload the seed.
2. Prompt (Chinese preferred):
`请完全按照原图重新生成这张图片,保持完全一致,不要添加或修改任何内容`
(English: `Recreate this image exactly as it is, keep it identical, do not add or change anything`)
3. Download the ORIGINAL output file (not a screenshot). Do not crop / edit / re-save.
Prior art confirms uploading a pure-black image and letting Doubao stamp it works.
If edit mode is unavailable and text-to-image refuses a solid color, fall back to generating 10-12
normal-content images at one fixed resolution; the mark is the only constant across them and can be
extracted by per-pixel min/median.
## What to capture (priority top to bottom)
| Aspect | black | white | gray128 | why |
|--------|-------|-------|---------|-----|
| 1:1 | 3 | 1 | 1 | primary alpha map + confirm the stamp is pixel-identical across runs + settle blend mode |
| 16:9 | 2 | 1 | 1 | anchor rule in landscape |
| 9:16 | 2 | 1 | 1 | anchor rule in portrait |
| 4:3, 3:4 | 1 each | - | - | optional, refines anchor rule |
- 3 blacks on 1:1: if the first two are byte-identical in the watermark region, the third is optional.
- gray128 is the blend-mode test: predict the gray result from the black capture under alpha vs screen;
whichever matches the real gray output is the true blend.
- If the UI offers multiple output resolutions (1K / 2K / 4K), capture one black per resolution on 1:1 -
needed to learn how the watermark scales.
- Also grab 3-5 normal-content images on 1:1 for end-to-end removal validation.
## Hygiene
- Original download, never a screenshot. PNG preferred; if Doubao only gives JPEG, note it.
- No crop / edit / re-save. Default settings, watermark left ON.
## Naming, drop into `captures/`
```
doubao_black_1x1_1.png
doubao_white_1x1_1.png
doubao_gray128_1x1_1.png
doubao_black_16x9_1.png
doubao_content_1x1_1.png
```
## Also report back
1. Which resolutions and aspect ratios the Doubao UI actually offers.
2. Whether there is a watermark on/off toggle in the UI.
3. Download format (PNG or JPEG).
+17 -2
View File
@@ -1,6 +1,6 @@
[project]
name = "remove-ai-watermarks"
version = "0.5.6"
version = "0.6.4"
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
readme = "README.md"
requires-python = ">=3.10"
@@ -36,6 +36,21 @@ gpu = [
detect = [
"invisible-watermark>=0.2.0",
]
# Adobe TrustMark decoder -- the open, keyless watermark behind Adobe Durable
# Content Credentials (soft-binding alg ``com.adobe.trustmark.P``). Optional
# because it pulls torch and downloads model weights on first use. identify()
# guards the import and skips the TrustMark signal when absent.
trustmark = [
"trustmark>=0.8.0",
]
# Universal region eraser backend -- big-LaMa via onnxruntime (Carve/LaMa-ONNX,
# Apache-2.0). CPU, no torch. Model (~200 MB) is downloaded on first use and
# cached by huggingface_hub; it is never bundled in this repo. The default cv2
# eraser backend needs none of this.
lama = [
"onnxruntime>=1.16.0",
"huggingface-hub>=0.20.0",
]
dev = [
"pytest>=8.0.0",
"pytest-cov>=4.1.0",
@@ -43,7 +58,7 @@ dev = [
"pyright>=1.1.0",
"invisible-watermark>=0.2.0",
]
all = ["remove-ai-watermarks[gpu,detect,dev]"]
all = ["remove-ai-watermarks[gpu,detect,trustmark,lama,dev]"]
# diffusers 0.38.0 (security fix for GHSA-98h9-4798-4q5v) declares a dependency
# on safetensors>=0.8.0rc0 — a pre-release. Allow pre-releases globally so the
+1 -1
View File
@@ -1,3 +1,3 @@
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
__version__ = "0.5.6"
__version__ = "0.6.4"
+177 -4
View File
@@ -12,7 +12,7 @@ import json
import logging
import time
from pathlib import Path
from typing import TYPE_CHECKING
from typing import TYPE_CHECKING, Literal
import click
from rich.console import Console
@@ -25,7 +25,7 @@ from remove_ai_watermarks import __version__
if TYPE_CHECKING:
import numpy as np
from remove_ai_watermarks.gemini_engine import DetectionResult
from remove_ai_watermarks.gemini_engine import DetectionResult, GeminiEngine
console = Console()
@@ -130,6 +130,72 @@ def _write_bgr_with_alpha(
cv2.imwrite(str(path), bgra)
def _run_doubao_if_selected(
ctx: click.Context,
image: np.ndarray,
alpha: np.ndarray | None,
output: Path,
mark: str,
gemini_engine: GeminiEngine,
detect: bool,
detect_threshold: float,
inpaint_method: str,
strip_metadata: bool,
) -> bool:
"""Run the Doubao text-strip removal path when it is the selected mark.
Returns True when this path handled the image (caller should stop). In
``auto`` mode the Doubao detector competes with the Gemini detector and wins
only when it is both positive and at least as confident.
"""
from remove_ai_watermarks.doubao_engine import DoubaoEngine
doubao = DoubaoEngine()
d_det = doubao.detect(image)
if mark == "auto":
g_det = gemini_engine.detect_watermark(image)
use_doubao = d_det.detected and d_det.confidence >= g_det.confidence
console.print(
f" [dim]Mark auto:[/] gemini={g_det.confidence:.2f} doubao={d_det.confidence:.2f} "
f"-> {'doubao' if use_doubao else 'gemini'}"
)
else:
use_doubao = mark == "doubao"
if not use_doubao:
return False
if detect and not d_det.detected and d_det.confidence < detect_threshold:
console.print(
f" [yellow]⚠[/] Doubao mark not detected [dim](coverage {d_det.coverage:.1%}). "
f"Use --no-detect to force.[/]"
)
raise SystemExit(0)
method: Literal["telea", "ns"] = "ns" if inpaint_method == "ns" else "telea"
t0 = time.monotonic()
with console.status("[cyan]Removing Doubao watermark…[/]"):
result = doubao.remove_watermark(image, inpaint_method=method)
elapsed = time.monotonic() - t0
output.parent.mkdir(parents=True, exist_ok=True)
_write_bgr_with_alpha(output, result, alpha, clear_region=d_det.region)
if strip_metadata:
try:
from remove_ai_watermarks.metadata import remove_ai_metadata
remove_ai_metadata(output, output)
except Exception as e:
if ctx.obj.get("verbose"):
console.print(f" [yellow]⚠[/] Failed to strip metadata: {e}")
size_kb = output.stat().st_size / 1024
console.print(f" [green]✓[/] Doubao mark removed → {output} [dim]({size_kb:.0f} KB, {elapsed:.2f}s)[/]")
return True
# ── Main group ───────────────────────────────────────────────────────
@@ -167,6 +233,12 @@ def main(ctx: click.Context, verbose: bool) -> None:
@click.option("--inpaint-strength", type=float, default=0.85, help="Inpainting blend strength (0.0-1.0).")
@click.option("--detect/--no-detect", default=True, help="Detect watermark before removal.")
@click.option("--detect-threshold", type=float, default=0.25, help="Detection confidence threshold.")
@click.option(
"--mark",
type=click.Choice(["auto", "gemini", "doubao"]),
default="auto",
help="Which visible mark to target. auto picks the stronger of the two detectors.",
)
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
@click.pass_context
def cmd_visible(
@@ -178,11 +250,14 @@ def cmd_visible(
inpaint_strength: float,
detect: bool,
detect_threshold: float,
mark: str,
strip_metadata: bool,
) -> None:
"""Remove visible Gemini watermark (sparkle logo) from an image.
"""Remove a visible AI watermark from an image.
Uses reverse alpha blending — fast, deterministic, offline.
Targets the Gemini sparkle logo (reverse alpha blending) or the Doubao
"豆包AI生成" text strip (locate -> mask -> inpaint). Fast, deterministic,
offline. ``--mark auto`` picks whichever detector fires stronger.
"""
from remove_ai_watermarks.gemini_engine import GeminiEngine
@@ -203,6 +278,12 @@ def cmd_visible(
h, w = image.shape[:2]
console.print(f" [dim]Input:[/] {source.name} ({w}x{h})")
# Resolve which visible mark to target, then run the Doubao path if chosen.
if _run_doubao_if_selected(
ctx, image, alpha, output, mark, engine, detect, detect_threshold, inpaint_method, strip_metadata
):
return
# Detection (we always detect softly, to find dynamic region for inpainting)
with console.status("[cyan]Detecting watermark…[/]"):
det = engine.detect_watermark(image)
@@ -256,6 +337,98 @@ def cmd_visible(
console.print(f" [green]✓[/] Saved: {output} [dim]({size_kb:.0f} KB, {elapsed:.2f}s)[/]")
# ── Universal region eraser ─────────────────────────────────────────
def _parse_region(spec: str) -> tuple[int, int, int, int]:
"""Parse an ``x,y,w,h`` region string into a 4-int tuple."""
parts = spec.replace(" ", "").split(",")
if len(parts) != 4:
raise click.BadParameter(f"region must be 'x,y,w,h', got: {spec!r}")
try:
x, y, w, h = (int(p) for p in parts)
except ValueError as e:
raise click.BadParameter(f"region values must be integers: {spec!r}") from e
if w <= 0 or h <= 0:
raise click.BadParameter(f"region width/height must be positive: {spec!r}")
return x, y, w, h
@main.command("erase")
@click.argument("source", type=click.Path(exists=True, path_type=Path))
@click.option("--region", "regions", multiple=True, required=True, help="x,y,w,h box to erase (repeatable).")
@click.option(
"-o", "--output", type=click.Path(path_type=Path), default=None, help="Output path (default: <source>_clean.<ext>)."
)
@click.option(
"--backend",
type=click.Choice(["cv2", "lama"]),
default="cv2",
help="Inpaint backend. cv2: instant, no deps. lama: onnxruntime big-LaMa, better quality (extra 'lama').",
)
@click.option("--inpaint-method", type=click.Choice(["telea", "ns"]), default="telea", help="cv2 inpaint method.")
@click.option("--dilate", type=int, default=3, help="Grow the box by this many px before inpainting.")
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
@click.pass_context
def cmd_erase(
ctx: click.Context,
source: Path,
regions: tuple[str, ...],
output: Path | None,
backend: str,
inpaint_method: str,
dilate: int,
strip_metadata: bool,
) -> None:
"""Erase arbitrary region(s) from an image via inpainting.
Universal and position-agnostic: removes any logo / watermark / object inside
the boxes you pass, regardless of colour or location. Runs on CPU. Use this
for marks the dedicated ``visible`` engines (Gemini, Doubao) do not cover.
"""
from remove_ai_watermarks.region_eraser import erase
_banner()
source = _validate_image(source)
if output is None:
output = source.with_stem(source.stem + "_clean")
boxes = [_parse_region(r) for r in regions]
image, alpha = _read_bgr_and_alpha(source)
if image is None:
console.print(f"[red]Error:[/] Failed to read image: {source}")
raise SystemExit(1)
h, w = image.shape[:2]
console.print(f" [dim]Input:[/] {source.name} ({w}x{h}) [dim]{len(boxes)} region(s), backend={backend}[/]")
t0 = time.monotonic()
method: Literal["telea", "ns"] = "ns" if inpaint_method == "ns" else "telea"
try:
with console.status(f"[cyan]Erasing ({backend})…[/]"):
result = erase(image, boxes=boxes, backend=backend, dilate=dilate, cv2_method=method)
except RuntimeError as e:
console.print(f" [red]Error:[/] {e}")
raise SystemExit(1) from e
elapsed = time.monotonic() - t0
output.parent.mkdir(parents=True, exist_ok=True)
clear = boxes[0] if len(boxes) == 1 else None
_write_bgr_with_alpha(output, result, alpha, clear_region=clear)
if strip_metadata:
try:
from remove_ai_watermarks.metadata import remove_ai_metadata
remove_ai_metadata(output, output)
except Exception as e:
if ctx.obj.get("verbose"):
console.print(f" [yellow]⚠[/] Failed to strip metadata: {e}")
size_kb = output.stat().st_size / 1024
console.print(f" [green]✓[/] Erased {len(boxes)} region(s) → {output} [dim]({size_kb:.0f} KB, {elapsed:.2f}s)[/]")
# ── Invisible watermark removal ─────────────────────────────────────
+245
View File
@@ -0,0 +1,245 @@
"""Doubao visible watermark removal engine.
Doubao (ByteDance) stamps every generated image with a visible "豆包AI生成"
(Doubao AI generated) text strip in the bottom-right corner. This is the
explicit AIGC label mandated by China's TC260 standard, rendered as a
near-white / light-gray, low-saturation text overlay.
Unlike the Gemini sparkle (a fixed square logo removed by reverse alpha
blending against a captured alpha map), the Doubao mark is a text strip whose
exact alpha map we do not yet have. This engine therefore removes it by:
locate -> mask -> inpaint
1. Locate: the mark scales with image WIDTH and sits in the bottom-right at a
fixed margin, so we anchor a generous box there (geometry only -- no bundled
template). Constants below are derived from measured Doubao output.
2. Mask: within the box, extract the light, low-saturation glyph pixels with a
polarity-aware rule (the mark is brighter than dark backgrounds and a
distinct off-white gray against light backgrounds).
3. Inpaint: cv2 inpainting (TELEA / NS) reconstructs the covered pixels.
This is fast, offline, deterministic, and needs no GPU. A future upgrade path
is per-pixel reverse alpha blending once a Doubao alpha map is captured on a
controlled black background (see data/doubao_capture/), which would recover the
true pixels instead of hallucinating them -- the same approach as the Gemini
engine.
"""
from __future__ import annotations
import logging
from dataclasses import dataclass
from typing import TYPE_CHECKING, Literal
import cv2
import numpy as np
if TYPE_CHECKING:
from pathlib import Path
from numpy.typing import NDArray
logger = logging.getLogger(__name__)
# Geometry as a fraction of image WIDTH. The Doubao mark scales with width and
# is anchored bottom-right. The box is intentionally generous (the glyph mask
# tightens it); values cover measured outputs across resolutions and aspect
# ratios (square 2048, portrait, ultra-wide). Margins are width-relative too.
WM_WIDTH_FRAC = 0.185
WM_HEIGHT_FRAC = 0.065
MARGIN_RIGHT_FRAC = 0.012
MARGIN_BOTTOM_FRAC = 0.014
# Glyph appearance: the label is a low-saturation light gray, rendered brighter
# than the surrounding content (the common case: a generated photo/illustration).
# We detect it as a local bright feature (white top-hat: brighter than a blurred
# local background) intersected with the grayish + minimum-brightness tests.
# This is polarity-correct for bright-on-darker backgrounds and, crucially,
# leaves white-paper documents untouched (there the mark is not brighter than
# its surroundings, so nothing is masked rather than damaging the document text).
MAX_SATURATION = 55 # max channel spread to count a pixel as "grayish"
LOGO_MIN_LUMA = 150 # glyphs are at least this bright in absolute terms
TOPHAT_DELTA = 12 # glyph must exceed the local background by this many levels
# Detection: a genuine label fills a meaningful fraction of the box. Measured
# coverage is >=0.20 on real Doubao outputs; random/textured corners stay <=0.06
# on large images but can spike to ~0.15 on tiny ones (small box -> high variance),
# so the threshold sits above that spike and below the real-mark floor.
DETECT_MIN_COVERAGE = 0.16
# Safety: a text strip fills a modest slice of the (generous) box. When the box
# is over a dense-text / document background the mask explodes and cv2 inpainting
# would smear the real content. Above this coverage we refuse to inpaint and
# leave the image untouched -- that hard case needs the neural path, not a guess.
MAX_INPAINT_COVERAGE = 0.50
@dataclass(frozen=True)
class DoubaoLocation:
"""Located watermark box (bottom-right), in absolute pixel coordinates."""
x: int
y: int
w: int
h: int
is_fallback: bool = True # geometry anchor (no template match) -> always True for now
@property
def bbox(self) -> tuple[int, int, int, int]:
return self.x, self.y, self.w, self.h
@dataclass
class DoubaoDetection:
"""Result of visible Doubao watermark detection."""
detected: bool = False
confidence: float = 0.0
region: tuple[int, int, int, int] = (0, 0, 0, 0)
coverage: float = 0.0 # fraction of the box occupied by glyph pixels
class DoubaoEngine:
"""Remove the visible Doubao "豆包AI生成" watermark (locate -> mask -> inpaint)."""
def __init__(
self,
*,
width_frac: float = WM_WIDTH_FRAC,
height_frac: float = WM_HEIGHT_FRAC,
margin_right_frac: float = MARGIN_RIGHT_FRAC,
margin_bottom_frac: float = MARGIN_BOTTOM_FRAC,
) -> None:
self.width_frac = width_frac
self.height_frac = height_frac
self.margin_right_frac = margin_right_frac
self.margin_bottom_frac = margin_bottom_frac
# ── Locate ────────────────────────────────────────────────────────
def locate(self, image: NDArray) -> DoubaoLocation:
"""Anchor the watermark box in the bottom-right corner by geometry."""
h, w = image.shape[:2]
wm_w = max(40, int(w * self.width_frac))
wm_h = max(16, int(w * self.height_frac))
margin_r = max(4, int(w * self.margin_right_frac))
margin_b = max(4, int(w * self.margin_bottom_frac))
x = max(0, w - margin_r - wm_w)
y = max(0, h - margin_b - wm_h)
wm_w = min(wm_w, w - x)
wm_h = min(wm_h, h - y)
return DoubaoLocation(x=x, y=y, w=wm_w, h=wm_h, is_fallback=True)
# ── Mask ──────────────────────────────────────────────────────────
def extract_mask(self, image: NDArray, loc: DoubaoLocation) -> NDArray:
"""Build a full-image uint8 mask (255 = watermark glyph) for the box.
Polarity-aware: the mark is a light, low-saturation gray. On a dark
background it is the bright region; on a light background it is the
off-white gray below paper-white. Both cases are captured by the logo
luminance band intersected with the grayish constraint, plus a
brighter-than-local-background test on dark backgrounds.
"""
h, w = image.shape[:2]
x, y, bw, bh = loc.bbox
roi = image[y : y + bh, x : x + bw].astype(np.float32)
luma = roi.mean(axis=2)
sat = roi.max(axis=2) - roi.min(axis=2)
grayish = sat < MAX_SATURATION
# Local background model: a strong Gaussian blur (sigma ~ box height)
# approximates the content under the glyphs. The white top-hat
# (luma - local_bg) lights up bright thin strokes regardless of the
# absolute background level.
sigma = max(4.0, bh * 0.4)
local_bg = cv2.GaussianBlur(luma, (0, 0), sigmaX=sigma, sigmaY=sigma)
tophat = luma - local_bg
cand = grayish & (tophat > TOPHAT_DELTA) & (luma > LOGO_MIN_LUMA)
glyph = cand.astype(np.uint8) * 255
# Connect glyph parts, then drop isolated specks (5x5 open clears the
# scattered grayish pixels that random/textured corners produce).
glyph = cv2.morphologyEx(glyph, cv2.MORPH_CLOSE, np.ones((5, 5), np.uint8))
glyph = cv2.morphologyEx(glyph, cv2.MORPH_OPEN, np.ones((5, 5), np.uint8))
mask = np.zeros((h, w), np.uint8)
mask[y : y + bh, x : x + bw] = glyph
return mask
# ── Detect ────────────────────────────────────────────────────────
def detect(self, image: NDArray) -> DoubaoDetection:
"""Detect the visible Doubao mark by glyph coverage in the corner box.
Heuristic: a genuine label fills a meaningful fraction of the box with
text-like glyph pixels. Coverage maps to a confidence score.
"""
det = DoubaoDetection()
if image is None or image.size == 0:
return det
loc = self.locate(image)
mask = self.extract_mask(image, loc)
x, y, bw, bh = loc.bbox
box = mask[y : y + bh, x : x + bw]
coverage = float((box > 0).sum()) / float(max(1, bw * bh))
det.region = loc.bbox
det.coverage = coverage
# Map coverage to a 0-1 confidence: ~0.06 (noise floor) -> 0, ~0.26 -> 1.
det.confidence = float(max(0.0, min(1.0, (coverage - 0.06) / 0.20)))
det.detected = coverage >= DETECT_MIN_COVERAGE
logger.debug("Doubao detect: coverage=%.3f conf=%.3f", coverage, det.confidence)
return det
# ── Remove ────────────────────────────────────────────────────────
def remove_watermark(
self,
image: NDArray,
*,
inpaint_method: Literal["telea", "ns"] = "telea",
inpaint_radius: int = 6,
dilate: int = 3,
) -> NDArray:
"""Remove the visible Doubao watermark by inpainting the glyph mask.
Returns an unmodified copy when no glyph pixels are found (so we never
smear a clean corner). ``dilate`` grows the mask to cover anti-aliased
glyph edges before inpainting.
"""
if image is None or image.size == 0:
return image
loc = self.locate(image)
mask = self.extract_mask(image, loc)
if not mask.any():
logger.debug("Doubao remove: no glyph pixels found; returning copy")
return image.copy()
x, y, bw, bh = loc.bbox
coverage = float((mask[y : y + bh, x : x + bw] > 0).sum()) / float(max(1, bw * bh))
if coverage > MAX_INPAINT_COVERAGE:
logger.warning(
"Doubao remove: box coverage %.2f exceeds %.2f (dense-text/document "
"background); leaving image untouched to avoid smearing content",
coverage,
MAX_INPAINT_COVERAGE,
)
return image.copy()
if dilate > 0:
k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * dilate + 1, 2 * dilate + 1))
mask = cv2.dilate(mask, k)
flag = cv2.INPAINT_TELEA if inpaint_method == "telea" else cv2.INPAINT_NS
return cv2.inpaint(image, mask, inpaint_radius, flag)
def load_image_bgr(path: str | Path) -> NDArray:
"""Read an image as BGR ndarray (helper for scripts/tests)."""
img = cv2.imread(str(path), cv2.IMREAD_COLOR)
if img is None:
raise FileNotFoundError(f"Failed to read image: {path}")
return img
+104 -5
View File
@@ -26,13 +26,15 @@ from remove_ai_watermarks.metadata import (
AI_METADATA_KEYS,
AIGC_MARKERS,
C2PA_UUID,
IPTC_AI_FIELD_MARKERS,
IPTC_AI_MARKERS,
aigc_label,
exif_generator,
get_ai_metadata,
iptc_ai_system,
xai_signature,
)
from remove_ai_watermarks.noai.c2pa import extract_c2pa_info
from remove_ai_watermarks.noai.c2pa import cbor_text_after, extract_c2pa_info, soft_binding_vendors_in
from remove_ai_watermarks.noai.constants import C2PA_AI_TOOLS, C2PA_ISSUERS
if TYPE_CHECKING:
@@ -125,6 +127,44 @@ def _ai_tools_in(data: bytes) -> list[str]:
return sorted({name for sig, name in C2PA_AI_TOOLS.items() if sig in data})
# Distinctive C2PA device/camera tokens (cert CN, cert org, or claim-generator
# substrings) scanned in the manifest bytes -> platform. This is more reliable
# than mapping an issuer name (which also matches incidental mentions: a
# timestamp authority like "Truepic" in a Leica chain, an XMP-toolkit "Adobe"
# string in a Nikon file, or "Google" in a Pixel camera's cert -- all verified
# on real samples), and more robust than parsing the claim generator (which
# lives under varying CBOR keys, e.g. `claim_generator` vs `claim_generator_info`,
# and is absent on the Pixel sample where only the cert CN "Pixel Camera"
# identifies it). Camera C2PA marks CAPTURE authenticity, not AI, so these never
# assert is_ai on their own (the verdict still comes from the digital-source-type:
# the Pixel sample carries `computationalCapture`, not `trainedAlgorithmicMedia`).
# Only tokens verified against a real signed file are listed (Leica, Nikon,
# Truepic, Google Pixel); add Sony/Canon/Samsung/Bria as real samples are captured.
_DEVICE_C2PA_PLATFORM: tuple[tuple[bytes, str], ...] = (
(b"lc_c2pa", "Leica (camera, C2PA capture)"),
(b"Leica Camera", "Leica (camera, C2PA capture)"),
(b"NIKON", "Nikon (camera, C2PA capture)"),
(b"Pixel Camera", "Google Pixel (camera, C2PA capture)"),
# Sony uses its own ``sony.*`` C2PA assertion namespace (sony.sig / sony.cert);
# match that, NOT bare "Sony" (which is an EXIF Make on countless photos).
# Verified on a real Sony-signed file (Sony PXW-Z300, signer "Sony Corporation").
(b"sony.sig", "Sony (camera, C2PA capture)"),
(b"sony.cert", "Sony (camera, C2PA capture)"),
# "Truepic_Lens" (from the Lens SDK claim generator), NOT bare "Truepic" --
# Truepic is a C2PA signing authority whose name appears in the trust chain
# of unrelated manifests (e.g. OpenAI), so the bare token mis-attributes.
(b"Truepic_Lens", "Truepic Lens (verified capture)"),
)
def _device_platform(head: bytes) -> str | None:
"""Map a distinctive C2PA device/camera token in the manifest bytes to a platform."""
for token, platform in _DEVICE_C2PA_PLATFORM:
if token in head:
return platform
return None
def _attribute_platform(issuers: list[str]) -> str | None:
"""Map a set of C2PA issuer names to a human-readable generating platform."""
joined = " ".join(issuers)
@@ -162,6 +202,17 @@ def _invisible_watermark(image_path: Path) -> str | None:
return detect_invisible_watermark(image_path)
def _trustmark(image_path: Path) -> str | None:
"""Adobe TrustMark scheme name or None.
Optional: needs the ``trustmark`` decoder (extra ``trustmark``). Returns None
if it is not installed or no TrustMark watermark decodes.
"""
from remove_ai_watermarks.trustmark_detector import detect_trustmark
return detect_trustmark(image_path)
def identify(image_path: Path, *, check_visible: bool = True, check_invisible: bool = True) -> ProvenanceReport:
"""Identify an image's origin platform and watermark inventory.
@@ -192,12 +243,23 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
# ── C2PA Content Credentials ────────────────────────────────────
has_c2pa = bool(info) or b"c2pa" in head.lower() or C2PA_UUID in head
issuers = [info["issuer"]] if info.get("issuer") else _issuers_in(head)
platform = _attribute_platform(issuers) if has_c2pa else None
c2pa_is_ai = "trainedAlgorithmicMedia" in info.get("source_type", "") or any(
m in head for m in (b"trainedAlgorithmicMedia", b"compositeWithTrainedAlgorithmicMedia")
)
# Generator: structured for PNG, binary-scanned for other containers.
generator = info.get("claim_generator") or (", ".join(tools) if (tools := _ai_tools_in(head)) else None)
# Generator string (for the signal detail): structured for PNG, CBOR-scanned
# for other containers. Best-effort -- some manifests key it as
# `claim_generator_info` (Pixel), so this can be None even when a device is
# identified by `_device_platform`.
generator = (
info.get("claim_generator")
or cbor_text_after(head, b"claim_generator")
or (", ".join(tools) if (tools := _ai_tools_in(head)) else None)
)
# Platform: a distinctive device/camera token in the manifest wins (it is the
# signer/producer), with the issuer byte-scan only as fallback. The issuer
# scan alone mis-attributed real samples (Leica->Truepic timestamp authority,
# Nikon->Adobe namespace, Pixel->Google Gemini) -- the device scan fixes that.
platform = (_device_platform(head) or _attribute_platform(issuers)) if has_c2pa else None
if has_c2pa:
detail = ", ".join(filter(None, [", ".join(issuers), generator, info.get("source_type")]))
signals.append(Signal("c2pa", detail or "C2PA manifest present", "high"))
@@ -213,6 +275,14 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
if "OpenAI" in (" ".join(issuers) + synthid):
caveats.append(_OPENAI_CAVEAT)
# ── C2PA soft-binding: a named forensic/third-party watermark vendor ─
# (Adobe TrustMark, Digimarc, Imatag, ...). Present in the manifest even when
# the watermark itself can't be decoded; names whose watermark stamped the pixels.
soft_binding = meta.get("soft_binding") or (", ".join(v) if (v := soft_binding_vendors_in(head)) else None)
if soft_binding:
signals.append(Signal("soft_binding", f"C2PA soft binding: {soft_binding}", "high"))
watermarks.append(f"Forensic watermark soft binding ({soft_binding})")
# ── IPTC "Made with AI" (Meta etc.), only meaningful without C2PA ─
iptc = any(m in head for m in IPTC_AI_MARKERS)
if iptc and not has_c2pa:
@@ -222,6 +292,18 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
if platform is None:
platform = "Made-with-AI tag (e.g. Meta AI); platform not specified"
# ── IPTC 2025.1 AI-disclosure fields (Iptc4xmpExt:AISystemUsed etc.) ─
iptc_ai = any(m in head for m in IPTC_AI_FIELD_MARKERS)
if iptc_ai:
system = iptc_ai_system(image_path)
named = bool(system) and system != "fields present"
signals.append(
Signal("iptc_ai_system", f"IPTC AI disclosure ({system})" if named else "IPTC AI disclosure fields", "high")
)
watermarks.append(f"IPTC 2025.1 AI disclosure ({system})" if named else "IPTC 2025.1 AI disclosure fields")
if platform is None and named:
platform = f"{system} (IPTC AISystemUsed)"
# ── China TC260 AIGC label (Doubao and other China-served gens) ──
aigc = any(m in head for m in AIGC_MARKERS)
if aigc:
@@ -266,12 +348,29 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
if platform is None:
platform = f"{scheme} (open DWT-DCT watermark)"
# ── Adobe TrustMark invisible watermark (open decoder, no key) ───
# The watermark behind Adobe Durable Content Credentials. Decoded locally,
# but it binds provenance for human-authored content too, so it enriches the
# watermark inventory without by itself asserting AI origin.
if check_invisible and (tm_scheme := _trustmark(image_path)) is not None:
signals.append(Signal("trustmark", tm_scheme, "high"))
watermarks.append(f"Adobe TrustMark invisible watermark ({tm_scheme})")
if platform is None:
platform = "Adobe (TrustMark / Content Credentials)"
# ── Verdict so far (metadata + embedded watermark) ──────────────
invisible_wm = any(s.name == "invisible_watermark" for s in signals)
exif_gen = any(s.name == "exif_generator" for s in signals)
xai_sig = any(s.name == "xai_signature" for s in signals)
ai_from_metadata = bool(
(has_c2pa and (c2pa_is_ai or synthid)) or iptc or aigc or local_keys or invisible_wm or exif_gen or xai_sig
(has_c2pa and (c2pa_is_ai or synthid))
or iptc
or iptc_ai
or aigc
or local_keys
or invisible_wm
or exif_gen
or xai_sig
)
# ── Visible Gemini sparkle (fallback for stripped-metadata case) ─
+110 -9
View File
@@ -74,6 +74,29 @@ IPTC_AI_MARKERS: tuple[bytes, ...] = (
b"compositeWithTrainedAlgorithmicMedia",
)
# IPTC Photo Metadata 2025.1 (published 2025-11-27) added explicit AI-disclosure
# XMP properties in the Iptc4xmpExt namespace. Their mere presence is an AI
# signal; ``AISystemUsed`` additionally carries the generator name. Property
# tokens verified against the IPTC 2025.1 specification.
IPTC_AI_FIELD_MARKERS: tuple[bytes, ...] = (
b"AISystemUsed",
b"AISystemVersionUsed",
b"AIPromptInformation",
b"AIPromptWriterName",
)
# ISOBMFF containers whose AI-provenance boxes ``remove_ai_metadata`` strips at
# the container level (image, video, audio -- all ISOBMFF). A content sniff
# (``ftyp``) is also accepted, so this is a fast-path hint, not the sole gate.
_ISOBMFF_EXTS: frozenset[str] = frozenset({".avif", ".heif", ".heic", ".jxl", ".mp4", ".mov", ".m4v", ".m4a"})
# Non-ISOBMFF audio/video the ISOBMFF box walker can't reach (EBML / framed /
# RIFF / Vorbis). remove_ai_metadata strips their container metadata losslessly
# via ffmpeg (`-c copy`), so it needs ffmpeg on PATH for these.
_FFMPEG_STRIP_EXTS: frozenset[str] = frozenset(
{".webm", ".mkv", ".mka", ".mp3", ".wav", ".flac", ".ogg", ".oga", ".opus", ".aac"}
)
# China's mandatory AI-content labeling (TC260, the national cybersecurity
# standards committee). AI generators serving China embed an XMP block in the
# TC260 namespace -- ``<TC260:AIGC>{"Label":"1",...}``. Doubao (ByteDance) uses
@@ -155,6 +178,9 @@ def has_ai_metadata(image_path: Path) -> bool:
return True
if any(marker in data for marker in IPTC_AI_MARKERS):
return True
# IPTC 2025.1 AI-disclosure XMP properties (their presence flags AI content).
if any(marker in data for marker in IPTC_AI_FIELD_MARKERS):
return True
# xAI / Grok: no C2PA/IPTC/XMP -- only the EXIF Signature + UUID-Artist pair.
return xai_signature(image_path)
@@ -183,6 +209,26 @@ def aigc_label(image_path: Path) -> dict[str, str] | None:
return {str(k): str(v) for k, v in parsed.items()} if isinstance(parsed, dict) else None
def iptc_ai_system(image_path: Path) -> str | None:
"""Return an IPTC 2025.1 AI-disclosure note if the file carries those XMP
properties, else None.
IPTC Photo Metadata 2025.1 added ``Iptc4xmpExt`` AI-disclosure properties
(see ``IPTC_AI_FIELD_MARKERS``); their presence alone flags AI content, and
``AISystemUsed`` names the generator. Returns the ``AISystemUsed`` value when
extractable, otherwise the literal ``"fields present"``. Container-agnostic
raw-byte scan; handles both XMP element and attribute serializations.
"""
with open(image_path, "rb") as f:
data = f.read(1024 * 1024)
if not any(marker in data for marker in IPTC_AI_FIELD_MARKERS):
return None
match = re.search(rb"AISystemUsed[=:\s]*[\"'>]\s*([^<\"']{1,120})", data)
if match and (value := match.group(1).decode("utf-8", "replace").strip()):
return value
return "fields present"
def synthid_source(image_path: Path) -> str | None:
"""Return the vendor name(s) if the image carries a SynthID pixel watermark.
@@ -380,7 +426,7 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
"""
from PIL import Image
from remove_ai_watermarks.noai.c2pa import extract_c2pa_info, synthid_verdict
from remove_ai_watermarks.noai.c2pa import extract_c2pa_info, soft_binding_vendors_in, synthid_verdict
result: dict[str, str] = {}
@@ -410,14 +456,21 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
"source_type",
"actions",
"synthid_watermark",
"soft_binding",
):
if key in c2pa:
result.setdefault(key, str(c2pa[key]))
# Non-PNG containers (JPEG/WebP/AVIF): extract_c2pa_info is PNG-only, so
# fall back to the format-agnostic source check for the SynthID verdict.
# Non-PNG containers (JPEG/WebP/AVIF/MP4): extract_c2pa_info is PNG-only, so
# fall back to the format-agnostic source check for the SynthID verdict and
# the soft-binding (forensic-watermark vendor) scan.
if "synthid_watermark" not in result and (vendor := synthid_source(image_path)):
result.setdefault("synthid_watermark", synthid_verdict(vendor))
if "soft_binding" not in result:
with open(image_path, "rb") as f:
head = f.read(1024 * 1024)
if vendors := soft_binding_vendors_in(head):
result["soft_binding"] = ", ".join(vendors)
# China TC260 AI-content label (Doubao and other China-served generators).
if aigc := aigc_label(image_path):
@@ -427,9 +480,46 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
# xAI / Grok EXIF signature scheme (its only provenance signal).
if xai_signature(image_path):
result.setdefault("xai_signature", "xAI/Grok EXIF signature (Artist UUID + Signature blob)")
# IPTC 2025.1 AI-disclosure XMP fields (Iptc4xmpExt:AISystemUsed etc.).
if system := iptc_ai_system(image_path):
result.setdefault("ai_system", f"IPTC 2025.1 AI disclosure ({system})")
return result
def _strip_with_ffmpeg(source_path: Path, output_path: Path) -> Path:
"""Strip container metadata from a non-ISOBMFF audio/video file via ffmpeg.
Uses a lossless stream copy (``-c copy``), so codec data is untouched and only
container-level tags/chapters are dropped -- the metadata strip for WebM /
Matroska (EBML), MP3 (ID3), WAV / FLAC / OGG (RIFF / Vorbis comments) that the
ISOBMFF box walker cannot reach. Requires ffmpeg on PATH (raises if absent).
The output extension should match the source so ``-c copy`` can re-mux.
"""
import shutil
import subprocess
ffmpeg = shutil.which("ffmpeg")
if ffmpeg is None:
raise RuntimeError(
f"ffmpeg is required to strip metadata from {source_path.suffix} files but was not found on "
"PATH; install ffmpeg (e.g. `brew install ffmpeg`) or re-encode the file with another tool"
)
output_path.parent.mkdir(parents=True, exist_ok=True)
cmd = [
ffmpeg, "-y", "-loglevel", "error",
"-i", str(source_path),
"-map_metadata", "-1", "-map_chapters", "-1",
"-c", "copy",
str(output_path),
]
result = subprocess.run(cmd, capture_output=True, text=True, check=False) # noqa: S603
if result.returncode != 0:
raise RuntimeError(f"ffmpeg failed to strip metadata from {source_path}: {result.stderr.strip()[:300]}")
logger.info("Stripped container metadata via ffmpeg -> %s", output_path)
return output_path
def remove_ai_metadata(
source_path: Path,
output_path: Path | None = None,
@@ -455,19 +545,30 @@ def remove_ai_metadata(
if output_path is None:
output_path = source_path
# AVIF/HEIF/JPEG-XL: strip C2PA boxes at the container level without
# re-encoding. Avoids needing PIL plugins (pillow-heif / pillow-jxl) and
# preserves pixel data bit-for-bit.
if source_path.suffix.lower() in (".avif", ".heif", ".heic", ".jxl"):
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
# ISOBMFF containers (AVIF/HEIF/JPEG-XL images, MP4/MOV/M4V video, M4A audio):
# strip C2PA + AI-label boxes at the container level without re-encoding.
# Avoids needing PIL plugins (pillow-heif / pillow-jxl) and preserves the
# codestream bit-for-bit. MP4/MOV/M4A are ISOBMFF too, so the same top-level
# uuid/jumb box walker applies. Route by suffix OR by an ``ftyp`` content
# sniff, so a correctly-shaped container is handled whatever its extension.
from remove_ai_watermarks.noai.isobmff import is_isobmff, strip_c2pa_boxes
with open(source_path, "rb") as f:
head = f.read(12)
if source_path.suffix.lower() in _ISOBMFF_EXTS or is_isobmff(head):
data = source_path.read_bytes()
cleaned, stripped = strip_c2pa_boxes(data)
output_path.parent.mkdir(parents=True, exist_ok=True)
output_path.write_bytes(cleaned)
logger.info("Stripped %d C2PA box(es) → %s", stripped, output_path)
logger.info("Stripped %d AI-provenance box(es) → %s", stripped, output_path)
return output_path
# Non-ISOBMFF audio/video (WebM/Matroska EBML, MP3 ID3, WAV/FLAC/OGG): the
# box walker can't reach these, so strip container metadata losslessly via
# ffmpeg (-c copy -- codec data untouched, only tags/chapters dropped).
if source_path.suffix.lower() in _FFMPEG_STRIP_EXTS:
return _strip_with_ffmpeg(source_path, output_path)
# Read image and filter metadata
with Image.open(source_path) as img:
img = img.copy()
+23 -3
View File
@@ -28,6 +28,7 @@ from remove_ai_watermarks.noai.constants import (
C2PA_CHUNK_TYPE,
C2PA_ISSUERS,
C2PA_SIGNATURES,
C2PA_SOFT_BINDINGS,
PNG_SIGNATURE,
SYNTHID_C2PA_ISSUERS,
)
@@ -130,7 +131,7 @@ def extract_c2pa_info(image_path: Path) -> dict[str, Any]:
return c2pa_info
def _cbor_text_after(payload: bytes, key: bytes) -> str | None:
def cbor_text_after(payload: bytes, key: bytes) -> str | None:
"""Return the CBOR text-string immediately following ``key`` in ``payload``.
Handles CBOR major-type 3 length prefixes: direct (0x60-0x77), 1-byte
@@ -174,6 +175,18 @@ def synthid_vendors_in(buffer: bytes) -> list[str]:
return sorted({name for sig, name in C2PA_ISSUERS.items() if sig in buffer and sig in SYNTHID_C2PA_ISSUERS})
def soft_binding_vendors_in(buffer: bytes) -> list[str]:
"""Return forensic-watermark vendor names whose C2PA soft-binding ``alg``
identifier appears in ``buffer``.
A ``c2pa.soft-binding`` assertion names the watermark scheme that stamped the
pixels (Adobe TrustMark, Digimarc, Imatag, Steg.AI, ...). Shared by the PNG
caBX parser and the format-agnostic binary scan so both apply the same
C2PA_SOFT_BINDINGS rule against their respective bytes.
"""
return sorted({name for sig, name in C2PA_SOFT_BINDINGS.items() if sig in buffer})
def _parse_c2pa_chunk(chunk_data: bytes, c2pa_info: dict[str, Any]) -> None:
"""Parse C2PA chunk data and populate info dictionary."""
c2pa_info["c2pa_manifest"] = f"C2PA manifest ({len(chunk_data)} bytes)"
@@ -199,9 +212,9 @@ def _parse_c2pa_chunk(chunk_data: bytes, c2pa_info: dict[str, Any]) -> None:
# Guard with isprintable(): on some manifests (e.g. Microsoft Designer) the
# first ``name`` key precedes a binary field (a hash), not the generator
# string, which would otherwise surface as control-char garbage.
if (generator := _cbor_text_after(chunk_data, b"name")) and generator.isprintable():
if (generator := cbor_text_after(chunk_data, b"name")) and generator.isprintable():
c2pa_info["claim_generator"] = generator
if (spec := _cbor_text_after(chunk_data, b"specVersion")) and spec.isprintable():
if (spec := cbor_text_after(chunk_data, b"specVersion")) and spec.isprintable():
c2pa_info["c2pa_spec"] = spec
# Find actions
@@ -238,6 +251,13 @@ def _parse_c2pa_chunk(chunk_data: bytes, c2pa_info: dict[str, Any]) -> None:
c2pa_info["synthid_vendors"] = synthid_vendors
c2pa_info["synthid_watermark"] = synthid_verdict(", ".join(synthid_vendors))
# Soft-binding: a forensic/third-party watermark vendor named in the
# manifest (Adobe TrustMark, Digimarc, ...), independent of the issuer.
soft_binding_vendors = soft_binding_vendors_in(chunk_data)
if soft_binding_vendors:
c2pa_info["soft_binding_vendors"] = soft_binding_vendors
c2pa_info["soft_binding"] = ", ".join(soft_binding_vendors)
def extract_c2pa_chunk(image_path: Path) -> bytes | None:
"""
@@ -122,6 +122,35 @@ C2PA_AI_TOOLS = {
b"Firefly": "Firefly",
}
# C2PA ``c2pa.soft-binding`` algorithm identifiers -> the forensic-watermark
# vendor that stamped the pixels. The manifest's ``alg`` field names the
# watermark scheme even when the watermark itself cannot be decoded locally, so
# a byte-scan for these (keyed on a distinctive prefix to catch all variants)
# tells us a third-party forensic watermark is present and whose. Verified
# against the official C2PA registry (github.com/c2pa-org/softbinding-algorithm-list).
# Adobe TrustMark is additionally decodable locally (see ``trustmark_detector``);
# the rest (Digimarc, Imatag, Steg.AI, etc.) are proprietary oracle-only decoders.
C2PA_SOFT_BINDINGS = {
b"com.adobe.trustmark": "Adobe TrustMark",
b"com.adobe.icn": "Adobe (content fingerprint)",
b"com.digimarc": "Digimarc",
b"com.imatag.lamark": "Imatag (Lamark)",
b"ai.steg": "Steg.AI",
b"com.microsoft.invismark": "Microsoft InvisMark",
b"com.microsoft.wavmark": "Microsoft WavMark",
b"com.verimatrix": "Verimatrix",
b"com.nagra.nexguard": "NAGRA NexGuard",
b"com.aiwatermark": "AIWatermark (Meta PixelSeal)",
b"ai.trufo": "Trufo",
b"app.overlai": "Overlai",
b"com.markany": "MarkAny",
b"com.mentaport": "Mentaport",
b"es.lumatrace": "LumaTrace",
b"ai.verda": "VerdaAI",
b"ai.contentlens": "ContentLens",
b"io.iscc": "ISCC (content code)",
}
# Lowercased substrings that mark an AI generator when found in an EXIF
# ``Software`` / XMP ``CreatorTool`` value. Conservative on purpose: plain
# editors like "Adobe Photoshop" or "GIMP" must NOT match (no AI token), so only
+39 -16
View File
@@ -23,13 +23,25 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from collections.abc import Iterator
from remove_ai_watermarks.metadata import C2PA_UUID
from remove_ai_watermarks.metadata import (
AIGC_MARKERS,
C2PA_UUID,
IPTC_AI_FIELD_MARKERS,
IPTC_AI_MARKERS,
)
# Top-level box types that carry C2PA payload. ``uuid`` boxes are checked
# against ``C2PA_UUID`` before being stripped; ``jumb`` boxes are always
# stripped (JPEG-XL uses them exclusively for JUMBF).
# Top-level box types that may carry AI provenance. ``uuid`` boxes are checked
# against ``C2PA_UUID`` / AI-label markers before being stripped; ``jumb`` boxes
# are always stripped (JPEG-XL uses them exclusively for JUMBF).
C2PA_BOX_TYPES: frozenset[bytes] = frozenset({b"uuid", b"jumb"})
# AI-label byte markers (TC260 AIGC, IPTC "Made with AI", IPTC 2025.1 AI fields)
# whose presence inside an XMP ``uuid`` box means the box carries an AI label.
# Matching the payload rather than a fixed XMP UUID avoids the XMP-box UUID
# byte-order ambiguity and stays surgical: only AI-bearing XMP is dropped, plain
# XMP (copyright, camera info) is kept.
_AI_LABEL_MARKERS: tuple[bytes, ...] = AIGC_MARKERS + IPTC_AI_MARKERS + IPTC_AI_FIELD_MARKERS
def _iter_top_level_boxes(data: bytes) -> Iterator[tuple[int, int, bytes, int]]:
"""Yield ``(start, end, type, payload_offset)`` for each top-level box.
@@ -67,12 +79,22 @@ def is_isobmff(data: bytes) -> bool:
def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
"""Return ``(cleaned_bytes, stripped_count)``.
"""Return ``(cleaned_bytes, stripped_count)`` with AI-provenance boxes removed.
Walks top-level boxes; drops any ``uuid`` box whose UUID equals
``C2PA_UUID`` and any ``jumb`` box (JPEG-XL JUMBF container). All other
boxes are emitted verbatim. If the input is not ISOBMFF-shaped, returns
it unchanged.
Walks top-level boxes and drops:
- any ``uuid`` box whose UUID equals ``C2PA_UUID`` (a C2PA manifest);
- any ``uuid`` box whose payload carries an AI-label marker (an XMP packet
with a TC260 / IPTC / IPTC-2025.1 AI field -- caught by content, not by the
XMP UUID, so it works regardless of the UUID's byte order, and leaves plain
non-AI XMP intact);
- any ``jumb`` box (JPEG-XL JUMBF container).
All other boxes (incl. ``mdat`` / codestream) are emitted verbatim, so pixel
and audio data is preserved bit-for-bit. Non-ISOBMFF input is returned
unchanged. Despite the name this also covers MP4/MOV/M4A video and audio
(all ISOBMFF). NOTE: EXIF/XMP stored as *items inside the ``meta`` box*
(typical for AVIF/HEIF images) is not removed -- that needs meta-box surgery
and is a documented limitation.
"""
if not is_isobmff(data):
return data, 0
@@ -80,14 +102,15 @@ def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
out = bytearray()
stripped = 0
for start, end, box_type, payload_off in _iter_top_level_boxes(data):
if box_type in C2PA_BOX_TYPES:
if box_type == b"uuid":
# uuid boxes carry the 16-byte UUID immediately after the type.
if payload_off + 16 <= end and data[payload_off : payload_off + 16] == C2PA_UUID:
stripped += 1
continue
else: # b"jumb"
if box_type == b"uuid":
# uuid boxes carry the 16-byte UUID immediately after the type.
is_c2pa = payload_off + 16 <= end and data[payload_off : payload_off + 16] == C2PA_UUID
has_ai_label = any(marker in data[payload_off:end] for marker in _AI_LABEL_MARKERS)
if is_c2pa or has_ai_label:
stripped += 1
continue
elif box_type == b"jumb":
stripped += 1
continue
out.extend(data[start:end])
return bytes(out), stripped
+179
View File
@@ -0,0 +1,179 @@
"""Universal region eraser: remove anything inside user-given boxes via inpainting.
Position- and content-agnostic. You supply the rectangle(s); the eraser inpaints
whatever is inside, so it removes any visible logo / watermark / object regardless
of colour, style, or location. Localisation is the user's responsibility (pass the
box); restoration runs on CPU. This is the universal fallback for marks the
deterministic per-generator engines (Gemini sparkle, Doubao) do not cover.
Backends:
- ``cv2`` (default): ``cv2.inpaint`` (Telea / Navier-Stokes). Instant, no extra
dependencies, lower quality on large or textured regions.
- ``lama`` (optional, extra ``lama``): big-LaMa via onnxruntime
(``Carve/LaMa-ONNX``, Apache-2.0). CPU, resolution-robust, much better on
texture. The model (~200 MB) is downloaded on first use and cached by
huggingface_hub; it is never bundled in this repo.
"""
from __future__ import annotations
import logging
from typing import TYPE_CHECKING, Literal
import cv2
import numpy as np
if TYPE_CHECKING:
from numpy.typing import NDArray
logger = logging.getLogger(__name__)
Backend = Literal["cv2", "lama"]
_LAMA_REPO = "Carve/LaMa-ONNX"
_LAMA_FILE = "lama_fp32.onnx"
# Cached onnxruntime session (loading is expensive; reuse across calls).
_lama_session: object | None = None
def boxes_to_mask(
shape: tuple[int, int],
boxes: list[tuple[int, int, int, int]],
dilate: int = 3,
) -> NDArray:
"""Build a uint8 mask (255 inside boxes) from ``(x, y, w, h)`` rectangles."""
h, w = shape
mask = np.zeros((h, w), np.uint8)
for x, y, bw, bh in boxes:
x0, y0 = max(0, x), max(0, y)
x1, y1 = min(w, x + bw), min(h, y + bh)
if x1 > x0 and y1 > y0:
mask[y0:y1, x0:x1] = 255
if dilate > 0 and mask.any():
k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (2 * dilate + 1, 2 * dilate + 1))
mask = cv2.dilate(mask, k)
return mask
def erase_cv2(
image_bgr: NDArray,
mask: NDArray,
*,
method: Literal["telea", "ns"] = "telea",
radius: int = 6,
) -> NDArray:
"""Inpaint ``mask`` with classical cv2 inpainting (CPU, no extra deps)."""
flag = cv2.INPAINT_TELEA if method == "telea" else cv2.INPAINT_NS
return cv2.inpaint(image_bgr, mask, radius, flag)
def lama_available() -> bool:
"""True when the optional LaMa-ONNX backend can run (onnxruntime installed)."""
try:
import onnxruntime # noqa: F401
return True
except ImportError:
return False
def _get_lama_session() -> object:
"""Load (once) the big-LaMa ONNX session, downloading the model on first use."""
global _lama_session
if _lama_session is not None:
return _lama_session
import onnxruntime as ort
from huggingface_hub import hf_hub_download
model_path = hf_hub_download(repo_id=_LAMA_REPO, filename=_LAMA_FILE)
logger.info("Loading LaMa-ONNX model: %s", model_path)
_lama_session = ort.InferenceSession(model_path, providers=["CPUExecutionProvider"])
return _lama_session
def erase_lama(image_bgr: NDArray, mask: NDArray) -> NDArray:
"""Inpaint ``mask`` with big-LaMa via onnxruntime (CPU).
LaMa runs at a fixed square input size. To preserve full-image resolution we
crop a padded region around the mask, inpaint that crop at the model size,
and paste only the masked pixels back -- so untouched areas stay pixel-exact.
"""
session = _get_lama_session()
inp = session.get_inputs() # type: ignore[attr-defined]
img_name = inp[0].name
mask_name = inp[1].name
# Model declares a fixed square spatial size (e.g. 512); fall back to 512.
dims = inp[0].shape
size = next((d for d in reversed(dims) if isinstance(d, int) and d > 1), 512)
h, w = image_bgr.shape[:2]
ys, xs = np.where(mask > 0)
if len(xs) == 0:
return image_bgr.copy()
# Padded crop around the mask (context for the inpainter).
pad = max(16, int(0.4 * max(xs.max() - xs.min() + 1, ys.max() - ys.min() + 1)))
cx0, cy0 = max(0, int(xs.min()) - pad), max(0, int(ys.min()) - pad)
cx1, cy1 = min(w, int(xs.max()) + 1 + pad), min(h, int(ys.max()) + 1 + pad)
crop = image_bgr[cy0:cy1, cx0:cx1]
crop_mask = mask[cy0:cy1, cx0:cx1]
ch, cw = crop.shape[:2]
# Resize crop + mask to the model size, normalise to [0,1] RGB CHW.
crop_rs = cv2.resize(crop, (size, size), interpolation=cv2.INTER_AREA)
mask_rs = cv2.resize(crop_mask, (size, size), interpolation=cv2.INTER_NEAREST)
img_in = cv2.cvtColor(crop_rs, cv2.COLOR_BGR2RGB).astype(np.float32) / 255.0
img_in = np.transpose(img_in, (2, 0, 1))[None] # (1,3,size,size)
mask_in = (mask_rs > 127).astype(np.float32)[None, None] # (1,1,size,size), 1=hole
out = session.run(None, {img_name: img_in, mask_name: mask_in})[0] # type: ignore[attr-defined]
out = np.asarray(out)[0] # (3,size,size)
out = np.transpose(out, (1, 2, 0))
if float(out.max()) <= 1.5: # model emits [0,1]; otherwise already [0,255]
out = out * 255.0
out = np.clip(out, 0, 255).astype(np.uint8)
out_bgr = cv2.cvtColor(out, cv2.COLOR_RGB2BGR)
# Resize back to crop size and paste only the masked pixels.
out_crop = cv2.resize(out_bgr, (cw, ch), interpolation=cv2.INTER_LINEAR)
result = image_bgr.copy()
region = result[cy0:cy1, cx0:cx1]
paste = crop_mask > 127
region[paste] = out_crop[paste]
result[cy0:cy1, cx0:cx1] = region
return result
def erase(
image_bgr: NDArray,
*,
boxes: list[tuple[int, int, int, int]] | None = None,
mask: NDArray | None = None,
backend: Backend = "cv2",
dilate: int = 3,
cv2_method: Literal["telea", "ns"] = "telea",
cv2_radius: int = 6,
) -> NDArray:
"""Erase the given boxes (or mask) via the chosen inpainting backend.
Provide either ``boxes`` (list of ``(x, y, w, h)``) or a precomputed ``mask``
(uint8, 255 = erase). Returns an unmodified copy when nothing is selected.
"""
if image_bgr is None or image_bgr.size == 0:
return image_bgr
if mask is None:
if not boxes:
return image_bgr.copy()
mask = boxes_to_mask(image_bgr.shape[:2], boxes, dilate=dilate)
if not mask.any():
return image_bgr.copy()
if backend == "lama":
if not lama_available():
raise RuntimeError(
"LaMa backend requires onnxruntime. Install the extra: pip install 'remove-ai-watermarks[lama]'"
)
return erase_lama(image_bgr, mask)
return erase_cv2(image_bgr, mask, method=cv2_method, radius=cv2_radius)
@@ -0,0 +1,72 @@
"""Detect Adobe TrustMark invisible watermarks.
TrustMark (github.com/adobe/trustmark, MIT) is the open, keyless image watermark
behind Adobe "Durable Content Credentials": when a C2PA manifest is stripped, a
TrustMark soft binding can still re-link the asset to its manifest in a
repository. Unlike SynthID it has a PUBLIC decoder with no secret key, so a
TrustMark-stamped image can be identified locally. Adobe's shipping products use
Variant P (the ``com.adobe.trustmark.P`` soft-binding ``alg``); this wrapper
loads that model.
Optional dependency (extra: ``trustmark``); the model weights download on first
use. ``detect_trustmark`` returns None when the package is absent. This detects
provenance (Adobe Content Credentials), NOT AI generation as such -- TrustMark
also marks human-authored content -- so callers should treat it as a watermark
signal, not proof of AI origin.
"""
# trustmark ships no type stubs; relax untyped-library diagnostics for this thin
# wrapper module only.
# pyright: reportMissingTypeStubs=false, reportUnknownMemberType=false, reportUnknownVariableType=false, reportUnknownArgumentType=false, reportMissingImports=false
from __future__ import annotations
import logging
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from pathlib import Path
log = logging.getLogger(__name__)
# Adobe ships Variant P in production (com.adobe.trustmark.P).
_MODEL_TYPE = "P"
# Lazily constructed singleton -- model load + first-use download is expensive.
_tm: Any = None
def is_available() -> bool:
"""True if the optional ``trustmark`` package is installed."""
import importlib.util
return importlib.util.find_spec("trustmark") is not None
def _decoder() -> Any:
global _tm
if _tm is None:
from trustmark import TrustMark
_tm = TrustMark(verbose=False, model_type=_MODEL_TYPE)
return _tm
def detect_trustmark(image_path: Path) -> str | None:
"""Return a TrustMark scheme note if a TrustMark watermark is decoded, else None.
Returns e.g. ``"Adobe TrustMark (variant P, schema 0)"`` when the decoder
reports the watermark present, or None if it is absent, the optional
``trustmark`` package is not installed, or the image cannot be read/decoded.
"""
if not is_available():
return None
try:
from PIL import Image
with Image.open(image_path) as img:
cover = img.convert("RGB")
_wm_secret, wm_present, wm_schema = _decoder().decode(cover)
except Exception as exc: # model download / decode failure / unreadable image
log.debug("TrustMark decode failed for %s: %s", image_path, exc)
return None
return f"Adobe TrustMark (variant {_MODEL_TYPE}, schema {wm_schema})" if wm_present else None
+7 -3
View File
@@ -27,7 +27,9 @@ def runner():
@pytest.fixture
def sample_png(tmp_path: Path) -> Path:
"""Create a sample PNG for CLI testing."""
img = np.random.randint(0, 255, (200, 200, 3), dtype=np.uint8)
# Seeded: an unseeded random corner can occasionally trip the Doubao
# visible-mark detector, making `visible --mark auto` flaky.
img = np.random.default_rng(0).integers(0, 255, (200, 200, 3), dtype=np.uint8)
path = tmp_path / "input.png"
cv2.imwrite(str(path), img)
return path
@@ -37,8 +39,9 @@ def _make_batch_dir(tmp_path: Path, count: int = 3) -> Path:
"""Create a directory with test images for batch testing."""
input_dir = tmp_path / "input"
input_dir.mkdir()
rng = np.random.default_rng(0)
for i in range(count):
img = np.random.randint(0, 255, (100, 100, 3), dtype=np.uint8)
img = rng.integers(0, 255, (100, 100, 3), dtype=np.uint8)
cv2.imwrite(str(input_dir / f"img_{i}.png"), img)
return input_dir
@@ -119,7 +122,8 @@ class TestVisibleCommand:
def test_visible_help(self, runner):
result = runner.invoke(main, ["visible", "--help"])
assert result.exit_code == 0
assert "Gemini watermark" in result.output
assert "visible AI watermark" in result.output
assert "--mark" in result.output
def test_visible_basic(self, runner, sample_png, tmp_path):
output = tmp_path / "clean.png"
+98
View File
@@ -0,0 +1,98 @@
"""Tests for the Doubao visible-watermark engine."""
from __future__ import annotations
from pathlib import Path
import cv2
import numpy as np
import pytest
from remove_ai_watermarks.doubao_engine import DoubaoEngine, load_image_bgr
SAMPLE = Path(__file__).resolve().parents[1] / "data" / "samples" / "doubao-1.png"
# ── Locate ──────────────────────────────────────────────────────────
class TestLocate:
def test_box_anchored_bottom_right(self):
eng = DoubaoEngine()
img = np.zeros((2048, 2048, 3), np.uint8)
loc = eng.locate(img)
# right and bottom edges sit close to the image corner (within margins)
assert 2048 - (loc.x + loc.w) < int(2048 * 0.03)
assert 2048 - (loc.y + loc.h) < int(2048 * 0.03)
assert loc.is_fallback # geometry anchor, no bundled template yet
def test_box_scales_with_width(self):
eng = DoubaoEngine()
small = eng.locate(np.zeros((1024, 1024, 3), np.uint8))
large = eng.locate(np.zeros((2048, 2048, 3), np.uint8))
# width-relative geometry: 2x wider image -> ~2x wider box
assert large.w == pytest.approx(small.w * 2, rel=0.1)
# ── Detect + remove on the real sample ──────────────────────────────
@pytest.mark.skipif(not SAMPLE.exists(), reason="sample image not present")
class TestRealSample:
def test_detects_watermark(self):
eng = DoubaoEngine()
det = eng.detect(load_image_bgr(SAMPLE))
assert det.detected
assert det.confidence > 0.0
assert det.coverage > 0.04
def test_remove_reduces_glyph_coverage(self):
eng = DoubaoEngine()
img = load_image_bgr(SAMPLE)
before = eng.detect(img).coverage
out = eng.remove_watermark(img)
after = eng.detect(out).coverage
# the inpaint should clear most glyph pixels from the corner box
assert after < before * 0.5
def test_pixels_outside_box_untouched(self):
eng = DoubaoEngine()
img = load_image_bgr(SAMPLE)
out = eng.remove_watermark(img)
# top-left quadrant is far from the bottom-right mark: must be identical
h, w = img.shape[:2]
assert np.array_equal(img[: h // 2, : w // 2], out[: h // 2, : w // 2])
# ── Negative + safety guard ─────────────────────────────────────────
class TestNegativeAndGuard:
def test_clean_image_not_detected(self):
eng = DoubaoEngine()
# smooth gradient, no watermark
ramp = np.tile(np.linspace(0, 255, 1024, dtype=np.uint8), (1024, 1))
img = cv2.cvtColor(ramp, cv2.COLOR_GRAY2BGR)
det = eng.detect(img)
assert not det.detected
def test_clean_image_returned_unchanged(self):
eng = DoubaoEngine()
ramp = np.tile(np.linspace(0, 255, 1024, dtype=np.uint8), (1024, 1))
img = cv2.cvtColor(ramp, cv2.COLOR_GRAY2BGR)
out = eng.remove_watermark(img)
assert np.array_equal(img, out)
def test_document_background_guard(self):
"""A dense high-frequency corner (document-like) trips the coverage
guard, so the image is left untouched rather than smeared."""
eng = DoubaoEngine()
rng = np.random.default_rng(0)
img = np.full((1024, 1024, 3), 255, np.uint8)
# fill the bottom-right box area with random grayish text-like noise
loc = eng.locate(img)
x, y, bw, bh = loc.bbox
noise = rng.integers(150, 246, size=(bh, bw), dtype=np.uint8)
img[y : y + bh, x : x + bw] = noise[:, :, None]
out = eng.remove_watermark(img)
assert np.array_equal(img, out)
+59
View File
@@ -311,6 +311,65 @@ class TestIdentifyXaiSignature:
assert any("xAI/Grok" in w for w in r.watermarks)
class TestIdentifySoftBinding:
"""A C2PA soft-binding alg names a forensic-watermark vendor in the inventory."""
def test_soft_binding_vendor_listed(self, tmp_path: Path):
p = tmp_path / "sb.jpg"
p.write_bytes(b"\xff\xd8\xff\xe1 c2pa jumb com.digimarc.validate.1 \xff\xd9")
r = identify(p, check_visible=False, check_invisible=False)
assert any("Digimarc" in w for w in r.watermarks)
assert any(s.name == "soft_binding" for s in r.signals)
class TestIdentifyIptcAi:
"""IPTC 2025.1 AISystemUsed drives an AI verdict + platform attribution."""
def test_iptc_ai_system_attributed(self, tmp_path: Path):
p = tmp_path / "iptc.jpg"
p.write_bytes(
b"\xff\xd8\xff\xe1<x:xmpmeta><Iptc4xmpExt:AISystemUsed>Google Gemini"
b"</Iptc4xmpExt:AISystemUsed></x:xmpmeta>\xff\xd9"
)
r = identify(p, check_visible=False, check_invisible=False)
assert r.is_ai_generated is True
assert r.platform is not None
assert "Gemini" in r.platform
class TestIdentifyC2paDevice:
"""A distinctive C2PA device token wins platform attribution over incidental
issuer-name mentions (regression guard for real-sample mis-attribution:
Leica->Truepic, Nikon->Adobe, Pixel->Google Gemini)."""
def test_leica_token_beats_incidental_tokens(self, tmp_path: Path):
# "Adobe"/"Google"/"Truepic" appear incidentally; Leica's lc_c2pa wins.
blob = b"\xff\xd8\xff\xe1 c2pa.claim jumbf Adobe Google Truepic lc_c2pa \xff\xd9"
p = tmp_path / "leica_like.jpg"
p.write_bytes(blob)
r = identify(p, check_visible=False, check_invisible=False)
assert r.platform == "Leica (camera, C2PA capture)"
def test_pixel_camera_cert_beats_incidental_google(self, tmp_path: Path):
# Pixel's cert CN is "Pixel Camera"; "Google LLC" appears as the cert org
# but must NOT yield "Google (Gemini / Imagen)" -- it is a camera capture.
blob = b"\xff\xd8\xff\xe1 c2pa.claim jumbf Google LLC Adobe Pixel Camera \xff\xd9"
p = tmp_path / "pixel_like.jpg"
p.write_bytes(blob)
r = identify(p, check_visible=False, check_invisible=False)
assert r.platform == "Google Pixel (camera, C2PA capture)"
assert r.is_ai_generated is None # camera capture, not AI
def test_sony_namespace_beats_bare_make(self, tmp_path: Path):
# Sony's own C2PA assertion namespace (sony.sig), not the bare "Sony"
# EXIF Make that appears on ordinary photos.
blob = b"\xff\xd8\xff\xe1 c2pa.claim jumbf Adobe Sony sony.sig.v1_1 \xff\xd9"
p = tmp_path / "sony_like.jpg"
p.write_bytes(blob)
r = identify(p, check_visible=False, check_invisible=False)
assert r.platform == "Sony (camera, C2PA capture)"
# ── Open invisible watermark (SD/SDXL/FLUX) integration ─────────────
from remove_ai_watermarks.invisible_watermark import is_available as _wm_available # noqa: E402
+166
View File
@@ -2,6 +2,8 @@
from __future__ import annotations
import shutil
import subprocess
from pathlib import Path
import piexif
@@ -14,6 +16,7 @@ from remove_ai_watermarks.metadata import (
exif_generator,
get_ai_metadata,
has_ai_metadata,
iptc_ai_system,
remove_ai_metadata,
synthid_source,
xai_signature,
@@ -567,3 +570,166 @@ class TestAIGCRealSample:
def test_doubao_detected_as_ai(self):
assert has_ai_metadata(SAMPLES_DIR / "doubao-1.png")
assert "aigc_label" in get_ai_metadata(SAMPLES_DIR / "doubao-1.png")
class TestSoftBinding:
"""C2PA soft-binding alg identifier -> forensic-watermark vendor name."""
def test_vendors_in_recognizes_known_algs(self):
from remove_ai_watermarks.noai.c2pa import soft_binding_vendors_in
assert soft_binding_vendors_in(b"...alg...com.adobe.trustmark.P...") == ["Adobe TrustMark"]
assert soft_binding_vendors_in(b"com.digimarc.validate.1") == ["Digimarc"]
assert soft_binding_vendors_in(b"ai.steg.api blah") == ["Steg.AI"]
# Registry-verified vendors added in v0.6.x.
assert soft_binding_vendors_in(b"ai.trufo.gen1.image") == ["Trufo"]
assert soft_binding_vendors_in(b"io.iscc.v0") == ["ISCC (content code)"]
def test_vendors_in_empty_when_absent(self):
from remove_ai_watermarks.noai.c2pa import soft_binding_vendors_in
assert soft_binding_vendors_in(b"no soft binding here") == []
def test_get_ai_metadata_surfaces_soft_binding(self, tmp_path: Path):
# Non-PNG binary-scan path: a manifest naming a soft-binding vendor.
p = tmp_path / "fake.jpg"
p.write_bytes(b"\xff\xd8\xff\xe1 c2pa jumb com.adobe.trustmark.P \xff\xd9")
assert get_ai_metadata(p).get("soft_binding") == "Adobe TrustMark"
class TestIptcAiFields:
"""IPTC 2025.1 AI-disclosure XMP properties (Iptc4xmpExt:AISystemUsed etc.)."""
def test_detects_ai_system_used_element_form(self, tmp_path: Path):
p = tmp_path / "iptc_ai.jpg"
p.write_bytes(
b"\xff\xd8\xff\xe1<x:xmpmeta><Iptc4xmpExt:AISystemUsed>ChatGPT DALL-E"
b"</Iptc4xmpExt:AISystemUsed></x:xmpmeta>\xff\xd9"
)
assert has_ai_metadata(p) is True
assert iptc_ai_system(p) == "ChatGPT DALL-E"
assert "ChatGPT DALL-E" in get_ai_metadata(p)["ai_system"]
def test_attribute_serialization(self, tmp_path: Path):
p = tmp_path / "attr.jpg"
p.write_bytes(b'\xff\xd8\xff\xe1 Iptc4xmpExt:AISystemUsed="Google Gemini" \xff\xd9')
assert iptc_ai_system(p) == "Google Gemini"
def test_present_without_value(self, tmp_path: Path):
# A disclosure field with no extractable value still flags presence.
p = tmp_path / "novalue.jpg"
p.write_bytes(b"\xff\xd8\xff\xe1 Iptc4xmpExt:AIPromptWriterName \xff\xd9")
assert iptc_ai_system(p) == "fields present"
assert has_ai_metadata(p) is True
def test_clean_image_none(self, tmp_clean_png: Path):
assert iptc_ai_system(tmp_clean_png) is None
# Synthetic MP4 (ISOBMFF): ftyp + C2PA uuid box + mdat. Same box format as AVIF.
_MP4_FTYP = b"\x00\x00\x00\x18ftypmp42\x00\x00\x00\x00mp42isom"
_MP4_MDAT = b"\x00\x00\x00\x10mdat" + b"videodat"
class TestVideoC2pa:
"""C2PA in MP4 (ISOBMFF) -- detect + strip, reusing the image box walker."""
def test_detects_c2pa_in_mp4(self, tmp_path: Path):
from remove_ai_watermarks.metadata import C2PA_UUID
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"manifest"
p = tmp_path / "ai.mp4"
p.write_bytes(_MP4_FTYP + uuid_box + _MP4_MDAT)
assert has_ai_metadata(p) is True
def test_strips_c2pa_in_mp4(self, tmp_path: Path):
from remove_ai_watermarks.metadata import C2PA_UUID
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"manifest"
src = tmp_path / "in.mp4"
src.write_bytes(_MP4_FTYP + uuid_box + _MP4_MDAT)
out = tmp_path / "out.mp4"
remove_ai_metadata(src, out)
assert out.read_bytes() == _MP4_FTYP + _MP4_MDAT
assert has_ai_metadata(out) is False
class TestIsobmffMetadataRemoval:
"""Container-level AI-provenance stripping across ISOBMFF image/video/audio."""
def test_strips_ai_xmp_uuid_box(self):
# A uuid box carrying a TC260 AIGC label is dropped by content match,
# regardless of the (non-C2PA) XMP UUID's byte order.
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
xmp_uuid = bytes(range(16)) # arbitrary, not the C2PA UUID
payload = b'<x:xmpmeta><TC260:AIGC>{"Label":"1"}</TC260:AIGC></x:xmpmeta>'
box = (24 + len(payload)).to_bytes(4, "big") + b"uuid" + xmp_uuid + payload
cleaned, stripped = strip_c2pa_boxes(_MP4_FTYP + box + _MP4_MDAT)
assert stripped == 1
assert cleaned == _MP4_FTYP + _MP4_MDAT
def test_keeps_plain_non_ai_xmp(self):
# A uuid box with ordinary (non-AI) XMP must be preserved.
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
xmp_uuid = bytes(range(16))
payload = b"<x:xmpmeta><dc:rights>(c) me</dc:rights></x:xmpmeta>"
box = (24 + len(payload)).to_bytes(4, "big") + b"uuid" + xmp_uuid + payload
cleaned, stripped = strip_c2pa_boxes(_MP4_FTYP + box + _MP4_MDAT)
assert stripped == 0
assert cleaned == _MP4_FTYP + box + _MP4_MDAT
def test_m4a_c2pa_stripped(self, tmp_path: Path):
from remove_ai_watermarks.metadata import C2PA_UUID
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"manifest"
src = tmp_path / "voice.m4a"
src.write_bytes(_MP4_FTYP + uuid_box + _MP4_MDAT)
out = tmp_path / "clean.m4a"
remove_ai_metadata(src, out)
assert out.read_bytes() == _MP4_FTYP + _MP4_MDAT
def test_content_sniff_routes_unknown_suffix(self, tmp_path: Path):
# An ISOBMFF file with a non-standard extension is still box-stripped.
from remove_ai_watermarks.metadata import C2PA_UUID
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"manifest"
src = tmp_path / "mystery.bin"
src.write_bytes(_MP4_FTYP + uuid_box + _MP4_MDAT)
out = tmp_path / "out.bin"
remove_ai_metadata(src, out)
assert out.read_bytes() == _MP4_FTYP + _MP4_MDAT
def test_unparseable_audio_raises(self, tmp_path: Path):
# Garbage that ffmpeg can't parse must raise a clear error, not crash in
# the image path. (When ffmpeg is absent this still raises RuntimeError.)
src = tmp_path / "audio.mp3"
src.write_bytes(b"ID3\x04\x00\x00\x00\x00\x00\x00 not real mp3 frames")
out = tmp_path / "out.mp3"
with pytest.raises(RuntimeError):
remove_ai_metadata(src, out)
@pytest.mark.skipif(shutil.which("ffmpeg") is None, reason="ffmpeg not installed")
class TestFfmpegMetadataStrip:
"""Lossless container-metadata strip for non-ISOBMFF audio/video via ffmpeg."""
def _wav_with_tag(self, path: Path, tag: str = "Suno AI") -> None:
subprocess.run( # noqa: S603
[
shutil.which("ffmpeg"), "-y", "-loglevel", "error",
"-f", "lavfi", "-i", "sine=frequency=440:duration=0.1",
"-metadata", f"title={tag}", str(path),
],
check=True,
)
def test_strips_wav_title_metadata(self, tmp_path: Path):
src = tmp_path / "in.wav"
self._wav_with_tag(src, "Suno AI generated")
assert b"Suno AI generated" in src.read_bytes() # tag is present pre-strip
out = tmp_path / "clean.wav"
remove_ai_metadata(src, out)
assert out.exists()
assert b"Suno AI generated" not in out.read_bytes() # tag stripped, audio kept
+9 -9
View File
@@ -8,8 +8,8 @@ from pathlib import Path
import pytest
from remove_ai_watermarks.noai.c2pa import (
_cbor_text_after,
_parse_c2pa_chunk,
cbor_text_after,
extract_c2pa_chunk,
extract_c2pa_info,
has_c2pa_metadata,
@@ -193,36 +193,36 @@ class TestC2PAInjectValidation:
class TestCborTextAfter:
"""_cbor_text_after handles the three CBOR text-string length prefixes."""
"""cbor_text_after handles the three CBOR text-string length prefixes."""
def test_direct_length(self):
# major-type 3, direct length (0x60 + len). "abc" -> 0x63.
payload = b"name" + bytes([0x63]) + b"abc"
assert _cbor_text_after(payload, b"name") == "abc"
assert cbor_text_after(payload, b"name") == "abc"
def test_one_byte_length(self):
s = b"x" * 30
payload = b"name" + bytes([0x78, 30]) + s
assert _cbor_text_after(payload, b"name") == "x" * 30
assert cbor_text_after(payload, b"name") == "x" * 30
def test_two_byte_length(self):
s = b"y" * 300
payload = b"name" + bytes([0x79]) + struct.pack(">H", 300) + s
assert _cbor_text_after(payload, b"name") == "y" * 300
assert cbor_text_after(payload, b"name") == "y" * 300
def test_key_not_found_returns_none(self):
assert _cbor_text_after(b"nothing here", b"name") is None
assert cbor_text_after(b"nothing here", b"name") is None
def test_key_at_end_returns_none(self):
assert _cbor_text_after(b"prefixname", b"name") is None
assert cbor_text_after(b"prefixname", b"name") is None
def test_invalid_head_returns_none(self):
# 0x00 is not a text-string head.
assert _cbor_text_after(b"name" + bytes([0x00]) + b"abc", b"name") is None
assert cbor_text_after(b"name" + bytes([0x00]) + b"abc", b"name") is None
def test_latin1_fallback_on_invalid_utf8(self):
payload = b"name" + bytes([0x61]) + b"\xff" # len 1, invalid utf-8
assert _cbor_text_after(payload, b"name") is not None
assert cbor_text_after(payload, b"name") is not None
class TestSynthIDVerdict:
+75
View File
@@ -0,0 +1,75 @@
"""Tests for the universal region eraser."""
from __future__ import annotations
import numpy as np
import pytest
from remove_ai_watermarks.region_eraser import boxes_to_mask, erase, lama_available
class TestBoxesToMask:
def test_mask_set_inside_box(self):
mask = boxes_to_mask((100, 100), [(10, 20, 30, 40)], dilate=0)
assert mask[25, 15] == 255 # inside
assert mask[0, 0] == 0 # outside
assert mask.shape == (100, 100)
def test_multiple_boxes(self):
mask = boxes_to_mask((100, 100), [(0, 0, 10, 10), (90, 90, 10, 10)], dilate=0)
assert mask[5, 5] == 255
assert mask[95, 95] == 255
assert mask[50, 50] == 0
def test_dilate_grows_mask(self):
m0 = boxes_to_mask((100, 100), [(40, 40, 10, 10)], dilate=0)
m5 = boxes_to_mask((100, 100), [(40, 40, 10, 10)], dilate=5)
assert m5.sum() > m0.sum()
def test_box_clipped_to_bounds(self):
# box partly outside the image must not raise and stays in-bounds
mask = boxes_to_mask((50, 50), [(40, 40, 100, 100)], dilate=0)
assert mask[45, 45] == 255
class TestEraseCv2:
def _image_with_logo(self) -> tuple[np.ndarray, tuple[int, int, int, int]]:
img = np.full((200, 200, 3), 120, np.uint8) # flat gray background
box = (140, 160, 50, 30)
x, y, w, h = box
img[y : y + h, x : x + w] = (255, 255, 255) # bright "logo"
return img, box
def test_erase_changes_region(self):
img, box = self._image_with_logo()
out = erase(img, boxes=[box], backend="cv2")
x, y, w, h = box
# on a flat background the logo region should be repainted near gray
region = out[y : y + h, x : x + w]
assert abs(float(region.mean()) - 120) < 20
assert not np.array_equal(out, img)
def test_pixels_outside_box_untouched(self):
img, box = self._image_with_logo()
out = erase(img, boxes=[box], backend="cv2", dilate=0)
# a far corner must be identical
assert np.array_equal(img[:50, :50], out[:50, :50])
def test_no_boxes_returns_copy(self):
img = np.full((100, 100, 3), 50, np.uint8)
out = erase(img, boxes=[], backend="cv2")
assert np.array_equal(img, out)
def test_empty_mask_returns_copy(self):
img = np.full((100, 100, 3), 50, np.uint8)
out = erase(img, mask=np.zeros((100, 100), np.uint8), backend="cv2")
assert np.array_equal(img, out)
class TestLamaBackend:
def test_lama_raises_when_unavailable(self):
img = np.full((100, 100, 3), 50, np.uint8)
if lama_available():
pytest.skip("onnxruntime installed; cannot test the unavailable path")
with pytest.raises(RuntimeError, match="onnxruntime"):
erase(img, boxes=[(10, 10, 20, 20)], backend="lama")
+36
View File
@@ -0,0 +1,36 @@
"""Tests for the optional Adobe TrustMark detector.
TrustMark is an optional dependency (extra ``trustmark``) that downloads model
weights on first use, so the decode path is only exercised when it is installed
(mirrors the imwatermark handling). The always-on test pins the graceful
absent/error behaviour: detect must return None, never raise.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
import pytest
from remove_ai_watermarks.trustmark_detector import detect_trustmark, is_available
if TYPE_CHECKING:
from pathlib import Path
def test_detect_never_raises(tmp_clean_png: Path):
# Whether or not trustmark is installed, a clean image must yield None
# (no watermark) without raising. When absent, the import guard returns None.
assert detect_trustmark(tmp_clean_png) is None
def test_unreadable_file_returns_none(tmp_path: Path):
bad = tmp_path / "not_an_image.txt"
bad.write_bytes(b"not an image")
assert detect_trustmark(bad) is None
@pytest.mark.skipif(not is_available(), reason="trustmark not installed")
def test_clean_image_reports_no_watermark(tmp_clean_png: Path):
# With the decoder present, an un-watermarked image must report absent.
assert detect_trustmark(tmp_clean_png) is None
Generated
+1780 -886
View File
File diff suppressed because it is too large. Load diff