mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-16 14:30:47 +02:00
Compare commits
No files matched your search
@@ -14,12 +14,12 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
|
||||
## Test and lint
|
||||
|
||||
- **CI** (`.github/workflows/test.yml`): runs on push to `main` + every PR. A `lint` job (ubuntu: `ruff check` + `ruff format --check`) plus a `test` matrix (ubuntu/macos/windows x py3.10/3.12) that does `uv sync --frozen --extra dev` then `pytest`. The matrix installs only core + dev (no `gpu` extra), so the GPU/model-running tests skip there and it exercises the metadata/identify/visible/cv2-eraser surface on all three OSes. Keep `uv.lock` valid (don't break `--frozen`) when editing `pyproject.toml`. `publish.yml` stays release-only and now verifies the release tag matches the `pyproject.toml` version (fails the build on a mismatch) before building.
|
||||
- **CI** (`.github/workflows/test.yml`): runs on push to `main` + every PR. A `lint` job (ubuntu: `ruff check` + `ruff format --check`) plus a `test` matrix (ubuntu/macos/windows x py3.10/3.12) that does `uv sync --frozen --extra dev` then `pytest`. The matrix installs only core + dev (no `gpu` extra), so the GPU/model-running tests skip there and it exercises the metadata/identify/visible/cv2-eraser surface on all three OSes. Keep `uv.lock` valid (don't break `--frozen`) when editing `pyproject.toml`. `publish.yml` stays release-only and now verifies the release tag matches the `pyproject.toml` version (fails the build on a mismatch) before building. **Release flow:** bump the version in `pyproject.toml` + `src/remove_ai_watermarks/__init__.py` + `uv.lock` (the project's own `[[package]]` entry, ~line 2868), commit `chore(release): vX.Y.Z`, `git tag -a vX.Y.Z -m vX.Y.Z` (annotated — `git tag` without `-m` errors here), push `main` + the tag, then `gh release create vX.Y.Z` — **PyPI publish triggers on the GitHub Release `published` event, NOT on the tag push**, so the tag alone does not publish. **Sdist must exclude `data/`** (`[tool.hatch.build.targets.sdist] exclude = ["/data"]`): hatchling's default sdist bundles all VCS-tracked files, so the committed `data/` test corpora (synthid_corpus images ~65 MB + the visible-mark captures) pushed the **0.8.0** sdist past PyPI's per-project file-size limit (400 "File too large") — the wheel uploaded but the sdist was rejected, so 0.8.0 shipped wheel-only and 0.8.1 carried the fix. The wheel only ships `src/` (via `[tool.hatch.build.targets.wheel] packages`), so it was never affected. **A failed PyPI upload of one artifact still leaves the other live and you cannot re-upload the same version** — fix the build and cut the next patch. **Build backend is pinned `hatchling<1.28`** (`[build-system] requires`): hatchling 1.28+ emits **Metadata-Version 2.5** (PEP 639), which the twine bundled in `pypa/gh-action-pypi-publish@release/v1` rejects (`"'2.5' is not a valid Metadata-Version"`) — this **failed the v0.8.3 PyPI upload on 2026-06-01** (tag-match + build passed, the upload step failed; nothing was uploaded, so the version stayed empty on PyPI). 1.27.x emits 2.4, which uploads fine (0.8.2 shipped on it). The pin is unpinned `requires = ["hatchling"]` no longer safe because `uv build` pulls the latest hatchling. Lift the pin only once the publish action's twine is ≥ 6.1.0 (2.5-aware) or the workflow moves to `uv publish`.
|
||||
- `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto
|
||||
- **Strict pyright is clean across `src/` (0 errors).** The cv2/torch/diffusers boundary files (`gemini_engine`, `region_eraser`, `doubao_engine`, `face_protector`, `humanizer`, `invisible_engine`, `noai/watermark_remover`, and the whole `noai/ctrlregen/` subpackage) carry a documented per-file `# pyright:` relax pragma (or, for `ctrlregen`, a `tool.pyright.executionEnvironments` entry) that turns off only the unknown-type / untyped-third-party rules — those libs ship no usable types, so strict typing there fights the ecosystem. Pure-logic files stay fully strict; `typings/piexif/__init__.pyi` is a local stub so `metadata.py`/`extractor.py` resolve piexif. Public ndarray-returning signatures on the relaxed engines are still annotated `NDArray[Any]` so strict consumers (`cli.py`) stay clean. When touching a relaxed file, prefer fixing real issues over widening the pragma; keep the pragma scoped to genuinely-untyped boundaries. (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.)
|
||||
- **Full-project `uv run pyright` (no path) OOMs/crashes node on this ML-heavy repo** (emits a `libnode` stack frame, no summary) — a known environment limit, not a code error. Gate with `uv run --extra dev --extra gpu pyright src/` (completes, authoritative) or scope to changed files; also run `uv run ruff check` and `uv run pytest` directly.
|
||||
- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch.
|
||||
- To add a dev tool (pytest/ruff/pyright) into the env, use `uv sync --frozen --extra dev --extra gpu`, **never `uv pip install`** — `uv pip install` re-resolves and rewrites `uv.lock`, which silently bumped `transformers` to a build incompatible with the pinned `diffusers` (`cannot import name 'Qwen3VLForConditionalGeneration'`) and broke every `identify`/metadata import. Recovery: `git checkout uv.lock && uv sync --frozen --extra gpu --extra dev`. The `gpu` extra holds `diffusers`/`transformers`/`torch`, so a bare `uv sync` (no extras) removes them and `noai/__init__` (eager pipeline import) then fails. `maintain.sh`'s `uv sync --all-extras` also pulls the heavy `trustmark`/`lama` wheels (pytorch-lightning, onnxruntime) — fine on a good connection, but on flaky DNS sync only `--extra gpu --extra dev` and run the lint/test steps by hand.
|
||||
- To add a dev tool (pytest/ruff/pyright) into the env, use `uv sync --frozen --extra dev --extra gpu`, **never `uv pip install`** — `uv pip install` re-resolves and rewrites `uv.lock`, which silently bumped `transformers` to a build incompatible with the pinned `diffusers` (`cannot import name 'Qwen3VLForConditionalGeneration'`) and broke every `identify`/metadata import. Recovery: `git checkout uv.lock && uv sync --frozen --extra gpu --extra dev`. The `gpu` extra holds `diffusers`/`transformers`/`torch`, so a bare `uv sync` (no extras) removes them; `noai/__init__` is now **lazy** (PEP 562 `__getattr__`, so importing `identify`/`metadata` no longer pulls `watermark_remover`/torch), so a bare env breaks only when the removal pipeline is actually invoked, not on import. `maintain.sh`'s `uv sync --all-extras` also pulls the heavy `trustmark`/`lama` wheels (pytorch-lightning, onnxruntime) — fine on a good connection, but on flaky DNS sync only `--extra gpu --extra dev` and run the lint/test steps by hand.
|
||||
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `mj-*` = Midjourney IPTC, `doubao-1.png` = ByteDance Doubao with the China TC260 `<TC260:AIGC>` XMP label **and** a visible "豆包AI生成" text mark bottom-right; `grok-1.jpg` = xAI Grok with its EXIF-only `Signature:` blob + UUID `Artist` and no C2PA/SynthID/IPTC); synthetic byte blobs cover the JPEG/ISOBMFF format paths. The "non-AI / clean photo" control is no longer in `data/samples/` -- the `clean_photo` conftest fixture serves a verified-negative image from the corpus `neg/` set (skips if the corpus is absent).
|
||||
- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/`. The labeled `images/` (`pos/` `neg/` `cleaned/`) are **committed** (public repo -- review every image for private content before adding; `manifest.csv` is kept in sync with the files on disk, one row per tracked image); only the synthetic `refs/` calibration fills are gitignored. See its README for the collection protocol and verification oracles.
|
||||
|
||||
@@ -34,16 +34,16 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
- `noai/c2pa.py` — PNG chunk parser; use `extract_c2pa_chunk(path)` to get raw caBX payload, `has_c2pa_metadata(path)` to detect. Do not reimplement chunk parsing. `extract_c2pa_info(path)` sets `synthid_watermark`/`synthid_vendors` when the manifest is signed by a SynthID-using vendor, and `soft_binding`/`soft_binding_vendors` when a `c2pa.soft-binding` `alg` names a forensic-watermark vendor (`soft_binding_vendors_in(buffer)` is the shared byte-scan, used by both the PNG parser and the non-PNG binary path). PNG/caBX chunk reads are clamped to the remaining file size (`safe_length = min(length, remaining)`; skipped chunks use seek) so a malformed huge `length` cannot drive a multi-GB allocation (shared safety discipline matching `isobmff.scan_c2pa_region`).
|
||||
- `noai/constants.py` — PNG_SIGNATURE, C2PA_CHUNK_TYPE, C2PA_SIGNATURES, C2PA_ISSUERS, `SYNTHID_C2PA_ISSUERS` (issuers that pair SynthID with C2PA: Google, OpenAI), and `C2PA_SOFT_BINDINGS` (soft-binding `alg` prefix → forensic-watermark vendor: Adobe TrustMark, Digimarc, Imatag, Steg.AI, Microsoft, ...). Add a new issuer/binding here, not inline.
|
||||
- `metadata.py` — `scan_head(path, size=1MB)` is the shared input for every C2PA/AIGC/IPTC byte scan: first `size` bytes plus the payloads of any provenance metadata found beyond that window — for ISOBMFF, the late provenance boxes from `isobmff.scan_c2pa_region` (catches a manifest after a large `mdat`); for **PNG**, the late `tEXt`/`iTXt`/`zTXt`/`eXIf`/`iCCP` chunks from `_png_late_metadata` (catches an XMP/EXIF packet appended after a large `IDAT`, e.g. a TC260 AIGC label at ~2.7 MB). Behavior-neutral (`f.read(size)`) for non-ISOBMFF inputs and for any file that fits within `size`. Use it instead of `open().read(1MB)` for any new marker scan. `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan. `xai_signature(path)` detects xAI/Grok's EXIF-only scheme (`ImageDescription` = `Signature: <base64>` + UUID `Artist`); it feeds `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify`. `iptc_ai_system(path)` detects the IPTC Photo Metadata 2025.1 AI-disclosure XMP properties (`IPTC_AI_FIELD_MARKERS` = `AISystemUsed`/`AISystemVersionUsed`/`AIPromptInformation`/`AIPromptWriterName`) and returns the `AISystemUsed` generator name (or `"fields present"`). `remove_ai_metadata` routes **ISOBMFF video** (`.mp4`/`.mov`/`.m4v`) through the same `isobmff.strip_c2pa_boxes` as AVIF/HEIF (MP4 is ISOBMFF), and `_scrub_ai_exif` removes the xAI signature + AI-generator EXIF tags on JPEG output. `strip_c2pa_boxes` is **fail-safe** on a malformed box: it returns the original bytes unchanged with a logged warning instead of truncating the tail to EOF (detection-only `scan_c2pa_region` still stops at a malformed box). `_png_late_metadata` clamps each late-chunk read to the remaining file size (`safe_length = min(length, remaining)`) so a malformed `length` cannot drive a multi-GB allocation.
|
||||
- `identify.py` — the OpenAI rollout caveat is keyed on `_vendor_of(synthid) == "OpenAI"` (not a raw substring over the issuer + verdict blob). `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, C2PA soft-binding forensic-watermark vendor, IPTC "Made with AI" + IPTC 2025.1 `AISystemUsed`, embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, the China TC260 AIGC label via `metadata.aigc_label`, the HuggingFace `hf-job-id` job marker via `metadata.huggingface_job`, the Samsung Galaxy AI editing marker via `metadata.samsung_genai`, visible Gemini sparkle, open invisible watermark, Adobe TrustMark via `trustmark_detector`) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). The `hf_job`, visible-sparkle, and Samsung `samsung_genai` signals are **medium** confidence: each lifts an otherwise-Unknown verdict to a tentative AI (`hf_only` / `visible_only` / `samsung_only`, parallel branches) but is excluded from the high-confidence `ai_from_metadata` set, so none overrides a hard metadata signal. Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. **C2PA platform attribution is device-token-first, issuer-scan fallback** (`_device_platform` scans manifest bytes for `_DEVICE_C2PA_PLATFORM` tokens, then `_attribute_platform`/`_ISSUER_PLATFORM`). **Why, verified on real signed files 2026-05-26:** the old issuer-only byte-scan matched ANY issuer substring anywhere, so multi-entity manifests mis-attributed -- Leica→"Truepic" (a signing authority in the trust chain), Nikon→"Adobe Firefly" (XMP-toolkit "Adobe" + the sample's "Adobe_MAX" name), Pixel→"Google (Gemini)" ("Google LLC" cert org), Truepic→"Google". A distinctive device token wins instead. **Token distinctiveness is load-bearing:** bare `b"Truepic"` mis-fires (it appears in unrelated trust chains -- it mis-attributed the OpenAI `chatgpt-1.png` fixture), so the token is the specific `b"Truepic_Lens"` from the Lens SDK claim generator; likewise `b"Pixel Camera"` (cert CN) not bare `b"Pixel"`. `_DEVICE_C2PA_PLATFORM` lists ONLY tokens **verified against a real C2PA file**: Leica (`lc_c2pa`/`Leica Camera`), Nikon (`NIKON`), Pixel (`Pixel Camera` -- from a real Pixel 10 Pro file attached to c2pa-rs issue #1609/#1554), Sony (`sony.sig`/`sony.cert` -- Sony's own C2PA assertion namespace, verified on a real Sony PXW-Z300 file; NOT bare "Sony" which is a common EXIF Make), Truepic (`Truepic_Lens`). Canon/Bria have **no public direct-download C2PA sample** (checked exhaustively: GitHub issue/PR attachments, contentcredentials gallery, HF datasets -- all upload-to-verify or token-gated; Canon's only public file was a self-signed hobbyist CR3, not factory), so they stay unmapped until a real file is captured (same fixture discipline as Grok/Doubao). The Sony sample is video (MP4) -- our ISOBMFF C2PA path detects it; Sony Alpha stills likely share the `sony.*` namespace but are not separately verified. **Samsung Galaxy + ASUS Gallery live in a separate `_SIGNER_C2PA_PLATFORM` (scanned after `_device_platform`, before the issuer fallback), NOT in `_DEVICE_C2PA_PLATFORM`** — verified on real signed files 2026-05-29. Reason: a Galaxy phone stamps BOTH its device cert AND a `trainedAlgorithmicMedia`/genAIType AI marker on a Generative-Edit image, so treating it as a "genuine camera capture" would false-fire integrity-clash rule 2 on every Galaxy AI edit. The signer tokens (`b"Samsung Galaxy"` cert org — distinct from the EXIF `SM-xxxx` model string on ordinary Samsung photos; `b"com.asus.gallery"` claim generator) only resolve the platform label; the AI verdict still comes from the source-type / genAIType. ASUS Gallery is a C2PA-signed edit with no AI marker, so it attributes the platform without asserting `is_ai`. **Samsung's `genAIType` (in the proprietary `PhotoEditor_Re_Edit_Data` JSON) is an undocumented Galaxy-AI editing marker** (`metadata.samsung_genai`, gated on the `PhotoEditor_Re_Edit_Data` container; non-zero value = AI tool used, values {1,5} observed): medium-confidence because the field has no public spec (verified 2026-05-29: absent from C2PA spec + Samsung docs), but it co-occurred with `trainedAlgorithmicMedia` in 3/3 verified files that record a source-type and was the SOLE AI marker on a Galaxy S24 file that omits the source type. Camera C2PA marks capture authenticity, not AI (Pixel carries `computationalCapture`, not `trainedAlgorithmicMedia`), so these never set `is_ai` -- that stays driven by digital-source-type. `c2pa.cbor_text_after` (now public) is best-effort for the `generator` detail string only and can be None when the manifest keys it `claim_generator_info` (Pixel). **Issuer→generator mapping iLine truncated
|
||||
- `identify.py` — the OpenAI rollout caveat is keyed on `_vendor_of(synthid) == "OpenAI"` (not a raw substring over the issuer + verdict blob). `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, C2PA soft-binding forensic-watermark vendor, IPTC "Made with AI" + IPTC 2025.1 `AISystemUsed`, embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, the China TC260 AIGC label via `metadata.aigc_label`, the HuggingFace `hf-job-id` job marker via `metadata.huggingface_job`, the Samsung Galaxy AI editing marker via `metadata.samsung_genai`, the visible marks — Gemini sparkle plus the ByteDance Doubao 豆包AI生成 / Jimeng 即梦AI text marks via the `watermark_registry` — open invisible watermark, Adobe TrustMark via `trustmark_detector`) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). The `hf_job`, visible-mark, and Samsung `samsung_genai` signals are **medium** confidence: each lifts an otherwise-Unknown verdict to a tentative AI (`hf_only` / `visible_only` / `samsung_only`, parallel branches; `visible_only` fires on any `visible_*` signal) but is excluded from the high-confidence `ai_from_metadata` set, so none overrides a hard metadata signal. **Visible-mark detection** (`check_visible`, signals `visible_sparkle` / `visible_doubao` / `visible_jimeng`): the Gemini sparkle keeps its own file-level path (`_visible_sparkle` → `gemini_engine.detect_sparkle_confidence`, promoted only at confidence ≥ `_SPARKLE_THRESHOLD` 0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49), while Doubao/Jimeng reuse the registry detectors (`_visible_text_marks` → `watermark_registry`), each gated by its own engine NCC threshold via `MarkDetection.detected` (Doubao 0.4, Jimeng 0.45). Doubao/Jimeng are normally also caught by the TC260 AIGC metadata label, so the visible path is their stripped-metadata fallback. Visible marks set `platform` only when no harder signal already did, and (like the sparkle) are excluded from integrity-clash vendor claims. The cv2 dependency lives in the engines, not here. **`import identify` is deliberately light** (~21 MB; ~36 MB with cv2 loaded by a visible-mark run, ~106 MB for a full `check_visible` run): it imports only the pure `noai.c2pa`/`noai.constants` submodules, and `noai/__init__` is lazy (see "Test and lint"), so torch/diffusers are NOT pulled at import even in a full `gpu`/`detect` install — fits a 512 MB host. The heavy paths are opt-in: `check_invisible=True` needs the `detect`/`trustmark` extras (each pulls **torch**; TrustMark also **downloads weights**), so on a core-only deploy leave `check_invisible` off (it is a no-op there anyway). Before the lazy `__init__`, the mere presence of torch in the env inflated `import identify` to ~420 MB. **C2PA platform attribution is device-token-first, issuer-scan fallback** (`_device_platform` scans manifest bytes for `_DEVICE_C2PA_PLATFORM` tokens, then `_attribute_platform`/`_ISSUER_PLATFORM`). **Why, verified on real signed files 2026-05-26:** the old issuer-only byte-scan matched ANY issuer substring anywhere, so multi-entity manifests mis-attributed -- Leica→"Truepic" (a signing authority in the trust chain), Nikon→"Adobe Firefly" (XMP-toolkit "Adobe" + the sample's "Adobe_MAX" name), Pixel→"Google (Gemini)" ("Google LLC" cert org), Truepic→"Google". A distinctive device token wins instead. **Token distinctiveness is load-bearing:** bare `b"Truepic"` mis-fires (it appears in unrelated trust chains -- it mis-attributed the OpenAI `chatgpt-1.png` fixture), so the token is the specific `b"Truepic_Lens"` from the Lens SDK claim generator; likewise `b"Pixel Camera"` (cert CN) not bare `b"Pixel"`. `_DEVICE_C2PA_PLATFORM` lists ONLY tokens **verified against a real C2PA file**: Leica (`lc_c2pa`/`Leica Camera`), Nikon (`NIKON`), Pixel (`Pixel Camera` -- from a real Pixel 10 Pro file attached to c2pa-rs issue #1609/#1554), Sony (`sony.sig`/`sony.cert` -- Sony's own C2PA assertion namespace, verified on a real Sony PXW-Z300 file; NOT bare "Sony" which is a common EXIF Make), Truepic (`Truepic_Lens`). Canon/Bria have **no public direct-download C2PA sample** (checked exhaustively: GitHub issue/PR attachments, contentcredentials gallery, HF datasets -- all upload-to-verify or token-gated; Canon's only public file was a self-signed hobbyist CR3, not factory), so they stay unmapped until a real file is captured (same fixture discipline as Grok/Doubao). The Sony sample is video (MP4) -- our ISOBMFF C2PA path detects it; Sony Alpha stills likely share the `sony.*` namespace but are not separately verified. **Samsung Galaxy + ASUS Gallery live in a separate `_SIGNER_C2PA_PLATFORM` (scanned after `_device_platform`, before the issuer fallback), NOT in `_DEVICE_C2PA_PLATFORM`** — verified on real signed files 2026-05-29. Reason: a Galaxy phone stamps BOLine truncated
|
||||
- `watermark_registry.py` — **single catalog of known visible watermarks**, the unified "find known marks in their usual places, recognize, remove" entry. **Reverse-alpha based by policy**: a mark is listed only once a real alpha map has been captured for it, and removal inverts that map (`original = (wm - a*logo)/(1-a)`) — Gemini recovers cleanly with no inpaint (its sparkle alpha comes from a pure-black capture, so it is near-exact), while **Doubao and Jimeng both add an always-on THIN residual inpaint** over the glyph footprint (their text marks re-rasterize + jitter a few px per image, so a single capture cannot pixel-cancel them; the inpaint blends into the reverse-alpha-recovered pixels). Arbitrary-region inpainting still lives in `region_eraser`/`erase`. Each `KnownMark` ties a key to {usual `location`, `in_auto` flag, `recovery` (="reverse-alpha"), a `detect` adapter → uniform `MarkDetection`, a `remove` adapter}. Entries today: `gemini` (bottom-right sparkle), `doubao` (bottom-right "豆包AI生成"), and `jimeng` (bottom-right "★ 即梦AI"). `detect_marks` scans all; `best_auto_mark` picks the highest-confidence detection. **Cross-engine confidences aren't directly comparable**, so the gemini adapter applies the corpus-validated 0.5 sparkle threshold (`_GEMINI_AUTO_MIN_CONF`) for its `detected` flag — otherwise the gemini engine's loose internal threshold weakly fires (~0.36) on the Doubao text and hijacks `auto`. The shape-keyed Doubao/Jimeng NCC detectors don't cross-fire (jimeng scores ~0.22 on the Doubao strip, well under its 0.45 threshold), so `auto` picks the right one on a Doubao vs Jimeng image. `cli.cmd_visible` is registry-driven: `--mark auto` → `best_auto_mark`, `--mark <key>` → that mark; `--mark` choices come from `mark_keys()`. `_doubao_remove`/`_jimeng_remove` apply reverse-alpha only when the mark is detected AND `reverse_alpha_available`; outside that, removal is **skipped** (not inpainted). Add a new visible mark = one `KnownMark` entry + its engine (with a captured alpha map); do not re-add per-mark `if` branches in the CLI. **Alpha-on-save policy (issue #30):** `cli._write_bgr_with_alpha` rejoins the input's alpha plane **unchanged** — it must NOT zero alpha in the watermark bbox. Reverse-alpha (and `erase` inpaint) recover real pixels there, so zeroing alpha punched a transparent hole that renders as a solid **white box** on any non-transparent viewer (Gemini app exports are opaque RGBA, so every user hit it; regression-guarded by `test_visible_keeps_alpha_opaque_in_watermark_region`). The registry `remove()` still returns its region (used for `inpaint_residual` positioning), but the CLI no longer uses it to clear alpha.
|
||||
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`. The public entry points normalize a grayscale (2D) or RGBA (4-channel) input to BGR up front so a non-BGR image does not crash the cv2 pipeline. **Removal is reverse-alpha with NO inpaint** (`remove_watermark` → `_reverse_alpha_blend`): the sparkle alpha is computed (`alpha = max(R,G,B)/255`) from the bundled sparkle-on-black captures `assets/gemini_bg_{96,48}.png`, which are PURE-BLACK so the alpha is near-exact — re-verified clean on `demo_banana_before.png` 2026-05-31 (the registry's optional `inpaint_residual` is a no-op on a clean removal; an earlier "Gemini smears" read was a misjudged soft-fur original, not an artifact). **The bg assets are now rebuilt from OUR OWN controlled captures** (`data/gemini_capture/captures/`, committed) by `scripts/visible_alpha_solve.py gemini`, which locates the 96px sparkle on the black capture and crops it to the two logo sizes; our capture matched the previously third-party-sourced `gemini_bg_96.png` to **NCC 0.9998**, validating the asset and making it reproducible. Gemini's multi-size fixed-slot model is genuinely different from the Doubao/Jimeng text-strip engines (so it stays a separate engine, not part of the shared-base refactor).
|
||||
- `doubao_engine.py` — visible Doubao "豆包AI生成" remover/detector (cv2/numpy, no GPU). `DoubaoEngine.locate` anchors a bottom-right box by **geometry** (mark scales with image WIDTH), `extract_mask` pulls the light, low-chroma glyphs (the detection candidate) using a per-pixel channel-spread proxy `sat = roi.max(axis=2) - roi.min(axis=2)` (no HSV conversion). `detect` is **shape-consistent**: it matches the bundled alpha glyph silhouette (`assets/doubao_alpha.png`) against the candidate via zero-mean normalized correlation (`_template_match_score`, cv2 `TM_CCOEFF_NORMED`), gated at `DETECT_NCC_THRESHOLD` 0.4 over a small `DETECT_MIN_COVERAGE` floor. Keying on glyph SHAPE (not coverage heuristics) fixed #23 (corpus FP 7/1243). **Removal = reverse-alpha + thin residual inpaint** (`remove_watermark_reverse_alpha`): `original = (wm - a*logo)/(1-a)` from the bundled alpha map + `_ALPHA_LOGO_BGR` (pure white) + `_ALPHA_*_FRAC` geometry, then a deliberately THIN inpaint (`_RESIDUAL_*`, `INPAINT_NS`) over the glyph footprint clears leftover edges without smearing. **Alpha is rebuilt by `scripts/visible_alpha_solve.py` (the careful gray-self solve: cubic background fit, mean over channels, full halo, unblurred), same recipe as Jimeng** — the captures are committed in `data/doubao_capture/captures/`. **Removal aligns ALWAYS** (no `_ALPHA_NATIVE_BAND` fast-path): it tries fixed geometry AND `_aligned_alpha_map`'s `TM_CCOEFF_NORMED` scale+position search and keeps the lower-residual one — the mark is re-rasterized and a few px off per image, so fixed geometry alone leaves a visible outline even at 2048. **The locate box (`WM_*`) is generous (0.22 wide, margins 0.004) and reaches close to the corner** — a tight box (the old 0.185 / margin 0.012) let a corner-ward shift fall OUTSIDE the alignment search, so the align missed and a readable outline survived; regression-guarded by `test_recovers_shifted_mark_on_texture` (composes the alpha shifted on a known texture; old box ~29 vs new ~1 mean residual). **Issue #13 follow-up defect (found 2026-05-31): the SHIPPED Doubao removal left a clearly READABLE "豆包AI生成" outline on the real `doubao-1.png` sample, while `detect` returned conf 0.0 (it is fooled by a thin outline) so `test_reverse_alpha_removes_mark` passed and the old "56/56 clean" claim was detector-measured, not visual.** Root cause: bad alpha (under-estimated, max ~0.65) + fixed-no-inpaint + tight box; the careful rebuild + always-align + thin inpaint + wide box takes it from a readable outline to faint texture-level traces (parity with Jimeng — a single capture cannot pixel-cancel a per-image re-rasterized mark). **Lesson: a detector-only removal test is insufficient; assert visual residual (the textured-shift test).** `reverse_alpha_available` is just "asset present"; the registry gates removal on `detect`. The shipped third-party `_refs/zhengsuanfa_doubao_alpha_120x20.png` is NOT a usable alpha (verified 2026-05-29). Arbitrary-region inpainting is `region_eraser`/`erase`.
|
||||
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`. The public entry points normalize a grayscale (2D) or RGBA (4-channel) input to BGR up front so a non-BGR image does not crash the cv2 pipeline. **Removal is reverse-alpha with an over-subtraction guard** (`remove_watermark` → `_reverse_alpha_blend`, else `_inpaint_footprint`): the sparkle alpha is computed (`alpha = max(R,G,B)/255`) from the bundled sparkle-on-black captures `assets/gemini_bg_{96,48}.png` (the capture max is ~130, NOT 255 — the sparkle is a ~51%-opaque white overlay, so `alpha` maxes at ~0.51, which is CORRECT for the capture, not under-exposed). The alpha is near-exact only when the real mark's effective opacity matches the capture, which holds on bright/flat backgrounds — re-verified clean on `demo_banana_before.png` 2026-05-31. **Issue #30 (dark-background black pit):** on a dark/textured background (e.g. grass, ~73) the real sparkle's effective opacity is LOWER than the captured 0.51, so the fixed-alpha reverse blend OVER-subtracts (`watermarked - a*logo` goes negative) and drives the footprint to black — the white sparkle becomes a black diamond. `remove_watermark` now detects this via `_reverse_alpha_oversubtracts` (fraction of footprint pixels with `alpha >= _FOOTPRINT_ALPHA` 0.1 whose numerator < 0 exceeds `_OVERSUB_FOOTPRINT_FRAC` 0.05) and **inpaints the footprint** (`_inpaint_footprint`, cv2 NS over the dilated alpha mask) from the surrounding pixels instead. **Behavior-neutral on the working case:** a bright background over-subtracts at ~0% so reverse-alpha is used and the output is byte-identical to before (verified: demo_banana 0.0 frac vs issue-#30 grass 0.61 frac; regression-guarded by `test_gemini_engine.py::TestOverSubtractionGuard`, which composites the sparkle at a reduced effective alpha to reproduce the mismatch). The registry's optional `inpaint_residual` (edge cleanup) is a no-op on a clean reverse-alpha removal; an earlier "Gemini smears" read was a misjudged soft-fur original, not an artifact. **The bg assets are now rebuilt from OUR OWN controlled captures** (`data/gemini_capture/captures/`, committed) by `scripts/visible_alpha_solve.py gemini`, which locates the 96px sparkle on the black capture and crops it to the two logo sizes; our capture matched the previously third-party-sourced `gemini_bg_96.png` to **NCC 0.9998**, validating the asset and making it reproducible. Gemini's multi-size fixed-slot model is genuinely different from the Doubao/Jimeng text-strip engines (so it stays a separate engine, not part of the shared-base refactor).
|
||||
- `doubao_engine.py` — visible Doubao "豆包AI生成" remover/detector (cv2/numpy, no GPU). `DoubaoEngine.locate` anchors a bottom-right box by **geometry** (mark scales with image WIDTH), `extract_mask` pulls the light, low-chroma glyphs (the detection candidate) using a per-pixel channel-spread proxy `sat = roi.max(axis=2) - roi.min(axis=2)` (no HSV conversion). `detect` is **shape-consistent**: it matches the bundled alpha glyph silhouette (`assets/doubao_alpha.png`) against the candidate via zero-mean normalized correlation (`_template_match_score`, cv2 `TM_CCOEFF_NORMED`), gated at `DETECT_NCC_THRESHOLD` 0.4 over a small `DETECT_MIN_COVERAGE` floor. Keying on glyph SHAPE (not coverage heuristics) fixed #23 (corpus FP 7/1243). **Removal = reverse-alpha + thin residual inpaint** (`remove_watermark_reverse_alpha`): `original = (wm - a*logo)/(1-a)` from the bundled alpha map + `_ALPHA_LOGO_BGR` (pure white) + `_ALPHA_*_FRAC` geometry, then a deliberately THIN inpaint (`_RESIDUAL_*`, `INPAINT_NS`) over the glyph footprint clears leftover edges without smearing. **Alpha is rebuilt by `scripts/visible_alpha_solve.py` (the careful gray-self solve: cubic background fit, mean over channels, full halo, unblurred), same recipe as Jimeng** — the captures are committed in `data/doubao_capture/captures/`. **Removal aligns ALWAYS** (no `_ALPHA_NATIVE_BAND` fast-path): it tries fixed geometry AND `_aligned_alpha_map`'s `TM_CCOEFF_NORMED` scale+position search and keeps the lower-residual one — the mark is re-rasterized and a few px off per image, so fixed geometry alone leaves a visible outline even at 2048. **The locate box (`WM_*`) is generous (0.22 wide, margins 0.004) and reaches close to the corner** — a tight box (the old 0.185 / margin 0.012) let a corner-ward shift fall OUTSIDE the alignment search, so the align missed and a readable outline survived; regression-guarded by `test_recovers_shifted_mark_on_texture` (composes the alpha shifted on a known texture; old box ~29 vs new ~1 mean residual). **Issue #13 follow-up defect (found 2026-05-31): the SHIPPED Doubao removal left a clearly READABLE "豆包AI生成" outline on the real `doubao-1.png` sample, while `detect` returned conf 0.0 (it is fooled by a thin outline) so `test_reverse_alpha_removes_mark` passed and the old "56/56 clean" claim was detector-measured, not visual.** Root cause: bad alpha (under-estimated, max ~0.65) + fixed-no-inpaint + tight box; the careful rebuild + always-align + thin inpaint + wide box takes it from a readable outline to faint texture-level traces (parity with Jimeng — a single capture cannot pixel-cancel a per-image re-rasterized mark). **Lesson: a detector-only removal test is insufficient; assert visual residual (the textured-shift test).** **`extract_mask` guards a degenerate ROI (`bh < 16 or bw < 16` -> empty mask, skips cv2):** the always-align removal scores each placement with a residual `detect(out)`, and on an extremely wide/short image (e.g. 2048x1, `test_wide_short_does_not_raise`) that fed cv2's GaussianBlur a ~1-px-tall ROI and **faulted natively on Windows py3.12 (access violation, non-deterministic — one CI cell went red while a re-run passed)**; the old at-native path never ran `detect` on degenerate sizes. Real images always clear the guard (the `WM_*` box floors are `max(16, …)` height / `max(40, …)` width), so it only short-circuits slivers. `reverse_alpha_available` is just "asset present"; the registry gates removal on `detect`. The shipped third-party `_refs/zhengsuanfa_doubao_alpha_120x20.png` is NOT a usable alpha (verified 2026-05-29). Arbitrary-region inpainting is `region_eraser`/`erase`.
|
||||
- `jimeng_engine.py` — visible Jimeng / Dreamina "★ 即梦AI" remover/detector (cv2/numpy, no GPU), built 2026-05-30 from issue #13's solid captures (@powersee). Mirrors `doubao_engine`: `locate` anchors a bottom-right box by **geometry** (scales with WIDTH), `extract_mask` pulls the light low-chroma glyphs (white top-hat + grayish + min-luma), `detect` matches the bundled "即梦AI" glyph silhouette (`assets/jimeng_alpha.png`) via `TM_CCOEFF_NORMED` over a coverage floor. Threshold `DETECT_NCC_THRESHOLD` **0.45** cleanly separates real Jimeng marks (>=0.81) from the Doubao strip (0.21) and other AI output (0.0), so the two ByteDance marks don't cross-fire in `--mark auto`. **Logo is pure white (255,255,255)** (`_ALPHA_LOGO_BGR`; the white capture + an L-pair-solve confirm ~254.6); compositing is **sRGB, not linear** (a linear-light solve tripled the cross-residual). **Alpha rebuilt by `scripts/visible_alpha_solve.py` from the GRAY capture** (`data/jimeng_capture/captures/`, the solid captures now committed): `a = (I - B)/(255 - B)`, B a per-capture **cubic** background fit over the non-glyph pixels, **averaged over channels, full halo extent (down to a~0.02), unblurred**. Gray (bg ~132) is the deliberate choice over black: it is the best proxy for real content (the mark sits on bright photo areas, not on black), and the careful build drops the gray self-residual to ~1.3. **The mask quality, not the method, was the earlier limit** — a max-channel / quadratic-bg / blurred / halo-truncated build (and a black-dominated LS) left a visible outline (lesson from issue #13: when reverse-alpha leaves a ghost, suspect the captured alpha map before adding heuristics or switching method). Geometry emitted by the solver at `_ALPHA_NATIVE_WIDTH` 2048: `_ALPHA_WIDTH_FRAC` 0.202, `_ALPHA_HEIGHT_FRAC` 0.058, margins ~0.029. **Removal = reverse-alpha + a deliberately THIN residual inpaint** (`remove_watermark_reverse_alpha`, `_RESIDUAL_DILATE` 5 over the `_RESIDUAL_ALPHA_FLOOR` 0.05 footprint, `_RESIDUAL_INPAINT_RADIUS` 2, `INPAINT_NS`): a single 2048 alpha cannot pixel-cancel the mark re-rasterized at another resolution (alpha maps from independent captures correlate 0.998, not 1.0; off-native reverse-alpha alone only halves the mark), so a tight inpaint clears the residual edges WITHOUT the texture/edge smear a wide full-footprint pass caused. **Placement ALWAYS tries fixed geometry AND `_aligned_alpha_map`'s NCC scale+position search, keeping the lower-residual** — the mark re-rasterizes + jitters a few px per image even at the captured width, so fixed geometry alone misses (there is no `_ALPHA_NATIVE_BAND` fast-path; the scale search `_ALPHA_ALIGN_SEARCH` is fine-stepped, and the `WM_*` locate box is generous so a corner-ward shift stays inside the search — the same widen that fixed Doubao). Verified clean on the solid captures (native 2048; faint self-residual ~1.3 visible only on a dead-flat field, hidden by real texture) and a real 1440-wide Jimeng download (off-native, table edge preserved). `reverse_alpha_available` is just "asset present"; the registry gates on `detect`. **No committed real sample** (the real content download stays gitignored; only the solid calibration captures are committed) — `tests/test_jimeng_engine.py` synthesizes a mark from the bundled alpha asset, and `test_recovers_shifted_mark_on_texture` guards the align-on-shift path that the Doubao defect exposed. Jimeng images are independently caught by the China TC260 AIGC label in `metadata`/`identify`, so this engine is the visible-mark *removal* path, not a new `identify` signal.
|
||||
- `region_eraser.py` — universal region eraser (`erase` CLI). `erase(image, boxes=|mask=, backend=)` normalizes grayscale (2D) and RGBA (4-channel) inputs up front (`erase_cv2` splits off any alpha plane and re-attaches it on the result): `boxes_to_mask` → `cv2.inpaint` (`cv2` backend, default, no deps) or big-LaMa via onnxruntime (`lama` backend, extra `lama`, `Carve/LaMa-ONNX` Apache-2.0 model downloaded on first use, never bundled). `erase_lama` crops a padded region around the mask, runs LaMa at its fixed 512² input, pastes only masked pixels back (untouched areas stay pixel-exact). Lazy `_get_lama_session` singleton; `lama_available()` guards the optional import. **LaMa-ONNX costs ~3.5-4 GB peak RAM and ~5-6 s/call on CPU** (FFC working set, not arena — `enable_cpu_mem_arena=False` does not help), so it does NOT fit a minimal droplet; the cv2 backend (tens of MB, ~30 ms) does. LaMa quality at low RAM = serverless/GPU, mirroring how raiw.cc offloads SDXL to fal.
|
||||
- `invisible_watermark.py` — `detect_invisible_watermark(path)` decodes the OPEN DWT-DCT watermarks (public decoder, no key) embedded by Stable Diffusion / SDXL / FLUX via the `imwatermark` library. Known fixed patterns (verified against upstream source) live in `_BITS_48` (SDXL 48-bit, FLUX.2 48-bit) and `_SD1_STRING` ("StableDiffusionV1", SD 1.x/2.x). Optional dep (extra `detect`); returns None when absent. The `detect` extra pulls **torch** transitively (invisible-watermark declares torch a hard dep, and `WatermarkDecoder` eagerly imports `rivaGan` -> `torch` at import time), so detection needs torch present even though dwtDct runs CPU-only on cv2/numpy/pywavelets — no GPU and no separate `gpu` extra required. **Unlike SynthID this is locally detectable**, but the watermark is fragile (does not survive JPEG re-encode/resize — verified gone after JPEG q90), so it confirms origin only on pristine files. Add new known patterns here. The file carries a top-of-module pyright pragma because imwatermark/cv2 ship no type stubs.
|
||||
- `trustmark_detector.py` — `detect_trustmark(path)` decodes the OPEN, keyless **Adobe TrustMark** watermark (the soft binding behind Adobe Durable Content Credentials, `alg` `com.adobe.trustmark.P`) via the optional `trustmark` package (extra `trustmark`; pulls torch, downloads model weights on first use). Mirrors `invisible_watermark.py` (lazy singleton guarded by a double-checked `threading.Lock` so concurrent callers do not double-download the weights, top-of-module pyright pragma, returns None when absent). It detects *provenance*, not AI origin as such (TrustMark also marks human-authored content), so `identify` lists it as a watermark without setting `is_ai_generated`. Other soft-binding vendors (Digimarc/Imatag/Steg.AI/...) have no public decoder — they are only *named* via the `C2PA_SOFT_BINDINGS` scan, not decoded. **False-positive gate (added 2026-05-29):** TrustMark's `wm_present` is a BCH error-correction validity flag that spuriously validates on a content-correlated fraction of un-watermarked images — AI-generated textures trip it far more than camera photos (verified 2026-05-29 on real files: it fires on Gemini/OpenAI/Doubao output that *cannot* carry Adobe's watermark, with a random-bytes decoded secret, while signal-free camera photos did not trip it). A genuine TrustMark is a *durable* soft binding engineered to survive re-encoding, so `detect_trustmark` re-decodes after a mild JPEG round-trip (`_survives_reencode`, `_REENCODE_QUALITY` 95) and requires the same schema both times; every observed false positive collapsed (none survived even q95), so the gate is the durability property the watermark guarantees. The second decode runs only on the rare initial hit, so the cost is negligible. Do NOT remove the gate to "catch more" — a lone TrustMark hit without it is almost always content noise.
|
||||
- `text_protector.py` — text-region protection for the `invisible` SDXL img2img pass (issue #21: CJK/small text deforms at watermark-removal strengths). `is_available()` gates on `cv2.dnn.TextDetectionModel_DB`; `TextProtector.detect_text_boxes(bgr)` runs the **PP-OCRv3 DB** ONNX detector (~2.4 MB, Apache-2.0, opencv_zoo, returns rotated quad polygons) — downloaded+cached to `~/.cache/remove-ai-watermarks` on first use via atomic temp-rename, never bundled, **no torch (cv2.dnn only)**. **Detection is script-agnostic** (DB segments text *regions*, not characters), so Latin / Cyrillic / CJK / Hangul / Arabic / digits all detect identically — language was never the recall lever, **resolution was**. `_detection_input_size(h, w)` (pure, unit-tested) detects at the **native long side capped at `_DET_MAX_LONG_SIDE` (1536), never upscaled**: the old fixed 736 downscaled large canvases so small text fell below the detector and was missed (issue #14, e.g. ~16 px text on a 2048 image). `scripts/text_detection_benchmark.py` measures recall across scripts × sizes × canvas: the cap fix lifts overall hit-rate 0.91 → 1.00 (worst cell 2048/16 px: 0.06 → 1.00) at ~100 ms CPU. Very large canvases with tiny text may still need tiling (documented limit, not built). `build_change_map(boxes, h, w, preserve=0.9, feather=15)` paints a Differential-Diffusion change map. **Polarity (verified empirically):** white(1.0)=PRESERVE original pixels, black(0.0)=MAX change; map is black bg + `preserve` inside text polygons, Gaussian-feathered edges, clipped to [0,1]. `preserve` stays below a hard 1.0 freeze by default so text still scrubs lightly (SynthID survives cropping). **Default text protection is `watermark_remover._run_region_hires`, NOT the differential change map.** Differential Diffusion froze text in latent space (`preserve`<1.0), so the watermark survived *inside* text — violating the "remove SynthID everywhere" requirement; and the SDXL VAE's 8px latent cell softens sub-8px strokes regardless of `preserve` (architectural limit, confirmed by the DD authors — see `docs/text-protection-research.md`). `_run_region_hires` instead: (1) scrubs the whole image (plain img2img), (2) RE-scrubs each detected text block at HIGH resolution and feather-composites it back. `merge_text_regions(boxes,h,w)` groups boxes into local blocks; each crop is upscaled by `_REGION_HIRES_SCALE` 3.0 (applied as an integer factor via int(...), capped so a region stays under `_REGION_MAX_MEGAPIXELS` 1.3 to avoid OOM; skipped if it can't reach 2x — very large text areas then fall back to the global scrub, tiling is the future fix), img2img-scrubbed, downscaled, **phase-correlated back to the original crop to null the ~1-2px round-trip offset** (the shift is applied only on a confident, small correlation -- `response > 0.3` and `|shift| < 4` -- so a spurious large offset on a flat crop no longer garbles the composite; and after a CPU fallback the generator is dropped before the per-region passes to avoid an MPS-vs-CPU generator device mismatch) (a sub-pixel shift garbles the composite even when text is crisp; integer scale alone did NOT fix it because the diffusion pipeline rounds dims to a multiple of 8), then `feather_paste`d. Every pixel is regenerated, so the watermark is removed everywhere AND small text stays crisp (high-res strokes span >1 latent cell). Validated on synthetic 18px multilingual text: text-region SSIM 0.28 (plain) → 0.48 (region-hires), visually garbled → readable across Latin/Cyrillic/CJK, residual shift ~0.5px. Gated to the SDXL `DEFAULT_MODEL_ID` + detector (`_can_protect_text`); no text → plain global scrub (text-free inputs pay only the cheap cv2 detection). CLI off-switch `--no-protect-text` on `invisible`/`all`. `merge_text_regions` + `feather_paste` are pure, unit-tested without a model (`tests/test_text_protector.py`). **MUST still be confirmed by the SynthID oracle** (openai.com/verify / Gemini app) that a region-rescrubbed text zone reads watermark-free before trusting it in prod. The legacy `_run_differential` / `build_change_map` / `_load_differential_pipeline` (community `pipeline_stable_diffusion_xl_differential_img2img`, `custom_revision="0.38.0"`) remain in the file but are no longer the default; the diff pipeline upcasts the VAE to fp32 internally, so do **not** add `upcast_vae()`/`enable_attention_slicing` there (NaN/black on fp16 MPS). `build_change_map` is still unit-tested.
|
||||
- `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features
|
||||
- `text_protector.py` — text-region protection for the `invisible` SDXL img2img pass (issue #21: CJK/small text deforms at watermark-removal strengths). `is_available()` gates on `cv2.dnn.TextDetectionModel_DB`; `TextProtector.detect_text_boxes(bgr)` runs the **PP-OCRv3 DB** ONNX detector (~2.4 MB, Apache-2.0, opencv_zoo, returns rotated quad polygons) — downloaded+cached to `~/.cache/remove-ai-watermarks` on first use via atomic temp-rename, never bundled, **no torch (cv2.dnn only)**. **Detection is script-agnostic** (DB segments text *regions*, not characters), so Latin / Cyrillic / CJK / Hangul / Arabic / digits all detect identically — language was never the recall lever, **resolution was**. `_detection_input_size(h, w)` (pure, unit-tested) detects at the **native long side capped at `_DET_MAX_LONG_SIDE` (1536), never upscaled**: the old fixed 736 downscaled large canvases so small text fell below the detector and was missed (issue #14, e.g. ~16 px text on a 2048 image). `scripts/text_detection_benchmark.py` measures recall across scripts × sizes × canvas: the cap fix lifts overall hit-rate 0.91 → 1.00 (worst cell 2048/16 px: 0.06 → 1.00) at ~100 ms CPU. Very large canvases with tiny text may still need tiling (documented limit, not built). `build_change_map(boxes, h, w, preserve=0.9, feather=15)` paints a Differential-Diffusion change map. **Polarity (verified empirically):** white(1.0)=PRESERVE original pixels, black(0.0)=MAX change; map is black bg + `preserve` inside text polygons, Gaussian-feathered edges, clipped to [0,1]. `preserve` stays below a hard 1.0 freeze by default so text still scrubs lightly (SynthID survives cropping). **Default text protection is `watermark_remover._run_region_hires`, NOT the differential change map.** Differential Diffusion froze text in latent space (`preserve`<1.0), so the watermark survived *inside* text — violating the "remove SynthID everywhere" requirement; and the SDXL VAE's 8px latent cell softens sub-8px strokes regardless of `preserve` (architectural limit, confirmed by the DD authors — see `docs/text-protection-research.md`). `_run_region_hires` instead: (1) scrubs the whole image (plain img2img), (2) RE-scrubs each detected text block at HIGH resolution and feather-composites it back. `merge_text_regions(boxes,h,w)` groups boxes into local blocks; each crop is upscaled by `_REGION_HIRES_SCALE` 3.0 (applied as an integer factor via int(...), capped so a region stays under `_REGION_MAX_MEGAPIXELS` 1.3 to avoid OOM; skipped if it can't reach 2x — very large text areas then fall back to the global scrub, tiling is the future fix), img2img-scrubbed, downscaled, **phase-correlated back to the original crop to null the ~1-2px round-trip offset** (the shift is applied only on a confident, small correlation -- `response > 0.3` and `|shift| < 4` -- so a spurious large offset on a flat crop no longer garbles the composite; and after a CPU fallback the generator is dropped before the per-region passes to avoid an MPS-vs-CPU generator device mismatch) (a sub-pixel shift garbles the composite even when text is crisp; integer scale alone did NOT fix it because the diffusion pipeline rounds dims to a multiple of 8), then `feather_paste`d. Every pixel is regenerated, so the watermark is removed everywhere AND small text stays crisp (high-res strokes span >1 latent cell). Validated on synthetic 18px multilingual text: text-region SSIM 0.28 (plain) → 0.48 (region-hires), visually garbled → readable across Latin/Cyrillic/CJK, residual shift ~0.5px. Gated to the SDXL `DEFAULT_MODEL_ID` + detector (`_can_protect_text`); no text → plain global scrub (text-free inputs pay only the cheap cv2 detection). CLI opt-in `--protect-text` on `invisible`/`all` (**OFF by default** — see SynthID bullet). `merge_text_regions` + `feather_paste` are pure, unit-tested without a model (`tests/test_text_protector.py`). **The high-res re-scrub can shield SynthID in text regions** (verified 2026-06-01: same gpt-image, with `--protect-text` → SynthID detected by oracle; without → SynthID removed). The mechanism: the global pass at step 1 removes SynthID everywhere, but the per-region high-res re-scrub at step 2 regenerates those pixels from a higher-resolution crop -- if the per-region strength is insufficient at the effective upscaled resolution, SynthID can reconstitute. Until this is resolved, `protect_text` is EXPERIMENTAL and OFF by default. The legacy `_run_differential` / `build_change_map` / `_load_differential_pipeline` (community `pipeline_stable_diffusion_xl_differential_img2img`, `custom_revision="0.38.0"`) remain in the file but are no longer the default; the diff pipeline upcasts the VAE to fp32 internally, so do **not** add `upcast_vae()`/`enable_attention_slicing` there (NaN/black on fp16 MPS). `build_change_map` is still unit-tested.
|
||||
- `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features. The expensive extract+blend already runs only when a face is found, but the YOLO detector itself always loads+runs to decide; CLI opt-in `--protect-faces` on `invisible`/`all` (**OFF by default**, experimental). **Face protection has an even more direct SynthID-preservation mechanism than text protection:** it extracts face regions from the ORIGINAL (watermarked) image BEFORE the diffusion pass, then blends those original pixels BACK after the global pass. Those restored pixels are the unprocessed originals -- SynthID is guaranteed to survive in face regions (not just possibly, as with text re-scrub). Any image with faces processed with `--protect-faces` will have SynthID intact in the face areas regardless of strength.
|
||||
- `humanizer.py` — optional post-process "humanize" effects (cv2/numpy). The chromatic-shift step replicates the border instead of wrapping opposite-edge pixels, so a shifted channel no longer bleeds the far edge into the near one.
|
||||
- `image_io.py` — Unicode-safe cv2 IO (issue #17). `imread(path, flags=None)` / `imwrite(path, img)` wrap `np.fromfile`+`cv2.imdecode` / `cv2.imencode`+`tofile` so non-ASCII paths work on Windows -- bare `cv2.imread`/`cv2.imwrite` use the platform ANSI code-page API there and fail (empty decode + `can't open/read file`) on Chinese/Cyrillic/accented filenames. `imread` keeps `cv2.imread` semantics (defaults to `IMREAD_COLOR`, returns `None` on missing/empty/undecodable). **Every cv2 file read/write in the package routes through here; do not call `cv2.imread`/`cv2.imwrite` directly.** `imwrite` returns `False` on an unwritable path (`OSError` caught) instead of raising, matching `cv2.imwrite` semantics. macOS/Linux already accept UTF-8 paths, so it is behavior-neutral there (the bug only reproduces on Windows). cv2/numpy are imported lazily inside the functions, so the module is cheap to import in a bare env.
|
||||
|
||||
@@ -76,12 +76,16 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
|
||||
|
||||
## Known limitations
|
||||
|
||||
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data point (verified 2026-05-25 on a 1254x1254 gpt-image SDXL run, fp32, 20 GB MPS ceiling):** native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. Adding `enable_vae_tiling()` alone does NOT prevent it (the peak is the UNet, not the VAE). The fast Mac workarounds are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
|
||||
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data points (the OOM is memory-tier-dependent, NOT a hard MPS limit):** on a ~24 GB unified-memory machine (verified 2026-05-25, 1254x1254 gpt-image SDXL, fp32) native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. On a **32 GB** unified-memory machine the same default SDXL pass runs entirely on MPS with **no CPU fallback** (verified 2026-05-31, 1122x1402 gpt-image, `all`/default, ~155 s end-to-end), so 32 GB clears the native-res UNet peak that 24 GB could not. Adding `enable_vae_tiling()` alone does NOT prevent the 24 GB OOM (the peak is the UNet, not the VAE). The fast Mac workarounds for memory-constrained machines are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. **ctrlregen is compute-bound, not memory-bound, on MPS:** the clean-noise profile tiles to 512px (e.g. ~12 tiles for a 1122x1402 image) and runs the full step count per tile (strength 1.0 -> ~50 effective steps/tile), so on a base-tier Apple-GPU laptop it is ~25-30 min/image after the one-time DINOv2-giant download, regardless of having 32 GB -- a discrete CUDA GPU (fp16, no tiling) is the right place for ctrlregen, while the default SDXL pass is comfortable on a 32 GB Mac. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
|
||||
- **fp16 VAE black-output fix (issue #29, 2026-05-30):** on a **CUDA/XPU fp16** backend the stock SDXL VAE overflows to NaN and the *plain* img2img path decodes to an **all-black** image (reproduced on the raiw.cc result: a 1086x1448 input -> a uniformly black 4.6 KB PNG, mean 0). `watermark_remover._load_pipeline` now swaps in the fp16-fixed SDXL VAE (`madebyollin/sdxl-vae-fp16-fix` = `_SDXL_FP16_VAE_ID`) when `_needs_fp16_vae_fix(model_id, DEFAULT_MODEL_ID, is_fp16)` is true -- only the default SDXL checkpoint on fp16. **cpu/mps run fp32** (the stock VAE is fine there, which is why the bug never reproduces on Mac), and the **differential / region-hires** pipeline already upcasts the VAE itself (see the `text_protector` bullet). A custom non-SDXL `model_id` keeps its own VAE (the fp16-fix VAE is SDXL-architecture-specific). The decision is a pure helper, unit-tested without a download (`tests/test_platform.py::TestFp16VaeFix`); the actual black->clean recovery needs a CUDA GPU and was NOT verifiable on this MPS machine -- confirm on the backend / an NVIDIA box.
|
||||
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files.
|
||||
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
|
||||
- **rich `console.print` parses `[word]` as a style tag and silently drops unknown ones.** A literal bracketed token in a print string disappears: `pip install 'remove-ai-watermarks[gpu]'` rendered as `...remove-ai-watermarks'` (the `[gpu]` extra eaten), which sent users a broken install command (surfaced via #19). Escape the literal bracket as `\[gpu]` (in a normal Python string that is `"\\[gpu]"`) in any rich string carrying user-facing brackets. Regression-guarded by `tests/test_cli.py::TestGpuHintMarkup`.
|
||||
- **rich was dropped (CLI + scripts print plain text via `click.echo`).** `cli.py` renders through small `_Console`/`_Table`/`_Progress` shims; the analysis scripts (`scripts/synthid_corpus.py`, `synthid_pixel_probe.py`, `text_detection_benchmark.py`, `corpus_gap_scan.py`) import `Console`/`Table` from the shared `scripts/_plain_console.py` shim (markup like `[bold]`/`[/]` is stripped, tables render aligned). Consequences: (1) `rich` is NOT a dependency, so anything that imports it breaks a clean `uv sync --frozen` (CI installs core+dev only) — this exact gap red-failed CI after the refactor when those 4 scripts still imported rich; if you add a script, use the `_plain_console` shim, not rich. (2) The old `[gpu]`-bracket-eaten bug (#19) is gone — plain `click.echo` prints `pip install 'remove-ai-watermarks[gpu]'` verbatim, no escaping needed (regression-guarded by `tests/test_cli.py::TestGpuHintMarkup`). (3) No Unicode glyphs / colors / progress bars in CLI output by design.
|
||||
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding), which now also drops a top-level XMP ``uuid`` box that carries an AI label (matched by AI-marker content, not by the XMP UUID, so byte-order-robust) and covers MP4/MOV/M4V/M4A by content sniff. **Non-ISOBMFF audio/video removal is via ffmpeg** (`_FFMPEG_STRIP_EXTS` -> `_strip_with_ffmpeg`): WebM/Matroska (EBML), MP3 (ID3), WAV/FLAC/OGG (RIFF/Vorbis) are stripped losslessly with `ffmpeg -map_metadata -1 -map_chapters -1 -c copy` (codec data untouched). Requires ffmpeg on PATH; raises `RuntimeError` if absent or if ffmpeg can't parse the file. Verified end-to-end (a real ffmpeg-made WAV/MP3 with a `title=Suno AI` tag -> tag gone, audio bytes preserved). **Meta-box XMP now handled (`isobmff.blank_ai_xmp_packets`, v0.6.9):** an AI-label XMP packet stored as a meta-box `mime` item (AVIF/HEIF) is blanked in place (overwritten with spaces of the same length, so `iloc` offsets and the coded image stay valid). **Still NOT built:** an `Exif` *item* inside the `meta` box (rare -- AI labels are XMP) needs full `iinf`/`iloc` surgery (offset rewrite) with corruption risk -- exiftool (R/W/C for HEIC/AVIF EXIF+XMP, verified on exiftool.org 2026-05-27) would do it but is a non-installed binary dep, so it stays a documented gap. **Audio watermark DETECTION (Resemble PerTh) was evaluated and NOT built (2026-05-26):** `resemble-perth`'s `PerthImplicitWatermarker.get_watermark()` returns a raw bit-array with **no presence/confidence flag** (clean audio decodes to arbitrary bits too), so reliably distinguishing watermarked-from-clean needs either Resemble's fixed payload or a confidence API -- neither is public, and there's no real Resemble sample to calibrate against. Same wall-class as the SynthID pixel detector: the decode exists, reliable presence-detection does not. (perth's top-level `PerthImplicitWatermarker` is also gated to None unless `librosa` is importable.)
|
||||
- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). Authoritative confirmation: Google DeepMind's own paper "SynthID-Image: Image watermarking at internet scale" (Gowal et al., arXiv:2510.09263) states the verification service is restricted to "trusted testers" and does not release detector weights or a reproducible algorithm — so a local pixel detector is infeasible by design, not just unbuilt. https://arxiv.org/abs/2510.09263 We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. **Surface-dependent blind spot (verified 2026-05-24):** the same Google model emits different metadata per surface -- the Gemini *app* wraps outputs in Google C2PA, but the *API/playground* (AI Studio, Nano Banana / gemini-2.5-flash-image) emits the SynthID *pixel* watermark (confirmed via the Gemini-app oracle) + the visible sparkle but **no C2PA/IPTC at all**, so `synthid_source` returns None despite SynthID being present. Only the pixel oracle or the visible-sparkle detector catches those. (Meta AI is another surface mismatch: it writes the IPTC `digitalSourceType=trainedAlgorithmicMedia` marker, not C2PA and not SynthID.) Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). **Each vendor's oracle detects only its OWN content (verified on the page 2026-05-31):** `openai.com/research/verify` states verbatim "OpenAI generation signals will only be detected if the image was generated with our tools" and "Content could also still be AI-generated by another company's model, which the tool currently does not detect" -- SynthID is shared tech but the verifier is keyed to its own vendor's payload, so a Google-SynthID image reads clean on OpenAI's verifier and vice-versa. **This explains the recurring "oracle says clean but `identify` still flags SynthID" report (#14):** the oracle reads the *pixel* watermark (gone after our SDXL pass), while `identify` reads the *C2PA-metadata proxy* (still present if the manifest survived). Different signals, not a contradiction -- strip the metadata too (`metadata --remove` / `all`) and the proxy goes quiet, but a quiet proxy is not proof the pixel watermark is gone. The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. A from-scratch gpt-image pilot (2026-05-24) confirmed this independently: 5 independent solid-black gpt-image outputs share a near-identical fixed signature (pairwise residual correlation **0.92**, avg-template retains 97% energy), so the watermark/carrier IS strongly present and consistent on flat content — but the carrier frequencies extracted from it do NOT discriminate real content (carrier-to-random ratio: cleaned 1.86 > watermarked 1.53; a non-gpt-image image scored highest at 3.67). The signature drowns in content texture. Net: a perfectly consistent solid-color signature still yields no real-content pixel detector with magnitude/carrier methods. A corpus discrimination test (2026-05-24, `scripts/synthid_pixel_probe.py`, raw zero-mean residual NCC) independently re-confirms this: at matched resolution, SynthID positives do NOT cluster apart from negatives (within-Gemini 0.07; at 1024 px pos-vs-neg >= pos-vs-pos). The only high correlations were near-duplicate *content* (5 ChatGPT renders of one prompt at ~0.92, while a distinct ChatGPT image scored ~0 against them) — content, not a carrier. The probe is solid-fills-only and EXPERIMENTAL/DIAGNOSTIC; do not use it on real content. **Correction (deeper re-examination 2026-05-25):** the carrier IS real on solid fills — the earlier "no carrier" was a *method* artifact of using spatial / FFT-magnitude NCC, which can't see it. The carrier is a fixed *phase* at specific low frequencies, so the right metric is **per-bin phase coherence**. On 8 white `gemini-2.5-flash-image` fills (generated via the reverse-SynthID trick: identity-edit prompt "Recreate this image exactly as it is" Line truncated
|
||||
- **SynthID technical reference: `docs/synthid.md`** — primary-source-cited doc covering mechanism (post-hoc encoder/decoder pair, 136-bit payload at 512x512, pixel-space, model weights NOT modified), robustness numbers (arXiv:2510.09263: ~99.98% TPR@0.1%FPR across 30 transforms including JPEG/crop/resize/color/noise), removal attacks and forensic detectability (arXiv:2605.09203: all 6 attacks detectable at >98% TPR@1%FPR), detectability limits (no public decoder, metadata-proxy only), oracle scope, and adoption landscape. Read that doc first before adding notes here.
|
||||
|
||||
- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). Authoritative confirmation: Google DeepMind's own paper "SynthID-Image: Image watermarking at internet scale" (Gowal et al., arXiv:2510.09263) states the verification service is restricted to "trusted testers" and does not release detector weights or a reproducible algorithm — so a local pixel detector is infeasible by design, not just unbuilt. https://arxiv.org/abs/2510.09263 We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. **Surface-dependent blind spot (verified 2026-05-24):** the same Google model emits different metadata per surface -- the Gemini *app* wraps outputs in Google C2PA, but the *API/playground* (AI Studio, Nano Banana / gemini-2.5-flash-image) emits the SynthID *pixel* watermark (confirmed via the Gemini-app oracle) + the visible sparkle but **no C2PA/IPTC at all**, so `synthid_source` returns None despite SynthID being present. Only the pixel oracle or the visible-sparkle detector catches those. (Meta AI is another surface mismatch: it writes the IPTC `digitalSourceType=trainedAlgorithmicMedia` marker, not C2PA and not SynthID.) Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). **Each vendor's oracle detects only its OWN content (verified on the page 2026-05-31):** `openai.com/research/verify` states verbatim "OpenAI generation signals will only be detected if the image was generated with our tools" and "Content could also still be AI-generated by another company's model, which the tool currently does not detect" -- SynthID is shared tech but the verifier is keyed to its own vendor's payload, so a Google-SynthID image reads clean on OpenAI's verifier and vice-versa. **This explains the recurring "oracle says clean but `identify` still flags SynthID" report (#14):** the oracle reads the *pixel* watermark (gone after our SDXL pass), while `identify` reads the *C2PA-metadata proxy* (still present if the manifest survived). Different signals, not a contradiction -- strip the metadata too (`metadata --remove` / `all`) and the proxy goes quiet, but a quiet proxy is not proof the pixel watermark is gone. **SynthID is durable to JPEG re-encode by design, so a GitHub-recompressed issue attachment is still a valid SynthID test subject** (verified 2026-06-01 on issue #14's pic3: the GitHub-served JPEG survived re-encoding and openai.com/verify still detected SynthID). Do NOT dismiss issue-attachment JPEGs as "not faithful originals" when reproducing a SynthID-survival report: the recompression strips the **C2PA metadata** (so `identify` reads Unknown on the attachment) but NOT the **pixel watermark** that openai.com/verify reads. A true byte-original only matters for the metadata/C2PA path, not for the pixel-SynthID-removal test. (Contrast the open imwatermark above, which IS fragile to JPEG.) The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. A from-scratch gpt-image pilot (2026-05-24) confirmed this independently: 5 independent solid-black gpt-image outputs share a near-identical fixed signature (pairwise residual correlation **0.92**, avg-template retains 97% energy), so the watermark/carrier IS strongly present and consistent on flat content — but the carrier frequencies extracted from it do NOT discriminate real content (carrier-to-random ratio: cleaned 1.86 > watermarked 1.53; a non-gpt-image image scored highest at 3.67). The signature drowns in content texture. Net: a perfectly consistent solid-color signature still yields no real-content pixel detector with magnitude/carrier methods. A corpus discrimination test (2026-05-24, `scripts/synthid_pixel_probe.py`, raw zero-mean residual NCC) independently re-confirms this: at matched resolution, SynthID positives do NOT cluster apart from negatives (within-Gemini 0.07; at 1024 px pos-vs-neg >= pos-vs-pos). The only high correlations Line truncated
|
||||
- **External AI-vs-real classifier models are out of scope (decided 2026-05-24).** Generic HuggingFace detectors (`Organika/sdxl-detector` Swin Transformer, `umm-maybe/AI-image-detector`, and fine-tunes) exist and report ~0.98 on their *own* SDXL-vs-real validation sets, but they are per-generator and the model cards themselves note degraded accuracy off-distribution; they are untested on gpt-image / Gemini Nano Banana (the metadata-stripped surfaces we care about), and our own light SDXL pass would likely defeat them the same way it defeats SynthID. Detection here stays local + signal-based (metadata + visible sparkle); do not add a bundled classifier dependency.
|
||||
- **SynthID v2 vs default pipeline:** **CORRECTION (2026-05-30): strength 0.05 does NOT remove the CURRENT Google SynthID (Nano Banana / Gemini 3).** Re-verified via the Gemini "Verify with SynthID" oracle on a real image: at 0.05 SynthID is still detected; at **0.10 it is removed** (OpenAI's SynthID was already cleared at 0.05). So the default strength was raised 0.05 -> **0.10** (`DEFAULT_STRENGTH` in `watermark_profiles.py`; CLI `--strength` defaults to 0.10), and that higher strength is exactly why text protection (`_run_region_hires`) runs by default (text deforms more at 0.10). Caveat: n=1 Google + n=1 OpenAI image so far -- broad oracle validation across the corpus is pending (different images may need a different strength). **Resolution dependence confirmed by a user report (#14, qw1212ss, 2026-05-31):** on 1600x1600 gpt-image outputs checked via openai.com/verify, 0.05 left SynthID detected on **7/8** images, while small images (376x429) cleared at ~100% -- so "OpenAI cleared at 0.05" was a low-resolution result; a larger canvas carries a stronger watermark and needs more strength. **Policy (do NOT chase a single magic number or build resolution/vendor-adaptive defaults): 0.10 is the default because it is what clears the watermark today; if the oracle still reads SynthID, the guidance is simply to raise `--strength` (0.12, then 0.15), using the lowest value that verifies clean.** There is no local SynthID detector, so the tool cannot self-check and auto-tune; both vendors tighten the watermark over time, so any fixed value is a moving target. README "Removing SynthID" documents the strength-ladder guidance for users. The original claim below (0.05 defeats SynthID v2) held for the specific May-2026 Gemini output tested then but is stale for current Google SynthID. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". Also confirmed against **OpenAI's** SynthID (2026-05-23): a fresh ChatGPT/gpt-image output read "SynthID detected" on openai.com/verify before the local SDXL run and "SynthID not detected" after (corpus regression chain: pos `4ef377bd` -> cleaned `47188e88`). The same configuration is used in raiw-app production (`fal-ai/fast-sdxl/image-to-image`, strength 0.05, steps 50, guidance 7.5, no pre-downscale). fal's own `llms.txt` for `fast-sdxl` names the base checkpoint as `stabilityai/stable-diffusion-xl-base-1.0` (verified 2026-05-25) -- the exact checkpoint the local CLI defaults to (`DEFAULT_MODEL_ID`). So the local `invisible` default is weight-for-weight identical to prod; "fast-sdxl" is fal's optimized serving, not different weights. After the native-resolution fix the local pipeline matches prod on weights + strength + steps + guidance + resolution. SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain. **Scope of the claim: defeating the SynthID verifier is NOT the same as forensic invisibility.** "Removing the Watermark Is Not Enough: Forensic Stealth in Generative-AI Watermark Removal" (arXiv:2605.09203, 2026-05) shows that six removal attacks across four families (UnMarker, CtrlRegen+, WatermarkAttacker, etc.) all leave forensic traces: independent detectors flag *removal-processed* images vs genuinely-clean ones at **>98% TPR at 1% FPR**. So our SDXL pass makes the oracle read "SynthID not detected," but the output can still be classifiable as "an image that went through a removal pipeline." Do not over-claim "indistinguishable from a real photo." https://arxiv.org/abs/2605.09203
|
||||
- **DEFAULT STRENGTH IS NOW VENDOR-ADAPTIVE (2026-06-01, SUPERSEDES every fixed-default claim in this bullet and the next).** `resolve_strength(strength, profile, vendor)` + `vendor_for_strength(path)` (`watermark_profiles.py`) read the C2PA issuer (`metadata.synthid_source`) on the ORIGINAL input and pick `OPENAI_STRENGTH` **0.10** / `GEMINI_STRENGTH` **0.15** / `UNKNOWN_STRENGTH` **0.15** when `--strength` is unset; explicit `--strength` always wins. The CLI detects the vendor from the pristine source (before the visible pass / metadata-strip removes C2PA from the temp file) and passes it to the engine, so display and execution agree; `cmd_invisible`/`cmd_all`/`batch` + the module-level `remove_watermark` all thread `vendor`. **This replaces the single 0.30 default AND the prior "do NOT build a vendor-adaptive default" policy** -- both came from the now-debunked protect-text-contaminated study. Basis: the oracle-verified June 2026 controlled study (clean v0.8.6, protect OFF): OpenAI clears at 0.05 across 1024-1600 (n=4, resolution-independent); Google needs 0.15 on the capped-1536 path (n=4). `docs/synthid.md` §2.2 (data) + §5.2 (the adaptive default) are authoritative. CAVEAT: Google's 0.15 was validated only on `--max-resolution 1536`; native large Gemini (2816) was not locally measurable (OOM on M-series) and is pending GPU validation on raiw.cc -- if it survives 0.15 native, raise `--strength`. **Everything below in this bullet about a fixed 0.10/0.30 default is HISTORICAL; trust the vendor-adaptive constants + docs/synthid.md.**
|
||||
- **SynthID v2 vs default pipeline (HISTORICAL, see the vendor-adaptive correction above):** **CORRECTION (2026-05-31, oracle-verified GPU study, SUPERSEDES the 0.10 claim below): the current Gemini SynthID survives 0.10/0.15/0.2 and is REMOVED only at strength 0.3** (Modal A100, native res, Gemini-app "Verify with SynthID", n=3 FRESH Gemini images, `protect_text`/`faces` OFF; 0.2 still present, 0.3 removed). `DEFAULT_STRENGTH` was raised **0.10 -> 0.30** to match. The "0.10 removes it" finding below was n=1 and is now stale -- the threshold has climbed 0.05 -> 0.10 -> ~0.3 as Google hardens SynthID, so re-test against fresh Gemini periodically (moving target). 0.3 costs SSIM ~0.97 vs original (modest) but softens dense/fine typography, and is **overkill for non-SynthID sources** (OpenAI/ChatGPT carry C2PA, NOT Google SynthID -- their pixels read negative on the Gemini verifier at every strength, so 0.10 is plenty there; a per-source strength via `identify` was considered and deferred in favour of the simpler single 0.3 default). **`protect_text` is OFF by default (CORRECTED 2026-06-01, supersedes earlier A/B findings):** verified on a gpt-image at 1600x1600 (issue #14, June 2026 oracle study): same image, with `--protect-text` → SynthID detected by openai.com/verify; without → SynthID removed. The 2026-05-31 A/B finding ("protect_text does not block removal") was Gemini-SynthID-only and did not generalize to OpenAI gpt-image at 1600x1600. Mechanism: the global pass removes SynthID everywhere, but the per-region hires re-scrub regenerates those pixels from an upscaled crop -- at that effective resolution the per-region pass may be insufficient to re-destroy the payload. Both `protect_text` and `protect_faces` are now **EXPERIMENTAL, opt-in** (`--protect-text` / `--protect-faces`), OFF by default. **Oracle scope (load-bearing):** the Gemini app "Verify with SynthID" is the ONLY valid SynthID oracle (it detects Google's mark on any image); `openai.com/verify` is scoped to OpenAI provenance (its own C2PA) and is **NOT** a SynthID oracle, so a negative there is meaningless for SynthID and the older "OpenAI cleared at 0.05 on openai.com/verify" notes below are about provenance, not a pixel-SynthID measurement. **CORRECTION (2026-05-30): strength 0.05 does NOT remove the CURRENT Google SynthID (Nano Banana / Gemini 3).** Re-verified via the Gemini "Verify with SynthID" oracle on a real image: at 0.05 SynthID is still detected; at **0.10 it is removed** (OpenAI's SynthID was already cleared at 0.05). So the default strength was raised 0.05 -> **0.10** (`DEFAULT_STRENGTH` in `watermark_profiles.py`; CLI `--strength` defaults to 0.10), and that higher strength is exactly why text protection (`_run_region_hires`) runs by default (text deforms more at 0.10). Caveat: n=1 Google + n=1 OpenAI image so far -- broad oracle validation across the corpus is pending (different images may need a different strength). **Resolution dependence confirmed by a user report (#14, qw1212ss, 2026-05-31):** on 1600x1600 gpt-image outputs checked via openai.com/verify, 0.05 left SynthID detected on **7/8** images, while small images (376x429) cleared at ~100% -- so "OpenAI cleared at 0.05" was a low-resolution result; a larger canvas carries a stronger watermark and needs more strength. **CORRECTION (2026-06-01, oracle-verified controlled study, SUPERSEDES the resolution-dependence claim above): the "1600 needs more strength" result was the protect-text bug, NOT resolution.** qw1212ss ran v0.7.2 with `protect_text` ON by default; on dense-text infographics the hires text re-scrub re-introduced SynthID. Re-running the SAME 1600x1600 gpt-image on clean v0.8.6 (protect OFF) removes SynthID at **0.05**. Controlled June 2026 study (clean v0.8.6, protect OFF, oracle-verified per image; test set in `data/synthid_corpus/`): **OpenAI clears at 0.05 across 1024-1600 (n=4), resolution-independent; Google (Gemini) needs 0.15 on the capped-1536 path (n=4), 0.05/0.10 do NOT clear.** The dominant real factor is **vendor** (~3x: Google harder), not resolution. Native large Gemini (2816) was not locally measured (OOM/CPU on M-series) -- it plausibly needs >= 0.30. **`docs/synthid.md` section 2.2 is the authoritative source for these numbers; the year-over-year "0.05 -> 0.10 -> 0.30" progression in this bullet conflates real Google hardening with the now-debunked protect-text artifact.** **Policy (do NOT chase a single magic number or build resolution/vendor-adaptive defaults): 0.10 is the default because it is what clears the watermark today; if the oracle still reads SynthID, the guidance is simply to raise `--strength` (0.12, then 0.15), using the lowest value that verifies clean.** There is no local SynthID detector, so the tool cannot self-check and auto-tune; both vendors tighten the watermark over time, so any fixed value is a moving target. README "Removing SynthID" documents the strength-ladder guidance for users. The original claim below (0.0Line truncated
|
||||
- **CtrlRegen profile uses a clean-noise default strength, NOT the SDXL 0.10 (fixed 2026-05-31).** **CORRECTION (2026-05-31, same oracle-verified GPU study): ctrlregen at its clean-noise strength DESTROYS real images** -- smooth/background regions fill with hallucinated micro-text garbage; the pipeline is binary (low strength = no-op, high = destroy, no usable middle) and heavy (~8.5 min / ~$0.30 vs ~25 s / ~$0.02 for SDXL). So the literature's "clean-noise is the lever" (detailed below) did NOT survive empirical testing on real content. ctrlregen is now flagged **EXPERIMENTAL** (CLI `--pipeline` help, README, and the `watermark_profiles` comment) and is **NOT for production** -- SDXL img2img at ~0.3 is the shippable path. The clean-noise-default plumbing below is kept (so the profile at least does real work if anyone opts in), but do not recommend ctrlregen. `--pipeline ctrlregen` no longer inherits the SDXL img2img `--strength` default. `resolve_strength(strength, profile, vendor)` (`watermark_profiles.py`, pure + unit-tested in `test_platform.py::TestResolveStrength`) resolves an unset `--strength` to `CTRLREGEN_DEFAULT_STRENGTH` (**1.0**) for ctrlregen and to the **vendor-adaptive** SDXL default (`OPENAI_STRENGTH` 0.10 / `GEMINI_STRENGTH` 0.15 / `UNKNOWN_STRENGTH` 0.15, see the vendor-adaptive bullet above); an explicit `--strength` always wins (including `0.0` -- the resolver checks `is None`, not falsiness, so it does not repeat the old `strength or DEFAULT` bug). CLI `--strength` for `invisible`/`all` defaults to **None** (batch already did); the display (`cli.py`) and the engine (`watermark_remover.remove_watermark`) both route through `resolve_strength` with the SAME detected `vendor` so they never disagree. **Why (deep-research pass 2026-05-31, primary sources):** CtrlRegen's removal power comes from regenerating from (near) clean Gaussian noise, not the light partial-noise img2img the SDXL pass uses. CtrlRegen (ICLR 2025, arXiv:2410.05470) diagnoses verbatim that prior partial-noise regeneration "struggles with high-perturbation watermarks" because a small noise step "retains" watermark info that diffuses back into the output; the fix is a clean-noise start, which with `StableDiffusionControlNetImg2ImgPipeline` maps to strength ~1.0 (image structure held by the canny ControlNet + DINOv2 IP-Adapter, not by the watermarked latent). **Before the fix `--pipeline ctrlregen` ran at 0.10** -- a near-identity pass that loaded ControlNet + DINOv2-giant and then barely changed the image (a removal no-op). **NOT yet oracle-verified** that clean-noise ctrlregen clears the stubborn high-texture gpt-image class that 0.20 SDXL img2img could not (issue #14, qw1212ss: pic3/6/7 survived SynthID through 0.05->0.20); that is the pending controlled test (via openai.com/verify with the IP-country rate-limit bypass). **Forensic-stealth caveat applies harder here:** regeneration-family removal is the MOST detectable as "an image that went through a removal pipeline" (CtrlRegen+ 99.97% TPR@1%FPR, arXiv:2605.09203). **Two #14-investigation hypotheses the literature did NOT confirm:** (1) our "VAE round-trip drives removal, denoising strength does not" framing is only PARTIALLY supported -- arXiv:2510.09263 confirms SynthID was hardened against *weak* VAE re-generation (explaining survival) but does not name the VAE round-trip as the removal vector; (2) our "survival correlates with high-frequency CONTENT texture (Laplacian 466 vs 236)" is unconfirmed by any primary source -- the literature establishes *watermark-perturbation-strength* dependence (a different axis), so the texture correlation stays our own unverified observation, not a literature-backed fact.
|
||||
@@ -17,7 +17,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
|
||||
## Features
|
||||
|
||||
- **Visible watermark removal** — a registry of known marks in their usual places: the Gemini / Nano Banana sparkle, the Doubao "豆包AI生成" text strip, and the Jimeng "★ 即梦AI" wordmark. Each is removed by **reverse-alpha blending** against a captured alpha map (`original = (wm − α·logo)/(1−α)`), recovering the true pixels rather than inpainting a guess. The Gemini sparkle recovers cleanly on its own; the Doubao and Jimeng text marks re-rasterize slightly per image, so a thin residual inpaint over the glyph footprint clears the leftover edges (the alpha maps are reproducibly rebuilt from controlled captures by `scripts/visible_alpha_solve.py`). Fast, offline, no GPU. `visible --mark auto` finds and removes the strongest detected mark. (For arbitrary logos/objects, see `erase`.)
|
||||
- **Visible watermark removal** — a registry of known marks in their usual places: the Gemini / Nano Banana sparkle, the Doubao "豆包AI生成" text strip, and the Jimeng "★ 即梦AI" wordmark. Each is removed by **reverse-alpha blending** against a captured alpha map (`original = (wm − α·logo)/(1−α)`), recovering the true pixels rather than inpainting a guess. The Gemini sparkle recovers cleanly on its own on bright backgrounds (on a dark background, where the fixed alpha would over-subtract and leave a dark spot, it automatically inpaints the small sparkle footprint instead); the Doubao and Jimeng text marks re-rasterize slightly per image, so a thin residual inpaint over the glyph footprint clears the leftover edges (the alpha maps are reproducibly rebuilt from controlled captures by `scripts/visible_alpha_solve.py`). Fast, offline, no GPU. `visible --mark auto` finds and removes the strongest detected mark. (For arbitrary logos/objects, see `erase`.)
|
||||
- **Universal region eraser (`erase`)** — remove any logo / watermark / object inside boxes you specify, regardless of position or colour. Default cv2 inpainting (CPU, instant); optional big-LaMa via onnxruntime (`lama` extra) for higher quality
|
||||
- **Invisible watermark removal** — SynthID, StableSignature, TreeRing via diffusion-based regeneration (needs a local GPU, or run it with no setup on [raiw.cc](https://raiw.cc))
|
||||
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests (PNG / JPEG / AVIF / HEIF / JPEG-XL, **MP4 / MOV / M4V / M4A** at the container level, and **WebM / MP3 / WAV / FLAC / OGG** losslessly via ffmpeg), XMP DigitalSourceType
|
||||
@@ -26,7 +26,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
|
||||
- **Batch processing** — process entire directories
|
||||
- **Detection** — three-stage NCC watermark detection with confidence scoring
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" text marks), the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -58,7 +58,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
|
||||
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo) and by ByteDance's Doubao ("豆包AI生成" corner text) and Jimeng / Dreamina ("★ 即梦AI" wordmark). All are removed on CPU by reverse-alpha against a captured alpha map (Jimeng adds a residual inpaint over the glyph footprint, since its mark re-rasterizes per image). Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any colour), use the universal `erase --region` command.
|
||||
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" text marks), and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -108,23 +108,30 @@ The removal pipeline (default profile, SDXL):
|
||||
```text
|
||||
image → encode to latent space (VAE) at native resolution
|
||||
→ add controlled noise (forward diffusion)
|
||||
→ denoise (reverse diffusion, ~50 steps at strength 0.10)
|
||||
→ denoise (reverse diffusion, ~50 steps; strength is vendor-adaptive:
|
||||
0.10 OpenAI / 0.15 Google / 0.15 unknown, override with --strength)
|
||||
→ decode back to pixels (VAE)
|
||||
```
|
||||
|
||||
- Native resolution avoids shrinking the input to 1024 px first; that down-then-up round-trip was the main quality loss (issue #10). Use `--max-resolution N` only to cap GPU/MPS memory on very large inputs.
|
||||
|
||||
> **If SynthID still verifies after the run, raise `--strength`.** The default `0.10` is the value that clears the watermark today, but SynthID is a moving target: both Google and OpenAI tighten it over time, and a larger image carries a stronger watermark (a 1600x1600 image needs more than a 400x400 one). There is no single permanent number, and there is no local SynthID detector, so the tool cannot self-check and auto-tune. The rule is simple: if the verifier still reads SynthID, step the strength up — try `--strength 0.12`, then `0.15`. Higher strength changes more detail and text, so use the lowest value that comes back clean on the oracle ([openai.com/research/verify](https://openai.com/research/verify/) or the Gemini app's "Verify with SynthID").
|
||||
> **Default strength is vendor-adaptive (no flag needed).** The tool reads the C2PA issuer to detect which vendor's SynthID is present and picks the strength that clears it with the least quality loss: **OpenAI gpt-image → `0.10`**, **Google Gemini → `0.15`**, **unknown source → `0.15`**. An oracle-verified June 2026 study (clean pipeline, per-image openai.com/verify or Gemini app) found OpenAI's watermark clears at `0.05` across `1024`-`1600` px (resolution-independent) while Google's is ~3x more robust and needs `0.15`. The dominant factor is the vendor, not resolution. There is no local SynthID detector, so if the oracle still reads SynthID, raise `--strength`; if you care more about preserving fine text, lower it. (Caveat: Google's `0.15` was validated on the capped `--max-resolution 1536` path; a very large native Gemini image may need more.)
|
||||
>
|
||||
> **Text and face protection are OFF by default.** The high-resolution text re-scrub can shield SynthID in text regions, leaving the watermark intact there even after the global pass clears it everywhere else (verified June 2026: same image, with `--protect-text` → SynthID detected; without → SynthID removed). Both features are opt-in with `--protect-text` / `--protect-faces` and considered **experimental**. If you enable them, verify the result with the oracle.
|
||||
>
|
||||
> **`--pipeline ctrlregen` is experimental and not recommended.** On paper CtrlRegen ([ICLR 2025](https://github.com/yepengliu/CtrlRegen)) regenerates from near-clean Gaussian noise to defeat robust watermarks, but in testing on real images it **destroys content** — smooth and background regions fill with hallucinated micro-text — and it is heavy (several GB of extra models, minutes per image). It has no usable middle setting (too low removes nothing, high enough to remove wrecks the image), so the shippable path is the default SDXL pipeline at the vendor-adaptive strength. CtrlRegen stays available for experimentation only.
|
||||
|
||||
SDXL is the default since May 2026: empirically defeats SynthID v2 on Gemini 3 Pro outputs, where the older SD-1.5 pipeline at 768 px did not. The SD-1.5 path was removed once it was verified not to handle v2. Note the scope: this defeats the SynthID *verifier*, which is not the same as being forensically indistinguishable from a real photo. Recent work ([arXiv:2605.09203](https://arxiv.org/abs/2605.09203)) shows watermark-removal pipelines leave detectable traces, so a separate "this image was processed" classifier can still flag the output.
|
||||
|
||||
> **Oracle vs `identify` can disagree, and that is expected.** An online verifier reads the actual SynthID *pixel* watermark and detects only its own vendor's content — [openai.com/research/verify](https://openai.com/research/verify/) states "OpenAI generation signals will only be detected if the image was generated with our tools". Our `identify` cannot decode the pixel watermark (no vendor ships a local decoder), so it infers SynthID from the **C2PA metadata** instead. So after the SDXL pass the oracle can read "no SynthID" (pixel watermark gone) while `identify` still reports SynthID from a surviving C2PA manifest. They measure different signals. Run `metadata --remove` (or `all`) to also strip the manifest; note that a quiet metadata proxy is not proof the pixel watermark itself is gone.
|
||||
|
||||
**Face Protection**: before diffusion, YOLO detects people in the image and extracts them. After diffusion, the original faces are blended back with a soft elliptical mask to prevent AI distortion of facial features.
|
||||
> **Technical deep-dive:** see [`docs/synthid.md`](docs/synthid.md) for a primary-source-cited breakdown of how SynthID works mechanically (post-hoc encoder/decoder, 136-bit payload, pixel-space embedding), what it empirically survives (JPEG, crop, resize: ~99.98% TPR at 0.1% FPR from arXiv:2510.09263), what removes it, and the forensic-stealth tradeoff (all known removal attacks are detectable at >98% TPR@1%FPR per arXiv:2605.09203).
|
||||
|
||||
**Face Protection** (experimental, opt-in `--protect-faces`): before diffusion, YOLO detects people in the image and extracts them; after diffusion the original faces are blended back. Off by default — enable only when face fidelity matters more than SynthID removal completeness.
|
||||
|
||||
**Analog Humanizer**: optional film grain and chromatic aberration injection that mimics a photo of a screen, raising the bar for AI-generated image classifiers. (It frustrates generic classifiers but does not guarantee forensic invisibility — see the [arXiv:2605.09203](https://arxiv.org/abs/2605.09203) note above.)
|
||||
|
||||
**Text Protection** (automatic): SDXL img2img regenerates every pixel, so small text and glyphs get deformed at the strengths that defeat SynthID. The SDXL pipeline guards against this by default: a PP-OCRv3 text detector (a 2.4 MB ONNX model run on CPU via OpenCV's DNN module, downloaded and cached on first use) locates text regions, and if any are found the whole image is scrubbed normally and then each text block is **re-scrubbed at high resolution** and composited back. Because the text is regenerated (not copied), the watermark is still removed inside it, while running the text region at higher resolution keeps small strokes crisp through the regeneration. Detection is **language-agnostic** (it finds text regions, not characters), so Latin, Cyrillic, CJK, Hangul, Arabic, and digits are all protected, and it runs at the image's native resolution so small text on large images is not missed. Text-free images run the standard pass at no extra cost. Pass `--no-protect-text` to turn it off. SDXL default pipeline only.
|
||||
**Text Protection** (experimental, opt-in `--protect-text`): re-scrubs detected text blocks at high resolution after the global pass to keep small glyphs crisp. **Off by default** because the high-resolution re-scrub can preserve SynthID in text regions even after the global pass removes it elsewhere. Enable only when text fidelity matters more than watermark removal completeness, and verify the oracle result. SDXL pipeline only.
|
||||
|
||||
### Stripping C2PA, EXIF, and "Made with AI" metadata
|
||||
|
||||
@@ -266,7 +273,9 @@ remove-ai-watermarks erase image.png --region 1640,1930,400,100 -o clean.png
|
||||
remove-ai-watermarks invisible image.png -o clean.png --humanize 4.0
|
||||
# Runs at native resolution by default. On a very large image that OOMs the
|
||||
# GPU/MPS, cap the long side: --max-resolution 2048
|
||||
# Text / CJK glyphs are preserved automatically; disable with --no-protect-text
|
||||
# Strength is vendor-adaptive by default (OpenAI 0.10 / Google 0.15); override
|
||||
# with --strength. Text/face protection is opt-in (--protect-text /
|
||||
# --protect-faces, experimental: they can shield SynthID).
|
||||
|
||||
# Check / strip AI metadata (C2PA, EXIF, "Made with AI" labels)
|
||||
# --check also flags SynthID-bearing sources: a C2PA manifest signed by
|
||||
|
||||
+460
@@ -0,0 +1,460 @@
|
||||
# SynthID-Image: technical reference
|
||||
|
||||
This document covers how Google SynthID for images works mechanically, what it
|
||||
survives, what removes it, and the current deployment landscape. It is written
|
||||
for engineers working on watermark detection and removal -- specifically to
|
||||
inform decisions about strength settings, test methodology, and what oracle
|
||||
results mean.
|
||||
|
||||
Primary sources are cited inline. Marketing-only claims are flagged separately
|
||||
from independently-verified results.
|
||||
|
||||
---
|
||||
|
||||
## 1. Mechanism
|
||||
|
||||
### 1.1 Post-hoc, model-independent design
|
||||
|
||||
SynthID-Image is **not** baked into a diffusion model's weights. It is a
|
||||
post-hoc, model-independent system: a separate encoder `f` is applied to an
|
||||
already-generated image, and a separate decoder `g` reads it back.
|
||||
|
||||
> "We deliberately designed SynthID-Image as a post-hoc, model-independent
|
||||
> approach, a choice largely based on deployment considerations."
|
||||
> -- Gowal et al., arXiv:2510.09263
|
||||
|
||||
The formal definition from the paper:
|
||||
|
||||
> "A post-hoc watermarking scheme is a pair f, g consisting of an encoder
|
||||
> function f: X -> X, which adds an identification mark, and a decoder
|
||||
> function g: X -> {+-1}, which tries to detect if the mark is present."
|
||||
|
||||
This is the key architectural fact: **the generative model (Imagen, Gemini's
|
||||
image model) is not modified**. The watermark is stamped onto the pixel output
|
||||
after generation, by a separate neural network. This means:
|
||||
|
||||
- The watermark is in **pixel space**, not in the model's latent activations.
|
||||
- Replacing the generative model does not remove the watermarking capability.
|
||||
- The encoder/decoder pair can be updated independently of the generative model.
|
||||
|
||||
The paper does not disclose the internal architecture of the encoder/decoder
|
||||
networks (layer types, capacity). The external variant SynthID-O is available
|
||||
to partners; the production internal variant is not published.
|
||||
|
||||
### 1.2 How it differs from classical DWT-DCT watermarks
|
||||
|
||||
The open watermarks used by Stable Diffusion / SDXL / FLUX (via the
|
||||
`imwatermark` library) use classical **DWT-DCT** frequency-domain embedding: a
|
||||
fixed bit pattern is added to specific frequency coefficients of the image's
|
||||
wavelet transform. This is fast, key-free, and locally detectable with a public
|
||||
decoder.
|
||||
|
||||
SynthID-Image uses **jointly-trained deep learning models**:
|
||||
|
||||
> "SynthID uses two deep learning models -- for watermarking and identifying --
|
||||
> that have been trained together on a diverse set of images. The combined model
|
||||
> is optimised on a range of objectives, including correctly identifying
|
||||
> watermarked content and improving imperceptibility by visually aligning the
|
||||
> watermark to the original content."
|
||||
> -- Google DeepMind blog, 2023
|
||||
|
||||
The practical difference for robustness: the deep learning encoder learns to
|
||||
spread the signal across the image in a way that is optimized to survive a
|
||||
specific perturbation distribution seen during training. Classical DWT-DCT
|
||||
embeds in fixed, predictable frequency bins, making it brittle to any
|
||||
operation that hits those bins (e.g., JPEG re-quantization wipes it cleanly at
|
||||
quality <= 90).
|
||||
|
||||
### 1.3 Payload capacity
|
||||
|
||||
SynthID-O (the external/partnership variant) encodes:
|
||||
|
||||
- **136 bits** within a **512x512 pixel image**
|
||||
|
||||
For comparison (from the same paper):
|
||||
|
||||
| Method | Bits | Resolution |
|
||||
|-------------|------|------------|
|
||||
| SynthID-O | 136 | 512x512 |
|
||||
| StegaStamp | 100 | 400x400 |
|
||||
| TrustMark | 100 | 256x256 |
|
||||
| WAM | 32 | 256x256 |
|
||||
|
||||
The payload carries an identification mark (not a user-readable secret). The
|
||||
paper separates watermark **detection** (is this watermarked?) from payload
|
||||
**recovery** (what does the payload say?): the detection path is what oracles
|
||||
like the Gemini app's "Verify with SynthID" exercise.
|
||||
|
||||
### 1.4 Where in the pipeline it lives
|
||||
|
||||
```
|
||||
[Diffusion model]
|
||||
|
|
||||
raw pixel output
|
||||
|
|
||||
[SynthID encoder f] <-- separate neural net, stamps the watermark
|
||||
|
|
||||
watermarked image
|
||||
|
|
||||
[served / downloaded]
|
||||
|
|
||||
[SynthID decoder g] <-- separate neural net, run by Google's verifier only
|
||||
|
|
||||
present / not present
|
||||
```
|
||||
|
||||
The VAE decoder of the diffusion model is **not** involved in watermarking.
|
||||
Some in-generation watermark approaches (like the research method "Tree Ring")
|
||||
inject the signal into the initial noise latent so it propagates through the
|
||||
diffusion process and appears in the final image; SynthID-Image does not do
|
||||
this -- it is applied after the VAE has already decoded latents to pixels.
|
||||
|
||||
---
|
||||
|
||||
## 2. Robustness
|
||||
|
||||
### 2.1 What the paper claims it survives (primary-source verified)
|
||||
|
||||
The SynthID-Image paper (arXiv:2510.09263) evaluates SynthID-O against **30
|
||||
image transformations** grouped into 6 categories:
|
||||
|
||||
| Category | Examples |
|
||||
|-------------|-----------------------------------------------|
|
||||
| Color | brightness, contrast, saturation, hue shifts |
|
||||
| Combination | combinations of multiple transforms |
|
||||
| Noise | Gaussian noise, impulse noise, median filter |
|
||||
| Overlay | text overlays, logos, stickers |
|
||||
| Quality | JPEG compression, WebP, format conversion |
|
||||
| Spatial | crop, resize, rotate, flip, padding |
|
||||
|
||||
**TPR at 0.1% FPR -- SynthID-O vs. baselines (resized to 512x512):**
|
||||
|
||||
| Category | SynthID-O | Best baseline (WAM) | Worst baseline (StegaStamp spatial) |
|
||||
|------------------|-----------|---------------------|--------------------------------------|
|
||||
| Identity (none) | 100.00% | 100.00% | 100.00% |
|
||||
| Aggregated | 99.98% | 90.62% | ~70% |
|
||||
| Color | 100.00% | 81.29% | ~75% |
|
||||
| Combination | 99.96% | 96.08% | ~22% |
|
||||
| Noise | 99.98% | 100.00% | ~92% |
|
||||
| Overlay | 100.00% | 100.00% | 100.00% |
|
||||
| Quality | 99.99% | -- | ~89% |
|
||||
| Spatial (worst) | 99.97% | 76.04% | 15.25% |
|
||||
|
||||
The "Spatial worst" row is the hardest case (aggressive crop + resize).
|
||||
SynthID-O retains 99.97% TPR; StegaStamp collapses to 15.25%. This is where
|
||||
the deep-learning approach gains the most over classical methods.
|
||||
|
||||
Google's marketing page states the watermark is:
|
||||
|
||||
> "designed to stand up to modifications like cropping, adding filters, changing
|
||||
> frame rates, or lossy compression."
|
||||
> -- deepmind.google/models/synthid/
|
||||
|
||||
The marketing claim is broadly consistent with the paper's numbers for these
|
||||
specific categories.
|
||||
|
||||
**JPEG and format conversion specifically** fall under the "Quality" category,
|
||||
where SynthID-O achieves 99.99% TPR. This is the empirical basis for the fact
|
||||
that **GitHub-recompressed JPEGs from issue attachments are valid SynthID test
|
||||
subjects**: the re-encoding does not remove the pixel watermark.
|
||||
|
||||
### 2.2 Stated limits (vendor claim, not independently verified)
|
||||
|
||||
> "SynthID isn't foolproof against extreme image manipulations."
|
||||
> -- Google DeepMind blog, 2023
|
||||
|
||||
This is the only public failure-mode statement Google has made. No specific
|
||||
perturbation type, threshold, or quantitative boundary is named. The
|
||||
Limitations section of the paper (Section 10) was not recoverable from the
|
||||
public HTML version of arXiv:2510.09263v1 due to a rendering failure in the
|
||||
conversion (the body text of Section 10 is absent from the HTML).
|
||||
|
||||
**What is known empirically from our own oracle-verified testing.**
|
||||
|
||||
A controlled study (June 2026, clean v0.8.6 with text/face protection OFF,
|
||||
native resolution on this repo's default SDXL pipeline) measured the minimum
|
||||
img2img strength that removes the SynthID pixel watermark, verified per image on
|
||||
the vendor's own oracle (openai.com/verify for OpenAI, the Gemini app "Verify
|
||||
with SynthID" for Google). The test set and per-image results are recorded in
|
||||
`data/synthid_corpus/` (manifest `verified_via` = `openai-verify` / `gemini-app`).
|
||||
|
||||
| Vendor | Images | Resolution(s) | Pipeline | Removed at |
|
||||
|--------|--------|---------------|----------|------------|
|
||||
| OpenAI (gpt-image) | n=4 | 1024x1536 .. 1600x1600 | native | **0.05** |
|
||||
| Google (Gemini) | n=4 | 2816x1536 -> capped 1536 | `--max-resolution 1536` | **0.15** (0.05 and 0.10 do NOT clear) |
|
||||
|
||||
**Two findings, both oracle-verified:**
|
||||
|
||||
1. **Vendor is the dominant factor, not resolution.** Google's SynthID is
|
||||
roughly 3x more robust than OpenAI's: at a comparable (small) working
|
||||
resolution, OpenAI clears at 0.05 while Google needs 0.15. This matches
|
||||
Google having hardened SynthID more aggressively over time.
|
||||
|
||||
2. **OpenAI SynthID removal is resolution-independent in the tested range.**
|
||||
All four OpenAI images (including a 1600x1600) cleared at 0.05.
|
||||
|
||||
**CORRECTION (supersedes the earlier "resolution dependence" claim).** A prior
|
||||
version of this doc and CLAUDE.md stated that strength 0.30 failed to remove
|
||||
SynthID on 1600x1600 gpt-image and that removal was resolution-dependent. That
|
||||
was an **artifact of the text-protection bug** (issue #14): those tests ran a
|
||||
build where `protect_text` was ON by default, and the high-resolution text
|
||||
re-scrub re-introduced SynthID in the dense-text regions of the infographic
|
||||
images tested. Re-running the *same* 1600x1600 image on clean v0.8.6 (protect
|
||||
OFF) removes SynthID at **0.05**. The "large images resist removal" conclusion
|
||||
was false; the resistance was the protect-text shielding, now fixed (v0.8.5).
|
||||
|
||||
**Open / not locally testable:**
|
||||
|
||||
- **Native large Gemini (2816x1536, ~4.3 MP).** The Gemini floor of 0.15 was
|
||||
measured on the *capped* (`--max-resolution 1536`) path, which is the
|
||||
practical local route on Apple-Silicon (native 2816 OOMs / falls back to slow
|
||||
CPU on a 32 GB M-series). Native large Gemini was not measured here; the
|
||||
vendor and resolution effects would stack, so it plausibly needs >= 0.30 or a
|
||||
discrete GPU. Confirm on a CUDA box if needed.
|
||||
- **Heavy JPEG compression** (quality < ~50-60): not oracle-tested; the DL
|
||||
approach is more robust than DWT-DCT but Google acknowledges limits at
|
||||
"extreme" manipulation.
|
||||
|
||||
### 2.3 Removal attacks and forensic detectability
|
||||
|
||||
The paper arXiv:2605.09203 ("Removing the Watermark Is Not Enough",
|
||||
Goonatilake & Ateniese, 2026) evaluates 6 removal attacks against a ResNet-50
|
||||
forensic detector. All attacks defeat the watermark verifier but are detected
|
||||
by the forensic classifier:
|
||||
|
||||
| Attack | Family | AUROC | TPR @ 1% FPR | TPR @ 0.1% FPR |
|
||||
|-----------------|------------------|--------|--------------|----------------|
|
||||
| UnMarker | Distortion | 0.9994 | 99.81% | 98.28% |
|
||||
| WatermarkAttacker| Regeneration | 0.9997 | 99.95% | 99.38% |
|
||||
| CtrlRegen+ | Regeneration | 0.9999 | 99.97% | 99.64% |
|
||||
| NFPA | Inversion/Pert. | 0.9984 | 99.24% | 62.10% |
|
||||
| Boundary Leak. | Inversion/Pert. | 0.9991 | 99.24% | 88.34% |
|
||||
| WiTS | Erosion | 0.9999 | 99.80% | 99.55% |
|
||||
|
||||
The forensic detector is a standard ResNet-50 fine-tuned end-to-end; no exotic
|
||||
architecture needed. The key finding:
|
||||
|
||||
> "These removers do not return images to a clean forensic state. They often
|
||||
> trade an explicit watermark for an implicit watermark: a detectable artifact
|
||||
> introduced by the removal process itself."
|
||||
|
||||
This means: even when our SDXL img2img pass defeats the SynthID pixel
|
||||
watermark (oracle reads negative), the output may still be classifiable as
|
||||
"an image that went through a removal pipeline" by an independent detector --
|
||||
even if that detector is not trained on SynthID specifically. **Defeating the
|
||||
verifier does not restore forensic deniability.**
|
||||
|
||||
CtrlRegen+ is the most detectable removal method (AUROC 0.9999), which is
|
||||
notable because it is also the most powerful removal attack. The paper notes
|
||||
that diffusion regeneration "leaves a strong reconstruction signature from the
|
||||
diffusion prior."
|
||||
|
||||
---
|
||||
|
||||
## 3. Detectability and verifier access
|
||||
|
||||
### 3.1 No public local detector
|
||||
|
||||
The SynthID decoder is proprietary and not released:
|
||||
|
||||
> "SynthID-Image has been used to watermark over ten billion images and video
|
||||
> frames across Google's services and its corresponding verification service is
|
||||
> available to trusted testers."
|
||||
> -- Gowal et al., arXiv:2510.09263
|
||||
|
||||
There is no public API, no released decoder weights, and no reproducible
|
||||
algorithm for local detection. The verification service (SynthID Detector) is:
|
||||
|
||||
> "a verification portal" in early testing with "journalists and media
|
||||
> professionals" on a waitlist
|
||||
> -- deepmind.google/models/synthid/
|
||||
|
||||
The external variant SynthID-O is available "through partnerships" only. Our
|
||||
tool cannot locally detect SynthID presence or absence -- this is by design,
|
||||
not a gap we can fill.
|
||||
|
||||
### 3.2 How our tool detects SynthID (metadata proxy)
|
||||
|
||||
We detect SynthID indirectly: if the image's C2PA manifest is signed by a
|
||||
known SynthID-using issuer (Google, OpenAI), we infer SynthID is present. This
|
||||
is a **metadata proxy**, not a pixel watermark decode. It works while the C2PA
|
||||
manifest is intact, and is silent once the manifest is stripped or the image
|
||||
is re-encoded without C2PA (e.g., a screenshot, a social-media re-upload, or
|
||||
after `metadata --remove`).
|
||||
|
||||
This is why:
|
||||
- `identify` on a GitHub-recompressed issue attachment returns Unknown (C2PA is
|
||||
gone) even though the pixel SynthID is still present and detectable by
|
||||
openai.com/verify.
|
||||
- A quiet `identify` output is not proof that SynthID was removed -- it only
|
||||
means the metadata signal is gone.
|
||||
|
||||
### 3.3 Oracle scope: each vendor detects only their own
|
||||
|
||||
From openai.com/research/verify (verbatim, verified 2026-05-31):
|
||||
|
||||
> "OpenAI generation signals will only be detected if the image was generated
|
||||
> with our tools."
|
||||
> "Content could also still be AI-generated by another company's model, which
|
||||
> the tool currently does not detect."
|
||||
|
||||
SynthID technology is used by multiple vendors, but each verifier is keyed to
|
||||
its own payload:
|
||||
|
||||
| Oracle | Detects | Does NOT detect |
|
||||
|-------------------------------|------------------|-------------------------|
|
||||
| Gemini app "Verify with SynthID" | Google SynthID | OpenAI SynthID |
|
||||
| openai.com/research/verify | OpenAI SynthID | Google SynthID |
|
||||
|
||||
A Google-SynthID image reads clean on openai.com/verify. An OpenAI image reads
|
||||
clean in the Gemini oracle. They are different payloads within the same
|
||||
framework.
|
||||
|
||||
---
|
||||
|
||||
## 4. Adoption and current state (as of June 2026)
|
||||
|
||||
### 4.1 Google products
|
||||
|
||||
Google has watermarked **over 10 billion** images and video frames. The
|
||||
deployment split by surface matters for our tool:
|
||||
|
||||
| Surface | SynthID pixel | C2PA metadata | Visible sparkle |
|
||||
|--------------------------------------|---------------|---------------|-----------------|
|
||||
| Gemini app (generated images) | YES | YES (Google) | YES |
|
||||
| Gemini API / AI Studio / Nano Banana | YES | NO | YES |
|
||||
|
||||
The Gemini API surface is a key blind spot: it embeds the pixel watermark and
|
||||
the visible sparkle but **no C2PA or IPTC at all**. Our `identify` returns
|
||||
Unknown on API-generated images unless the visible sparkle is detected (via
|
||||
`check_visible=True`) or the user runs the Gemini app oracle.
|
||||
|
||||
### 4.2 OpenAI
|
||||
|
||||
OpenAI confirmed SynthID adoption (Help Center, updated 2026-05-21):
|
||||
|
||||
> "ChatGPT images include both C2PA metadata and SynthID watermarks."
|
||||
|
||||
This is time-gated: pre-rollout ChatGPT/gpt-image images carry C2PA without
|
||||
SynthID. Our C2PA proxy therefore over-reports SynthID presence on old images
|
||||
(hence the `_OPENAI_CAVEAT` hedging flag in the codebase).
|
||||
|
||||
### 4.3 Other vendors
|
||||
|
||||
- **Kakao** (South Korea): SynthID adopter as of May 2026 (Google announcement)
|
||||
- **NVIDIA Cosmos**: SynthID for video (not still images; different pipeline)
|
||||
- **Meta AI**: does NOT use SynthID; uses IPTC `digitalSourceType` marker instead
|
||||
|
||||
### 4.4 Version evolution (v1 vs v2 hardening)
|
||||
|
||||
Google has not publicly documented version numbers for the SynthID image
|
||||
watermark in a way that maps to our testing observations. What is known
|
||||
empirically from oracle tests:
|
||||
|
||||
- **Before May 2026 (Gemini)**: strength 0.05 removed the watermark
|
||||
- **May 2026 (Gemini)**: strength 0.05 insufficient; 0.10 required
|
||||
- **Current (Gemini, June 2026)**: on the capped 1536 path, 0.05 and 0.10 do
|
||||
NOT clear; 0.15 clears (n=4, Gemini app oracle). See section 2.2.
|
||||
- **OpenAI (June 2026)**: clears at 0.05 across 1024-1600 (n=4, clean v0.8.6).
|
||||
The earlier "0.30 still detected on 1600x1600" report (issue #14) was the
|
||||
text-protection bug, not a hardening of the watermark -- see the correction in
|
||||
section 2.2.
|
||||
|
||||
Google has hardened SynthID relative to OpenAI's (vendor gap measured at ~3x
|
||||
strength), but the year-over-year "0.05 -> 0.10 -> 0.30" progression above
|
||||
conflates a real hardening trend with the now-debunked protect-text artifact;
|
||||
treat only the section 2.2 controlled numbers as authoritative.
|
||||
|
||||
---
|
||||
|
||||
## 5. Practical implications for this tool
|
||||
|
||||
### 5.1 Text and face protection: OFF by default
|
||||
|
||||
**Text protection (`--protect-text`) can preserve SynthID in text regions.**
|
||||
Verified June 2026 on gpt-image at 1600x1600: same image processed twice --
|
||||
with `--protect-text` the oracle detected SynthID; without it, SynthID was
|
||||
removed. The mechanism: the global img2img pass clears SynthID everywhere, but
|
||||
the text-protection high-resolution re-scrub regenerates those pixels from an
|
||||
upscaled crop. At the effective resolution of the upscaled crop, the per-region
|
||||
pass may be insufficient to re-destroy the payload, reconstituting SynthID in
|
||||
text regions.
|
||||
|
||||
**Face protection (`--protect-faces`) has an even more direct preservation
|
||||
mechanism.** The pipeline extracts face regions from the ORIGINAL (watermarked)
|
||||
image BEFORE the diffusion pass, runs the global pass (which removes SynthID
|
||||
everywhere), then blends the original face pixels BACK onto the result
|
||||
(`invisible_engine.py`: `original_faces = protector.extract_faces(cv_img)`
|
||||
before `remove_watermark`, then `protector.restore_faces(out_cv, original_faces)`
|
||||
after). Those restored pixels are the original watermarked pixels -- SynthID is
|
||||
guaranteed to survive in face regions, not just possibly. The text-protection
|
||||
case is at least re-generating (uncertain); face protection is literally
|
||||
restoring the original SynthID-bearing pixels.
|
||||
|
||||
Both `--protect-text` and `--protect-faces` are therefore **experimental and
|
||||
OFF by default**. Enable only when text/face fidelity matters more than
|
||||
watermark removal completeness, and always verify the result with the oracle.
|
||||
|
||||
### 5.2 Strength setting
|
||||
|
||||
There is no single permanent correct strength, but the controlled June 2026
|
||||
study (section 2.2) gives empirical floors:
|
||||
|
||||
- **OpenAI**: 0.05 clears across 1024-1600 (n=4). 0.30 is large overkill here.
|
||||
- **Google (capped 1536)**: 0.15 (n=4); 0.05 and 0.10 do not clear.
|
||||
- **Google native 2816**: not locally measured; likely needs >= 0.30 (vendor +
|
||||
resolution stack). Use a GPU or `--max-resolution 1536`.
|
||||
|
||||
The default is **vendor-adaptive** (`watermark_profiles.resolve_strength` +
|
||||
`vendor_for_strength`): the tool reads the C2PA issuer on the original input and
|
||||
picks `OPENAI_STRENGTH` 0.10 / `GEMINI_STRENGTH` 0.15 / `UNKNOWN_STRENGTH` 0.15.
|
||||
This uses the vendor signal we DO have locally (the C2PA SynthID proxy) to avoid
|
||||
the overkill of a single high default on OpenAI images, without needing a local
|
||||
pixel detector. An explicit `--strength` always wins. If the watermark still
|
||||
survives (e.g. a large native Gemini beyond the capped-1536 validation), raise
|
||||
toward 0.30 then 0.35-0.40 (0.40 visibly corrupts dense text), using the lowest
|
||||
value that reads clean on the oracle.
|
||||
|
||||
### 5.3 Test methodology
|
||||
|
||||
- **GitHub-recompressed JPEGs from issue attachments are valid SynthID test
|
||||
subjects.** JPEG re-encoding removes C2PA metadata but does NOT remove the
|
||||
SynthID pixel watermark (verified June 2026 on issue #14 pic3). Do not
|
||||
dismiss these as "not faithful originals" for SynthID-removal tests.
|
||||
- **The correct oracle for OpenAI images is openai.com/verify**, not the Gemini
|
||||
app. The two oracles detect different payloads.
|
||||
- **A quiet `identify` output after processing is not proof of removal.** It
|
||||
means the metadata proxy is gone. The pixel watermark state is unknown without
|
||||
an oracle check.
|
||||
- **After removal, the output may carry forensic artifacts** detectable by an
|
||||
independent classifier even if the vendor oracle reads negative. Defeating the
|
||||
verifier is not the same as being forensically indistinguishable from clean
|
||||
content (arXiv:2605.09203).
|
||||
|
||||
### 5.4 ctrlregen and img2img: the tradeoff
|
||||
|
||||
Both the paper and our testing confirm: higher img2img strength removes the
|
||||
watermark but introduces detectable regeneration artifacts. The Goonatilake &
|
||||
Ateniese paper shows CtrlRegen+ (the most powerful remover) is simultaneously
|
||||
the most forensically detectable (AUROC 0.9999). The tradeoff is unavoidable
|
||||
with current diffusion-based approaches.
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
1. Gowal et al. (2025). **SynthID-Image: Image watermarking at internet scale.**
|
||||
arXiv:2510.09263. https://arxiv.org/abs/2510.09263
|
||||
|
||||
2. Google DeepMind. **Identifying AI-generated images with SynthID.** Blog post,
|
||||
2023. https://deepmind.google/blog/identifying-ai-generated-images-with-synthid/
|
||||
|
||||
3. Google DeepMind. **SynthID.** Product page.
|
||||
https://deepmind.google/models/synthid/
|
||||
|
||||
4. Goonatilake & Ateniese (2026). **Removing the Watermark Is Not Enough:
|
||||
Forensic Stealth in Generative-AI Watermark Removal.** arXiv:2605.09203.
|
||||
https://arxiv.org/abs/2605.09203
|
||||
|
||||
5. OpenAI. **Verify tool for AI-generated images.** openai.com/research/verify.
|
||||
Accessed 2026-05-31.
|
||||
@@ -14,8 +14,10 @@ filter applied to every candidate below.
|
||||
|
||||
## Problem recap
|
||||
|
||||
The `invisible` pipeline is SDXL base 1.0 img2img at low strength (~0.05) to defeat
|
||||
SynthID with minimal visible change. Text is protected via Differential Diffusion with a
|
||||
The `invisible` pipeline is SDXL base 1.0 img2img to defeat SynthID. The default
|
||||
strength has risen over time as Google hardens SynthID (0.05 -> 0.10 -> **~0.30**, the
|
||||
current threshold for fresh Gemini output); higher strength deforms text more, which is
|
||||
exactly why text protection matters. Text is protected via Differential Diffusion with a
|
||||
per-pixel change map (`preserve` ~0.9) driven by the PP-OCRv3 DB detector
|
||||
(`text_protector.py`). Large text survives; **small text (sub ~8 px strokes) softens or
|
||||
garbles** (issue #14, confirmed on real content).
|
||||
|
||||
+7
-3
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.8.1"
|
||||
version = "0.8.8"
|
||||
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
@@ -21,7 +21,6 @@ dependencies = [
|
||||
"numpy>=1.24.0",
|
||||
"opencv-python-headless>=4.8.0",
|
||||
"click>=8.0.0",
|
||||
"rich>=13.0.0",
|
||||
"python-dotenv>=1.0.0",
|
||||
]
|
||||
|
||||
@@ -112,7 +111,12 @@ remove-ai-watermarks = "remove_ai_watermarks.cli:main"
|
||||
Repository = "https://github.com/wiltodelta/remove-ai-watermarks"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
# Pin hatchling < 1.28: 1.28+ emits Metadata-Version 2.5 (PEP 639), which the twine
|
||||
# bundled in pypa/gh-action-pypi-publish@release/v1 rejects ("'2.5' is not a valid
|
||||
# Metadata-Version"), failing the PyPI upload (v0.8.3, 2026-06-01). 1.27.x emits 2.4,
|
||||
# which uploads fine (0.8.2 shipped on it). Lift this pin once the publish action's
|
||||
# twine is upgraded to >= 6.1.0 (2.5-aware) or the workflow moves to `uv publish`.
|
||||
requires = ["hatchling<1.28"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
"""Minimal plain-text stand-ins for the rich Console/Table API.
|
||||
|
||||
rich was dropped as a project dependency (see the CLI plain-text refactor), but
|
||||
the analysis scripts still printed through it. These shims keep the scripts
|
||||
runnable without rich: ``[bold]``/``[/]``-style markup is stripped and tables
|
||||
render as aligned plain text. Output goes through ``click.echo`` to match the
|
||||
package CLI (no bare ``print`` in tooling).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Any
|
||||
|
||||
import click
|
||||
|
||||
# Matches rich style tags: the bare close ``[/]`` and named open/close tags such
|
||||
# as ``[yellow]``, ``[bold yellow]``, ``[/green]``. Anchored to lowercase-letter
|
||||
# starts so numeric/data brackets (``[1024]``, ``[file.png]``) are left intact.
|
||||
_MARKUP = re.compile(r"\[(?:/|/?[a-z][a-z ]*)\]")
|
||||
|
||||
|
||||
def _strip(obj: Any) -> str:
|
||||
return _MARKUP.sub("", str(obj))
|
||||
|
||||
|
||||
class Table:
|
||||
"""Drop-in for ``rich.table.Table`` covering add_column/add_row + render."""
|
||||
|
||||
def __init__(self, *args: Any, title: str | None = None, **kwargs: Any) -> None:
|
||||
self.title = title
|
||||
self._headers: list[str] = []
|
||||
self._rows: list[list[str]] = []
|
||||
|
||||
def add_column(self, header: str = "", *args: Any, **kwargs: Any) -> None:
|
||||
self._headers.append(_strip(header))
|
||||
|
||||
def add_row(self, *cells: Any) -> None:
|
||||
self._rows.append([_strip(c) for c in cells])
|
||||
|
||||
def render(self) -> str:
|
||||
all_rows = ([self._headers] if self._headers else []) + self._rows
|
||||
cols = max((len(r) for r in all_rows), default=0)
|
||||
widths = [0] * cols
|
||||
for row in all_rows:
|
||||
for i, cell in enumerate(row):
|
||||
widths[i] = max(widths[i], len(cell))
|
||||
lines: list[str] = []
|
||||
if self.title:
|
||||
lines.append(_strip(self.title))
|
||||
if self._headers:
|
||||
lines.append(" ".join(h.ljust(widths[i]) for i, h in enumerate(self._headers)))
|
||||
lines.extend(" ".join(c.ljust(widths[i]) for i, c in enumerate(row)) for row in self._rows)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
class Console:
|
||||
"""Drop-in for ``rich.console.Console`` covering ``print`` (with Table)."""
|
||||
|
||||
def print(self, *objects: Any, **kwargs: Any) -> None:
|
||||
click.echo(" ".join(o.render() if isinstance(o, Table) else _strip(o) for o in objects))
|
||||
@@ -30,8 +30,7 @@ from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
from _plain_console import Console, Table
|
||||
|
||||
from remove_ai_watermarks.identify import identify
|
||||
from remove_ai_watermarks.metadata import _png_late_metadata
|
||||
|
||||
@@ -26,9 +26,8 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from _plain_console import Console, Table
|
||||
from PIL import Image
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from remove_ai_watermarks.noai.c2pa import extract_c2pa_info
|
||||
|
||||
|
||||
@@ -30,8 +30,8 @@ from typing import TYPE_CHECKING
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from _plain_console import Console
|
||||
from PIL import Image
|
||||
from rich.console import Console
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from numpy.typing import NDArray
|
||||
|
||||
@@ -33,8 +33,8 @@ from typing import Any
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
from _plain_console import Console
|
||||
from PIL import Image, ImageDraw, ImageFont
|
||||
from rich.console import Console
|
||||
|
||||
from remove_ai_watermarks import text_protector as tp
|
||||
|
||||
|
||||
@@ -1,3 +1,14 @@
|
||||
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
|
||||
|
||||
__version__ = "0.8.1"
|
||||
import os as _os
|
||||
import warnings as _warnings
|
||||
|
||||
# transformers prints a noisy deprecation for the Siglip2ImageProcessorFast
|
||||
# alias when it is imported (by the optional GPU/ML path). Silence it before
|
||||
# any submodule pulls transformers in, so the CLI startup stays quiet. Uses
|
||||
# setdefault so a user-set TRANSFORMERS_VERBOSITY still wins.
|
||||
_os.environ.setdefault("TRANSFORMERS_VERBOSITY", "error")
|
||||
_warnings.filterwarnings("ignore", message=r".*ImageProcessorFast.*")
|
||||
|
||||
|
||||
__version__ = "0.8.8"
|
||||
+219
-117
@@ -1,13 +1,14 @@
|
||||
"""Unified CLI for remove-ai-watermarks.
|
||||
|
||||
Provides commands for:
|
||||
- Visible watermark removal (Gemini sparkle) — works offline, fast
|
||||
- Invisible watermark removal (SynthID etc.) — requires GPU/diffusion models
|
||||
- AI metadata stripping — lightweight, no ML deps needed
|
||||
- Visible watermark removal (Gemini sparkle) - works offline, fast
|
||||
- Invisible watermark removal (SynthID etc.) - requires GPU/diffusion models
|
||||
- AI metadata stripping - lightweight, no ML deps needed
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
@@ -15,20 +16,92 @@ from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.progress import BarColumn, Progress, SpinnerColumn, TextColumn, TimeElapsedColumn
|
||||
from rich.table import Table
|
||||
|
||||
from remove_ai_watermarks import __version__, watermark_registry
|
||||
from remove_ai_watermarks.noai.watermark_profiles import DEFAULT_STRENGTH
|
||||
from remove_ai_watermarks.noai.watermark_profiles import resolve_strength, vendor_for_strength
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Generator
|
||||
|
||||
from numpy.typing import NDArray
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import DetectionResult
|
||||
|
||||
console = Console()
|
||||
# --- plain-text output layer (replaces rich: no colors, no markup, no boxes) ---
|
||||
|
||||
|
||||
class _Table:
|
||||
"""Plain-text stand-in for rich.Table."""
|
||||
|
||||
def __init__(self, *args: Any, title: str | None = None, **kwargs: Any) -> None:
|
||||
self._title = title
|
||||
self._headers: list[str] = []
|
||||
self._rows: list[list[str]] = []
|
||||
|
||||
def add_column(self, header: str = "", *args: Any, **kwargs: Any) -> None:
|
||||
self._headers.append(str(header))
|
||||
|
||||
def add_row(self, *cells: Any) -> None:
|
||||
self._rows.append([str(c) for c in cells])
|
||||
|
||||
def render(self) -> str:
|
||||
lines: list[str] = []
|
||||
if self._title:
|
||||
lines.append(self._title)
|
||||
if any(self._headers):
|
||||
lines.append(" ".join(self._headers))
|
||||
lines.extend(" ".join(row) for row in self._rows)
|
||||
return "\n".join(f" {line}" for line in lines)
|
||||
|
||||
|
||||
class _Progress:
|
||||
"""No-op stand-in for rich.Progress; results are printed directly instead."""
|
||||
|
||||
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||
pass
|
||||
|
||||
def __enter__(self) -> _Progress:
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc: object) -> bool:
|
||||
return False
|
||||
|
||||
def add_task(self, *args: Any, **kwargs: Any) -> int:
|
||||
return 0
|
||||
|
||||
def advance(self, *args: Any, **kwargs: Any) -> None:
|
||||
pass
|
||||
|
||||
def update(self, *args: Any, **kwargs: Any) -> None:
|
||||
pass
|
||||
|
||||
|
||||
class _Console:
|
||||
"""Minimal plain-text replacement for rich.Console."""
|
||||
|
||||
def print(self, *objects: Any, **kwargs: Any) -> None:
|
||||
click.echo(" ".join(o.render() if isinstance(o, _Table) else str(o) for o in objects))
|
||||
|
||||
@contextlib.contextmanager
|
||||
def status(self, message: str = "", **kwargs: Any) -> Generator[None, None, None]:
|
||||
if message:
|
||||
click.echo(message)
|
||||
yield
|
||||
|
||||
|
||||
def _panel(text: str = "", *args: Any, **kwargs: Any) -> str:
|
||||
return text
|
||||
|
||||
|
||||
def _column(*args: Any, **kwargs: Any) -> None:
|
||||
return None
|
||||
|
||||
|
||||
Panel = _panel
|
||||
Table = _Table
|
||||
Progress = _Progress
|
||||
SpinnerColumn = BarColumn = TextColumn = TimeElapsedColumn = _column
|
||||
console = _Console()
|
||||
|
||||
SUPPORTED_FORMATS = {".png", ".jpg", ".jpeg", ".webp"}
|
||||
|
||||
@@ -45,7 +118,7 @@ def _setup_logging(verbose: bool) -> None:
|
||||
def _banner() -> None:
|
||||
console.print(
|
||||
Panel(
|
||||
f"[bold cyan]Remove-AI-Watermarks[/] [dim]v{__version__}[/]\n[dim]Visible & invisible watermark removal[/]",
|
||||
f"Remove-AI-Watermarks v{__version__}\nVisible & invisible watermark removal",
|
||||
border_style="cyan",
|
||||
padding=(0, 2),
|
||||
)
|
||||
@@ -54,17 +127,34 @@ def _banner() -> None:
|
||||
|
||||
def _validate_image(path: Path) -> Path:
|
||||
if not path.exists():
|
||||
console.print(f"[red]Error:[/] File not found: {path}")
|
||||
console.print(f"Error: File not found: {path}")
|
||||
raise SystemExit(1)
|
||||
if path.suffix.lower() not in SUPPORTED_FORMATS:
|
||||
console.print(
|
||||
f"[yellow]Warning:[/] {path.suffix} may not be supported (expected: {', '.join(SUPPORTED_FORMATS)})"
|
||||
)
|
||||
console.print(f"Warning: {path.suffix} may not be supported (expected: {', '.join(SUPPORTED_FORMATS)})")
|
||||
return path
|
||||
|
||||
|
||||
_ALPHA_FORMATS = {".png", ".webp"}
|
||||
|
||||
# Shared option decorators for commands that run the invisible-watermark pipeline.
|
||||
# Both cmd_invisible and cmd_all expose these flags; defining them once avoids
|
||||
# copy-paste drift.
|
||||
_protect_text_option = click.option(
|
||||
"--protect-text",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help=(
|
||||
"Enable text region protection (experimental: re-scrubs text blocks at high resolution). "
|
||||
"May prevent SynthID removal in text areas -- verify with oracle before relying on it."
|
||||
),
|
||||
)
|
||||
_protect_faces_option = click.option(
|
||||
"--protect-faces",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Enable face protection (experimental: YOLO detect + blend original faces back).",
|
||||
)
|
||||
|
||||
|
||||
def _watermark_region(det: DetectionResult, width: int, height: int) -> tuple[int, int, int, int]:
|
||||
"""Pick a watermark bbox: detector's region if confident, else the default config slot."""
|
||||
@@ -124,7 +214,7 @@ def _write_bgr_with_alpha(
|
||||
image_io.imwrite(path, bgra)
|
||||
|
||||
|
||||
# ── Main group ───────────────────────────────────────────────────────
|
||||
# -- Main group -------------------------------------------------------
|
||||
|
||||
|
||||
@click.group(invoke_without_command=True)
|
||||
@@ -146,7 +236,7 @@ def main(ctx: click.Context, verbose: bool) -> None:
|
||||
click.echo(ctx.get_help())
|
||||
|
||||
|
||||
# ── Visible (Gemini) watermark removal ───────────────────────────────
|
||||
# -- Visible (Gemini) watermark removal -------------------------------
|
||||
|
||||
|
||||
@main.command("visible")
|
||||
@@ -198,11 +288,11 @@ def cmd_visible(
|
||||
# Load image (preserving any alpha channel separately)
|
||||
image, alpha = _read_bgr_and_alpha(source)
|
||||
if image is None:
|
||||
console.print(f"[red]Error:[/] Failed to read image: {source}")
|
||||
console.print(f"Error: Failed to read image: {source}")
|
||||
raise SystemExit(1)
|
||||
|
||||
h, w = image.shape[:2]
|
||||
console.print(f" [dim]Input:[/] {source.name} ({w}x{h})")
|
||||
console.print(f" Input: {source.name} ({w}x{h})")
|
||||
|
||||
# Resolve the target mark from the known-watermark registry. ``auto`` scans
|
||||
# every in-auto mark in its usual place and picks the strongest; an explicit
|
||||
@@ -210,31 +300,28 @@ def cmd_visible(
|
||||
if mark == "auto":
|
||||
best = registry.best_auto_mark(image)
|
||||
if best is None:
|
||||
console.print(" [yellow]⚠[/] No known visible mark detected (gemini / doubao).")
|
||||
console.print(" Warning: No known visible mark detected (gemini / doubao).")
|
||||
if detect:
|
||||
console.print(" [dim]Skipping. Use --mark <name> --no-detect to force.[/]")
|
||||
console.print(" Skipping. Use --mark <name> --no-detect to force.")
|
||||
raise SystemExit(0)
|
||||
target = "gemini" # forced (no-detect): fall back to the default mark
|
||||
else:
|
||||
target = best.key
|
||||
console.print(f" [dim]Mark auto:[/] {best.label} [dim]({best.location}, conf {best.confidence:.2f})[/]")
|
||||
console.print(f" Mark auto: {best.label} ({best.location}, conf {best.confidence:.2f})")
|
||||
else:
|
||||
target = mark
|
||||
|
||||
chosen = registry.get_mark(target)
|
||||
det = chosen.detect(image)
|
||||
if detect and not det.detected:
|
||||
console.print(
|
||||
f" [yellow]⚠[/] {chosen.label} not detected "
|
||||
f"[dim](conf {det.confidence:.2f}). Use --no-detect to force.[/]"
|
||||
)
|
||||
console.print(f" Warning: {chosen.label} not detected (conf {det.confidence:.2f}). Use --no-detect to force.")
|
||||
raise SystemExit(0)
|
||||
if det.detected:
|
||||
console.print(f" [green]✓[/] {chosen.label} detected [dim]({chosen.location}, conf {det.confidence:.2f})[/]")
|
||||
console.print(f" {chosen.label} detected ({chosen.location}, conf {det.confidence:.2f})")
|
||||
|
||||
method: Literal["telea", "ns"] = "ns" if inpaint_method == "ns" else "telea"
|
||||
t0 = time.monotonic()
|
||||
with console.status(f"[cyan]Removing {chosen.label}… ({chosen.recovery})[/]"):
|
||||
with console.status(f"Removing {chosen.label}... ({chosen.recovery})"):
|
||||
result, _ = chosen.remove(
|
||||
image,
|
||||
inpaint_method=method,
|
||||
@@ -256,13 +343,13 @@ def cmd_visible(
|
||||
remove_ai_metadata(output, output)
|
||||
except Exception as e:
|
||||
if ctx.obj.get("verbose"):
|
||||
console.print(f" [yellow]⚠[/] Failed to strip metadata: {e}")
|
||||
console.print(f" Warning: Failed to strip metadata: {e}")
|
||||
|
||||
size_kb = output.stat().st_size / 1024
|
||||
console.print(f" [green]✓[/] Saved: {output} [dim]({size_kb:.0f} KB, {elapsed:.2f}s)[/]")
|
||||
console.print(f" Saved: {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
||||
|
||||
|
||||
# ── Universal region eraser ─────────────────────────────────────────
|
||||
# -- Universal region eraser -----------------------------------------
|
||||
|
||||
|
||||
def _parse_region(spec: str) -> tuple[int, int, int, int]:
|
||||
@@ -322,18 +409,18 @@ def cmd_erase(
|
||||
|
||||
image, alpha = _read_bgr_and_alpha(source)
|
||||
if image is None:
|
||||
console.print(f"[red]Error:[/] Failed to read image: {source}")
|
||||
console.print(f"Error: Failed to read image: {source}")
|
||||
raise SystemExit(1)
|
||||
h, w = image.shape[:2]
|
||||
console.print(f" [dim]Input:[/] {source.name} ({w}x{h}) [dim]{len(boxes)} region(s), backend={backend}[/]")
|
||||
console.print(f" Input: {source.name} ({w}x{h}) {len(boxes)} region(s), backend={backend}")
|
||||
|
||||
t0 = time.monotonic()
|
||||
method: Literal["telea", "ns"] = "ns" if inpaint_method == "ns" else "telea"
|
||||
try:
|
||||
with console.status(f"[cyan]Erasing ({backend})…[/]"):
|
||||
with console.status(f"Erasing ({backend})..."):
|
||||
result = erase(image, boxes=boxes, backend=backend, dilate=dilate, cv2_method=method)
|
||||
except RuntimeError as e:
|
||||
console.print(f" [red]Error:[/] {e}")
|
||||
console.print(f" Error: {e}")
|
||||
raise SystemExit(1) from e
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
@@ -347,13 +434,13 @@ def cmd_erase(
|
||||
remove_ai_metadata(output, output)
|
||||
except Exception as e:
|
||||
if ctx.obj.get("verbose"):
|
||||
console.print(f" [yellow]⚠[/] Failed to strip metadata: {e}")
|
||||
console.print(f" Warning: Failed to strip metadata: {e}")
|
||||
|
||||
size_kb = output.stat().st_size / 1024
|
||||
console.print(f" [green]✓[/] Erased {len(boxes)} region(s) → {output} [dim]({size_kb:.0f} KB, {elapsed:.2f}s)[/]")
|
||||
console.print(f" Erased {len(boxes)} region(s) -> {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
||||
|
||||
|
||||
# ── Invisible watermark removal ─────────────────────────────────────
|
||||
# -- Invisible watermark removal -------------------------------------
|
||||
|
||||
|
||||
@main.command("invisible")
|
||||
@@ -361,13 +448,19 @@ def cmd_erase(
|
||||
@click.option(
|
||||
"-o", "--output", type=click.Path(path_type=Path), default=None, help="Output path (default: <source>_clean.<ext>)."
|
||||
)
|
||||
@click.option("--strength", type=float, default=DEFAULT_STRENGTH, help="Denoising strength (0.0-1.0). Default: 0.10.")
|
||||
@click.option(
|
||||
"--strength",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Denoising strength (0.0-1.0). Default: vendor-adaptive (OpenAI 0.10 / Google 0.15 / "
|
||||
"unknown 0.15, from the C2PA issuer); ctrlregen uses 1.0.",
|
||||
)
|
||||
@click.option("--steps", type=int, default=50, help="Number of denoising steps. Default: 50.")
|
||||
@click.option(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
help="Pipeline profile (default=SDXL; ctrlregen=CtrlRegen, EXPERIMENTAL/destructive at clean-noise).",
|
||||
)
|
||||
@click.option(
|
||||
"--device",
|
||||
@@ -386,18 +479,14 @@ def cmd_erase(
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@click.option(
|
||||
"--no-protect-text",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Disable automatic text protection (text/CJK is preserved by default on the SDXL pipeline).",
|
||||
)
|
||||
@_protect_text_option
|
||||
@_protect_faces_option
|
||||
@click.pass_context
|
||||
def cmd_invisible(
|
||||
ctx: click.Context,
|
||||
source: Path,
|
||||
output: Path | None,
|
||||
strength: float,
|
||||
strength: float | None,
|
||||
steps: int,
|
||||
pipeline: str,
|
||||
device: str,
|
||||
@@ -405,7 +494,8 @@ def cmd_invisible(
|
||||
hf_token: str | None,
|
||||
humanize: float,
|
||||
max_resolution: int,
|
||||
no_protect_text: bool,
|
||||
protect_text: bool,
|
||||
protect_faces: bool,
|
||||
) -> None:
|
||||
"""Remove invisible AI watermarks (SynthID, StableSignature, TreeRing).
|
||||
|
||||
@@ -416,8 +506,7 @@ def cmd_invisible(
|
||||
|
||||
if not invisible_available():
|
||||
console.print(
|
||||
"[red]Error:[/] GPU dependencies not installed.\n"
|
||||
" Install them with: [bold]pip install 'remove-ai-watermarks\\[gpu]'[/]"
|
||||
"Error: GPU dependencies not installed.\n Install them with: pip install 'remove-ai-watermarks[gpu]'"
|
||||
)
|
||||
raise SystemExit(1)
|
||||
|
||||
@@ -430,7 +519,7 @@ def cmd_invisible(
|
||||
device_str = None if device == "auto" else device
|
||||
|
||||
def progress_cb(msg: str) -> None:
|
||||
console.print(f" [dim]{msg}[/]")
|
||||
console.print(f" {msg}")
|
||||
|
||||
engine = InvisibleEngine(
|
||||
device=device_str,
|
||||
@@ -439,9 +528,12 @@ def cmd_invisible(
|
||||
progress_callback=progress_cb,
|
||||
)
|
||||
|
||||
console.print(f" [dim]Input:[/] {source.name}")
|
||||
console.print(f" [dim]Pipeline:[/] {pipeline}")
|
||||
console.print(f" [dim]Strength:[/] {strength} Steps: {steps}")
|
||||
# Detect the SynthID vendor from the ORIGINAL (before processing strips C2PA) so the
|
||||
# displayed and executed strength agree on the vendor-adaptive default.
|
||||
vendor = vendor_for_strength(source)
|
||||
console.print(f" Input: {source.name}")
|
||||
console.print(f" Pipeline: {pipeline}")
|
||||
console.print(f" Strength: {resolve_strength(strength, pipeline, vendor)} Steps: {steps}")
|
||||
|
||||
t0 = time.monotonic()
|
||||
result_path = engine.remove_watermark(
|
||||
@@ -452,16 +544,18 @@ def cmd_invisible(
|
||||
guidance_scale=None,
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
protect_text=not no_protect_text,
|
||||
protect_text=protect_text,
|
||||
protect_faces=protect_faces,
|
||||
max_resolution=max_resolution,
|
||||
vendor=vendor,
|
||||
)
|
||||
elapsed = time.monotonic() - t0
|
||||
|
||||
size_kb = result_path.stat().st_size / 1024
|
||||
console.print(f"\n [green]✓[/] Saved: {result_path} [dim]({size_kb:.0f} KB, {elapsed:.1f}s)[/]")
|
||||
console.print(f"\n Saved: {result_path} ({size_kb:.0f} KB, {elapsed:.1f}s)")
|
||||
|
||||
|
||||
# ── Metadata operations ─────────────────────────────────────────────
|
||||
# -- Metadata operations ---------------------------------------------
|
||||
|
||||
|
||||
@main.command("metadata")
|
||||
@@ -499,10 +593,10 @@ def cmd_metadata(
|
||||
if check or (not remove):
|
||||
has_ai = has_ai_metadata(source)
|
||||
if has_ai:
|
||||
console.print(f" [yellow]⚠[/] AI metadata detected in {source.name}:")
|
||||
console.print(f" Warning: AI metadata detected in {source.name}:")
|
||||
meta = get_ai_metadata(source)
|
||||
if synthid := meta.get("synthid_watermark"):
|
||||
console.print(f" [bold yellow]⚠ SynthID watermark (inferred from C2PA metadata) {synthid}[/]")
|
||||
console.print(f" Warning: SynthID watermark (inferred from C2PA metadata) {synthid}")
|
||||
table = Table(show_header=True, header_style="bold")
|
||||
table.add_column("Key", style="cyan")
|
||||
table.add_column("Value")
|
||||
@@ -510,17 +604,17 @@ def cmd_metadata(
|
||||
table.add_row(k, str(v)[:80])
|
||||
console.print(table)
|
||||
else:
|
||||
console.print(f" [green]✓[/] No AI metadata found in {source.name}")
|
||||
console.print(f" No AI metadata found in {source.name}")
|
||||
|
||||
if not remove:
|
||||
return
|
||||
|
||||
# Remove
|
||||
out = remove_ai_metadata(source, output, keep_standard=keep_standard)
|
||||
console.print(f" [green]✓[/] AI metadata stripped → {out}")
|
||||
console.print(f" AI metadata stripped -> {out}")
|
||||
|
||||
|
||||
# ── Provenance identification ───────────────────────────────────────
|
||||
# -- Provenance identification ---------------------------------------
|
||||
|
||||
|
||||
@main.command("identify")
|
||||
@@ -552,24 +646,22 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
return
|
||||
|
||||
_banner()
|
||||
verdict = {True: "[yellow]AI-generated[/]", False: "[green]not AI[/]", None: "[dim]unknown[/]"}[
|
||||
report.is_ai_generated
|
||||
]
|
||||
console.print(f"\n Verdict: {verdict} [dim](confidence: {report.confidence})[/]")
|
||||
console.print(f" Platform: {report.platform or '[dim]undetermined[/]'}")
|
||||
verdict = {True: "AI-generated", False: "not AI", None: "unknown"}[report.is_ai_generated]
|
||||
console.print(f"\n Verdict: {verdict} (confidence: {report.confidence})")
|
||||
console.print(f" Platform: {report.platform or 'undetermined'}")
|
||||
|
||||
if report.is_ai_generated is None:
|
||||
console.print(
|
||||
" [dim]No locally-readable AI signal found. This is not the same as 'clean': "
|
||||
" No locally-readable AI signal found. This is not the same as 'clean': "
|
||||
"metadata is often stripped by re-encoding, screenshots, or upload, and SynthID-class "
|
||||
"pixel watermarks (Gemini / Nano Banana / gpt-image) have no local detector. "
|
||||
"See caveats below.[/]"
|
||||
"See caveats below."
|
||||
)
|
||||
|
||||
if report.integrity_clashes:
|
||||
console.print("\n [bold red]⚠ Integrity clash[/] [dim](provenance signals contradict each other)[/]")
|
||||
console.print("\n Warning: Integrity clash (provenance signals contradict each other)")
|
||||
for clash in report.integrity_clashes:
|
||||
console.print(f" [red]- {clash}[/]")
|
||||
console.print(f" - {clash}")
|
||||
|
||||
if report.watermarks:
|
||||
table = Table(show_header=True, header_style="bold", title="Watermarks / provenance markers")
|
||||
@@ -578,15 +670,15 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
table.add_row(wm)
|
||||
console.print(table)
|
||||
else:
|
||||
console.print(" [dim]No watermarks or provenance markers found.[/]")
|
||||
console.print(" No watermarks or provenance markers found.")
|
||||
|
||||
if report.caveats:
|
||||
console.print("\n [dim]Caveats:[/]")
|
||||
console.print("\n Caveats:")
|
||||
for c in report.caveats:
|
||||
console.print(f" [dim]- {c}[/]")
|
||||
console.print(f" - {c}")
|
||||
|
||||
|
||||
# ── Combined "all" mode ──────────────────────────────────────────────
|
||||
# -- Combined "all" mode ----------------------------------------------
|
||||
|
||||
|
||||
@main.command("all")
|
||||
@@ -599,14 +691,18 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
"--inpaint-method", type=click.Choice(["ns", "telea", "gaussian"]), default="ns", help="Inpainting method."
|
||||
)
|
||||
@click.option(
|
||||
"--strength", type=float, default=DEFAULT_STRENGTH, help="Invisible watermark denoising strength. Default: 0.10."
|
||||
"--strength",
|
||||
type=float,
|
||||
default=None,
|
||||
help="Invisible watermark denoising strength. Default: vendor-adaptive "
|
||||
"(OpenAI 0.10 / Google 0.15 / unknown 0.15); ctrlregen uses 1.0.",
|
||||
)
|
||||
@click.option("--steps", type=int, default=50, help="Number of denoising steps for invisible removal.")
|
||||
@click.option(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
help="Pipeline profile (default=SDXL; ctrlregen=CtrlRegen, EXPERIMENTAL/destructive at clean-noise).",
|
||||
)
|
||||
@click.option("--model", type=str, default=None, help="HuggingFace model ID for invisible removal.")
|
||||
@click.option(
|
||||
@@ -626,12 +722,8 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@click.option(
|
||||
"--no-protect-text",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="Disable automatic text protection (text/CJK is preserved by default on the SDXL pipeline).",
|
||||
)
|
||||
@_protect_text_option
|
||||
@_protect_faces_option
|
||||
@click.pass_context
|
||||
def cmd_all(
|
||||
ctx: click.Context,
|
||||
@@ -639,7 +731,7 @@ def cmd_all(
|
||||
output: Path | None,
|
||||
inpaint: bool,
|
||||
inpaint_method: Literal["ns", "telea", "gaussian"],
|
||||
strength: float,
|
||||
strength: float | None,
|
||||
steps: int,
|
||||
pipeline: str,
|
||||
model: str | None,
|
||||
@@ -648,7 +740,8 @@ def cmd_all(
|
||||
hf_token: str | None,
|
||||
humanize: float,
|
||||
max_resolution: int,
|
||||
no_protect_text: bool,
|
||||
protect_text: bool,
|
||||
protect_faces: bool,
|
||||
) -> None:
|
||||
"""Remove ALL watermarks: visible + invisible + metadata.
|
||||
|
||||
@@ -680,40 +773,40 @@ def cmd_all(
|
||||
|
||||
os.close(tmp_fd)
|
||||
|
||||
# ── Step 1: Visible watermark ────────────────────────────────
|
||||
console.print("\n [bold cyan]① Visible watermark removal[/]")
|
||||
# -- Step 1: Visible watermark --------------------------------
|
||||
console.print("\n 1) Visible watermark removal")
|
||||
engine = GeminiEngine()
|
||||
image, alpha = _read_bgr_and_alpha(source)
|
||||
if image is None:
|
||||
console.print(f"[red]Error:[/] Failed to read image: {source}")
|
||||
console.print(f"Error: Failed to read image: {source}")
|
||||
raise SystemExit(1)
|
||||
|
||||
h, w = image.shape[:2]
|
||||
console.print(f" [dim]Input:[/] {source.name} ({w}x{h})")
|
||||
console.print(f" Input: {source.name} ({w}x{h})")
|
||||
|
||||
with console.status("[cyan]Removing visible watermark…[/]"):
|
||||
with console.status("Removing visible watermark..."):
|
||||
det = engine.detect_watermark(image)
|
||||
if det.detected:
|
||||
result = engine.remove_watermark(image)
|
||||
if inpaint:
|
||||
region = _watermark_region(det, w, h)
|
||||
result = engine.inpaint_residual(result, region, method=inpaint_method)
|
||||
console.print(" [green]✓[/] Visible watermark removed")
|
||||
console.print(" Visible watermark removed")
|
||||
else:
|
||||
result = image.copy()
|
||||
console.print(" [dim]Skipped (no visible watermark detected)[/]")
|
||||
console.print(" Skipped (no visible watermark detected)")
|
||||
|
||||
# Save to temp file for invisible engine input (preserve alpha if present)
|
||||
_write_bgr_with_alpha(tmp_path, result, alpha)
|
||||
|
||||
# ── Step 2: Invisible watermark ──────────────────────────────
|
||||
console.print("\n [bold cyan]② Invisible watermark removal[/]")
|
||||
# -- Step 2: Invisible watermark ------------------------------
|
||||
console.print("\n 2) Invisible watermark removal")
|
||||
from remove_ai_watermarks.invisible_engine import is_available as invisible_available
|
||||
|
||||
if not invisible_available():
|
||||
console.print(
|
||||
" [yellow]⚠[/] Skipped — GPU dependencies not installed.\n"
|
||||
" Install them with: [bold]pip install 'remove-ai-watermarks\\[gpu]'[/]"
|
||||
" Warning: Skipped - GPU dependencies not installed.\n"
|
||||
" Install them with: pip install 'remove-ai-watermarks[gpu]'"
|
||||
)
|
||||
else:
|
||||
from remove_ai_watermarks.invisible_engine import InvisibleEngine
|
||||
@@ -721,7 +814,7 @@ def cmd_all(
|
||||
device_str = None if device == "auto" else device
|
||||
|
||||
def progress_cb(msg: str) -> None:
|
||||
console.print(f" [dim]{msg}[/]")
|
||||
console.print(f" {msg}")
|
||||
|
||||
inv_engine = InvisibleEngine(
|
||||
model_id=model,
|
||||
@@ -731,7 +824,11 @@ def cmd_all(
|
||||
progress_callback=progress_cb,
|
||||
)
|
||||
|
||||
console.print(f" [dim]Strength:[/] {strength} Steps: {steps}")
|
||||
# Detect the vendor from the pristine ORIGINAL (`source`); `tmp_path` has
|
||||
# already lost its C2PA to the visible-removal pass, so reading it would
|
||||
# always resolve to the unknown-vendor default.
|
||||
vendor = vendor_for_strength(source)
|
||||
console.print(f" Strength: {resolve_strength(strength, pipeline, vendor)} Steps: {steps}")
|
||||
inv_engine.remove_watermark(
|
||||
image_path=tmp_path,
|
||||
output_path=tmp_path,
|
||||
@@ -739,29 +836,31 @@ def cmd_all(
|
||||
num_inference_steps=steps,
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
protect_text=not no_protect_text,
|
||||
protect_text=protect_text,
|
||||
protect_faces=protect_faces,
|
||||
max_resolution=max_resolution,
|
||||
vendor=vendor,
|
||||
)
|
||||
console.print(" [green]✓[/] Invisible watermark removed")
|
||||
console.print(" Invisible watermark removed")
|
||||
|
||||
# ── Step 3: Metadata ─────────────────────────────────────────
|
||||
console.print("\n [bold cyan]③ AI metadata stripping[/]")
|
||||
# -- Step 3: Metadata -----------------------------------------
|
||||
console.print("\n 3) AI metadata stripping")
|
||||
try:
|
||||
from remove_ai_watermarks.metadata import remove_ai_metadata
|
||||
|
||||
remove_ai_metadata(tmp_path, tmp_path)
|
||||
console.print(" [green]✓[/] AI metadata stripped")
|
||||
console.print(" AI metadata stripped")
|
||||
except Exception as e:
|
||||
console.print(f" [yellow]⚠[/] Metadata strip failed: {e}")
|
||||
console.print(f" Warning: Metadata strip failed: {e}")
|
||||
|
||||
# ── Write final result ────────────────────────────────────────
|
||||
# -- Write final result ----------------------------------------
|
||||
# The invisible step (and downstream cv2.IMREAD_COLOR paths) drops alpha,
|
||||
# so re-attach the original alpha plane unchanged when writing the final
|
||||
# output for transparent formats.
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
final_bgr, _ = _read_bgr_and_alpha(tmp_path)
|
||||
if final_bgr is None:
|
||||
console.print(f"[red]Error:[/] Failed to read intermediate file: {tmp_path}")
|
||||
console.print(f"Error: Failed to read intermediate file: {tmp_path}")
|
||||
raise SystemExit(1)
|
||||
_write_bgr_with_alpha(output, final_bgr, alpha)
|
||||
|
||||
@@ -770,13 +869,13 @@ def cmd_all(
|
||||
if tmp_path.exists():
|
||||
tmp_path.unlink()
|
||||
|
||||
# ── Done ─────────────────────────────────────────────────────
|
||||
# -- Done -----------------------------------------------------
|
||||
elapsed = time.monotonic() - t0
|
||||
size_kb = output.stat().st_size / 1024
|
||||
console.print(f"\n [bold green]✓ Done:[/] {output} [dim]({size_kb:.0f} KB, {elapsed:.1f}s total)[/]")
|
||||
console.print(f"\n Done: {output} ({size_kb:.0f} KB, {elapsed:.1f}s total)")
|
||||
|
||||
|
||||
# ── Batch command ────────────────────────────────────────────────────
|
||||
# -- Batch command ----------------------------------------------------
|
||||
|
||||
|
||||
def _process_batch_image(
|
||||
@@ -853,6 +952,9 @@ def _process_batch_image(
|
||||
seed=seed,
|
||||
humanize=humanize,
|
||||
max_resolution=max_resolution,
|
||||
# Detect the vendor from the pristine original (`img_path`), not the
|
||||
# visible-processed `out_path` whose C2PA is already gone.
|
||||
vendor=vendor_for_strength(img_path),
|
||||
)
|
||||
|
||||
if mode in ("metadata", "all"):
|
||||
@@ -890,7 +992,7 @@ def _process_batch_image(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
help="Pipeline profile (default=SDXL; ctrlregen=CtrlRegen, EXPERIMENTAL/destructive at clean-noise).",
|
||||
)
|
||||
@click.option(
|
||||
"--device",
|
||||
@@ -932,12 +1034,12 @@ def cmd_batch(
|
||||
images = sorted(p for p in directory.iterdir() if p.suffix.lower() in SUPPORTED_FORMATS)
|
||||
|
||||
if not images:
|
||||
console.print(f"[yellow]No supported images found in {directory}[/]")
|
||||
console.print(f"No supported images found in {directory}")
|
||||
return
|
||||
|
||||
console.print(f" Found [bold]{len(images)}[/] images in {directory}")
|
||||
console.print(f" Output → {output_dir}")
|
||||
console.print(f" Mode: [cyan]{mode}[/]")
|
||||
console.print(f" Found {len(images)} images in {directory}")
|
||||
console.print(f" Output -> {output_dir}")
|
||||
console.print(f" Mode: {mode}")
|
||||
|
||||
processed = 0
|
||||
errors = 0
|
||||
@@ -950,11 +1052,11 @@ def cmd_batch(
|
||||
TimeElapsedColumn(),
|
||||
console=console,
|
||||
) as progress:
|
||||
task = progress.add_task("Processing…", total=len(images))
|
||||
task = progress.add_task("Processing...", total=len(images))
|
||||
|
||||
for img_path in images:
|
||||
out_path = output_dir / img_path.name
|
||||
progress.update(task, description=f"[cyan]{img_path.name}[/]")
|
||||
progress.update(task, description=f"{img_path.name}")
|
||||
|
||||
try:
|
||||
_process_batch_image(
|
||||
@@ -977,11 +1079,11 @@ def cmd_batch(
|
||||
except Exception as e:
|
||||
errors += 1
|
||||
if ctx.obj.get("verbose"):
|
||||
console.print(f" [red]✗[/] {img_path.name}: {e}")
|
||||
console.print(f" {img_path.name}: {e}")
|
||||
|
||||
progress.advance(task)
|
||||
|
||||
console.print(f"\n [green]✓[/] {processed} processed" + (f" [red]✗[/] {errors} errors" if errors else ""))
|
||||
console.print(f"\n {processed} processed" + (f" {errors} errors" if errors else ""))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -4,21 +4,25 @@ Doubao (ByteDance) stamps every generated image with a visible "豆包AI生成"
|
||||
(Doubao AI generated) text strip in the bottom-right corner -- the explicit AIGC
|
||||
label mandated by China's TC260 standard, a near-white semi-transparent overlay.
|
||||
|
||||
Like the Gemini sparkle, it is a fixed overlay, so it is removed by **exact
|
||||
reverse-alpha blending** against a captured alpha map (``remove_watermark_reverse_alpha``):
|
||||
``original = (wm - a*logo)/(1-a)`` -- recovering the true pixels, not an inpaint
|
||||
guess. The alpha map + logo colour were solved from black+gray Doubao captures
|
||||
(see data/doubao_capture/ and the reverse-alpha section below) and bundled as
|
||||
``assets/doubao_alpha.png``.
|
||||
Like the Gemini sparkle and the Jimeng wordmark, it is a fixed overlay, so removal
|
||||
starts from **reverse-alpha blending** against a captured alpha map
|
||||
(``remove_watermark_reverse_alpha``): ``original = (wm - a*logo)/(1-a)``. The alpha
|
||||
map is rebuilt by ``scripts/visible_alpha_solve.py`` from black/gray Doubao captures
|
||||
(the careful gray-self solve; logo is pure white) and bundled as
|
||||
``assets/doubao_alpha.png``. The mark re-rasterizes a few px off per image, so
|
||||
removal ALWAYS NCC-aligns the template to the actual mark and then clears the
|
||||
residual edges with a deliberately THIN inpaint over the glyph footprint (an
|
||||
earlier under-estimated alpha + fixed-no-inpaint left a readable outline that the
|
||||
detector did not flag -- see the reverse-alpha section below).
|
||||
|
||||
Detection (``detect``) is reverse-alpha-consistent: it matches that same alpha
|
||||
glyph silhouette against the corner via normalized correlation, so it keys on
|
||||
the actual "豆包AI生成" shape rather than coverage/structure heuristics.
|
||||
Detection (``detect``) is shape-consistent: it matches that same alpha glyph
|
||||
silhouette against the corner via normalized correlation, so it keys on the actual
|
||||
"豆包AI生成" shape rather than coverage/structure heuristics.
|
||||
|
||||
``locate`` (geometry box, scales with image WIDTH) and ``extract_mask`` (the
|
||||
candidate glyph mask the detector correlates) remain; there is no inpaint-based
|
||||
removal here -- arbitrary-region inpainting lives in ``region_eraser`` / the
|
||||
``erase`` command. Fast, offline, no GPU.
|
||||
candidate glyph mask the detector correlates) mirror the Jimeng engine.
|
||||
Arbitrary-region inpainting still lives in ``region_eraser`` / the ``erase``
|
||||
command. Fast, offline, no GPU.
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: third-party libs ship no usable element types; relax the
|
||||
@@ -227,6 +231,13 @@ class DoubaoEngine:
|
||||
"""
|
||||
h, w = image.shape[:2]
|
||||
x, y, bw, bh = loc.bbox
|
||||
# A degenerate ROI (a sliver from an extremely wide/short image) cannot hold
|
||||
# the mark and would feed cv2's GaussianBlur/morphology a ~1-px-tall array,
|
||||
# which can fault the native code on some platforms (observed: a Windows
|
||||
# access violation via the always-align removal's residual `detect`). Skip
|
||||
# the cv2 pipeline and return an empty mask there.
|
||||
if bh < 16 or bw < 16:
|
||||
return np.zeros((h, w), np.uint8)
|
||||
# Normalize the ROI to 3-channel BGR: a 2D grayscale or 4-channel BGRA
|
||||
# input would otherwise break the axis=2 channel reductions below.
|
||||
roi = image[y : y + bh, x : x + bw]
|
||||
@@ -372,6 +383,13 @@ class DoubaoEngine:
|
||||
image = cv2.cvtColor(image, cv2.COLOR_GRAY2BGR)
|
||||
elif image.shape[2] == 4:
|
||||
image = cv2.cvtColor(image, cv2.COLOR_BGRA2BGR)
|
||||
# An image too small to hold the mark would make the geometry boxes
|
||||
# degenerate and feed cv2.resize a ~1-px-tall target / GaussianBlur a sliver
|
||||
# ROI, which faults natively on Windows (access violation / "Unknown C++
|
||||
# exception"). No real watermarked image is this small; skip cv2 entirely.
|
||||
h, w = image.shape[:2]
|
||||
if h < 32 or w < 64:
|
||||
return image.copy()
|
||||
maps = [c for c in (self._fixed_alpha_map(image), self._aligned_alpha_map(image)) if c is not None]
|
||||
if not maps:
|
||||
return image.copy()
|
||||
|
||||
@@ -127,6 +127,17 @@ class GeminiEngine:
|
||||
This is a Python port of the GeminiWatermarkTool C++ engine.
|
||||
"""
|
||||
|
||||
# Footprint pixels with alpha at/above this are the sparkle body; below it the
|
||||
# mark barely affects the pixel, so those are excluded from both the
|
||||
# over-subtraction test and the inpaint mask.
|
||||
_FOOTPRINT_ALPHA = 0.1
|
||||
# If more than this fraction of footprint pixels over-subtract (numerator < 0),
|
||||
# the fixed alpha does not match this image's sparkle and reverse-alpha would
|
||||
# punch a dark pit -- inpaint instead. demo_banana measures 0.0 (reverse-alpha
|
||||
# kept), the issue #30 dark-grass image measures ~0.61 (inpaint), so the 0.05
|
||||
# gate separates them with a wide margin.
|
||||
_OVERSUB_FOOTPRINT_FRAC = 0.05
|
||||
|
||||
def __init__(self, logo_value: float = 255.0) -> None:
|
||||
"""Initialize the engine with embedded alpha maps.
|
||||
|
||||
@@ -190,8 +201,10 @@ class GeminiEngine:
|
||||
# Use large alpha template (96x96) as the high-quality source for downscaling
|
||||
source_alpha = self._alpha_large
|
||||
|
||||
# Dynamically search bottom-right corner (search up to 256x256 region)
|
||||
search_size = int(min(min(w, h), 256))
|
||||
# Dynamically search bottom-right corner. 512 covers up to 512px from the
|
||||
# corner -- enough for known Gemini margin variations (standard: 64+96=160px;
|
||||
# observed variants up to ~300px). 256 was too tight and caused misses.
|
||||
search_size = int(min(min(w, h), 512))
|
||||
sx1 = max(0, w - search_size)
|
||||
sy1 = max(0, h - search_size)
|
||||
|
||||
@@ -363,7 +376,21 @@ class GeminiEngine:
|
||||
detection.confidence,
|
||||
)
|
||||
|
||||
self._reverse_alpha_blend(result, alpha_map, pos)
|
||||
# The captured alpha map (max ~0.51 = a ~50%-opaque white sparkle) is exact
|
||||
# only when the real mark's effective opacity matches it. On a dark/textured
|
||||
# background the sparkle's effective alpha is lower than the capture, so the
|
||||
# fixed-alpha reverse blend OVER-subtracts and drives the footprint to black --
|
||||
# the "white sparkle turns into a black pit" bug (issue #30). The signature is
|
||||
# a large fraction of footprint pixels whose numerator (watermarked - a*logo)
|
||||
# goes negative, which is physically impossible under a brightening overlay.
|
||||
# In that case inpaint the small footprint from the surrounding pixels instead;
|
||||
# on a bright background no pixel over-subtracts, so reverse-alpha is used and
|
||||
# the result is byte-identical to before (verified on demo_banana: 0% vs 61%).
|
||||
if self._reverse_alpha_oversubtracts(result, alpha_map, pos):
|
||||
logger.debug("Reverse-alpha over-subtracts on this background; inpainting sparkle footprint.")
|
||||
self._inpaint_footprint(result, alpha_map, pos)
|
||||
else:
|
||||
self._reverse_alpha_blend(result, alpha_map, pos)
|
||||
return result
|
||||
|
||||
def remove_watermark_custom(
|
||||
@@ -397,6 +424,84 @@ class GeminiEngine:
|
||||
self._reverse_alpha_blend(result, alpha, (x, y))
|
||||
return result
|
||||
|
||||
def _footprint_indices(
|
||||
self,
|
||||
alpha_map: NDArray[Any],
|
||||
position: tuple[int, int],
|
||||
image_shape: tuple[int, ...],
|
||||
) -> tuple[NDArray[Any], tuple[int, int, int, int]] | None:
|
||||
"""Return (alpha_roi, (y1, y2, x1, x2)) for the placed footprint, or None.
|
||||
|
||||
Shared by the over-subtraction test and the inpaint mask so both operate on
|
||||
exactly the same clipped, in-bounds region.
|
||||
"""
|
||||
x, y = position
|
||||
ah, aw = alpha_map.shape[:2]
|
||||
ih, iw = image_shape[:2]
|
||||
x1, y1 = max(0, x), max(0, y)
|
||||
x2, y2 = min(iw, x + aw), min(ih, y + ah)
|
||||
if x1 >= x2 or y1 >= y2:
|
||||
return None
|
||||
ax1, ay1 = x1 - x, y1 - y
|
||||
alpha_roi = alpha_map[ay1 : ay1 + (y2 - y1), ax1 : ax1 + (x2 - x1)]
|
||||
return alpha_roi, (y1, y2, x1, x2)
|
||||
|
||||
def _reverse_alpha_oversubtracts(
|
||||
self,
|
||||
image: NDArray[Any],
|
||||
alpha_map: NDArray[Any],
|
||||
position: tuple[int, int],
|
||||
) -> bool:
|
||||
"""True when reverse-alpha would drive the footprint dark (issue #30).
|
||||
|
||||
Tests the numerator ``watermarked - alpha*logo`` over the sparkle body: a
|
||||
brightening overlay can never make it negative, so a large negative fraction
|
||||
means the fixed alpha over-estimates this image's opacity.
|
||||
"""
|
||||
placed = self._footprint_indices(alpha_map, position, image.shape)
|
||||
if placed is None:
|
||||
return False
|
||||
alpha_roi, (y1, y2, x1, x2) = placed
|
||||
body = alpha_roi >= self._FOOTPRINT_ALPHA
|
||||
if not bool(body.any()):
|
||||
return False
|
||||
roi = image[y1:y2, x1:x2].astype(np.float32)
|
||||
numerator = roi.mean(axis=2) - np.clip(alpha_roi, 0.0, 0.99) * self.logo_value
|
||||
frac = float((numerator[body] < 0).sum()) / float(body.sum())
|
||||
return frac > self._OVERSUB_FOOTPRINT_FRAC
|
||||
|
||||
def _inpaint_footprint(
|
||||
self,
|
||||
image: NDArray[Any],
|
||||
alpha_map: NDArray[Any],
|
||||
position: tuple[int, int],
|
||||
) -> None:
|
||||
"""Inpaint the sparkle body from surrounding pixels, in-place.
|
||||
|
||||
Fallback for backgrounds where reverse-alpha over-subtracts: a small mask of
|
||||
the footprint (alpha >= threshold, dilated) is reconstructed by cv2 NS inpaint
|
||||
from the continuous surroundings, so the sparkle is replaced by plausible
|
||||
background instead of a black pit.
|
||||
"""
|
||||
placed = self._footprint_indices(alpha_map, position, image.shape)
|
||||
if placed is None:
|
||||
return
|
||||
alpha_roi, (y1, y2, x1, x2) = placed
|
||||
# Inpaint only a padded crop around the footprint, not the whole image: the
|
||||
# mask is zero outside a ~96x96 corner, so inpainting the full (multi-MP)
|
||||
# image would be ~hundreds of times more work for an identical result. The
|
||||
# padding gives cv2 enough surrounding context to reconstruct the sparkle.
|
||||
ih, iw = image.shape[:2]
|
||||
pad = 24
|
||||
cy1, cy2 = max(0, y1 - pad), min(ih, y2 + pad)
|
||||
cx1, cx2 = max(0, x1 - pad), min(iw, x2 + pad)
|
||||
crop = image[cy1:cy2, cx1:cx2]
|
||||
mask = np.zeros(crop.shape[:2], dtype=np.uint8)
|
||||
mask[y1 - cy1 : y2 - cy1, x1 - cx1 : x2 - cx1] = (alpha_roi >= self._FOOTPRINT_ALPHA).astype(np.uint8) * 255
|
||||
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
|
||||
mask = cv2.dilate(mask, kernel, iterations=2)
|
||||
image[cy1:cy2, cx1:cx2] = cv2.inpaint(crop, mask, 6, cv2.INPAINT_NS)
|
||||
|
||||
def _reverse_alpha_blend(
|
||||
self,
|
||||
image: NDArray[Any],
|
||||
@@ -407,22 +512,10 @@ class GeminiEngine:
|
||||
|
||||
Formula: original = (watermarked - a * logo) / (1 - a)
|
||||
"""
|
||||
x, y = position
|
||||
ah, aw = alpha_map.shape[:2]
|
||||
ih, iw = image.shape[:2]
|
||||
|
||||
# Clip to bounds
|
||||
x1 = max(0, x)
|
||||
y1 = max(0, y)
|
||||
x2 = min(iw, x + aw)
|
||||
y2 = min(ih, y + ah)
|
||||
|
||||
if x1 >= x2 or y1 >= y2:
|
||||
placed = self._footprint_indices(alpha_map, position, image.shape)
|
||||
if placed is None:
|
||||
return
|
||||
|
||||
# Get ROIs
|
||||
ax1, ay1 = x1 - x, y1 - y
|
||||
alpha_roi = alpha_map[ay1 : ay1 + (y2 - y1), ax1 : ax1 + (x2 - x1)]
|
||||
alpha_roi, (y1, y2, x1, x2) = placed
|
||||
image_roi = image[y1:y2, x1:x2].astype(np.float32)
|
||||
|
||||
alpha_threshold = 0.002
|
||||
|
||||
@@ -8,7 +8,8 @@ Aggregates every locally-readable signal into a single :class:`ProvenanceReport`
|
||||
- **PNG text / EXIF generation parameters** (Stable Diffusion, ComfyUI, InvokeAI).
|
||||
- **SynthID metadata proxy** -- a C2PA companion from a SynthID-using vendor
|
||||
(Google / OpenAI) implies the invisible pixel watermark.
|
||||
- **Visible Gemini sparkle** (optional; needs cv2/numpy, no GPU).
|
||||
- **Visible marks** (optional; needs cv2/numpy, no GPU): the Gemini sparkle and
|
||||
the ByteDance Doubao 豆包AI生成 / Jimeng 即梦AI text marks.
|
||||
|
||||
Hard limit: a stripped image (re-encoded, screenshotted, social-media upload)
|
||||
loses all metadata, and the SynthID *pixel* watermark is not locally decodable
|
||||
@@ -43,6 +44,8 @@ from remove_ai_watermarks.noai.constants import C2PA_AI_TOOLS, C2PA_ISSUERS
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
from remove_ai_watermarks.watermark_registry import MarkDetection
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
# How much of a non-PNG container to binary-scan for the C2PA issuer.
|
||||
@@ -334,6 +337,46 @@ def _visible_sparkle(image_path: Path) -> float | None:
|
||||
return detect_sparkle_confidence(image_path)
|
||||
|
||||
|
||||
# Visible text marks (registry keys) -> human-readable platform, mirroring the
|
||||
# Gemini-sparkle phrasing. These are the stripped-metadata visual fallback for
|
||||
# the China-served ByteDance generators (normally also caught by the TC260 AIGC
|
||||
# metadata label); the per-engine detection thresholds live in the registry.
|
||||
_VISIBLE_MARK_PLATFORM = {
|
||||
"doubao": "ByteDance Doubao (visible 豆包AI生成 mark detected)",
|
||||
"jimeng": "ByteDance Jimeng / Dreamina (visible 即梦AI mark detected)",
|
||||
}
|
||||
|
||||
|
||||
def _visible_text_marks(image_path: Path) -> list[MarkDetection]:
|
||||
"""Detected visible Doubao/Jimeng marks (registry ``MarkDetection`` list).
|
||||
|
||||
The Gemini sparkle keeps its own ``_visible_sparkle`` path (file-level
|
||||
confidence); these two text marks reuse the registry detectors, which apply
|
||||
each engine's calibrated NCC threshold via ``MarkDetection.detected``.
|
||||
Optional: needs cv2/numpy; returns ``[]`` if the engines/assets are missing
|
||||
or the image can't be read.
|
||||
"""
|
||||
try:
|
||||
from remove_ai_watermarks.image_io import imread
|
||||
from remove_ai_watermarks.watermark_registry import get_mark
|
||||
except Exception as exc: # cv2/engine assets missing
|
||||
log.debug("visible-mark detectors unavailable: %s", exc)
|
||||
return []
|
||||
image = imread(image_path)
|
||||
if image is None:
|
||||
return []
|
||||
detections: list[MarkDetection] = []
|
||||
for key in _VISIBLE_MARK_PLATFORM:
|
||||
try:
|
||||
det = get_mark(key).detect(image)
|
||||
except Exception as exc: # one engine failing must not break identify
|
||||
log.debug("visible-mark %s detector failed: %s", key, exc)
|
||||
continue
|
||||
if det.detected:
|
||||
detections.append(det)
|
||||
return detections
|
||||
|
||||
|
||||
def _invisible_watermark(image_path: Path) -> str | None:
|
||||
"""Open invisible-watermark scheme name (SD/SDXL/FLUX) or None.
|
||||
|
||||
@@ -361,7 +404,8 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
|
||||
Args:
|
||||
image_path: Path to the image (PNG, JPEG, WebP, or ISOBMFF container).
|
||||
check_visible: Also run the visible Gemini-sparkle detector (cv2). Set
|
||||
check_visible: Also run the visible-mark detectors (cv2) -- the Gemini
|
||||
sparkle and the Doubao/Jimeng text marks from the registry. Set
|
||||
False for a pure-metadata, dependency-light scan.
|
||||
check_invisible: Also decode open invisible watermarks (SD/SDXL/FLUX) via
|
||||
the optional imwatermark library. No-op when it is not installed.
|
||||
@@ -580,7 +624,16 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
if platform is None:
|
||||
platform = "Google Gemini family (visible sparkle detected)"
|
||||
|
||||
visible_only = any(s.name == "visible_sparkle" for s in signals) and not ai_from_metadata
|
||||
# ── Visible Doubao / Jimeng text marks (registry; same stripped-metadata
|
||||
# fallback role as the Gemini sparkle above) ─
|
||||
if check_visible:
|
||||
for det in _visible_text_marks(image_path):
|
||||
signals.append(Signal(f"visible_{det.key}", f"NCC confidence {det.confidence:.2f}", "medium"))
|
||||
watermarks.append(f"Visible {det.label} (confidence {det.confidence:.2f})")
|
||||
if platform is None:
|
||||
platform = _VISIBLE_MARK_PLATFORM[det.key]
|
||||
|
||||
visible_only = any(s.name.startswith("visible_") for s in signals) and not ai_from_metadata
|
||||
hf_only = bool(hf_job) and not ai_from_metadata
|
||||
samsung_only = samsung_genai_type is not None and not ai_from_metadata
|
||||
|
||||
|
||||
@@ -19,8 +19,6 @@ import warnings
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from remove_ai_watermarks.noai.watermark_profiles import DEFAULT_STRENGTH
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
@@ -72,9 +70,10 @@ class InvisibleEngine:
|
||||
to break watermark patterns, and reconstructs via reverse diffusion.
|
||||
"""
|
||||
|
||||
# SDXL base is the default since May 2026: empirically defeats SynthID v2
|
||||
# at strength=0.10 / steps=50 / native ~1024px. See CLAUDE.md "Known
|
||||
# limitations" for the regression evidence ruling out SD-1.5 pipelines.
|
||||
# SDXL base is the default since May 2026; the current Google SynthID is
|
||||
# removed at strength ~0.30 / steps=50 / native res (oracle-verified, n=3 fresh
|
||||
# Gemini -- 0.10/0.15/0.2 still detected). See CLAUDE.md "Known limitations" for
|
||||
# the strength study and the regression evidence ruling out SD-1.5 pipelines.
|
||||
DEFAULT_MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
CTRLREGEN_MODEL_ID = "yepengliu/ctrlregen"
|
||||
|
||||
@@ -126,16 +125,18 @@ class InvisibleEngine:
|
||||
guidance_scale: float | None = None,
|
||||
seed: int | None = None,
|
||||
humanize: float = 0.0,
|
||||
protect_faces: bool = True,
|
||||
protect_text: bool = True,
|
||||
protect_faces: bool = False,
|
||||
protect_text: bool = False,
|
||||
max_resolution: int = 0,
|
||||
vendor: str | None = None,
|
||||
) -> Path:
|
||||
"""Remove invisible watermark from an image.
|
||||
|
||||
Args:
|
||||
image_path: Path to the watermarked image.
|
||||
output_path: Output path (None = overwrite source).
|
||||
strength: Denoising strength (0.0-1.0). None -> DEFAULT_STRENGTH (0.10).
|
||||
strength: Denoising strength (0.0-1.0). None -> profile default
|
||||
(0.10 for SDXL, 1.0 clean-noise for ctrlregen).
|
||||
steps: Number of denoising steps.
|
||||
guidance_scale: Classifier-free guidance scale.
|
||||
seed: Random seed for reproducibility.
|
||||
@@ -217,6 +218,7 @@ class InvisibleEngine:
|
||||
guidance_scale=guidance_scale,
|
||||
seed=seed,
|
||||
protect_text=protect_text,
|
||||
vendor=vendor,
|
||||
)
|
||||
|
||||
# Optional: Face restoration & Humanizer (Phase 2 - Post-processing)
|
||||
@@ -281,7 +283,7 @@ class InvisibleEngine:
|
||||
self,
|
||||
input_dir: Path,
|
||||
output_dir: Path,
|
||||
strength: float = DEFAULT_STRENGTH,
|
||||
strength: float | None = None,
|
||||
steps: int = 50,
|
||||
) -> list[Path]:
|
||||
"""Remove invisible watermarks from all images in a directory."""
|
||||
|
||||
@@ -238,6 +238,13 @@ class JimengEngine:
|
||||
"""
|
||||
h, w = image.shape[:2]
|
||||
x, y, bw, bh = loc.bbox
|
||||
# A degenerate ROI (a sliver from an extremely wide/short image) cannot hold
|
||||
# the mark and would feed cv2's GaussianBlur/morphology a ~1-px-tall array,
|
||||
# which can fault the native code on some platforms (observed: a Windows
|
||||
# access violation via the always-align removal's residual `detect`). Skip
|
||||
# the cv2 pipeline and return an empty mask there.
|
||||
if bh < 16 or bw < 16:
|
||||
return np.zeros((h, w), np.uint8)
|
||||
# Normalize the ROI to 3-channel BGR: a 2D grayscale or 4-channel BGRA
|
||||
# input would otherwise break the axis=2 channel reductions below.
|
||||
roi = image[y : y + bh, x : x + bw]
|
||||
@@ -364,6 +371,13 @@ class JimengEngine:
|
||||
image = cv2.cvtColor(image, cv2.COLOR_GRAY2BGR)
|
||||
elif image.shape[2] == 4:
|
||||
image = cv2.cvtColor(image, cv2.COLOR_BGRA2BGR)
|
||||
# An image too small to hold the mark would make the geometry boxes
|
||||
# degenerate and feed cv2.resize a ~1-px-tall target / GaussianBlur a sliver
|
||||
# ROI, which faults natively on Windows (access violation / "Unknown C++
|
||||
# exception"). No real watermarked image is this small; skip cv2 entirely.
|
||||
h, w = image.shape[:2]
|
||||
if h < 32 or w < 64:
|
||||
return image.copy()
|
||||
# Always try fixed geometry AND the NCC-aligned placement and keep
|
||||
# whichever leaves the least residual mark (re-detect confidence on the
|
||||
# bare reverse-alpha). Unlike Doubao's deterministic overlay, Jimeng jitters
|
||||
|
||||
@@ -1,9 +1,36 @@
|
||||
"""Vendored noai-watermark code for invisible watermark removal.
|
||||
|
||||
Original: https://github.com/mertizci/noai-watermark (MIT License)
|
||||
|
||||
The public API (``WatermarkRemover`` / ``remove_watermark`` / ``remove_ai_metadata``)
|
||||
is exposed **lazily** via PEP 562 ``__getattr__``: importing a light submodule
|
||||
(e.g. ``noai.c2pa`` / ``noai.constants`` from ``identify``) must NOT eagerly pull
|
||||
``watermark_remover``, which imports torch + diffusers at module top. Keeping this
|
||||
lazy is what lets ``import remove_ai_watermarks.identify`` stay cheap (~36 MB, no
|
||||
torch) even in a full install where the ``gpu``/``detect`` extras are present --
|
||||
otherwise the mere presence of torch in the env inflated identify to ~420 MB and
|
||||
risked OOM on a 512 MB host.
|
||||
"""
|
||||
|
||||
from remove_ai_watermarks.noai.cleaner import remove_ai_metadata
|
||||
from remove_ai_watermarks.noai.watermark_remover import WatermarkRemover, remove_watermark
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from remove_ai_watermarks.noai.cleaner import remove_ai_metadata
|
||||
from remove_ai_watermarks.noai.watermark_remover import WatermarkRemover, remove_watermark
|
||||
|
||||
__all__ = ["WatermarkRemover", "remove_ai_metadata", "remove_watermark"]
|
||||
|
||||
|
||||
def __getattr__(name: str) -> object:
|
||||
"""Resolve the public API on first access (PEP 562), not at package import."""
|
||||
if name == "remove_ai_metadata":
|
||||
from remove_ai_watermarks.noai.cleaner import remove_ai_metadata
|
||||
|
||||
return remove_ai_metadata
|
||||
if name in ("WatermarkRemover", "remove_watermark"):
|
||||
from remove_ai_watermarks.noai import watermark_remover
|
||||
|
||||
return getattr(watermark_remover, name)
|
||||
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
|
||||
@@ -5,21 +5,105 @@ Pure configuration and lookup functions with no ML dependencies.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
|
||||
DEFAULT_MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
CTRLREGEN_MODEL_ID = "yepengliu/ctrlregen"
|
||||
|
||||
# Single default denoising strength for the SDXL img2img scrub, overridable from
|
||||
# the CLI (`--strength`). Raised from the old 0.04/0.05 because that no longer
|
||||
# removes the CURRENT Google SynthID (Nano Banana / Gemini 3): verified 2026-05-30
|
||||
# via the Gemini "Verify with SynthID" oracle on a real image -- 0.05 still
|
||||
# detected, 0.10 not detected (OpenAI's SynthID was already cleared at 0.05). 0.10
|
||||
# keeps the visible change modest while removing both. CAVEAT: confirmed on n=1
|
||||
# Google + n=1 OpenAI image; broad oracle validation across the corpus is pending
|
||||
# (different images may need a different strength). At this higher strength small
|
||||
# text deforms more -- which is exactly why text protection (`_run_region_hires`)
|
||||
# runs by default. (Fixed LOW/MEDIUM/HIGH presets were removed -- unused; the one
|
||||
# knob is this default plus the per-call `--strength` override.)
|
||||
DEFAULT_STRENGTH = 0.10
|
||||
# Vendor-adaptive default denoising strength for the SDXL img2img scrub, overridable
|
||||
# from the CLI (`--strength`). The right strength depends on which vendor's SynthID is
|
||||
# present, detected from the C2PA issuer (metadata.synthid_source). Oracle-verified
|
||||
# controlled study (2026-06-01, clean v0.8.6 with protect_text/faces OFF, per-image
|
||||
# openai.com/verify or Gemini-app verdict; see docs/synthid.md section 2.2):
|
||||
# - OpenAI gpt-image: removed at 0.05 across 1024-1600 (n=4), resolution-independent.
|
||||
# OPENAI_STRENGTH 0.10 = the 0.05 floor plus a 2x margin (keeps quality high).
|
||||
# - Google Gemini: removed at 0.15 on the capped-1536 path (n=4); 0.05/0.10 do NOT
|
||||
# clear. GEMINI_STRENGTH 0.15. CAVEAT: 0.15 was validated only on
|
||||
# `--max-resolution 1536`; native 2816 (the default path) was not locally
|
||||
# measurable (OOM on Apple Silicon) and may need more -- pending GPU validation on
|
||||
# the raiw.cc backend. If a native large Gemini still verifies positive at 0.15,
|
||||
# raise `--strength`.
|
||||
# - Unknown vendor (metadata stripped, or non-OpenAI/Google C2PA): UNKNOWN_STRENGTH
|
||||
# 0.15, the safe middle that clears both vendors at the tested resolutions.
|
||||
# The dominant factor is VENDOR, not resolution: Google's SynthID is ~3x more robust
|
||||
# than OpenAI's. The earlier single 0.30 default (and the "resolution dependence" lore)
|
||||
# came from contaminated tests run with protect_text ON -- see docs/synthid.md 2.2.
|
||||
OPENAI_STRENGTH = 0.10
|
||||
GEMINI_STRENGTH = 0.15
|
||||
UNKNOWN_STRENGTH = 0.15
|
||||
# Backwards-compatible alias: the vendor-unknown default (what a caller gets without a
|
||||
# detected vendor). Kept as DEFAULT_STRENGTH for existing references.
|
||||
DEFAULT_STRENGTH = UNKNOWN_STRENGTH
|
||||
|
||||
# Detected-vendor -> default strength. Vendor strings come from `vendor_for_strength`.
|
||||
_VENDOR_STRENGTH = {"openai": OPENAI_STRENGTH, "google": GEMINI_STRENGTH}
|
||||
|
||||
# CtrlRegen removes watermarks by regenerating from (near) clean Gaussian noise,
|
||||
# NOT by the light-touch partial-noise img2img the SDXL default uses. The research
|
||||
# is explicit (CtrlRegen, ICLR 2025, arXiv:2410.05470): partial-noise regeneration
|
||||
# "struggles with high-perturbation watermarks" because a small noise step "retains"
|
||||
# watermark information that diffuses back into the output; the fix is to start from
|
||||
# clean noise. With the StableDiffusionControlNetImg2ImgPipeline that maps to a high
|
||||
# strength (~1.0 = full noise at the first timestep, structure held by the canny
|
||||
# ControlNet + DINOv2 IP-Adapter, not by the watermarked latent). So the ctrlregen
|
||||
# profile must NOT inherit the SDXL default (`DEFAULT_STRENGTH`, a partial-noise
|
||||
# value) -- at that low strength it loads ControlNet + DINOv2-giant and then barely
|
||||
# changes the image (a no-op for removal). Tunable via
|
||||
# `--strength`; lower it to trade removal strength for fidelity (the CtrlRegen+ regime).
|
||||
#
|
||||
# EXPERIMENTAL -- NOT recommended for production. The same GPU study that set the 0.3
|
||||
# SDXL threshold tested ctrlregen at its clean-noise strength and found it DESTROYS
|
||||
# images: smooth/background regions fill with hallucinated micro-text garbage, and it
|
||||
# is heavy (~8.5 min / ~$0.30 vs ~25 s / ~$0.02 for SDXL on a large image). The pipeline
|
||||
# is effectively binary -- low strength = no-op, high strength = destroys -- with no
|
||||
# usable middle, so the literature's "clean-noise is the lever" (arXiv:2410.05470) did
|
||||
# NOT survive empirical testing on real content. SDXL img2img at ~0.3 is the shippable
|
||||
# path; ctrlregen stays opt-in and flagged experimental.
|
||||
CTRLREGEN_DEFAULT_STRENGTH = 1.0
|
||||
|
||||
|
||||
def resolve_strength(strength: float | None, profile: str, vendor: str | None = None) -> float:
|
||||
"""Resolve the denoising strength, applying the profile/vendor default when unset.
|
||||
|
||||
``None`` means "the user did not pass ``--strength``". ``ctrlregen`` resolves to
|
||||
``CTRLREGEN_DEFAULT_STRENGTH`` (clean-noise regeneration). The SDXL default profile
|
||||
resolves **vendor-adaptively**: ``vendor`` (``"openai"`` / ``"google"`` / None, from
|
||||
``vendor_for_strength``) selects ``OPENAI_STRENGTH`` / ``GEMINI_STRENGTH`` /
|
||||
``UNKNOWN_STRENGTH``. An explicit value always wins (including ``0.0`` -- the check is
|
||||
``is None``, not falsiness). Shared by the CLI (for display) and the engine (for
|
||||
execution) so the two never disagree -- both must pass the SAME ``vendor``.
|
||||
"""
|
||||
if strength is not None:
|
||||
return strength
|
||||
if profile == "ctrlregen":
|
||||
return CTRLREGEN_DEFAULT_STRENGTH
|
||||
return _VENDOR_STRENGTH.get(vendor or "", UNKNOWN_STRENGTH)
|
||||
|
||||
|
||||
def vendor_for_strength(image_path: Path) -> Literal["openai", "google"] | None:
|
||||
"""Detect the SynthID vendor for strength selection: ``"openai"`` / ``"google"`` / None.
|
||||
|
||||
Reads the C2PA SynthID proxy (``metadata.synthid_source``) on the ORIGINAL input,
|
||||
so it must run before any pass that strips metadata. When both issuers appear (a
|
||||
rare multi-sign anomaly) Google wins -- the more-robust watermark -> safer (higher)
|
||||
strength. Returns None when metadata is stripped or the issuer is neither vendor,
|
||||
which maps to ``UNKNOWN_STRENGTH``. Lazy-imports ``metadata`` to keep this module
|
||||
dependency-light.
|
||||
"""
|
||||
try:
|
||||
from remove_ai_watermarks.metadata import synthid_source
|
||||
|
||||
src = (synthid_source(image_path) or "").lower()
|
||||
except Exception: # metadata unreadable -> treat as unknown vendor
|
||||
return None
|
||||
if "google" in src:
|
||||
return "google"
|
||||
if "openai" in src:
|
||||
return "openai"
|
||||
return None
|
||||
|
||||
|
||||
def get_model_id_for_profile(profile: str) -> str:
|
||||
|
||||
@@ -33,6 +33,7 @@ from remove_ai_watermarks.noai.watermark_profiles import (
|
||||
DEFAULT_MODEL_ID,
|
||||
DEFAULT_STRENGTH,
|
||||
detect_model_profile,
|
||||
resolve_strength,
|
||||
)
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
@@ -446,13 +447,15 @@ class WatermarkRemover:
|
||||
guidance_scale: float | None = None,
|
||||
seed: int | None = None,
|
||||
protect_text: bool = True,
|
||||
vendor: str | None = None,
|
||||
) -> Path:
|
||||
"""Remove watermark from an image using regeneration attack.
|
||||
|
||||
Args:
|
||||
image_path: Path to the watermarked image.
|
||||
output_path: Path for the cleaned image. If None, modifies in place.
|
||||
strength: Denoising strength (0.0-1.0).
|
||||
strength: Denoising strength (0.0-1.0). None -> the vendor-adaptive
|
||||
default (see ``vendor``).
|
||||
num_inference_steps: Number of denoising steps.
|
||||
guidance_scale: Classifier-free guidance scale.
|
||||
seed: Random seed for reproducibility.
|
||||
@@ -460,6 +463,11 @@ class WatermarkRemover:
|
||||
Diffusion when any are found (SDXL default profile only). On by
|
||||
default; the detector decides per image, and text-free inputs run
|
||||
the standard pass at no extra cost.
|
||||
vendor: SynthID vendor (``"openai"`` / ``"google"`` / None) used to pick the
|
||||
default strength when ``strength`` is None. Detect it from the ORIGINAL
|
||||
input with ``watermark_profiles.vendor_for_strength`` before processing
|
||||
strips the metadata; the caller passes it down so display and execution
|
||||
agree.
|
||||
|
||||
Returns:
|
||||
Path to the cleaned image.
|
||||
@@ -474,7 +482,7 @@ class WatermarkRemover:
|
||||
if output_path is None:
|
||||
output_path = image_path
|
||||
|
||||
strength = strength or self.DEFAULT_STRENGTH
|
||||
strength = resolve_strength(strength, self.model_profile, vendor)
|
||||
|
||||
if not 0.0 <= strength <= 1.0:
|
||||
raise ValueError(f"Strength must be between 0.0 and 1.0, got {strength}")
|
||||
@@ -894,15 +902,23 @@ class WatermarkRemover:
|
||||
def remove_watermark(
|
||||
image_path: Path,
|
||||
output_path: Path | None = None,
|
||||
strength: float = DEFAULT_STRENGTH,
|
||||
strength: float | None = None,
|
||||
model_id: str | None = None,
|
||||
device: str | None = None,
|
||||
hf_token: str | None = None,
|
||||
) -> Path:
|
||||
"""Convenience function to remove watermark from an image."""
|
||||
"""Convenience function to remove watermark from an image.
|
||||
|
||||
``strength=None`` lets the profile pick its default: vendor-adaptive for SDXL
|
||||
(0.10 OpenAI / 0.15 Google / 0.15 unknown, from the C2PA SynthID proxy on the
|
||||
input), clean-noise 1.0 for ctrlregen. Pass a value to override.
|
||||
"""
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
remover = WatermarkRemover(model_id=model_id, device=device, hf_token=hf_token)
|
||||
return remover.remove_watermark(
|
||||
image_path=image_path,
|
||||
output_path=output_path,
|
||||
strength=strength,
|
||||
vendor=vendor_for_strength(image_path),
|
||||
)
|
||||
@@ -8,7 +8,9 @@ present.
|
||||
**Reverse-alpha based.** A known mark is a fixed semi-transparent overlay, so it
|
||||
is removed by inverting the alpha blend against a captured alpha map
|
||||
(``original = (wm - a*logo)/(1-a)``) -- recovering the true pixels rather than
|
||||
inpainting a guess. Gemini and Doubao recover exactly with no inpaint at native;
|
||||
inpainting a guess. Gemini and Doubao recover exactly with no inpaint at native on
|
||||
bright/flat backgrounds (Gemini falls back to inpainting the sparkle footprint when
|
||||
reverse-alpha would over-subtract on a dark background -- issue #30, see gemini_engine);
|
||||
Jimeng adds a thin residual inpaint over the glyph footprint to clear the outline
|
||||
its per-image render variation leaves behind (still seeded by the reverse-alpha
|
||||
recovery, not a blind inpaint). Detection is consistent with that: each mark is
|
||||
|
||||
+2
-2
@@ -526,8 +526,8 @@ class TestBatchCommand:
|
||||
|
||||
|
||||
class TestGpuHintMarkup:
|
||||
"""The GPU-extra install hint must survive rich markup (the ``[gpu]`` token
|
||||
is otherwise parsed as a style tag and silently dropped)."""
|
||||
"""The GPU-extra install hint must reach the user with the ``[gpu]`` token
|
||||
intact (plain output prints it verbatim, with no markup parsing)."""
|
||||
|
||||
def test_invisible_install_hint_keeps_gpu_extra(self, runner, sample_png):
|
||||
with patch("remove_ai_watermarks.invisible_engine.is_available", return_value=False):
|
||||
|
||||
@@ -230,3 +230,57 @@ class TestInpainting:
|
||||
original = image.copy()
|
||||
self.engine.inpaint_residual(image, (150, 150, 48, 48))
|
||||
np.testing.assert_array_equal(image, original)
|
||||
|
||||
|
||||
class TestOverSubtractionGuard:
|
||||
"""Issue #30: reverse-alpha must not turn the sparkle into a black pit.
|
||||
|
||||
On a dark background the captured alpha over-estimates the real sparkle opacity,
|
||||
so the fixed-alpha reverse blend over-subtracts and drives the footprint to black.
|
||||
The engine detects this and inpaints the footprint instead.
|
||||
"""
|
||||
|
||||
# Composite the mark at ~60% of the captured opacity: the engine's alpha maxes at
|
||||
# ~0.51, real dark-background sparkles sit nearer ~0.31, so 0.6x reproduces the
|
||||
# capture-over-estimates-reality mismatch that triggers the bug.
|
||||
_REALISTIC_ALPHA_SCALE = 0.6
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _setup_engine(self):
|
||||
self.engine = GeminiEngine()
|
||||
|
||||
def _composite_sparkle(self, bg_value: int, size: int = 1400, alpha_scale: float = _REALISTIC_ALPHA_SCALE):
|
||||
"""Build a flat BGR image of ``bg_value`` with the sparkle composited in.
|
||||
|
||||
The mark is composited at a LOWER effective opacity than the engine's captured
|
||||
alpha map (``alpha_scale`` < 1), reproducing the real-world mismatch behind
|
||||
issue #30: the captured alpha (~0.51) over-estimates a real sparkle whose
|
||||
effective opacity is lower, so the fixed-alpha reverse blend over-subtracts.
|
||||
Placed at the configured large-image position so the detector locates it.
|
||||
"""
|
||||
img = np.full((size, size, 3), bg_value, dtype=np.float32)
|
||||
config = get_watermark_config(size, size)
|
||||
x, y = config.get_position(size, size)
|
||||
alpha = self.engine.get_alpha_map(WatermarkSize.LARGE)
|
||||
ah, aw = alpha.shape[:2]
|
||||
a = (alpha * alpha_scale)[:, :, None]
|
||||
roi = img[y : y + ah, x : x + aw]
|
||||
img[y : y + ah, x : x + aw] = a * 255.0 + (1.0 - a) * roi
|
||||
return np.clip(img, 0, 255).astype(np.uint8), (x, y, aw, ah)
|
||||
|
||||
def test_dark_background_does_not_leave_black_pit(self):
|
||||
image, (x, y, w, h) = self._composite_sparkle(bg_value=60)
|
||||
out = self.engine.remove_watermark(image)
|
||||
footprint = out[y : y + h, x : x + w]
|
||||
# The recovered footprint must read like the dark background, not a black hole.
|
||||
assert footprint.min() > 25, f"black pit: min={footprint.min()}"
|
||||
assert abs(float(footprint.mean()) - 60.0) < 25.0
|
||||
|
||||
def test_bright_background_keeps_reverse_alpha(self):
|
||||
"""A bright background does not over-subtract, so reverse-alpha is used."""
|
||||
bright, pos = self._composite_sparkle(bg_value=230)
|
||||
alpha = self.engine.get_interpolated_alpha(pos[2])
|
||||
assert self.engine._reverse_alpha_oversubtracts(bright, alpha, (pos[0], pos[1])) is False
|
||||
dark, dpos = self._composite_sparkle(bg_value=60)
|
||||
dalpha = self.engine.get_interpolated_alpha(dpos[2])
|
||||
assert self.engine._reverse_alpha_oversubtracts(dark, dalpha, (dpos[0], dpos[1])) is True
|
||||
@@ -7,6 +7,8 @@ against the real committed C2PA / IPTC fixtures in data/samples/.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
@@ -371,6 +373,68 @@ class TestIdentifyVisibleSparkle:
|
||||
assert r.confidence == "high"
|
||||
|
||||
|
||||
class TestIdentifyImportIsLight:
|
||||
"""`import identify` must stay torch-free (lazy noai/__init__): the package
|
||||
is deployed on a 512 MB host where eagerly pulling torch/diffusers OOMs."""
|
||||
|
||||
def test_import_identify_does_not_pull_torch(self):
|
||||
# Only meaningful where torch is installed (the gpu/detect extra); on a
|
||||
# core-only CI runner torch can't be in sys.modules anyway.
|
||||
pytest.importorskip("torch")
|
||||
code = "import sys, remove_ai_watermarks.identify; sys.exit(1 if 'torch' in sys.modules else 0)"
|
||||
result = subprocess.run([sys.executable, "-c", code], capture_output=True, check=False) # noqa: S603
|
||||
assert result.returncode == 0, f"import identify pulled torch: {result.stderr.decode()[-500:]}"
|
||||
|
||||
|
||||
# Where the registry-backed Doubao/Jimeng visible detector resolves.
|
||||
_TEXT_MARKS_TARGET = "remove_ai_watermarks.identify._visible_text_marks"
|
||||
|
||||
|
||||
class TestIdentifyVisibleTextMarks:
|
||||
"""The visible Doubao/Jimeng marks are a stripped-metadata visual fallback,
|
||||
parallel to the Gemini sparkle: each lifts an Unknown verdict to medium."""
|
||||
|
||||
@staticmethod
|
||||
def _detection(key: str, label: str, conf: float):
|
||||
from remove_ai_watermarks.watermark_registry import MarkDetection
|
||||
|
||||
return MarkDetection(key, label, "bottom-right", True, conf, (0, 0, 10, 10))
|
||||
|
||||
def test_doubao_promotes_to_medium(self, tmp_clean_png: Path):
|
||||
det = self._detection("doubao", "Doubao 豆包AI生成 text", 0.8)
|
||||
with patch(_SPARKLE_TARGET, return_value=None), patch(_TEXT_MARKS_TARGET, return_value=[det]):
|
||||
r = identify(tmp_clean_png, check_visible=True)
|
||||
assert r.is_ai_generated is True
|
||||
assert r.confidence == "medium"
|
||||
assert r.platform is not None
|
||||
assert "Doubao" in r.platform
|
||||
signal = next(s for s in r.signals if s.name == "visible_doubao")
|
||||
assert signal.confidence == "medium"
|
||||
|
||||
def test_jimeng_promotes_to_medium(self, tmp_clean_png: Path):
|
||||
det = self._detection("jimeng", "Jimeng 即梦AI wordmark", 0.9)
|
||||
with patch(_SPARKLE_TARGET, return_value=None), patch(_TEXT_MARKS_TARGET, return_value=[det]):
|
||||
r = identify(tmp_clean_png, check_visible=True)
|
||||
assert r.is_ai_generated is True
|
||||
assert r.confidence == "medium"
|
||||
assert r.platform is not None
|
||||
assert "Jimeng" in r.platform
|
||||
assert any(s.name == "visible_jimeng" for s in r.signals)
|
||||
|
||||
def test_check_visible_false_skips_text_marks(self, tmp_clean_png: Path):
|
||||
det = self._detection("doubao", "Doubao 豆包AI生成 text", 0.99)
|
||||
with patch(_SPARKLE_TARGET, return_value=None), patch(_TEXT_MARKS_TARGET, return_value=[det]) as mock:
|
||||
r = identify(tmp_clean_png, check_visible=False)
|
||||
mock.assert_not_called()
|
||||
assert not any(s.name == "visible_doubao" for s in r.signals)
|
||||
|
||||
def test_metadata_keeps_high_even_with_text_mark(self, tmp_png_with_ai_metadata: Path):
|
||||
det = self._detection("doubao", "Doubao 豆包AI生成 text", 0.8)
|
||||
with patch(_SPARKLE_TARGET, return_value=None), patch(_TEXT_MARKS_TARGET, return_value=[det]):
|
||||
r = identify(tmp_png_with_ai_metadata, check_visible=True)
|
||||
assert r.confidence == "high"
|
||||
|
||||
|
||||
# ── Caveats and serialization ───────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -6,6 +6,7 @@ code paths work correctly on CPU, MPS (macOS), and CUDA (Linux/Windows).
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
@@ -13,8 +14,14 @@ import pytest
|
||||
from remove_ai_watermarks.noai.progress import is_mps_error
|
||||
from remove_ai_watermarks.noai.utils import get_image_format, is_supported_format
|
||||
from remove_ai_watermarks.noai.watermark_profiles import (
|
||||
CTRLREGEN_DEFAULT_STRENGTH,
|
||||
DEFAULT_STRENGTH,
|
||||
GEMINI_STRENGTH,
|
||||
OPENAI_STRENGTH,
|
||||
UNKNOWN_STRENGTH,
|
||||
detect_model_profile,
|
||||
get_model_id_for_profile,
|
||||
resolve_strength,
|
||||
)
|
||||
from remove_ai_watermarks.noai.watermark_remover import get_device, is_watermark_removal_available
|
||||
|
||||
@@ -121,6 +128,81 @@ class TestModelProfiles:
|
||||
assert detect_model_profile("yepengliu/ctrlregen") == "ctrlregen"
|
||||
|
||||
|
||||
class TestResolveStrength:
|
||||
"""resolve_strength applies the profile/vendor default only when strength is unset."""
|
||||
|
||||
def test_none_default_profile_is_vendor_adaptive(self):
|
||||
# No vendor -> unknown default; OpenAI lower, Google == unknown.
|
||||
assert resolve_strength(None, "default") == UNKNOWN_STRENGTH
|
||||
assert resolve_strength(None, "default", "openai") == OPENAI_STRENGTH
|
||||
assert resolve_strength(None, "default", "google") == GEMINI_STRENGTH
|
||||
assert resolve_strength(None, "default", None) == UNKNOWN_STRENGTH
|
||||
# An unrecognized vendor string falls through to the unknown default.
|
||||
assert resolve_strength(None, "default", "adobe") == UNKNOWN_STRENGTH
|
||||
|
||||
def test_default_strength_alias_is_unknown_vendor_value(self):
|
||||
assert DEFAULT_STRENGTH == UNKNOWN_STRENGTH
|
||||
assert OPENAI_STRENGTH < UNKNOWN_STRENGTH
|
||||
|
||||
def test_none_ctrlregen_uses_clean_noise_default(self):
|
||||
# ctrlregen must NOT inherit the SDXL vendor defaults (that makes it a no-op);
|
||||
# clean-noise regeneration is the lever against robust marks. Vendor is ignored.
|
||||
assert resolve_strength(None, "ctrlregen") == CTRLREGEN_DEFAULT_STRENGTH
|
||||
assert resolve_strength(None, "ctrlregen", "openai") == CTRLREGEN_DEFAULT_STRENGTH
|
||||
assert CTRLREGEN_DEFAULT_STRENGTH > DEFAULT_STRENGTH
|
||||
|
||||
def test_explicit_value_overrides_profile_and_vendor(self):
|
||||
assert resolve_strength(0.3, "default") == 0.3
|
||||
assert resolve_strength(0.3, "default", "openai") == 0.3
|
||||
assert resolve_strength(0.3, "ctrlregen") == 0.3
|
||||
|
||||
def test_explicit_zero_is_respected_not_treated_as_unset(self):
|
||||
# 0.0 is falsy but explicit -- must not fall through to the profile default
|
||||
# (the old `strength or DEFAULT` bug would have). Range validation lives in
|
||||
# remove_watermark, not here.
|
||||
assert resolve_strength(0.0, "ctrlregen") == 0.0
|
||||
assert resolve_strength(0.0, "default", "google") == 0.0
|
||||
|
||||
|
||||
class TestVendorForStrength:
|
||||
"""vendor_for_strength normalizes the C2PA SynthID proxy to openai/google/None."""
|
||||
|
||||
@staticmethod
|
||||
def _patch(value):
|
||||
return patch("remove_ai_watermarks.metadata.synthid_source", return_value=value)
|
||||
|
||||
def test_openai(self):
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
with self._patch("OpenAI"):
|
||||
assert vendor_for_strength(Path("x.png")) == "openai"
|
||||
|
||||
def test_google(self):
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
with self._patch("Google"):
|
||||
assert vendor_for_strength(Path("x.png")) == "google"
|
||||
|
||||
def test_both_issuers_google_wins(self):
|
||||
# The more-robust watermark wins -> safer (higher) strength.
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
with self._patch("OpenAI, Google"):
|
||||
assert vendor_for_strength(Path("x.png")) == "google"
|
||||
|
||||
def test_none_when_no_synthid_source(self):
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
with self._patch(None):
|
||||
assert vendor_for_strength(Path("x.png")) is None
|
||||
|
||||
def test_unreadable_metadata_is_none(self):
|
||||
from remove_ai_watermarks.noai.watermark_profiles import vendor_for_strength
|
||||
|
||||
with patch("remove_ai_watermarks.metadata.synthid_source", side_effect=OSError):
|
||||
assert vendor_for_strength(Path("x.png")) is None
|
||||
|
||||
|
||||
# ── Format utilities ────────────────────────────────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -2865,7 +2865,7 @@ wheels = [
|
||||
|
||||
[[package]]
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.8.1"
|
||||
version = "0.8.8"
|
||||
source = { editable = "." }
|
||||
dependencies = [
|
||||
{ name = "click" },
|
||||
@@ -2874,7 +2874,6 @@ dependencies = [
|
||||
{ name = "piexif" },
|
||||
{ name = "pillow" },
|
||||
{ name = "python-dotenv" },
|
||||
{ name = "rich" },
|
||||
]
|
||||
|
||||
[package.optional-dependencies]
|
||||
@@ -2948,7 +2947,6 @@ requires-dist = [
|
||||
{ name = "pytest-cov", marker = "extra == 'dev'", specifier = ">=4.1.0" },
|
||||
{ name = "python-dotenv", specifier = ">=1.0.0" },
|
||||
{ name = "remove-ai-watermarks", extras = ["gpu", "detect", "trustmark", "lama", "dev"], marker = "extra == 'all'" },
|
||||
{ name = "rich", specifier = ">=13.0.0" },
|
||||
{ name = "ruff", marker = "extra == 'dev'", specifier = ">=0.4.0" },
|
||||
{ name = "safetensors", marker = "extra == 'gpu'" },
|
||||
{ name = "tokenizers", marker = "extra == 'gpu'", specifier = ">=0.22,<0.23" },
|
||||
|
||||
Reference in new issue
Block a user