mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-16 06:20:48 +02:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
96b3653b9e | ||
|
|
9aaa53fe32 | ||
|
|
41e4365cd4 | ||
|
|
888c8c2556 | ||
|
|
f326bab189 | ||
|
|
223cbcf171 |
No files matched your search
@@ -14,8 +14,10 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
## Test and lint
|
||||
|
||||
- `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto
|
||||
- `maintain.sh` may not finish fully green (pre-existing, not per-change): strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.) To gate a change, run `uv run ruff check`, `uv run pyright <changed files>`, `uv run pytest` directly.
|
||||
- **Strict pyright is clean across `src/` (0 errors).** The cv2/torch/diffusers boundary files (`gemini_engine`, `region_eraser`, `doubao_engine`, `face_protector`, `humanizer`, `invisible_engine`, `noai/watermark_remover`, and the whole `noai/ctrlregen/` subpackage) carry a documented per-file `# pyright:` relax pragma (or, for `ctrlregen`, a `tool.pyright.executionEnvironments` entry) that turns off only the unknown-type / untyped-third-party rules — those libs ship no usable types, so strict typing there fights the ecosystem. Pure-logic files stay fully strict; `typings/piexif/__init__.pyi` is a local stub so `metadata.py`/`extractor.py` resolve piexif. Public ndarray-returning signatures on the relaxed engines are still annotated `NDArray[Any]` so strict consumers (`cli.py`) stay clean. When touching a relaxed file, prefer fixing real issues over widening the pragma; keep the pragma scoped to genuinely-untyped boundaries. (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.)
|
||||
- **Full-project `uv run pyright` (no path) OOMs/crashes node on this ML-heavy repo** (emits a `libnode` stack frame, no summary) — a known environment limit, not a code error. Gate with `uv run --extra dev --extra gpu pyright src/` (completes, authoritative) or scope to changed files; also run `uv run ruff check` and `uv run pytest` directly.
|
||||
- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch.
|
||||
- To add a dev tool (pytest/ruff/pyright) into the env, use `uv sync --frozen --extra dev --extra gpu`, **never `uv pip install`** — `uv pip install` re-resolves and rewrites `uv.lock`, which silently bumped `transformers` to a build incompatible with the pinned `diffusers` (`cannot import name 'Qwen3VLForConditionalGeneration'`) and broke every `identify`/metadata import. Recovery: `git checkout uv.lock && uv sync --frozen --extra gpu --extra dev`. The `gpu` extra holds `diffusers`/`transformers`/`torch`, so a bare `uv sync` (no extras) removes them and `noai/__init__` (eager pipeline import) then fails. `maintain.sh`'s `uv sync --all-extras` also pulls the heavy `trustmark`/`lama` wheels (pytorch-lightning, onnxruntime) — fine on a good connection, but on flaky DNS sync only `--extra gpu --extra dev` and run the lint/test steps by hand.
|
||||
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `mj-*` = Midjourney IPTC, `doubao-1.png` = ByteDance Doubao with the China TC260 `<TC260:AIGC>` XMP label **and** a visible "豆包AI生成" text mark bottom-right; `grok-1.jpg` = xAI Grok with its EXIF-only `Signature:` blob + UUID `Artist` and no C2PA/SynthID/IPTC); synthetic byte blobs cover the JPEG/ISOBMFF format paths. The "non-AI / clean photo" control is no longer in `data/samples/` -- the `clean_photo` conftest fixture serves a verified-negative image from the corpus `neg/` set (skips if the corpus is absent).
|
||||
- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/`. The labeled `images/` (`pos/` `neg/` `cleaned/`) are **committed** (public repo -- review every image for private content before adding; `manifest.csv` is kept in sync with the files on disk, one row per tracked image); only the synthetic `refs/` calibration fills are gitignored. See its README for the collection protocol and verification oracles.
|
||||
|
||||
@@ -30,7 +32,7 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
- `noai/c2pa.py` — PNG chunk parser; use `extract_c2pa_chunk(path)` to get raw caBX payload, `has_c2pa_metadata(path)` to detect. Do not reimplement chunk parsing. `extract_c2pa_info(path)` sets `synthid_watermark`/`synthid_vendors` when the manifest is signed by a SynthID-using vendor, and `soft_binding`/`soft_binding_vendors` when a `c2pa.soft-binding` `alg` names a forensic-watermark vendor (`soft_binding_vendors_in(buffer)` is the shared byte-scan, used by both the PNG parser and the non-PNG binary path).
|
||||
- `noai/constants.py` — PNG_SIGNATURE, C2PA_CHUNK_TYPE, C2PA_SIGNATURES, C2PA_ISSUERS, `SYNTHID_C2PA_ISSUERS` (issuers that pair SynthID with C2PA: Google, OpenAI), and `C2PA_SOFT_BINDINGS` (soft-binding `alg` prefix → forensic-watermark vendor: Adobe TrustMark, Digimarc, Imatag, Steg.AI, Microsoft, ...). Add a new issuer/binding here, not inline.
|
||||
- `metadata.py` — `scan_head(path, size=1MB)` is the shared input for every C2PA/AIGC/IPTC byte scan: first `size` bytes plus, for ISOBMFF, the late provenance-box payloads from `isobmff.scan_c2pa_region` (catches a manifest after a large `mdat`); behavior-neutral (`f.read(size)`) for non-ISOBMFF. Use it instead of `open().read(1MB)` for any new marker scan. `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. Both `get_ai_metadata` and `has_ai_metadata` guard the PIL open with `except Exception` (HEIC/unknown formats raise non-OSError) and fall through to the binary scan. `xai_signature(path)` detects xAI/Grok's EXIF-only scheme (`ImageDescription` = `Signature: <base64>` + UUID `Artist`); it feeds `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify`. `iptc_ai_system(path)` detects the IPTC Photo Metadata 2025.1 AI-disclosure XMP properties (`IPTC_AI_FIELD_MARKERS` = `AISystemUsed`/`AISystemVersionUsed`/`AIPromptInformation`/`AIPromptWriterName`) and returns the `AISystemUsed` generator name (or `"fields present"`). `remove_ai_metadata` routes **ISOBMFF video** (`.mp4`/`.mov`/`.m4v`) through the same `isobmff.strip_c2pa_boxes` as AVIF/HEIF (MP4 is ISOBMFF), and `_scrub_ai_exif` removes the xAI signature + AI-generator EXIF tags on JPEG output.
|
||||
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, C2PA soft-binding forensic-watermark vendor, IPTC "Made with AI" + IPTC 2025.1 `AISystemUsed`, embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, visible Gemini sparkle, open invisible watermark, Adobe TrustMark via `trustmark_detector`) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. **C2PA platform attribution is device-token-first, issuer-scan fallback** (`_device_platform` scans manifest bytes for `_DEVICE_C2PA_PLATFORM` tokens, then `_attribute_platform`/`_ISSUER_PLATFORM`). **Why, verified on real signed files 2026-05-26:** the old issuer-only byte-scan matched ANY issuer substring anywhere, so multi-entity manifests mis-attributed -- Leica→"Truepic" (a signing authority in the trust chain), Nikon→"Adobe Firefly" (XMP-toolkit "Adobe" + the sample's "Adobe_MAX" name), Pixel→"Google (Gemini)" ("Google LLC" cert org), Truepic→"Google". A distinctive device token wins instead. **Token distinctiveness is load-bearing:** bare `b"Truepic"` mis-fires (it appears in unrelated trust chains -- it mis-attributed the OpenAI `chatgpt-1.png` fixture), so the token is the specific `b"Truepic_Lens"` from the Lens SDK claim generator; likewise `b"Pixel Camera"` (cert CN) not bare `b"Pixel"`. `_DEVICE_C2PA_PLATFORM` lists ONLY tokens **verified against a real C2PA file**: Leica (`lc_c2pa`/`Leica Camera`), Nikon (`NIKON`), Pixel (`Pixel Camera` -- from a real Pixel 10 Pro file attached to c2pa-rs issue #1609/#1554), Sony (`sony.sig`/`sony.cert` -- Sony's own C2PA assertion namespace, verified on a real Sony PXW-Z300 file; NOT bare "Sony" which is a common EXIF Make), Truepic (`Truepic_Lens`). Canon/Samsung/Bria have **no public direct-download C2PA sample** (checked exhaustively: GitHub issue/PR attachments, contentcredentials gallery, HF datasets -- all upload-to-verify or token-gated; Canon's only public file was a self-signed hobbyist CR3, not factory), so they stay unmapped until a real file is captured (same fixture discipline as Grok/Doubao). The Sony sample is video (MP4) -- our ISOBMFF C2PA path detects it; Sony Alpha stills likely share the `sony.*` namespace but are not separately verified. Camera C2PA marks capture authenticity, not AI (Pixel carries `computationalCapture`, not `trainedAlgorithmicMedia`), so these never set `is_ai` -- that stays driven by digital-source-type. `c2pa.cbor_text_after` (now public) is best-effort for the `generator` detail string only and can be None when the manifest keys it `claim_generator_info` (Pixel). **Issuer→generator mapping is `is_ai`-gated** (`_attribute_platform(issuers, is_ai=c2pa_is_ai)`): a specific AI-generator platform is named only when the digital-source-type is `trainedAlgorithmicMedia`; on a non-AI source an issuer substring is treated as incidental (an "Adobe XMP" toolkit string in an *unmapped* Canon/Sony capture would otherwise mislabel it "Adobe Firefly"), so it degrades to the neutral "C2PA signer: X" label. Real Firefly/OpenAI/Google output carries the AI source-type, so it is unaffected (verified: chatgpt-1.png→OpenAI, firefly-1.png→Adobe Firefly still attribute). `_attribute_platform` defaults `is_ai=True` so the mapping stays unit-testable in isolation. Add device tokens to `_DEVICE_C2PA_PLATFORM`, generator/issuer platforms to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read. **Integrity-clash detection** (`_integrity_clashes`, surfaced as `ProvenanceReport.integrity_clashes`, printed in red by `identify` and serialized to `--json`): contradictions between independent generator stamps are a laundering/spoofing tell. Two rules: (1) two or more distinct AI-origin vendors named by independent signals (e.g. C2PA OpenAI + EXIF `Make="Ideogram AI"`), and (2) a camera-capture C2PA device (`_DEVICE_C2PA_PLATFORM`) coexisting with any AI-generation marker. Vendor normalizatLine truncated
|
||||
- `identify.py` — `identify(path)` aggregates every locally-readable signal (C2PA issuer→platform, C2PA soft-binding forensic-watermark vendor, IPTC "Made with AI" + IPTC 2025.1 `AISystemUsed`, embedded SD/ComfyUI params, SynthID proxy, xAI/Grok EXIF signature via `metadata.xai_signature`, the China TC260 AIGC label via `metadata.aigc_label`, the HuggingFace `hf-job-id` job marker via `metadata.huggingface_job`, visible Gemini sparkle, open invisible watermark, Adobe TrustMark via `trustmark_detector`) into one `ProvenanceReport`. `is_ai_generated` is True or None (never asserted False — stripped metadata is not proof of clean origin). The `hf_job` and visible-sparkle signals are **medium** confidence: each lifts an otherwise-Unknown verdict to a tentative AI (`hf_only` / `visible_only`, parallel branches) but is excluded from the high-confidence `ai_from_metadata` set, so neither overrides a hard metadata signal. Visible-sparkle is promoted only at confidence ≥ `_SPARKLE_THRESHOLD` (0.5; corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49). The cv2 dependency lives in `gemini_engine.detect_sparkle_confidence`, not here. **C2PA platform attribution is device-token-first, issuer-scan fallback** (`_device_platform` scans manifest bytes for `_DEVICE_C2PA_PLATFORM` tokens, then `_attribute_platform`/`_ISSUER_PLATFORM`). **Why, verified on real signed files 2026-05-26:** the old issuer-only byte-scan matched ANY issuer substring anywhere, so multi-entity manifests mis-attributed -- Leica→"Truepic" (a signing authority in the trust chain), Nikon→"Adobe Firefly" (XMP-toolkit "Adobe" + the sample's "Adobe_MAX" name), Pixel→"Google (Gemini)" ("Google LLC" cert org), Truepic→"Google". A distinctive device token wins instead. **Token distinctiveness is load-bearing:** bare `b"Truepic"` mis-fires (it appears in unrelated trust chains -- it mis-attributed the OpenAI `chatgpt-1.png` fixture), so the token is the specific `b"Truepic_Lens"` from the Lens SDK claim generator; likewise `b"Pixel Camera"` (cert CN) not bare `b"Pixel"`. `_DEVICE_C2PA_PLATFORM` lists ONLY tokens **verified against a real C2PA file**: Leica (`lc_c2pa`/`Leica Camera`), Nikon (`NIKON`), Pixel (`Pixel Camera` -- from a real Pixel 10 Pro file attached to c2pa-rs issue #1609/#1554), Sony (`sony.sig`/`sony.cert` -- Sony's own C2PA assertion namespace, verified on a real Sony PXW-Z300 file; NOT bare "Sony" which is a common EXIF Make), Truepic (`Truepic_Lens`). Canon/Samsung/Bria have **no public direct-download C2PA sample** (checked exhaustively: GitHub issue/PR attachments, contentcredentials gallery, HF datasets -- all upload-to-verify or token-gated; Canon's only public file was a self-signed hobbyist CR3, not factory), so they stay unmapped until a real file is captured (same fixture discipline as Grok/Doubao). The Sony sample is video (MP4) -- our ISOBMFF C2PA path detects it; Sony Alpha stills likely share the `sony.*` namespace but are not separately verified. Camera C2PA marks capture authenticity, not AI (Pixel carries `computationalCapture`, not `trainedAlgorithmicMedia`), so these never set `is_ai` -- that stays driven by digital-source-type. `c2pa.cbor_text_after` (now public) is best-effort for the `generator` detail string only and can be None when the manifest keys it `claim_generator_info` (Pixel). **Issuer→generator mapping is `is_ai`-gated** (`_attribute_platform(issuers, is_ai=c2pa_is_ai)`): a specific AI-generator platform is named only when the digital-source-type is `trainedAlgorithmicMedia`; on a non-AI source an issuer substring is treated as incidental (an "Adobe XMP" toolkit string in an *unmapped* Canon/Sony capture would otherwise mislabel it "Adobe Firefly"), so it degrades to the neutral "C2PA signer: X" label. Real Firefly/OpenAI/Google output carries the AI source-type, so it is unaffected (verified: chatgpt-1.png→OpenAI, firefly-1.png→Adobe Firefly still attribute). `_attribute_platform` defaults `is_ai=True` so the mapping stays unit-testable in isolation. Add device tokens to `_DEVICE_C2PA_PLATFORM`, generator/issuer platforms to `_ISSUER_PLATFORM`, not inline. For non-PNG containers (JPEG/WebP/AVIF/HEIF/JXL) the caBX parser returns nothing, so issuer (`_issuers_in`) and generator (`_ai_tools_in`, reusing `C2PA_AI_TOOLS`) are recovered by binary-scanning the first MB. EXIF `Software` / `Make` / `Artist` / `ImageDescription` and XMP `CreatorTool` generator tags are read by `metadata.exif_generator` (PIL+piexif for any format PIL opens incl. AVIF, plus a container-agnostic XMP raw-byte scan that also covers HEIF/JXL), matched against `AI_GENERATOR_TOKENS` so ordinary editors (plain "Adobe Photoshop") and real-camera `Make` ("Apple"/"Canon") are not flagged. **Ideogram tags its output with EXIF `Make="Ideogram AI"`** (verified on a real download 2026-05-24) — that's why `Make` is read. **Integrity-clash detection** (`_integrity_clashes`, surfaced as `ProvenanceReport.integrLine truncated
|
||||
- `gemini_engine.py` — visible Gemini-sparkle remover/detector (cv2/numpy, no GPU). `detect_sparkle_confidence(path)` is the file-level entry point used by `identify.py`.
|
||||
- `doubao_engine.py` — visible Doubao "豆包AI生成" remover/detector (cv2/numpy, no GPU). `DoubaoEngine.locate` anchors a bottom-right box by **geometry** (mark scales with image WIDTH, fractions in module constants; no bundled template), `extract_mask` pulls the light low-saturation glyphs with a **polarity-aware white top-hat** (brighter-than-blurred-local-bg, so white-paper documents are left untouched instead of smeared), `detect` thresholds glyph coverage (`DETECT_MIN_COVERAGE` 0.16 separates real marks ≥0.20 from corner noise, which stays ≤0.06 on large images but can spike to ~0.15 on tiny ones), `remove_watermark` inpaints (cv2 Telea/NS) and **bails when coverage > `MAX_INPAINT_COVERAGE` 0.50** (dense-text background → would smear). Wired into `visible --mark` via `cli._run_doubao_if_selected`. **Logo is near-white (~253), not the gray some third-party tools assume.** Best on photo/illustration backgrounds; high-contrast edges leave faint residue (cv2-inpaint limit). Clean per-pixel reverse-alpha (Gemini-style) is the future upgrade but needs a captured/distilled alpha map — see below.
|
||||
- `region_eraser.py` — universal region eraser (`erase` CLI). `erase(image, boxes=|mask=, backend=)`: `boxes_to_mask` → `cv2.inpaint` (`cv2` backend, default, no deps) or big-LaMa via onnxruntime (`lama` backend, extra `lama`, `Carve/LaMa-ONNX` Apache-2.0 model downloaded on first use, never bundled). `erase_lama` crops a padded region around the mask, runs LaMa at its fixed 512² input, pastes only masked pixels back (untouched areas stay pixel-exact). Lazy `_get_lama_session` singleton; `lama_available()` guards the optional import. **LaMa-ONNX costs ~3.5-4 GB peak RAM and ~5-6 s/call on CPU** (FFC working set, not arena — `enable_cpu_mem_arena=False` does not help), so it does NOT fit a minimal droplet; the cv2 backend (tens of MB, ~30 ms) does. LaMa quality at low RAM = serverless/GPU, mirroring how raiw.cc offloads SDXL to fal.
|
||||
@@ -51,7 +53,8 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
|
||||
- **C2PA / IPTC (covered by the issuer/marker scan):** OpenAI, Google, Adobe Firefly, Microsoft (Designer + **Bing Image Creator** — collected 2026-05-24; Bing now runs Microsoft's own **MAI-Image** model, signs C2PA as "Microsoft", NOT OpenAI/DALL-E), and **Stability AI** (collected from Brand Studio / DreamStudio successor; signs C2PA as "Stability AI Ltd", no SynthID, no imwatermark on its current Stable Image model — issuer added to `C2PA_ISSUERS`). Still unsampled: Canva (its downloads are re-encoded design *exports* that strip C2PA, so a Canva "positive" is inconclusive — skipped), Getty, Shutterstock. Midjourney embeds NO C2PA and no invisible watermark (our `mj-*` sample carried only the IPTC tag).
|
||||
- **EXIF/XMP generator tag (caught by `exif_generator`):** **Ideogram** writes EXIF `Make="Ideogram AI"` (collected 2026-05-24 — no C2PA, no SynthID, no imwatermark; the Make tag is the only signal).
|
||||
- **xAI / Grok — its own EXIF signature scheme, NOT C2PA (DETECTED by `metadata.xai_signature`, built 2026-05-26).** Grok JPEG downloads (Aurora model) carry **no C2PA, no XMP, no SynthID, no IPTC** — only EXIF `Artist` = a UUID and EXIF `ImageDescription` = `Signature: <base64>` (a crypto signature, unverifiable locally without xAI's public key). This empirically kills the earlier unverified "xAI signs C2PA as xAI" lead — xAI is not even a C2PA member. `exif_generator` misses it (neither field holds an `AI_GENERATOR_TOKENS` token), so a dedicated detector `xai_signature(path)` matches the pair (`ImageDescription ~ ^Signature: [A-Za-z0-9+/=]{64,}` AND UUID `Artist`); wired into `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify` (signal `xai_signature`, platform "xAI (Grok / Aurora)"). **Format confirmed stable across n=3 genuine generations:** exactly three EXIF tags (`Artist`, `ExifOffset`, `ImageDescription`), `Signature:` prefix constant, base64 payload 300-1004 chars. Two capture facts: (a) the `Artist` UUID **equals the public image id** in the asset URL (`https://imagine-public.x.ai/imagine-public/images/<uuid>.jpg`), so it is NOT a private per-user secret — only the `Signature` blob is; (b) the Grok web-UI image is a re-encoded **WebP with no signature** — the EXIF survives only in the *original* JPEG (download button or that public tokenless URL), which is why screenshots / re-encodes are metadata-stripped. A real fixture `data/samples/grok-1.jpg` plus **synthetic** JPEG fixtures (fake UUID + fake `Signature:` blob) cover the detector; never add a real Grok image carrying private content (the repo is public). **Stripped on removal too:** `remove_ai_metadata` now calls `_scrub_ai_exif` on the JPEG EXIF, which deletes the xAI Signature+UUID-Artist pair **and** any `Software`/`Make`/`Artist`/`ImageDescription` tag holding an `AI_GENERATOR_TOKENS` token (so Ideogram's `Make="Ideogram AI"` is scrubbed too), while keeping genuine camera/editor EXIF. The shared `_is_xai_signature_pair` helper (module-level compiled regexes) is the single source of truth for the pattern, used by both `xai_signature` and `_scrub_ai_exif`. (AVIF/HEIF/JXL still strip only C2PA boxes via `isobmff`, not EXIF — unchanged.)
|
||||
- **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `<TC260:AIGC>{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). **Doubao** (ByteDance) uses it (verified on the real #13 sample 2026-05-25; `ContentProducer` `001191110102MACQD9K64010000`, no C2PA/SynthID/imwatermark — the XMP block is the only signal; GitHub attachment upload did NOT strip it). The same standard is mandatory for Jimeng/Kling/Qwen/Ernie etc., so the one marker covers the whole China-AIGC-labeled ecosystem. `aigc_label` json-decodes the (HTML-entity-encoded) block; container-agnostic raw-byte scan.
|
||||
- **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `<TC260:AIGC>{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). **Doubao** (ByteDance) uses it (verified on the real #13 sample 2026-05-25; `ContentProducer` `001191110102MACQD9K64010000`, no C2PA/SynthID/imwatermark — the XMP block is the only signal; GitHub attachment upload did NOT strip it). The same standard is mandatory for Jimeng/Kling/Qwen/Ernie etc., so the one marker covers the whole China-AIGC-labeled ecosystem. `aigc_label` reads **two serializations** through a shared `_parse` helper: the HTML-entity-encoded XMP `<TC260:AIGC>` block (container-agnostic raw-byte scan, any JSON object accepted) **and** a raw-JSON PNG `AIGC` tEXt chunk — Doubao also writes the label this way, with no namespaced marker at all (confirmed on the corpus 2026-05-28, `ContentProducer="doubao"`). The PNG-chunk path is gated on at least one TC260 field (`_TC260_FIELDS`) so a generic `AIGC` key cannot false-positive. In `identify`, `aigc` fires on the parsed label **or** the `AIGC_MARKERS` byte scan (the latter preserves the laundering-tell case where the JSON payload is truncated).
|
||||
- **HuggingFace-hosted job (caught by `metadata.huggingface_job`, surfaced by `identify` as the `hf_job` signal, MEDIUM confidence):** HuggingFace Jobs / Spaces stamp generated PNGs with an `hf-job-id` tEXt chunk holding the job UUID (3 on the corpus 2026-05-28, no other signal). It marks the *hosting job*, not a model — most commonly diffusion output — so it lifts an Unknown verdict to a tentative AI via `hf_only` (parallel to the visible sparkle) but never overrides a hard metadata signal; `_HF_JOB_CAVEAT` states the limit (job, not model; not proof of AI pixels). Stripped on removal (the PNG save whitelist keeps only `STANDARD_METADATA_KEYS`, so `hf-job-id` and the `AIGC` chunk are both dropped). The exact writer is not authoritatively documented (HF Jobs are generic GPU jobs), hence medium not high.
|
||||
- **No detectable signal on download (correctly reported `unknown`):** **Recraft** (PNG export is a re-encoded design export — strips everything), **Krea hosting FLUX 2** (no imwatermark despite FLUX — the host omits the encoder, same as Stability's hosted SDXL), and Midjourney (embeds nothing). Lesson: the imwatermark detector only fires on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL), not from re-hosts (Krea/Stability) or re-encoded exports (Recraft/Canva).
|
||||
- **Invisible but NOT locally detectable (proprietary, API/oracle only — same wall as SynthID):** Amazon Titan Image Generator + Nova Canvas (Bedrock `DetectGeneratedContent` API), Kakao (new SynthID image adopter, May 2026), NVIDIA Cosmos (SynthID video). No local detector possible; treat like SynthID.
|
||||
- **C2PA 2.4 "Durable Content Credentials" (April 2026; verified against the spec) raise the bar for metadata stripping.** 2.4 defines soft bindings (an invisible watermark or a content fingerprint) plus a server-side manifest repository and a new `c2pa.repository-receipt` assertion. Per the spec: "if a C2PA manifest is removed from an asset, but a copy of that manifest remains in a provenance store elsewhere, the manifest and asset may be matched using available soft bindings." So our local `metadata --remove` deletes the *embedded* manifest, but a fingerprint/watermark soft binding can still re-link the image to its manifest in a repository server-side. Stripping the file is becoming necessary-but-not-sufficient against durable provenance. (Our parsers target the stable embedded-manifest format documented in C2PA 2.1 §11; that format is unchanged in 2.4 -- the new pieces are repository/soft-binding infra, not the on-file box layout, so no parser change is implied.) Spec: https://spec.c2pa.org/specifications/specifications/2.4/specs/C2PA_Specification.html We now READ the soft-binding `alg` (`C2PA_SOFT_BINDINGS` / `soft_binding_vendors_in`) to name the forensic-watermark vendor, and locally DECODE the one open scheme, Adobe TrustMark (`trustmark_detector`); the rest (Digimarc/Imatag/Steg.AI/...) stay name-only (proprietary decoders).
|
||||
|
||||
@@ -25,7 +25,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
|
||||
- **Batch processing** — process entire directories
|
||||
- **Detection** — three-stage NCC watermark detection with confidence scoring
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP or PNG chunk), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -48,13 +48,13 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
| **xAI Grok (Aurora)** | — | — | ✅ EXIF signature scheme (no C2PA): `Signature:` blob + UUID `Artist` | Detected (`identify`); metadata strip |
|
||||
| **Midjourney** | — | — | ✅ EXIF + XMP (prompt, model, seed) | Metadata strip |
|
||||
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
|
||||
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 `<TC260:AIGC>` XMP label (China's mandatory AI labeling) | Locate + mask + inpaint (cv2, CPU) + metadata strip |
|
||||
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 AIGC label — `<TC260:AIGC>` XMP **or** `AIGC` PNG chunk (China's mandatory AI labeling) | Locate + mask + inpaint (cv2, CPU) + metadata strip |
|
||||
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
|
||||
| **TreeRing** | — | ✅ Latent space watermark | — | Diffusion regeneration |
|
||||
|
||||
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo) and by Doubao / China AIGC generators (the mandated "...AI生成" corner text). Both are removed deterministically on CPU. Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any colour), use the universal `erase --region` command.
|
||||
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP or PNG chunk), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
|
||||
## How it works
|
||||
|
||||
@@ -326,7 +326,7 @@ Tracked but not yet implemented:
|
||||
- **Local SynthID *pixel* detector**. Not feasible today: Google's decoder is proprietary, and magnitude/carrier spectral methods do not separate real content (confirmed by three independent evaluations, including a from-scratch gpt-image pilot; see CLAUDE.md). Blocked on either (a) a programmatic generation path (OpenAI / Gemini API) to build a per-(model, resolution) labeled corpus at scale, or (b) a raw watermarked-output dataset. If data arrives, the next approach to try is a learned classifier on diverse content rather than a fixed carrier codebook.
|
||||
- **Grow the SynthID reference corpus** (`data/synthid_corpus/`) with oracle-labeled samples per model and resolution (Gemini app for Google, openai.com/verify for OpenAI). Prerequisite for any pixel-detector attempt and for an automated removal-regression set.
|
||||
- **Real non-PNG C2PA fixtures**. SynthID-source detection for JPEG / WebP / AVIF is currently covered only by synthetic byte blobs; replace with real vendor-emitted files to ground the binary-scan path.
|
||||
- **Maintenance debt**. Clear strict-pyright debt in `remove_ai_metadata` / `cli.py` (untyped piexif / PIL / click / rich) so `maintain.sh` can finish green. (`uv-secure` is already clean since `idna` was bumped to 3.16.)
|
||||
- **Maintenance debt**. Strict pyright is now clean across `src/` (0 errors): pure-logic files are fully typed, the cv2 / torch / diffusers boundary files carry a documented per-file relax pragma, and a local `typings/piexif` stub covers piexif. Remaining: full-project `pyright` (no path) still OOMs node on this ML-heavy repo, so it must be scoped to `src/`; narrowing the boundary pragmas back toward full strict (as upstream stubs improve) is the long tail. (`uv-secure` is already clean since `idna` was bumped to 3.16.)
|
||||
- **AVIF / HEIF `Exif` item inside the `meta` box**. An AI-label *XMP* packet in a `meta`-box item is now blanked in place (v0.6.9), but EXIF stored as a `meta`-box `Exif` *item* is still not removed — it needs full `iinf`/`iloc` surgery (offset rewrite, corruption risk) or `exiftool` (a non-bundled binary dependency). Low priority: the AI labels we target are XMP, not EXIF, so an EXIF-only meta-box case is rare.
|
||||
- **More C2PA device signers**. Leica, Nikon, Google Pixel, Sony, and Truepic are mapped (each verified against a real signed file). Canon and Samsung Galaxy (AI-edit) are deferred until a real signed sample surfaces — no public direct-download C2PA file exists for them today (upload-to-verify / news-agency-licensed only).
|
||||
- **Resemble PerTh audio detection** — evaluated, not feasible with the public API: `get_watermark()` returns a raw bit array with no presence/confidence flag, so watermarked vs. clean audio can't be reliably separated without Resemble's fixed payload or a confidence service. Same wall as the SynthID pixel detector.
|
||||
|
||||
+31
-1
@@ -1,6 +1,6 @@
|
||||
[project]
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.6.10"
|
||||
version = "0.6.12"
|
||||
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
@@ -133,3 +133,33 @@ reportUnknownMemberType = false
|
||||
reportUnknownArgumentType = false
|
||||
reportUnknownVariableType = false
|
||||
reportMissingTypeArgument = false
|
||||
|
||||
# CtrlRegen is a torch/diffusers/controlnet-aux boundary: those libs ship no
|
||||
# usable types, so strict pyright cannot know the tensor element types. Relax the
|
||||
# unknown-type rules for this subpackage only (mirrors the per-file pragmas used
|
||||
# for the cv2 engines); the rest of the codebase stays strict.
|
||||
[[tool.pyright.executionEnvironments]]
|
||||
root = "src/remove_ai_watermarks/noai/ctrlregen"
|
||||
reportUnknownMemberType = false
|
||||
reportUnknownArgumentType = false
|
||||
reportUnknownVariableType = false
|
||||
reportUnknownParameterType = false
|
||||
reportMissingTypeArgument = false
|
||||
reportMissingTypeStubs = false
|
||||
reportMissingImports = false
|
||||
reportArgumentType = false
|
||||
reportAssignmentType = false
|
||||
reportReturnType = false
|
||||
reportCallIssue = false
|
||||
reportIndexIssue = false
|
||||
reportOperatorIssue = false
|
||||
reportOptionalMemberAccess = false
|
||||
reportOptionalCall = false
|
||||
reportOptionalSubscript = false
|
||||
reportOptionalOperand = false
|
||||
reportAttributeAccessIssue = false
|
||||
reportPrivateImportUsage = false
|
||||
reportPrivateUsage = false
|
||||
reportInvalidTypeForm = false
|
||||
reportConstantRedefinition = false
|
||||
reportUnnecessaryComparison = false
|
||||
@@ -1,3 +1,3 @@
|
||||
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
|
||||
|
||||
__version__ = "0.6.9"
|
||||
__version__ = "0.6.12"
|
||||
@@ -12,7 +12,7 @@ import json
|
||||
import logging
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
@@ -23,7 +23,7 @@ from rich.table import Table
|
||||
from remove_ai_watermarks import __version__
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy as np
|
||||
from numpy.typing import NDArray
|
||||
|
||||
from remove_ai_watermarks.gemini_engine import DetectionResult, GeminiEngine
|
||||
|
||||
@@ -76,7 +76,7 @@ def _watermark_region(det: DetectionResult, width: int, height: int) -> tuple[in
|
||||
return (px, py, config.logo_size, config.logo_size)
|
||||
|
||||
|
||||
def _read_bgr_and_alpha(path: Path) -> tuple[np.ndarray | None, np.ndarray | None]:
|
||||
def _read_bgr_and_alpha(path: Path) -> tuple[NDArray[Any] | None, NDArray[Any] | None]:
|
||||
"""Read an image preserving its alpha channel separately.
|
||||
|
||||
Returns ``(bgr, alpha)`` where ``alpha`` is a single-channel ndarray when the
|
||||
@@ -99,8 +99,8 @@ def _read_bgr_and_alpha(path: Path) -> tuple[np.ndarray | None, np.ndarray | Non
|
||||
|
||||
def _write_bgr_with_alpha(
|
||||
path: Path,
|
||||
bgr: np.ndarray,
|
||||
alpha: np.ndarray | None,
|
||||
bgr: NDArray[Any],
|
||||
alpha: NDArray[Any] | None,
|
||||
clear_region: tuple[int, int, int, int] | None = None,
|
||||
pad: int = 6,
|
||||
) -> None:
|
||||
@@ -135,8 +135,8 @@ def _write_bgr_with_alpha(
|
||||
|
||||
def _run_doubao_if_selected(
|
||||
ctx: click.Context,
|
||||
image: np.ndarray,
|
||||
alpha: np.ndarray | None,
|
||||
image: NDArray[Any],
|
||||
alpha: NDArray[Any] | None,
|
||||
output: Path,
|
||||
mark: str,
|
||||
gemini_engine: GeminiEngine,
|
||||
@@ -249,7 +249,7 @@ def cmd_visible(
|
||||
source: Path,
|
||||
output: Path | None,
|
||||
inpaint: bool,
|
||||
inpaint_method: str,
|
||||
inpaint_method: Literal["ns", "telea", "gaussian"],
|
||||
inpaint_strength: float,
|
||||
detect: bool,
|
||||
detect_threshold: float,
|
||||
@@ -378,7 +378,7 @@ def cmd_erase(
|
||||
source: Path,
|
||||
regions: tuple[str, ...],
|
||||
output: Path | None,
|
||||
backend: str,
|
||||
backend: Literal["cv2", "lama"],
|
||||
inpaint_method: str,
|
||||
dilate: int,
|
||||
strip_metadata: bool,
|
||||
@@ -626,6 +626,14 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
console.print(f"\n Verdict: {verdict} [dim](confidence: {report.confidence})[/]")
|
||||
console.print(f" Platform: {report.platform or '[dim]undetermined[/]'}")
|
||||
|
||||
if report.is_ai_generated is None:
|
||||
console.print(
|
||||
" [dim]No locally-readable AI signal found. This is not the same as 'clean': "
|
||||
"metadata is often stripped by re-encoding, screenshots, or upload, and SynthID-class "
|
||||
"pixel watermarks (Gemini / Nano Banana / gpt-image) have no local detector. "
|
||||
"See caveats below.[/]"
|
||||
)
|
||||
|
||||
if report.integrity_clashes:
|
||||
console.print("\n [bold red]⚠ Integrity clash[/] [dim](provenance signals contradict each other)[/]")
|
||||
for clash in report.integrity_clashes:
|
||||
@@ -691,7 +699,7 @@ def cmd_all(
|
||||
source: Path,
|
||||
output: Path | None,
|
||||
inpaint: bool,
|
||||
inpaint_method: str,
|
||||
inpaint_method: Literal["ns", "telea", "gaussian"],
|
||||
strength: float,
|
||||
steps: int,
|
||||
pipeline: str,
|
||||
@@ -856,7 +864,7 @@ def _process_batch_image(
|
||||
Raises:
|
||||
ValueError: If the image cannot be opened.
|
||||
"""
|
||||
saved_alpha: np.ndarray | None = None
|
||||
saved_alpha: NDArray[Any] | None = None
|
||||
saved_region: tuple[int, int, int, int] | None = None
|
||||
|
||||
if mode in ("visible", "all"):
|
||||
|
||||
@@ -26,11 +26,14 @@ true pixels instead of hallucinating them -- the same approach as the Gemini
|
||||
engine.
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: third-party libs ship no usable element types; relax the
|
||||
# unknown-type rules for this file only.
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
@@ -119,7 +122,7 @@ class DoubaoEngine:
|
||||
|
||||
# ── Locate ────────────────────────────────────────────────────────
|
||||
|
||||
def locate(self, image: NDArray) -> DoubaoLocation:
|
||||
def locate(self, image: NDArray[Any]) -> DoubaoLocation:
|
||||
"""Anchor the watermark box in the bottom-right corner by geometry."""
|
||||
h, w = image.shape[:2]
|
||||
wm_w = max(40, int(w * self.width_frac))
|
||||
@@ -134,7 +137,7 @@ class DoubaoEngine:
|
||||
|
||||
# ── Mask ──────────────────────────────────────────────────────────
|
||||
|
||||
def extract_mask(self, image: NDArray, loc: DoubaoLocation) -> NDArray:
|
||||
def extract_mask(self, image: NDArray[Any], loc: DoubaoLocation) -> NDArray[Any]:
|
||||
"""Build a full-image uint8 mask (255 = watermark glyph) for the box.
|
||||
|
||||
Polarity-aware: the mark is a light, low-saturation gray. On a dark
|
||||
@@ -172,7 +175,7 @@ class DoubaoEngine:
|
||||
|
||||
# ── Detect ────────────────────────────────────────────────────────
|
||||
|
||||
def detect(self, image: NDArray) -> DoubaoDetection:
|
||||
def detect(self, image: NDArray[Any]) -> DoubaoDetection:
|
||||
"""Detect the visible Doubao mark by glyph coverage in the corner box.
|
||||
|
||||
Heuristic: a genuine label fills a meaningful fraction of the box with
|
||||
@@ -198,12 +201,12 @@ class DoubaoEngine:
|
||||
|
||||
def remove_watermark(
|
||||
self,
|
||||
image: NDArray,
|
||||
image: NDArray[Any],
|
||||
*,
|
||||
inpaint_method: Literal["telea", "ns"] = "telea",
|
||||
inpaint_radius: int = 6,
|
||||
dilate: int = 3,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Remove the visible Doubao watermark by inpainting the glyph mask.
|
||||
|
||||
Returns an unmodified copy when no glyph pixels are found (so we never
|
||||
@@ -237,7 +240,7 @@ class DoubaoEngine:
|
||||
return cv2.inpaint(image, mask, inpaint_radius, flag)
|
||||
|
||||
|
||||
def load_image_bgr(path: str | Path) -> NDArray:
|
||||
def load_image_bgr(path: str | Path) -> NDArray[Any]:
|
||||
"""Read an image as BGR ndarray (helper for scripts/tests)."""
|
||||
from remove_ai_watermarks import image_io
|
||||
|
||||
|
||||
@@ -1,5 +1,8 @@
|
||||
"""YOLO-based face detection and soft-blend restoration for diffusion pipelines."""
|
||||
|
||||
# cv2/numpy/ultralytics boundary: these libs ship no usable element types; relax
|
||||
# the unknown-type rules for this file only.
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false, reportPossiblyUnboundVariable=false
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@@ -13,13 +13,17 @@ The alpha maps are derived from background captures of the Gemini watermark
|
||||
on pure-black backgrounds (48x48 for small images, 96x96 for large images).
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: cv2 and numpy ship no usable type info for the array ops
|
||||
# below, so strict pyright cannot know their element types. Relax the unknown-type
|
||||
# rules for this file only; the public signatures are still annotated with NDArray[Any].
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from dataclasses import dataclass
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
@@ -86,7 +90,7 @@ def get_watermark_size(width: int, height: int) -> WatermarkSize:
|
||||
return WatermarkSize.SMALL
|
||||
|
||||
|
||||
def _calculate_alpha_map(bg_capture: NDArray) -> NDArray:
|
||||
def _calculate_alpha_map(bg_capture: NDArray[Any]) -> NDArray[Any]:
|
||||
"""Calculate alpha map from a background capture.
|
||||
|
||||
The alpha map represents how much the watermark affects each pixel.
|
||||
@@ -103,7 +107,7 @@ def _calculate_alpha_map(bg_capture: NDArray) -> NDArray:
|
||||
return gray / 255.0
|
||||
|
||||
|
||||
def _load_embedded_asset(name: str) -> NDArray:
|
||||
def _load_embedded_asset(name: str) -> NDArray[Any]:
|
||||
"""Load an embedded PNG asset and decode it with OpenCV."""
|
||||
asset_path = Path(__file__).parent / "assets" / name
|
||||
if not asset_path.exists():
|
||||
@@ -151,13 +155,13 @@ class GeminiEngine:
|
||||
self._alpha_large.shape,
|
||||
)
|
||||
|
||||
def get_alpha_map(self, size: WatermarkSize) -> NDArray:
|
||||
def get_alpha_map(self, size: WatermarkSize) -> NDArray[Any]:
|
||||
"""Get the base alpha map for a specific standard size."""
|
||||
if size == WatermarkSize.SMALL:
|
||||
return self._alpha_small
|
||||
return self._alpha_large
|
||||
|
||||
def get_interpolated_alpha(self, size_px: int) -> NDArray:
|
||||
def get_interpolated_alpha(self, size_px: int) -> NDArray[Any]:
|
||||
"""Create an interpolated alpha map dynamically scaled from the high-res 96x96 base."""
|
||||
source = self._alpha_large
|
||||
if size_px == source.shape[1]:
|
||||
@@ -170,7 +174,7 @@ class GeminiEngine:
|
||||
|
||||
def detect_watermark(
|
||||
self,
|
||||
image: NDArray,
|
||||
image: NDArray[Any],
|
||||
force_size: WatermarkSize | None = None,
|
||||
) -> DetectionResult:
|
||||
"""Detect Gemini watermark using multi-scale Snap Engine logic (ported from C++ vendor algorithm)."""
|
||||
@@ -304,9 +308,9 @@ class GeminiEngine:
|
||||
|
||||
def remove_watermark(
|
||||
self,
|
||||
image: NDArray,
|
||||
image: NDArray[Any],
|
||||
force_size: WatermarkSize | None = None,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Remove Gemini visible watermark from an image using reverse alpha blending.
|
||||
|
||||
No-op when the detector does not find a watermark: returns an unmodified
|
||||
@@ -359,9 +363,9 @@ class GeminiEngine:
|
||||
|
||||
def remove_watermark_custom(
|
||||
self,
|
||||
image: NDArray,
|
||||
image: NDArray[Any],
|
||||
region: tuple[int, int, int, int],
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Remove watermark from a custom region with interpolated alpha map.
|
||||
|
||||
Args:
|
||||
@@ -390,8 +394,8 @@ class GeminiEngine:
|
||||
|
||||
def _reverse_alpha_blend(
|
||||
self,
|
||||
image: NDArray,
|
||||
alpha_map: NDArray,
|
||||
image: NDArray[Any],
|
||||
alpha_map: NDArray[Any],
|
||||
position: tuple[int, int],
|
||||
) -> None:
|
||||
"""Apply reverse alpha blending in-place.
|
||||
@@ -442,13 +446,13 @@ class GeminiEngine:
|
||||
|
||||
def inpaint_residual(
|
||||
self,
|
||||
image: NDArray,
|
||||
image: NDArray[Any],
|
||||
region: tuple[int, int, int, int],
|
||||
strength: float = 0.85,
|
||||
method: Literal["gaussian", "telea", "ns"] = "ns",
|
||||
inpaint_radius: int = 10,
|
||||
padding: int = 32,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Apply inpaint cleanup on residual artifacts after reverse alpha blend.
|
||||
|
||||
Uses a sparse mask derived from alpha map gradient to repair only
|
||||
|
||||
@@ -4,6 +4,9 @@ Simulates analog film imperfections to defeat digital AI perfection
|
||||
classifiers. Ported from NeuralBleach.
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: third-party libs ship no usable element types; relax the
|
||||
# unknown-type rules for this file only.
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
import cv2
|
||||
import numpy as np
|
||||
from numpy.typing import NDArray
|
||||
|
||||
@@ -31,6 +31,7 @@ from remove_ai_watermarks.metadata import (
|
||||
aigc_label,
|
||||
exif_generator,
|
||||
get_ai_metadata,
|
||||
huggingface_job,
|
||||
iptc_ai_system,
|
||||
scan_head,
|
||||
xai_signature,
|
||||
@@ -89,6 +90,11 @@ _INVISIBLE_WM_CAVEAT = (
|
||||
"The open invisible watermark is fragile: it does not survive JPEG re-encoding "
|
||||
"or resizing, so it confirms origin only on a pristine (un-re-encoded) file."
|
||||
)
|
||||
_HF_JOB_CAVEAT = (
|
||||
"The hf-job-id tag marks a HuggingFace-hosted job (commonly diffusion "
|
||||
"generation) but names neither the model nor the content type, so it is a "
|
||||
"medium-confidence signal, not proof the pixels are AI-generated."
|
||||
)
|
||||
|
||||
|
||||
@dataclass
|
||||
@@ -423,9 +429,14 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
ai_vendor_claims["iptc_ai_system"] = v
|
||||
|
||||
# ── China TC260 AIGC label (Doubao and other China-served gens) ──
|
||||
aigc = any(m in head for m in AIGC_MARKERS)
|
||||
# Fire on either the namespaced byte marker (``TC260:AIGC`` / the TC260 ns
|
||||
# URL, present in XMP and as a laundering tell even when the JSON payload is
|
||||
# truncated) OR the parsed label, which additionally catches the raw-JSON
|
||||
# PNG ``AIGC`` tEXt chunk that carries no namespaced marker at all.
|
||||
aigc_data = aigc_label(image_path)
|
||||
aigc = aigc_data is not None or any(m in head for m in AIGC_MARKERS)
|
||||
if aigc:
|
||||
producer = (aigc_label(image_path) or {}).get("ContentProducer", "")
|
||||
producer = (aigc_data or {}).get("ContentProducer", "")
|
||||
signals.append(Signal("aigc", f"TC260 AIGC label{f' (producer {producer})' if producer else ''}", "high"))
|
||||
watermarks.append("China AIGC label (TC260 standard)")
|
||||
if platform is None:
|
||||
@@ -461,6 +472,18 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
platform = "xAI (Grok / Aurora)"
|
||||
ai_vendor_claims["xai"] = "xAI"
|
||||
|
||||
# ── HuggingFace-hosted job marker (hf-job-id PNG text chunk) ─────
|
||||
# Marks the hosting job, not a model -- medium confidence (commonly diffusion
|
||||
# output). Like the visible sparkle, it lifts an otherwise-Unknown verdict to
|
||||
# a tentative AI, but never overrides a high-confidence metadata signal.
|
||||
hf_job = huggingface_job(image_path)
|
||||
if hf_job:
|
||||
signals.append(Signal("hf_job", f"HuggingFace job {hf_job}", "medium"))
|
||||
watermarks.append("HuggingFace-hosted job (hf-job-id)")
|
||||
caveats.append(_HF_JOB_CAVEAT)
|
||||
if platform is None:
|
||||
platform = "HuggingFace-hosted job (model not identified)"
|
||||
|
||||
# ── Open invisible watermark (SD / SDXL / FLUX, dwtDct) ──────────
|
||||
# Public decoder, no key -- a definitive embedded signal on pristine files.
|
||||
if check_invisible and (scheme := _invisible_watermark(image_path)) is not None:
|
||||
@@ -503,11 +526,12 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
platform = "Google Gemini family (visible sparkle detected)"
|
||||
|
||||
visible_only = any(s.name == "visible_sparkle" for s in signals) and not ai_from_metadata
|
||||
hf_only = bool(hf_job) and not ai_from_metadata
|
||||
|
||||
if ai_from_metadata:
|
||||
is_ai: bool | None = True
|
||||
confidence = "high"
|
||||
elif visible_only:
|
||||
elif visible_only or hf_only:
|
||||
is_ai = True
|
||||
confidence = "medium"
|
||||
else:
|
||||
|
||||
@@ -7,6 +7,10 @@ This module requires the 'gpu' extra dependencies:
|
||||
uv pip install 'remove-ai-watermarks[gpu]'
|
||||
"""
|
||||
|
||||
# cv2/torch boundary: this engine wraps cv2 (resize/imwrite/cvtColor), the YOLO
|
||||
# face protector, and the humanizer, none of which carry usable element types;
|
||||
# relax the unknown-type rules for this file only.
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
@@ -33,13 +37,9 @@ logger = logging.getLogger(__name__)
|
||||
|
||||
def is_available() -> bool:
|
||||
"""Check if invisible watermark removal dependencies are installed."""
|
||||
try:
|
||||
import diffusers # noqa: F401
|
||||
import torch # noqa: F401
|
||||
import importlib.util
|
||||
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
return importlib.util.find_spec("diffusers") is not None and importlib.util.find_spec("torch") is not None
|
||||
|
||||
|
||||
def _target_size(width: int, height: int, max_resolution: int) -> tuple[int, int] | None:
|
||||
|
||||
@@ -11,7 +11,7 @@ from __future__ import annotations
|
||||
import contextlib
|
||||
import logging
|
||||
import re
|
||||
from typing import TYPE_CHECKING
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
@@ -108,6 +108,27 @@ AIGC_MARKERS: tuple[bytes, ...] = (
|
||||
b"TC260:AIGC",
|
||||
)
|
||||
|
||||
# TC260 AIGC-label JSON fields (the standard's labeling object). Doubao writes
|
||||
# the same object as a PNG ``tEXt`` chunk keyed ``AIGC`` (raw JSON, not XMP), so
|
||||
# a JSON object carrying at least one of these is accepted as a valid TC260
|
||||
# label even when the namespaced XMP element is absent.
|
||||
_TC260_FIELDS: frozenset[str] = frozenset(
|
||||
{
|
||||
"Label",
|
||||
"ContentProducer",
|
||||
"ProduceID",
|
||||
"ContentPropagator",
|
||||
"PropagateID",
|
||||
"ReservedCode1",
|
||||
"ReservedCode2",
|
||||
}
|
||||
)
|
||||
|
||||
# HuggingFace-hosted GPU jobs (Jobs / Spaces) stamp generated PNGs with this
|
||||
# ``tEXt`` chunk key holding the job UUID. It marks the hosting job, not a
|
||||
# specific model -- a medium-confidence AI signal (commonly diffusion output).
|
||||
_HF_JOB_KEY: str = "hf-job-id"
|
||||
|
||||
STANDARD_METADATA_KEYS: frozenset[str] = frozenset(
|
||||
[
|
||||
"Author",
|
||||
@@ -172,7 +193,7 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
try:
|
||||
with Image.open(image_path) as img:
|
||||
for key in img.info:
|
||||
if _is_ai_key(key):
|
||||
if isinstance(key, str) and _is_ai_key(key):
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.debug("PIL could not open %s for metadata scan: %s", image_path, exc)
|
||||
@@ -181,7 +202,8 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
# binary scan that also catches AVIF/HEIF/JPEG-XL containers (PIL doesn't
|
||||
# expose their metadata uniformly).
|
||||
try:
|
||||
from c2pa import has_c2pa_metadata
|
||||
# optional official lib, not a declared dep -> falls back to the binary scan
|
||||
from c2pa import has_c2pa_metadata # pyright: ignore[reportMissingImports, reportUnknownVariableType]
|
||||
|
||||
if has_c2pa_metadata(image_path):
|
||||
return True
|
||||
@@ -202,31 +224,90 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
# IPTC 2025.1 AI-disclosure XMP properties (their presence flags AI content).
|
||||
if any(marker in data for marker in IPTC_AI_FIELD_MARKERS):
|
||||
return True
|
||||
# China TC260 AIGC label as a PNG text chunk (the byte scan above catches
|
||||
# only the XMP form; the raw-JSON tEXt chunk needs the PIL-based parse).
|
||||
if aigc_label(image_path):
|
||||
return True
|
||||
# HuggingFace-hosted job marker (hf-job-id PNG text chunk).
|
||||
if huggingface_job(image_path):
|
||||
return True
|
||||
# xAI / Grok: no C2PA/IPTC/XMP -- only the EXIF Signature + UUID-Artist pair.
|
||||
return xai_signature(image_path)
|
||||
|
||||
|
||||
def aigc_label(image_path: Path) -> dict[str, str] | None:
|
||||
"""Parse a China TC260 ``<TC260:AIGC>`` AI-labeling block, if present.
|
||||
"""Parse a China TC260 AI-labeling block, if present.
|
||||
|
||||
Two serializations are recognized:
|
||||
|
||||
- a PNG ``tEXt``/``iTXt`` chunk keyed ``AIGC`` carrying the raw JSON object
|
||||
(as written by Doubao / ByteDance), read via PIL; and
|
||||
- an XMP ``<TC260:AIGC>{...}</TC260:AIGC>`` block (HTML-entity encoded text),
|
||||
found by a container-agnostic raw-byte scan (PNG/JPEG/WebP alike).
|
||||
|
||||
Returns the decoded JSON (e.g. ``{"Label": "1", "ContentProducer": ...}``)
|
||||
or None. The block is XMP text (HTML-entity encoded), so it is found by a
|
||||
container-agnostic raw-byte scan and works for PNG/JPEG/WebP alike.
|
||||
or None. The PNG-chunk key ``AIGC`` is generic, so a JSON object there is
|
||||
accepted only if it carries at least one known TC260 field (``_TC260_FIELDS``);
|
||||
the namespaced XMP element is unambiguous, so any JSON object is accepted.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
import re
|
||||
from typing import cast
|
||||
|
||||
def _parse(text: str, *, require_tc260_field: bool) -> dict[str, str] | None:
|
||||
try:
|
||||
parsed = json.loads(text)
|
||||
except ValueError:
|
||||
return None
|
||||
if not isinstance(parsed, dict):
|
||||
return None
|
||||
fields = {str(k): str(v) for k, v in cast("dict[object, object]", parsed).items()}
|
||||
if require_tc260_field and not (_TC260_FIELDS & fields.keys()):
|
||||
return None
|
||||
return fields
|
||||
|
||||
# PNG tEXt chunk keyed "AIGC" with raw JSON (Doubao and other China gens).
|
||||
# The key is generic, so require a TC260 field to avoid a false positive.
|
||||
try:
|
||||
from PIL import Image
|
||||
|
||||
with Image.open(image_path) as img:
|
||||
value = img.info.get("AIGC")
|
||||
except Exception as exc:
|
||||
logger.debug("PIL could not open %s for AIGC chunk scan: %s", image_path, exc)
|
||||
value = None
|
||||
if isinstance(value, str) and (result := _parse(value, require_tc260_field=True)):
|
||||
return result
|
||||
|
||||
# XMP <TC260:AIGC>{...}</TC260:AIGC> block (namespaced element, unambiguous).
|
||||
data = scan_head(image_path)
|
||||
match = re.search(rb"<TC260:AIGC>(.*?)</TC260:AIGC>", data, re.DOTALL)
|
||||
if not match:
|
||||
return None
|
||||
raw = html.unescape(match.group(1).decode("utf-8", "replace"))
|
||||
return _parse(html.unescape(match.group(1).decode("utf-8", "replace")), require_tc260_field=False)
|
||||
|
||||
|
||||
def huggingface_job(image_path: Path) -> str | None:
|
||||
"""Return the HuggingFace job id if the image carries an ``hf-job-id`` PNG
|
||||
text chunk, else None.
|
||||
|
||||
HuggingFace-hosted GPU jobs (Jobs / Spaces) stamp generated PNGs with an
|
||||
``hf-job-id`` ``tEXt`` chunk holding the job's UUID. It identifies the
|
||||
*hosting job*, not a specific model, and is most commonly seen on diffusion-
|
||||
generation output -- a medium-confidence AI signal, not proof of AI pixels
|
||||
on its own.
|
||||
"""
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except ValueError:
|
||||
from PIL import Image
|
||||
|
||||
with Image.open(image_path) as img:
|
||||
value = img.info.get(_HF_JOB_KEY)
|
||||
except Exception as exc:
|
||||
logger.debug("PIL could not open %s for hf-job-id scan: %s", image_path, exc)
|
||||
return None
|
||||
return {str(k): str(v) for k, v in parsed.items()} if isinstance(parsed, dict) else None
|
||||
if isinstance(value, str) and value.strip():
|
||||
return value.strip()
|
||||
return None
|
||||
|
||||
|
||||
def iptc_ai_system(image_path: Path) -> str | None:
|
||||
@@ -353,7 +434,7 @@ def _is_xai_signature_pair(description: str, artist: str) -> bool:
|
||||
return _XAI_SIGNATURE_RE.match(description) is not None and _UUID_RE.fullmatch(artist) is not None
|
||||
|
||||
|
||||
def _exif_text(ifd: dict, tag: int) -> str:
|
||||
def _exif_text(ifd: dict[int, Any], tag: int) -> str:
|
||||
"""Decode a piexif 0th-IFD byte tag to a stripped string ('' if absent)."""
|
||||
value = ifd.get(tag)
|
||||
return value.decode("latin1", "replace").strip() if isinstance(value, bytes) else ""
|
||||
@@ -389,7 +470,7 @@ def xai_signature(image_path: Path) -> bool:
|
||||
)
|
||||
|
||||
|
||||
def _scrub_ai_exif(exif_dict: dict) -> list[str]:
|
||||
def _scrub_ai_exif(exif_dict: dict[str, Any]) -> list[str]:
|
||||
"""Delete AI-provenance tags from a piexif dict's ``0th`` IFD, in place.
|
||||
|
||||
Removes (a) the xAI/Grok signature pair (``ImageDescription`` "Signature: ..."
|
||||
@@ -453,7 +534,7 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
|
||||
try:
|
||||
with Image.open(image_path) as img:
|
||||
for key, value in img.info.items():
|
||||
if _is_ai_key(key):
|
||||
if isinstance(key, str) and _is_ai_key(key):
|
||||
if isinstance(value, bytes):
|
||||
result[key] = f"<binary {len(value)} bytes>"
|
||||
elif isinstance(value, str) and len(value) > 200:
|
||||
@@ -500,6 +581,10 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
|
||||
# IPTC 2025.1 AI-disclosure XMP fields (Iptc4xmpExt:AISystemUsed etc.).
|
||||
if system := iptc_ai_system(image_path):
|
||||
result.setdefault("ai_system", f"IPTC 2025.1 AI disclosure ({system})")
|
||||
|
||||
# HuggingFace-hosted job marker (hf-job-id PNG text chunk).
|
||||
if job := huggingface_job(image_path):
|
||||
result.setdefault("huggingface_job", f"HuggingFace-hosted job ({job})")
|
||||
return result
|
||||
|
||||
|
||||
@@ -607,11 +692,23 @@ def remove_ai_metadata(
|
||||
img = img.copy()
|
||||
fmt = output_path.suffix.lower()
|
||||
|
||||
save_kwargs: dict = {}
|
||||
save_kwargs: dict[str, Any] = {}
|
||||
if fmt in (".jpg", ".jpeg"):
|
||||
save_kwargs["format"] = "JPEG"
|
||||
# JPEG output is unavoidably lossy, so minimize the loss: high quality
|
||||
# and no chroma subsampling (4:4:4). Without these PIL defaults to
|
||||
# quality 75 + 4:2:0, which visibly degrades a re-saved image.
|
||||
save_kwargs["quality"] = 95
|
||||
save_kwargs["subsampling"] = 0
|
||||
if img.mode in ("RGBA", "P"):
|
||||
img = img.convert("RGB")
|
||||
elif fmt == ".webp":
|
||||
# Preserve the WebP container losslessly instead of silently rewriting
|
||||
# it as PNG (which changes the format and bloats the file).
|
||||
save_kwargs["format"] = "WEBP"
|
||||
save_kwargs["lossless"] = True
|
||||
if img.mode == "P": # WebP cannot encode palette mode
|
||||
img = img.convert("RGBA" if "transparency" in img.info else "RGB")
|
||||
else:
|
||||
save_kwargs["format"] = "PNG"
|
||||
|
||||
@@ -620,6 +717,8 @@ def remove_ai_metadata(
|
||||
exif_data = None
|
||||
|
||||
for key, value in img.info.items():
|
||||
if not isinstance(key, str):
|
||||
continue
|
||||
if _is_ai_key(key):
|
||||
continue
|
||||
if key == "exif":
|
||||
|
||||
@@ -94,6 +94,8 @@ def _extract_non_ai_metadata(source_path: Path, keep_standard: bool) -> dict[str
|
||||
|
||||
# Extract non-AI metadata
|
||||
for key, value in img.info.items():
|
||||
if not isinstance(key, str):
|
||||
continue
|
||||
if _is_ai_metadata_key(key):
|
||||
continue
|
||||
|
||||
@@ -127,7 +129,7 @@ def _is_ai_metadata_key(key: str) -> bool:
|
||||
|
||||
def _prepare_clean_png_kwargs(save_kwargs: dict[str, Any], metadata: dict[str, Any]) -> dict[str, Any]:
|
||||
"""Prepare save kwargs for clean PNG."""
|
||||
pnginfo = {}
|
||||
pnginfo: dict[str, Any] = {}
|
||||
exclude_keys = ["exif", "exif_raw", "dpi", "gamma"]
|
||||
|
||||
for key, value in metadata.items():
|
||||
|
||||
@@ -6,7 +6,7 @@ human-readable summary without modifying the source file.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING, Any
|
||||
from typing import TYPE_CHECKING, Any, cast
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from pathlib import Path
|
||||
@@ -46,6 +46,8 @@ def extract_metadata(source_path: Path) -> dict[str, Any]:
|
||||
|
||||
# Extract all other metadata including AI-specific
|
||||
for key, value in img.info.items():
|
||||
if not isinstance(key, str):
|
||||
continue
|
||||
if key not in metadata and key not in ["exif"]:
|
||||
metadata[key] = value
|
||||
|
||||
@@ -83,6 +85,8 @@ def extract_ai_metadata(source_path: Path) -> dict[str, Any]:
|
||||
ai_metadata[key] = img.info[key]
|
||||
|
||||
for key, value in img.info.items():
|
||||
if not isinstance(key, str):
|
||||
continue
|
||||
key_lower = key.lower()
|
||||
if key not in ai_metadata and any(kw in key_lower for kw in AI_KEYWORDS):
|
||||
ai_metadata[key] = value
|
||||
@@ -138,7 +142,7 @@ def get_ai_metadata_summary(source_path: Path) -> str:
|
||||
continue
|
||||
if key == "c2pa" and isinstance(value, dict):
|
||||
lines.append("C2PA Metadata:")
|
||||
for ck, cv in value.items():
|
||||
for ck, cv in cast("dict[str, Any]", value).items():
|
||||
lines.append(f" {ck}: {cv}")
|
||||
elif isinstance(value, str) and len(value) > 100:
|
||||
value = value[:100] + "..."
|
||||
|
||||
@@ -259,7 +259,7 @@ def make_pipeline_progress(
|
||||
label: str = "Denoising",
|
||||
pre_phases: list[tuple[int, str]] | None = None,
|
||||
post_phases: list[tuple[int, str]] | None = None,
|
||||
) -> tuple[Callable, threading.Event, threading.Event, Callable[[], threading.Thread]]:
|
||||
) -> tuple[Callable[..., None], threading.Event, threading.Event, Callable[[], threading.Thread]]:
|
||||
"""Create step callback and background updater for a diffusion pipeline.
|
||||
|
||||
Returns:
|
||||
|
||||
@@ -10,6 +10,9 @@ This module implements a simple regeneration attack that:
|
||||
4. Decodes back to pixel space
|
||||
"""
|
||||
|
||||
# torch/diffusers/cv2 boundary: these libs ship no usable types for the tensor and
|
||||
# array ops below; relax the unknown-type rules for this file only.
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
from __future__ import annotations
|
||||
|
||||
import contextlib
|
||||
|
||||
@@ -15,10 +15,14 @@ Backends:
|
||||
huggingface_hub; it is never bundled in this repo.
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: cv2 ships no usable type info, so strict pyright cannot know
|
||||
# its array element types. Relax the unknown-type rules for this file only; the
|
||||
# public signatures are still annotated with NDArray[Any].
|
||||
# pyright: reportUnknownMemberType=false, reportUnknownArgumentType=false, reportUnknownVariableType=false, reportUnknownParameterType=false, reportMissingTypeArgument=false, reportMissingTypeStubs=false, reportMissingImports=false, reportArgumentType=false, reportAssignmentType=false, reportReturnType=false, reportCallIssue=false, reportIndexIssue=false, reportOperatorIssue=false, reportOptionalMemberAccess=false, reportOptionalCall=false, reportOptionalSubscript=false, reportOptionalOperand=false, reportAttributeAccessIssue=false, reportPrivateImportUsage=false, reportPrivateUsage=false, reportInvalidTypeForm=false, reportConstantRedefinition=false, reportUnnecessaryComparison=false
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
from typing import TYPE_CHECKING, Literal
|
||||
from typing import TYPE_CHECKING, Any, Literal
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
@@ -41,7 +45,7 @@ def boxes_to_mask(
|
||||
shape: tuple[int, int],
|
||||
boxes: list[tuple[int, int, int, int]],
|
||||
dilate: int = 3,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Build a uint8 mask (255 inside boxes) from ``(x, y, w, h)`` rectangles."""
|
||||
h, w = shape
|
||||
mask = np.zeros((h, w), np.uint8)
|
||||
@@ -57,12 +61,12 @@ def boxes_to_mask(
|
||||
|
||||
|
||||
def erase_cv2(
|
||||
image_bgr: NDArray,
|
||||
mask: NDArray,
|
||||
image_bgr: NDArray[Any],
|
||||
mask: NDArray[Any],
|
||||
*,
|
||||
method: Literal["telea", "ns"] = "telea",
|
||||
radius: int = 6,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Inpaint ``mask`` with classical cv2 inpainting (CPU, no extra deps)."""
|
||||
flag = cv2.INPAINT_TELEA if method == "telea" else cv2.INPAINT_NS
|
||||
return cv2.inpaint(image_bgr, mask, radius, flag)
|
||||
@@ -70,12 +74,9 @@ def erase_cv2(
|
||||
|
||||
def lama_available() -> bool:
|
||||
"""True when the optional LaMa-ONNX backend can run (onnxruntime installed)."""
|
||||
try:
|
||||
import onnxruntime # noqa: F401
|
||||
import importlib.util
|
||||
|
||||
return True
|
||||
except ImportError:
|
||||
return False
|
||||
return importlib.util.find_spec("onnxruntime") is not None
|
||||
|
||||
|
||||
def _get_lama_session() -> object:
|
||||
@@ -93,7 +94,7 @@ def _get_lama_session() -> object:
|
||||
return _lama_session
|
||||
|
||||
|
||||
def erase_lama(image_bgr: NDArray, mask: NDArray) -> NDArray:
|
||||
def erase_lama(image_bgr: NDArray[Any], mask: NDArray[Any]) -> NDArray[Any]:
|
||||
"""Inpaint ``mask`` with big-LaMa via onnxruntime (CPU).
|
||||
|
||||
LaMa runs at a fixed square input size. To preserve full-image resolution we
|
||||
@@ -147,15 +148,15 @@ def erase_lama(image_bgr: NDArray, mask: NDArray) -> NDArray:
|
||||
|
||||
|
||||
def erase(
|
||||
image_bgr: NDArray,
|
||||
image_bgr: NDArray[Any],
|
||||
*,
|
||||
boxes: list[tuple[int, int, int, int]] | None = None,
|
||||
mask: NDArray | None = None,
|
||||
mask: NDArray[Any] | None = None,
|
||||
backend: Backend = "cv2",
|
||||
dilate: int = 3,
|
||||
cv2_method: Literal["telea", "ns"] = "telea",
|
||||
cv2_radius: int = 6,
|
||||
) -> NDArray:
|
||||
) -> NDArray[Any]:
|
||||
"""Erase the given boxes (or mask) via the chosen inpainting backend.
|
||||
|
||||
Provide either ``boxes`` (list of ``(x, y, w, h)``) or a precomputed ``mask``
|
||||
|
||||
@@ -375,6 +375,14 @@ class TestIdentifyCommand:
|
||||
assert result.exit_code == 0
|
||||
assert "unknown" in result.output
|
||||
|
||||
def test_identify_unknown_explains_why(self, runner, tmp_clean_png):
|
||||
# An unknown verdict must explain itself inline (issue #22: users read a bare
|
||||
# "unknown" as the tool being broken) rather than only in the caveats section.
|
||||
result = runner.invoke(main, ["identify", str(tmp_clean_png), "--no-visible"])
|
||||
assert result.exit_code == 0
|
||||
assert "No locally-readable AI signal found" in result.output
|
||||
assert "not the same as 'clean'" in result.output
|
||||
|
||||
def test_identify_ai_png_reports_platform(self, runner, tmp_png_with_ai_metadata):
|
||||
result = runner.invoke(main, ["identify", str(tmp_png_with_ai_metadata), "--no-visible"])
|
||||
assert result.exit_code == 0
|
||||
|
||||
@@ -201,6 +201,78 @@ class TestIdentifyLocalParams:
|
||||
assert r.signals == []
|
||||
|
||||
|
||||
# ── China TC260 AIGC label as a PNG text chunk (Doubao) ─────────────
|
||||
|
||||
|
||||
class TestIdentifyAigcPngChunk:
|
||||
"""The raw-JSON ``AIGC`` PNG chunk (no namespaced XMP marker) is a high-
|
||||
confidence AI verdict, same as the XMP form."""
|
||||
|
||||
def _aigc_chunk_png(self, tmp_path: Path) -> Path:
|
||||
from PIL import Image
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
|
||||
p = tmp_path / "doubao_chunk.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("AIGC", json.dumps({"Label": "1", "ContentProducer": "doubao"}))
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=pnginfo)
|
||||
return p
|
||||
|
||||
def test_png_chunk_detected_high(self, tmp_path: Path):
|
||||
r = identify(self._aigc_chunk_png(tmp_path), check_visible=False)
|
||||
assert r.is_ai_generated is True
|
||||
assert r.confidence == "high"
|
||||
assert r.platform is not None
|
||||
assert "AIGC" in r.platform
|
||||
signal = next(s for s in r.signals if s.name == "aigc")
|
||||
assert "doubao" in signal.detail
|
||||
|
||||
|
||||
# ── HuggingFace-hosted job marker (medium confidence) ───────────────
|
||||
|
||||
|
||||
class TestIdentifyHuggingFaceJob:
|
||||
"""The hf-job-id chunk lifts an otherwise-Unknown verdict to a tentative
|
||||
(medium) AI, never overriding a high-confidence metadata signal."""
|
||||
|
||||
def _hf_png(self, tmp_path: Path) -> Path:
|
||||
from PIL import Image
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
|
||||
p = tmp_path / "hfjob.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("hf-job-id", "ec8380a6-2091-423a-b835-209420f99ee1")
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=pnginfo)
|
||||
return p
|
||||
|
||||
def test_hf_job_promotes_to_medium(self, tmp_path: Path):
|
||||
r = identify(self._hf_png(tmp_path), check_visible=False)
|
||||
assert r.is_ai_generated is True
|
||||
assert r.confidence == "medium"
|
||||
assert r.platform is not None
|
||||
assert "HuggingFace" in r.platform
|
||||
signal = next(s for s in r.signals if s.name == "hf_job")
|
||||
assert signal.confidence == "medium"
|
||||
|
||||
def test_hf_job_caveat_present(self, tmp_path: Path):
|
||||
r = identify(self._hf_png(tmp_path), check_visible=False)
|
||||
assert any("hf-job-id" in c for c in r.caveats)
|
||||
|
||||
def test_metadata_keeps_high_even_with_hf_job(self, tmp_png_with_ai_metadata: Path):
|
||||
# A high-confidence metadata verdict is not downgraded by an hf-job hit.
|
||||
from PIL import Image
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
|
||||
img = Image.open(tmp_png_with_ai_metadata)
|
||||
pnginfo = PngInfo()
|
||||
for k, v in img.text.items():
|
||||
pnginfo.add_text(k, v)
|
||||
pnginfo.add_text("hf-job-id", "ec8380a6-2091-423a-b835-209420f99ee1")
|
||||
img.save(tmp_png_with_ai_metadata, pnginfo=pnginfo)
|
||||
r = identify(tmp_png_with_ai_metadata, check_visible=False)
|
||||
assert r.confidence == "high"
|
||||
|
||||
|
||||
# ── Visible-sparkle fallback (mocked detector) ──────────────────────
|
||||
|
||||
|
||||
|
||||
@@ -322,6 +322,44 @@ class TestRemoveAiMetadata:
|
||||
assert result == jpg_path
|
||||
assert jpg_path.exists()
|
||||
|
||||
def test_jpeg_output_is_high_quality(self, tmp_path):
|
||||
"""JPEG output uses high quality + 4:4:4 (no chroma subsampling), not the
|
||||
lossy PIL defaults (quality 75, 4:2:0) that visibly degrade the image."""
|
||||
from PIL.JpegImagePlugin import get_sampling
|
||||
|
||||
img = Image.new("RGB", (64, 64), color=(100, 150, 200))
|
||||
png_path = tmp_path / "source.png"
|
||||
img.save(png_path)
|
||||
|
||||
jpg_path = tmp_path / "output.jpg"
|
||||
remove_ai_metadata(png_path, jpg_path)
|
||||
|
||||
with Image.open(jpg_path) as out:
|
||||
assert get_sampling(out) == 0 # 4:4:4, no chroma subsampling
|
||||
# quality 95 quantization tables stay well below the q75 defaults
|
||||
# (whose max quant value is ~40+); q95 tops out around 12.
|
||||
assert max(max(t) for t in out.quantization.values()) <= 15
|
||||
|
||||
def test_webp_output_preserves_format_losslessly(self, tmp_path):
|
||||
"""A .webp output keeps the WebP format (not silently rewritten to PNG)
|
||||
and is pixel-identical to the source (lossless)."""
|
||||
import numpy as np
|
||||
|
||||
rng = np.random.default_rng(0)
|
||||
arr = rng.integers(0, 255, (48, 48, 3), dtype=np.uint8)
|
||||
src = Image.fromarray(arr, "RGB")
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("parameters", "ai stuff")
|
||||
png_path = tmp_path / "source.png"
|
||||
src.save(png_path, pnginfo=pnginfo)
|
||||
|
||||
webp_path = tmp_path / "output.webp"
|
||||
remove_ai_metadata(png_path, webp_path)
|
||||
|
||||
with Image.open(webp_path) as out:
|
||||
assert out.format == "WEBP"
|
||||
assert np.array_equal(np.asarray(out.convert("RGB")), arr)
|
||||
|
||||
def test_creates_parent_directories(self, tmp_path):
|
||||
img = Image.new("RGB", (32, 32))
|
||||
pnginfo = PngInfo()
|
||||
@@ -554,6 +592,88 @@ class TestAIGCLabel:
|
||||
assert "aigc_label" in meta
|
||||
assert "TC260" in meta["aigc_label"]
|
||||
|
||||
def _aigc_chunk_png(self, tmp_path: Path, producer: str = "doubao") -> Path:
|
||||
"""Doubao writes the TC260 object as a PNG ``tEXt`` chunk keyed ``AIGC``
|
||||
with raw JSON (no XMP, no namespaced marker)."""
|
||||
import json
|
||||
|
||||
p = tmp_path / "doubao_chunk.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text(
|
||||
"AIGC",
|
||||
json.dumps({"Label": "1", "ContentProducer": producer, "ProduceID": "abc123"}),
|
||||
)
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=pnginfo)
|
||||
return p
|
||||
|
||||
def test_parses_png_text_chunk_form(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
info = aigc_label(self._aigc_chunk_png(tmp_path))
|
||||
assert info is not None
|
||||
assert info["Label"] == "1"
|
||||
assert info["ContentProducer"] == "doubao"
|
||||
|
||||
def test_png_chunk_without_tc260_field_ignored(self, tmp_path: Path):
|
||||
"""A generic ``AIGC`` chunk with no TC260 field must not false-positive."""
|
||||
import json
|
||||
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
p = tmp_path / "unrelated.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("AIGC", json.dumps({"unrelated": "value"}))
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=pnginfo)
|
||||
assert aigc_label(p) is None
|
||||
|
||||
def test_has_ai_metadata_detects_png_chunk_form(self, tmp_path: Path):
|
||||
assert has_ai_metadata(self._aigc_chunk_png(tmp_path))
|
||||
|
||||
def test_remove_strips_png_chunk_form(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import aigc_label, remove_ai_metadata
|
||||
|
||||
out = tmp_path / "clean.png"
|
||||
remove_ai_metadata(self._aigc_chunk_png(tmp_path), out)
|
||||
assert aigc_label(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
|
||||
class TestHuggingFaceJob:
|
||||
"""HuggingFace-hosted job marker (``hf-job-id`` PNG text chunk)."""
|
||||
|
||||
def _hf_png(self, tmp_path: Path, job_id: str = "ec8380a6-2091-423a-b835-209420f99ee1") -> Path:
|
||||
p = tmp_path / "hfjob.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("hf-job-id", job_id)
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=pnginfo)
|
||||
return p
|
||||
|
||||
def test_returns_job_id(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import huggingface_job
|
||||
|
||||
assert huggingface_job(self._hf_png(tmp_path)) == "ec8380a6-2091-423a-b835-209420f99ee1"
|
||||
|
||||
def test_none_when_absent(self, tmp_clean_png):
|
||||
from remove_ai_watermarks.metadata import huggingface_job
|
||||
|
||||
assert huggingface_job(tmp_clean_png) is None
|
||||
|
||||
def test_has_ai_metadata_detects_hf_job(self, tmp_path: Path):
|
||||
assert has_ai_metadata(self._hf_png(tmp_path))
|
||||
|
||||
def test_get_ai_metadata_surfaces_hf_job(self, tmp_path: Path):
|
||||
meta = get_ai_metadata(self._hf_png(tmp_path))
|
||||
assert "huggingface_job" in meta
|
||||
assert "ec8380a6" in meta["huggingface_job"]
|
||||
|
||||
def test_remove_strips_hf_job(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import huggingface_job, remove_ai_metadata
|
||||
|
||||
out = tmp_path / "clean.png"
|
||||
remove_ai_metadata(self._hf_png(tmp_path), out)
|
||||
assert huggingface_job(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
|
||||
@pytest.mark.skipif(not (SAMPLES_DIR / "doubao-1.png").exists(), reason="doubao sample not present")
|
||||
class TestAIGCRealSample:
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Minimal local type stub for the untyped ``piexif`` package.
|
||||
|
||||
Covers only the API surface used in this repo (``load``/``dump`` plus the
|
||||
``ImageIFD`` tag ids), so strict pyright resolves the otherwise-Unknown values
|
||||
that ``piexif.load`` returns instead of carrying type debt. piexif ships no
|
||||
``py.typed`` marker; extend this stub if new piexif APIs are adopted.
|
||||
"""
|
||||
|
||||
from typing import Any
|
||||
|
||||
class ImageIFD:
|
||||
Software: int
|
||||
Make: int
|
||||
Artist: int
|
||||
ImageDescription: int
|
||||
|
||||
class ExifIFD: ...
|
||||
class GPSIFD: ...
|
||||
|
||||
def load(input_data: bytes | str, key_is_name: bool = ...) -> dict[str, Any]: ...
|
||||
def dump(exif_dict_original: dict[str, Any]) -> bytes: ...
|
||||
def insert(exif: bytes, image: str, new_file: str | None = ...) -> None: ...
|
||||
def remove(src: str, new_file: str | None = ...) -> None: ...
|
||||
Reference in new issue
Block a user