mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-14 13:30:55 +02:00
fix(metadata): strip bare AIGC in APP11 and AIGC in a standard PNG text chunk
Full-corpus strip audit surfaced 24 china_aigc survivors on real uploads:
- 19 JPEG carried the bare AIGC{...} blob in APP11 (0xEB). That marker's branch
in _jpeg_app_carries_ai only tested for a C2PA/JUMBF manifest and RETURNED, so
a bare AIGC there slipped past the generic AIGC check. The specific
C2PA(APP11)/XMP(APP1)/IPTC(APP13) checks now fall through to the generic
_is_aigc_exif_value drop, which runs for every APP marker they did not claim.
- PNG carried the {"AIGC":{...}} block in a STANDARD text chunk (Description).
_is_ai_key keeps that key, so removal now also drops a text value carrying an
AIGC block (_is_aigc_exif_value broadened to accept str; wired into the PNG
re-save value filter), in parity with aigc_label's detection.
Verified on the corpus: decodable china_aigc survivors 27 -> 0; the 3 remaining
are truncated files the strip fail-safe (v0.15.1) copies through by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
1 parent
72f6734b45
commit
2bdaa09e5f
3 files changed
+77
-25
No files matched your search
@@ -54,7 +54,7 @@ Compact map. The full per-module detail (design decisions, tuned thresholds, cal
|
||||
|
||||
- `noai/c2pa.py` — C2PA reading. `extract_c2pa_info(path)` uses the official **c2pa-python `Reader`** first (core dep, any container; `read_manifest_store_json` returns the WHOLE store JSON — active + ingredient manifests — so an AI marker on a parent manifest is seen), and falls back to the hand-rolled caBX/CBOR parser (`has_c2pa_metadata` / `extract_c2pa_chunk` / `_extract_c2pa_info_png`) for synthetic/partial blobs the validator rejects or a broken/absent wheel. The registry scan (issuer / source-type / SynthID / soft-binding) is shared by both paths via `_populate_registry_fields`, so the return-dict shape is identical. Do not reimplement chunk parsing; chunk reads are clamped to the remaining file size by design. `extract_c2pa_chunk`/`inject_c2pa_chunk` stay PNG-only (raw caBX bytes, test/extractor use).
|
||||
- `noai/constants.py` — the single `C2PA_AI_VENDORS` registry (+ `C2PA_SOFT_BINDINGS`) from which `C2PA_ISSUERS` / `SYNTHID_C2PA_ISSUERS` / `C2PA_IDENTITY_AI_ORGS` / `identify._ISSUER_PLATFORM` are all derived. Add a new vendor as one registry entry; never edit the derived dicts and never add inline. A vendor's `asserts_ai=True` flag means its mere presence asserts AI generation even without a `trainedAlgorithmicMedia` digital-source-type (a pure-generator brand with a distinctive issuer/generator string, e.g. **Dreamina** — ByteDance's international Jimeng brand, signed as "Bytedance Pte. Ltd." with a "Dreamina/x.y" claim generator and no source-type); NEVER set it for common-word issuers (Adobe/Google/OpenAI/Microsoft) that appear incidentally in unrelated bytes — those stay source-type-gated in `identify._attribute_platform`.
|
||||
- `metadata.py` — `scan_head(path)` is the shared (memoized) input for every C2PA/AIGC/IPTC byte scan; use it instead of `open().read(1MB)` for any new marker scan. Also home to `synthid_source`, `xai_signature`, `iptc_ai_system`, `aigc_label`, `huggingface_job`, `samsung_genai`, and `remove_ai_metadata` (fail-safe `strip_c2pa_boxes`). **`remove_ai_metadata` is the SINGLE metadata stripper** (the legacy PIL-re-encoding `noai/cleaner` was deleted; the diffusion core and the public `noai.remove_ai_metadata` re-export now point here). It strips **losslessly** per container: ISOBMFF (HEIC/AVIF/MP4) blanks tokens / strips boxes in place; **JPEG uses `_strip_jpeg_metadata_lossless`** — a marker-segment walk that drops the AI-bearing APP segments (C2PA APP11; XMP APP1 carrying C2PA, a China-AIGC token, OR an IPTC `digitalSourceType` / 2025.1 AI-disclosure marker; IPTC-IIM APP13) and scrubs AI EXIF tags via piexif, copying the entropy-coded scan verbatim so **the pixels are bit-identical** (no DCT re-encode). **Detection<->removal parity across every marker placement is load-bearing** — anything a scanner flags, the strip must reach, or a re-served file still reads as AI: (a) the APP1-XMP branch of `_jpeg_app_carries_ai` checks the IPTC marker sets too, not only C2PA/AIGC (the Instagram/MidJourney/Meta "Made with AI" `digitalSourceType` lives in XMP, not the APP13 IIM record); (b) a bare `AIGC{...}` / `{"AIGC":{...}}` block in a NON-standard APP segment (near JFIF on some China gens) is dropped via `_is_aigc_exif_value`, not only the APP11/APP1/APP13 cases; (c) the China TC260 `{"AIGC":{...}}` block in EXIF `UserComment`/`ImageDescription` is scrubbed by `_scrub_ai_exif` (Doubao producer + Tencent service-provider schemas); (d) the Samsung Galaxy AI `PhotoEditor_Re_Edit_Data` trailer past the JPEG EOI is truncated by `_strip_samsung_trailer` (and `samsung_genai` reads the file tail so a multi-MB photo's trailer past the 512 KB quick-scan window is still DETECTED). Pixels stay bit-identical throughout, so a `--strip-metadata` on a q100 removal output does NOT crush it back to q75; PNG/WebP re-saves are pixel-lossless (WebP written at cv2 lossless mode, quality 101 — quality 1-100 is lossy). **`remove_ai_metadata` is fail-safe on an undecodable image:** a truncated/corrupt file (PIL raises `OSError` decoding it; ~0.2% of real uploads) is copied through UNCHANGED rather than crashing a direct library caller (a web worker would 500 on a partial upload), mirroring `strip_c2pa_boxes` — we cannot strip what we cannot parse, but we never raise. Regression: `tests/test_metadata.py::TestHasAiMetadata::test_remove_ai_metadata_failsafe_on_truncated_png`. Regression: `tests/test_metadata.py::TestHasAiMetadata::{test_jpeg_metadata_strip_is_pixel_lossless, test_jpeg_strip_removes_iptc_marker_in_xmp}`, `TestSamsungGenai::{test_remove_strips_post_eoi_trailer, test_detects_trailer_past_scan_window}`, the AIGC-EXIF/bare-APP removal tests, and `tests/test_noai.py::TestISOBMFF::{test_blank_aigc_block_in_exif, test_blank_xai_signature_pair_in_exif}`. `exif_generator` matches a VALUE against `AI_GENERATOR_TOKENS` across EXIF `Software`/`Make`/`Artist`/`ImageDescription`, XMP `CreatorTool`, AND PNG `tEXt` chunks (`Software`/`Source`/`Title`/`Description` — NovelAI stamps there, not EXIF). **Detection and removal must stay in parity:** a generator that stamps an AI-shaped VALUE under a non-AI KEY (NovelAI's `Title`/`Source`) is dropped on removal by `_is_ai_value` (value-token match, mirrors `exif_generator`), NOT by `_is_ai_key` alone — else the cleaned file still reads as that generator. Add a new no-C2PA generator = one `AI_GENERATOR_TOKENS` entry (use a distinctive token, e.g. `reve.com` not bare `reve`); detection and removal then both follow. Regression: `tests/test_metadata.py::TestExifGenerator::{test_novelai_png_text_chunk_detected,test_novelai_removal_parity}`.
|
||||
- `metadata.py` — `scan_head(path)` is the shared (memoized) input for every C2PA/AIGC/IPTC byte scan; use it instead of `open().read(1MB)` for any new marker scan. Also home to `synthid_source`, `xai_signature`, `iptc_ai_system`, `aigc_label`, `huggingface_job`, `samsung_genai`, and `remove_ai_metadata` (fail-safe `strip_c2pa_boxes`). **`remove_ai_metadata` is the SINGLE metadata stripper** (the legacy PIL-re-encoding `noai/cleaner` was deleted; the diffusion core and the public `noai.remove_ai_metadata` re-export now point here). It strips **losslessly** per container: ISOBMFF (HEIC/AVIF/MP4) blanks tokens / strips boxes in place; **JPEG uses `_strip_jpeg_metadata_lossless`** — a marker-segment walk that drops the AI-bearing APP segments (C2PA APP11; XMP APP1 carrying C2PA, a China-AIGC token, OR an IPTC `digitalSourceType` / 2025.1 AI-disclosure marker; IPTC-IIM APP13) and scrubs AI EXIF tags via piexif, copying the entropy-coded scan verbatim so **the pixels are bit-identical** (no DCT re-encode). **Detection<->removal parity across every marker placement is load-bearing** — anything a scanner flags, the strip must reach, or a re-served file still reads as AI: (a) the APP1-XMP branch of `_jpeg_app_carries_ai` checks the IPTC marker sets too, not only C2PA/AIGC (the Instagram/MidJourney/Meta "Made with AI" `digitalSourceType` lives in XMP, not the APP13 IIM record); (b) a bare `AIGC{...}` / `{"AIGC":{...}}` block in ANY JPEG APP segment — the specific C2PA(APP11)/XMP(APP1)/IPTC(APP13) checks FALL THROUGH to a generic `_is_aigc_exif_value` drop, so a bare AIGC in APP11 (the common real-corpus placement, NOT a C2PA manifest) is caught, not swallowed by the C2PA-only 0xEB branch — plus the same AIGC block in a STANDARD **PNG text chunk** value (e.g. `Description`, which `_is_ai_key` keeps) is dropped on the value; (c) the China TC260 `{"AIGC":{...}}` block in EXIF `UserComment`/`ImageDescription` is scrubbed by `_scrub_ai_exif` (Doubao producer + Tencent service-provider schemas); (d) the Samsung Galaxy AI `PhotoEditor_Re_Edit_Data` trailer past the JPEG EOI is truncated by `_strip_samsung_trailer` (and `samsung_genai` reads the file tail so a multi-MB photo's trailer past the 512 KB quick-scan window is still DETECTED). Pixels stay bit-identical throughout, so a `--strip-metadata` on a q100 removal output does NOT crush it back to q75; PNG/WebP re-saves are pixel-lossless (WebP written at cv2 lossless mode, quality 101 — quality 1-100 is lossy). **`remove_ai_metadata` is fail-safe on an undecodable image:** a truncated/corrupt file (PIL raises `OSError` decoding it; ~0.2% of real uploads) is copied through UNCHANGED rather than crashing a direct library caller (a web worker would 500 on a partial upload), mirroring `strip_c2pa_boxes` — we cannot strip what we cannot parse, but we never raise. Regression: `tests/test_metadata.py::TestHasAiMetadata::test_remove_ai_metadata_failsafe_on_truncated_png`. Regression: `tests/test_metadata.py::TestHasAiMetadata::{test_jpeg_metadata_strip_is_pixel_lossless, test_jpeg_strip_removes_iptc_marker_in_xmp}`, `TestSamsungGenai::{test_remove_strips_post_eoi_trailer, test_detects_trailer_past_scan_window}`, the AIGC-EXIF/bare-APP removal tests, and `tests/test_noai.py::TestISOBMFF::{test_blank_aigc_block_in_exif, test_blank_xai_signature_pair_in_exif}`. `exif_generator` matches a VALUE against `AI_GENERATOR_TOKENS` across EXIF `Software`/`Make`/`Artist`/`ImageDescription`, XMP `CreatorTool`, AND PNG `tEXt` chunks (`Software`/`Source`/`Title`/`Description` — NovelAI stamps there, not EXIF). **Detection and removal must stay in parity:** a generator that stamps an AI-shaped VALUE under a non-AI KEY (NovelAI's `Title`/`Source`) is dropped on removal by `_is_ai_value` (value-token match, mirrors `exif_generator`), NOT by `_is_ai_key` alone — else the cleaned file still reads as that generator. Add a new no-C2PA generator = one `AI_GENERATOR_TOKENS` entry (use a distinctive token, e.g. `reve.com` not bare `reve`); detection and removal then both follow. Regression: `tests/test_metadata.py::TestExifGenerator::{test_novelai_png_text_chunk_detected,test_novelai_removal_parity}`.
|
||||
- `identify.py` — aggregates every locally-readable signal into one `ProvenanceReport`; `is_ai_generated` is True or None, never asserted False. `ProvenanceReport.ai_source_kind` exposes the C2PA digital-source-type split — `"generated"` (trainedAlgorithmicMedia, fully AI) vs `"enhanced"` (compositeWithTrainedAlgorithmicMedia, a real photo with an AI-composited region), else None — so a caller branches full-frame scrub vs region-targeted clean (see `noai/tiling.feather_region_composite` + `WatermarkRemover.remove_watermark(region=...)`). The sparkle provenance threshold is the SHARED `watermark_registry.GEMINI_SPARKLE_TRUST_CONF` (imported, not a private copy) so the provenance "is there a sparkle" verdict and the removal "take the sparkle" decision can never drift. `import identify` is deliberately light (lazy `noai/__init__`, fits a 512 MB host) — keep heavy imports out (the `watermark_registry` constant import stays light: engines are lazy there). Add capture-camera tokens to `_DEVICE_C2PA_PLATFORM` only when verified against a real C2PA file; editing-app/AI-device signer tokens go to `_SIGNER_C2PA_PLATFORM`; generator/issuer platforms to `C2PA_AI_VENDORS` in `constants.py`. The IPTC `digitalSourceType` **`algorithmicMedia`** (bare) is PROCEDURAL (an algorithm not trained on sampled data), NOT AI/ML generation, so it is deliberately absent from `IPTC_AI_MARKERS` — flagging it made `identify` assert AI + `has_invisible_target` True, scrubbing clean procedural content (it is a distinct token from `trainedAlgorithmicMedia`, so real "Made with AI" labels are unaffected; regression `test_metadata.py::...test_bare_algorithmic_media_not_flagged_ai`). Integrity-clash detection is high-precision by design (only hard generator stamps feed it, source-grouped independence). `_vendor_of` normalizes ByteDance/Canva/ElevenLabs/Black Forest Labs (as well as OpenAI/Google/... ) so their C2PA claims participate in the clash check; the generic **China TC260 AIGC label names no specific vendor**, so when a TC260-applying vendor (ByteDance, `_TC260_VENDORS`) is co-attributed the label is attributed to it (a legit Doubao image carrying its own TC260 label must NOT clash), while a NON-TC260 vendor next to a TC260 label still clashes as a laundering tell. Corpus-validated: adding the vendors introduced 0 new clashes on 5000 carriers.
|
||||
- `watermark_registry.py` — the single catalog of known visible watermarks (gemini / doubao / jimeng / samsung / jimeng_pill). **Removal is LOCALIZE -> FILL for every mark:** each mark is localized to a binary full-frame footprint mask (a `Localization`), then ONE shared, swappable fill inpaints that mask via `fill(image, mask, backend=...)` (delegates to `region_eraser.erase`). Reverse-alpha (the old `original = (wm - a*logo)/(1-a)` inversion of a captured alpha map + thin residual inpaint) is GONE for ALL marks; why it was dropped is recorded in `docs/module-internals.md`. Backends: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN ONNX, light, the memory-tight pick where LaMa will not fit), `lama` (big-LaMa ONNX, best quality, heavier, auto-preferred when a learned backend is available); `auto` = LaMa > MI-GAN > cv2, best available. The captured alpha maps (`scripts/visible_alpha_solve.py`) are still used to DETECT the marks and to shape the mask, but NOT for pixel recovery. **`--mark auto` removes EVERY detected mark in one pass** via `remove_auto_marks(image, *, sensitivity="auto", provenance=frozenset(), backend="auto")` (marks coexist -- a Jimeng-basic image has the top-left pill AND the bottom-right wordmark; a single-strongest pick would leave one). **Three orthogonal axes:** `backend` (the fill), `sensitivity` (how hard to trust a borderline mark: `auto`/`strict`/`assume_ai`, see the `Sensitivity` literal), and `provenance` (vendor keys metadata confirms -- the evidence that drives `auto`). **Perception / decision / action are separated:** `_build_candidates(image)` runs every detector at BOTH trust levels (strict + relaxed) and packages raw verdicts + features into `Candidate`s (no policy); the pure arbiter `decide(candidates, Context(sensitivity, provenance)) -> [Decision]` makes every keep/drop call (per-mark `resolve_relax` + the pill gate) with no image/IO, so it is unit-testable in isolation; then each winner is localized -> filled. Do NOT put policy back into the engines (the one exception, the Gemini FP gate, stays in `gemini_engine` because `identify` shares that confidence). `detect_marks(..., provenance=frozenset())` stays strict (identify verdict, precision over recall); `KnownMark.remove/detect/localize(..., provenance: bool)` take the already-resolved boolean. **How auto/assume decide (this is metadata-INDEPENDENT for recall):** the visual detectors are pixel-based and need no metadata; the recall gain comes from RELAXING the false-positive gate, not from metadata. `strict` never relaxes (clean images untouched); `auto` relaxes a mark only on same-product evidence -- metadata provenance for that vendor OR a confidently detected sibling mark of the SAME product (`_PRODUCT_OF`; Doubao and Jimeng are both bottom-right ByteDance but distinct products, so they do NOT cross-relax); `assume_ai` relaxes every mark (the caller asserts AI -- a metadata-stripped screenshot uploaded to a remover). Corpus finding: Gemini sparkle removal on Google-C2PA images is ~46% under `strict`/metadata-free `auto` and ~92% under `assume_ai` (recovering marks the vendor moved or re-rendered); the library CANNOT infer AI from a stripped image, so only the caller's `assume_ai` reaches the high recall there. A wrong relaxation just fills a small corner near-losslessly (the localize -> fill benign failure mode), which is what makes `assume_ai` acceptable. Metadata provenance mapping (feeds `auto`, read by `cli._visible_provenance`): Google/Gemini C2PA issuer -> gemini; China-AIGC (TC260) label -> doubao/jimeng; `samsung_genai` -> samsung. **The `jimeng_pill` is CAPTURE-LESS** (`pill_engine.py`): the top-left "AI生成" label has no captured alpha map, so it is detect-by-synthetic-silhouette; its footprint is a fixed top-left geometry box. Its weak edge-NCC detector (~7% raw false-fire) is gated in `remove_auto_marks` via **`_keep_pill`** (32k real-upload corpus validation 2026-07): the pill never rides on a **Doubao** detection, and has confirmation arms because metadata/intent confirms the platform, not pill presence. **(1) Bottom-right "★ 即梦AI" wordmark fired** — ~94% precise and survives **metadata-STRIPPED uploads** (screenshots / re-saves, ~61% of pills carry a detectable wordmark): remove **unrestricted**. **(2) TC260 metadata confirms Jimeng** (`"jimeng" in provenance`, no wordmark) **OR the caller asserts AI** (`sensitivity == "assume_ai"`) — the metadata-only arm is only ~27% precise and its false fires are **textured ceilings/walls that the fill visibly SMEARS**, so remove **only when the top-left footprint is flat enough for an invisible fill** (`pill_engine.footprint_is_flat`, median-Sobel texture ≤ `_FLAT_TEXTURE_MAX`) — the flatness guard holds even under `assume_ai`. This keeps real flat-scene pills (incl. metadata-only ones the wordmark misses) plus harmless flat false fires, and leaves the damaging textured false fires untouched. Do NOT drop the wordmark arm or loosenLine truncated
|
||||
- `gemini_engine.py` — visible Gemini-sparkle detector + localizer (cv2/numpy, no GPU): top-K size-weighted fusion candidate selection (`_SELECT_TOPK`), corner-promote, false-positive gate (the provenance prior relaxes the gate + lowers the trust threshold when a Google/Gemini C2PA issuer confirms the vendor). **White-core rescue:** the FP gate demotes a low-gradient match (soft edges), but a real FAINT sparkle also has soft edges -- so the gate keeps a low-grad match that is a strong (conf ≥ `_SPARKLE_KEEP_CONF` 0.52), bright (margin), near-WHITE-core sparkle (`_core_saturation` ≤ `_SPARKLE_WHITE_SAT` 0.20): a real sparkle core is white, a clean bright corner that shape-matches (sky/sun) is colored. This recovers ~14/20 metadata-stripped faint sparkles under the DEFAULT strict/auto (no flag, no metadata) at ~1.25% clean false-fire (baseline 0.55%); the ~0.51-scoring bright-bg FPs stay demoted (below 0.52). A learned classifier on the SAME features was measured WORSE than the tuned gate (2026-07 tier-1: MLP 86.7% recall vs 90.8% at equal FP), so the heuristic stays; a patch-CNN with richer features is the only lever left (roadmapped P2, low expected value -- the wall is fundamental). Detection scores the top-K size-weighted matches by full fusion (spatial+gradient+variance) and keeps the highest — NOT the raw-NCC argmax, which re-admits the tiny-patch FPs the size weight suppresses (the osachub 2026-06-12 sub-0.85 corner-sparkle regression; see `docs/module-internals.md`). Keep the 0.85 corner-promote NCC gate; a margin/chroma-gated lower promote was measured and REJECTED 2026-06-11 (~33% FP on non-Google content). Removal is localize -> fill: `footprint_mask` returns the sparkle footprint (the captured alpha thresholded LOW so the faint halo is included, then dilated by a sparkle-relative margin), and the shared `watermark_registry.fill` inpaints it. The captured alpha maps are used only to detect and to shape the mask, not for pixel recovery. `detect_sparkle_confidence` reuses a process-wide `_shared_engine()` singleton (lru_cache) — the engine holds only constant assets (captures, alpha maps, a precomputed 16..118 template ladder) and takes the image as an arg, so do NOT reconstruct `GeminiEngine()` per call: that reloaded assets + recomputed alpha maps + rebuilt the template cache on every one of ~34k `identify` calls (−24% on the sparkle path once made a singleton, output byte-identical). `detect_watermark`/`footprint_mask` guard `image.size == 0` before `to_bgr`, and return an empty (detected=False) result when no template scale fits (short side < 16 px), rather than dereferencing an empty candidate list.
|
||||
|
||||
@@ -751,8 +751,11 @@ def _is_aigc_exif_value(raw: object) -> bool:
|
||||
``UserComment`` / ``ImageDescription`` by China-served generators (Doubao's
|
||||
producer schema AND Tencent Cloud's service-provider schema, both keyed under
|
||||
``_TC260_FIELDS``). Gated on both the ``AIGC`` marker and a TC260 field so a
|
||||
coincidental token cannot false-drop a genuine caption/comment.
|
||||
coincidental token cannot false-drop a genuine caption/comment. Accepts a ``str``
|
||||
too (a PNG ``tEXt``/``iTXt`` value), not only EXIF bytes.
|
||||
"""
|
||||
if isinstance(raw, str):
|
||||
raw = raw.encode("latin-1", "ignore")
|
||||
if not isinstance(raw, (bytes, bytearray)):
|
||||
return False
|
||||
if b"AIGC" not in raw:
|
||||
@@ -951,25 +954,37 @@ def _jpeg_app_carries_ai(marker: int, payload: bytes) -> bool:
|
||||
APP11, an AI XMP packet in APP1, an IPTC "Made with AI" record in APP13). EXIF
|
||||
(APP1 ``Exif``) is NOT dropped here -- it is scrubbed tag-by-tag via piexif so
|
||||
genuine camera EXIF survives."""
|
||||
if marker == 0xEB: # APP11: C2PA / JUMBF manifest
|
||||
return c2pa_marker_in(payload) or b"jumb" in payload[:256].lower()
|
||||
if marker == 0xE1 and payload.startswith(b"http://ns.adobe.com/xap/"): # APP1 XMP
|
||||
return (
|
||||
if not (0xE0 <= marker <= 0xEF): # only APPn segments carry these
|
||||
return False
|
||||
# C2PA / JUMBF manifest (APP11).
|
||||
if marker == 0xEB and (c2pa_marker_in(payload) or b"jumb" in payload[:256].lower()):
|
||||
return True
|
||||
# AI XMP packet (APP1): C2PA, a China-AIGC token, or an IPTC digitalSourceType /
|
||||
# 2025.1 AI-disclosure marker (which live in XMP, not only the APP13 IIM record).
|
||||
if (
|
||||
marker == 0xE1
|
||||
and payload.startswith(b"http://ns.adobe.com/xap/")
|
||||
and (
|
||||
c2pa_marker_in(payload)
|
||||
or any(m in payload for m in AIGC_MARKERS)
|
||||
or any(m in payload for m in IPTC_AI_MARKERS) # digitalSourceType in XMP, not only APP13
|
||||
or any(m in payload for m in IPTC_AI_FIELD_MARKERS) # IPTC 2025.1 AI-disclosure fields
|
||||
or any(m in payload for m in IPTC_AI_MARKERS)
|
||||
or any(m in payload for m in IPTC_AI_FIELD_MARKERS)
|
||||
)
|
||||
if marker == 0xED: # APP13: Photoshop / IPTC
|
||||
return any(m in payload for m in IPTC_AI_MARKERS) or any(m in payload for m in IPTC_AI_FIELD_MARKERS)
|
||||
# A bare / wrapped China TC260 AIGC block (``AIGC{...}`` or ``{"AIGC":{...}}``) that
|
||||
# some China-served generators glue into a non-standard APP segment near the JFIF
|
||||
# header. ``aigc_label`` detects it anywhere in the scan head, so removal must drop
|
||||
# the carrying segment too (detection<->removal parity). Skip APP1-EXIF (0xE1
|
||||
# ``Exif``): its camera tags are scrubbed tag-by-tag via piexif, and the AIGC-in-
|
||||
# UserComment/ImageDescription placement is handled there, so it must not be dropped
|
||||
# wholesale here.
|
||||
if 0xE0 <= marker <= 0xEF and not (marker == 0xE1 and payload.startswith(b"Exif")):
|
||||
):
|
||||
return True
|
||||
# IPTC "Made with AI" record (APP13).
|
||||
if marker == 0xED and (
|
||||
any(m in payload for m in IPTC_AI_MARKERS) or any(m in payload for m in IPTC_AI_FIELD_MARKERS)
|
||||
):
|
||||
return True
|
||||
# A bare / wrapped China TC260 AIGC block (``AIGC{...}`` or ``{"AIGC":{...}}``) glued
|
||||
# into ANY APP segment -- some China gens use APP11, APP1, or a near-JFIF APPn. This
|
||||
# runs for every APP marker the specific checks above did NOT already claim, so a bare
|
||||
# AIGC in APP11 (not a C2PA manifest) is no longer missed by the 0xEB C2PA-only check.
|
||||
# ``aigc_label`` detects it anywhere, so removal must drop the carrying segment too
|
||||
# (detection<->removal parity). Skip APP1-EXIF (0xE1 ``Exif``): its camera tags are
|
||||
# scrubbed tag-by-tag via piexif, not dropped wholesale.
|
||||
if not (marker == 0xE1 and payload.startswith(b"Exif")):
|
||||
return _is_aigc_exif_value(payload)
|
||||
return False
|
||||
|
||||
@@ -1171,10 +1186,12 @@ def remove_ai_metadata(
|
||||
continue
|
||||
if _is_ai_key(key):
|
||||
continue
|
||||
# Drop a generic text chunk whose VALUE names an AI generator (NovelAI
|
||||
# writes its stamp into Title/Source under non-AI keys) -- keeps removal
|
||||
# in parity with exif_generator's value-based detection.
|
||||
if isinstance(value, str) and _is_ai_value(value):
|
||||
# Drop a text chunk whose VALUE names an AI generator (NovelAI writes its
|
||||
# stamp into Title/Source under non-AI keys) OR carries a China TC260 AIGC
|
||||
# block (some China gens put `{"AIGC":{...}}` in a STANDARD chunk like
|
||||
# Description, which _is_ai_key would keep) -- keeps removal in parity with
|
||||
# exif_generator / aigc_label's value-based detection.
|
||||
if isinstance(value, str) and (_is_ai_value(value) or _is_aigc_exif_value(value)):
|
||||
continue
|
||||
if key == "exif":
|
||||
with contextlib.suppress(Exception):
|
||||
|
||||
+38
-3
@@ -1067,16 +1067,19 @@ class TestAIGCLabel:
|
||||
assert aigc_label(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
def _aigc_bare_jpeg(self, tmp_path: Path, producer: str = "00119144030008867405X210002") -> Path:
|
||||
def _aigc_bare_jpeg(
|
||||
self, tmp_path: Path, producer: str = "00119144030008867405X210002", marker: bytes = b"\xff\xe9"
|
||||
) -> Path:
|
||||
"""Some China-served generators glue the TC260 label straight to its JSON
|
||||
as a bare ``AIGC{...}`` blob inside a JPEG APP segment (no ``"AIGC":``
|
||||
key wrapper, no PNG chunk, no namespaced XMP) -- seen near the JFIF
|
||||
header on real 2026-06 downloads."""
|
||||
header on real 2026-06 downloads. ``marker`` selects the APP segment
|
||||
(default APP9; the real corpus also uses APP11)."""
|
||||
p = tmp_path / "aigc_bare.jpg"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
raw = p.read_bytes()
|
||||
blob = b'AIGC{"Label":"1","ContentProducer":"' + producer.encode() + b'","ProduceID":"8F995586"}'
|
||||
segment = b"\xff\xe9" + (len(blob) + 2).to_bytes(2, "big") + blob # APP9
|
||||
segment = marker + (len(blob) + 2).to_bytes(2, "big") + blob
|
||||
p.write_bytes(raw[:2] + segment + raw[2:]) # splice after SOI
|
||||
return p
|
||||
|
||||
@@ -1103,6 +1106,38 @@ class TestAIGCLabel:
|
||||
assert aigc_label(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
def test_remove_strips_bare_aigc_in_app11(self, tmp_path: Path):
|
||||
"""Regression (real corpus, 19/27 survivors): the bare ``AIGC{...}`` blob lives
|
||||
in APP11 (0xEB) on many China gens. That marker's branch in _jpeg_app_carries_ai
|
||||
only checked for a C2PA/JUMBF manifest and RETURNED, so the AIGC blob slipped past
|
||||
the generic check -> survived the strip. The specific checks must fall through to
|
||||
the generic AIGC check."""
|
||||
from remove_ai_watermarks.metadata import aigc_label, remove_ai_metadata
|
||||
|
||||
src = self._aigc_bare_jpeg(tmp_path, marker=b"\xff\xeb") # APP11
|
||||
assert aigc_label(src) is not None
|
||||
out = tmp_path / "clean.jpg"
|
||||
remove_ai_metadata(src, out)
|
||||
assert aigc_label(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
def test_remove_strips_aigc_in_png_text_chunk(self, tmp_path: Path):
|
||||
"""Regression (real corpus, 2 survivors): the TC260 ``{"AIGC":{...}}`` block in a
|
||||
STANDARD PNG text chunk (Description) -- _is_ai_key keeps that key, so removal
|
||||
must also drop it on the VALUE carrying an AIGC block."""
|
||||
from PIL.PngImagePlugin import PngInfo
|
||||
|
||||
from remove_ai_watermarks.metadata import aigc_label, remove_ai_metadata
|
||||
|
||||
p = tmp_path / "aigc_desc.png"
|
||||
info = PngInfo()
|
||||
info.add_text("Description", '{"AIGC":{"Label":"1","ContentProducer":"00119144030008867405X210002"}}')
|
||||
Image.new("RGB", (32, 32)).save(p, pnginfo=info)
|
||||
assert aigc_label(p) is not None
|
||||
out = tmp_path / "clean.png"
|
||||
remove_ai_metadata(p, out)
|
||||
assert aigc_label(out) is None
|
||||
|
||||
def test_bare_aigc_without_tc260_field_ignored(self, tmp_path: Path):
|
||||
"""A bare ``AIGC{...}`` blob with no TC260 field must not false-positive."""
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
Reference in new issue
Block a user