diff --git a/README.md b/README.md index d23a020..2318637 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,8 @@ inspection includes the native TC260 `AIGC` tag in `moov.udta.meta.keys/ilst`, including a `moov` placed after the media payload, plus the QuickTime-form `meta` variants Doubao's iOS export writes (a bare `meta` box as a direct `moov` child, and a keyless `hdlr=mdir` metadata list). +Keyed `workflow` and `prompt` entries in the same metadata-list structure are +also detected and removed, including ComfyUI exports. MKV and WebM inspection reads the normative `Segment.Tags.Tag.SimpleTag` placement. AVI uses `LIST/INFO/AIGC`, while FLV uses `script.onMetaData.AIGC`. The non-ISOBMFF formats are remuxed with stream @@ -350,9 +352,9 @@ Visible mark support includes: - one calibrated Samsung Galaxy AI label variant. Metadata and provenance inspection covers C2PA, EXIF, XMP, IPTC, common -generator parameters, China TC260 AIGC labels, and several vendor specific -signals. Optional decoders add support for open DWT-DCT watermarks and Adobe -TrustMark. +generator parameters in image and video containers, China TC260 AIGC labels, +and several vendor specific signals. Optional decoders add support for open +DWT-DCT watermarks and Adobe TrustMark. The exact support matrix, including important locale and detector limits, lives in [supported signals](docs/supported-signals.md). The @@ -369,8 +371,9 @@ Visible removal follows three steps: Metadata removal uses format aware stripping. JPEG metadata removal preserves the encoded image scan instead of recompressing it. Native MP4/MOV TC260 values -are blanked without changing box sizes or media offsets. Other supported -containers use their corresponding metadata path. +and keyed generation parameters are blanked without changing box sizes or +media offsets. Other supported containers use their corresponding metadata +path. Invisible removal is different. It regenerates the image through a diffusion pipeline to disrupt pixel and frequency domain watermarks. This changes the diff --git a/docs/cli.md b/docs/cli.md index 12146fd..a733106 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -316,10 +316,11 @@ delegates to the same verified metadata scanner and stripper as the generic streams are not transcoded. For MP4 and MOV, this includes the native TC260 `AIGC` key and JSON value stored in `moov.udta.meta.keys/ilst`, plus the QuickTime-form `meta` variants Doubao's iOS export writes (a bare `meta` box -as a direct `moov` child, and a keyless `hdlr=mdir` metadata list). The inspector -seeks past a large `mdat` to find a tail `moov`. Removal stream-copies the +as a direct `moov` child, and a keyless `hdlr=mdir` metadata list). Keyed +`workflow` and `prompt` entries in the same metadata list are also inspected. +The inspector seeks past a large `mdat` to find a tail `moov`. Removal stream-copies the container in bounded chunks, converts supported top-level provenance boxes to -same-size `free` boxes, and blanks the TC260 key/value in place. Box sizes, +same-size `free` boxes, and blanks the TC260 or generation key/value in place. Box sizes, media offsets, and encoded stream bytes do not move; the result is atomically published only after the complete copy succeeds. diff --git a/docs/module-internals.md b/docs/module-internals.md index f2410f1..33632fe 100644 --- a/docs/module-internals.md +++ b/docs/module-internals.md @@ -222,7 +222,9 @@ alone is spaced out). The MP4/MOV/M4V/M4A removal path first validates the top-l box walk, then copies the source to a sibling temporary file in bounded chunks. Supported C2PA/JUMBF/AI-label boxes become same-size `free` boxes with blank payloads; TC260 removal changes the four-byte key to `free` and blanks only the -validated JSON value with same-length spaces. This preserves every box size, +validated JSON value with same-length spaces. Keyed generation metadata such +as `workflow` and `prompt` uses the same structural walk; removal blanks the +complete key and value in place. This preserves every box size, `stco`/`co64` offset, encoded stream byte, and source-sized memory bound. Publication is atomic, and a malformed top-level walk is copied unchanged. A generic `AIGC` key whose value has no TC260 field is ignored. @@ -541,10 +543,18 @@ Key contracts: asserting that the pixels were generated; explicit `aigc_info` discriminator values and Dreamina `exportType=generation` do assert AI origin. Ordinary Aweme, retouch, and `lv` editor exports are preserved. +- EXIF reads fall back to Pillow's decoded `getexif()` mapping when a container + exposes no ordinary `info["exif"]` blob. This covers ImageMagick PNG raw EXIF + profiles and keeps portable-record identification aligned with the file path. +- Structured `generationParams.modelName` JSON is reduced to its recognized + model name before attribution, so prompts and the surrounding JSON never + become a platform label. - ISOBMFF containers use [`_internal/isobmff.py`](../src/remove_ai_watermarks/_internal/isobmff.py). - Native MP4/MOV TC260 `AIGC` entries are read from `moov.udta.meta.keys/ilst` and blanked without changing box sizes. +- Native MP4/MOV `workflow` and `prompt` entries use that keyed metadata-list + reader and the same offset-preserving removal path. - Native MKV/WebM TC260 `AIGC` entries are read from `Segment.Tags.Tag.SimpleTag` and removed through the ffmpeg stream-copy path. - Native AVI and FLV TC260 entries are read from `LIST/INFO/AIGC` and diff --git a/docs/python-api.md b/docs/python-api.md index 48431d8..9192096 100644 --- a/docs/python-api.md +++ b/docs/python-api.md @@ -568,7 +568,8 @@ field records that source audio is copied if present and not watermark-verified. MP4/MOV inspection recognizes the native TC260 `AIGC` entry in `moov.udta.meta.keys/ilst` and the QuickTime-form `meta` variants Doubao's iOS -export writes; its removal preserves container size and encoded +export writes. It also recognizes keyed `workflow` and `prompt` generation +metadata in the same metadata-list structure; removal preserves container size and encoded stream bytes. MP4/MOV/M4V are copied in bounded chunks, so a large `mdat` is not loaded into memory; publication is atomic. MKV/WebM inspection recognizes the corresponding `Segment.Tags.Tag.SimpleTag` representation; its removal requires ffmpeg for a diff --git a/docs/supported-signals.md b/docs/supported-signals.md index b927e97..1393b61 100644 --- a/docs/supported-signals.md +++ b/docs/supported-signals.md @@ -80,17 +80,20 @@ The learned backends download model files on first use. The inspection and stripping code handles signals in these groups: - C2PA Content Credentials and supported cloud manifest references; -- EXIF and XMP generator fields; +- EXIF and XMP generator fields, including structured + `generationParams.modelName` values from supported generators; - exact app-export provenance and AIGC disclosures from supported ByteDance-family products, with product-only provenance excluded from the generated-image verdict; - IPTC AI disclosure fields; -- PNG text chunks and embedded generation parameters; +- PNG text chunks and embedded generation parameters, including keyed + `workflow` and `prompt` tags in MP4/MOV metadata lists; - China TC260 AIGC labels in supported image placements and the normative MP4/MOV `moov.udta.meta.keys/ilst`, MKV/WebM `Segment.Tags.Tag.SimpleTag`, AVI `LIST/INFO/AIGC`, and FLV `script.onMetaData.AIGC` placements; -- xAI and Grok EXIF signature fields; +- xAI and Grok EXIF signature fields in ordinary EXIF and ImageMagick PNG raw + EXIF profiles; - Samsung AI editing markers; - Hugging Face job metadata; - open Stable Diffusion style DWT-DCT watermarks with the `detect` extra; diff --git a/docs/watermarking-landscape.md b/docs/watermarking-landscape.md index 37ad375..378ef37 100644 --- a/docs/watermarking-landscape.md +++ b/docs/watermarking-landscape.md @@ -27,6 +27,9 @@ provider. For AI claims, exact product mappings in `claim_generator` therefore take precedence over issuer attribution. Supported mappings include Higgsfield AI, Recraft, Topaz Labs Image API, and TikTok Ad Creative Toolbox; an unknown claim generator still falls back to the certificate issuer. +Exact claim-generator mappings also cover `Grok Imagine` as xAI Grok Imagine +and `Suno` as Suno. Both can carry a valid AI source claim without a separately +readable issuer, so issuer-only attribution leaves the platform blank. **OpenRouter raster metadata survey (2026-09-02):** a paid Recraft V3 API output carried a valid C2PA claim with `claim_generator` `recraft.ai`, a @@ -37,11 +40,11 @@ container metadata. Their absence is evidence about those exact files and API routes, not a claim that every Krea or Riverflow export is metadata-free. **ByteDance Volcano Engine (Volcengine)** — the cloud behind Doubao / Jimeng — signs its AI image output with a cert from `certificate_center@volcengine.com` + `trainedAlgorithmicMedia` (issuer `b"volcengine"` → "ByteDance (Volcano Engine)", platform "ByteDance Volcano Engine"); note this is the C2PA-signed surface, distinct from the XMP/PNG TC260 `AIGC` label Doubao also uses. ByteDance's **international brand (BytePlus / Seedream / Seededit)** signs the same content as **"Byteplus Pte. Ltd."**. The bare `volcengine` needle missed it, so BytePlus output was mis-attributed to "Adobe Firefly" through an incidental "Adobe XMP" toolkit string. Issuer `b"Byteplus"` maps directly to "BytePlus (ByteDance)". ByteDance's consumer app **Dreamina** (the international Jimeng brand) signs as **"Bytedance Pte. Ltd."** with a `Dreamina/x.y` claim generator but, unlike the Volcano Engine surface, ships **no `trainedAlgorithmicMedia`**. Issuer `b"Dreamina"` maps to "ByteDance Dreamina" with **`asserts_ai=True`**. Registering the broader **issuer** `b"Bytedance Pte"` was deliberately avoided because that same entity also signs non-AI CapCut edits; keying on the `Dreamina` generator token is precise. -- **EXIF/XMP/PNG-text generator tag (caught by `exif_generator`):** **Ideogram** writes EXIF `Make="Ideogram AI"` (collected 2026-05-24 — no C2PA, no SynthID, no imwatermark; the Make tag is the only signal). Additional verified generator stamps include **NovelAI** (`Software`, `Source`, and `Title` PNG text chunks), **Reve** (`Software` or XMP `CreatorTool` = `reve.com`), and **Aphrodite AI** (`Make` or `Software` = `Aphrodite AI`). +- **EXIF/XMP/PNG-text generator tag (caught by `exif_generator`):** **Ideogram** writes EXIF `Make="Ideogram AI"` (collected 2026-05-24 — no C2PA, no SynthID, no imwatermark; the Make tag is the only signal). Additional verified generator stamps include **NovelAI** (`Software`, `Source`, and `Title` PNG text chunks), **Reve** (`Software` or XMP `CreatorTool` = `reve.com`), and **Aphrodite AI** (`Make` or `Software` = `Aphrodite AI`). A JSON `generationParams.modelName` value is parsed structurally and reduced to a recognized model name, rather than exposing its surrounding prompt JSON as a platform label. - **App-export provenance and AIGC JSON:** supported ByteDance-family exports can place a JSON object in EXIF `ImageDescription` or `UserComment`, independently of C2PA or TC260. Exact `product` values for Doubao, Xinghui, and Dreamina are removable product provenance, but do not alone prove that the pixels were generated. Dreamina additionally requires `exportType=generation` for that verdict. A nested Aweme `aigc_type=1` or private ByteDance `aigc_label_type=1` / `2` is an AIGC disclosure; `0` is inconclusive and can occur on a Dreamina generation export. Plain Aweme, retouch, and `lv` exports are preserved. The lower-case private field is deliberately not interpreted as the normative TC260 `Label`, whose values `1` / `2` / `3` mean generated / possibly generated / suspected generated under [GB 45438-2025](https://www.tc260.org.cn/upload/2025-03-15/1742009439794081593.pdf). -- **xAI / Grok — its own EXIF signature scheme, NOT C2PA (DETECTED by `metadata.xai_signature`, built 2026-05-26).** +- **xAI / Grok legacy JPEG EXIF signature scheme (DETECTED by `metadata.xai_signature`, built 2026-05-26).** -Grok JPEG downloads (Aurora model) carry **no C2PA, no XMP, no SynthID, no IPTC** — only EXIF `Artist` = a UUID and EXIF `ImageDescription` = `Signature: ` (a crypto signature, unverifiable locally without xAI's public key). This empirically kills the earlier unverified "xAI signs C2PA as xAI" lead — xAI is not even a C2PA member. `exif_generator` misses it (neither field holds an `AI_GENERATOR_TOKENS` token), so a dedicated detector `xai_signature(path)` matches the pair (`ImageDescription ~ ^Signature: [A-Za-z0-9+/=]{64,}` AND UUID `Artist`); wired into `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify` (signal `xai_signature`, platform "xAI (Grok / Aurora)"). +The sampled Grok JPEG downloads from the Aurora route carry **no C2PA, no XMP, no SynthID, no IPTC** — only EXIF `Artist` = a UUID and EXIF `ImageDescription` = `Signature: ` (a crypto signature, unverifiable locally without xAI's public key). This is route-specific, not a product-wide no-C2PA claim: newer Grok Imagine output can carry a valid AI-source C2PA claim whose `claim_generator` is `Grok Imagine`. `exif_generator` misses the legacy pair (neither field holds an `AI_GENERATOR_TOKENS` token), so a dedicated detector `xai_signature(path)` matches it (`ImageDescription ~ ^Signature: [A-Za-z0-9+/=]{64,}` AND UUID `Artist`); wired into `has_ai_metadata`, `get_ai_metadata` (key `xai_signature`), and `identify` (signal `xai_signature`, platform "xAI (Grok / Aurora)"). **Format confirmed stable across n=3 genuine generations:** exactly three EXIF tags (`Artist`, `ExifOffset`, `ImageDescription`), `Signature:` prefix constant, base64 payload 300-1004 chars. Two capture facts: (a) the `Artist` UUID **equals the public image id** in the asset URL (`https://imagine-public.x.ai/imagine-public/images/.jpg`), so it is NOT a private per-user secret — only the `Signature` blob is; (b) the Grok web-UI image is a re-encoded **WebP with no signature** — the EXIF survives only in the *original* JPEG (download button or that public tokenless URL), which is why screenshots / re-encodes are metadata-stripped. A real fixture `data/fixtures/provenance/grok-1.jpg` plus **synthetic** JPEG fixtures (fake UUID + fake `Signature:` blob) cover the detector; never add a real Grok image carrying private content (the repo is public). @@ -49,7 +52,10 @@ Grok JPEG downloads (Aurora model) carry **no C2PA, no XMP, no SynthID, no IPTC* JPEG EXIF, which deletes the xAI Signature and UUID Artist pair plus supported AI generator values while retaining unrelated camera and editor EXIF. The shared `xai_signature_pair` helper is the single source of truth for the -pair. On the ISOBMFF path, `blank_ai_exif_tokens` provides the corresponding +pair. Converted PNGs can retain the same pair in an ImageMagick raw EXIF +profile; Pillow exposes that form through `getexif()` even when `info["exif"]` +is absent, so detection, removal, and portable records share that fallback. On +the ISOBMFF path, `blank_ai_exif_tokens` provides the corresponding in-place scrub for supported EXIF values, TC260 AIGC blocks, and the xAI pair. - **China TC260 AIGC label (caught by `AIGC_MARKERS` / `metadata.aigc_label`, surfaced by `identify` as the `aigc` signal):** China-served generators embed an XMP `{"Label":"1","ContentProducer":...}` block — China's mandatory AI-content labeling (TC260 namespace `tc260.org.cn/ns/AIGC`). The label says only "this is AI", but its `ContentProducer` names the signing entity — `001` + `1` + an 18-char Unified Social Credit Code + a 5-digit product suffix, normalized by `metadata.uscc_of`, or for a few generators a bare product name. `KnownMark.tc260_producer_codes` maps the codes settled per vendor by `scripts/vendor_cohort_harvest.py` to registry mark keys, so an AIGC image relaxes only the detector of the product it actually carries; an unmapped or absent producer leaves every product strict. The same mapping supplies the manufacturer for provenance-conflict checks. A code identifies a legal entity, not necessarily one brand, so a hosting or aggregating platform that signs for several apps is a recall bet rather than a proof. @@ -88,7 +94,10 @@ payloads. Removal remuxes either container through ffmpeg with stream copy. Native TC260 MP4/MOV tags do not live in those top-level provenance boxes. `tc260_aigc_payloads` separately seeks through `moov.udta.meta.keys/ilst`, so -the normative tag is also found when a large `mdat` precedes `moov`. +the normative tag is also found when a large `mdat` precedes `moov`. The parser +canonicalizes known fields across PascalCase and lowerCamelCase serializations. +The same keyed metadata-list walk exposes generation `workflow` and `prompt` +entries, including ComfyUI exports, and blanks them in place during removal. **Meta-box XMP and EXIF removal are handled in place:** an AI-label XMP packet stored as a meta-box `mime` item is blanked by diff --git a/scripts/corpus_gap_scan.py b/scripts/corpus_gap_scan.py index 219d6d2..0ba8ad0 100644 --- a/scripts/corpus_gap_scan.py +++ b/scripts/corpus_gap_scan.py @@ -29,6 +29,7 @@ from __future__ import annotations import csv import logging +import re from collections import Counter from datetime import date from importlib.metadata import version @@ -40,7 +41,14 @@ from _plain_console import Console, Table from remove_ai_watermarks.identify import _metadata_region as identify_metadata_region from remove_ai_watermarks.identify import identify -from remove_ai_watermarks.metadata import scan_head +from remove_ai_watermarks.metadata import ( + IPTC_AI_FIELD_MARKERS, + IPTC_AI_MARKERS, + aigc_label_from_metadata, + c2pa_marker_in, + samsung_genai_in, + scan_head, +) log = logging.getLogger(__name__) console = Console() @@ -51,22 +59,10 @@ console = Console() # below is the primary guard, this list is the second. Group: C2PA/JUMBF infra, # AI source-type / labeling schemes, and distinctive generator name strings. MARKERS: tuple[bytes, ...] = ( - # C2PA / JUMBF infrastructure and AI source-type / labeling schemes. - b"c2pa", - b"jumbf", - b"contentauth", + # AI source-type / labeling schemes that are meaningful by presence. b"trainedAlgorithmicMedia", - b"digitalSourceType", - b'"AIGC"', - b"", - b"TC260:AIGC", - b"tc260.org.cn", b"AISystemUsed", - b"SynthID", b"hf-job-id", - b"genAIType", - b"PhotoEditor_Re_Edit", - b"Signature:", # Distinctive multi-word generator strings only. Bare single words (Luma, # Gemini, Sora, ...) are omitted: they collide with unrelated metadata prose # (e.g. "Luma" in Lightroom's EnhanceDenoiseLumaAmount), defeating precision. @@ -80,12 +76,11 @@ MARKERS: tuple[bytes, ...] = ( b"Adobe Firefly", b"Black Forest", b"volcengine", - b"Doubao", - b"\xe8\xb1\x86\xe5\x8c\x85", b"Nano Banana", b"Stability AI", - b"Samsung Galaxy", ) +_XAI_SIGNATURE_RE = re.compile(rb"Signature:\s*[A-Za-z0-9+/=]{64,}") +_UUID_RE = re.compile(rb"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", re.IGNORECASE) REPORT_FIELDS: tuple[str, ...] = ( "path", @@ -122,9 +117,20 @@ def _row(rep, *, path: str, suffix: str, lib_version: str) -> dict[str, str]: # def _marker_hits(region: bytes) -> list[str]: - """Return marker labels found case-insensitively in one metadata region.""" + """Return high-precision marker labels found in one metadata region.""" folded = region.lower() - return sorted({marker.decode("latin-1", "replace") for marker in MARKERS if marker.lower() in folded}) + hits = {marker.decode("latin-1", "replace") for marker in MARKERS if marker.lower() in folded} + if c2pa_marker_in(region): + hits.add("C2PA/JUMBF") + if aigc_label_from_metadata(region) is not None: + hits.add("TC260 AIGC") + if any(marker in region for marker in IPTC_AI_MARKERS + IPTC_AI_FIELD_MARKERS): + hits.add("IPTC AI disclosure") + if samsung_genai_in(region) is not None: + hits.add("Samsung genAIType") + if _XAI_SIGNATURE_RE.search(region) and _UUID_RE.search(region): + hits.add("xAI signature pair") + return sorted(hits) def _marker_hits_for_path(path: Path) -> list[str]: diff --git a/src/remove_ai_watermarks/_internal/constants.py b/src/remove_ai_watermarks/_internal/constants.py index 320bc1c..328a10e 100644 --- a/src/remove_ai_watermarks/_internal/constants.py +++ b/src/remove_ai_watermarks/_internal/constants.py @@ -119,6 +119,8 @@ C2PA_IDENTITY_AI_ORGS = frozenset(vendor.org for vendor in C2PA_AI_VENDORS if ve # Keep this attribution beside the issuer registry so every C2PA consumer has one # canonical source rather than maintaining a derived product map in identify.py. C2PA_CLAIM_GENERATOR_PLATFORMS: tuple[tuple[str, str], ...] = ( + ("grok imagine", "xAI Grok Imagine"), + ("suno", "Suno"), ("adobe_firefly", "Adobe Firefly"), ("firefly", "Adobe Firefly"), ("dreamina", "ByteDance Dreamina"), @@ -211,6 +213,7 @@ AI_GENERATOR_TOKENS = frozenset( "invokeai", "imagen", "gpt-image", + "nano banana", "nightcafe", "ideogram", "leonardo", diff --git a/src/remove_ai_watermarks/_internal/isobmff.py b/src/remove_ai_watermarks/_internal/isobmff.py index 2ce9355..90f66ab 100644 --- a/src/remove_ai_watermarks/_internal/isobmff.py +++ b/src/remove_ai_watermarks/_internal/isobmff.py @@ -43,6 +43,7 @@ from remove_ai_watermarks.metadata import ( C2PA_UUID, IPTC_AI_FIELD_MARKERS, IPTC_AI_MARKERS, + LOCAL_GENERATION_METADATA_KEYS, MAX_TC260_VALUE_BYTES, parse_tc260_aigc_json, ) @@ -152,12 +153,12 @@ def iter_file_boxes( pos = box_end -def _tc260_key_indices( +def _metadata_key_indices( stream: BinaryIO, payload_off: int, box_end: int, -) -> dict[int, tuple[int, int]]: - """Map every exact ``AIGC`` key index to its byte span.""" +) -> dict[int, tuple[bytes, tuple[int, int]]]: + """Map bounded metadata key indices to their names and byte spans.""" if payload_off + 8 > box_end: return {} stream.seek(payload_off) @@ -166,7 +167,7 @@ def _tc260_key_indices( return {} entry_count = struct.unpack(">I", prefix[4:8])[0] pos = payload_off + 8 - found: dict[int, tuple[int, int]] = {} + found: dict[int, tuple[bytes, tuple[int, int]]] = {} for index in range(1, entry_count + 1): if pos + 8 > box_end: return {} @@ -179,10 +180,12 @@ def _tc260_key_indices( if entry_size < 8 or entry_end > box_end: return {} name_start = pos + 8 - if entry_end - name_start == 4: + name_size = entry_end - name_start + if 0 < name_size <= 128: stream.seek(name_start) - if stream.read(4) == b"AIGC": - found[index] = (name_start, entry_end) + name = stream.read(name_size) + if len(name) == name_size: + found[index] = (name, (name_start, entry_end)) pos = entry_end return found @@ -248,12 +251,23 @@ def _tc260_aigc_regions( iOS export writes, where the JSON sits in a bare ``ilst`` data item with no key name to blank. """ - regions: list[tuple[tuple[int, int] | None, int, int, bytes]] = [] + return [ + (key_span, value_start, value_end, value) + for key_name, key_span, value_start, value_end, value in _metadata_items(stream, file_size) + if (key_name is None or key_name == b"AIGC") and parse_tc260_aigc_json(value) is not None + ] + + +def _metadata_items( + stream: BinaryIO, + file_size: int, +) -> Iterator[tuple[bytes | None, tuple[int, int] | None, int, int, bytes]]: + """Yield bounded keyed or keyless metadata-list values from a media file.""" for _moov_start, moov_end, moov_type, moov_payload in iter_file_boxes(stream, 0, file_size): if moov_type != b"moov": continue for meta_payload, meta_end in _iter_tc260_meta_boxes(stream, moov_payload, moov_end): - keys: dict[int, tuple[int, int]] = {} + keys: dict[int, tuple[bytes, tuple[int, int]]] = {} ilst_boxes: list[tuple[int, int]] = [] keyed = False for _child_start, child_end, child_type, child_payload in _meta_child_boxes( @@ -263,7 +277,7 @@ def _tc260_aigc_regions( ): if child_type == b"keys": keyed = True - keys.update(_tc260_key_indices(stream, child_payload, child_end)) + keys.update(_metadata_key_indices(stream, child_payload, child_end)) elif child_type == b"ilst": ilst_boxes.append((child_payload, child_end)) if not ilst_boxes: @@ -275,8 +289,8 @@ def _tc260_aigc_regions( ilst_end, ): index = int.from_bytes(item_type, "big") - key_span = keys.get(index) - if keyed and key_span is None: + key = keys.get(index) + if keyed and key is None: # A keyed (ISO) meta box maps items through ``keys``; # an unmapped index is not an AIGC entry, and reading # its value would pull arbitrary metadata (e.g. cover @@ -284,6 +298,7 @@ def _tc260_aigc_regions( # keyless QuickTime list falls through to content # validation below. continue + key_name, key_span = key if key is not None else (None, None) for _data_start, data_end, data_type, data_payload in iter_file_boxes( stream, item_payload, @@ -295,9 +310,8 @@ def _tc260_aigc_regions( continue stream.seek(value_start) value = stream.read(value_size) - if len(value) == value_size and parse_tc260_aigc_json(value) is not None: - regions.append((key_span, value_start, data_end, value)) - return regions + if len(value) == value_size: + yield key_name, key_span, value_start, data_end, value def tc260_aigc_payloads(path: str | Path, *, strict: bool = False) -> tuple[bytes, ...]: @@ -315,6 +329,42 @@ def tc260_aigc_payloads(path: str | Path, *, strict: bool = False) -> tuple[byte return () +def _ai_metadata_regions( + stream: BinaryIO, + file_size: int, +) -> list[tuple[tuple[int, int], int, int, str, bytes]]: + """Locate keyed generation metadata without reading media payloads.""" + regions: list[tuple[tuple[int, int], int, int, str, bytes]] = [] + for name_bytes, key_span, value_start, value_end, value in _metadata_items(stream, file_size): + if name_bytes is None or key_span is None: + continue + try: + name = name_bytes.decode("utf-8") + except UnicodeDecodeError: + continue + if name.lower() in LOCAL_GENERATION_METADATA_KEYS: + regions.append((key_span, value_start, value_end, name, value)) + return regions + + +def ai_metadata_tags(path: str | Path, *, strict: bool = False) -> dict[str, str]: + """Read keyed AI-generation metadata from an ISOBMFF container.""" + try: + with open(path, "rb") as stream: + if not is_isobmff(stream.read(8)): + return {} + stream.seek(0, 2) + file_size = stream.tell() + return { + name: value.decode("utf-8", "replace") + for _key_span, _value_start, _value_end, name, value in _ai_metadata_regions(stream, file_size) + } + except OSError: + if strict: + raise + return {} + + def blank_tc260_aigc_tags(data: bytes) -> tuple[bytes, int]: """Blank native TC260 values in place while preserving every box offset. @@ -576,7 +626,9 @@ def strip_isobmff_media_file( max_scan=max_box_scan, ) tc260_regions = _tc260_aigc_regions(stream, file_size) if targets is not None else [] + ai_metadata_regions = _ai_metadata_regions(stream, file_size) if targets is not None else [] tc260_key_spans = {region[0] for region in tc260_regions if region[0] is not None} + ai_metadata_key_spans = {region[0] for region in ai_metadata_regions} with atomic_video_output(output_path) as temporary_path: with source_path.open("rb") as source_stream, temporary_path.open("r+b") as temporary: @@ -591,6 +643,10 @@ def strip_isobmff_media_file( temporary.seek(key_span[0]) temporary.write(b"free") _overwrite_range(temporary, value_start, value_end, byte=b" ") + for key_span, value_start, value_end, _name, _value in ai_metadata_regions: + temporary.seek(key_span[0]) + temporary.write(b"free" + b" " * (key_span[1] - key_span[0] - 4)) + _overwrite_range(temporary, value_start, value_end, byte=b" ") temporary.flush() os.fsync(temporary.fileno()) shutil.copymode(source_path, temporary_path) @@ -601,7 +657,7 @@ def strip_isobmff_media_file( source_path, ) return 0, 0 - return len(targets), len(tc260_key_spans) + return len(targets), len(tc260_key_spans | ai_metadata_key_spans) def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]: diff --git a/src/remove_ai_watermarks/identify.py b/src/remove_ai_watermarks/identify.py index 358043c..e8721e7 100644 --- a/src/remove_ai_watermarks/identify.py +++ b/src/remove_ai_watermarks/identify.py @@ -51,6 +51,7 @@ from remove_ai_watermarks.metadata import ( AIGC_MARKERS, IPTC_AI_FIELD_MARKERS, IPTC_AI_MARKERS, + LOCAL_GENERATION_METADATA_KEYS, aigc_label, aigc_label_from_metadata, c2pa_cloud_manifest_in, @@ -106,9 +107,7 @@ _ISSUER_PLATFORM: tuple[tuple[str, str], ...] = tuple( # PNG-text / EXIF keys that indicate a local diffusion pipeline (vs. a hosted # platform's C2PA). Subset of AI_METADATA_KEYS; excludes the C2PA/Software keys. -_LOCAL_GEN_KEYS = frozenset( - AI_METADATA_KEYS & {"parameters", "prompt", "negative_prompt", "workflow", "comfyui", "invokeai_metadata", "dream"} -) +_LOCAL_GEN_KEYS = LOCAL_GENERATION_METADATA_KEYS _STRIP_CAVEAT = ( "Absence of metadata is not proof the image is clean: C2PA, EXIF, and PNG " diff --git a/src/remove_ai_watermarks/metadata.py b/src/remove_ai_watermarks/metadata.py index 97d33f9..238b7e0 100644 --- a/src/remove_ai_watermarks/metadata.py +++ b/src/remove_ai_watermarks/metadata.py @@ -59,6 +59,9 @@ AI_METADATA_KEYS: frozenset[str] = frozenset( "Software", ] ) +LOCAL_GENERATION_METADATA_KEYS: frozenset[str] = frozenset( + AI_METADATA_KEYS & {"parameters", "prompt", "negative_prompt", "workflow", "comfyui", "invokeai_metadata", "dream"} +) AI_KEYWORDS: tuple[str, ...] = ( "stable_diffusion", @@ -169,6 +172,7 @@ TC260_AIGC_FIELDS: frozenset[str] = frozenset( "ServiceUser", } ) +_TC260_CANONICAL_FIELDS: dict[str, str] = {field.casefold(): field for field in TC260_AIGC_FIELDS} MAX_TC260_VALUE_BYTES = 1024 * 1024 @@ -197,7 +201,10 @@ def parse_tc260_aigc_json(value: bytes) -> dict[str, str] | None: return None if not isinstance(parsed, dict): return None - fields = {str(key): str(item) for key, item in cast("dict[object, object]", parsed).items()} + fields = { + _TC260_CANONICAL_FIELDS.get(str(key).casefold(), str(key)): str(item) + for key, item in cast("dict[object, object]", parsed).items() + } return fields if TC260_AIGC_FIELDS & fields.keys() else None @@ -877,11 +884,47 @@ def generator_from_metadata(candidates: Iterable[str], scan: bytes = b"") -> str for value in itertools.chain(candidates, creator_tools): if app_generator := app_generator_from_metadata(value): return app_generator + if structured_generator := _structured_generator_from_metadata(value): + return structured_generator if any(token in value.lower() for token in AI_GENERATOR_TOKENS): return value.strip() return None +def _structured_generator_from_metadata(value: str) -> str | None: + """Extract a bounded generator name without returning its surrounding JSON.""" + from remove_ai_watermarks._internal.constants import AI_GENERATOR_TOKENS + + if len(value) > MAX_TC260_VALUE_BYTES: + return None + try: + parsed = json.loads(value) + except ValueError: + return None + if not isinstance(parsed, dict): + return None + generation_params = cast("dict[object, object]", parsed).get("generationParams") + if not isinstance(generation_params, dict): + return None + model_name = cast("dict[object, object]", generation_params).get("modelName") + if not isinstance(model_name, str) or not model_name.strip() or len(model_name) > 200: + return None + normalized = model_name.strip() + return normalized if any(token in normalized.lower() for token in AI_GENERATOR_TOKENS) else None + + +def exif_bytes_from_image(image: Any) -> bytes | None: + """Return EXIF bytes from either a normal EXIF field or a decoded raw profile.""" + value = image.info.get("exif") + if isinstance(value, bytes) and value: + return value + with contextlib.suppress(Exception): + exif = image.getexif() + if exif: + return exif.tobytes() + return None + + _APP_PROVENANCE_PRODUCTS: dict[str, str] = { "doubao": "ByteDance Doubao", "xinghui": "ByteDance Xinghui", @@ -894,12 +937,17 @@ _APP_AIGC_LABEL_RE = re.compile(r'"aigc_label_type"\s*:\s*[12](?=\s*[,}])', re.I _APP_AIGC_TYPE_RE = re.compile(r'"aigc_type"\s*:\s*1(?=\s*[,}])', re.IGNORECASE) -def _normalized_app_metadata(value: str | bytes) -> str: - text = value.decode("latin-1", "ignore") if isinstance(value, bytes) else value +def _normalized_app_metadata(value: object) -> str: + if isinstance(value, bytes): + text = value.decode("latin-1", "ignore") + elif isinstance(value, str): + text = value + else: + return "" return text.replace('\\"', '"') -def _app_metadata_evidence(value: str | bytes) -> tuple[str | None, str | None]: +def _app_metadata_evidence(value: object) -> tuple[str | None, str | None]: """Return removable product provenance and stronger AI-origin evidence.""" normalized = _normalized_app_metadata(value) products = tuple(match.group(1).lower() for match in _APP_PRODUCT_RE.finditer(normalized)) @@ -962,7 +1010,7 @@ def exif_generator(image_path: Path) -> str | None: with Image.open(image_path) as img: info = img.info - exif_bytes = info.get("exif") + exif_bytes = exif_bytes_from_image(img) # PNG tEXt/iTXt chunks land in img.info too (same idiom as the other # PNG-text readers in this module); NovelAI stamps Software/Source/Title. for key in ("Software", "Source", "Title", "Description"): @@ -1034,7 +1082,7 @@ def _xai_signature_impl(image_path: Path) -> bool: from PIL import Image with Image.open(image_path) as img: - exif_bytes = img.info.get("exif") + exif_bytes = exif_bytes_from_image(img) if not exif_bytes: return False tags = piexif.load(exif_bytes).get("0th", {}) @@ -1063,8 +1111,7 @@ def _is_aigc_exif_value(raw: object) -> bool: return False if b"AIGC" not in raw: return False - text = bytes(raw).decode("latin-1", "ignore") - return any(field in text for field in TC260_AIGC_FIELDS) + return aigc_label_from_metadata(bytes(raw)) is not None def _ai_exif_targets(loaded: dict[str, Any]) -> list[tuple[str, int, bytes, str]]: @@ -1100,15 +1147,18 @@ def _ai_exif_targets(loaded: dict[str, Any]) -> list[tuple[str, int, bytes, str] if xai_signature_pair(_exif_text(ifd0, piexif.ImageIFD.ImageDescription), _exif_text(ifd0, piexif.ImageIFD.Artist)): add("0th", ifd0, piexif.ImageIFD.ImageDescription, "ImageDescription") add("0th", ifd0, piexif.ImageIFD.Artist, "Artist") - # (b) known AI generator token in a 0th text tag. - for tag, name in ( - (piexif.ImageIFD.Software, "Software"), - (piexif.ImageIFD.Make, "Make"), - (piexif.ImageIFD.Artist, "Artist"), - (piexif.ImageIFD.ImageDescription, "ImageDescription"), + # (b) known AI generator token or structured generator name in a text tag. + for ifd_key, ifd, tag, name in ( + ("0th", ifd0, piexif.ImageIFD.Software, "Software"), + ("0th", ifd0, piexif.ImageIFD.Make, "Make"), + ("0th", ifd0, piexif.ImageIFD.Artist, "Artist"), + ("0th", ifd0, piexif.ImageIFD.ImageDescription, "ImageDescription"), + ("Exif", ifde, piexif.ExifIFD.UserComment, "UserComment"), ): - if any(token in _exif_text(ifd0, tag).lower() for token in AI_GENERATOR_TOKENS): - add("0th", ifd0, tag, name) + value = ifd.get(tag) + text = value.decode("latin-1", "replace") if isinstance(value, bytes) else "" + if any(token in text.lower() for token in AI_GENERATOR_TOKENS) or _structured_generator_from_metadata(text): + add(ifd_key, ifd, tag, name) # (c) TC260 AIGC block in ImageDescription (0th) or UserComment (Exif sub-IFD). if _is_aigc_exif_value(ifd0.get(piexif.ImageIFD.ImageDescription)): add("0th", ifd0, piexif.ImageIFD.ImageDescription, "ImageDescription") @@ -1171,6 +1221,11 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]: except Exception as exc: logger.debug("PIL could not open %s for AI-metadata scan: %s", image_path, exc) + from remove_ai_watermarks._internal.isobmff import ai_metadata_tags + + for key, value in ai_metadata_tags(image_path).items(): + result.setdefault(key, value[:200] + ("…" if len(value) > 200 else "")) + # C2PA manifest fields from the single canonical parser (_internal/c2pa.py). c2pa = extract_c2pa_info(image_path) for key in ( @@ -1632,11 +1687,11 @@ def remove_ai_metadata( with open(source_path, "rb") as f: head = f.read(26) if source_path.suffix.lower() in _STREAMING_ISOBMFF_EXTS and is_isobmff(head): - stripped, tc260_blanked = strip_isobmff_media_file(source_path, output_path) + stripped, native_ai_blanked = strip_isobmff_media_file(source_path, output_path) logger.info( - "Stream-blanked %d AI-provenance box(es) and %d native TC260 tag(s) → %s", + "Stream-blanked %d AI-provenance box(es) and %d native AI metadata tag(s) → %s", stripped, - tc260_blanked, + native_ai_blanked, output_path, ) return output_path @@ -1714,8 +1769,9 @@ def remove_ai_metadata( return output_path # Read image and filter metadata - with Image.open(source_path) as img: - img = img.copy() + with Image.open(source_path) as source_image: + source_exif_bytes = exif_bytes_from_image(source_image) + img = source_image.copy() # Pick the save format. Honor the caller's output extension (so a deliberate # source.png -> output.jpg conversion still works) UNLESS the SOURCE is misnamed # -- a lossless PNG/WebP whose extension lies (served as .jpg). There the output @@ -1762,6 +1818,10 @@ def remove_ai_metadata( if _keep_standard_text_metadata(key, value, keep_standard): kept_meta[key] = str(value) if not isinstance(value, str) else value + if source_exif_bytes: + with contextlib.suppress(Exception): + exif_data = piexif.load(source_exif_bytes) + # Apply cleaned metadata if save_kwargs["format"] == "PNG" and kept_meta: pnginfo = PngInfo() diff --git a/src/remove_ai_watermarks/metadata_record.py b/src/remove_ai_watermarks/metadata_record.py index 9b3c291..51c6fff 100644 --- a/src/remove_ai_watermarks/metadata_record.py +++ b/src/remove_ai_watermarks/metadata_record.py @@ -50,6 +50,7 @@ from remove_ai_watermarks._internal.schema import require_schema_version from remove_ai_watermarks.metadata import ( QUICK_SCAN_BYTES, SAMSUNG_EDITOR_MARKER, + exif_bytes_from_image, exif_text, read_file_tail, ) @@ -267,7 +268,10 @@ def _decoder_info(image_path: Path) -> dict[str, Any]: with Image.open(image_path) as img: # PIL types this mapping with a non-string key union (a DPI tuple key # exists), so the keys are normalized here rather than assumed. - return {str(key): value for key, value in img.info.items()} + info = {str(key): value for key, value in img.info.items()} + if "exif" not in info and (exif_bytes := exif_bytes_from_image(img)): + info["exif"] = exif_bytes + return info except (UnidentifiedImageError, ImportError) as exc: # unsupported optional decoder logger.debug("PIL info unavailable for %s: %s", image_path, exc) return {} @@ -392,6 +396,10 @@ def collect_metadata_record( issues.append({"stage": "trailer", "code": "collection-failed"}) try: info = _decoder_info(image_path) + if container == "isobmff": + from remove_ai_watermarks._internal.isobmff import ai_metadata_tags + + info.update(ai_metadata_tags(image_path, strict=True)) exif = _exif_pairs(info) except Exception as exc: logger.debug("metadata decoder collection failed for %s: %s", image_path, exc) diff --git a/tests/test_identify.py b/tests/test_identify.py index 68d541a..318adbd 100644 --- a/tests/test_identify.py +++ b/tests/test_identify.py @@ -464,6 +464,8 @@ class TestIdentifyNonPng: @pytest.mark.parametrize( ("claim_generator", "platform"), [ + ("Grok Imagine", "xAI Grok Imagine"), + ("Suno", "Suno"), ("Higgsfield AI", "Higgsfield AI"), ("recraft.ai", "Recraft"), ("Topaz Labs Image API", "Topaz Labs"), diff --git a/tests/test_metadata.py b/tests/test_metadata.py index 805cdfd..fa4b396 100644 --- a/tests/test_metadata.py +++ b/tests/test_metadata.py @@ -945,6 +945,30 @@ class TestExifGenerator: def test_clean_image_is_none(self, tmp_clean_png: Path): assert exif_generator(tmp_clean_png) is None + def test_nested_generation_model_in_webp_user_comment(self, tmp_path: Path): + import json + + payload = json.dumps( + { + "generationParams": { + "modelName": "Nano Banana synthetic fixture", + "prompt": "private prompt must not become the platform label", + } + } + ) + exif = piexif.dump({"Exif": {piexif.ExifIFD.UserComment: payload.encode("utf-8")}}) + source = tmp_path / "generated.webp" + Image.new("RGB", (32, 32)).save(source, "WEBP", lossless=True, exif=exif) + + assert exif_generator(source) == "Nano Banana synthetic fixture" + assert ( + identify(source, check_visible=False).platform == "Nano Banana synthetic fixture (embedded generator tag)" + ) + + output = tmp_path / "clean.webp" + remove_ai_metadata(source, output) + assert exif_generator(output) is None + _FAKE_SIG = "A" * 120 # 64+ base64 chars; real Grok payloads are 300-1004 _FAKE_UUID = "12345678-1234-1234-1234-123456789abc" @@ -970,12 +994,48 @@ def _grok_jpeg(tmp_path: Path, *, signature: str = _FAKE_SIG, artist: str = _FAK return path +def _grok_imagemagick_png(tmp_path: Path) -> Path: + """Write the ImageMagick raw-profile EXIF form seen on converted PNGs.""" + exif = piexif.dump( + { + "0th": { + piexif.ImageIFD.ImageDescription: f"Signature: {_FAKE_SIG}".encode("latin1"), + piexif.ImageIFD.Artist: _FAKE_UUID.encode("latin1"), + piexif.ImageIFD.Make: b"Canon", + }, + "Exif": {}, + "GPS": {}, + "1st": {}, + } + ) + profile = ( + "\nexif\n" + + f"{len(exif):8d}\n" + + "\n".join(exif.hex()[start : start + 72] for start in range(0, len(exif) * 2, 72)) + ) + info = PngInfo() + info.add_text("Raw profile type exif", profile) + path = tmp_path / "grok-imagemagick.png" + Image.new("RGB", (64, 64), (70, 80, 90)).save(path, pnginfo=info) + return path + + class TestXaiSignature: """xAI / Grok's EXIF Signature + UUID-Artist provenance scheme.""" def test_signature_plus_uuid_detected(self, tmp_path: Path): assert xai_signature(_grok_jpeg(tmp_path)) is True + def test_imagemagick_png_raw_exif_profile_detected(self, tmp_path: Path): + path = _grok_imagemagick_png(tmp_path) + + with Image.open(path) as image: + assert "exif" not in image.info + assert image.getexif() + + assert xai_signature(path) is True + assert "xai_signature" in get_ai_metadata(path) + def test_real_grok_sample_detected(self): # Real committed Grok download (data/fixtures/provenance/grok-1.jpg); the EXIF # Signature + UUID-Artist pair is the only AI signal it carries. @@ -1169,6 +1229,21 @@ class TestRemoveAiExif: assert xai_signature(out) is False assert has_ai_metadata(out) is False + def test_grok_raw_profile_stripped_on_png_output(self, tmp_path: Path): + src = _grok_imagemagick_png(tmp_path) + out = tmp_path / "clean.png" + + remove_ai_metadata(src, out) + + assert xai_signature(out) is False + with Image.open(out) as image: + assert image.getexif().get(piexif.ImageIFD.Make) == "Canon" + + def test_non_text_exif_value_does_not_break_app_metadata_scrub(self): + from remove_ai_watermarks.metadata import _app_metadata_evidence + + assert _app_metadata_evidence((1, 2)) == (None, None) + def test_generator_make_token_stripped(self, tmp_path: Path): # Ideogram's EXIF Make="Ideogram AI" must be scrubbed on removal. exif = piexif.dump({"0th": {piexif.ImageIFD.Make: b"Ideogram AI"}, "Exif": {}, "GPS": {}, "1st": {}}) @@ -1391,6 +1466,37 @@ class TestAIGCLabel: assert info["Label"] == "1" assert info["ContentProducer"] == "001191440300708461136T1308L" + def test_parses_lower_camel_case_tc260_fields(self, tmp_path: Path): + import json + + from remove_ai_watermarks.metadata import aigc_label + + p = tmp_path / "aigc-lower-camel.jpg" + payload = json.dumps( + { + "data": { + "AIGC": { + "label": "1", + "contentProducer": "synthetic-producer", + "produceId": "synthetic-id", + } + } + } + ) + exif = {"Exif": {piexif.ExifIFD.UserComment: payload.encode("ascii")}} + Image.new("RGB", (32, 32)).save(p, exif=piexif.dump(exif)) + + info = aigc_label(p) + + assert info == { + "Label": "1", + "ContentProducer": "synthetic-producer", + "ProduceID": "synthetic-id", + } + out = tmp_path / "clean-lower-camel.jpg" + remove_ai_metadata(p, out) + assert aigc_label(out) is None + def test_has_ai_metadata_detects_raw_json_exif_form(self, tmp_path: Path): assert has_ai_metadata(self._aigc_exif_jpeg(tmp_path)) diff --git a/tests/test_metadata_record.py b/tests/test_metadata_record.py index 130be6f..c6cfaf3 100644 --- a/tests/test_metadata_record.py +++ b/tests/test_metadata_record.py @@ -15,6 +15,7 @@ from pathlib import Path import numpy as np import pytest from PIL import Image +from PIL.PngImagePlugin import PngInfo from remove_ai_watermarks.identify import ( PROVENANCE_REPORT_SCHEMA_VERSION, @@ -157,6 +158,54 @@ class TestPlacementsTheRecordCouldDrop: assert "xai_signature" in {s.name for s in identify(path, check_visible=False, check_invisible=False).signals} _assert_same_verdict(path) + def test_an_imagemagick_raw_exif_profile_survives(self, tmp_path: Path): + """Pillow decodes this PNG EXIF profile through getexif, not img.info.""" + import piexif + + exif = piexif.dump( + { + "0th": { + piexif.ImageIFD.ImageDescription: b"Signature: " + b"A" * 80, + piexif.ImageIFD.Artist: b"3f2504e0-4f89-11d3-9a0c-0305e82c3301", + } + } + ) + profile = ( + "\nexif\n" + + f"{len(exif):8d}\n" + + "\n".join(exif.hex()[start : start + 72] for start in range(0, len(exif) * 2, 72)) + ) + info = PngInfo() + info.add_text("Raw profile type exif", profile) + path = tmp_path / "grok-converted.png" + Image.fromarray(np.zeros((64, 64, 3), dtype=np.uint8)).save(path, "PNG", pnginfo=info) + + assert "xai_signature" in {s.name for s in identify(path, check_visible=False, check_invisible=False).signals} + _assert_same_verdict(path) + + def test_isobmff_generation_tags_survive(self, tmp_path: Path): + def box(kind: bytes, payload: bytes) -> bytes: + return (8 + len(payload)).to_bytes(4, "big") + kind + payload + + def key(name: bytes) -> bytes: + return box(b"mdta", name) + + def value(index: int, payload: bytes) -> bytes: + return box(index.to_bytes(4, "big"), box(b"data", b"\x00" * 8 + payload)) + + keys = box(b"keys", b"\x00" * 4 + (2).to_bytes(4, "big") + key(b"workflow") + key(b"title")) + ilst = box(b"ilst", value(1, b'{"app":"ComfyUI"}') + value(2, b"standard title")) + meta = box(b"meta", b"\x00" * 4 + keys + ilst) + path = tmp_path / "workflow.mp4" + path.write_bytes( + b"\x00\x00\x00\x18ftypmp42\x00\x00\x00\x00mp42isom" + + box(b"mdat", b"synthetic-video") + + box(b"moov", box(b"udta", meta)) + ) + + assert "gen_params" in {s.name for s in identify(path, check_visible=False, check_invisible=False).signals} + _assert_same_verdict(path) + class TestRecordShape: def test_the_record_survives_json(self, tmp_path: Path): diff --git a/tests/test_signal_discovery_scripts.py b/tests/test_signal_discovery_scripts.py index b2b9e11..976667d 100644 --- a/tests/test_signal_discovery_scripts.py +++ b/tests/test_signal_discovery_scripts.py @@ -25,7 +25,16 @@ def _report(**overrides): def test_gap_markers_are_case_insensitive(): - assert "c2pa" in gap._marker_hits(b"UPPERCASE C2PA MANIFEST") + assert "ComfyUI" in gap._marker_hits(b"COMFYUI WORKFLOW") + + +def test_gap_markers_require_structural_provenance_evidence(): + noise = b"c2pa digitalSourceType Samsung Galaxy Doubao PhotoEditor_Re_Edit Signature: " + b"A" * 80 + assert gap._marker_hits(noise) == [] + + tc260 = b'{"AIGC":{"label":"1","contentProducer":"synthetic"}}' + assert "TC260 AIGC" in gap._marker_hits(tc260) + assert "Samsung genAIType" in gap._marker_hits(b'PhotoEditor_Re_Edit_Data{"genAIType":1}') def test_gap_candidates_cover_more_than_blind_unknowns(): diff --git a/tests/test_video.py b/tests/test_video.py index b373b51..216ebb0 100644 --- a/tests/test_video.py +++ b/tests/test_video.py @@ -65,6 +65,30 @@ def _video_with_tc260(path: Path, *, media_payload: bytes = _VIDEO_PAYLOAD) -> P return path +def _video_with_comfyui_metadata(path: Path, *, media_payload: bytes = _VIDEO_PAYLOAD) -> Path: + workflow = b'{"nodes":[{"type":"KSampler"}],"app":"ComfyUI"}' + prompt = b'{"1":{"class_type":"KSampler"}}' + keys = _box( + b"keys", + b"\x00\x00\x00\x00" + + (4).to_bytes(4, "big") + + _metadata_key(b"workflow") + + _metadata_key(b"prompt") + + _metadata_key(b"title") + + _metadata_key(b"software"), + ) + ilst = _box( + b"ilst", + _metadata_value(1, workflow) + + _metadata_value(2, prompt) + + _metadata_value(3, b"standard title") + + _metadata_value(4, b"ordinary editor"), + ) + meta = _box(b"meta", b"\x00\x00\x00\x00" + keys + ilst) + path.write_bytes(_MP4_FTYP + _box(b"mdat", media_payload) + _box(b"moov", _box(b"udta", meta))) + return path + + def _hdlr(handler: bytes) -> bytes: # version/flags + pre_defined + handler_type + reserved[3]; the TC260 walker # never parses it, but a real-shaped hdlr keeps the fixture honest. @@ -564,6 +588,24 @@ class TestVideoDependencies: class TestVideoMetadataApi: + def test_inspects_and_removes_comfyui_mp4_metadata(self, tmp_path: Path): + from remove_ai_watermarks.video import inspect_video_metadata, remove_video_metadata + + source = _video_with_comfyui_metadata(tmp_path / "source.mp4") + output = tmp_path / "clean.mp4" + + report = inspect_video_metadata(source) + assert set(report.markers) >= {"workflow", "prompt"} + + result = remove_video_metadata(source, output) + cleaned = output.read_bytes() + assert result.remaining == {} + assert len(cleaned) == source.stat().st_size + assert _VIDEO_PAYLOAD in cleaned + assert b"standard title" in cleaned + assert b"ordinary editor" in cleaned + assert b"ComfyUI" not in cleaned + def test_top_level_api_is_lazy_exported(self): import remove_ai_watermarks as raiw