mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-30 01:00:35 +02:00
Detect TC260 AIGC label in JPEG EXIF and late/attribute PNG XMP
A corpus audit surfaced China TC260 AIGC-labeled images that `identify`
missed. Three detection gaps in `aigc_label`, all fixed:
- raw-JSON `{"AIGC":{...}}` in JPEG EXIF (UserComment): brace-matched from
the scan head with `json.raw_decode`, gated on a TC260 field like the
PNG-chunk path. (Doubao-class output via that export surface.)
- XMP attribute form `TC260:AIGC="{...}"` (PicWish): folded into the
element regex as a second alternation.
- TC260 XMP packet appended after a large `IDAT`, past the 1 MB scan
window: `scan_head` now appends late PNG metadata chunks via
`_png_late_metadata`, mirroring the existing ISOBMFF late-box scan.
Adds `scripts/corpus_gap_scan.py`: runs `identify` over a corpus, writes
the per-file report CSV, and flags `unknown` files that carry a known
marker in their metadata region (the audit that found these gaps).
Scanning only the metadata region — not the whole file — avoids the
random short-token collisions inside compressed PNG/JPEG streams.
On the local corpus this lifts 3 files from `unknown` to AI (China AIGC)
and leaves zero false gap candidates. Synthetic piexif/PngInfo fixtures
cover all three forms.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
58bdf51c59
commit
25a1acc53b
@@ -26,7 +26,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
|
||||
- **Batch processing** — process entire directories
|
||||
- **Detection** — three-stage NCC watermark detection with confidence scoring
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP or PNG chunk), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the visible sparkle, the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
|
||||
|
||||
## Examples
|
||||
|
||||
@@ -49,7 +49,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
| **xAI Grok (Aurora)** | — | — | ✅ EXIF signature scheme (no C2PA): `Signature:` blob + UUID `Artist` | Detected (`identify`); metadata strip |
|
||||
| **Midjourney** | — | — | ✅ EXIF + XMP (prompt, model, seed) | Metadata strip |
|
||||
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
|
||||
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 AIGC label (`<TC260:AIGC>` XMP **or** `AIGC` PNG chunk) **+ C2PA** signed by ByteDance Volcano Engine (`volcengine`) | Exact reverse-alpha (captured α map): pixel-exact at native width, NCC-aligned at other resolutions, + metadata strip |
|
||||
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 AIGC label (`<TC260:AIGC>` XMP, `AIGC` PNG chunk, or EXIF JSON) **+ C2PA** signed by ByteDance Volcano Engine (`volcengine`) | Exact reverse-alpha (captured α map): pixel-exact at native width, NCC-aligned at other resolutions, + metadata strip |
|
||||
| **Samsung Galaxy AI** (Generative Edit, Sketch to Image, ...) | — | — | ✅ C2PA (signer "Samsung Galaxy") + `trainedAlgorithmicMedia` / proprietary `genAIType` marker | Detected (`identify`) + metadata strip |
|
||||
| **Black Forest Labs** (FLUX API) | — | — | ✅ C2PA (`Black Forest Labs API` + `c2pa.ai_generated_content` + `trainedAlgorithmicMedia`) | Metadata strip |
|
||||
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
|
||||
@@ -57,7 +57,7 @@ If this tool saves you time, consider [sponsoring its development](https://githu
|
||||
|
||||
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo) and by Doubao / China AIGC generators (the mandated "...AI生成" corner text). Both are removed deterministically on CPU. Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any colour), use the universal `erase --region` command.
|
||||
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP or PNG chunk), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP, PNG chunk, or EXIF), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the visible sparkle, and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
|
||||
|
||||
## How it works
|
||||
|
||||
|
||||
@@ -0,0 +1,215 @@
|
||||
"""Audit a local image corpus against the library's own ``identify`` detector.
|
||||
|
||||
Two jobs in one pass:
|
||||
|
||||
1. **Report** -- run ``identify`` over every image and write one CSV row per file
|
||||
(verdict, platform, confidence, watermarks, signals, integrity clashes).
|
||||
2. **Gap audit** -- for every ``unknown``-verdict file, scan only its *metadata
|
||||
region* (PNG text/eXIf chunks, JPEG APPn segments before SOS, or the file
|
||||
head for other containers) for known provenance markers. A marker found there
|
||||
on a file the detector calls ``unknown`` is a concrete lib gap: a serialization
|
||||
or generator we do not yet parse. Scanning the metadata region -- not the whole
|
||||
file -- is deliberate: short tokens collide randomly inside compressed PNG
|
||||
``IDAT`` / JPEG scan data, which produced false "xAI/Flux/AIGC" hits when the
|
||||
first audit naively scanned the first megabyte.
|
||||
|
||||
This is how new detector gaps get found (it is what surfaced the JPEG-EXIF
|
||||
``{"AIGC":{...}}`` form). Re-run after collecting a fresh corpus batch.
|
||||
|
||||
Usage:
|
||||
uv run python scripts/corpus_gap_scan.py --corpus data/spaces/originals
|
||||
uv run python scripts/corpus_gap_scan.py --corpus data/spaces/originals \\
|
||||
--report data/spaces/detector_report.csv
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import logging
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from remove_ai_watermarks.identify import identify
|
||||
from remove_ai_watermarks.metadata import _png_late_metadata
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
console = Console()
|
||||
|
||||
# Distinctive, multi-byte provenance markers worth flagging when they appear in a
|
||||
# file the detector calls `unknown`. Kept long enough that a random collision in a
|
||||
# (non-scanned) compressed stream is implausible; the metadata-region restriction
|
||||
# below is the primary guard, this list is the second. Group: C2PA/JUMBF infra,
|
||||
# AI source-type / labeling schemes, and distinctive generator name strings.
|
||||
MARKERS: tuple[bytes, ...] = (
|
||||
# C2PA / JUMBF infrastructure and AI source-type / labeling schemes.
|
||||
b"c2pa",
|
||||
b"jumbf",
|
||||
b"contentauth",
|
||||
b"trainedAlgorithmicMedia",
|
||||
b"digitalSourceType",
|
||||
b'"AIGC"',
|
||||
b"<TC260:AIGC>",
|
||||
b"TC260:AIGC",
|
||||
b"tc260.org.cn",
|
||||
b"AISystemUsed",
|
||||
b"SynthID",
|
||||
b"hf-job-id",
|
||||
b"genAIType",
|
||||
b"PhotoEditor_Re_Edit",
|
||||
b"Signature:",
|
||||
# Distinctive multi-word generator strings only. Bare single words (Luma,
|
||||
# Gemini, Sora, ...) are omitted: they collide with unrelated metadata prose
|
||||
# (e.g. "Luma" in Lightroom's EnhanceDenoiseLumaAmount), defeating precision.
|
||||
b"Midjourney",
|
||||
b"Stable Diffusion",
|
||||
b"StableDiffusion",
|
||||
b"ComfyUI",
|
||||
b"Automatic1111",
|
||||
b"DALL-E",
|
||||
b"Ideogram AI",
|
||||
b"Adobe Firefly",
|
||||
b"Black Forest",
|
||||
b"volcengine",
|
||||
b"Doubao",
|
||||
b"\xe8\xb1\x86\xe5\x8c\x85",
|
||||
b"Nano Banana",
|
||||
b"Stability AI",
|
||||
b"Samsung Galaxy",
|
||||
)
|
||||
|
||||
|
||||
def _metadata_region(path: Path) -> bytes:
|
||||
"""Return only the bytes where provenance metadata can live, never the
|
||||
compressed pixel stream (which produces random short-token collisions)."""
|
||||
try:
|
||||
head = path.read_bytes()
|
||||
except OSError:
|
||||
return b""
|
||||
if head[:8] == b"\x89PNG\r\n\x1a\n":
|
||||
# All ancillary metadata chunks (window=0), via the library's own walker.
|
||||
return _png_late_metadata(path, 0)
|
||||
if head[:2] == b"\xff\xd8": # JPEG: APPn segments up to Start-Of-Scan
|
||||
out = bytearray()
|
||||
p = 2
|
||||
n = len(head)
|
||||
while p + 4 <= n and head[p] == 0xFF:
|
||||
marker = head[p + 1]
|
||||
if marker == 0xDA: # SOS -> compressed scan data follows
|
||||
break
|
||||
seg_len = (head[p + 2] << 8) | head[p + 3]
|
||||
out += head[p + 4 : p + 2 + seg_len]
|
||||
p += 2 + seg_len
|
||||
return bytes(out)
|
||||
return head[:65536] # webp/avif/heif/jxl: metadata sits near the head
|
||||
|
||||
|
||||
def _row(rep) -> dict[str, str]: # noqa: ANN001 (ProvenanceReport)
|
||||
return {
|
||||
"path": "", # filled by caller (relative)
|
||||
"is_ai": str(rep.is_ai_generated),
|
||||
"platform": rep.platform or "",
|
||||
"confidence": rep.confidence,
|
||||
"watermarks": "|".join(rep.watermarks),
|
||||
"signals": "|".join(s.name for s in rep.signals),
|
||||
"integrity_clashes": "|".join(rep.integrity_clashes),
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option(
|
||||
"--corpus",
|
||||
type=click.Path(exists=True, file_okay=False, path_type=Path),
|
||||
default=Path("data/spaces/originals"),
|
||||
show_default=True,
|
||||
help="Directory of images to scan (recursively).",
|
||||
)
|
||||
@click.option(
|
||||
"--report",
|
||||
type=click.Path(path_type=Path),
|
||||
default=None,
|
||||
help="Write the per-file CSV here (default: <corpus>/../detector_report.csv).",
|
||||
)
|
||||
@click.option("--limit", type=int, default=0, help="Scan at most N files (0 = all).")
|
||||
def main(corpus: Path, report: Path | None, limit: int) -> None:
|
||||
logging.basicConfig(level=logging.WARNING, format="%(message)s")
|
||||
report = report or corpus.parent / "detector_report.csv"
|
||||
|
||||
files = sorted(p for p in corpus.rglob("*") if p.is_file())
|
||||
if limit:
|
||||
files = files[:limit]
|
||||
console.print(f"Scanning [bold]{len(files)}[/bold] files under {corpus} ...")
|
||||
|
||||
verdicts: Counter[str] = Counter()
|
||||
platforms: Counter[str] = Counter()
|
||||
gap_tokens: Counter[str] = Counter()
|
||||
gaps: list[tuple[str, list[str]]] = []
|
||||
rows: list[dict[str, str]] = []
|
||||
errors = 0
|
||||
|
||||
with click.progressbar(files, label="identify") as bar:
|
||||
for p in bar:
|
||||
rel = str(p.relative_to(corpus))
|
||||
try:
|
||||
rep = identify(p)
|
||||
except Exception as exc:
|
||||
log.warning("identify failed on %s: %s", rel, exc)
|
||||
errors += 1
|
||||
continue
|
||||
row = _row(rep)
|
||||
row["path"] = rel
|
||||
rows.append(row)
|
||||
if rep.is_ai_generated:
|
||||
verdicts["ai"] += 1
|
||||
platforms[rep.platform or "?"] += 1
|
||||
continue
|
||||
verdicts["unknown"] += 1
|
||||
# A gap candidate is a file identify is *blind* to (no signal at all)
|
||||
# yet whose metadata carries a known marker. A file that produced a
|
||||
# signal but no AI verdict (e.g. an ASUS Gallery C2PA signer, which we
|
||||
# attribute but do not call AI) is handled correctly -- not a gap.
|
||||
if rep.signals:
|
||||
continue
|
||||
region = _metadata_region(p)
|
||||
hits = sorted({m.decode("latin-1", "replace") for m in MARKERS if m in region})
|
||||
if hits:
|
||||
gaps.append((rel, hits))
|
||||
gap_tokens.update(hits)
|
||||
|
||||
with report.open("w", newline="") as f:
|
||||
writer = csv.DictWriter(
|
||||
f,
|
||||
fieldnames=["path", "is_ai", "platform", "confidence", "watermarks", "signals", "integrity_clashes"],
|
||||
)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
console.print(f"\nWrote [bold]{len(rows)}[/bold] rows -> {report}")
|
||||
|
||||
console.print(f"\n[bold]Verdicts:[/bold] AI {verdicts['ai']} | unknown {verdicts['unknown']} | errors {errors}")
|
||||
plat = Table(title="AI platforms", show_header=False)
|
||||
for name, n in platforms.most_common():
|
||||
plat.add_row(str(n), name)
|
||||
console.print(plat)
|
||||
|
||||
if gaps:
|
||||
console.print(
|
||||
f"\n[bold red]Gap candidates[/bold red]: {len(gaps)} unknown files carry a known "
|
||||
f"marker in their metadata region (potential undetected serialization/generator):"
|
||||
)
|
||||
tok = Table(title="markers seen in unknown files")
|
||||
tok.add_column("count", justify="right")
|
||||
tok.add_column("marker")
|
||||
for name, n in gap_tokens.most_common():
|
||||
tok.add_row(str(n), name)
|
||||
console.print(tok)
|
||||
for rel, hits in gaps:
|
||||
console.print(f" {rel} -> {', '.join(hits)}")
|
||||
else:
|
||||
console.print("\n[green]No gap candidates: every unknown file is metadata-free.[/green]")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -11,6 +11,7 @@ from __future__ import annotations
|
||||
import contextlib
|
||||
import logging
|
||||
import re
|
||||
import struct
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -169,15 +170,58 @@ def _is_ai_key(key: str) -> bool:
|
||||
return any(kw in key_lower for kw in AI_KEYWORDS)
|
||||
|
||||
|
||||
# PNG ancillary chunks that can carry provenance metadata (XMP, EXIF, text).
|
||||
# Never IDAT -- that is the compressed pixel stream.
|
||||
_PNG_META_CHUNKS: frozenset[bytes] = frozenset({b"tEXt", b"iTXt", b"zTXt", b"eXIf", b"iCCP"})
|
||||
|
||||
|
||||
def _png_late_metadata(image_path: Path, window: int) -> bytes:
|
||||
"""Payloads of PNG metadata chunks that start *beyond* the first ``window``
|
||||
bytes, found by seeking past the (large) ``IDAT`` pixel stream.
|
||||
|
||||
A PNG encoder may append the XMP/EXIF packet after the image data, so a
|
||||
fixed first-``size`` read misses it (e.g. a TC260 AIGC label in an XMP
|
||||
``iTXt`` chunk at ~2.7 MB). This is the PNG analogue of the ISOBMFF
|
||||
late-box scan in :func:`scan_head`. Returns only chunks past ``window`` so
|
||||
bytes already in the head are not duplicated; empty when there are none.
|
||||
"""
|
||||
out = bytearray()
|
||||
try:
|
||||
with open(image_path, "rb") as f:
|
||||
if f.read(8) != b"\x89PNG\r\n\x1a\n":
|
||||
return b""
|
||||
pos = 8
|
||||
while True:
|
||||
f.seek(pos)
|
||||
header = f.read(8)
|
||||
if len(header) < 8:
|
||||
break
|
||||
(length,) = struct.unpack(">I", header[:4])
|
||||
chunk_type = header[4:8]
|
||||
if chunk_type == b"IEND":
|
||||
break
|
||||
data_start = pos + 8
|
||||
if chunk_type in _PNG_META_CHUNKS and data_start >= window:
|
||||
f.seek(data_start)
|
||||
out += f.read(length)
|
||||
pos = data_start + length + 4 # data + CRC
|
||||
except OSError as exc:
|
||||
logger.debug("PNG late-metadata scan failed on %s: %s", image_path, exc)
|
||||
return b""
|
||||
return bytes(out)
|
||||
|
||||
|
||||
def scan_head(image_path: Path, size: int = 1024 * 1024) -> bytes:
|
||||
"""First ``size`` bytes of the file, plus -- for ISOBMFF containers -- the
|
||||
payloads of any provenance (``uuid`` / ``jumb``) boxes found beyond that
|
||||
window by seeking past large boxes like ``mdat``.
|
||||
"""First ``size`` bytes of the file, plus the payloads of any provenance
|
||||
metadata found beyond that window: ISOBMFF ``uuid`` / ``jumb`` boxes (seeking
|
||||
past large boxes like ``mdat``) and PNG ``tEXt`` / ``iTXt`` / ``eXIf`` chunks
|
||||
(seeking past ``IDAT``).
|
||||
|
||||
This is the shared input for every C2PA / AIGC / IPTC byte scan. The
|
||||
ISOBMFF extension catches a manifest placed AFTER the media data in a
|
||||
streaming / non-faststart MP4, which a fixed first-MB read would miss. For
|
||||
non-ISOBMFF inputs it is exactly ``f.read(size)`` -- behavior-neutral.
|
||||
extensions catch a manifest or XMP packet placed AFTER the media data -- a
|
||||
non-faststart MP4 manifest, or a PNG XMP packet appended after the pixels --
|
||||
which a fixed first-MB read would miss. For other inputs, and for files that
|
||||
fit within ``size``, it is exactly ``f.read(size)`` -- behavior-neutral.
|
||||
"""
|
||||
with open(image_path, "rb") as f:
|
||||
head = f.read(size)
|
||||
@@ -188,6 +232,10 @@ def scan_head(image_path: Path, size: int = 1024 * 1024) -> bytes:
|
||||
region = isobmff.scan_c2pa_region(image_path)
|
||||
if region:
|
||||
head += region
|
||||
elif head[:8] == b"\x89PNG\r\n\x1a\n" and len(head) == size:
|
||||
# len(head) == size means the file is at least `size` bytes, so metadata
|
||||
# chunks may lie beyond the window; otherwise the whole PNG is in `head`.
|
||||
head += _png_late_metadata(image_path, size)
|
||||
return head
|
||||
|
||||
|
||||
@@ -252,17 +300,21 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
def aigc_label(image_path: Path) -> dict[str, str] | None:
|
||||
"""Parse a China TC260 AI-labeling block, if present.
|
||||
|
||||
Two serializations are recognized:
|
||||
Three serializations are recognized:
|
||||
|
||||
- a PNG ``tEXt``/``iTXt`` chunk keyed ``AIGC`` carrying the raw JSON object
|
||||
(as written by Doubao / ByteDance), read via PIL; and
|
||||
(as written by Doubao / ByteDance), read via PIL;
|
||||
- an XMP ``<TC260:AIGC>{...}</TC260:AIGC>`` block (HTML-entity encoded text),
|
||||
found by a container-agnostic raw-byte scan (PNG/JPEG/WebP alike).
|
||||
found by a container-agnostic raw-byte scan (PNG/JPEG/WebP alike); and
|
||||
- a raw-JSON ``{"AIGC":{...}}`` block with no namespace, as embedded in JPEG
|
||||
EXIF (UserComment) by some China-served generators, brace-matched from the
|
||||
scan head.
|
||||
|
||||
Returns the decoded JSON (e.g. ``{"Label": "1", "ContentProducer": ...}``)
|
||||
or None. The PNG-chunk key ``AIGC`` is generic, so a JSON object there is
|
||||
accepted only if it carries at least one known TC260 field (``_TC260_FIELDS``);
|
||||
the namespaced XMP element is unambiguous, so any JSON object is accepted.
|
||||
or None. The generic forms (the PNG-chunk key ``AIGC`` and the bare
|
||||
``{"AIGC":...}`` object) are accepted only if they carry at least one known
|
||||
TC260 field (``_TC260_FIELDS``); the namespaced XMP element is unambiguous,
|
||||
so any JSON object is accepted.
|
||||
"""
|
||||
import html
|
||||
import json
|
||||
@@ -293,12 +345,35 @@ def aigc_label(image_path: Path) -> dict[str, str] | None:
|
||||
if isinstance(value, str) and (result := _parse(value, require_tc260_field=True)):
|
||||
return result
|
||||
|
||||
# XMP <TC260:AIGC>{...}</TC260:AIGC> block (namespaced element, unambiguous).
|
||||
# XMP TC260:AIGC, namespaced (unambiguous) in either serialization RDF allows:
|
||||
# an element <TC260:AIGC>{...}</TC260:AIGC> or an attribute TC260:AIGC="{...}"
|
||||
# (the attribute form is what PicWish writes). Both are HTML-entity encoded.
|
||||
data = scan_head(image_path)
|
||||
match = re.search(rb"<TC260:AIGC>(.*?)</TC260:AIGC>", data, re.DOTALL)
|
||||
if not match:
|
||||
return None
|
||||
return _parse(html.unescape(match.group(1).decode("utf-8", "replace")), require_tc260_field=False)
|
||||
match = re.search(
|
||||
rb'<TC260:AIGC>(.*?)</TC260:AIGC>|TC260:AIGC\s*=\s*"(.*?)"',
|
||||
data,
|
||||
re.DOTALL,
|
||||
)
|
||||
if match:
|
||||
body = match.group(1) if match.group(1) is not None else match.group(2)
|
||||
return _parse(html.unescape(body.decode("utf-8", "replace")), require_tc260_field=False)
|
||||
|
||||
# Raw-JSON {"AIGC":{...}} block (no namespace), as written into JPEG EXIF
|
||||
# (UserComment) by some China-served generators -- the PNG-chunk and XMP
|
||||
# paths above both miss it. The bytes pre-check keeps the common (no-AIGC)
|
||||
# path off the full-buffer decode; raw_decode then brace-matches the inner
|
||||
# object (respecting nested braces / quoted strings) and `_parse` applies the
|
||||
# same dict coercion + TC260-field gate as the generic PNG-chunk path.
|
||||
if b'"AIGC"' in data:
|
||||
text = data.decode("latin-1")
|
||||
brace = text.find("{", text.find('"AIGC"') + len('"AIGC"'))
|
||||
if brace != -1:
|
||||
try:
|
||||
_, end = json.JSONDecoder().raw_decode(text, brace)
|
||||
except ValueError:
|
||||
return None
|
||||
return _parse(text[brace:end], require_tc260_field=True)
|
||||
return None
|
||||
|
||||
|
||||
def huggingface_job(image_path: Path) -> str | None:
|
||||
|
||||
@@ -705,6 +705,82 @@ class TestAIGCLabel:
|
||||
assert aigc_label(out) is None
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
def _aigc_exif_jpeg(self, tmp_path: Path, producer: str = "001191440300708461136T1308L") -> Path:
|
||||
"""Some China-served generators embed the raw-JSON ``{"AIGC":{...}}``
|
||||
block in JPEG EXIF (UserComment) -- no PNG chunk, no namespaced XMP."""
|
||||
import json
|
||||
|
||||
import piexif
|
||||
|
||||
p = tmp_path / "aigc_exif.jpg"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
payload = json.dumps({"AIGC": {"Label": "1", "ContentProducer": producer, "ProduceID": "abc123"}})
|
||||
exif = {"Exif": {piexif.ExifIFD.UserComment: payload.encode("ascii")}}
|
||||
piexif.insert(piexif.dump(exif), str(p))
|
||||
return p
|
||||
|
||||
def test_parses_raw_json_exif_form(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
info = aigc_label(self._aigc_exif_jpeg(tmp_path))
|
||||
assert info is not None
|
||||
assert info["Label"] == "1"
|
||||
assert info["ContentProducer"] == "001191440300708461136T1308L"
|
||||
|
||||
def test_has_ai_metadata_detects_raw_json_exif_form(self, tmp_path: Path):
|
||||
assert has_ai_metadata(self._aigc_exif_jpeg(tmp_path))
|
||||
|
||||
def test_raw_json_without_tc260_field_ignored(self, tmp_path: Path):
|
||||
"""A bare ``{"AIGC":{...}}`` object with no TC260 field must not fire."""
|
||||
import json
|
||||
|
||||
import piexif
|
||||
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
p = tmp_path / "unrelated.jpg"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
payload = json.dumps({"AIGC": {"unrelated": "value"}})
|
||||
exif = {"Exif": {piexif.ExifIFD.UserComment: payload.encode("ascii")}}
|
||||
piexif.insert(piexif.dump(exif), str(p))
|
||||
assert aigc_label(p) is None
|
||||
|
||||
def _aigc_attr_png(self, tmp_path: Path, producer: str = "picwish") -> Path:
|
||||
"""PicWish writes the TC260 label as an XMP *attribute*
|
||||
(``TC260:AIGC="{...}"``), not the nested element form."""
|
||||
p = tmp_path / "picwish.png"
|
||||
Image.new("RGB", (32, 32)).save(p)
|
||||
xmp = (
|
||||
'<rdf:Description rdf:about="" '
|
||||
'xmlns:TC260="http://www.tc260.org.cn/ns/AIGC/1.0/" '
|
||||
f'TC260:AIGC="{{"Label":"1","ContentProducer":"{producer}"}}"/>'
|
||||
)
|
||||
with open(p, "ab") as f:
|
||||
f.write(xmp.encode())
|
||||
return p
|
||||
|
||||
def test_parses_xmp_attribute_form(self, tmp_path: Path):
|
||||
from remove_ai_watermarks.metadata import aigc_label
|
||||
|
||||
info = aigc_label(self._aigc_attr_png(tmp_path))
|
||||
assert info is not None
|
||||
assert info["ContentProducer"] == "picwish"
|
||||
|
||||
def test_scan_head_collects_png_metadata_past_window(self, tmp_path: Path):
|
||||
"""A PNG metadata chunk beyond the read window is still reachable -- the
|
||||
regression for a TC260 XMP packet appended after a large IDAT."""
|
||||
import json
|
||||
|
||||
from remove_ai_watermarks.metadata import _png_late_metadata, scan_head
|
||||
|
||||
p = tmp_path / "late.png"
|
||||
pnginfo = PngInfo()
|
||||
pnginfo.add_text("AIGC", json.dumps({"Label": "1", "ContentProducer": "doubao"}))
|
||||
Image.new("RGB", (16, 16)).save(p, pnginfo=pnginfo)
|
||||
# window = 8 (just the signature) makes the text chunk "late".
|
||||
assert b"ContentProducer" in _png_late_metadata(p, 8)
|
||||
assert b"ContentProducer" in scan_head(p, 8)
|
||||
|
||||
|
||||
class TestHuggingFaceJob:
|
||||
"""HuggingFace-hosted job marker (``hf-job-id`` PNG text chunk)."""
|
||||
|
||||
Reference in New Issue
Block a user