mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-06-04 01:58:00 +02:00
feat: SDXL default; AVIF/HEIF/JPEG-XL C2PA stripping
SD-1.5 dreamshaper at 768 px did not defeat SynthID v2 on Gemini 3 Pro outputs (verified May 2026 via Gemini app's "Verify with SynthID"). Switch the default invisible engine to SDXL at 1024 px, matching the raiw-app production config (strength 0.05, steps 50). Drop the SD-1.5 pipeline. Metadata layer: add C2PA UUID and IPTC AI marker byte-scan detection across all formats, plus an ISOBMFF box walker (noai/isobmff.py) that strips top-level C2PA uuid and JUMBF jumb boxes from AVIF/HEIF/JPEG-XL containers without re-encoding. README gets a Legal table and a Threat-model section about SynthID v2's 136-bit payload. CLAUDE.md tracks the SD-1.5 regression as historical context. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
test-user
@@ -21,5 +21,8 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
||||
|
||||
## Known limitations
|
||||
|
||||
- `invisible` pipeline downscales to 768 px before diffusion → degrades fine text in infographics. Tracked; fix is tile-based or skip-downscale approach.
|
||||
- `invisible` pipeline downscales to model-native resolution (1024 px for SDXL) before diffusion. Degrades fine text in infographics. Tracked; fix is tile-based diffusion.
|
||||
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs)
|
||||
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
|
||||
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`. C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are not yet scrubbed.
|
||||
- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2 per the deployment in raiw-app (`fal-ai/fast-sdxl` at native ~1024 px, strength 0.05, steps 50). SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against Gemini app's "Verify with SynthID" feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain.
|
||||
|
||||
@@ -8,7 +8,7 @@ Strips SynthID, C2PA Content Credentials, EXIF/XMP "Made with AI" labels, and vi
|
||||
|
||||
- **Visible watermark removal** — Gemini / Nano Banana sparkle logo via reverse alpha blending (fast, offline, deterministic)
|
||||
- **Invisible watermark removal** — SynthID, StableSignature, TreeRing via diffusion-based regeneration
|
||||
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests, XMP DigitalSourceType
|
||||
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests (PNG / JPEG / AVIF / HEIF / JPEG-XL), XMP DigitalSourceType
|
||||
- **"Made with AI" label removal** — removes the metadata that triggers AI labels on Instagram, Facebook, X (Twitter)
|
||||
- **Analog Humanizer** — film grain and chromatic aberration to bypass AI image classifiers
|
||||
- **Smart Face Protection** — automatic extraction and blending of human faces to prevent AI distortion
|
||||
@@ -27,7 +27,7 @@ Strips SynthID, C2PA Content Credentials, EXIF/XMP "Made with AI" labels, and vi
|
||||
|
||||
| AI model | Visible watermark | Invisible watermark | Metadata | Our approach |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Google Gemini / Nano Banana** | ✅ Sparkle logo | ✅ SynthID | ✅ C2PA + EXIF | Alpha reversal + diffusion + metadata strip |
|
||||
| **Google Gemini / Nano Banana / Gemini 3 Pro** | ✅ Sparkle logo | ✅ SynthID v1 + v2 (default SDXL pipeline at native ~1024 px) | ✅ C2PA + EXIF | Alpha reversal + diffusion + metadata strip |
|
||||
| **OpenAI DALL-E 3 / ChatGPT** | — | — | ✅ C2PA manifest | Metadata strip |
|
||||
| **OpenAI ChatGPT Images 2.0** (gpt-image-2) | — | ⚠️ imperceptible pixel watermark (no public detector yet) | ✅ C2PA manifest (verified) | Diffusion regeneration + metadata strip |
|
||||
| **Stable Diffusion (AUTOMATIC1111, ComfyUI)** | — | ✅ DWT / steganographic | ✅ PNG text chunks | Diffusion regeneration + metadata strip |
|
||||
@@ -62,16 +62,16 @@ A three-stage NCC (Normalized Cross-Correlation) detector finds the watermark po
|
||||
|
||||
Google embeds **SynthID** into every image generated by Gemini / Nano Banana. Other AI services use StableSignature, TreeRing, and similar schemes. These imperceptible frequency-domain patterns survive cropping, resizing, and JPEG compression.
|
||||
|
||||
The removal pipeline:
|
||||
The removal pipeline (default profile, SDXL):
|
||||
|
||||
```text
|
||||
image → downscale to 768px → encode to latent space (VAE)
|
||||
image → resize to ~1024px (SDXL native) → encode to latent space (VAE)
|
||||
→ add controlled noise (forward diffusion)
|
||||
→ denoise (reverse diffusion, ~2 steps at strength 0.02)
|
||||
→ denoise (reverse diffusion, ~50 steps at strength 0.05)
|
||||
→ decode back to pixels (VAE) → upscale to original resolution
|
||||
```
|
||||
|
||||
The key insight: even minimal noise injection (strength 0.02 = 2% perturbation) breaks the watermark signal while preserving visual quality. The diffusion model acts as a learned image prior — it reconstructs the image faithfully while destroying the watermark pattern.
|
||||
SDXL is the default since May 2026: empirically defeats SynthID v2 on Gemini 3 Pro outputs, where the older SD-1.5 pipeline at 768 px did not. The SD-1.5 path was removed once it was verified not to handle v2.
|
||||
|
||||
**Face Protection**: before diffusion, YOLO detects people in the image and extracts them. After diffusion, the original faces are blended back with a soft elliptical mask to prevent AI distortion of facial features.
|
||||
|
||||
@@ -249,17 +249,53 @@ pip install certifi
|
||||
- [CtrlRegen](https://github.com/yepengliu/CtrlRegen) by Liu et al. (ICLR 2025) — controllable regeneration pipeline
|
||||
- NeuralBleach (MIT) — analog humanizer technique
|
||||
|
||||
## ⚠️ Disclaimer
|
||||
## Roadmap
|
||||
|
||||
This tool is provided for **educational and research purposes only**.
|
||||
Tracked but not yet implemented:
|
||||
|
||||
Removing AI watermarks to misrepresent AI-generated content as human-created
|
||||
may violate applicable laws, including the U.S. Digital Millennium Copyright Act
|
||||
(DMCA) and the COPIED Act. Users are solely responsible for ensuring their use
|
||||
complies with all applicable laws and platform terms of service.
|
||||
- **SynthID-Image v2 automated regression test**. The default SDXL profile defeats v2 per manual checks against the [Gemini app](https://support.google.com/gemini/answer/16722517)'s "Verify with SynthID" feature on a Gemini 3 Pro output (May 2026). An automated end-to-end test would need either programmatic access to the [SynthID Detector portal](https://blog.google/innovation-and-ai/products/google-synthid-ai-content-detector/) (waitlist for media professionals and researchers) or an offline surrogate detector. Open.
|
||||
- **AVIF / HEIF / JPEG-XL detection limits**. Removal strips top-level C2PA `uuid` and JUMBF `jumb` boxes. EXIF/XMP boxes inside these containers are not yet scrubbed (PNG and JPEG are fully covered).
|
||||
- **Video pipeline (`noai-video`)**: per-frame inpainting and tracking for Sora 2 dynamic logo, Veo 3.1 badge, Kling, Runway. Separate package, not folded into this repo.
|
||||
|
||||
The authors do not condone the use of this tool for deception, fraud,
|
||||
or any activity that violates applicable laws or regulations.
|
||||
Won't fix:
|
||||
|
||||
- **Nightshade / Glaze / PhotoGuard removal**. These are defensive perturbations used by artists to protect their work from being scraped into AI training sets. Removing them attacks artists, not AI provenance. Out of scope.
|
||||
|
||||
## Legal
|
||||
|
||||
Watermarking and provenance for AI-generated content is now regulated in several jurisdictions. The table below summarises the May 2026 status. None of this is legal advice.
|
||||
|
||||
| Jurisdiction | Instrument | Status (May 2026) | Relevance |
|
||||
| --- | --- | --- | --- |
|
||||
| EU | AI Act, Article 50(2) | Marking obligations postponed to **2 December 2026** under the December 2025 omnibus agreement. Code of Practice finalising May/June 2026. | Removing mandated provenance markers with intent to deceive may be sanctioned under national implementations. |
|
||||
| US (federal) | COPIED Act | Enacted 2025. | Criminalises removal of provenance information with intent to deceive about content origin. The tool itself is lawful; usage may not be. |
|
||||
| US (state) | CA AB 2655, TX SB 751, similar | In force. | Content-specific (election deepfakes, sexual deepfakes). Not tool-specific. |
|
||||
| China | Deep Synthesis Regulation, 2025 updates | In force. | Mandatory visible label for AI content. Removal is an administrative offence. |
|
||||
| UK | Online Safety Act, 2025 transparency extension | In force. | Platform obligations, not user obligations. |
|
||||
|
||||
## Threat model
|
||||
|
||||
This tool defends already-distributed AI imagery against automatic detection systems (social-platform "Made with AI" labels, third-party classifiers, content-policy filters). It does **not** retroactively anonymise generation.
|
||||
|
||||
In particular, **SynthID-Image v2** (Google, deployed October 2025 with Gemini 3 Pro / Nano Banana Pro / Imagen 4 / Veo) embeds a **136-bit payload** ([arxiv 2510.09263](https://arxiv.org/abs/2510.09263)). The payload is believed to encode a user / session identifier. If the original watermarked file ever passed through a system controlled by the prompt originator (a saved Gemini account history, a screenshot uploaded to a Google product, a backup), Google retains the ability to link that original to the generating account. Stripping the watermark from a copy you possess does not erase Google's server-side record.
|
||||
|
||||
Use cases where the threat model fits:
|
||||
- You generated the image yourself, want to publish it as your own work, and accept the consequences if Google ever publishes their detector logs.
|
||||
- You are running a security / robustness evaluation.
|
||||
- You are preserving art or historical record against false-positive "AI-generated" labels.
|
||||
|
||||
Use cases where the threat model **does not** fit:
|
||||
- Generating an image, expecting that removing the watermark anonymises you to Google. It doesn't.
|
||||
- Distributing AI-generated content while claiming human authorship. The watermark is one of several traceability layers.
|
||||
|
||||
This tool is intended for legitimate purposes such as:
|
||||
|
||||
- Privacy protection (removing metadata that leaks user account identifiers).
|
||||
- Art preservation and fair-use research.
|
||||
- Removing false-positive "Made with AI" labels from human-edited photographs.
|
||||
- Security research and watermark robustness study.
|
||||
|
||||
Removing AI provenance markers to misrepresent AI-generated content as human-created may violate the laws above, the DMCA, and platform terms of service. Users are solely responsible for ensuring their use complies with all applicable laws. The authors do not condone use of this tool for deception, fraud, or any activity that violates applicable laws or regulations.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -197,9 +197,14 @@ def cmd_visible(
|
||||
@click.option(
|
||||
"-o", "--output", type=click.Path(path_type=Path), default=None, help="Output path (default: <source>_clean.<ext>)."
|
||||
)
|
||||
@click.option("--strength", type=float, default=0.02, help="Denoising strength (0.0-1.0). Default: 0.02.")
|
||||
@click.option("--steps", type=int, default=100, help="Number of denoising steps. Default: 100.")
|
||||
@click.option("--pipeline", type=click.Choice(["default", "ctrlregen"]), default="default", help="Pipeline profile.")
|
||||
@click.option("--strength", type=float, default=0.05, help="Denoising strength (0.0-1.0). Default: 0.05.")
|
||||
@click.option("--steps", type=int, default=50, help="Number of denoising steps. Default: 50.")
|
||||
@click.option(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
)
|
||||
@click.option("--device", type=click.Choice(["auto", "cpu", "mps", "cuda"]), default="auto", help="Inference device.")
|
||||
@click.option("--seed", type=int, default=None, help="Random seed for reproducibility.")
|
||||
@click.option("--hf-token", type=str, default=None, help="HuggingFace API token.")
|
||||
@@ -334,13 +339,13 @@ def cmd_metadata(
|
||||
@click.option(
|
||||
"--inpaint-method", type=click.Choice(["ns", "telea", "gaussian"]), default="ns", help="Inpainting method."
|
||||
)
|
||||
@click.option("--strength", type=float, default=0.02, help="Invisible watermark denoising strength (0.0-1.0).")
|
||||
@click.option("--steps", type=int, default=100, help="Number of denoising steps for invisible removal.")
|
||||
@click.option("--strength", type=float, default=0.05, help="Invisible watermark denoising strength (0.0-1.0).")
|
||||
@click.option("--steps", type=int, default=50, help="Number of denoising steps for invisible removal.")
|
||||
@click.option(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile for invisible removal.",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
)
|
||||
@click.option("--model", type=str, default=None, help="HuggingFace model ID for invisible removal.")
|
||||
@click.option("--device", type=click.Choice(["auto", "cpu", "mps", "cuda"]), default="auto", help="Inference device.")
|
||||
@@ -600,7 +605,12 @@ def _process_batch_image(
|
||||
@click.option(
|
||||
"--humanize", type=float, default=0.0, help="Analog Humanizer film grain intensity (0 = off, typical: 2.0-6.0)."
|
||||
)
|
||||
@click.option("--pipeline", type=click.Choice(["default", "ctrlregen"]), default="default", help="Pipeline profile.")
|
||||
@click.option(
|
||||
"--pipeline",
|
||||
type=click.Choice(["default", "ctrlregen"]),
|
||||
default="default",
|
||||
help="Pipeline profile (default=SDXL, ctrlregen=CtrlRegen).",
|
||||
)
|
||||
@click.option("--device", type=click.Choice(["auto", "cpu", "mps", "cuda"]), default="auto", help="Inference device.")
|
||||
@click.option("--seed", type=int, default=None, help="Random seed for reproducibility.")
|
||||
@click.option("--hf-token", type=str, default=None, help="HuggingFace API token.")
|
||||
@@ -667,7 +677,7 @@ def cmd_batch(
|
||||
seed=seed,
|
||||
hf_token=hf_token,
|
||||
humanize=humanize,
|
||||
)
|
||||
)
|
||||
processed += 1
|
||||
|
||||
except Exception as e:
|
||||
|
||||
@@ -52,7 +52,10 @@ class InvisibleEngine:
|
||||
to break watermark patterns, and reconstructs via reverse diffusion.
|
||||
"""
|
||||
|
||||
DEFAULT_MODEL_ID = "Lykon/dreamshaper-8"
|
||||
# SDXL base is the default since May 2026: empirically defeats SynthID v2
|
||||
# at strength=0.05 / steps=50 / native ~1024px. See CLAUDE.md "Known
|
||||
# limitations" for the regression evidence ruling out SD-1.5 pipelines.
|
||||
DEFAULT_MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
CTRLREGEN_MODEL_ID = "yepengliu/ctrlregen"
|
||||
|
||||
def __init__(
|
||||
@@ -68,7 +71,8 @@ class InvisibleEngine:
|
||||
Args:
|
||||
model_id: HuggingFace model ID. None = use default for pipeline.
|
||||
device: Device for inference (auto/cpu/mps/cuda). None = auto.
|
||||
pipeline: Pipeline profile ("default" or "ctrlregen").
|
||||
pipeline: Pipeline profile. "default" (SDXL base, defeats SynthID
|
||||
v2) or "ctrlregen" (CtrlRegen).
|
||||
hf_token: HuggingFace API token.
|
||||
progress_callback: Optional callback for progress messages.
|
||||
"""
|
||||
@@ -123,7 +127,9 @@ class InvisibleEngine:
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
max_dimension = 768
|
||||
# SDXL is trained at 1024px and degrades both quality and watermark-removal
|
||||
# efficacy below that.
|
||||
max_dimension = 1024
|
||||
image = Image.open(image_path)
|
||||
image = ImageOps.exif_transpose(image)
|
||||
orig_size = image.size # (width, height)
|
||||
|
||||
@@ -60,6 +60,19 @@ AI_KEYWORDS: tuple[str, ...] = (
|
||||
"c2pa",
|
||||
)
|
||||
|
||||
# C2PA UUID used in ISOBMFF (AVIF, HEIF, MP4) ``uuid`` boxes.
|
||||
# Reference: https://spec.c2pa.org/specifications/specifications/2.1/specs/C2PA_Specification.html
|
||||
C2PA_UUID: bytes = bytes.fromhex("d8fec3d61b0e483c92975828877ec481")
|
||||
|
||||
# IPTC ``digitalSourceType`` values (IPTC 2025.1) that flag AI provenance.
|
||||
# Used by Instagram, Facebook, X (Twitter) to show "Made with AI" labels.
|
||||
IPTC_AI_MARKERS: tuple[bytes, ...] = (
|
||||
b"trainedAlgorithmicMedia",
|
||||
b"compositeSynthetic",
|
||||
b"algorithmicMedia",
|
||||
b"compositeWithTrainedAlgorithmicMedia",
|
||||
)
|
||||
|
||||
STANDARD_METADATA_KEYS: frozenset[str] = frozenset(
|
||||
[
|
||||
"Author",
|
||||
@@ -95,25 +108,38 @@ def has_ai_metadata(image_path: Path) -> bool:
|
||||
"""
|
||||
from PIL import Image
|
||||
|
||||
with Image.open(image_path) as img:
|
||||
for key in img.info:
|
||||
if _is_ai_key(key):
|
||||
return True
|
||||
# PIL may not handle AVIF/HEIF/JPEG-XL without the optional plugins
|
||||
# (ultralytics also monkey-patches Image.open in a way that can raise
|
||||
# ModuleNotFoundError when pi_heif autoload fails), so any open failure
|
||||
# falls through to the binary scan.
|
||||
try:
|
||||
with Image.open(image_path) as img:
|
||||
for key in img.info:
|
||||
if _is_ai_key(key):
|
||||
return True
|
||||
except Exception as exc:
|
||||
logger.debug("PIL could not open %s for metadata scan: %s", image_path, exc)
|
||||
|
||||
# Check C2PA
|
||||
# Check C2PA — via the official ``c2pa`` lib if available, otherwise via a
|
||||
# binary scan that also catches AVIF/HEIF/JPEG-XL containers (PIL doesn't
|
||||
# expose their metadata uniformly).
|
||||
try:
|
||||
from c2pa import has_c2pa_metadata
|
||||
|
||||
if has_c2pa_metadata(image_path):
|
||||
return True
|
||||
except ImportError:
|
||||
# Try simple binary scan (read only first 512KB to avoid OOM on huge files)
|
||||
with open(image_path, "rb") as f:
|
||||
data = f.read(512 * 1024)
|
||||
if b"c2pa" in data.lower() or b"C2PA" in data:
|
||||
return True
|
||||
pass
|
||||
|
||||
return False
|
||||
# Binary scan covers C2PA (PNG caBX, JPEG APP11, AVIF/HEIF/JXL uuid boxes)
|
||||
# and IPTC AI markers in XMP. Read only the first 512KB to bound memory.
|
||||
with open(image_path, "rb") as f:
|
||||
data = f.read(512 * 1024)
|
||||
if b"c2pa" in data.lower() or b"C2PA" in data:
|
||||
return True
|
||||
if C2PA_UUID in data:
|
||||
return True
|
||||
return any(marker in data for marker in IPTC_AI_MARKERS)
|
||||
|
||||
|
||||
def _scan_png_c2pa_chunk(image_path: Path) -> dict[str, str]:
|
||||
@@ -244,6 +270,19 @@ def remove_ai_metadata(
|
||||
if output_path is None:
|
||||
output_path = source_path
|
||||
|
||||
# AVIF/HEIF/JPEG-XL: strip C2PA boxes at the container level without
|
||||
# re-encoding. Avoids needing PIL plugins (pillow-heif / pillow-jxl) and
|
||||
# preserves pixel data bit-for-bit.
|
||||
if source_path.suffix.lower() in (".avif", ".heif", ".heic", ".jxl"):
|
||||
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
|
||||
|
||||
data = source_path.read_bytes()
|
||||
cleaned, stripped = strip_c2pa_boxes(data)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
output_path.write_bytes(cleaned)
|
||||
logger.info("Stripped %d C2PA box(es) → %s", stripped, output_path)
|
||||
return output_path
|
||||
|
||||
# Read image and filter metadata
|
||||
with Image.open(source_path) as img:
|
||||
img = img.copy()
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Minimal ISOBMFF box walker for stripping C2PA from AVIF / HEIF / MP4 / JPEG-XL.
|
||||
|
||||
The ISO Base Media File Format wraps content in nested ``[size:4][type:4][...]``
|
||||
boxes. C2PA stores its manifest in a top-level ``uuid`` box keyed by the
|
||||
C2PA UUID; JPEG-XL uses a ``jumb`` box (JUMBF) instead. To strip provenance
|
||||
without re-encoding the image, we walk the top-level box list, drop boxes that
|
||||
carry C2PA, and emit the rest verbatim. The codestream (``mdat`` for ISOBMFF,
|
||||
``jxlc`` / ``jxlp`` for JPEG-XL) is untouched, so pixel data is preserved
|
||||
bit-for-bit.
|
||||
|
||||
This file intentionally avoids dependencies on format-specific libraries
|
||||
(pillow-heif, pillow-jxl, pymp4) so it works on systems where they aren't
|
||||
installed.
|
||||
|
||||
Reference: ISO/IEC 14496-12 (ISOBMFF) and C2PA 2.1 spec §11.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import struct
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Iterator
|
||||
|
||||
from remove_ai_watermarks.metadata import C2PA_UUID
|
||||
|
||||
# Top-level box types that carry C2PA payload. ``uuid`` boxes are checked
|
||||
# against ``C2PA_UUID`` before being stripped; ``jumb`` boxes are always
|
||||
# stripped (JPEG-XL uses them exclusively for JUMBF).
|
||||
C2PA_BOX_TYPES: frozenset[bytes] = frozenset({b"uuid", b"jumb"})
|
||||
|
||||
|
||||
def _iter_top_level_boxes(data: bytes) -> Iterator[tuple[int, int, bytes, int]]:
|
||||
"""Yield ``(start, end, type, payload_offset)`` for each top-level box.
|
||||
|
||||
Handles all three ISOBMFF box-size encodings:
|
||||
- ``size > 1``: 32-bit size field is the total box length.
|
||||
- ``size == 1``: 64-bit ``largesize`` follows after the type field.
|
||||
- ``size == 0``: box runs to end of file.
|
||||
"""
|
||||
pos = 0
|
||||
n = len(data)
|
||||
while pos + 8 <= n:
|
||||
size32 = struct.unpack_from(">I", data, pos)[0]
|
||||
box_type = data[pos + 4 : pos + 8]
|
||||
if size32 == 1:
|
||||
if pos + 16 > n:
|
||||
return
|
||||
size = struct.unpack_from(">Q", data, pos + 8)[0]
|
||||
payload_off = pos + 16
|
||||
elif size32 == 0:
|
||||
size = n - pos
|
||||
payload_off = pos + 8
|
||||
else:
|
||||
size = size32
|
||||
payload_off = pos + 8
|
||||
if size < (payload_off - pos) or pos + size > n:
|
||||
return
|
||||
yield pos, pos + size, box_type, payload_off
|
||||
pos += size
|
||||
|
||||
|
||||
def is_isobmff(data: bytes) -> bool:
|
||||
"""Cheap sniff: ISOBMFF files start with an ``ftyp`` box."""
|
||||
return len(data) >= 8 and data[4:8] == b"ftyp"
|
||||
|
||||
|
||||
def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
|
||||
"""Return ``(cleaned_bytes, stripped_count)``.
|
||||
|
||||
Walks top-level boxes; drops any ``uuid`` box whose UUID equals
|
||||
``C2PA_UUID`` and any ``jumb`` box (JPEG-XL JUMBF container). All other
|
||||
boxes are emitted verbatim. If the input is not ISOBMFF-shaped, returns
|
||||
it unchanged.
|
||||
"""
|
||||
if not is_isobmff(data):
|
||||
return data, 0
|
||||
|
||||
out = bytearray()
|
||||
stripped = 0
|
||||
for start, end, box_type, payload_off in _iter_top_level_boxes(data):
|
||||
if box_type in C2PA_BOX_TYPES:
|
||||
if box_type == b"uuid":
|
||||
# uuid boxes carry the 16-byte UUID immediately after the type.
|
||||
if payload_off + 16 <= end and data[payload_off : payload_off + 16] == C2PA_UUID:
|
||||
stripped += 1
|
||||
continue
|
||||
else: # b"jumb"
|
||||
stripped += 1
|
||||
continue
|
||||
out.extend(data[start:end])
|
||||
return bytes(out), stripped
|
||||
@@ -5,7 +5,7 @@ Pure configuration and lookup functions with no ML dependencies.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
DEFAULT_MODEL_ID = "Lykon/dreamshaper-8"
|
||||
DEFAULT_MODEL_ID = "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
CTRLREGEN_MODEL_ID = "yepengliu/ctrlregen"
|
||||
|
||||
LOW_STRENGTH = 0.04
|
||||
|
||||
@@ -21,7 +21,8 @@ class TestInvisibleEngineInit:
|
||||
"""Tests for InvisibleEngine construction (no GPU required)."""
|
||||
|
||||
def test_default_model_id(self):
|
||||
assert InvisibleEngine.DEFAULT_MODEL_ID == "Lykon/dreamshaper-8"
|
||||
# SDXL base became the default in May 2026 (defeats SynthID v2).
|
||||
assert InvisibleEngine.DEFAULT_MODEL_ID == "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
|
||||
def test_ctrlregen_model_id(self):
|
||||
assert InvisibleEngine.CTRLREGEN_MODEL_ID == "yepengliu/ctrlregen"
|
||||
|
||||
@@ -54,6 +54,75 @@ class TestHasAiMetadata:
|
||||
def test_clean_image_no_ai(self, tmp_clean_png):
|
||||
assert not has_ai_metadata(tmp_clean_png)
|
||||
|
||||
def test_detects_c2pa_uuid_in_isobmff_container(self, tmp_path: Path):
|
||||
"""C2PA in AVIF/HEIF/MP4 lives in a ``uuid`` box identified by a fixed UUID.
|
||||
|
||||
Real AVIF/HEIF fixtures aren't shipped, so simulate the container by
|
||||
prepending an ISOBMFF-shaped ftyp box and the C2PA UUID bytes.
|
||||
"""
|
||||
from remove_ai_watermarks.metadata import C2PA_UUID
|
||||
|
||||
path = tmp_path / "fake.avif"
|
||||
# ftyp box: size(4) + 'ftyp' + 'avif' + minor_version(4) + 'avif'
|
||||
ftyp = b"\x00\x00\x00\x18ftypavif\x00\x00\x00\x00avifmif1"
|
||||
# uuid box: size(4) + 'uuid' + 16-byte UUID + minimal payload
|
||||
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"jumb-payload"
|
||||
path.write_bytes(ftyp + uuid_box + b"\x00" * 64)
|
||||
assert has_ai_metadata(path)
|
||||
|
||||
def test_strip_c2pa_boxes_removes_uuid_box(self, tmp_path: Path):
|
||||
"""ISOBMFF strip should drop the C2PA uuid box and keep everything else."""
|
||||
from remove_ai_watermarks.metadata import C2PA_UUID
|
||||
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
|
||||
|
||||
ftyp = b"\x00\x00\x00\x18ftypavif\x00\x00\x00\x00avifmif1"
|
||||
# uuid box: size(4) + 'uuid' + 16-byte UUID + minimal payload (8 bytes -> total 32)
|
||||
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"payload!"
|
||||
mdat = b"\x00\x00\x00\x10mdat" + b"pixeldat"
|
||||
cleaned, stripped = strip_c2pa_boxes(ftyp + uuid_box + mdat)
|
||||
assert stripped == 1
|
||||
assert cleaned == ftyp + mdat
|
||||
|
||||
def test_strip_c2pa_boxes_passthrough_for_non_isobmff(self):
|
||||
"""Non-ISOBMFF input must be returned unchanged."""
|
||||
from remove_ai_watermarks.noai.isobmff import strip_c2pa_boxes
|
||||
|
||||
data = b"\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR" + b"\x00" * 100
|
||||
cleaned, stripped = strip_c2pa_boxes(data)
|
||||
assert stripped == 0
|
||||
assert cleaned == data
|
||||
|
||||
def test_remove_ai_metadata_strips_c2pa_in_avif(self, tmp_path: Path):
|
||||
"""End-to-end: ``remove_ai_metadata`` on a fake .avif drops the C2PA box."""
|
||||
from remove_ai_watermarks.metadata import C2PA_UUID, remove_ai_metadata
|
||||
|
||||
src = tmp_path / "in.avif"
|
||||
ftyp = b"\x00\x00\x00\x18ftypavif\x00\x00\x00\x00avifmif1"
|
||||
uuid_box = b"\x00\x00\x00\x20uuid" + C2PA_UUID + b"payload!"
|
||||
mdat = b"\x00\x00\x00\x10mdat" + b"pixeldat"
|
||||
src.write_bytes(ftyp + uuid_box + mdat)
|
||||
|
||||
out = tmp_path / "out.avif"
|
||||
result = remove_ai_metadata(src, out)
|
||||
assert result == out
|
||||
assert out.read_bytes() == ftyp + mdat
|
||||
# And after stripping, detection must no longer flag the cleaned file.
|
||||
from remove_ai_watermarks.metadata import has_ai_metadata
|
||||
|
||||
assert not has_ai_metadata(out)
|
||||
|
||||
def test_detects_iptc_trained_algorithmic_media_marker(self, tmp_path: Path):
|
||||
"""Some pipelines embed only the IPTC AI marker in XMP, no C2PA manifest."""
|
||||
path = tmp_path / "fake.jpg"
|
||||
# Minimal JPEG-ish bytes containing the IPTC AI marker in an XMP-like blob.
|
||||
xmp = (
|
||||
b"<x:xmpmeta><Iptc4xmpExt:DigitalSourceType>"
|
||||
b"trainedAlgorithmicMedia"
|
||||
b"</Iptc4xmpExt:DigitalSourceType></x:xmpmeta>"
|
||||
)
|
||||
path.write_bytes(b"\xff\xd8\xff\xe1" + xmp + b"\xff\xd9")
|
||||
assert has_ai_metadata(path)
|
||||
|
||||
|
||||
class TestGetAiMetadata:
|
||||
"""Tests for extracting AI metadata."""
|
||||
|
||||
@@ -63,7 +63,7 @@ class TestModelProfiles:
|
||||
"""Tests for watermark_profiles.py."""
|
||||
|
||||
def test_default_profile(self):
|
||||
assert get_model_id_for_profile("default") == "Lykon/dreamshaper-8"
|
||||
assert get_model_id_for_profile("default") == "stabilityai/stable-diffusion-xl-base-1.0"
|
||||
|
||||
def test_ctrlregen_profile(self):
|
||||
assert get_model_id_for_profile("ctrlregen") == "yepengliu/ctrlregen"
|
||||
@@ -73,7 +73,7 @@ class TestModelProfiles:
|
||||
get_model_id_for_profile("nonexistent")
|
||||
|
||||
def test_detect_default(self):
|
||||
assert detect_model_profile("Lykon/dreamshaper-8") == "default"
|
||||
assert detect_model_profile("stabilityai/stable-diffusion-xl-base-1.0") == "default"
|
||||
|
||||
def test_detect_ctrlregen(self):
|
||||
assert detect_model_profile("yepengliu/ctrlregen") == "ctrlregen"
|
||||
|
||||
Reference in New Issue
Block a user