Compare commits

...
5 Commits
Author SHA1 Message Date
test-userandClaude Opus 4.7 d24d8a4b14 Extract _target_size helper + regression-test native resolution (v0.5.4)
The native-vs-downscale decision in InvisibleEngine.remove_watermark (the
issue #10/#15 fix: max_resolution=0 must not pre-downscale, since any
downscale both loses quality and lets SynthID survive) had no test. Extract
it into a pure helper invisible_engine._target_size(w, h, max_resolution)
and cover it with tests/test_invisible_engine.py::TestTargetSize so a
re-introduced forced downscale fails CI instead of silently regressing #15.

Also:
- Clamp the short side to >=1 in _target_size: extreme aspect ratios (e.g.
  5000x3 with --max-resolution 1024) truncated it to 0 and crashed
  image.resize(). Pre-existing in the inline math; fixed now that it is a
  named, tested function.
- Consolidate the two duplicated temp-file save blocks into one
  unconditional save (behavior unchanged: the EXIF-transposed image is
  still always persisted before WatermarkRemover reloads it by path), and
  drop the now-redundant `_tmp_path is not None` guard in finally.
- Bump version 0.5.3 -> 0.5.4 (pyproject, __init__, uv.lock); document the
  helper as the regression guard in CLAUDE.md.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 14:09:33 -07:00
test-userandClaude Opus 4.7 28fe13db8f Document native-res MPS OOM -> CPU-fallback behavior in limitations
Concrete data point from the 2026-05-25 gpt-image SDXL run: native
1254x1254 fp32 OOMs at the UNet step (not just VAE) on a 20 GB MPS
ceiling, and img2img_runner auto-falls back to CPU and completes
(slow, weight-identical, still defeats SynthID). enable_vae_tiling()
alone does not prevent it. Fast Mac workarounds: fp16 on MPS or
--max-resolution; neither is the default.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:57:13 -07:00
test-userandClaude Opus 4.7 59d72c5db7 Record verified gpt-image-2 SynthID-cleaned chain in corpus
Add manifest row for the 4ef377bd -> f6dd47a5 chain: a gpt-image-2 sample
(openai.com/verify: SynthID + C2PA detected) cleaned via v0.5.3 `all` at
native 1254x1254 (prod-equivalent SDXL base, strength 0.05, 50 steps).
openai.com/verify reports SynthID NOT detected after the run, re-confirming
that the #10 native-resolution default defeats OpenAI SynthID and resolving
the #15 root cause (older SD-1.5/768px downscale default did not).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 13:55:10 -07:00
test-userandClaude Opus 4.7 e27f24f520 test(samples): commit real Doubao fixture + AIGC real-sample test
data/samples/doubao-1.png is the real #13 sample: carries the China TC260
<TC260:AIGC> XMP label and a visible '豆包AI生成' text mark (bottom-right).
Grounds the AIGC detection on a real file (alongside the synthetic tests)
and serves as the fixture for visible-watermark removal work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:37:15 -07:00
test-userandClaude Opus 4.7 1afc1e60ef test(samples): add real Doubao TC260 AIGC reference sample
2048x2048 PNG carrying China's TC260 <TC260:AIGC> label; identify reports
it as a China AIGC-labeled generator (TC260). Reference fixture for manual
re-verification of the TC260 detection path -- the automated tests use
synthetic blobs, so nothing depends on this file being present.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-25 12:36:28 -07:00
9 changed files with 99 additions and 31 deletions

No files matched your search

+2 -2
View File
@@ -14,7 +14,7 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
- `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto
- `maintain.sh` may not finish fully green (pre-existing, not per-change): strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). (`uv-secure` is clean since idna was bumped 3.11 -> 3.16, fixing GHSA-65pc-fj4g-8rjx.) To gate a change, run `uv run ruff check`, `uv run pyright <changed files>`, `uv run pytest` directly.
- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch.
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `mj-*` = Midjourney IPTC, `doubao-1.png` = ByteDance Doubao with the China TC260 `<TC260:AIGC>` XMP label **and** a visible "豆包AI生成" text mark bottom-right, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths.
- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/` (`manifest.csv` tracked, `images/` gitignored); see its README for the collection protocol and verification oracles.
## Configuration
@@ -44,7 +44,7 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are
## Known limitations
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix.
- `invisible` pipeline processes at **native resolution by default** (`max_resolution=0`), matching the hosted raiw.cc backend (fal fast-sdxl, no pre-downscale). The old forced downscale-to-1024 -> upscale-back round-trip was the main quality loss (issue #10) and is gone; at strength ~0.05 SDXL img2img does not need the ~1024 downscale. `--max-resolution N` re-introduces an opt-in long-side cap purely to bound GPU/MPS memory on very large inputs (it reintroduces the lossy round-trip). For huge images that OOM at native, tile-based diffusion is still the proper long-term fix. **Concrete MPS data point (verified 2026-05-25 on a 1254x1254 gpt-image SDXL run, fp32, 20 GB MPS ceiling):** native res OOMs at the *UNet* step (peak ~17 GiB), not only the VAE decode, and the auto-fallback in `img2img_runner` reloads on CPU and finishes (slow, ~13 min) -- the output is still weight-identical and defeats SynthID, so "looks hung/crashed" on Mac is usually this CPU fallback, not a pipeline error. Adding `enable_vae_tiling()` alone does NOT prevent it (the peak is the UNet, not the VAE). The fast Mac workarounds are fp16 on MPS (roughly halves memory) or `--max-resolution` to cap the long side; neither is wired as the default. The native-vs-downscale decision lives in the pure helper `invisible_engine._target_size(w, h, max_resolution)` (returns `None` for native, a clamped target tuple otherwise) so it is unit-tested (`tests/test_invisible_engine.py::TestTargetSize`, the #10/#15 regression guard) without loading the model -- keep that logic in the helper, don't re-inline it.
- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files.
- `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`.
- Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`, plus EXIF `Software` / XMP `CreatorTool` generator tags via `metadata.exif_generator` (validated with synthesized AVIF/JPEG fixtures + an XMP raw-scan fixture). C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are read for detection but not yet **scrubbed** on removal.
Binary file not shown.

After

Width:  |  Height:  |  Size: 5.1 MiB

+1
View File
@@ -116,3 +116,4 @@ f9bb6b1039ee228221d2eead7daebd46422a36a4a8959a74fa0198e1aa19f9b8,f9bb6b10-a_red_
ca2162624607b8f030c63cb1371d6afdd3f0d7af3d9b17514f7ae64a2e8f378c,ca216262-0001_2_a-red-ceramic-coffee-mug-sits-on-weather_rs0Lt1beXV-Mwjs33vWKVw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
8a90f2ddf83ceedd7944531d27248191c79dc321ec76ea1f36df6cff9f6b8916,8a90f2dd-0001_3_a-red-ceramic-coffee-mug-sits-on-weather_Lm96lBwcXsmdeCfCpm52vw_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
87fca198d5c436b27a16685972802a2ff0a310a941248c9ec8d81560d7f387a1,87fca198-0001_4_a-red-ceramic-coffee-mug-sits-on-weather_pGmlgcH4XqSaq41hc0a9AQ_F7yI1GFGTayDA1NAUmkraw.jpeg,neg,Ideogram,ideogram-3,1024,1024,jpeg,,,none,2026-05-25T01:36:38Z,EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
f6dd47a5ffd319aea21bf10dcf9877097666420b02c2620080bac12b03976e7e,f6dd47a5-4ef377bd-gpt-image-2-cleaned.png,cleaned,"our pipeline (invisible/SDXL, native-res default)",stabilityai/stable-diffusion-xl-base-1.0,1254,1254,png,,,openai-verify,2026-05-25T20:50:38Z,"cleaned from 4ef377bd via v0.5.3 'all' at native 1254x1254 (prod-equivalent); openai.com/verify: SynthID NOT detected. Re-confirms #10 native-res default defeats OpenAI SynthID (closes #15 root cause). Note: native res OOMs on 20GB MPS, auto-fell back to CPU."
1 sha256 filename label source model width height format c2pa_issuer synthid_metadata verified_via added notes
116 ca2162624607b8f030c63cb1371d6afdd3f0d7af3d9b17514f7ae64a2e8f378c ca216262-0001_2_a-red-ceramic-coffee-mug-sits-on-weather_rs0Lt1beXV-Mwjs33vWKVw_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
117 8a90f2ddf83ceedd7944531d27248191c79dc321ec76ea1f36df6cff9f6b8916 8a90f2dd-0001_3_a-red-ceramic-coffee-mug-sits-on-weather_Lm96lBwcXsmdeCfCpm52vw_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
118 87fca198d5c436b27a16685972802a2ff0a310a941248c9ec8d81560d7f387a1 87fca198-0001_4_a-red-ceramic-coffee-mug-sits-on-weather_pGmlgcH4XqSaq41hc0a9AQ_F7yI1GFGTayDA1NAUmkraw.jpeg neg Ideogram ideogram-3 1024 1024 jpeg none 2026-05-25T01:36:38Z EXIF Make='Ideogram AI'; no C2PA/SynthID; caught via exif_generator
119 f6dd47a5ffd319aea21bf10dcf9877097666420b02c2620080bac12b03976e7e f6dd47a5-4ef377bd-gpt-image-2-cleaned.png cleaned our pipeline (invisible/SDXL, native-res default) stabilityai/stable-diffusion-xl-base-1.0 1254 1254 png openai-verify 2026-05-25T20:50:38Z cleaned from 4ef377bd via v0.5.3 'all' at native 1254x1254 (prod-equivalent); openai.com/verify: SynthID NOT detected. Re-confirms #10 native-res default defeats OpenAI SynthID (closes #15 root cause). Note: native res OOMs on 20GB MPS, auto-fell back to CPU.
+1 -1
View File
@@ -1,6 +1,6 @@
[project]
name = "remove-ai-watermarks"
version = "0.5.3"
version = "0.5.4"
description = "Remove visible and invisible AI watermarks from images (Gemini / Nano Banana, ChatGPT, Stable Diffusion)"
readme = "README.md"
requires-python = ">=3.10"
+1 -1
View File
@@ -1,3 +1,3 @@
"""Remove-AI-Watermarks: Unified tool for removing visible and invisible AI watermarks."""
__version__ = "0.5.3"
__version__ = "0.5.4"
+33 -25
View File
@@ -42,6 +42,24 @@ def is_available() -> bool:
return False
def _target_size(width: int, height: int, max_resolution: int) -> tuple[int, int] | None:
"""Compute the downscaled (width, height) for a long-side cap, or None for native.
Returns None when no pre-downscale is needed: ``max_resolution <= 0`` (native
resolution, the default that matches the raiw.cc backend -- see issue #10) or
the long side already fits the cap. Otherwise scales the long side down to
``max_resolution`` preserving aspect ratio (integer-truncated, matching the
PIL ``resize`` call site). Pure function so the native-vs-downscale decision
is unit-testable without loading the diffusion model.
"""
if max_resolution > 0 and max(width, height) > max_resolution:
ratio = max_resolution / max(width, height)
# Clamp the short side to >=1: extreme aspect ratios (e.g. 5000x3 capped
# at 1024) would otherwise truncate it to 0 and crash image.resize().
return (max(1, int(width * ratio)), max(1, int(height * ratio)))
return None
class InvisibleEngine:
"""Remove invisible AI watermarks using diffusion model regeneration.
@@ -142,37 +160,26 @@ class InvisibleEngine:
image = Image.open(image_path)
image = ImageOps.exif_transpose(image)
orig_size = image.size # (width, height)
_tmp_path = None
if max_resolution > 0 and max(image.width, image.height) > max_resolution:
ratio = max_resolution / max(image.width, image.height)
new_size = (int(image.width * ratio), int(image.height * ratio))
# Optional long-side downscale; native resolution by default (issue #10).
target = _target_size(image.width, image.height, max_resolution)
if target is not None:
if self._progress_callback:
self._progress_callback(
f"Downscaling {image.width}x{image.height} "
f"to {new_size[0]}x{new_size[1]} "
f"to {target[0]}x{target[1]} "
f"(max-resolution cap {max_resolution}px)..."
)
image = image.resize(new_size, Image.Resampling.LANCZOS)
image = image.resize(target, Image.Resampling.LANCZOS)
# Save to a temp file instead of overwriting the original
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
import os as _os
_os.close(_tmp_fd)
image_path = _tmp_path
else:
# We must save the transposed image back to a tmp file if it was rotated
# otherwise WatermarkRemover will reload it without EXIF rotation!
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
import os as _os
_os.close(_tmp_fd)
image_path = _tmp_path
# Always persist to a temp file, even without downscaling: WatermarkRemover
# reloads by path, so the EXIF-transposed pixels must be saved or rotation
# is lost. Cleaned up in the finally block via _tmp_path.
_tmp_fd, _tmp_str = tempfile.mkstemp(suffix=image_path.suffix)
_tmp_path = Path(_tmp_str)
image.save(_tmp_path)
os.close(_tmp_fd)
image_path = _tmp_path
try:
# Optional: Face protection (Phase 1 - Extraction)
@@ -253,7 +260,8 @@ class InvisibleEngine:
return out_path
finally:
if _tmp_path is not None and _tmp_path.exists():
# _tmp_path is always set above (we persist the image unconditionally).
if _tmp_path.exists():
_tmp_path.unlink()
def remove_watermark_batch(
+43 -1
View File
@@ -2,7 +2,7 @@
from __future__ import annotations
from remove_ai_watermarks.invisible_engine import InvisibleEngine, is_available
from remove_ai_watermarks.invisible_engine import InvisibleEngine, _target_size, is_available
class TestIsAvailable:
@@ -26,3 +26,45 @@ class TestInvisibleEngineInit:
def test_ctrlregen_model_id(self):
assert InvisibleEngine.CTRLREGEN_MODEL_ID == "yepengliu/ctrlregen"
class TestTargetSize:
"""Regression guard for the native-resolution decision (issues #10 / #15).
max_resolution=0 must NOT downscale -- the forced downscale->upscale
round-trip was the quality loss in #10, and downscaling at all let SynthID
survive in #15 (the native SDXL pass at strength ~0.05 is what defeats it).
"""
def test_native_default_no_downscale(self):
# The default (0) means native resolution: no resize, regardless of size.
assert _target_size(4096, 4096, 0) is None
assert _target_size(123, 456, 0) is None
def test_negative_cap_treated_as_native(self):
assert _target_size(4096, 4096, -1) is None
def test_cap_below_long_side_downscales(self):
# 2000x1000, cap 1024 -> long side scaled to 1024, aspect preserved.
assert _target_size(2000, 1000, 1024) == (1024, 512)
def test_cap_uses_long_side_for_portrait(self):
# Portrait: height is the long side, so it drives the ratio.
assert _target_size(1000, 2000, 1024) == (512, 1024)
def test_cap_at_or_above_long_side_no_downscale(self):
# Already within the cap (and exactly equal) -> no resize.
assert _target_size(800, 600, 1024) is None
assert _target_size(1024, 768, 1024) is None
def test_integer_truncation_matches_pil_call_site(self):
# 1254x1254 (the gpt-image sample) capped at 1000: int(1254*1000/1254)=1000.
assert _target_size(1254, 1254, 1000) == (1000, 1000)
# Non-divisible ratio truncates toward zero like int() at the call site.
assert _target_size(1000, 333, 500) == (500, 166)
def test_extreme_aspect_ratio_clamps_short_side_to_one(self):
# 5000x3 capped at 1024: int(3 * 1024/5000) = 0 would crash resize();
# the short side must clamp to 1, never 0.
assert _target_size(5000, 3, 1024) == (1024, 1)
assert _target_size(3, 5000, 1024) == (1, 1024)
+17
View File
@@ -433,3 +433,20 @@ class TestAIGCLabel:
meta = get_ai_metadata(self._aigc_png(tmp_path))
assert "aigc_label" in meta
assert "TC260" in meta["aigc_label"]
@pytest.mark.skipif(not (SAMPLES_DIR / "doubao-1.png").exists(), reason="doubao sample not present")
class TestAIGCRealSample:
"""Real Doubao (ByteDance) sample carries the China TC260 AIGC XMP label."""
def test_doubao_aigc_label(self):
from remove_ai_watermarks.metadata import aigc_label
info = aigc_label(SAMPLES_DIR / "doubao-1.png")
assert info is not None
assert info["Label"] == "1"
assert info["ContentProducer"] # ByteDance producer code present
def test_doubao_detected_as_ai(self):
assert has_ai_metadata(SAMPLES_DIR / "doubao-1.png")
assert "aigc_label" in get_ai_metadata(SAMPLES_DIR / "doubao-1.png")
Generated
+1 -1
View File
@@ -2150,7 +2150,7 @@ wheels = [
[[package]]
name = "remove-ai-watermarks"
version = "0.5.3"
version = "0.5.4"
source = { editable = "." }
dependencies = [
{ name = "click" },