diff --git a/docs/module-internals.md b/docs/module-internals.md index ef29973..db18240 100644 --- a/docs/module-internals.md +++ b/docs/module-internals.md @@ -53,7 +53,7 @@ module. **Independence is source-grouped (`_CLASH_SOURCE`, added 2026-06-02):** the C2PA issuer attribution (`c2pa`) and the SynthID proxy (`synthid`) are NOT independent — the proxy is inferred from the *same* manifest — so they share one source and two vendors named within a single manifest do not clash. This killed a false-positive class found on the spaces corpus: legitimate multi-actor manifests where a product wraps another vendor's engine (Microsoft Designer on OpenAI → `OpenAI, Microsoft`; Microsoft on Google → `Microsoft, Google LLC, Google C2PA Core Generator Library`) or an edit chain re-signs (Adobe over a Gemini original → Adobe c2pa + Google synthid) — 19 such files across the 2026-06-01/02 batches read as clashes before the fix. Rule 1 still fires when a manifest vendor disagrees with a genuinely independent stamp (EXIF/XMP generator, IPTC `AISystemUsed`, AIGC, xAI); each non-`c2pa`/`synthid` family is its own source (`test_identify.py::TestIntegrityClashes::{test_multi_actor_manifest_no_clash,test_manifest_vendor_vs_independent_signal_clashes}`). Vendor normalization is `_vendor_of` over `_AI_VENDOR_TOKENS` (so a C2PA "Google (Gemini)" issuer and a SynthID-Google proxy agree, while different vendors clash). `_AI_VENDOR_TOKENS` covers ByteDance (all brands: bytedance/doubao/jimeng/dreamina/volcengine), Canva, ElevenLabs, and Black Forest Labs in addition to the OpenAI/Google/Adobe/... set — without them a transplanted ByteDance/Canva/BFL C2PA manifest next to an independent conflicting stamp was silently missed. **The generic `China AIGC (TC260)` label names no SPECIFIC vendor** (any Chinese generator applies it), so it cannot vendor-conflict in the spoofing sense: when a Chinese TC260-applying vendor (`_TC260_VENDORS`, today `{ByteDance}`) is co-attributed, Rule 1 attributes the label to that vendor (a legit Doubao image carries BOTH a ByteDance C2PA manifest and its own TC260 label and must not clash); against a NON-TC260 vendor (OpenAI etc.) the label stays generic and still clashes as a laundering tell (`test_bytedance_c2pa_plus_own_aigc_no_clash`, `test_foreign_vendor_plus_aigc_still_clashes`, `test_bytedance_c2pa_plus_foreign_generator_clashes`). Corpus-validated: 0 new clashes on 5000 ByteDance/AIGC/Canva/FLUX carriers. -**High-precision by design:** only hard generator stamps feed it (C2PA-issuer when source is AI, SynthID, EXIF/XMP generator, IPTC `AISystemUsed`, xAI, AIGC); the fuzzy visible sparkle and the open invisible watermark are **excluded** (the latter can be a by-product of our own SDXL removal pass). The c2pa vendor is classified from the issuer attribution / generator, NOT the resolved `platform` (a camera label like "Google Pixel" would mis-normalize to "Google"). All real single-origin fixtures (chatgpt/firefly/doubao/grok/mj) verified to produce **zero** clashes (false-positive guard in `test_identify.py::TestRealSamplesHaveNoClash`). +**High-precision by design:** only hard generator stamps feed it (C2PA-issuer when source is AI, SynthID, EXIF/XMP generator, IPTC `AISystemUsed`, xAI, AIGC); the fuzzy visible sparkle and the open invisible watermark are **excluded** (both are low-precision/positive-only signals; the open watermark was also historically a by-product of our own SDXL removal pass, until `watermark_remover` was fixed to load the SDXL pipelines with `add_watermarker=False` — it stays excluded as a fuzzy signal regardless). The c2pa vendor is classified from the issuer attribution / generator, NOT the resolved `platform` (a camera label like "Google Pixel" would mis-normalize to "Google"). All real single-origin fixtures (chatgpt/firefly/doubao/grok/mj) verified to produce **zero** clashes (false-positive guard in `test_identify.py::TestRealSamplesHaveNoClash`). **`ai_from_metadata` field + `has_invisible_target` helper (P0#5, 2026-06-22):** the high-confidence union (everything that sets `confidence == "high"`: C2PA AI-issuer / SynthID proxy, IPTC, AIGC, local gen params, EXIF/xAI, open DWT-DCT / TrustMark — the medium-confidence `hf_only`/`visible_only`/`samsung_only` are excluded) is now surfaced as the public `ProvenanceReport.ai_from_metadata` boolean, so callers gate on intent rather than on the `confidence` string. `has_invisible_target(path)` wraps `identify(path, check_visible=False, check_invisible=True)` and returns that field — it is the decision gate for the diffusion scrub (the CLI `invisible`/`all`/`batch` no-signal skip, `cli._no_invisible_signal_exit`): a visible-only or no-signal image has it False, so regeneration (which would only degrade a clean image) does not run. It fails SAFE — any detector exception returns True so the removal still runs (leaving a watermark on a paid removal is worse than over-regenerating). It does NOT prove a pixel SynthID is absent (SynthID is detectable only via its metadata proxy, gone once stripped), so a False means "no locally-detectable target", never "clean". Guarded by `test_identify.py::{TestIdentifyRealSamples::test_has_invisible_target_*,TestHasInvisibleTargetFailSafe}`. diff --git a/src/remove_ai_watermarks/noai/watermark_remover.py b/src/remove_ai_watermarks/noai/watermark_remover.py index f8aea0a..df705fe 100644 --- a/src/remove_ai_watermarks/noai/watermark_remover.py +++ b/src/remove_ai_watermarks/noai/watermark_remover.py @@ -447,6 +447,19 @@ class WatermarkRemover: self._set_progress("Loading fp16-fixed SDXL VAE (avoids black output)...") load_kwargs["vae"] = AutoencoderKL.from_pretrained(_SDXL_FP16_VAE_ID, torch_dtype=torch.float16) + @staticmethod + def _disable_sdxl_watermarker(load_kwargs: dict[str, Any]) -> None: + """Turn off the diffusers default invisible watermarker on an SDXL pipeline. + + diffusers embeds an open "Stable Diffusion XL" DWT-DCT invisible watermark on + EVERY SDXL output whenever ``invisible-watermark`` is installed (the ``detect`` + extra). A watermark REMOVER must not re-stamp a detectable AI watermark, or the + cleaned output re-reads as AI (``identify`` -> "Open invisible watermark: Stable + Diffusion XL"). Shared by both SDXL loaders; the ``ControlNetModel`` sub-model + and the Qwen loader never call it (only the pipeline accepts the kwarg). + """ + load_kwargs["add_watermarker"] = False + def _move_to_device_and_optimize(self, pipeline: Any) -> Any: """Move a freshly-loaded pipeline to ``self.device`` + enable memory opts. @@ -520,6 +533,7 @@ class WatermarkRemover: load_kwargs = self._base_load_kwargs() load_kwargs["safety_checker"] = None load_kwargs["requires_safety_checker"] = False + self._disable_sdxl_watermarker(load_kwargs) self._maybe_add_fp16_vae(load_kwargs) pipeline = self._load_from_pretrained(AutoImg2ImgPipeline, self.model_id, **load_kwargs) # type: ignore @@ -549,6 +563,7 @@ class WatermarkRemover: load_kwargs = self._base_load_kwargs() load_kwargs["controlnet"] = controlnet + self._disable_sdxl_watermarker(load_kwargs) self._maybe_add_fp16_vae(load_kwargs) self._set_progress(f"Loading model weights: {self.model_id}") diff --git a/tests/test_platform.py b/tests/test_platform.py index f3b2fde..37799ba 100644 --- a/tests/test_platform.py +++ b/tests/test_platform.py @@ -178,6 +178,54 @@ class TestFp16WeightVariant: assert "variant" not in cls.from_pretrained.call_args.kwargs +class TestNoReembeddedWatermark: + """F2 regression: the SDXL removal pipelines must disable the diffusers default + invisible watermarker (``add_watermarker=False``). + + diffusers stamps an open "Stable Diffusion XL" DWT-DCT watermark onto every SDXL + output whenever ``invisible-watermark`` is installed. A watermark REMOVER that left + it on would replace one detectable AI watermark (SynthID) with another -- the cleaned + output re-reads as AI. The ControlNet sub-model load must NOT receive the kwarg (it + is not a pipeline and does not accept it). + """ + + def _remover(self, profile: str): + if not is_watermark_removal_available(): + pytest.skip("torch/diffusers not installed") + from remove_ai_watermarks.noai.watermark_remover import WatermarkRemover + + return WatermarkRemover(device="cpu", pipeline=profile) + + def _capture(self, monkeypatch, remover): + from remove_ai_watermarks.noai.watermark_remover import WatermarkRemover + + calls: list[tuple[str, dict]] = [] + + def fake_load(self, cls, model_id, **kwargs): + calls.append((getattr(cls, "__name__", str(cls)), kwargs)) + return MagicMock() + + monkeypatch.setattr(WatermarkRemover, "_load_from_pretrained", fake_load) + monkeypatch.setattr(WatermarkRemover, "_move_to_device_and_optimize", lambda self, p: p) + return calls + + def test_sdxl_pipeline_disables_watermarker(self, monkeypatch: pytest.MonkeyPatch): + remover = self._remover("sdxl") + calls = self._capture(monkeypatch, remover) + remover._load_pipeline() + assert any(kw.get("add_watermarker") is False for _, kw in calls), calls + + def test_controlnet_pipeline_disables_watermarker(self, monkeypatch: pytest.MonkeyPatch): + remover = self._remover("controlnet") + calls = self._capture(monkeypatch, remover) + remover._load_controlnet_pipeline() + by_cls = dict(calls) + # the SDXL pipeline load disables the watermarker... + assert by_cls["StableDiffusionXLControlNetImg2ImgPipeline"].get("add_watermarker") is False + # ...but the ControlNet sub-model load must not carry the kwarg (it would error). + assert "add_watermarker" not in by_cls["ControlNetModel"] + + class _StubImage: """Minimal PIL.Image stand-in: just the ``width``/``height`` the pure helper reads."""