mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-06-05 18:46:34 +02:00
58bdf51c59
* fix(trustmark): gate detection on re-encode durability to kill false positives TrustMark's wm_present flag is a BCH validity check that spuriously validates on a content-correlated fraction of un-watermarked images (AI textures trip it more than camera photos). On a 1343-image set all 20 raw detections were false, several on Gemini/OpenAI/Doubao output that cannot carry Adobe's watermark, with random-bytes secrets. A genuine TrustMark is a durable soft binding that survives re-encoding, so detect_trustmark now re-decodes after a mild JPEG round-trip and requires the same schema both times. Every observed false positive collapsed under this gate; the second decode runs only on the rare hit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(identify): Samsung Galaxy AI, FLUX, ByteDance C2PA; fix C2PA substring FP Detection extensions verified on real signed files (2026-05-29): - Samsung Galaxy AI: signer attribution via a new _SIGNER_C2PA_PLATFORM (Samsung Galaxy / ASUS Gallery) kept separate from the capture-camera _DEVICE_C2PA_PLATFORM so a Galaxy AI edit (device cert + AI source type) does not trip the camera-vs-AI integrity clash. Plus metadata.samsung_genai: the proprietary genAIType marker in PhotoEditor_Re_Edit_Data, a medium- confidence AI-editing signal (samsung_only branch). - Black Forest Labs (FLUX) and ByteDance Volcano Engine (Doubao/Jimeng) added as C2PA issuers + issuer->platform mappings. - fix: C2PA presence required only the bare 4-byte 'c2pa' substring, which false-positives on compressed pixel data (a recompressed PNG IDAT re-flagged C2PA after its manifest was correctly stripped). New c2pa_marker_in() requires the JUMBF wrapper (jumb+c2pa) or the C2PA uuid box; applied in identify + metadata. Verified: all 535 real C2PA files carry jumb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): gate detection on text structure to cut ~95% of false positives (#23) Coverage alone over-fired: any textured bottom-right corner cleared the threshold, so the detector false-positived on ~28% of arbitrary images. The real '豆包AI生成' mark is six glyphs in one row, so detect now also requires the text-structure signature (_glyph_structure): many connected components, no single dominant blob, concentration in a thin horizontal band. False positives dropped 343 -> 17 across the corpus while keeping real-mark recall and the doubao-1.png sample. Also accept a no-op force kwarg for remover-interface symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(samsung): add Samsung Galaxy AI visible-badge remover New samsung_engine.py removes the bottom-left sparkle + localized 'AI-generated content' badge that Galaxy AI tools stamp. Mirrors the Doubao locate->mask->inpaint pattern but bottom-left, with a dual-polarity top-hat mask (the badge is light-on-dark or dark-on-light). Detection gates on a band + left-anchor signature (the Doubao CJK-component gate does not transfer: Latin badge letters connect into few blobs). Explicit-only -- tuned on few real badges with a ~4% FP floor, so it is not used in auto. Synthetic byte-blob fixtures (real badges are user content, not shipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(visible): unified known-watermark registry + LaMa inpaint backend watermark_registry.py is a single catalog of known visible marks, each tying {usual location, in_auto flag, recovery strategy, detect adapter, remove adapter}: gemini (reverse-alpha, exact), doubao, samsung. cmd_visible is now registry-driven (best_auto_mark for --mark auto; mark_keys() feeds the CLI choices) -- the per-mark _run_doubao/_run_samsung helper branches are gone. Cross-engine confidences are not comparable, so the gemini adapter applies the corpus-validated 0.5 sparkle threshold for auto arbitration (its engine flag is loose and weakly fired ~0.36 on Doubao text, hijacking auto). --backend auto|cv2|lama chooses background reconstruction for the mask-based marks; auto = LaMa when onnxruntime is present, else cv2. For LaMa the mask is the FILLED glyph bounding box (sparse glyph masks leave anti-aliased edges behind). cv2 stays the zero-dependency fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: watermark registry, Samsung/FLUX/ByteDance detection, LaMa backend, trustmark gate Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(doubao): exact reverse-alpha removal from captured alpha map The Doubao '豆包AI生成' mark is a fixed semi-transparent white overlay, so given its alpha map the original pixels are recovered exactly: original = (wm - a*logo)/(1-a) -- no inpaint hallucination. The alpha map + logo colour were solved from real black+gray Doubao captures on a controlled background: on black captured = a*logo, and the black/gray pair solves a per-pixel without assuming the logo colour (a_max~0.65, logo near-white); the white capture cross-validates (mark vanishes to a flat fill). Bundled as assets/doubao_alpha.png + geometry constants. remove_watermark_reverse_alpha applies it scaled to image width; exact at the captured width, so the registry routes doubao through it only when reverse_alpha_available (width within the calibrated band) and the mark is detected, falling back to mask inpaint (cv2/LaMa) otherwise. A light residual inpaint cleans the sub-pixel rescaling error. Add captures at more resolutions to widen exact coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(visible): reverse-alpha only -- drop inpaint removal + heuristic detection Per the principle that we only remove/detect what we can do exactly, the visible-mark path is now reverse-alpha only: - Doubao detect is reverse-alpha-consistent: match the bundled alpha glyph silhouette against the corner via TM_CCOEFF_NORMED (DETECT_NCC_THRESHOLD 0.4) -- keys on the '豆包AI生成' SHAPE, not coverage/structure heuristics. FP 7/1243 (0.6%). Removes the cv2 inpaint path + the _glyph_structure gate. - Registry is reverse-alpha only: dropped the cv2/LaMa backend (_glyph_remove, _lama_box_inpaint, default_backend, --backend) and the Samsung entry. Doubao outside the alpha resolution band is skipped, never inpainted. - Removed samsung_engine.py + tests + --mark samsung (no alpha map captured; Samsung C2PA/genAIType metadata detection in identify is unaffected). - The universal erase --region (cv2/LaMa) is unchanged -- arbitrary-region inpainting stays a user-directed tool, separate from the known-mark registry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(doubao): NCC sub-pixel alignment -> reverse-alpha at any resolution A pure width-scale of the captured alpha map is only sub-pixel-accurate at the captured width and leaves a faint ghost elsewhere. remove_watermark_reverse_alpha now registers the alpha glyph to the actual mark via a TM_CCOEFF_NORMED scale+position search (_aligned_alpha_map) before inverting the blend, so the single 2048 capture works at any resolution -- verified clean on the 1773x2364 (3:4) corpus size, the biggest coverage gap (23 files). reverse_alpha_available is now just 'asset present' (no width band); the registry still gates removal on detect so a clean corner is never touched. Drops the _ALPHA_WIDTH_TOLERANCE gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): keep native recovery exact -- fixed geometry at captured width Integer-pixel NCC alignment landed ~1px off at the captured width, degrading the otherwise-exact native reverse-alpha (synthetic recovery error 0.94 -> 1.39). remove_watermark_reverse_alpha now uses exact width-relative geometry within _ALPHA_NATIVE_BAND of the captured width and the NCC search only off it -- best of both: native back to 0.94, other resolutions still aligned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): harden alignment -- try fixed+aligned, keep least residual (56/56) On a faint/busy-background mark the NCC alignment peak can wander a few px off the true mark and leave a residual (2/56 real corpus files). Off the captured width, remove_watermark_reverse_alpha now builds BOTH the fixed-geometry and the NCC-aligned alpha map, applies each, and keeps whichever leaves the least residual mark (re-detect confidence on the bare reverse-alpha) -- geometry wins on faint marks, alignment on clear ones, no magic threshold. Real-file round-trip now removes 56/56 detected Doubao clean across every corpus resolution (was 54). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(doubao): skip residual inpaint at native width for exact recovery At the captured width the fixed-geometry reverse-alpha is pixel-exact, so inpainting over it only replaced exactly-recovered interior pixels with a cv2 hallucination -- measured worse on a textured background (native error vs true bg 1.6 reverse-alpha-only vs 2.6 with the old always-on full-footprint inpaint). Native now returns the bare recovery untouched; off-native, where NCC alignment is only sub-pixel-approximate, the footprint inpaint stays to clean the seam. Real round-trip still 56/56 across all corpus resolutions; negatives 0/60, Gemini unaffected. Add test_native_returns_exact_reverse_alpha_no_inpaint as the regression guard. Sync CLAUDE.md + README (the table cell and prose described the pre-NCC "skipped off native / cv2-LaMa" behavior, now stale). Gitignore the session scheduled_tasks.lock, and add the text-protection research note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
164 lines
6.6 KiB
Python
164 lines
6.6 KiB
Python
"""Tests for the Doubao visible-watermark engine (reverse-alpha only)."""
|
|
|
|
from __future__ import annotations
|
|
|
|
from pathlib import Path
|
|
|
|
import cv2
|
|
import numpy as np
|
|
import pytest
|
|
|
|
from remove_ai_watermarks.doubao_engine import (
|
|
_ALPHA_HEIGHT_FRAC,
|
|
_ALPHA_LOGO_BGR,
|
|
_ALPHA_MARGIN_BOTTOM_FRAC,
|
|
_ALPHA_MARGIN_RIGHT_FRAC,
|
|
_ALPHA_NATIVE_WIDTH,
|
|
_ALPHA_WIDTH_FRAC,
|
|
DETECT_NCC_THRESHOLD,
|
|
DoubaoEngine,
|
|
_alpha_template,
|
|
_glyph_silhouette,
|
|
_template_match_score,
|
|
load_image_bgr,
|
|
)
|
|
|
|
SAMPLE = Path(__file__).resolve().parents[1] / "data" / "samples" / "doubao-1.png"
|
|
|
|
|
|
class TestLocate:
|
|
def test_box_anchored_bottom_right(self):
|
|
eng = DoubaoEngine()
|
|
img = np.zeros((2048, 2048, 3), np.uint8)
|
|
loc = eng.locate(img)
|
|
assert 2048 - (loc.x + loc.w) < int(2048 * 0.03)
|
|
assert 2048 - (loc.y + loc.h) < int(2048 * 0.03)
|
|
|
|
def test_box_scales_with_width(self):
|
|
eng = DoubaoEngine()
|
|
small = eng.locate(np.zeros((1024, 1024, 3), np.uint8))
|
|
large = eng.locate(np.zeros((2048, 2048, 3), np.uint8))
|
|
assert large.w == pytest.approx(small.w * 2, rel=0.1)
|
|
|
|
|
|
# ── Detection: alpha-template NCC ───────────────────────────────────
|
|
|
|
|
|
class TestDetect:
|
|
def test_clean_gradient_not_detected(self):
|
|
eng = DoubaoEngine()
|
|
ramp = np.tile(np.linspace(0, 255, 1024, dtype=np.uint8), (1024, 1))
|
|
img = cv2.cvtColor(ramp, cv2.COLOR_GRAY2BGR)
|
|
assert not eng.detect(img).detected
|
|
|
|
def test_solid_blob_corner_not_detected(self):
|
|
"""A bright blob is not the glyph shape -> low correlation, not detected."""
|
|
eng = DoubaoEngine()
|
|
img = np.zeros((1024, 1024, 3), np.uint8)
|
|
x, y, bw, bh = eng.locate(img).bbox
|
|
img[y + bh // 4 : y + bh * 3 // 4, x : x + bw // 2] = 200
|
|
assert not eng.detect(img).detected
|
|
|
|
def test_silhouette_loads(self):
|
|
sil = _glyph_silhouette()
|
|
assert sil is not None
|
|
assert set(np.unique(sil)).issubset({0, 255})
|
|
|
|
def test_match_score_shape_sensitive(self):
|
|
"""The glyph silhouette correlates with itself, not with a filled block."""
|
|
sil = _glyph_silhouette()
|
|
h, w = sil.shape
|
|
# box that contains the silhouette -> high score
|
|
box = np.zeros((h + 8, int(w / _ALPHA_WIDTH_FRAC * 0.2) + w), np.uint8)
|
|
box[4 : 4 + h, 4 : 4 + w] = sil
|
|
assert _template_match_score(box, _ALPHA_NATIVE_WIDTH) >= DETECT_NCC_THRESHOLD
|
|
# a uniformly filled box has no glyph structure -> low score
|
|
solid = np.full_like(box, 255)
|
|
assert _template_match_score(solid, _ALPHA_NATIVE_WIDTH) < DETECT_NCC_THRESHOLD
|
|
|
|
|
|
@pytest.mark.skipif(not SAMPLE.exists(), reason="sample image not present")
|
|
class TestRealSample:
|
|
def test_detects_watermark(self):
|
|
det = DoubaoEngine().detect(load_image_bgr(SAMPLE))
|
|
assert det.detected
|
|
assert det.confidence >= DETECT_NCC_THRESHOLD
|
|
|
|
def test_reverse_alpha_removes_mark(self):
|
|
eng = DoubaoEngine()
|
|
img = load_image_bgr(SAMPLE)
|
|
assert eng.reverse_alpha_available(img) # sample is at the captured width
|
|
out = eng.remove_watermark_reverse_alpha(img)
|
|
assert not eng.detect(out).detected # mark gone after recovery
|
|
|
|
def test_far_region_untouched(self):
|
|
eng = DoubaoEngine()
|
|
img = load_image_bgr(SAMPLE)
|
|
out = eng.remove_watermark_reverse_alpha(img)
|
|
h, w = img.shape[:2]
|
|
assert np.array_equal(img[: h // 2, : w // 2], out[: h // 2, : w // 2])
|
|
|
|
|
|
# ── Reverse-alpha (exact recovery) ──────────────────────────────────
|
|
|
|
|
|
class TestReverseAlpha:
|
|
def test_alpha_asset_loads(self):
|
|
at = _alpha_template()
|
|
assert at is not None
|
|
assert at.dtype.kind == "f"
|
|
assert float(at.min()) >= 0.0
|
|
assert float(at.max()) <= 1.0
|
|
|
|
def test_available_whenever_asset_present(self):
|
|
# NCC alignment generalizes to any resolution, so availability is just
|
|
# "asset loadable" (any non-empty image); the caller gates on detect.
|
|
eng = DoubaoEngine()
|
|
assert eng.reverse_alpha_available(np.zeros((1024, 1024, 3), np.uint8))
|
|
assert eng.reverse_alpha_available(np.zeros((1773, 1535, 3), np.uint8))
|
|
assert not eng.reverse_alpha_available(np.zeros((0, 0, 3), np.uint8))
|
|
|
|
@staticmethod
|
|
def _compose(w: int, h: int, bg: float = 100.0):
|
|
"""Composite the real alpha (scaled to width ``w``) onto a flat bg.
|
|
Returns ``(watermarked_uint8, mark_bool_mask)``."""
|
|
img = np.full((h, w, 3), bg, np.float32)
|
|
at = _alpha_template()
|
|
gw, gh = int(_ALPHA_WIDTH_FRAC * w), int(_ALPHA_HEIGHT_FRAC * w)
|
|
ax = w - int(_ALPHA_MARGIN_RIGHT_FRAC * w) - gw
|
|
ay = h - int(_ALPHA_MARGIN_BOTTOM_FRAC * w) - gh
|
|
amap = np.zeros((h, w), np.float32)
|
|
amap[ay : ay + gh, ax : ax + gw] = cv2.resize(at, (gw, gh))
|
|
a3 = amap[:, :, None]
|
|
wm = (a3 * np.array(_ALPHA_LOGO_BGR, np.float32) + (1 - a3) * img).clip(0, 255).astype(np.uint8)
|
|
return wm, amap > 0.2
|
|
|
|
def test_native_returns_exact_reverse_alpha_no_inpaint(self):
|
|
"""At native width the recovery is exact, so it must be returned untouched
|
|
-- inpainting over exactly-recovered interior pixels degrades quality
|
|
(regression: native textured error 1.6 reverse-alpha-only vs 2.6 with the
|
|
old full-footprint inpaint). The output must equal pure reverse-alpha."""
|
|
eng = DoubaoEngine()
|
|
wm, _mark = self._compose(_ALPHA_NATIVE_WIDTH, _ALPHA_NATIVE_WIDTH)
|
|
out = eng.remove_watermark_reverse_alpha(wm)
|
|
amap = eng._fixed_alpha_map(wm)
|
|
assert amap is not None
|
|
expected = eng._apply_reverse_alpha(wm, amap[0])
|
|
assert np.array_equal(out, expected) # no inpaint touched the recovery
|
|
|
|
@pytest.mark.parametrize(
|
|
("w", "h", "max_err"),
|
|
[
|
|
(_ALPHA_NATIVE_WIDTH, _ALPHA_NATIVE_WIDTH, 5.0), # native 1:1 -> fixed geometry, ~exact
|
|
(1773, 2364, 8.0), # 3:4 portrait -> NCC alignment generalizes the single capture
|
|
],
|
|
)
|
|
def test_recovers_flat_background(self, w, h, max_err):
|
|
"""Recovers the flat background at native (fixed geometry, exact) AND a
|
|
non-native resolution (NCC alignment generalizes the single capture)."""
|
|
eng = DoubaoEngine()
|
|
wm, mark = self._compose(w, h)
|
|
assert float(np.abs(wm.astype(np.float32)[mark] - 100.0).mean()) > 15 # mark visible
|
|
out = eng.remove_watermark_reverse_alpha(wm).astype(np.float32)
|
|
assert float(np.abs(out[mark] - 100.0).mean()) < max_err
|