mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-07 14:38:35 +02:00
Restructure documentation, validate metadata removal, consolidate assets
This commit is contained in:
@@ -1,22 +1,13 @@
|
||||
"""Shared base for the visible text-mark detectors/localizers (localize -> fill).
|
||||
"""Shared base for visible text-mark detectors and localizers.
|
||||
|
||||
The Doubao "豆包AI生成", Jimeng "★ 即梦AI", and Samsung "✦ Contenuti generati
|
||||
dall'AI" marks are the SAME algorithm: anchor a bottom-corner box by geometry
|
||||
relative to the image's SHORT side, extract the light low-saturation glyph candidate (white top-hat), detect
|
||||
by matching the bundled alpha-glyph silhouette via ``TM_CCOEFF_NORMED``, and build a
|
||||
removal MASK from the glyph blob's bounding box (:meth:`footprint_mask`) for the
|
||||
shared fill (region_eraser). The mask is template-FREE -- the top-hat glyph bbox, not
|
||||
a fixed alpha-template placement -- so a re-rendered or differently-placed mark (e.g.
|
||||
a non-Italian Samsung string) is still masked. The old reverse-alpha pixel recovery
|
||||
(``original = (wm - a*logo)/(1-a)``) is gone.
|
||||
Each mark supplies a :class:`TextMarkConfig` with its silhouette, expected area,
|
||||
scale ladder, detector frontend, and calibrated gates. The shared engine locates the
|
||||
candidate, scores the silhouette with normalized correlation, and builds a removal
|
||||
footprint for the common fill backend. Individual engines may override detection or
|
||||
footprint behavior when their measured variant requires it.
|
||||
|
||||
They differ ONLY in a bounded set of tuned values captured by :class:`TextMarkConfig`:
|
||||
the constants, the bundled silhouette asset, the corner (Doubao/Jimeng bottom-right,
|
||||
Samsung bottom-left), and a few structural knobs. Each engine module is a thin
|
||||
:class:`TextMarkEngine` subclass plus the test-facing module constants/helpers.
|
||||
|
||||
Gemini stays a SEPARATE engine (``gemini_engine``): its multi-size sparkle model is
|
||||
genuinely different, not a tuned variant of this one.
|
||||
The removal path never performs reverse-alpha pixel recovery. Gemini and the Jimeng
|
||||
pill remain separate engines because their geometry and gating differ from text marks.
|
||||
"""
|
||||
|
||||
# cv2/numpy boundary: third-party libs ship no usable element types; relax the
|
||||
@@ -61,7 +52,7 @@ _MIN_DETECT_SHORT_SIDE = 200
|
||||
# This used to be ONE shared 0.7 for every text mark. Measured 2026-07-18 on the
|
||||
# `auto` path (the default -- no flag, driven by TC260 metadata), it turned out to
|
||||
# mean two completely different things per mark. Blind hand-label of the ADDITIONS
|
||||
# (accepted with provenance, rejected without) over 4417 unique TC260 carriers,
|
||||
# (accepted with provenance, rejected without) over a labelled TC260 evaluation set,
|
||||
# two-sided control (labeller sensitivity 100%/96%, specificity 100%/100%):
|
||||
#
|
||||
# mark band precision 95% CI n
|
||||
@@ -182,9 +173,9 @@ class TextMarkDetection:
|
||||
coverage: float = 0.0 # fraction of the box occupied by glyph pixels
|
||||
|
||||
|
||||
# Alpha / silhouette templates, cached per asset name (the originals cached per
|
||||
# module global; this keys by asset so the three engines share the loader without
|
||||
# re-reading). Only SUCCESSFUL loads are cached, so a missing asset is retried.
|
||||
# Alpha / silhouette templates, cached per asset name. This shared cache lets every
|
||||
# text-mark engine reuse the loader without re-reading an asset. Only SUCCESSFUL loads
|
||||
# are cached, so a missing asset is retried.
|
||||
_alpha_cache: dict[str, NDArray[Any]] = {}
|
||||
_silhouette_cache: dict[str, NDArray[Any]] = {}
|
||||
|
||||
@@ -289,21 +280,18 @@ class TextMarkEngine:
|
||||
Two marks sharing a corner and a script (Doubao "豆包AI生成" and Jimeng
|
||||
"★ 即梦AI", both bottom-right, both near-white CJK) survive binarization into
|
||||
very similar blobs, so an absolute gate cannot separate them -- and under the
|
||||
provenance relaxation it stopped trying: 33 of jimeng's 68 false additions
|
||||
were Doubao marks (corpus-measured 2026-07-18).
|
||||
provenance relaxation it stopped trying because many Jimeng false additions
|
||||
were actually Doubao marks.
|
||||
|
||||
Measured separability on hand-labelled examples, scoring BOTH templates
|
||||
against the same glyph blob (n=40 jimeng / 75 doubao / 20 other-vendor labels
|
||||
/ 89 clean):
|
||||
against the same glyph blob:
|
||||
|
||||
feature separability (0.5 = useless, 1.0 = perfect)
|
||||
absolute ncc_jimeng 0.96
|
||||
ncc_jimeng MINUS ncc_doubao 0.99
|
||||
|
||||
At a 0.10 margin: real Jimeng wordmarks pass 100%, Doubao strips 8%, other
|
||||
vendors' AI labels (千问/百度/星绘/抖音) 55%, no-mark corners 12%. Because the
|
||||
real marks pass at 100%, this costs NO recall -- it is a pure precision gain,
|
||||
unlike raising the threshold, which trades recall away.
|
||||
A 0.10 margin separated Jimeng wordmarks from the rival and clean examples
|
||||
without reducing recall, unlike raising the absolute threshold.
|
||||
|
||||
Marks with no same-corner rival declare `rivals=()` and are unaffected.
|
||||
"""
|
||||
@@ -327,8 +315,7 @@ class TextMarkEngine:
|
||||
thin translucent overlay shatters into specks under the threshold, and no
|
||||
template can match a blob that is not there.
|
||||
|
||||
Measured 2026-07-18 on hand-verified corpus positives (40 doubao, 14 千问, 60
|
||||
verified-clean), scoring each mark with its own template:
|
||||
Calibration on hand-verified examples, scoring each mark with its own template:
|
||||
|
||||
front-end doubao clean neg AUC doubao/neg
|
||||
binary 0.723 ~0.12 --
|
||||
@@ -484,19 +471,15 @@ class TextMarkEngine:
|
||||
were all calibrated on PORTRAIT captures, where width and short side coincide,
|
||||
so the basis was never exercised until landscape inputs were measured.
|
||||
|
||||
Corpus-measured 2026-07-18 (2572 unique TC260 carriers; the harness is
|
||||
`scripts/visible_eval.py`). Before any fix, doubao detection by aspect ratio:
|
||||
portrait 60%, square 41%, **landscape 0% (0 of 435)** -- the width-scaled box
|
||||
is inflated by the aspect ratio on a wide image and the glyph never lands in
|
||||
it. Re-running the previously-undetected set with a short-side basis recovered
|
||||
**56% of landscape** images (12% square, 4% portrait).
|
||||
Detector calibration showed that a width-scaled box is inflated by the aspect
|
||||
ratio on a wide image and can miss the glyph entirely. A short-side basis
|
||||
recovered the affected Doubao landscape cases.
|
||||
|
||||
But the same switch took JIMENG's labelled landscape positives from 13/13 to
|
||||
0/13: its wordmark tracks the WIDTH. Both marks are ByteDance and share a
|
||||
The same switch broke Jimeng landscape positives because its wordmark tracks
|
||||
the WIDTH. Both marks are ByteDance and share a
|
||||
corner, and they still scale differently -- so this is a per-mark measurement,
|
||||
not a house rule to generalize. Samsung keeps ``width`` because there is no
|
||||
corpus evidence either way (1 addition corpus-wide) and an unmeasured change
|
||||
is not an improvement.
|
||||
not a house rule to generalize. Samsung keeps ``width`` because it has not been
|
||||
calibrated for a different basis, and an unmeasured change is not an improvement.
|
||||
|
||||
China's GB 45438-2025 clause 5.2(e) mandates glyph height >= 5% of "the
|
||||
shortest side" for CN marks, which is why a short-side basis is the natural
|
||||
@@ -708,8 +691,7 @@ class TextMarkEngine:
|
||||
elif self.config.detect_frontend == "tophat" and self.detect(image).detected:
|
||||
# A mark found only by the CONTINUOUS front-end has no binary glyph blob to
|
||||
# bound, so the mask came back empty and removal was a silent no-op while
|
||||
# `identify` still reported the mark (corpus-measured 2026-07-20: 57 of 60
|
||||
# sampled still-detected Doubao marks were untouched, ~8% of its detections).
|
||||
# `identify` still reported the mark while removal left it untouched.
|
||||
# Use the DETECTOR'S OWN best-match box: the correlation already located the
|
||||
# mark at a position and scale, and thresholding the response was a strictly
|
||||
# worse proxy for that. An earlier fix thresholded the max-normalized uint8
|
||||
|
||||
@@ -128,9 +128,11 @@ def remove_visible(
|
||||
strip_metadata: bool = True,
|
||||
write_noop: bool = True,
|
||||
) -> tuple[NDArray[Any], list[str]]:
|
||||
"""Remove every detected known visible AI mark (Gemini sparkle, the registered
|
||||
vendor text marks including Tencent Yuanbao, and the Jimeng pill) via
|
||||
localize -> fill, returning ``(result_bgr, [labels removed])``.
|
||||
"""Remove every detected known visible AI mark through localize then fill.
|
||||
|
||||
The registry currently covers the Gemini sparkle; Doubao, Jimeng, Qwen, Kling,
|
||||
Yuanbao, Samsung, RunningHub, Baidu, and LibLibAI text marks; and the Jimeng
|
||||
pill. Returns ``(result_bgr, [labels removed])``.
|
||||
|
||||
``source`` is a file path OR a BGR ndarray. For a PATH, metadata provenance is read
|
||||
automatically (so ``sensitivity="auto"`` recovers a moved/faint mark whenever the
|
||||
@@ -141,8 +143,7 @@ def remove_visible(
|
||||
known was found (e.g. route to the diffusion ``all`` path or ``erase``).
|
||||
|
||||
``sensitivity`` (``auto``/``strict``) and ``backend``
|
||||
(``auto``/``cv2``/``migan``/``lama``) are the same knobs as the CLI. Pass
|
||||
corner if the guess is wrong).
|
||||
(``auto``/``cv2``/``migan``/``lama``) are the same controls as the CLI.
|
||||
|
||||
``strip_metadata`` (default True, matching the CLI ``visible --strip-metadata``)
|
||||
also strips AI provenance metadata (C2PA/EXIF/XMP/IPTC) from the written output via
|
||||
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 2.8 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 2.6 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 3.3 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 4.1 KiB |
@@ -16,23 +16,9 @@ Removal is the shared **localize -> fill** (:meth:`footprint_mask` ->
|
||||
synthetic silhouette from ``scripts/render_vendor_silhouettes.py``, never cut
|
||||
from an upload).
|
||||
|
||||
Measured on the vendor cohort (16 TC260 carriers whose producer USCC
|
||||
91110000802100433B names Baidu, harvested 2026-07-22 by
|
||||
``scripts/vendor_cohort_harvest.py``), NOT inherited from Doubao:
|
||||
|
||||
* The 百度 text run is 0.090 of the SHORT side wide (measured on 720/768/
|
||||
1024-px frames), with its right edge ~0.099 of short off the right edge
|
||||
(the pill tag sits between the text and the corner), bottom margin
|
||||
~0.006; the locate box below covers the whole mark (text + tag).
|
||||
* Gate 0.43 (tophat front-end): on 278 hand-labelled clean frames
|
||||
(cohort-contamination-guarded) the max is 0.352 / p99 0.314, and the
|
||||
visibly-marked cohort frames score 0.386-0.65. Picked over the clean-arm
|
||||
0.37 after a full-corpus check on the 741-frame blind-labelled eval set
|
||||
surfaced 13 cross-fires at 0.38-0.43 (12 Qwen marks + one 抖音 mark) --
|
||||
see DETECT_NCC_THRESHOLD below. At 0.43 the cohort keeps 7 detections
|
||||
(0.61-0.65) and the whole 741 set fires only on the true Baidu frame.
|
||||
* STRICT ONLY (``provenance_ncc_factor`` 1.0): the cohort is small (16) and
|
||||
the sub-gate band is unmeasured, so no provenance relaxation exists.
|
||||
The detector uses a synthetic silhouette, short-side geometry, a strict
|
||||
confidence gate, and a Qwen rival margin. The footprint covers both the text
|
||||
run and its adjacent pill tag.
|
||||
"""
|
||||
# The module-level _alpha_template / _glyph_silhouette / _template_match_score below
|
||||
# are thin test-facing shims (imported by tests/), so pyright's src-only pass sees them
|
||||
@@ -67,20 +53,9 @@ LOGO_MIN_LUMA = 150
|
||||
TOPHAT_DELTA = 12
|
||||
|
||||
DETECT_MIN_COVERAGE = 0.04 # unused by the tophat front-end (kept for config parity)
|
||||
# Calibrated 2026-07-22 on the vendor cohort vs 278 hand-labelled clean frames
|
||||
# (clean p99 0.314 / max 0.352), THEN raised 0.37 -> 0.43 after a full-corpus
|
||||
# check: on the 741-frame blind-labelled eval set the 0.37 gate fired 14 times
|
||||
# outside the cohort, and only ONE was the vendor -- 12 were 千问AI生成 (Qwen)
|
||||
# marks (the 百/千 first glyphs are near-identical after binarization) and one
|
||||
# was a 抖音 AI创作 mark at 0.425. The Qwen fires are handled by the rival
|
||||
# margin (Qwen scores 0.58-0.76 there, beating Baidu by 0.17-0.35), but the
|
||||
# 抖音 one named no registered rival, so the gate moved above it. Cost: the
|
||||
# cohort's low trio at 0.386 (3 genuine marks) -- precision over recall on a
|
||||
# small cohort. Raised again 0.43 -> 0.48 after the full-corpus sweep
|
||||
# (2026-07-22): outside-cohort true Baidu carriers score 0.50-0.66 while the
|
||||
# false fires (大众点评 UI, a math blackboard, an 80s banner, a checkerboard)
|
||||
# top out at 0.47. Remaining cohort detections: 7 at 0.61-0.65, plus the 6
|
||||
# metadata-stripped true carriers the cohort cannot see.
|
||||
# Calibrated against vendor, rival-mark, and clean compatibility examples.
|
||||
# The Qwen rival margin handles visually similar marks; the threshold rejects
|
||||
# remaining unrelated bottom-right text.
|
||||
DETECT_NCC_THRESHOLD = 0.48
|
||||
|
||||
# Detection-silhouette geometry (fraction of the short side): the 百度 text run
|
||||
|
||||
@@ -193,9 +193,9 @@ _auto_option = click.option(
|
||||
"--auto",
|
||||
is_flag=True,
|
||||
default=False,
|
||||
help="DEPRECATED: controlnet is already the default pipeline, so --auto now only "
|
||||
"enables --adaptive-polish (the content detectors were removed). Use "
|
||||
"--adaptive-polish instead.",
|
||||
help="DEPRECATED: controlnet and adaptive polish are already the defaults, so "
|
||||
"--auto only emits a warning and changes nothing. Use --no-adaptive-polish "
|
||||
"to disable polishing.",
|
||||
)
|
||||
|
||||
_adaptive_polish_option = click.option(
|
||||
@@ -210,8 +210,8 @@ _adaptive_polish_option = click.option(
|
||||
|
||||
|
||||
# Tiled-diffusion knobs, shared by the diffusion commands (invisible/all/batch).
|
||||
# Tiling is the lossless alternative to --max-resolution for large inputs that OOM
|
||||
# on MPS/GPU: process at native resolution in overlapping, feather-blended tiles.
|
||||
# Tiling avoids an explicit resolution cap for large inputs that OOM on MPS/GPU:
|
||||
# it regenerates overlapping tiles at the input's native dimensions.
|
||||
def _tile_options(f: Any) -> Any:
|
||||
"""Apply the --tile / --tile-size / --tile-overlap options to a command."""
|
||||
f = click.option(
|
||||
@@ -229,9 +229,9 @@ def _tile_options(f: Any) -> Any:
|
||||
return click.option(
|
||||
"--tile/--no-tile",
|
||||
default=False,
|
||||
help="Process large images in overlapping tiles instead of one forward pass -- the lossless "
|
||||
"alternative to --max-resolution for inputs that OOM on MPS/GPU. Engages only when the long "
|
||||
"side exceeds --tile-size; pair with --max-resolution 0 (default) to keep native resolution. Default off.",
|
||||
help="Process large images in overlapping tiles instead of one forward pass. This keeps "
|
||||
"the input's native dimensions instead of applying --max-resolution, but still regenerates "
|
||||
"every tile. Engages only when the long side exceeds --tile-size. Default off.",
|
||||
)(f)
|
||||
|
||||
|
||||
@@ -417,8 +417,8 @@ def _remove_visible_auto(
|
||||
"""Remove every auto-detected visible mark via the registry (localize -> fill).
|
||||
|
||||
Routes the ``all``/``batch`` visible step through the same registry path the
|
||||
standalone ``visible`` command uses, so EVERY registered mark is handled (the
|
||||
Gemini sparkle and all registered vendor text marks), not just the sparkle.
|
||||
standalone ``visible`` command uses, so every registered mark is handled rather
|
||||
than only the Gemini sparkle.
|
||||
Returns ``(result, label-or-None)``; when no ``in_auto`` mark fires the image is
|
||||
returned unchanged with ``None``. ``backend`` selects the shared fill; ``sensitivity``
|
||||
controls how hard a borderline mark is trusted (auto reads metadata provenance)."""
|
||||
@@ -474,10 +474,9 @@ def _write_output_or_exit(output: Path, bgr: NDArray[Any], alpha: NDArray[Any] |
|
||||
def _no_visible_mark_exit(source: Path) -> NoReturn:
|
||||
"""Explain why no visible watermark was removed, then exit non-zero.
|
||||
|
||||
The visible registry handles only known visual marks (the Gemini sparkle and
|
||||
the registered vendor text marks). Most real uploads carry no such mark
|
||||
-- frequently an invisible/metadata watermark instead (e.g. an OpenAI or
|
||||
Gemini image whose only signal is C2PA + SynthID). Returning the input
|
||||
The visible registry handles only known visual marks. Most images carry no
|
||||
registered mark and may instead have an invisible or metadata watermark.
|
||||
Returning the input
|
||||
unchanged with exit 0 reads as success to a caller and re-serves the
|
||||
watermarked image -- the recurring "it didn't work" report. Instead, run a
|
||||
cheap metadata-only :func:`identify`, tell the user what the image actually
|
||||
@@ -533,9 +532,9 @@ def _no_visible_mark_exit(source: Path) -> NoReturn:
|
||||
|
||||
|
||||
# Same value as EXIT_NO_VISIBLE_MARK (2): a distinct-from-success / distinct-from-
|
||||
# error code that tells a wrapping service (raiw.cc) "the diffusion scrub was skipped
|
||||
# because no invisible watermark was locally detectable", so it can surface the
|
||||
# message instead of charging for and serving an unchanged image as done.
|
||||
# error code that tells a wrapping service "the diffusion scrub was skipped because
|
||||
# no invisible watermark was locally detectable", so it can surface the message
|
||||
# instead of treating an unchanged image as a completed removal.
|
||||
EXIT_NO_INVISIBLE_SIGNAL = 2
|
||||
|
||||
|
||||
@@ -639,7 +638,7 @@ def _run_visible_auto(
|
||||
console.print(f" Input: {source.name} ({w}x{h})")
|
||||
if not removed:
|
||||
# write_noop=False means nothing was written, so a pre-existing output is intact.
|
||||
console.print(" No registered visible mark detected.")
|
||||
console.print(f" No known visible mark detected. Checked: {', '.join(watermark_registry.mark_keys())}.")
|
||||
_no_visible_mark_exit(source)
|
||||
console.print(f" Removed: {', '.join(removed)}")
|
||||
size_kb = output.stat().st_size / 1024
|
||||
@@ -737,11 +736,11 @@ def cmd_visible(
|
||||
) -> None:
|
||||
"""Remove a known visible AI watermark from an image.
|
||||
|
||||
Finds a known mark in its usual place (Gemini sparkle / Doubao-Jimeng-Qwen-Samsung
|
||||
text) via the watermark registry and removes it by LOCALIZING the mark to a mask
|
||||
and filling that mask with the chosen ``--backend`` (auto: best available, LaMa >
|
||||
MI-GAN > cv2). ``--mark auto`` removes every detected mark in one
|
||||
pass. For arbitrary logos/objects, use ``erase``.
|
||||
Finds registered marks in their expected areas and removes them by localizing
|
||||
each mark to a mask, then filling that mask with the selected ``--backend``.
|
||||
``--mark auto`` removes every detected registry entry in one pass. Run
|
||||
``--help`` to see the current mark keys. For arbitrary logos and objects, use
|
||||
``erase``.
|
||||
"""
|
||||
_banner()
|
||||
source = _validate_image(source)
|
||||
@@ -822,7 +821,7 @@ def cmd_erase(
|
||||
|
||||
Universal and position-agnostic: removes any logo / watermark / object inside
|
||||
the boxes you pass, regardless of color or location. Runs on CPU. Use this
|
||||
for marks the dedicated ``visible`` engines (Gemini, Doubao) do not cover.
|
||||
for marks the dedicated ``visible`` registry does not cover.
|
||||
"""
|
||||
from remove_ai_watermarks.region_eraser import erase
|
||||
|
||||
@@ -899,7 +898,7 @@ def cmd_erase(
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
help="Cap long side (px) before diffusion; 0 = native and preserves the most detail. Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@_controlnet_scale_option
|
||||
@_min_resolution_option
|
||||
@@ -1096,10 +1095,10 @@ def cmd_metadata(
|
||||
def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bool) -> None:
|
||||
"""Identify where an image was made and what watermarks it carries.
|
||||
|
||||
Aggregates C2PA Content Credentials, IPTC "Made with AI" tags, embedded
|
||||
generation parameters, the SynthID metadata proxy, and the visible Gemini
|
||||
sparkle into a single provenance verdict. Absence of signals is reported as
|
||||
"unknown", never as "clean" (stripped metadata leaves no local proof).
|
||||
Aggregates supported C2PA, IPTC, EXIF, XMP, generator, visible-mark, and
|
||||
optional invisible-watermark signals into one provenance verdict. Absence of
|
||||
signals is reported as "unknown", never as "clean" because stripped metadata
|
||||
leaves no local proof.
|
||||
"""
|
||||
from dataclasses import asdict
|
||||
|
||||
@@ -1189,7 +1188,7 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
help="Cap long side (px) before diffusion; 0 = native and preserves the most detail. Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@_controlnet_scale_option
|
||||
@_min_resolution_option
|
||||
@@ -1361,12 +1360,16 @@ def cmd_all(
|
||||
# ── Step 3: Metadata ──
|
||||
console.print("\n 3) AI metadata stripping")
|
||||
try:
|
||||
from remove_ai_watermarks.metadata import remove_ai_metadata
|
||||
from remove_ai_watermarks.metadata import strip_and_verify
|
||||
|
||||
remove_ai_metadata(tmp_path, tmp_path)
|
||||
console.print(" AI metadata stripped")
|
||||
_, leftover = strip_and_verify(tmp_path, tmp_path)
|
||||
except Exception as e:
|
||||
console.print(f" Warning: Metadata strip failed: {e}")
|
||||
console.print(f" Error: metadata strip failed: {e}")
|
||||
raise SystemExit(1) from e
|
||||
if leftover:
|
||||
console.print(f" Error: metadata stripping was incomplete; {', '.join(sorted(leftover))} survived")
|
||||
raise SystemExit(1)
|
||||
console.print(" AI metadata stripped")
|
||||
|
||||
# ── Write final result ──
|
||||
# The invisible step (and downstream cv2.IMREAD_COLOR paths) drops alpha,
|
||||
@@ -1629,7 +1632,7 @@ def _process_batch_image(
|
||||
"--max-resolution",
|
||||
type=int,
|
||||
default=0,
|
||||
help="Cap long side (px) before diffusion; 0 = native (best quality, like raiw.cc). Raise only on GPU/MPS OOM.",
|
||||
help="Cap long side (px) before diffusion; 0 = native and preserves the most detail. Raise only on GPU/MPS OOM.",
|
||||
)
|
||||
@_min_resolution_option
|
||||
@_unsharp_option
|
||||
|
||||
@@ -6,8 +6,8 @@ label mandated by China's TC260 standard, a near-white semi-transparent overlay.
|
||||
|
||||
Detection matches the bundled glyph silhouette against the corner candidate; removal
|
||||
is the shared **localize -> fill** (the glyph-bbox :meth:`footprint_mask` feeds
|
||||
``region_eraser``), NOT reverse-alpha. This is one of the registered text-mark engines that
|
||||
share :class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine`; this module
|
||||
``region_eraser``), NOT reverse-alpha. This module shares
|
||||
:class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine` and
|
||||
supplies only Doubao's tuned :class:`TextMarkConfig` (bottom-right corner,
|
||||
``assets/doubao_alpha.png`` -- the detection silhouette, rebuilt by
|
||||
``scripts/visible_alpha_solve.py``). Arbitrary-region inpainting still lives in
|
||||
|
||||
@@ -150,9 +150,9 @@ class GeminiEngine:
|
||||
# or flat content (text, banners, hatching) can score >0.5 without that lift.
|
||||
# Demote a detection that is BOTH low-confidence AND low core-ring brightness
|
||||
# margin -- the joint signature of a content false positive (verified on the
|
||||
# spaces corpus: of 16 demoted, 13 carried no AI metadata and the 3 AI-meta ones
|
||||
# were visually FPs / a near-invisible white-on-white sparkle whose AI verdict is
|
||||
# held by metadata anyway). Real sparkles escape via EITHER high confidence
|
||||
# detector calibration: demoted examples were visual false positives or a
|
||||
# near-invisible white-on-white sparkle whose AI verdict is held by metadata
|
||||
# anyway). Real sparkles escape via EITHER high confidence
|
||||
# (white-bg sparkles score >=0.79 despite a low margin) OR high margin (dark/mid
|
||||
# backgrounds, incl. the #36 faint-corner case, lift well clear), so both must
|
||||
# fail to demote.
|
||||
@@ -184,8 +184,8 @@ class GeminiEngine:
|
||||
# confidence (_SPARKLE_KEEP_CONF -- the registry's 0.5 sparkle gate plus a small
|
||||
# margin so the ~0.51 bright-background FPs the grad gate was added for stay demoted)
|
||||
# AND has a bright (margin) near-neutral core (_core_saturation <= _SPARKLE_WHITE_SAT).
|
||||
# Corpus-measured on metadata-stripped faint sparkles: recovers ~14/20 the low-grad
|
||||
# demotion would drop, at ~0.8% clean false-fire vs the ~0.55% baseline.
|
||||
# Calibrated on metadata-stripped faint sparkles to recover low-gradient marks
|
||||
# without materially increasing clean false fires.
|
||||
_SPARKLE_KEEP_CONF = 0.52
|
||||
_SPARKLE_WHITE_SAT = 0.20
|
||||
|
||||
|
||||
@@ -89,7 +89,7 @@ def unsharp_mask(image: NDArray, amount: float = 0.5, sigma: float = 1.0) -> NDA
|
||||
|
||||
# ── Adaptive polish (target the input's detail level; spare text) ──────────────
|
||||
# A capped unsharp scaled to the sharpness deficit, then edge-masked grain to close
|
||||
# the rest -- tunable constants. Validated 2026-06-03 on the spaces corpus: a soft
|
||||
# the rest -- tunable constants. Compatibility testing showed that a soft
|
||||
# gemini_3 face/photo (lap-var 84 vs the 592 of its original) is pulled up to ~327
|
||||
# with full polish, while a sharp openai_1 text card (1175 vs 1644) gets near-zero
|
||||
# (the deficit is tiny) so text is left alone -- the polish self-limits on text.
|
||||
|
||||
@@ -8,8 +8,8 @@ Aggregates every locally-readable signal into a single :class:`ProvenanceReport`
|
||||
- **PNG text / EXIF generation parameters** (Stable Diffusion, ComfyUI, InvokeAI).
|
||||
- **SynthID metadata proxy** -- a C2PA companion from a SynthID-using vendor
|
||||
(Google / OpenAI) implies the invisible pixel watermark.
|
||||
- **Visible marks** (optional; needs cv2/numpy, no GPU): the Gemini sparkle and
|
||||
the ByteDance Doubao 豆包AI生成 / Jimeng 即梦AI text marks.
|
||||
- **Registered visible marks** (optional; needs cv2/numpy, no GPU) through the
|
||||
shared watermark registry.
|
||||
|
||||
Hard limit: a stripped image (re-encoded, screenshotted, social-media upload)
|
||||
loses all metadata, and the SynthID *pixel* watermark is not locally decodable
|
||||
@@ -65,9 +65,8 @@ _SCAN_BYTES = 1024 * 1024
|
||||
# Visible-sparkle confidence above which the signal is trusted as provenance.
|
||||
# Shared with the removal arbitration (watermark_registry.GEMINI_SPARKLE_TRUST_CONF)
|
||||
# so the provenance "is there a sparkle" verdict and the removal "take the sparkle"
|
||||
# decision can never drift apart -- the detect-vs-remove desync the retained-corpus
|
||||
# mining surfaced (2026-06-20). On the corpus Gemini-family sparkles score >= 0.56
|
||||
# while non-sparkle images top out at 0.49, so 0.5 cleanly separates them and avoids
|
||||
# decision can never drift apart. Calibration showed that 0.5 separates Gemini-family
|
||||
# sparkles from non-sparkle images and avoids
|
||||
# false positives when the sparkle is the only signal (e.g. an OpenAI image scored
|
||||
# 0.37 -- below threshold, correctly dropped).
|
||||
_SPARKLE_THRESHOLD = GEMINI_SPARKLE_TRUST_CONF
|
||||
@@ -463,7 +462,7 @@ def _visible_text_marks(image_path: Path, *, image: NDArray[Any] | None = None)
|
||||
"""Detected visible text marks (registry ``MarkDetection`` list).
|
||||
|
||||
The Gemini sparkle keeps its own ``_visible_sparkle`` path (file-level
|
||||
confidence); these two text marks reuse the registry detectors, which apply
|
||||
confidence); the text marks reuse the registry detectors, which apply
|
||||
each engine's calibrated NCC threshold via ``MarkDetection.detected``.
|
||||
Optional: needs cv2/numpy; returns ``[]`` if the engines/assets are missing
|
||||
or the image can't be read. ``image`` is a pre-decoded BGR array shared
|
||||
@@ -555,9 +554,8 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
|
||||
Args:
|
||||
image_path: Path to the image (PNG, JPEG, WebP, or ISOBMFF container).
|
||||
check_visible: Also run the visible-mark detectors (cv2) -- the Gemini
|
||||
sparkle and vendor text marks from the registry. Set
|
||||
False for a pure-metadata, dependency-light scan.
|
||||
check_visible: Also run the registered visible-mark detectors through cv2.
|
||||
Set False for a pure-metadata, dependency-light scan.
|
||||
check_invisible: Also decode open invisible watermarks (SD/SDXL/FLUX) via
|
||||
the optional imwatermark library. No-op when it is not installed.
|
||||
|
||||
@@ -679,8 +677,8 @@ def identify(image_path: Path, *, check_visible: bool = True, check_invisible: b
|
||||
if platform is None:
|
||||
# Apple Photos Clean Up (Apple Intelligence object removal) marks
|
||||
# the edit with photoshop:Credit / IPTC "Apple Photos Clean Up"
|
||||
# next to compositeWithTrainedAlgorithmicMedia -- corpus-measured
|
||||
# 2026-07-23 (35 files); it was detected but never attributed.
|
||||
# next to compositeWithTrainedAlgorithmicMedia. It was detected but
|
||||
# previously never attributed.
|
||||
platform = (
|
||||
"Apple Photos (Clean Up AI edit)"
|
||||
if b"Apple Photos Clean Up" in head
|
||||
|
||||
@@ -226,10 +226,9 @@ class InvisibleEngine:
|
||||
case); a ``max_resolution`` downscale always uses Lanczos. Falls back
|
||||
to Lanczos if the extra is absent.
|
||||
tile: Process the diffusion pass in overlapping tiles instead of one
|
||||
forward pass -- the lossless alternative to ``max_resolution`` for
|
||||
large inputs that OOM on MPS/GPU. Engages only when the long side
|
||||
exceeds ``tile_size``. Pair with ``max_resolution=0`` (the default)
|
||||
so the input keeps its native resolution.
|
||||
forward pass. This retains the input's native dimensions instead
|
||||
of applying ``max_resolution``, but each tile is still regenerated.
|
||||
Engages only when the long side exceeds ``tile_size``.
|
||||
tile_size: Tile dimension in px (default 1024).
|
||||
tile_overlap: Overlap between adjacent tiles in px (default 128).
|
||||
|
||||
|
||||
@@ -6,13 +6,13 @@ class as the Doubao text strip.
|
||||
|
||||
Detection matches the bundled glyph silhouette against the corner; removal is the
|
||||
shared **localize -> fill** (the glyph-bbox :meth:`footprint_mask` feeds
|
||||
``region_eraser``), NOT reverse-alpha. This is one of the registered text-mark engines that
|
||||
share :class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine`; this module
|
||||
``region_eraser``), NOT reverse-alpha. This module shares
|
||||
:class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine` and
|
||||
supplies only Jimeng's tuned :class:`TextMarkConfig` (bottom-right corner,
|
||||
``assets/jimeng_alpha.png`` -- the detection silhouette, rebuilt by
|
||||
``scripts/visible_alpha_solve.py`` from the gray capture). Jimeng images are also caught
|
||||
by the China TC260 AIGC metadata label, so this is the visible-mark *removal* path, not
|
||||
a new ``identify`` signal.
|
||||
by the China TC260 AIGC metadata label. The visual detector also feeds
|
||||
``identify`` when metadata is absent.
|
||||
"""
|
||||
# The module-level _alpha_template / _glyph_silhouette / _template_match_score below
|
||||
# are thin test-facing shims (imported by tests/), so pyright's src-only pass sees them
|
||||
@@ -44,9 +44,8 @@ TOPHAT_DELTA = 12
|
||||
# (>=0.81) from the Doubao strip (0.21), so the two ByteDance marks do not cross-fire.
|
||||
#
|
||||
# That separation holds at the STRICT threshold ONLY. Relaxed under provenance it
|
||||
# collapses: at the old shared 0.7 factor (gate 0.315) the arm ran at 17% precision
|
||||
# and 33 of its 68 false additions were Doubao marks (corpus-measured 2026-07-18 --
|
||||
# full table at _DEFAULT_PROVENANCE_NCC_FACTOR). That was patched with a tighter 0.85
|
||||
# collapses: at the old shared 0.7 factor, many false additions were Doubao marks.
|
||||
# That was patched with a tighter 0.85
|
||||
# factor at the time; the competitive rival margin replaced it -- see below.
|
||||
DETECT_MIN_COVERAGE = 0.02
|
||||
DETECT_NCC_THRESHOLD = 0.45
|
||||
|
||||
@@ -11,19 +11,9 @@ This module supplies only LibLibAI's tuned :class:`TextMarkConfig`
|
||||
(``assets/liblib_alpha.png`` from ``scripts/render_vendor_silhouettes.py``,
|
||||
never cut from an upload).
|
||||
|
||||
Measured on the vendor cohort (15 TC260 carriers, harvested 2026-07-22 by
|
||||
``scripts/vendor_cohort_harvest.py``), NOT inherited from Doubao:
|
||||
|
||||
* The wordmark is ~0.10 of the frame WIDTH wide, centered horizontally, its
|
||||
baseline ~0.94-0.95 of the height; consistent across 768..2240-px frames.
|
||||
* The silhouette font is Arial, NOT the STHeiti the CJK marks use: the real
|
||||
wordmark is a grotesque, and measured across 7 candidate fonts Arial lifts
|
||||
the cohort positives from 0.31-0.47 to 0.42-0.73 while the full-corpus
|
||||
false arm (latin UI text) drops to max 0.398. A 200x200 icon false-fired
|
||||
at 0.444, so a per-mark size floor (``_MIN_SHORT_SIDE``) backs the gate.
|
||||
* Gate 0.42 (tophat front-end): false arm max 0.398, cohort 0.43-0.59.
|
||||
* STRICT ONLY (``provenance_ncc_factor`` 1.0): small cohort, the relaxed band
|
||||
is unmeasured.
|
||||
The detector uses an Arial-class synthetic silhouette, width-based geometry, a
|
||||
strict confidence gate, and a minimum image size. The footprint includes both
|
||||
the logo and wordmark.
|
||||
"""
|
||||
# The module-level _alpha_template / _glyph_silhouette / _template_match_score below
|
||||
# are thin test-facing shims (imported by tests/), so pyright's src-only pass sees them
|
||||
@@ -57,13 +47,8 @@ LOGO_MIN_LUMA = 150
|
||||
TOPHAT_DELTA = 12
|
||||
|
||||
DETECT_MIN_COVERAGE = 0.04 # unused by the tophat front-end (kept for config parity)
|
||||
# Calibrated 2026-07-22 on the vendor cohort vs 286 hand-labelled clean frames
|
||||
# (clean p99 0.315 / max 0.367) and re-measured after the font fix: the wordmark
|
||||
# is set in an Arial-class grotesque, and the Arial silhouette lifts the cohort
|
||||
# positives to 0.43-0.59 while the full-corpus false arm (latin UI text bands,
|
||||
# website screenshots) drops to max 0.398 -- generic latin text matches the
|
||||
# wrong font less, which is exactly where the discrimination comes from. Gate
|
||||
# 0.42 keeps all 8 marked cohort frames with a 0.022 margin over the false arm.
|
||||
# Calibrated against vendor and clean compatibility examples. The Arial-class
|
||||
# silhouette separates the wordmark from generic Latin UI text.
|
||||
DETECT_NCC_THRESHOLD = 0.42
|
||||
|
||||
# Detection-silhouette geometry (fraction of the frame width): the wordmark,
|
||||
@@ -120,11 +105,7 @@ def _template_match_score(box_mask: NDArray[Any], scale_base: int) -> float:
|
||||
class LibLibEngine(TextMarkEngine):
|
||||
"""Detect/localize the visible LibLibAI wordmark (bottom-center; localize -> fill)."""
|
||||
|
||||
# Per-mark size floor: the wordmark template is 0.10 of the frame width, so
|
||||
# below ~480px short side it degrades under ~48px -- the one full-corpus
|
||||
# false fire with the final Arial template was a 200x200 icon (0.444, above
|
||||
# the gate, on a 20px template; measured 2026-07-22). The smallest true
|
||||
# carrier in the cohort is 768px.
|
||||
# Per-mark size floor prevents small generic icons from matching the wordmark.
|
||||
_MIN_SHORT_SIDE = 480
|
||||
|
||||
def __init__(self) -> None:
|
||||
|
||||
@@ -153,8 +153,8 @@ _TC260_FIELDS: frozenset[str] = frozenset(
|
||||
"PropagateID",
|
||||
"ReservedCode1",
|
||||
"ReservedCode2",
|
||||
# Service-provider schema (Tencent Cloud's AIGC variant, mined from the
|
||||
# retained corpus 2026-07): the same ``{"AIGC":{...}}`` wrapper but keyed
|
||||
# Service-provider schema (Tencent Cloud's AIGC variant): the same
|
||||
# ``{"AIGC":{...}}`` wrapper but keyed
|
||||
# ``ServiceProvider`` / ``ServiceUser`` (+ generic ``Time`` / ``ContentId``,
|
||||
# not gated on), embedded in EXIF ``ImageDescription``.
|
||||
"ServiceProvider",
|
||||
@@ -1062,8 +1062,8 @@ _EXT_TO_PIL_FORMAT = {".jpg": "JPEG", ".jpeg": "JPEG", ".webp": "WEBP", ".png":
|
||||
def _sniff_image_format(head: bytes) -> str | None:
|
||||
"""Actual raster format from a file's leading magic bytes (>= 12 bytes), as a PIL
|
||||
format name ("JPEG"/"PNG"/"WEBP"), or None when unrecognized. The file EXTENSION is
|
||||
unreliable: ~2% of real uploads carry a mismatched one (a PNG served as ``.jpg`` is
|
||||
common). Choosing the save format by extension re-encodes a lossless PNG/WebP into a
|
||||
unreliable because an input may carry a mismatched one (for example, PNG content
|
||||
served as ``.jpg``). Choosing the save format by extension re-encodes a lossless PNG/WebP into a
|
||||
real JPEG, silently degrading the pixels -- so the strip routes on content instead.
|
||||
ISOBMFF/GIF are handled before this point or fall through to PNG; only the
|
||||
lossy-vs-lossless distinction that matters here is resolved."""
|
||||
@@ -1087,9 +1087,9 @@ def strip_and_verify(
|
||||
:func:`remove_ai_metadata` is deliberately fail-safe: a file PIL cannot decode is
|
||||
copied through UNCHANGED rather than crashing a caller, and the path it returns is
|
||||
indistinguishable from a real strip. Any caller that reports an outcome to a user
|
||||
therefore cannot tell a no-op from a success -- corpus-observed on real Samsung
|
||||
Galaxy S22 C2PA PNGs, where `metadata --remove` printed "stripped" and exited 0 while
|
||||
the output still read as AI (2026-07-19 parity audit).
|
||||
therefore cannot tell a no-op from a success. A Samsung C2PA compatibility case
|
||||
exposed this when `metadata --remove` reported success while the output still read
|
||||
as AI.
|
||||
|
||||
Returns ``(output_path, surviving_markers)``; an empty mapping means a real strip.
|
||||
"""
|
||||
@@ -1164,8 +1164,8 @@ def remove_ai_metadata(
|
||||
if source_path.suffix.lower() in _FFMPEG_STRIP_EXTS:
|
||||
return _strip_with_ffmpeg(source_path, output_path)
|
||||
|
||||
# Route on the ACTUAL content format, not the extension (which lies on ~2% of real
|
||||
# uploads -- a PNG served as .jpg, etc.). Trusting the extension would push a
|
||||
# Route on the actual content format, not the extension. PNG content may be served
|
||||
# as .jpg, and trusting the extension would push a
|
||||
# lossless PNG/WebP through the lossy JPEG re-encode below just because its name
|
||||
# ends .jpg, breaking the "work with originals" invariant.
|
||||
true_fmt = _sniff_image_format(head) # reuse the 12 bytes already read above
|
||||
|
||||
@@ -135,12 +135,11 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
),
|
||||
# Some Volcano Engine certs name the signer with the Chinese legal entity
|
||||
# "北京火山引擎科技有限公司" (Beijing Volcano Engine Technology Co., Ltd.) rather
|
||||
# than the latin "volcengine" -- the latin needle misses it entirely, so real
|
||||
# ByteDance output was un-attributed in production traffic. The issuer is the
|
||||
# than the latin "volcengine" -- the latin needle misses it entirely. The issuer is the
|
||||
# UTF-8 of the Chinese name (it appears UTF-8-encoded in the manifest-store
|
||||
# JSON and the raw caBX bytes alike); it normalizes to the same "ByteDance"
|
||||
# needle and platform as the volcengine row, so the two collapse together for
|
||||
# clash detection. Verified against the mined retained corpus, 2026-06-20.
|
||||
# clash detection. Verified against compatible signed samples.
|
||||
C2paAiVendor(
|
||||
"北京火山引擎科技有限公司".encode(),
|
||||
"ByteDance (Volcano Engine)",
|
||||
@@ -154,7 +153,7 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
# clean manifest issuer matches "BytePlus (ByteDance)" directly. The platform
|
||||
# string mirrors the volcengine row: both share the "ByteDance" needle, so the
|
||||
# earlier row's label wins anyway -- they normalize together for clash
|
||||
# detection. Verified on real signed files in production traffic, 2026-06-19.
|
||||
# detection. Verified on compatible signed samples.
|
||||
C2paAiVendor(b"Byteplus", "BytePlus (ByteDance)", "ByteDance (Doubao / Jimeng / Volcano Engine)", "ByteDance"),
|
||||
# Dreamina (ByteDance's international Jimeng brand) signs C2PA as "Bytedance
|
||||
# Pte. Ltd." with a "Dreamina/x.y" claim generator and, unlike the Volcano
|
||||
@@ -164,8 +163,8 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
# active manifest is often a plain c2pa-tool transcode). ``asserts_ai`` lets the
|
||||
# issuer alone flag AI without trainedAlgorithmicMedia; "Dreamina" is a
|
||||
# distinctive brand string, so it does not risk the incidental-mention problem
|
||||
# the common-word issuers have. Verified on real signed files in the retained
|
||||
# corpus, 2026-07. Normalizes to the same "ByteDance" needle/platform as the
|
||||
# the common-word issuers have. Verified on compatible signed samples.
|
||||
# Normalizes to the same "ByteDance" needle/platform as the
|
||||
# volcengine row (they collapse together for clash detection).
|
||||
C2paAiVendor(
|
||||
b"Dreamina",
|
||||
@@ -176,18 +175,17 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
),
|
||||
# Canva Magic Media signs AI-generated images as "Canva" with a generic
|
||||
# c2pa-rs claim generator + trainedAlgorithmicMedia; without this entry the
|
||||
# source read AI but no platform was attributed. Verified on real signed files
|
||||
# in production traffic, 2026-06-19. Canva does not use SynthID.
|
||||
# source read AI but no platform was attributed. Verified on compatible signed
|
||||
# samples. Canva does not use SynthID.
|
||||
C2paAiVendor(b"Canva", "Canva", "Canva (Magic Media)", "Canva"),
|
||||
# ElevenLabs is a pure generative-AI company (AI voice / audio, and image /
|
||||
# video via its API); it signs output as "Eleven Labs Inc.", so the C2PA
|
||||
# manifest alone marks AI generation. Verified against the mined retained
|
||||
# corpus, 2026-06-20. ElevenLabs does not use SynthID.
|
||||
# manifest alone marks AI generation. Verified on compatible signed samples.
|
||||
# ElevenLabs does not use SynthID.
|
||||
C2paAiVendor(b"Eleven Labs", "ElevenLabs", "ElevenLabs", "ElevenLabs"),
|
||||
# fal.ai (generative inference platform, issuer "fal - Features & Labels
|
||||
# Inc." / common name "fal.ai", claim generators like "fal-ai/seedvr",
|
||||
# "fal-ai/gpt-image-2"). Corpus-measured 2026-07-23 on the retained
|
||||
# uploads (17 files): the files carry trainedAlgorithmicMedia, so the
|
||||
# "fal-ai/gpt-image-2"). The files carry trainedAlgorithmicMedia, so the
|
||||
# verdict already fired, but the platform stayed unattributed. fal.ai is
|
||||
# a pure generative platform, so ``asserts_ai`` also covers its output
|
||||
# that omits the source-type.
|
||||
@@ -195,7 +193,7 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
# Bria AI (bria.ai, generative platform) signs as "Bria Artificial
|
||||
# Intelligence" with a "Bria Ai" claim generator and source type
|
||||
# ``empty`` (NOT trainedAlgorithmicMedia), so a real signed file was
|
||||
# completely missed by identify -- corpus-found 2026-07-23. A pure-AI
|
||||
# completely missed by identify. A pure-AI
|
||||
# vendor with distinctive strings, so ``asserts_ai`` is safe here.
|
||||
C2paAiVendor(b"Bria", "Bria Artificial Intelligence", "Bria AI", "Bria", asserts_ai=True),
|
||||
# Truepic is a C2PA signing authority, not an AI generator: no platform label,
|
||||
@@ -203,8 +201,7 @@ C2PA_AI_VENDORS: tuple[C2paAiVendor, ...] = (
|
||||
C2paAiVendor(b"Truepic", "Truepic", None, None),
|
||||
)
|
||||
|
||||
# Deliberately NOT registered as AI-generation vendors (mined-corpus candidates
|
||||
# evaluated 2026-06-20):
|
||||
# Deliberately NOT registered as AI-generation vendors:
|
||||
# - TikTok Inc.: signs C2PA as a content-provenance / AI-labeling authority on
|
||||
# uploads, not as an image generator. The is_ai verdict keys off the
|
||||
# digitalSourceType (trainedAlgorithmicMedia), which is already honored; a
|
||||
@@ -234,12 +231,11 @@ C2PA_IDENTITY_AI_ORGS: frozenset[str] = frozenset(v.org for v in C2PA_AI_VENDORS
|
||||
# images", updated 2026-05-21): "Images generated with ChatGPT, Codex, and
|
||||
# our API include both C2PA metadata and SynthID watermarks." OpenAI also
|
||||
# notes a signal may be absent if "the image was created before these
|
||||
# signals were available" -- so OpenAI images from BEFORE the rollout carry
|
||||
# C2PA WITHOUT SynthID (e.g. data/samples/openai-images-2/amur-leopard.png,
|
||||
# C2PA timestamp 2026-04-22). For OpenAI the proxy is therefore "likely",
|
||||
# signals were available" -- so OpenAI images from before the rollout can
|
||||
# carry C2PA without SynthID. For OpenAI the proxy is therefore "likely",
|
||||
# not certain; the verdict string is hedged accordingly. OpenAI's own oracle
|
||||
# is openai.com/verify (Google's is the Gemini app "Verify with SynthID").
|
||||
# The issuer byte ("OpenAI"/"Google") is verified locally against data/samples;
|
||||
# The issuer byte ("OpenAI"/"Google") is verified locally against data/fixtures/provenance;
|
||||
# the SynthID pairing is documented behavior (Google: DeepMind; OpenAI: above).
|
||||
# Adobe Firefly and Microsoft Designer sign C2PA but do NOT use SynthID, so a
|
||||
# C2PA manifest alone is not a SynthID signal -- the issuer is. The pixel
|
||||
@@ -311,8 +307,7 @@ AI_GENERATOR_TOKENS: frozenset[str] = frozenset(
|
||||
"leonardo",
|
||||
"flux",
|
||||
"dreamstudio",
|
||||
# Mined from the retained corpus 2026-06-22 (no C2PA -- a plain EXIF/PNG
|
||||
# generator stamp was the only signal and we read none of them):
|
||||
# Generator stamps without C2PA:
|
||||
# - NovelAI (anime SD): PNG tEXt Software="NovelAI", Source="NovelAI
|
||||
# Diffusion V4.5 <hash>", Title="NovelAI generated image".
|
||||
# - Reve Image (reve.com): EXIF Software / XMP CreatorTool = "reve.com"
|
||||
@@ -321,11 +316,11 @@ AI_GENERATOR_TOKENS: frozenset[str] = frozenset(
|
||||
"novelai",
|
||||
"reve.com",
|
||||
"aphrodite ai",
|
||||
# Corpus-mined 2026-07-23:
|
||||
# Additional verified markers:
|
||||
# - Apple Photos Clean Up (Apple Intelligence object removal): XMP
|
||||
# photoshop:Credit / IPTC credit value; composite source-type
|
||||
# covered detection, this token covers removal parity (35 files).
|
||||
# - fal-ai: generative-platform generator string (17 files).
|
||||
# covered detection, this token covers removal parity.
|
||||
# - fal-ai: generative-platform generator string.
|
||||
"apple photos clean up",
|
||||
"fal-ai",
|
||||
}
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
The img2img / ControlNet pipeline denoises the WHOLE image in one forward pass,
|
||||
so it OOMs on MPS/GPU above ~2K (issue #10). Tiling splits the image into
|
||||
overlapping tiles -- each kept near SDXL's ~1024 training size -- regenerates
|
||||
each tile independently, and feather-blends the overlaps. The result is processed
|
||||
at NATIVE resolution with no seam: the lossless alternative to the
|
||||
``--max-resolution`` downscale (which trades quality for a smaller forward pass).
|
||||
each tile independently, and feather-blends the overlaps. The result retains the
|
||||
input's native dimensions without an explicit ``--max-resolution`` downscale, but
|
||||
it is not pixel-lossless because every tile is regenerated.
|
||||
|
||||
The geometry (``plan_tiles``) and the blend weighting (``feather_weights``) are
|
||||
pure functions, unit-tested without the diffusion model. ``run_tiled`` is the
|
||||
|
||||
@@ -75,7 +75,7 @@ CONTROLNET_CANNY_MODEL = "xinsir/controlnet-canny-sdxl-1.0"
|
||||
# ladder" below.
|
||||
#
|
||||
# Data basis (see docs/synthid.md sections 2.2 / 5.5): ORACLE-CERTIFIED controlnet floors.
|
||||
# A 2026-06-14 re-test on the deployed Modal worker (the production controlnet pipeline)
|
||||
# Oracle re-testing
|
||||
# LOWERED the ladder back to OpenAI 0.10 / Google 0.15: each output verified on its own
|
||||
# oracle (openai.com/verify for OpenAI, the Google Gemini app for Google), all clean ->
|
||||
# - OpenAI 0.10: 2 photoreal images (1402 / 1448 px), SynthID not found on either.
|
||||
@@ -98,9 +98,8 @@ CONTROLNET_CANNY_MODEL = "xinsir/controlnet-canny-sdxl-1.0"
|
||||
# case (flat fills) `sdxl` is the WEAKER remover -- plain img2img at low strength barely
|
||||
# perturbs a flat region -- so it needs AT LEAST as much strength as controlnet, not
|
||||
# less. Hence the certified controlnet floor is the right floor for `sdxl` too. The
|
||||
# higher strength costs little quality where it matters: `controlnet` is now the default
|
||||
# pipeline, so `sdxl` is reached only for structure-less inputs (via `--auto`) or an
|
||||
# explicit `--pipeline sdxl`, where over-regeneration has no faces/text to damage. NOTE:
|
||||
# higher strength costs little quality where it matters. `controlnet` is now the default
|
||||
# pipeline and `sdxl` is reached only through an explicit `--pipeline sdxl`. NOTE:
|
||||
# this is a MARGIN argument for `sdxl`, not a fresh certification -- there is no local
|
||||
# SynthID detector, so if an oracle still reads SynthID on a flat `sdxl` output, raise
|
||||
# `--strength`.
|
||||
|
||||
@@ -85,8 +85,8 @@ def is_watermark_removal_available() -> bool:
|
||||
|
||||
|
||||
# Drop-in fp16-safe replacement for the SDXL VAE. The stock SDXL VAE overflows
|
||||
# to NaN in fp16 and decodes to an all-black image (issue #29: the raiw.cc black
|
||||
# result on a CUDA fp16 backend). This community VAE is numerically rescaled to
|
||||
# to NaN in fp16 and decodes to an all-black image (issue #29, reproduced on a
|
||||
# CUDA fp16 backend). This community VAE is numerically rescaled to
|
||||
# stay in fp16 range. SDXL-architecture only.
|
||||
_SDXL_FP16_VAE_ID = "madebyollin/sdxl-vae-fp16-fix"
|
||||
|
||||
@@ -692,9 +692,10 @@ class WatermarkRemover:
|
||||
strips the metadata; the caller passes it down so display and execution
|
||||
agree.
|
||||
tile: Process the image in overlapping tiles instead of one forward pass.
|
||||
The lossless alternative to a ``--max-resolution`` downscale for large
|
||||
inputs that OOM on MPS/GPU (issue #10). Only engages when the long side
|
||||
exceeds ``tile_size``; smaller images run a single pass unchanged.
|
||||
This keeps the input's native dimensions instead of applying a
|
||||
``--max-resolution`` downscale, but every tile is still regenerated.
|
||||
Only engages when the long side exceeds ``tile_size``; smaller images
|
||||
run a single pass unchanged.
|
||||
tile_size: Tile dimension in px (default 1024).
|
||||
tile_overlap: Overlap between adjacent tiles in px (default 128), feather-
|
||||
blended so there is no visible seam.
|
||||
|
||||
@@ -7,14 +7,14 @@ silhouette; like every mark it is then removed by the shared localize -> fill:
|
||||
|
||||
* Detect: edge-NCC of a font-rendered SILHOUETTE (``assets/jimeng_pill.png``,
|
||||
synthetic, data-safe -- see ``scripts/render_pill_silhouette.py``) against the
|
||||
top-left ROI, at the pill's known width fraction. Corpus-calibrated threshold
|
||||
(61 real positives + jimeng negatives): ``_DETECT_THRESHOLD`` 0.22.
|
||||
top-left ROI, at the pill's known width fraction. The calibrated
|
||||
``_DETECT_THRESHOLD`` is 0.22.
|
||||
* Remove: place the pill footprint at the matched location and inpaint it
|
||||
(MI-GAN / cv2 via the registry). Quality comes from the inpaint backend, so the
|
||||
silhouette need not be pixel-accurate -- which is why a synthetic render is
|
||||
sufficient and no corpus-derived asset is committed.
|
||||
sufficient and no source-derived asset is committed.
|
||||
|
||||
Geometry measured on 51 real examples (8 resolutions, all 3:4): width ~0.161*W,
|
||||
Geometry uses width ~0.161*W,
|
||||
height ~0.091*W, top-left, margins ~0.02-0.05.
|
||||
"""
|
||||
|
||||
@@ -42,7 +42,7 @@ _ASSET = Path(__file__).parent / "assets" / "jimeng_pill.png"
|
||||
_WIDTH_FRAC = 0.161
|
||||
_ROI_W_FRAC = 0.34 # search window width (of W)
|
||||
_ROI_H_FRAC = 0.14 # search window height (of H)
|
||||
_DETECT_THRESHOLD = 0.22 # edge-NCC gate, corpus-calibrated
|
||||
_DETECT_THRESHOLD = 0.22 # calibrated edge-NCC gate
|
||||
# Inpaint mask GEOMETRY (fractions of W unless noted): a generous fixed top-left box
|
||||
# covering the pill (measured ~0.167*W wide, ~0.09*W tall, margin ~0.02-0.05) plus
|
||||
# margin. The mask uses stable geometry, NOT the NCC match position -- the synthetic
|
||||
@@ -59,9 +59,8 @@ _MASK_W, _MASK_H = 0.205, 0.115 # width of W, height of W
|
||||
# corners (sky / wall / solid) where inpaint is invisible. So the metadata-only arm
|
||||
# removes the pill only when the footprint background is flat enough for a safe,
|
||||
# invisible inpaint. Threshold = median Sobel magnitude over the footprint box at a
|
||||
# normalized width; corpus-validated on 32k real uploads 2026-07 (real pills median
|
||||
# ~3.2, textured-ceiling false fires median ~8+). The reliable bottom-right wordmark
|
||||
# arm is NOT texture-gated -- a wordmark-confirmed pill is removed regardless.
|
||||
# normalized width. The reliable bottom-right wordmark arm is NOT texture-gated:
|
||||
# a wordmark-confirmed pill is removed regardless.
|
||||
_FLAT_TEXTURE_MAX = 6.0
|
||||
|
||||
_silhouette: NDArray[Any] | None = None
|
||||
|
||||
@@ -4,7 +4,7 @@ Position- and content-agnostic. You supply the rectangle(s); the eraser inpaints
|
||||
whatever is inside, so it removes any visible logo / watermark / object regardless
|
||||
of color, style, or location. Localization is the user's responsibility (pass the
|
||||
box); restoration runs on CPU. This is the universal fallback for marks the
|
||||
deterministic per-generator engines (Gemini sparkle, Doubao) do not cover.
|
||||
registered visible detectors do not cover.
|
||||
|
||||
Backends:
|
||||
- ``cv2`` (default): ``cv2.inpaint`` (Telea / Navier-Stokes). Instant, no extra
|
||||
@@ -235,8 +235,8 @@ def erase_migan(image_bgr: NDArray[Any], mask: NDArray[Any]) -> NDArray[Any]:
|
||||
|
||||
Mask polarity: the shipped ``andraniksargsyan/migan`` ONNX expects 0 = hole
|
||||
(inpaint) / 255 = known (keep) -- the INVERSE of this package's 255-erase
|
||||
convention -- so the mask is inverted before feeding the model (corpus-validated
|
||||
2026-07; feeding 255=hole regenerates the whole frame into stripes).
|
||||
convention -- so the mask is inverted before feeding the model. Feeding 255=hole
|
||||
regenerates the whole frame into stripes.
|
||||
|
||||
Accepts 1-channel (grayscale) and 4-channel (BGRA) input.
|
||||
"""
|
||||
|
||||
@@ -135,15 +135,8 @@ class RunningHubEngine(TextMarkEngine):
|
||||
def __init__(self) -> None:
|
||||
super().__init__(_CONFIG)
|
||||
|
||||
# Anchor window for the match position, as a fraction of the FRAME: the true
|
||||
# mark hugs the corner (measured x 0.008-0.014, y 0.005-0.007 of the frame on
|
||||
# every cohort positive), while the full-corpus false fires (hair, shelves,
|
||||
# window frames, CJK banners -- 37 of 42009 outside-cohort frames at the 0.34
|
||||
# gate, 2026-07-22 sweep) match off-anchor at x 0.013-0.150 / y 0.009-0.045.
|
||||
# No NCC gate separates them (false max 0.384 vs two positives at 0.381), but
|
||||
# position does: every false fire sits outside this window, every positive
|
||||
# inside. Contrast-dependent raw-gray NCC keys on "some text-like structure
|
||||
# anywhere in the box"; the anchor is what makes it about THIS mark.
|
||||
# Anchor window for the match position, as a fraction of the frame. The true
|
||||
# mark hugs the corner, while common false matches sit farther from the anchor.
|
||||
_ANCHOR_MAX_X = 0.025
|
||||
_ANCHOR_MAX_Y = 0.015
|
||||
|
||||
|
||||
@@ -9,8 +9,8 @@ Doubao/Jimeng marks but bottom-left.
|
||||
|
||||
Detection matches the bundled glyph silhouette against the corner; removal is the
|
||||
shared **localize -> fill** (the glyph-bbox :meth:`footprint_mask` feeds
|
||||
``region_eraser``), NOT reverse-alpha. This is one of the registered text-mark engines that
|
||||
share :class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine`; this module
|
||||
``region_eraser``), NOT reverse-alpha. This module shares
|
||||
:class:`remove_ai_watermarks._text_mark_engine.TextMarkEngine` and
|
||||
supplies only Samsung's tuned :class:`TextMarkConfig` (bottom-LEFT corner, a lower glyph
|
||||
luma since the mark is faint, ``assets/samsung_alpha.png`` -- the detection silhouette,
|
||||
solved from the flat captures by ``scripts/visible_alpha_solve.py``). Samsung Galaxy AI
|
||||
|
||||
@@ -104,8 +104,8 @@ _PRODUCT_OF: dict[str, str] = {
|
||||
# trust, which bypasses the sibling's false-positive gate outright -- so a detector
|
||||
# that false-fires often must not be able to hand that bypass to anyone.
|
||||
#
|
||||
# The pill qualifies on its own documented numbers: ~7% raw false-fire (5.5% measured
|
||||
# on 578 vendor negatives, 2026-07-18). Letting it corroborate produced a closed loop
|
||||
# The pill detector has a meaningful raw false-fire rate. Letting it corroborate
|
||||
# produced a closed loop
|
||||
# on the DEFAULT auto path, no user flag involved:
|
||||
# pill false-fires on a clean non-ByteDance image
|
||||
# -> _PRODUCT_OF maps it to "jimeng", so jimeng resolves to `confirmed`
|
||||
@@ -113,10 +113,9 @@ _PRODUCT_OF: dict[str, str] = {
|
||||
# -> _keep_pill now sees "jimeng" in keys and takes the WORDMARK arm, which
|
||||
# removes the pill unrestricted -- skipping the flatness guard that exists
|
||||
# precisely to stop the fill smearing a textured corner.
|
||||
# Measured on the corpus: 3 of 578 negatives ran the full loop, one of them with
|
||||
# footprint_flat=0 (the exact case the guard was written to block). Cutting the pill
|
||||
# out of corroboration removed all 3 and cost NOTHING on 4417 TC260 carriers
|
||||
# (jimeng fires 398 -> 398), so this is a defect fix, not a recall trade.
|
||||
# Calibration reproduced the full loop, including a textured footprint. Cutting the
|
||||
# pill out of corroboration removed the loop without reducing Jimeng detections, so
|
||||
# this is a defect fix, not a recall trade.
|
||||
#
|
||||
# `_keep_pill` already encodes the same distrust for the pill's own ACTION; this
|
||||
# closes the gap that its TESTIMONY was never gated.
|
||||
@@ -293,11 +292,11 @@ class KnownMark:
|
||||
# confidence, shared by BOTH the removal arbitration here (`_gemini_detect`) and
|
||||
# the provenance detector in `identify` (which imports it as its sparkle threshold).
|
||||
# Defining it once removes the detect-vs-remove
|
||||
# threshold drift the retained-corpus mining surfaced (2026-06-20): identify
|
||||
# threshold drift found during compatibility testing: identify
|
||||
# would report a sparkle while removal declined it, or vice versa, whenever the
|
||||
# two independently-maintained 0.5 constants fell out of step. Now they cannot.
|
||||
#
|
||||
# Value 0.5 is corpus-validated: the gemini engine's own `detected` flag uses a
|
||||
# Value 0.5 is calibrated: the Gemini engine's own `detected` flag uses a
|
||||
# looser internal threshold (0.35) and weakly fires (~0.36-0.42) on unrelated
|
||||
# bottom-right text -- a real Doubao mark scores ~0.40-0.42 as a gemini match,
|
||||
# and its core-ring brightness margin is HIGHER than a genuine faint sparkle's,
|
||||
@@ -451,7 +450,8 @@ def resolve_backend(backend: Backend) -> Literal["cv2", "migan", "lama"]:
|
||||
def fill(image: NDArray[Any], mask: NDArray[Any], *, backend: Backend = "auto") -> NDArray[Any]:
|
||||
"""The ONE shared, mark-agnostic removal: erase ``mask`` (255 = remove) via the
|
||||
chosen inpaint backend. Delegates to :func:`region_eraser.erase`; ``auto``
|
||||
resolves to MI-GAN when installed else cv2 (see :func:`resolve_backend`)."""
|
||||
resolves in quality order, LaMa then MI-GAN then cv2 (see
|
||||
:func:`resolve_backend`)."""
|
||||
from remove_ai_watermarks import region_eraser
|
||||
|
||||
return region_eraser.erase(image, mask=mask, backend=resolve_backend(backend))
|
||||
@@ -479,8 +479,8 @@ def _gemini_mask(
|
||||
return _engine("gemini").footprint_mask(image, force=force, region=region)
|
||||
|
||||
|
||||
# The registered text-mark engines share the TextMarkEngine interface, so one
|
||||
# parameterized adapter pair drives all of them -- a new
|
||||
# The text-mark engines share the TextMarkEngine interface, so one parameterized
|
||||
# adapter pair drives all of them -- a new
|
||||
# text mark is one `_text_mark(...)` row below, not another copy-paste of these
|
||||
# bodies. Detection matches the glyph silhouette; the mask is the template-free
|
||||
# glyph-bbox footprint (see TextMarkEngine.footprint_mask).
|
||||
@@ -505,8 +505,7 @@ def _text_mark_mask(key: str) -> Callable[..., NDArray[Any] | None]:
|
||||
|
||||
|
||||
def _text_mark(key: str, label: str, location: str) -> KnownMark:
|
||||
"""A text-mark registry row (Doubao/Jimeng/Samsung): glyph-silhouette detect +
|
||||
template-free glyph-bbox mask."""
|
||||
"""Build a text-mark registry row from its shared detector and mask adapters."""
|
||||
return KnownMark(key, label, location, True, _text_mark_detect(key, label, location), _text_mark_mask(key))
|
||||
|
||||
|
||||
@@ -611,12 +610,11 @@ def _keep_pill(keys: set[str], *, provenance: frozenset[str], footprint_flat: bo
|
||||
"""Whether to auto-remove the capture-less 'AI生成' pill given the fired marks.
|
||||
|
||||
Pure decision (the flatness feature is precomputed at perception time and passed
|
||||
in). The pill detector is weak (~7% raw false-fire) and metadata/intent confirms
|
||||
the platform, not pill presence, so a naive confirmation-OR gate over-fires: on a
|
||||
32k real-upload corpus (2026-07) the metadata-only arm was only ~27% precise and
|
||||
its false fires were textured ceilings/walls that the fill visibly SMEARS. Arms:
|
||||
in). The pill detector is weak and metadata/intent confirms the platform, not pill
|
||||
presence, so a naive confirmation-OR gate over-fires on textured ceilings and walls
|
||||
that the fill visibly smears. Arms:
|
||||
* bottom-right "★ 即梦AI" wordmark fired -> ~94% precise, and it survives
|
||||
metadata-STRIPPED uploads: remove the pill unrestricted;
|
||||
metadata-STRIPPED images: remove the pill unrestricted;
|
||||
* TC260 metadata confirms Jimeng (``"jimeng" in provenance``, no wordmark) -> remove ONLY when the
|
||||
top-left footprint is flat enough for an invisible fill (``footprint_flat``),
|
||||
so real flat-scene pills (and harmless flat false fires) are cleaned while the
|
||||
|
||||
Reference in New Issue
Block a user