mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-07 06:28:36 +02:00
Mask faint text marks with the detector's match box, not a response threshold
The faint-mask fallback added for the tophat front-end thresholded the max-normalized uint8 response at 0.5 -- which selects every non-zero pixel, not "half the peak" as its comment claimed -- and filled ~120% of the corner box on textured frames. Measured on 14 real faint-path frames (cv2 fill, detector re-run after): the detector's own best-match box fills a 58.7%-median corner box vs 120.9% for the threshold, both 100% detector-clean. Detection and the mask now read one method, _tophat_best, whose score gates detection and whose argmax box bounds the fill, so the two cannot drift by construction -- which is how the mismatch arose. The 0.5 constant is deleted. Parity could not catch this (a mask that fills everything is trivially detector-clean) and the regression test could not either: its flat fixture gives every threshold the same box, so mutating the constant to 99.0 stayed green. The fixture now carries texture and asserts the mask area is bounded, not merely non-empty; it reproduces the corpus number (127% pre-fix). Also lands the Tier B2 verification harnesses that found and bounded this: - detector_response.py: response curves (detected AND maskable per cell); found the size response is a comb, contrast is near-irrelevant, no unmaskable cells. - ladder_headroom.py: measured that a denser scale ladder recovers 7.6% of misses for a 2.52%->3.05% false-fire rise, and the one landscape rung that helps is a geometry shift that helps and hurts equally (1.7:1) -- do not add. - cjk_tail_probe.py: a generic shared-tail (AI生成) template does not separate uncovered vendors from clean corners (0.407 vs clean p99 0.298). Records the visible-parity re-run confirming the earlier front-end fix (doubao 91.8% -> 99.3%), and dedups the thrice-written stamp forward model into one fill_quality.composite. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
co-authored by
Claude Opus 4.8
Claude Opus 4.8
parent
7d00debdca
commit
52eb40c2ca
@@ -0,0 +1,272 @@
|
||||
"""Can one GENERIC template cover the CJK AI labels no per-vendor detector fires on?
|
||||
|
||||
THE OPPORTUNITY
|
||||
Corpus inspection of doubao-provenance misses turned up `千问AI生成` (Alibaba Qwen) and
|
||||
`百度 AI生成` (Baidu) sitting in the same bottom-right corner as the marks we do cover,
|
||||
bold and plainly legible, with no detector able to fire on either. `docs/...landscape`
|
||||
puts uncovered vendors at ~6% of sampled images -- larger than any tuning gain left in
|
||||
the covered ones (a dense scale ladder recovers 1 miss in 26; see ladder_headroom.py).
|
||||
|
||||
WHY A SHARED TEMPLATE IS EVEN PLAUSIBLE
|
||||
China's GB 45438-2025 mandates the label, and every compliant vendor ends it with the
|
||||
same run: `AI生成`. The vendor PREFIX varies (豆包 / 千问 / 百度 / 星绘) and is what makes
|
||||
per-vendor silhouettes expensive; the TAIL is guaranteed. So the tail is the part worth
|
||||
detecting, and vendor attribution -- which the shared tail cannot provide anyway
|
||||
(measured 千问-vs-doubao AUC ~0.5) -- moves to metadata, where it is reliable.
|
||||
|
||||
WHY IT IS WORTH RETRYING NOW
|
||||
`render_vendor_silhouettes.py` records 千问 as ruled out, and its reasoning was sound:
|
||||
the then-current front-end binarized the glyph, and a thin translucent overlay shatters
|
||||
into specks that no template can match. It closes by naming what would be needed --
|
||||
"a detection front-end that does not depend on binarizing the glyph (grayscale/edge
|
||||
correlation on the raw top-hat)". That front-end now EXISTS: `detect_frontend="tophat"`
|
||||
was built for doubao and correlates a soft template against the continuous response.
|
||||
The blocker was removed by unrelated work, so the ruling deserves re-measurement rather
|
||||
than inheritance.
|
||||
|
||||
THE MEASUREMENT
|
||||
Three arms, one scorer (the tail template on the tophat response, swept over a dense
|
||||
size ladder because an unregistered vendor's glyph size is genuinely unknown -- the one
|
||||
place a dense ladder earns its cost):
|
||||
|
||||
covered a registered mark already fires -- a sanity arm; the tail is inside those
|
||||
marks too, so a tail template that cannot score THESE is simply broken
|
||||
uncovered TC260 provenance but no detector fires -- the target population
|
||||
clean no metadata signal at all -- the precision arm
|
||||
|
||||
The decision rests on whether `uncovered` carries a high-scoring subpopulation that
|
||||
`clean` does not. It does NOT rest on the raw rate: TC260 provenance does not imply a
|
||||
visible mark, so a modest fire rate on that arm is expected even if the detector is
|
||||
perfect. Any threshold this suggests must then be confirmed by eyeballing the crops it
|
||||
selects -- the script writes a contact sheet for exactly that.
|
||||
|
||||
DATA SAFETY
|
||||
Corpus images are user uploads: read-only, local analysis, gitignored output. The
|
||||
template is font-rendered synthetic, never cut from a user upload.
|
||||
|
||||
uv run python scripts/cjk_tail_probe.py --n 6000
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
CORPUS = REPO / "data" / "spaces" / "originals"
|
||||
OUT = REPO / "data" / "spaces" / "_cjk_tail_probe.jsonl"
|
||||
# Cached under the gitignored data dir, not in scripts/: this is a probe artifact, not a
|
||||
# product asset. If the tail mark is ever registered, `render_vendor_silhouettes.py` is
|
||||
# what writes the committed silhouette into src/.../assets/.
|
||||
TAIL_PNG = REPO / "data" / "spaces" / "_cjk_tail_silhouette.png"
|
||||
|
||||
# The tail is a fraction of a full vendor mark's width (`豆包AI生成` is ~5 CJK widths,
|
||||
# `AI生成` ~3), and the prefix length differs per vendor, so the size is genuinely
|
||||
# unknown here -- unlike a registered mark, whose fraction is calibrated. Hence a wide,
|
||||
# dense ladder: this is the case a dense ladder is actually for.
|
||||
TAIL_SCALES = tuple(round(0.30 * (1.08**i), 4) for i in range(18)) # ~0.30 .. ~1.15
|
||||
|
||||
|
||||
def build_tail_silhouette() -> np.ndarray:
|
||||
"""Font-rendered `AI生成` silhouette, cached next to this script (data-safe)."""
|
||||
if TAIL_PNG.exists():
|
||||
img = cv2.imread(str(TAIL_PNG), cv2.IMREAD_GRAYSCALE)
|
||||
if img is not None:
|
||||
return img
|
||||
from render_vendor_silhouettes import render
|
||||
|
||||
sil = render("AI生成", width=200)
|
||||
cv2.imwrite(str(TAIL_PNG), sil)
|
||||
return sil
|
||||
|
||||
|
||||
def tail_score(engine: Any, image: np.ndarray, sil: np.ndarray) -> tuple[float, float]:
|
||||
"""Best tail correlation over the size ladder; returns (score, winning_scale)."""
|
||||
loc = engine.locate(image)
|
||||
resp = engine.tophat_response(image, loc)
|
||||
if resp is None:
|
||||
return (0.0, 0.0)
|
||||
c = engine.config
|
||||
base = engine.scale_base(image)
|
||||
ar = sil.shape[0] / max(1, sil.shape[1])
|
||||
best, best_s = 0.0, 0.0
|
||||
for s in TAIL_SCALES:
|
||||
gw = max(12, int(c.alpha_width_frac * base * s))
|
||||
gh = max(6, int(gw * ar))
|
||||
if gw >= resp.shape[1] or gh >= resp.shape[0]:
|
||||
continue
|
||||
t = cv2.resize(sil, (gw, gh), interpolation=cv2.INTER_AREA)
|
||||
v = float(cv2.matchTemplate(resp, t, cv2.TM_CCOEFF_NORMED).max())
|
||||
if v > best:
|
||||
best, best_s = v, s
|
||||
return (best, best_s)
|
||||
|
||||
|
||||
def _one(path_str: str) -> dict[str, Any] | None:
|
||||
from remove_ai_watermarks.api import visible_provenance
|
||||
from remove_ai_watermarks.doubao_engine import DoubaoEngine
|
||||
from remove_ai_watermarks.identify import identify
|
||||
from remove_ai_watermarks.image_io import imread
|
||||
from remove_ai_watermarks.watermark_registry import detect_marks
|
||||
|
||||
path = Path(path_str)
|
||||
try:
|
||||
prov = visible_provenance(path)
|
||||
except Exception:
|
||||
return None
|
||||
tc260 = bool({"doubao", "jimeng"} & prov)
|
||||
|
||||
if not tc260:
|
||||
try:
|
||||
if identify(path, check_visible=False).signals:
|
||||
return None # some other provenance: neither target nor clean control
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
img = imread(path_str)
|
||||
if img is None or min(img.shape[:2]) < 200:
|
||||
return None
|
||||
try:
|
||||
fired = [d.key for d in detect_marks(img) if d.detected]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
if fired:
|
||||
arm = "covered"
|
||||
elif tc260:
|
||||
arm = "uncovered"
|
||||
else:
|
||||
arm = "clean"
|
||||
|
||||
score, scale = tail_score(DoubaoEngine(), img, build_tail_silhouette())
|
||||
return {
|
||||
"src": path.name,
|
||||
"arm": arm,
|
||||
"fired": fired,
|
||||
"tail_score": round(score, 4),
|
||||
"tail_scale": scale,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--n", type=int, default=6000)
|
||||
ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 4) - 2))
|
||||
ap.add_argument("--out", type=Path, default=OUT)
|
||||
ap.add_argument("--report-only", action="store_true")
|
||||
ap.add_argument(
|
||||
"--sheet-at", type=float, default=0.0, help="write a contact sheet of `uncovered` crops scoring >= this"
|
||||
)
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.report_only:
|
||||
rows = [json.loads(x) for x in a.out.read_text(encoding="utf-8").splitlines() if x.strip()]
|
||||
report(rows)
|
||||
if a.sheet_at:
|
||||
contact_sheet(rows, a.sheet_at)
|
||||
return
|
||||
|
||||
build_tail_silhouette()
|
||||
pool = glob.glob(str(CORPUS / "*" / "*"))
|
||||
random.Random(19).shuffle(pool) # noqa: S311 -- deterministic sampling, not crypto
|
||||
pool = pool[: a.n]
|
||||
print(f"scanning {len(pool)} corpus files workers={a.workers}")
|
||||
print(f"tail ladder {TAIL_SCALES[0]} .. {TAIL_SCALES[-1]} ({len(TAIL_SCALES)} rungs)\n", flush=True)
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
a.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(a.out, "w", encoding="utf-8") as fh, ProcessPoolExecutor(max_workers=a.workers) as ex:
|
||||
futures = [ex.submit(_one, p) for p in pool]
|
||||
for i, fut in enumerate(as_completed(futures), 1):
|
||||
try:
|
||||
rec = fut.result()
|
||||
except Exception: # noqa: S112 -- one bad file must not kill the sweep
|
||||
continue
|
||||
if rec is None:
|
||||
continue
|
||||
fh.write(json.dumps(rec) + "\n")
|
||||
rows.append(rec)
|
||||
if i % 500 == 0:
|
||||
fh.flush()
|
||||
print(f" {i}/{len(pool)} usable={len(rows)}", flush=True)
|
||||
report(rows)
|
||||
if a.sheet_at:
|
||||
contact_sheet(rows, a.sheet_at)
|
||||
|
||||
|
||||
def report(rows: list[dict[str, Any]]) -> None:
|
||||
arms = {k: [r for r in rows if r["arm"] == k] for k in ("covered", "uncovered", "clean")}
|
||||
print(f"\n{'=' * 80}\nGENERIC CJK TAIL PROBE ({', '.join(f'{k}={len(v)}' for k, v in arms.items())})\n{'=' * 80}")
|
||||
print("\nScore distribution of the shared `AI生成` tail on the tophat response\n")
|
||||
print(f"{'arm':11s} {'n':>6s} {'p50':>7s} {'p75':>7s} {'p90':>7s} {'p95':>7s} {'p99':>7s} {'max':>7s}")
|
||||
for k, v in arms.items():
|
||||
if not v:
|
||||
continue
|
||||
s = np.array([r["tail_score"] for r in v])
|
||||
qs = [np.percentile(s, q) for q in (50, 75, 90, 95, 99)]
|
||||
print(f"{k:11s} {len(v):6d} " + " ".join(f"{q:7.3f}" for q in qs) + f" {s.max():7.3f}")
|
||||
|
||||
if arms["clean"] and arms["uncovered"]:
|
||||
clean = np.array([r["tail_score"] for r in arms["clean"]])
|
||||
unc = np.array([r["tail_score"] for r in arms["uncovered"]])
|
||||
print("\n\nOPERATING POINTS -- threshold set on the CLEAN arm, yield read on `uncovered`")
|
||||
print("`clean` fires are the cost; `uncovered` fires are candidate recoveries that")
|
||||
print("still need eyeballing, since TC260 provenance does not imply a visible mark.\n")
|
||||
print(f"{'threshold':>10s} {'clean fire':>12s} {'uncovered fire':>16s} {'candidates':>12s}")
|
||||
for q in (95, 97.5, 99, 99.5, 99.9):
|
||||
t = float(np.percentile(clean, q))
|
||||
cf = float((clean >= t).mean())
|
||||
uf = float((unc >= t).mean())
|
||||
print(f"{t:10.3f} {100 * cf:11.2f}% {100 * uf:15.1f}% {int((unc >= t).sum()):12d}")
|
||||
|
||||
cov = arms["covered"]
|
||||
if cov:
|
||||
s = np.array([r["tail_score"] for r in cov])
|
||||
print(f"\n\nSANITY: on frames where a registered mark fires, the tail scores median {np.median(s):.3f}.")
|
||||
print("A tail template that cannot score these is broken, whatever it does elsewhere.")
|
||||
|
||||
|
||||
def contact_sheet(rows: list[dict[str, Any]], thresh: float, limit: int = 30) -> None:
|
||||
"""Crop the corner of the top-scoring `uncovered` frames so the fires can be judged."""
|
||||
from remove_ai_watermarks.doubao_engine import DoubaoEngine
|
||||
from remove_ai_watermarks.image_io import imread
|
||||
|
||||
eng = DoubaoEngine()
|
||||
picks = sorted(
|
||||
(r for r in rows if r["arm"] == "uncovered" and r["tail_score"] >= thresh),
|
||||
key=lambda r: -r["tail_score"],
|
||||
)[:limit]
|
||||
tiles = []
|
||||
for r in picks:
|
||||
hits = glob.glob(str(CORPUS / "*" / r["src"]))
|
||||
if not hits:
|
||||
continue
|
||||
img = imread(hits[0])
|
||||
if img is None:
|
||||
continue
|
||||
loc = eng.locate(img)
|
||||
crop = img[loc.y : loc.y + loc.h, loc.x : loc.x + loc.w]
|
||||
if crop.size:
|
||||
tiles.append(cv2.resize(crop, (320, 96), interpolation=cv2.INTER_AREA))
|
||||
if tiles:
|
||||
dest = REPO / "data" / "spaces" / "_cjk_tail_sheet.png"
|
||||
cv2.imwrite(str(dest), np.vstack(tiles))
|
||||
print(f"\ncontact sheet ({len(tiles)} crops, score >= {thresh:.3f}) -> {dest}")
|
||||
print("scores: " + ", ".join(f"{r['tail_score']:.2f}" for r in picks[: len(tiles)]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,319 @@
|
||||
"""Tier B2: detector response curves -- recall vs mark size, contrast, background and aspect.
|
||||
|
||||
WHY THIS EXISTS
|
||||
Every recall number the project quotes rests on found positives: doubao 89% on n=240,
|
||||
jimeng 71% on n=14, the pill 50% on n=6. Those tell you how the detectors do on the
|
||||
marks a corpus happened to contain, and nothing about WHERE they fall over. Two bugs
|
||||
have now shipped in exactly that blind spot:
|
||||
|
||||
* `scale_basis` -- doubao detected 0 of 435 LANDSCAPE marks, a 100% miss, because
|
||||
every fraction was calibrated on portrait captures where width == short side.
|
||||
* the front-end mismatch -- detection moved to the continuous `tophat` response while
|
||||
the removal mask still came from the BINARIZED blob, so ~8% of doubao detections
|
||||
produced an empty mask and removal was a silent no-op.
|
||||
|
||||
Both are geometry/plumbing failures at the edge of the operating range, and both were
|
||||
invisible to a corpus sweep because the corpus is concentrated in the middle of it.
|
||||
This harness constructs the edges instead of waiting for them to turn up.
|
||||
|
||||
THE AGGREGATE HERE IS NOT RECALL
|
||||
The grid deliberately visits sizes and opacities the engines were never calibrated for,
|
||||
so a mark counted as missed at size=0.6 is not evidence of a weak detector -- it is the
|
||||
point of the sweep. Quote the NOMINAL cell (size=1.0, alpha=1.0) as the reference, and
|
||||
read the per-axis tables for shape. Production recall comes from an unbiased corpus
|
||||
sample (`scripts/visible_recall_sample.py`), never from here.
|
||||
|
||||
WHAT IT MEASURES -- TWO NUMBERS, NOT ONE
|
||||
`detected` the detector fires.
|
||||
`maskable` the same call path then yields a NON-EMPTY removal mask.
|
||||
|
||||
The second is the one that has no natural home anywhere else. A mark that is detected
|
||||
but not maskable is reported by `identify` and silently skipped by `visible` -- the user
|
||||
sees a mark the tool says it removed. Anything that measures only detection scores that
|
||||
bug as a success. They are reported side by side and their GAP is the headline.
|
||||
|
||||
THE CONSTRUCTION
|
||||
Take a verified-clean corpus image, stamp a real mark onto it with the forward model the
|
||||
reverse-alpha work established (`stamped = (1-a)*bg + a*white`), and sweep two knobs the
|
||||
engine's own geometry normally pins:
|
||||
|
||||
size multiplies the glyph box the engine would use (a re-rendered or rescaled mark)
|
||||
alpha multiplies the captured opacity (bold stamp -> faint translucent overlay)
|
||||
|
||||
Background texture and frame aspect are not swept -- they come from the corpus and are
|
||||
recorded, so the grid is crossed with real backgrounds rather than synthetic flats.
|
||||
|
||||
WHY EVERY SOURCE ALSO RUNS UNSTAMPED
|
||||
A recall curve with no false-fire baseline is uninterpretable in the same way a fill
|
||||
PSNR with no damage baseline is: "detected on 70% of faint marks" is a triumph or a
|
||||
scandal depending on whether the detector also fires on 70% of the clean frames those
|
||||
marks were stamped onto. The control runs the identical call path on the identical
|
||||
image with no stamp.
|
||||
|
||||
READING `mask_hit`
|
||||
Fraction of the stamped glyph box the removal mask covers -- real localization accuracy
|
||||
for the text marks and gemini, because the mask has to be placed from a detection.
|
||||
For `jimeng_pill` it is CONSTRUCTED and means nothing: the pill's footprint is a fixed
|
||||
top-left geometry box, so it covers the stamp by definition. Reported, and flagged.
|
||||
|
||||
DATA SAFETY
|
||||
Corpus images are user uploads: read-only, local analysis, gitignored output under
|
||||
data/spaces/. Records source filenames and measurements, never image content.
|
||||
|
||||
uv run python scripts/detector_response.py --n 12 # trial, measures throughput
|
||||
uv run python scripts/detector_response.py --n 150 # the real run, resumable
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from fill_quality import SLOT_STAMPABLE, STAMPABLE, clean_sources, stamp_any, texture_of
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
OUT = REPO / "data" / "spaces" / "_detector_response.jsonl"
|
||||
|
||||
# 1.0 = the geometry/opacity the engine's own constants assume. The sweep reaches below
|
||||
# it (a mark rendered smaller, or a faint translucent overlay -- the class the tophat
|
||||
# front-end was introduced for) and above it (a larger re-render, or an upscaled upload).
|
||||
SIZES = (0.6, 0.8, 1.0, 1.3)
|
||||
ALPHAS = (0.3, 0.5, 0.75, 1.0)
|
||||
MARKS = (*STAMPABLE, *SLOT_STAMPABLE)
|
||||
|
||||
|
||||
def aspect_of(image: np.ndarray) -> str:
|
||||
h, w = image.shape[:2]
|
||||
r = w / max(1, h)
|
||||
return "portrait" if r < 0.9 else ("landscape" if r > 1.1 else "square")
|
||||
|
||||
|
||||
def _probe(image: np.ndarray, mark_key: str, box: tuple[int, int, int, int] | None) -> dict[str, Any]:
|
||||
"""Run the product detect -> localize path once and score it.
|
||||
|
||||
``box`` is the stamped glyph box, or None for the unstamped control.
|
||||
"""
|
||||
from remove_ai_watermarks.watermark_registry import get_mark
|
||||
|
||||
mark = get_mark(mark_key)
|
||||
loc = mark.localize(image, force=False)
|
||||
mask_px = 0 if loc.mask is None else int(np.count_nonzero(loc.mask))
|
||||
hit = None
|
||||
if box is not None and loc.mask is not None:
|
||||
x, y, w, h = box
|
||||
inside = loc.mask[y : y + h, x : x + w]
|
||||
hit = round(float(np.count_nonzero(inside)) / max(1, w * h), 3)
|
||||
return {
|
||||
"detected": bool(loc.detected),
|
||||
"confidence": round(float(loc.confidence), 4),
|
||||
"maskable": mask_px > 0,
|
||||
"mask_px": mask_px,
|
||||
"mask_hit": hit,
|
||||
}
|
||||
|
||||
|
||||
def _one_source(path_str: str) -> list[dict[str, Any]]:
|
||||
"""Every (mark, size, alpha) cell for one clean background, plus its controls."""
|
||||
from remove_ai_watermarks.image_io import imread
|
||||
|
||||
base = imread(path_str)
|
||||
if base is None:
|
||||
return []
|
||||
name = Path(path_str).name
|
||||
asp = aspect_of(base)
|
||||
h, w = base.shape[:2]
|
||||
rows: list[dict[str, Any]] = []
|
||||
|
||||
for key in MARKS:
|
||||
common = {"src": name, "mark": key, "aspect": asp, "px": w * h}
|
||||
# Control first: the same call path on the same frame with nothing stamped.
|
||||
rows.append({**common, "cell": "control", "size": None, "alpha": None, **_probe(base, key, None)})
|
||||
for size in SIZES:
|
||||
for alpha in ALPHAS:
|
||||
st = stamp_any(base, key, size_mult=size, alpha_mult=alpha)
|
||||
if st is None:
|
||||
continue
|
||||
stamped, box = st
|
||||
# Texture of the region the mark sits on, from the CLEAN frame -- the
|
||||
# same median-Sobel proxy the pill's flatness gate uses.
|
||||
x, y, gw, gh = box
|
||||
rows.append(
|
||||
{
|
||||
**common,
|
||||
"cell": "stamped",
|
||||
"size": size,
|
||||
"alpha": alpha,
|
||||
"texture": round(texture_of(base[y : y + gh, x : x + gw]), 2),
|
||||
**_probe(stamped, key, box),
|
||||
}
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
|
||||
if n == 0:
|
||||
return (0.0, 0.0)
|
||||
p, d = k / n, 1 + z * z / n
|
||||
c = (p + z * z / (2 * n)) / d
|
||||
hw = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
|
||||
return (100 * (c - hw), 100 * (c + hw))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--n", type=int, default=150, help="verified-clean source images")
|
||||
ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 4) - 2))
|
||||
ap.add_argument("--out", type=Path, default=OUT)
|
||||
ap.add_argument("--restart", action="store_true")
|
||||
ap.add_argument("--report-only", action="store_true", help="re-read --out and print the report")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.report_only:
|
||||
rows = [json.loads(line) for line in a.out.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
report(rows)
|
||||
return
|
||||
|
||||
if a.restart and a.out.exists():
|
||||
a.out.unlink()
|
||||
done: set[str] = set()
|
||||
if a.out.exists():
|
||||
with open(a.out, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
try:
|
||||
done.add(json.loads(line)["src"])
|
||||
except Exception: # noqa: S112 -- tolerate a torn last line
|
||||
continue
|
||||
|
||||
print(f"selecting {a.n} verified-clean sources (already measured: {len(done)})...", flush=True)
|
||||
sources = [p for p in clean_sources(a.n) if p.name not in done]
|
||||
print(f"to do {len(sources)} workers {a.workers} grid {len(MARKS)}x{len(SIZES)}x{len(ALPHAS)}\n", flush=True)
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
a.out.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(a.out, "a", encoding="utf-8") as fh, ProcessPoolExecutor(max_workers=a.workers) as ex:
|
||||
futures = {ex.submit(_one_source, str(p)): p for p in sources}
|
||||
for i, fut in enumerate(as_completed(futures), 1):
|
||||
try:
|
||||
got = fut.result()
|
||||
except Exception as e: # a crashed source must not kill the sweep
|
||||
print(f" source failed: {type(e).__name__}: {e}", flush=True)
|
||||
continue
|
||||
for r in got:
|
||||
fh.write(json.dumps(r) + "\n")
|
||||
rows += got
|
||||
if i % 10 == 0:
|
||||
fh.flush()
|
||||
print(f" {i}/{len(sources)}", flush=True)
|
||||
|
||||
if done: # merge the resumed rows so the report covers the whole file
|
||||
rows = [json.loads(line) for line in a.out.read_text(encoding="utf-8").splitlines() if line.strip()]
|
||||
report(rows)
|
||||
|
||||
|
||||
def _rate(rs: list[dict[str, Any]], field: str) -> tuple[float, float, float, int]:
|
||||
k, n = sum(bool(r[field]) for r in rs), len(rs)
|
||||
lo, hi = wilson(k, n)
|
||||
return (100 * k / n if n else 0.0, lo, hi, n)
|
||||
|
||||
|
||||
def report(rows: list[dict[str, Any]]) -> None:
|
||||
stamped = [r for r in rows if r["cell"] == "stamped"]
|
||||
control = [r for r in rows if r["cell"] == "control"]
|
||||
if not stamped:
|
||||
print("no measurements")
|
||||
return
|
||||
srcs = len({r["src"] for r in rows})
|
||||
print(f"\n{'=' * 84}\nDETECTOR RESPONSE {len(stamped)} stamped cells over {srcs} clean backgrounds\n{'=' * 84}")
|
||||
|
||||
# Lead with the nominal cell. The grid deliberately includes sizes and opacities the
|
||||
# engines were never calibrated for, so an aggregate over the whole grid is an
|
||||
# adversarial score, NOT production recall -- quoting it as recall would understate
|
||||
# the detectors badly. The nominal cell is the honest reference point: the mark at the
|
||||
# exact geometry and opacity the engine assumes, on a real clean background.
|
||||
print("\nNOMINAL CELL (size=1.0, alpha=1.0) -- a mark exactly as the engine expects it.")
|
||||
print("Read every other number against THIS, not as production recall: the grid is")
|
||||
print("adversarial by design and its aggregate is not a recall figure.\n")
|
||||
print(f"{'mark':13s} {'n':>5s} {'detected':>10s} {'maskable':>10s}")
|
||||
for key in MARKS:
|
||||
rs = [r for r in stamped if r["mark"] == key and r["size"] == 1.0 and r["alpha"] == 1.0]
|
||||
if rs:
|
||||
print(f"{key:13s} {len(rs):5d} {_rate(rs, 'detected')[0]:9.1f}% {_rate(rs, 'maskable')[0]:9.1f}%")
|
||||
|
||||
print("\n\nFALSE FIRE on the unstamped controls (the baseline every recall below is read against)\n")
|
||||
print(f"{'mark':13s} {'n':>5s} {'fires':>8s} {'95% CI':>16s}")
|
||||
for key in MARKS:
|
||||
rs = [r for r in control if r["mark"] == key]
|
||||
if rs:
|
||||
pct, lo, hi, n = _rate(rs, "detected")
|
||||
print(f"{key:13s} {n:5d} {pct:7.1f}% {f'{lo:.1f}-{hi:.1f}':>16s}")
|
||||
|
||||
print("\n\nDETECTED vs MASKABLE by mark. The GAP is the silent-no-op rate:")
|
||||
print("a mark counted here as detected-but-not-maskable is reported by `identify` and")
|
||||
print("skipped by `visible`, so the user is told it was removed when it was not.\n")
|
||||
print(f"{'mark':13s} {'n':>6s} {'detected':>10s} {'maskable':>10s} {'gap':>7s} {'mask_hit':>9s}")
|
||||
for key in MARKS:
|
||||
rs = [r for r in stamped if r["mark"] == key]
|
||||
if not rs:
|
||||
continue
|
||||
det = [r for r in rs if r["detected"]]
|
||||
d, m = _rate(rs, "detected")[0], _rate(rs, "maskable")[0]
|
||||
gap = (100 * sum(1 for r in det if not r["maskable"]) / len(det)) if det else 0.0
|
||||
hits = [r["mask_hit"] for r in det if r["mask_hit"] is not None]
|
||||
note = " (constructed)" if key == "jimeng_pill" else ""
|
||||
hv = f"{float(np.median(hits)):.2f}" if hits else "-"
|
||||
print(f"{key:13s} {len(rs):6d} {d:9.1f}% {m:9.1f}% {gap:6.1f}% {hv:>9s}{note}")
|
||||
|
||||
for axis, label in (("alpha", "CONTRAST (alpha multiplier)"), ("size", "SIZE (glyph box multiplier)")):
|
||||
print(f"\n\nRECALL BY {label} -- 1.0 is the geometry/opacity the engine assumes\n")
|
||||
vals = sorted({r[axis] for r in stamped})
|
||||
print(f"{'mark':13s}" + "".join(f"{v:>12}" for v in vals))
|
||||
for key in MARKS:
|
||||
cells = []
|
||||
for v in vals:
|
||||
rs = [r for r in stamped if r["mark"] == key and r[axis] == v]
|
||||
cells.append(f"{_rate(rs, 'detected')[0]:10.0f}%" if rs else f"{'-':>11s}")
|
||||
print(f"{key:13s}" + "".join(f"{c:>12}" for c in cells))
|
||||
|
||||
print("\n\nRECALL BY FRAME ASPECT -- the axis the `scale_basis` bug lived on\n")
|
||||
asps = ["portrait", "square", "landscape"]
|
||||
print(f"{'mark':13s}" + "".join(f"{x:>14s}" for x in asps))
|
||||
for key in MARKS:
|
||||
cells = []
|
||||
for asp in asps:
|
||||
rs = [r for r in stamped if r["mark"] == key and r["aspect"] == asp]
|
||||
cells.append(f"{_rate(rs, 'detected')[0]:.0f}% (n={len(rs)})" if rs else "-")
|
||||
print(f"{key:13s}" + "".join(f"{c:>14s}" for c in cells))
|
||||
|
||||
tex = sorted(r["texture"] for r in stamped if "texture" in r)
|
||||
if tex:
|
||||
t1, t2 = tex[len(tex) // 3], tex[2 * len(tex) // 3]
|
||||
print(f"\n\nRECALL BY BACKGROUND TEXTURE (terciles at {t1:.1f} / {t2:.1f} median-Sobel)\n")
|
||||
print(f"{'mark':13s}" + "".join(f"{x:>14s}" for x in ("flat", "mid", "textured")))
|
||||
for key in MARKS:
|
||||
cells = []
|
||||
for lo, hi in ((-1, t1), (t1, t2), (t2, 1e9)):
|
||||
rs = [r for r in stamped if r["mark"] == key and lo < r.get("texture", -2) <= hi]
|
||||
cells.append(f"{_rate(rs, 'detected')[0]:.0f}% (n={len(rs)})" if rs else "-")
|
||||
print(f"{key:13s}" + "".join(f"{c:>14s}" for c in cells))
|
||||
|
||||
worst = collections.Counter((r["mark"], r["size"], r["alpha"]) for r in stamped if not r["detected"]).most_common(8)
|
||||
if worst:
|
||||
print("\n\nWORST CELLS (most misses)\n")
|
||||
for (mark, size, alpha), n in worst:
|
||||
print(f" {mark:13s} size={size} alpha={alpha} {n} misses")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
+46
-24
@@ -89,8 +89,31 @@ def engine_for(mark_key: str) -> Any:
|
||||
return cls()
|
||||
|
||||
|
||||
def stamp(image: np.ndarray, mark_key: str) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
"""Composite a mark's alpha glyph into its canonical corner. Returns (stamped, bbox)."""
|
||||
def composite(image: np.ndarray, alpha: np.ndarray, x: int, y: int) -> np.ndarray:
|
||||
"""The single forward model every stamp uses: ``stamped = (1-a)*bg + a*white``.
|
||||
|
||||
``alpha`` is float and clipped to [0,1] here, so a caller may pre-multiply it by an
|
||||
opacity factor without worrying about overflow. Shared by ``stamp``/``stamp_slot``
|
||||
here and by ``scripts/detector_response.py`` (which imports it) so the model is
|
||||
written ONCE.
|
||||
"""
|
||||
gh, gw = alpha.shape[:2]
|
||||
out = image.copy()
|
||||
roi = out[y : y + gh, x : x + gw].astype(np.float32)
|
||||
a3 = np.clip(alpha, 0.0, 1.0)[..., None]
|
||||
out[y : y + gh, x : x + gw] = np.clip(roi * (1 - a3) + 255.0 * a3, 0, 255).astype(np.uint8)
|
||||
return out
|
||||
|
||||
|
||||
def stamp(
|
||||
image: np.ndarray, mark_key: str, *, size_mult: float = 1.0, alpha_mult: float = 1.0
|
||||
) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
"""Composite a mark's alpha glyph into its canonical corner. Returns (stamped, bbox).
|
||||
|
||||
``size_mult`` scales the glyph box off the engine's nominal geometry and ``alpha_mult``
|
||||
its opacity -- both default to 1.0 (the geometry the engine assumes), and
|
||||
`scripts/detector_response.py` sweeps them to build response curves.
|
||||
"""
|
||||
from remove_ai_watermarks._text_mark_engine import load_alpha_template
|
||||
|
||||
engine = engine_for(mark_key)
|
||||
@@ -101,19 +124,15 @@ def stamp(image: np.ndarray, mark_key: str) -> tuple[np.ndarray, tuple[int, int,
|
||||
|
||||
loc = engine.locate(image)
|
||||
base = engine.scale_base(image)
|
||||
gw = max(cfg.min_gw, int(cfg.alpha_width_frac * base))
|
||||
gh = max(4, int(cfg.alpha_height_frac * base))
|
||||
gw = max(cfg.min_gw, int(cfg.alpha_width_frac * base * size_mult))
|
||||
gh = max(4, int(cfg.alpha_height_frac * base * size_mult))
|
||||
if gw < 8 or gh < 4 or gw > loc.w or gh > loc.h:
|
||||
return None
|
||||
|
||||
a = cv2.resize(alpha, (gw, gh), interpolation=cv2.INTER_AREA).astype(np.float32)
|
||||
a = cv2.resize(alpha, (gw, gh), interpolation=cv2.INTER_AREA).astype(np.float32) * alpha_mult
|
||||
x = loc.x + (loc.w - gw) // 2
|
||||
y = loc.y + (loc.h - gh) // 2
|
||||
out = image.copy()
|
||||
roi = out[y : y + gh, x : x + gw].astype(np.float32)
|
||||
a3 = a[..., None]
|
||||
out[y : y + gh, x : x + gw] = np.clip(roi * (1 - a3) + 255.0 * a3, 0, 255).astype(np.uint8)
|
||||
return out, (x, y, gw, gh)
|
||||
return composite(image, a, x, y), (x, y, gw, gh)
|
||||
|
||||
|
||||
def _slot_alpha(mark_key: str) -> np.ndarray | None:
|
||||
@@ -134,12 +153,15 @@ def _slot_alpha(mark_key: str) -> np.ndarray | None:
|
||||
return None
|
||||
|
||||
|
||||
def stamp_slot(image: np.ndarray, mark_key: str) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
def stamp_slot(
|
||||
image: np.ndarray, mark_key: str, *, size_mult: float = 1.0, alpha_mult: float = 1.0
|
||||
) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
"""Stamp a mark into its OWN default footprint slot (the `--no-detect` geometry).
|
||||
|
||||
Used for gemini and the pill, which have no bundled alpha asset with corner
|
||||
fractions. The mark is fitted into the middle of its slot so the fill has to
|
||||
recover the same kind of region it would in production.
|
||||
recover the same kind of region it would in production. ``size_mult``/``alpha_mult``
|
||||
match ``stamp`` (both default to 1.0).
|
||||
"""
|
||||
from remove_ai_watermarks.watermark_registry import get_mark
|
||||
|
||||
@@ -154,21 +176,21 @@ def stamp_slot(image: np.ndarray, mark_key: str) -> tuple[np.ndarray, tuple[int,
|
||||
return None
|
||||
y0, y1, x0, x1 = int(ys.min()), int(ys.max()), int(xs.min()), int(xs.max())
|
||||
bw, bh = x1 - x0 + 1, y1 - y0 + 1
|
||||
gw, gh = max(8, int(bw * 0.7)), max(8, int(bh * 0.7))
|
||||
if gw < 8 or gh < 8 or gw > image.shape[1] or gh > image.shape[0]:
|
||||
return None
|
||||
a = cv2.resize(alpha, (gw, gh), interpolation=cv2.INTER_AREA).astype(np.float32)
|
||||
a = np.clip(a, 0.0, 1.0)
|
||||
gw, gh = max(8, int(bw * 0.7 * size_mult)), max(8, int(bh * 0.7 * size_mult))
|
||||
a = cv2.resize(alpha, (gw, gh), interpolation=cv2.INTER_AREA).astype(np.float32) * alpha_mult
|
||||
x, y = x0 + (bw - gw) // 2, y0 + (bh - gh) // 2
|
||||
out = image.copy()
|
||||
roi = out[y : y + gh, x : x + gw].astype(np.float32)
|
||||
a3 = a[..., None]
|
||||
out[y : y + gh, x : x + gw] = np.clip(roi * (1 - a3) + 255.0 * a3, 0, 255).astype(np.uint8)
|
||||
return out, (x, y, gw, gh)
|
||||
# A large size_mult can grow the glyph past its slot and push x/y negative; guard so
|
||||
# composite never writes out of bounds (numpy would wrap a negative index silently).
|
||||
if x < 0 or y < 0 or x + gw > image.shape[1] or y + gh > image.shape[0]:
|
||||
return None
|
||||
return composite(image, a, x, y), (x, y, gw, gh)
|
||||
|
||||
|
||||
def stamp_any(image: np.ndarray, mark_key: str) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
return stamp(image, mark_key) if mark_key in STAMPABLE else stamp_slot(image, mark_key)
|
||||
def stamp_any(
|
||||
image: np.ndarray, mark_key: str, *, size_mult: float = 1.0, alpha_mult: float = 1.0
|
||||
) -> tuple[np.ndarray, tuple[int, int, int, int]] | None:
|
||||
fn = stamp if mark_key in STAMPABLE else stamp_slot
|
||||
return fn(image, mark_key, size_mult=size_mult, alpha_mult=alpha_mult)
|
||||
|
||||
|
||||
def clean_sources(n: int, seed: int = 11) -> list[Path]:
|
||||
|
||||
@@ -0,0 +1,305 @@
|
||||
"""How much recall is the coarse scale ladder costing, and what would a denser one cost?
|
||||
|
||||
THE FINDING THIS MEASURES
|
||||
`_tophat_score` sweeps three rungs -- (0.8, 1.0, 1.25) -- and the `binary` front-end
|
||||
sweeps none at all. Measured on stamped marks over controlled backgrounds
|
||||
(`scripts/detector_response.py`), the response is a COMB: doubao scores 0.99 exactly at
|
||||
each rung and collapses to 0.37-0.48 between them, against a 0.50 gate. So a mark whose
|
||||
rendered size lands mid-gap is missed at FULL contrast. Jimeng (binary, one nominal
|
||||
size) holds a single 0.90-1.20 lobe; samsung only 0.95-1.05.
|
||||
|
||||
WHY A SYNTHETIC SWEEP IS NOT ENOUGH TO JUSTIFY A FIX
|
||||
Dead zones only cost recall if real marks land in them. The fractions were calibrated on
|
||||
real captures, so it is entirely possible that real marks cluster at ratio 1.0 and the
|
||||
gaps are never visited. That question is not answerable by stamping -- it needs the
|
||||
corpus.
|
||||
|
||||
THE MEASUREMENT
|
||||
Positives are images carrying INDEPENDENT vendor provenance (TC260 / C2PA metadata
|
||||
naming the vendor), which is evidence that does not come from the pixel detector we are
|
||||
grading -- the same discriminator that settled the pill gate. For every such image the
|
||||
detector currently MISSES, rescore at a dense ladder and ask whether it would now cross
|
||||
the gate, and at which scale.
|
||||
|
||||
Negatives are images with NO metadata signal at all. That is deliberately NOT "and no
|
||||
mark was detected": defining the set by the detector's own verdict would make its
|
||||
false-fire rate 0 by construction and the comparison vacuous. The cost is that a
|
||||
metadata-stripped screenshot of a marked image sits in the negative set and its correct
|
||||
detection is counted as a false fire -- but that impurity is identical under both
|
||||
ladders, so the DELTA between them, which is what the fix is judged on, stays sound.
|
||||
|
||||
Two more impurities to keep in view when reading the positive arm. Provenance says the
|
||||
vendor produced the file, not that a visible mark is on it. And the TC260 label names no
|
||||
specific vendor, so `visible_provenance` maps it to BOTH doubao and jimeng -- a "doubao
|
||||
positive" may carry a jimeng mark or none. Both inflate the miss count, so the recovery
|
||||
percentage is an upper bound on what a denser ladder buys.
|
||||
|
||||
DATA SAFETY
|
||||
Corpus images are user uploads: read-only, local analysis, gitignored output.
|
||||
|
||||
uv run python scripts/ladder_headroom.py --mark doubao --n 4000
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import collections
|
||||
import glob
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import random
|
||||
import sys
|
||||
from concurrent.futures import ProcessPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import cv2
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).parent.parent))
|
||||
|
||||
REPO = Path(__file__).resolve().parents[1]
|
||||
CORPUS = REPO / "data" / "spaces" / "originals"
|
||||
OUT = REPO / "data" / "spaces" / "_ladder_headroom.jsonl"
|
||||
|
||||
# The rungs the product ships today, and the dense ladder under evaluation. The dense one
|
||||
# is geometric with a ~6% step, chosen from the measured half-width of a rung's lobe
|
||||
# (doubao holds >=0.5 for about +/-6% around each peak), so no gap is left uncovered.
|
||||
SHIPPED = (0.8, 1.0, 1.25)
|
||||
LADDERS: dict[str, tuple[float, ...]] = {
|
||||
# A geometric ~6% step across the whole plausible range. Measures the CEILING of what
|
||||
# any ladder change can buy, and the worst case of what it costs in false fires.
|
||||
"dense": tuple(round(0.70 * (1.06**i), 4) for i in range(13)), # 0.70 .. ~1.49
|
||||
# The targeted alternative. On the full run, 26 of 28 recoveries came from ONE rung
|
||||
# (~1.116) and 22 of 28 were LANDSCAPE frames -- that is a geometry gap, not a density
|
||||
# gap, so the honest comparison is one extra rung against thirteen.
|
||||
"plus_one": (0.8, 1.0, 1.1157, 1.25),
|
||||
"shipped": SHIPPED,
|
||||
}
|
||||
DENSE = LADDERS["dense"]
|
||||
# Score every rung any candidate ladder might use and store them all, so a new candidate
|
||||
# is evaluated by re-reading the JSONL instead of re-running the hour-long sweep.
|
||||
PROBE_SCALES: tuple[float, ...] = tuple(sorted({s for rungs in LADDERS.values() for s in rungs}))
|
||||
|
||||
|
||||
def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
|
||||
if n == 0:
|
||||
return (0.0, 0.0)
|
||||
p, d = k / n, 1 + z * z / n
|
||||
c = (p + z * z / (2 * n)) / d
|
||||
hw = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
|
||||
return (100 * (c - hw), 100 * (c + hw))
|
||||
|
||||
|
||||
def score_at_scales(engine: Any, image: np.ndarray, scales: tuple[float, ...]) -> dict[float, float]:
|
||||
"""`_tophat_score`'s inner loop, opened up so the ladder is a parameter.
|
||||
|
||||
Deliberately reaches into the engine (`tophat_response`, `_glyph_silhouette`): a
|
||||
measurement script may, product code may not. Kept a faithful copy of the shipped
|
||||
scoring -- if it drifts, the numbers below stop describing the product.
|
||||
"""
|
||||
c = engine.config
|
||||
loc = engine.locate(image)
|
||||
resp = engine.tophat_response(image, loc)
|
||||
sil = engine._glyph_silhouette()
|
||||
if resp is None or sil is None:
|
||||
return {}
|
||||
base = engine.scale_base(image)
|
||||
out: dict[float, float] = {}
|
||||
for scale in scales:
|
||||
gw = max(c.min_gw, int(c.alpha_width_frac * base * scale))
|
||||
gh = max(4, int(c.alpha_height_frac * base * scale))
|
||||
if gw >= resp.shape[1] or gh >= resp.shape[0]:
|
||||
continue
|
||||
tmpl = cv2.resize(sil, (gw, gh), interpolation=cv2.INTER_AREA).astype(np.float32)
|
||||
if c.template_blur > 0:
|
||||
tmpl = cv2.GaussianBlur(tmpl, (0, 0), sigmaX=c.template_blur, sigmaY=c.template_blur)
|
||||
out[scale] = float(cv2.matchTemplate(resp, tmpl.astype(np.uint8), cv2.TM_CCOEFF_NORMED).max())
|
||||
return out
|
||||
|
||||
|
||||
def _one(args: tuple[str, str]) -> dict[str, Any] | None:
|
||||
path_str, mark_key = args
|
||||
from remove_ai_watermarks.api import visible_provenance
|
||||
from remove_ai_watermarks.identify import identify
|
||||
from remove_ai_watermarks.image_io import imread
|
||||
from remove_ai_watermarks.watermark_registry import get_mark
|
||||
|
||||
path = Path(path_str)
|
||||
try:
|
||||
prov = visible_provenance(path)
|
||||
except Exception:
|
||||
return None
|
||||
positive = mark_key in prov
|
||||
|
||||
if not positive:
|
||||
# A negative is a frame with NO metadata signal at all -- evidence independent of
|
||||
# the detector being graded. Do NOT also require that no mark was detected: that
|
||||
# defines the negative set by the very verdict under test, and "false fire today"
|
||||
# would then be 0 by construction on every run.
|
||||
try:
|
||||
if identify(path, check_visible=False).signals:
|
||||
return None
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
img = imread(path_str)
|
||||
if img is None or min(img.shape[:2]) < 200:
|
||||
return None
|
||||
|
||||
mark = get_mark(mark_key)
|
||||
det = mark.detect(img)
|
||||
|
||||
import importlib
|
||||
|
||||
mod = importlib.import_module(f"remove_ai_watermarks.{mark_key}_engine")
|
||||
cls = next(
|
||||
o for n, o in vars(mod).items() if isinstance(o, type) and n.endswith("Engine") and n != "TextMarkEngine"
|
||||
)
|
||||
engine = cls()
|
||||
scores = score_at_scales(engine, img, PROBE_SCALES)
|
||||
if not scores:
|
||||
return None
|
||||
gate = engine.config.detect_ncc_threshold
|
||||
dense = {s: v for s, v in scores.items() if s in LADDERS["dense"]}
|
||||
best_scale = max(dense, key=lambda s: dense[s]) if dense else 0.0
|
||||
h, w = img.shape[:2]
|
||||
return {
|
||||
"src": path.name,
|
||||
"arm": "positive" if positive else "negative",
|
||||
"detected_now": bool(det.detected),
|
||||
"conf_now": round(float(det.confidence), 4),
|
||||
"gate": gate,
|
||||
# Every rung, so a new candidate ladder is evaluated by re-reading this file.
|
||||
"scores": {str(s): round(v, 4) for s, v in scores.items()},
|
||||
"dense_best": round(dense[best_scale], 4) if dense else 0.0,
|
||||
"dense_best_scale": best_scale,
|
||||
"dense_crosses": bool(dense) and dense[best_scale] >= gate,
|
||||
"aspect": "portrait" if w / h < 0.9 else ("landscape" if w / h > 1.1 else "square"),
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser()
|
||||
# doubao only, and that is not laziness. `score_at_scales` reproduces the TOPHAT
|
||||
# front-end, which is the only one with a ladder to widen; scoring it for jimeng or
|
||||
# samsung would measure a code path their detectors never run. Doubao is also the one
|
||||
# mark that declares NO rival, so crossing its NCC gate really is the whole verdict --
|
||||
# for a mark with rivals, `dense_crosses` would ignore the competitive margin and read
|
||||
# optimistically. Widening the binary front-end is a separate experiment.
|
||||
ap.add_argument("--mark", default="doubao", choices=["doubao"])
|
||||
ap.add_argument("--n", type=int, default=4000, help="corpus files to scan")
|
||||
ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 4) - 2))
|
||||
ap.add_argument("--out", type=Path, default=OUT)
|
||||
ap.add_argument("--report-only", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
out = a.out.with_name(f"{a.out.stem}_{a.mark}{a.out.suffix}")
|
||||
if a.report_only:
|
||||
report([json.loads(x) for x in out.read_text(encoding="utf-8").splitlines() if x.strip()], a.mark)
|
||||
return
|
||||
|
||||
pool = glob.glob(str(CORPUS / "*" / "*"))
|
||||
random.Random(19).shuffle(pool) # noqa: S311 -- deterministic sampling, not crypto
|
||||
pool = pool[: a.n]
|
||||
print(f"mark={a.mark} scanning {len(pool)} corpus files workers={a.workers}")
|
||||
print(f"shipped ladder {SHIPPED}\ndense ladder {DENSE}\n", flush=True)
|
||||
|
||||
rows: list[dict[str, Any]] = []
|
||||
out.parent.mkdir(parents=True, exist_ok=True)
|
||||
with open(out, "w", encoding="utf-8") as fh, ProcessPoolExecutor(max_workers=a.workers) as ex:
|
||||
futures = [ex.submit(_one, (p, a.mark)) for p in pool]
|
||||
for i, fut in enumerate(as_completed(futures), 1):
|
||||
try:
|
||||
rec = fut.result()
|
||||
except Exception: # noqa: S112 -- one bad file must not kill the sweep
|
||||
continue
|
||||
if rec is None:
|
||||
continue
|
||||
fh.write(json.dumps(rec) + "\n")
|
||||
rows.append(rec)
|
||||
if i % 500 == 0:
|
||||
fh.flush()
|
||||
pos = sum(1 for r in rows if r["arm"] == "positive")
|
||||
print(f" {i}/{len(pool)} usable={len(rows)} (pos={pos})", flush=True)
|
||||
report(rows, a.mark)
|
||||
|
||||
|
||||
def report(rows: list[dict[str, Any]], mark: str) -> None:
|
||||
pos = [r for r in rows if r["arm"] == "positive"]
|
||||
neg = [r for r in rows if r["arm"] == "negative"]
|
||||
print(f"\n{'=' * 82}\nLADDER HEADROOM mark={mark} positives={len(pos)} verified-clean negatives={len(neg)}")
|
||||
print(f"{'=' * 82}")
|
||||
if not pos and not neg:
|
||||
print("no usable rows")
|
||||
return
|
||||
|
||||
if pos:
|
||||
miss = [r for r in pos if not r["detected_now"]]
|
||||
rec = [r for r in miss if r["dense_crosses"]]
|
||||
now = len(pos) - len(miss)
|
||||
lo, hi = wilson(len(rec), len(miss)) if miss else (0.0, 0.0)
|
||||
print("\nPOSITIVE ARM (images whose METADATA names this vendor -- an upper bound,")
|
||||
print("provenance says the vendor produced the file, not that a mark is visible on it)\n")
|
||||
print(f" detected today {now:5d} / {len(pos)} ({100 * now / len(pos):.1f}%)")
|
||||
print(f" of the {len(miss)} misses, the dense ladder crosses the gate on {len(rec)}")
|
||||
if miss:
|
||||
print(f" ({100 * len(rec) / len(miss):.1f}%, 95% CI {lo:.1f}-{hi:.1f})")
|
||||
after = now + len(rec)
|
||||
print(f" detection would go {100 * now / len(pos):.1f}% -> {100 * after / len(pos):.1f}%")
|
||||
if rec:
|
||||
by_scale = collections.Counter(r["dense_best_scale"] for r in rec)
|
||||
print("\n which rung recovers them (a rung near a SHIPPED one recovering many means")
|
||||
print(" the gain is the finer STEP, not the wider RANGE):")
|
||||
for s, n in sorted(by_scale.items()):
|
||||
near = " <- shipped" if any(abs(s - x) < 0.02 for x in SHIPPED) else ""
|
||||
print(f" scale {s:<6} {n:4d}{near}")
|
||||
by_asp = collections.Counter(r["aspect"] for r in rec)
|
||||
print(f"\n by aspect: {dict(by_asp)}")
|
||||
|
||||
if neg:
|
||||
fires_now = sum(1 for r in neg if r["detected_now"])
|
||||
fires_dense = sum(1 for r in neg if r["dense_crosses"])
|
||||
lo0, hi0 = wilson(fires_now, len(neg))
|
||||
lo1, hi1 = wilson(fires_dense, len(neg))
|
||||
print("\n\nNEGATIVE ARM (no metadata signal; NOT filtered on the detector's own verdict)\n")
|
||||
p0, p1 = 100 * fires_now / len(neg), 100 * fires_dense / len(neg)
|
||||
print(f" false fire today {fires_now:4d} / {len(neg)} {p0:.2f}% (CI {lo0:.2f}-{hi0:.2f})")
|
||||
print(f" false fire, dense ladder {fires_dense:4d} / {len(neg)} {p1:.2f}% (CI {lo1:.2f}-{hi1:.2f})")
|
||||
print("\n This is the price. A denser ladder gives a spurious blob more chances to")
|
||||
print(" match at SOME scale, so the gain above is only real if this line barely moves.")
|
||||
|
||||
ladder_table(pos, neg)
|
||||
|
||||
|
||||
def _crosses(row: dict[str, Any], rungs: tuple[float, ...]) -> bool:
|
||||
"""Would this frame cross its gate on the given ladder, from the stored per-rung scores."""
|
||||
sc = row.get("scores") or {}
|
||||
gate = row["gate"]
|
||||
return any(v >= gate for k, v in sc.items() if float(k) in rungs)
|
||||
|
||||
|
||||
def ladder_table(pos: list[dict[str, Any]], neg: list[dict[str, Any]]) -> None:
|
||||
"""Every candidate ladder side by side: what it recovers against what it costs."""
|
||||
if not (pos and neg) or "scores" not in pos[0]:
|
||||
return # older run without per-rung scores
|
||||
miss = [r for r in pos if not r["detected_now"]]
|
||||
base_fire = sum(1 for r in neg if r["detected_now"])
|
||||
print("\n\nLADDER CANDIDATES -- recovered marks against the false fires they cost\n")
|
||||
print("A candidate is only worth shipping if the recovered column beats the added-fires")
|
||||
print("column by enough to survive the base rates: clean frames vastly outnumber marked")
|
||||
print("ones in real traffic, so a small percentage on the negative arm is a large count.\n")
|
||||
hdr = f"{'ladder':10s} {'rungs':>6s} {'recovered':>10s} {'of misses':>10s}"
|
||||
print(f"{hdr} {'false fire':>11s} {'added':>7s} {'ratio':>7s}")
|
||||
for name, rungs in LADDERS.items():
|
||||
rec = sum(1 for r in miss if _crosses(r, rungs))
|
||||
fire = sum(1 for r in neg if _crosses(r, rungs) or r["detected_now"])
|
||||
added = fire - base_fire
|
||||
ratio = f"{rec / added:.1f}:1" if added > 0 else ("inf" if rec else "-")
|
||||
pct = 100 * rec / len(miss) if miss else 0.0
|
||||
print(f"{name:10s} {len(rungs):6d} {rec:10d} {pct:9.1f}% {100 * fire / len(neg):10.2f}% {added:7d} {ratio:>7s}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -45,6 +45,35 @@ almost nothing and, at any threshold low enough to fire, fires on arbitrary corn
|
||||
|
||||
星绘 additionally has only ONE confirmed example in the corpus, so even a working
|
||||
front-end could not have its threshold calibrated yet.
|
||||
|
||||
UPDATE 2026-07-20: the named blocker is GONE, and the retry is still inconclusive.
|
||||
`detect_frontend="tophat"` (built later, for doubao) is exactly the "grayscale correlation
|
||||
on the raw top-hat" this note asked for, so the 2026-07-18 ruling rests on a premise that
|
||||
no longer holds and must not simply be inherited. Two things were measured against it, and
|
||||
neither settles the question:
|
||||
|
||||
* A GENERIC template of the shared `AI生成` tail -- attractive because GB 45438-2025
|
||||
guarantees that run across vendors, so one template would cover 千问 / 百度 / 星绘 and
|
||||
anything compliant that ships next. Measured on the tophat front-end at the shipped
|
||||
3-rung ladder: a bold 千问 positive scores 0.407 against clean corners at p99 0.298 /
|
||||
max 0.321. It separates on that one frame, but only by a hair, and a 4-glyph template
|
||||
is inherently less specific than a 6-glyph one -- the shorter the run, the more
|
||||
arbitrary corner structure correlates with it.
|
||||
* The FULL 千问 template on the same front-end scores 0.248 against a clean max of 0.537,
|
||||
i.e. no separation at all -- WORSE than the generic tail, which is the opposite of
|
||||
what the specificity argument predicts and is itself a reason to distrust n=1.
|
||||
|
||||
The blocker is now EVIDENCE, not architecture: this session found exactly one 千问 and one
|
||||
百度 positive (both by eyeballing doubao-provenance misses), and the 14 positives quoted
|
||||
above were not preserved anywhere the current scripts can reach. Nothing should be
|
||||
registered off a single frame. What it takes: harvest 30+ confirmed positives per vendor
|
||||
-- `scripts/cjk_tail_probe.py` exists for exactly this, scoring TC260-provenance frames
|
||||
that no detector fires on and writing a contact sheet of the top scorers to label -- then
|
||||
calibrate a gate against the clean arm. Two traps worth knowing before repeating this
|
||||
measurement: score with `alpha_height_frac`, not the silhouette's own aspect ratio (the
|
||||
latter inflated the clean p99 from 0.30 to 0.58 and made every comparison meaningless),
|
||||
and keep the ladder at the shipped 3 rungs, since a wide sweep hands clean corners many
|
||||
extra chances to match.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
Reference in New Issue
Block a user