CalvinBackup/remove-ai-watermarks
1
0
Fork 0
mirror of https://github.com/wiltodelta/remove-ai-watermarks.git synced 2026-08-06 22:18:36 +02:00
Code Issues Packages Projects Releases Wiki Activity

Register the Qwen 千问AI生成 visible text mark

Browse Source 
Calibrated on the 117-frame TC260-producer cohort (vendor_cohort_harvest +
vendor_mark_calibrate, both committed here): per-mark 2-rung ladder
(0.78, 1.27) for the two measured size modes, fitted locate box (the mark
sits ~0.025 of the short side off the edge; doubao's box clipped the first
glyph), measured template aspect 0.26, gate 0.45 (clean p99 0.301).
Strict-only (the sub-gate band is non-Qwen banners), no rival margin
(0 cross-fires on 400 doubao / 298 jimeng / 286 clean frames).
83/83 real marks detector-clean after cv2 fill.

TextMarkConfig gains a per-mark ladder field; the shipped 3-rung default
is unchanged for every other mark.
This commit is contained in:
Victor Kuznetsov
2026-07-21 16:41:16 -07:00
parent 1d93704a4d
commit 008319c6a5
17 changed files with 1400 additions and 75 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+5 -4
View File
File diff suppressed because one or more lines are too long
README.md
+8 -7
View File
@@ -21,13 +21,13 @@ If this tool saves you time, consider [sponsoring its development](https://githu
## Scope
This tool removes **AI-provenance watermarks** that a platform stamps onto content **you generated yourself** — SynthID, the Gemini / Nano Banana sparkle, the Doubao / Jimeng / Samsung visible AI labels, the Chinese TC260 "由…AI生成" label, and C2PA / IPTC / EXIF "Made with AI" metadata. The point is your autonomy over your own output.
This tool removes **AI-provenance watermarks** that a platform stamps onto content **you generated yourself** — SynthID, the Gemini / Nano Banana sparkle, the Doubao / Jimeng / Qwen / Samsung visible AI labels, the Chinese TC260 "由…AI生成" label, and C2PA / IPTC / EXIF "Made with AI" metadata. The point is your autonomy over your own output.
It does **not** target watermarks that protect someone else's paid or copyrighted content — stock-agency overlays (Shutterstock, Getty, iStock, Adobe Stock), classifieds-site marks, or any tiled "preview" watermark whose job is to gate a purchase. Removing those is out of scope by design. `erase` is a generic, user-driven region tool for your own objects, not an automatic stock-watermark remover.
## Features
- **Visible watermark removal** — a registry of known marks in their usual places: the Gemini / Nano Banana sparkle, the Doubao "豆包AI生成" text strip, the Jimeng "★ 即梦AI" wordmark, and the Samsung Galaxy AI "✦ Contenuti generati dall'AI" strip (bottom-left, locale-specific). Each mark is **localized to a footprint mask, then filled**: the engine finds the mark, builds a binary mask over its footprint, and one shared, swappable fill inpaints that region. Choose the fill with `--backend`: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN, light, the memory-tight pick where LaMa will not fit), or `lama` (big-LaMa, best quality, heavier, auto-preferred when a learned backend is available); the default `auto` uses LaMa > MI-GAN > cv2, best available. The localizer is cheap CPU (cv2/numpy), so it runs anywhere; the heavier MI-GAN/LaMa fill is opt-in. Detection keys on each mark's own shape (NCC against a captured silhouette; the alpha captures rebuilt by `scripts/visible_alpha_solve.py` are used to detect and to shape the mask, not for pixel recovery). The visual detector needs no metadata, but a borderline (faint or moved) mark is only trusted with corroboration: `--sensitivity` (default `auto`) relaxes a mark's gate when local metadata confirms the vendor or a same-product sibling mark is found; `strict` never relaxes. There is deliberately no "assume this is AI" mode: asserting that an image is AI says nothing about *which* vendor made it or *where* the mark is, which is exactly what a gate bypass needs. If you can SEE a mark the detector missed, point at it with `erase --region x,y,w,h`, or force a known text mark by name with `--mark <name> --no-detect` -- both act on what you actually see instead of relaxing every detector at once. `visible --mark auto` finds and removes every detected mark in one pass. Fast, offline, no GPU. (For arbitrary logos/objects, see `erase`.)
- **Visible watermark removal** — a registry of known marks in their usual places: the Gemini / Nano Banana sparkle, the Doubao "豆包AI生成" text strip, the Jimeng "★ 即梦AI" wordmark, the Qwen "千问AI生成" text strip, and the Samsung Galaxy AI "✦ Contenuti generati dall'AI" strip (bottom-left, locale-specific). Each mark is **localized to a footprint mask, then filled**: the engine finds the mark, builds a binary mask over its footprint, and one shared, swappable fill inpaints that region. Choose the fill with `--backend`: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN, light, the memory-tight pick where LaMa will not fit), or `lama` (big-LaMa, best quality, heavier, auto-preferred when a learned backend is available); the default `auto` uses LaMa > MI-GAN > cv2, best available. The localizer is cheap CPU (cv2/numpy), so it runs anywhere; the heavier MI-GAN/LaMa fill is opt-in. Detection keys on each mark's own shape (NCC against a captured silhouette; the alpha captures rebuilt by `scripts/visible_alpha_solve.py` are used to detect and to shape the mask, not for pixel recovery). The visual detector needs no metadata, but a borderline (faint or moved) mark is only trusted with corroboration: `--sensitivity` (default `auto`) relaxes a mark's gate when local metadata confirms the vendor or a same-product sibling mark is found; `strict` never relaxes. There is deliberately no "assume this is AI" mode: asserting that an image is AI says nothing about *which* vendor made it or *where* the mark is, which is exactly what a gate bypass needs. If you can SEE a mark the detector missed, point at it with `erase --region x,y,w,h`, or force a known text mark by name with `--mark <name> --no-detect` -- both act on what you actually see instead of relaxing every detector at once. `visible --mark auto` finds and removes every detected mark in one pass. Fast, offline, no GPU. (For arbitrary logos/objects, see `erase`.)
- **Universal region eraser (`erase`)** — remove any logo / watermark / object inside boxes you specify, regardless of position or color. Default cv2 inpainting (CPU, instant); optional big-LaMa via onnxruntime (`lama` extra) for higher quality
- **Invisible watermark removal** — SynthID, StableSignature, TreeRing via diffusion-based regeneration (needs a local GPU, or run it with no setup on [raiw.cc](https://raiw.cc))
- **AI metadata stripping** — EXIF, PNG text chunks, C2PA provenance manifests (PNG / JPEG / AVIF / HEIF / JPEG-XL, **MP4 / MOV / M4V / M4A** at the container level, and **WebM / MP3 / WAV / FLAC / OGG** losslessly via ffmpeg), XMP DigitalSourceType
@@ -36,7 +36,7 @@ It does **not** target watermarks that protect someone else's paid or copyrighte
- **Text and face preservation (default)** — the default pipeline is a canny ControlNet that keeps text and face structure sharp through the removal pass (without copying original pixels, so SynthID is still removed). Use `--pipeline sdxl` for plain SDXL img2img (lighter, no extra model download) on inputs without text or faces. An experimental `--pipeline qwen` runs Qwen-Image (20B, Apache-2.0) img2img, which preserves **text** (including CJK and small text) better than SDXL at equal strength; it is CUDA/cloud-class (does not fit MPS), and its strength floors are not yet certified (pass an explicit `--strength`, especially for Gemini content). Note: measured fidelity (`scripts/fidelity_metrics.py`) shows Qwen wins on text but controlnet preserves **faces** better (Qwen smooths skin more), so Qwen is not a universal upgrade. Canny preserves face *structure*, not *identity* (the regenerated face drifts in likeness). The library does not ship a face-restore extra: every approach evaluated (GFPGAN-on-cleaned, PhotoMaker-V2, InstantID txt2img, InstantID img2img-on-cleaned) regenerated the face via SDXL and made the output look more AI-generated than the cleaned image. The cleaned controlnet output is the least-AI face state achievable without re-introducing SynthID.
- **Batch processing** — process entire directories
- **Detection** — three-stage NCC watermark detection with confidence scoring
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP, PNG chunk, EXIF, or JPEG segment), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the C2PA cloud-manifest reference (Adobe Durable Content Credentials, when the embedded manifest is stripped), the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" / Samsung Galaxy AI "Contenuti generati dall'AI" text marks), the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
- **Provenance detection (`identify`)** — aggregate C2PA issuer, the C2PA soft-binding forensic-watermark vendor (Adobe TrustMark, Digimarc, Imatag, ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, embedded SD/ComfyUI params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the China TC260 AIGC label (XMP, PNG chunk, EXIF, or JPEG segment), the HuggingFace `hf-job-id` job marker, the SynthID metadata proxy, the C2PA cloud-manifest reference (Adobe Durable Content Credentials, when the embedded manifest is stripped), the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" / Qwen "千问AI生成" / Samsung Galaxy AI "Contenuti generati dall'AI" text marks), the open SD/SDXL/FLUX invisible watermark, and (with the `trustmark` extra) the open Adobe TrustMark watermark into one origin-platform + watermark-inventory verdict (`--json` for machine output)
## Examples
@@ -61,14 +61,15 @@ It does **not** target watermarks that protect someone else's paid or copyrighte
| **Meta AI** | — | — | ✅ IPTC "Made with AI" (digitalSourceType) | Metadata strip (removes the label) |
| **Doubao** (ByteDance) / China AIGC generators | ✅ "豆包AI生成" text strip (bottom-right) | — | ✅ TC260 AIGC label (`<TC260:AIGC>` XMP, `AIGC` PNG chunk, or EXIF JSON) **+ C2PA** signed by ByteDance Volcano Engine (`volcengine`) | Localize glyph footprint + fill + metadata strip |
| **Jimeng / Dreamina** (即梦AI, ByteDance) | ✅ "★ 即梦AI" wordmark (bottom-right) | — | ✅ TC260 AIGC label + C2PA (Volcano Engine) | Localize glyph footprint + fill + metadata strip |
| **Qwen / Tongyi Qianwen** (千问, Alibaba) | ✅ "千问AI生成" text strip (bottom-right) | — | ✅ TC260 AIGC label | Localize glyph footprint + fill + metadata strip |
| **Samsung Galaxy AI** (Generative Edit, Sketch to Image, ...) | ✅ "✦ Contenuti generati dall'AI" strip (bottom-left, Italian-locale detection) | — | ✅ C2PA (signer "Samsung Galaxy") + `trainedAlgorithmicMedia` / proprietary `genAIType` marker | Localize glyph footprint + fill + metadata strip |
| **Black Forest Labs** (FLUX API) | — | — | ✅ C2PA (`Black Forest Labs API` + `c2pa.ai_generated_content` + `trainedAlgorithmicMedia`) | Metadata strip |
| **StableSignature** (Meta) | — | ✅ In-model watermark | — | Diffusion regeneration |
| **TreeRing** | — | ✅ Latent space watermark | — | Diffusion regeneration |
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo), by ByteDance's Doubao ("豆包AI生成" corner text) and Jimeng / Dreamina ("★ 即梦AI" wordmark), and by Samsung Galaxy AI ("✦ Contenuti generati dall'AI" strip, bottom-left, locale-specific). All are removed by localizing the mark to a footprint mask and inpainting it with one shared fill (cv2 by default, MI-GAN or big-LaMa via `--backend`); the localizer is CPU-cheap and the heavier fills are opt-in. Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any color), use the universal `erase --region` command.
> Visible overlays are used by Google Gemini / Nano Banana (sparkle logo), by ByteDance's Doubao ("豆包AI生成" corner text) and Jimeng / Dreamina ("★ 即梦AI" wordmark), by Alibaba's Qwen / Tongyi Qianwen ("千问AI生成" corner text), and by Samsung Galaxy AI ("✦ Contenuti generati dall'AI" strip, bottom-left, locale-specific). All are removed by localizing the mark to a footprint mask and inpainting it with one shared fill (cv2 by default, MI-GAN or big-LaMa via `--backend`); the localizer is CPU-cheap and the heavier fills are opt-in. Other services rely on invisible watermarks and/or metadata; our diffusion-based regeneration works against any invisible watermark in pixel or frequency domain. For a visible mark from any other source (any position, any color), use the universal `erase --region` command.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP, PNG chunk, EXIF, or JPEG segment), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the C2PA cloud-manifest reference (Adobe Durable Content Credentials, when the embedded manifest is stripped), the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" / Samsung Galaxy AI "Contenuti generati dall'AI" text marks), and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
> **Detection:** `remove-ai-watermarks identify <image>` reports the origin platform and watermark inventory for all the signals above — C2PA issuer, the C2PA soft-binding forensic-watermark vendor (TrustMark / Digimarc / Imatag / ...), IPTC "Made with AI" plus the IPTC 2025.1 `AISystemUsed` field, the China TC260 AIGC label (XMP, PNG chunk, EXIF, or JPEG segment), the HuggingFace `hf-job-id` job marker, embedded generation params, EXIF/XMP generator tags, the xAI/Grok EXIF signature, the SynthID metadata proxy, the C2PA cloud-manifest reference (Adobe Durable Content Credentials, when the embedded manifest is stripped), the visible marks (Gemini sparkle plus the Doubao "豆包AI生成" / Jimeng "即梦AI" / Qwen "千问AI生成" / Samsung Galaxy AI "Contenuti generati dall'AI" text marks), and (with the `[detect]` / `[trustmark]` extras) the open SD/SDXL/FLUX and Adobe TrustMark invisible watermarks. SynthID and the proprietary soft-binding watermarks (Digimarc etc.) have no local decoder, so they are reported by metadata proxy / vendor name only.
## How it works
@@ -332,7 +333,7 @@ remove-ai-watermarks identify image.png
# Visible watermark only — fast, offline, CPU. --mark auto (default) removes every
# detected known mark (Gemini sparkle / Doubao "豆包AI生成" / Jimeng "即梦AI" /
# Samsung Galaxy AI "Contenuti generati dall'AI"); force one with
# --mark gemini / doubao / jimeng / samsung. Removal localizes each mark to a
# --mark gemini / doubao / jimeng / qwen / samsung. Removal localizes each mark to a
# footprint mask and inpaints it with a shared fill; --backend auto|cv2|migan|lama
# (default auto) picks the fill (auto = LaMa > MI-GAN > cv2, best available).
# --sensitivity auto|strict (default auto) sets how hard a borderline mark is
@@ -393,7 +394,7 @@ remove-ai-watermarks batch ./images/ --mode all
### Python API
One high-level call removes every detected visible mark (Gemini sparkle, Doubao / Jimeng / Samsung text, the Jimeng pill) by localize then fill. For a file it reads metadata provenance automatically and preserves the alpha channel; `import remove_ai_watermarks` stays cheap (the heavy deps load lazily on first use).
One high-level call removes every detected visible mark (Gemini sparkle, Doubao / Jimeng / Qwen / Samsung text, the Jimeng pill) by localize then fill. For a file it reads metadata provenance automatically and preserves the alpha channel; `import remove_ai_watermarks` stays cheap (the heavy deps load lazily on first use).
```python
import remove_ai_watermarks as raiw
docs/module-internals.md
+26 -8
View File
@@ -29,7 +29,7 @@ module.
**AI-generated vs AI-enhanced** (`ProvenanceReport.ai_source_kind`, roadmap item): the C2PA digital-source-type is split into `"generated"` (trainedAlgorithmicMedia, fully synthetic) vs `"enhanced"` (compositeWithTrainedAlgorithmicMedia, a real photo with an AI-composited region) — the two byte strings are unambiguous (`compositeWithTrainedAlgorithmicMedia` capitalizes the inner "Trained", so a lowercase `trainedAlgorithmicMedia` match is standalone full generation; full generation wins when both appear). `ai_source_kind` is set only when the AI verdict actually came from the C2PA source type (a non-C2PA AI signal — IPTC/AIGC/local gen/xAI — leaves it None). It lets a caller branch a full-frame scrub (`generated`) from a region-targeted clean that preserves the real photo (`enhanced`; see `noai/tiling.feather_region_composite`). The CLI verdict line reads "AI-generated (fully synthetic)" vs "AI-enhanced (real content with an AI-composited region)".
**Visible-mark detection** (`check_visible`, signals `visible_sparkle` / `visible_doubao` / `visible_jimeng` / `visible_samsung`): the Gemini sparkle keeps its own file-level path (`_visible_sparkle` → `gemini_engine.detect_sparkle_confidence`, promoted only at confidence ≥ `_SPARKLE_THRESHOLD`, which is the SHARED `watermark_registry.GEMINI_SPARKLE_TRUST_CONF` (0.5) — imported, not a private copy, so the provenance detect threshold and the removal `detect_marks` / `_gemini_detect` arbitration gate can never drift (the detect-vs-remove desync from roadmap P0#7; regression-guarded by `tests/test_identify.py::TestSparkleDetectRemoveAlignment`, which composites the real demo sparkle at borderline opacities and asserts identify and `detect_marks` AGREE on either side of the line). Lowering the gate to recover faint sub-0.5 sparkles was evaluated 2026-06-20 and REJECTED: a real Doubao text mark scores ~0.40-0.42 as a gemini match with a HIGHER core-ring brightness margin than a genuine faint sparkle, so neither confidence nor the brightness gate separates them in the [0.35, 0.5) band — lowering trades a rare miss for false-positive removals on clean images. Corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49), while Doubao/Jimeng/Samsung reuse the registry detectors (`_visible_text_marks` → `watermark_registry`, iterating `_VISIBLE_MARK_PLATFORM`), each gated by its own engine NCC threshold via `MarkDetection.detected` (Doubao 0.4, Jimeng 0.45, Samsung 0.4). Doubao/Jimeng are normally also caught by the TC260 AIGC metadata label and Samsung by its C2PA + `genAIType` marker, so the visible path is their stripped-metadata fallback. Visible marks set `platform` only when no harder signal already did, and (like the sparkle) are excluded from integrity-clash vendor claims. The cv2 dependency lives in the engines, not here.
**Visible-mark detection** (`check_visible`, signals `visible_sparkle` / `visible_doubao` / `visible_jimeng` / `visible_qwen` / `visible_samsung`): the Gemini sparkle keeps its own file-level path (`_visible_sparkle` → `gemini_engine.detect_sparkle_confidence`, promoted only at confidence ≥ `_SPARKLE_THRESHOLD`, which is the SHARED `watermark_registry.GEMINI_SPARKLE_TRUST_CONF` (0.5) — imported, not a private copy, so the provenance detect threshold and the removal `detect_marks` / `_gemini_detect` arbitration gate can never drift (the detect-vs-remove desync from roadmap P0#7; regression-guarded by `tests/test_identify.py::TestSparkleDetectRemoveAlignment`, which composites the real demo sparkle at borderline opacities and asserts identify and `detect_marks` AGREE on either side of the line). Lowering the gate to recover faint sub-0.5 sparkles was evaluated 2026-06-20 and REJECTED: a real Doubao text mark scores ~0.40-0.42 as a gemini match with a HIGHER core-ring brightness margin than a genuine faint sparkle, so neither confidence nor the brightness gate separates them in the [0.35, 0.5) band — lowering trades a rare miss for false-positive removals on clean images. Corpus-tuned to separate Gemini sparkles ≥0.56 from non-sparkle ≤0.49), while Doubao/Jimeng/Qwen/Samsung reuse the registry detectors (`_visible_text_marks` → `watermark_registry`, iterating `_VISIBLE_MARK_PLATFORM`), each gated by its own engine NCC threshold via `MarkDetection.detected` (Doubao 0.50, Jimeng 0.45, Qwen 0.45, Samsung 0.40). Doubao/Jimeng are normally also caught by the TC260 AIGC metadata label and Samsung by its C2PA + `genAIType` marker, so the visible path is their stripped-metadata fallback. Visible marks set `platform` only when no harder signal already did, and (like the sparkle) are excluded from integrity-clash vendor claims. The cv2 dependency lives in the engines, not here.
**`import identify` is deliberately light** (~26 MB; ~36 MB with cv2 loaded by a visible-mark run, ~106 MB for a full `check_visible` run): it imports the `noai.c2pa`/`noai.constants` submodules, and `noai/__init__` is lazy (see "Test and lint"), so torch/diffusers are NOT pulled at import even in a full `gpu`/`detect` install — fits a 512 MB host. `noai.c2pa` does eagerly import the **c2pa-python** binary (Rust + cryptography, ~+5 MB RSS, no torch) for the primary `Reader` path — light enough to stay on the dependency-light host; a broken/absent wheel degrades to the byte-scan parser (`reader_available()` False). The heavy paths are opt-in: `check_invisible=True` needs the `detect`/`trustmark` extras (each pulls **torch**; TrustMark also **downloads weights**), so on a core-only deploy leave `check_invisible` off (it is a no-op there anyway). Before the lazy `__init__`, the mere presence of torch in the env inflated `import identify` to ~420 MB.
@@ -61,7 +61,7 @@ module.
`watermark_registry.py` — **single catalog of known visible watermarks**, the unified "find known marks in their usual places, recognize, remove" entry.
**Localize -> fill by policy (replaced reverse-alpha):** each mark is localized to a binary full-frame footprint mask (a `Localization`), and one shared, swappable fill inpaints that mask via `fill(image, mask, backend=...)` (delegates to `region_eraser.erase`). This replaced the old reverse-alpha removal (invert a captured alpha map, `original = (wm - a*logo)/(1-a)`, plus a thin residual inpaint) for ALL marks — gemini, doubao, jimeng, samsung, and jimeng_pill. **Why it changed:** reverse-alpha depended on a fixed captured alpha map at a fixed position, so it broke whenever a vendor moved or re-rendered its mark; and it was not color-lossless even with the right map (it amplifies 8-bit quantization and JPEG-chroma error by `1/(1-a)`), which showed up as "the color just changed, not removed" reports. Localize -> fill has a benign failure mode: a slightly-off localization just inpaints a small region near-losslessly instead of leaving a color-shifted smear. The captured alpha maps are still used to DETECT the marks and to shape the mask (gemini's footprint), but NOT for pixel recovery. Fill backends: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN ONNX, light, the memory-tight pick where LaMa will not fit), `lama` (big-LaMa ONNX, best quality, heavier, auto-preferred when a learned backend is available); `auto` = LaMa > MI-GAN > cv2, best available. Each `KnownMark` ties a key to {usual `location`, `in_auto` flag, a `_detect` callable → uniform `MarkDetection`, a `_mask` callable → full-frame footprint mask}; `KnownMark.remove(image, *, backend="auto", provenance=False, force=False)`. Entries today: `gemini` (bottom-right sparkle), `doubao` (bottom-right "豆包AI生成"), `jimeng` (bottom-right "★ 即梦AI"), `samsung` (bottom-**LEFT** "✦ Contenuti generati dall'AI", Samsung Galaxy AI, Italian locale), and the capture-less `jimeng_pill` (top-left "AI生成"). `detect_marks(image, *, provenance=frozenset())` scans all (strict, for the identify verdict); `remove_auto_marks(image, *, sensitivity="auto", provenance=frozenset(), backend="auto")` removes every detected mark in one pass. **Sensitivity (`auto`/`strict`/`assume_ai`)** decides how hard a borderline mark is trusted: the visual detectors are pixel-based (no metadata needed) and the recall gain comes from relaxing the false-positive gate, not from metadata. `resolve_trust` turns the policy + evidence into the per-mark trust level the engines consume as `provenance = level != "strict"` — `strict` never relaxes; `auto` relaxes only on same-product evidence (metadata provenance for that vendor, or a confidently strict-detected sibling of the same `_PRODUCT_OF` — Doubao and Jimeng are both bottom-right ByteDance but distinct products, so they do NOT cross-relax); `assume_ai` relaxes every mark (the caller asserts AI, e.g. a metadata-stripped screenshot). **Three levels, not two: `strict` / `assumed` / `confirmed`.** Relaxing bypasses the engine's false-positive gate outright, and that bypass is contracted to mean the vendor is CONFIRMED (`GeminiEngine.detect_watermark`'s `trust_provenance`: "external metadata already proves this is a Google generation"). An `assume_ai` caller asserts the image is AI, which says nothing about WHICH vendor, so a mark relaxed on assumption alone must also clear `_ASSUMED_CONF_FLOOR` (`assumed_floor_ok`; gemini 0.50) — see "Assumed-trust confidence floor" below. **Perception / decision / action are separated three ways** (the removal path only; `identify` keeps calling `KnownMark.detect` directly, so its verdict is untouched): `_build_candidates(image)` is PERCEPTION — it runs each detector at both trust levels and packages raw verdicts + the pill's flatness feature into `Candidate`s, no policy; `decide(candidates, Context(sensitivity, provenance)) -> [Decision]` is the pure DECISION arbiter — all keep/drop policy (`resolve_trust` cross-mark corroboration + the assumed-trust floor + the pill gate) in one image-free, unit-testable function (`tests/test_watermark_registry.py::TestArbiter`); then `remove_auto_marks` does the ACTION, localizing -> filling each winner. The Gemini FP gate deliberately stays inside `gemini_engine` (not the arbiter) because `identify` reads that same gated confidence — pulling it out would drift the provenance verdict. Behavior was byte-identical to the pre-arbiter two-pass when the arbiter landed; `assume_ai` has since gained the assumed-trust confidence floor (see below), which deliberately changes its verdict on weak gate-bypassed matches.
**Localize -> fill by policy (replaced reverse-alpha):** each mark is localized to a binary full-frame footprint mask (a `Localization`), and one shared, swappable fill inpaints that mask via `fill(image, mask, backend=...)` (delegates to `region_eraser.erase`). This replaced the old reverse-alpha removal (invert a captured alpha map, `original = (wm - a*logo)/(1-a)`, plus a thin residual inpaint) for ALL marks — gemini, doubao, jimeng, samsung, and jimeng_pill. **Why it changed:** reverse-alpha depended on a fixed captured alpha map at a fixed position, so it broke whenever a vendor moved or re-rendered its mark; and it was not color-lossless even with the right map (it amplifies 8-bit quantization and JPEG-chroma error by `1/(1-a)`), which showed up as "the color just changed, not removed" reports. Localize -> fill has a benign failure mode: a slightly-off localization just inpaints a small region near-losslessly instead of leaving a color-shifted smear. The captured alpha maps are still used to DETECT the marks and to shape the mask (gemini's footprint), but NOT for pixel recovery. Fill backends: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN ONNX, light, the memory-tight pick where LaMa will not fit), `lama` (big-LaMa ONNX, best quality, heavier, auto-preferred when a learned backend is available); `auto` = LaMa > MI-GAN > cv2, best available. Each `KnownMark` ties a key to {usual `location`, `in_auto` flag, a `_detect` callable → uniform `MarkDetection`, a `_mask` callable → full-frame footprint mask}; `KnownMark.remove(image, *, backend="auto", provenance=False, force=False)`. Entries today: `gemini` (bottom-right sparkle), `doubao` (bottom-right "豆包AI生成"), `jimeng` (bottom-right "★ 即梦AI"), `qwen` (bottom-right "千问AI生成", Alibaba Tongyi Qianwen), `samsung` (bottom-**LEFT** "✦ Contenuti generati dall'AI", Samsung Galaxy AI, Italian locale), and the capture-less `jimeng_pill` (top-left "AI生成"). `detect_marks(image, *, provenance=frozenset())` scans all (strict, for the identify verdict); `remove_auto_marks(image, *, sensitivity="auto", provenance=frozenset(), backend="auto")` removes every detected mark in one pass. **Sensitivity (`auto`/`strict`/`assume_ai`)** decides how hard a borderline mark is trusted: the visual detectors are pixel-based (no metadata needed) and the recall gain comes from relaxing the false-positive gate, not from metadata. `resolve_trust` turns the policy + evidence into the per-mark trust level the engines consume as `provenance = level != "strict"` — `strict` never relaxes; `auto` relaxes only on same-product evidence (metadata provenance for that vendor, or a confidently strict-detected sibling of the same `_PRODUCT_OF` — Doubao and Jimeng are both bottom-right ByteDance but distinct products, so they do NOT cross-relax); `assume_ai` relaxes every mark (the caller asserts AI, e.g. a metadata-stripped screenshot). **Three levels, not two: `strict` / `assumed` / `confirmed`.** Relaxing bypasses the engine's false-positive gate outright, and that bypass is contracted to mean the vendor is CONFIRMED (`GeminiEngine.detect_watermark`'s `trust_provenance`: "external metadata already proves this is a Google generation"). An `assume_ai` caller asserts the image is AI, which says nothing about WHICH vendor, so a mark relaxed on assumption alone must also clear `_ASSUMED_CONF_FLOOR` (`assumed_floor_ok`; gemini 0.50) — see "Assumed-trust confidence floor" below. **Perception / decision / action are separated three ways** (the removal path only; `identify` keeps calling `KnownMark.detect` directly, so its verdict is untouched): `_build_candidates(image)` is PERCEPTION — it runs each detector at both trust levels and packages raw verdicts + the pill's flatness feature into `Candidate`s, no policy; `decide(candidates, Context(sensitivity, provenance)) -> [Decision]` is the pure DECISION arbiter — all keep/drop policy (`resolve_trust` cross-mark corroboration + the assumed-trust floor + the pill gate) in one image-free, unit-testable function (`tests/test_watermark_registry.py::TestArbiter`); then `remove_auto_marks` does the ACTION, localizing -> filling each winner. The Gemini FP gate deliberately stays inside `gemini_engine` (not the arbiter) because `identify` reads that same gated confidence — pulling it out would drift the provenance verdict. Behavior was byte-identical to the pre-arbiter two-pass when the arbiter landed; `assume_ai` has since gained the assumed-trust confidence floor (see below), which deliberately changes its verdict on weak gate-bypassed matches.
**Head-to-head validation (v0.12.1 reverse-alpha vs the current localize -> fill):** run over the full labelled visible-mark set, with the cv2 / MI-GAN / LaMa fills each compared against the old reverse-alpha. **doubao and jimeng are identical** across every backend -- 100% coverage and 100% clearance either way. **gemini** strict coverage is a few points below reverse-alpha's (the deliberate false-positive tightening), but the metadata-stripped faint ones are now mostly recovered by the DEFAULT white-core rescue in the FP gate (`gemini_engine`: a bright near-WHITE core distinguishes a real faint sparkle from a colored bright corner -- ~14/20 recovered at ~1.25% clean false-fire; a learned classifier on the same features measured worse, 2026-07 tier-1), the residual under `assume_ai`; clearance is equal (~98% both), and neither version touches pixels outside the mark box (outside-box PSNR ~99). **Clearance is fill-independent** -- cv2, MI-GAN and LaMa all strip the mark's shape equally, so the re-detect metric does not separate them; the difference is purely the *visual fill quality* on the recovered region, and it is background-dependent. reverse-alpha recovered textured and especially regular/structured backgrounds (a lattice, a grid) more cleanly than any inpaint; **LaMa closes most of that gap** (the best learned backend), **MI-GAN can ghost or hallucinate structure**, and **cv2 smears** (the last-resort floor). This is why `auto` resolves `LaMa > MI-GAN > cv2` (`preferred_inpaint_backend`) and warns once on the cv2 fallback; on flat backgrounds every backend is clean.
@@ -94,9 +94,9 @@ Doubao is switched to it; jimeng and samsung stay `binary` until measured, becau
**The removal MASK must ride the same front-end, and how it does so was fixed twice.** `tophat` detection does not binarize, but `extract_mask` (which bounds the fill) still does, so a mark faint enough to be found only by the continuous response produced an EMPTY binary blob: `localize` returned `mask=None`, `remove()` was a silent no-op, and `identify` reported `visible_doubao` while `visible` said "no visible mark" on the same file (corpus-measured 2026-07-20: 57 of 60 sampled still-detected Doubao marks untouched, ~8% of its detections). The FIRST fallback (2026-07-19) thresholded the continuous response and took the bounding box of everything above the level -- but the level was `0.5` compared against the max-normalized **uint8 0..255** response, so it selected every non-zero pixel and filled ~120% of the corner box on textured frames (a padded whole-ROI box). It passed parity (a mask that fills everything is trivially detector-clean) and its regression test (a FLAT fixture, where the response is non-zero only on the glyph, so every threshold yields the same box). The SECOND fallback (2026-07-20) uses the detector's OWN best-match box instead: `_tophat_score` was split into **`_tophat_best(image, loc) -> (score, box)`**, the single method whose score gates detection and whose argmax box bounds the mask -- so the two cannot drift by construction, which is how the mismatch arose in the first place. Measured over 14 real faint-path frames (cv2 fill, detector re-run after): the match box fills a **58.7%**-median corner box vs the threshold's **120.9%**, both 100% detector-clean. The largest-connected-component alternative was tighter (10.5%) but removed the mark on only 21% of frames, so it does not cover it and was rejected. Regression: `tests/test_text_mark_faint_mask.py`, whose fixture now carries texture (the flatness of the old one is exactly why it could not see the threshold bug -- mutating the constant to 99.0 left it green). **Any future front-end change must move both the detection and the mask path, or re-check this.**
**What this does NOT solve: vendor ATTRIBUTION for the shared-suffix marks.** With the continuous front-end 千问 becomes separable from clean corners (AUC 0.92) but NOT from Doubao (**AUC 0.41-0.59, i.e. a coin flip**), because "千问AI生成" and "豆包AI生成" share the `AI生成` tail -- three of five glyphs, same face, same corner. So the front-end removes the *detection* blocker and exposes an *attribution* one. Since removal is identical for either (localize the glyph blob -> fill), the natural next design is a GENERIC "CJK AI-generation text mark" detector covering 千问/百度/星绘/小云雀/TRAE and any future GB 45438-2025-compliant vendor in one template, with per-vendor attribution treated as optional metadata rather than a detection requirement -- the standard mandates that every compliant string contain (人工智能|AI) and (生成|合成), so the shared tail is guaranteed. That needs its own precision-labelling round before it ships.
**Vendor ATTRIBUTION for the shared-suffix marks -- solved for 千问 by exact sizing, not by a generic template.** The 2026-07-18 worry was that "千问AI生成" and "豆包AI生成" share the `AI生成` tail (four of six glyph cells), so their templates would cross-fire. That held while Qwen was scored at the WRONG SIZE (AUC 0.41-0.59, a coin flip). Re-measured 2026-07-21 at the fitted geometry on real pools (`scripts/vendor_mark_calibrate.py --crossfire`): Qwen's template scores p50 0.224 / p90 0.242 on 400 Doubao-marked frames against a 0.45 gate -- **0 cross-fires**, and 0 on 298 Jimeng-marked frames and 286 clean frames -- because an exact-size 6-glyph template is specific enough that the 2-glyph prefix mismatch dominates the shared tail. Qwen was therefore registered WITHOUT a rival margin (a 0.10 margin would have suppressed ~10% of genuine Qwen detections, whose margin p10 sits at 0.00). A GENERIC shared-tail template remains a harvesting aid, not a detector (measured: 0.407 on a bold positive vs clean p99 0.298).
**Why 千问 / 星绘 are NOT registered (measured 2026-07-18).** Adding a text mark is documented as "a `TextMarkConfig` + a thin subclass + one registry row", and that is true only when the mark is stamped like Doubao's. It does not hold for a FAINT mark, and 千问 is the counter-example. Measured on 14 hand-verified corpus positives, same pipeline, each mark scored with its OWN template:
**千问 is registered since 2026-07-21; 星绘 is NOT (still one confirmed example).** The 2026-07-18 measurement below is why 千问 was originally held out -- it is kept because it records the failure CLASS that the registration then had to solve. Measured back then on 14 hand-verified corpus positives, same pipeline, each mark scored with its OWN template:
| mark | n | mean NCC | median | above the 0.40 gate |
|---|---|---|---|---|
@@ -108,7 +108,7 @@ Three candidate explanations were ruled out in order, each by measurement:
2. **Not the morphology kernel.** `MORPH_OPEN`/`MORPH_CLOSE` use fixed 5px kernels regardless of mark size (~9% of a 57px-tall box, ~2.7% of a 188px one). Scaling them with the box height gained +0.014 mean and moved nothing across the gate.
3. **Not the appearance thresholds.** Sweeping `tophat_delta` / `logo_min_luma` / kernel size peaked at mean 0.35 with 4/14 over the gate.
The blocker is SEGMENTATION on a faint mark. Doubao is stamped bold and opaque so the white top-hat returns a clean glyph blob; the Qwen mark is a thin translucent overlay that shatters into specks, and no template can match a blob that is not there. **So the registry's cheap-to-add promise is conditional on mark contrast, and that condition should be checked before promising a new mark.** Adding 千问 needs a detection front-end that does not binarize the glyph first -- grayscale/edge correlation on the raw top-hat, or a learned patch classifier -- not a new silhouette. 星绘 additionally has only ONE confirmed corpus example, so even a working front-end could not calibrate its threshold yet. The synthetic renderer and the full evidence chain are kept in `scripts/render_vendor_silhouettes.py`; researched vendor specs are in `docs/watermarking-landscape.md`.
The blocker was named as SEGMENTATION on a faint mark -- and the `tophat` front-end (built for Doubao the same week) removed exactly that blocker, yet 千问 still did not register, because the real residual was never segmentation alone: it was **mis-sized geometry** (two size modes the shared 3-rung ladder cannot straddle, plus a locate box that clipped the first glyph). The 2026-07-21 cohort harvest (117 labelled frames) and the full calibration chain are in the `qwen_engine.py` section below and `docs/verification-plan.md`. 星绘 remains where it was: ONE confirmed corpus example, so nothing to calibrate a gate against -- do not register it off a single frame. The synthetic renderer and the evidence chain are kept in `scripts/render_vendor_silhouettes.py`; researched vendor specs are in `docs/watermarking-landscape.md`.
**RECALL, measured at last (unbiased random sample, 2026-07-18).** Every earlier round sampled where detectors FIRED, so recall was structurally unmeasurable. This round draws 240 images at RANDOM within each provenance class (160 TC260, 80 Google-C2PA) and labels them EXHAUSTIVELY -- both corners shown at native scale, so a missed mark is visible as a miss rather than absent from the data. Build it with `scripts/visible_recall_sample.py`; labels live in the gitignored research dir.
@@ -187,7 +187,7 @@ The factor is now a per-mark `TextMarkConfig.provenance_ncc_factor`. Doubao stay
**Cross-engine confidences aren't directly comparable**, so the gemini adapter applies the corpus-validated 0.5 sparkle threshold (`_GEMINI_AUTO_MIN_CONF`) for its `detected` flag (lowered to 0.35 under the Google/Gemini provenance prior) — otherwise the gemini engine's loose internal threshold weakly fires (~0.36) on the Doubao text and hijacks `auto`. The shape-keyed Doubao/Jimeng/Samsung NCC detectors don't cross-fire (jimeng scores ~0.22 on the Doubao strip, well under its 0.45 threshold; Samsung is bottom-left so it shares no corner with the others, and scored 0.0 on Doubao/Jimeng captures and they 0.0 on a real Samsung photo), so `auto` picks the right one. `cli.cmd_visible` is registry-driven: `--mark auto` → `remove_auto_marks` (removes every detected mark), `--mark <key>` → that mark; `--mark` choices come from `mark_keys()`.
**`cli._remove_visible_auto` is the shared visible-removal helper used by `cmd_all`/`cmd_batch` too** (they no longer hardcode `GeminiEngine`), so `all`/`batch` remove Doubao/Jimeng/Samsung text marks, not just the Gemini sparkle (regression-guarded by `test_all_visible_step_uses_registry`). The three text-mark adapters were consolidated 2026-06-09: a single `_text_mark(key, label, location)` builds the registry row from one parameterized `_text_mark_detect`/`_text_mark_remove` pair (the remove adapter localizes the glyph footprint and hands it to the shared `fill` only when detected/forced, else skipped); the gemini adapters stay bespoke. Add a new visible mark = one `_text_mark(...)` row + its `TextMarkConfig` (with a captured alpha map for the detection silhouette); do not re-add per-mark `if` branches or copy-paste adapters.
**`cli._remove_visible_auto` is the shared visible-removal helper used by `cmd_all`/`cmd_batch` too** (they no longer hardcode `GeminiEngine`), so `all`/`batch` remove Doubao/Jimeng/Qwen/Samsung text marks, not just the Gemini sparkle (regression-guarded by `test_all_visible_step_uses_registry`). The three text-mark adapters were consolidated 2026-06-09: a single `_text_mark(key, label, location)` builds the registry row from one parameterized `_text_mark_detect`/`_text_mark_remove` pair (the remove adapter localizes the glyph footprint and hands it to the shared `fill` only when detected/forced, else skipped); the gemini adapters stay bespoke. Add a new visible mark = one `_text_mark(...)` row + its `TextMarkConfig` (with a captured alpha map for the detection silhouette); do not re-add per-mark `if` branches or copy-paste adapters.
**Alpha-on-save policy (issue #30):** `image_io.write_bgr_with_alpha` (it lives in `image_io`, not `cli` — moved so the CLI and the library `api` share ONE implementation) rejoins the input's alpha plane **unchanged** — it must NOT zero alpha in the watermark bbox. The fill reconstructs real pixels there, so zeroing alpha punched a transparent hole that renders as a solid **white box** on any non-transparent viewer (Gemini app exports are opaque RGBA, so every user hit it; regression-guarded by `test_visible_keeps_alpha_opaque_in_watermark_region`). The registry `remove()` still returns its region, but the CLI no longer uses it to clear alpha. **It returns `imwrite`'s success flag and callers must check it** (2026-07-20): `imwrite` is contractually non-raising, so that bool is the only signal the file was not created. The wrapper previously returned `None` and swallowed it, so every CLI write site ran `output.stat()` to report the size and a read-only destination died with a bare `FileNotFoundError` traceback pointing at the stat instead of the write. The CLI now writes through the shared `cli._write_output_or_exit`. Regression: `tests/test_cli_robustness.py::TestFailedWriteIsReported`.
@@ -215,7 +215,7 @@ The cost (mislabel ~8-33% of non-Gemini content as Gemini) outweighs the benefit
## `_text_mark_engine.py`
`_text_mark_engine.py` — **shared base for the three text-mark engines (Doubao/Jimeng/Samsung), extracted 2026-06-09** (they were ~90% byte-identical clones). `TextMarkEngine(config: TextMarkConfig)` owns the `locate → extract_mask → detect` detection pipeline plus the removal that localizes the glyph blob to a footprint mask and hands it to the shared `watermark_registry.fill` (+ the asset-keyed `load_alpha_template`/`glyph_silhouette`/`template_match_score` caches). Detection still matches the glyph silhouette (NCC against the captured template); the removal MASK is TEMPLATE-FREE — it is the bounding box of the top-hat glyph blob from `extract_mask`, filled solid + dilated, so a re-rendered or differently-placed mark is still masked. This dropped the fixed alpha-template placement; the captured alpha maps are now used only for the detection silhouette, not for removal. Each engine module is a thin subclass supplying only its `TextMarkConfig` (the tuned constants, the bundled asset, and the bounded structural deltas — `corner` br/bl, `margin_floor` 4/2, `morph_open_size` 5/3, `min_gw` 8/16) plus the test-facing module shims (`_alpha_template`/`_glyph_silhouette`/`_template_match_score` + the constants). Gemini stays a SEPARATE engine (its multi-size fixed-slot sparkle model is genuinely different). Add a new text mark = a new `TextMarkConfig` + a thin subclass + one registry `_text_mark(...)` row. The engine bullets below describe each mark's calibration history; the LOGIC lives here. **Small-image detection guard (`_MIN_DETECT_SHORT_SIDE` 200, added 2026-06-26):** `detect` returns not-detected when the image short side is below 200px. Below that the glyph template degrades to the `min_gw` floor (~8px) and `TM_CCOEFF_NORMED` on a few pixels is noise, so an unrelated small geometric shape can spuriously correlate with the CJK silhouette — a 48×48 app-icon chevron scored Doubao 0.41 / Jimeng 0.47 (both above their thresholds), a pure small-size artifact (the same icon upscaled collapses to ~0.06–0.10 NCC at ≥256px). A real AI-generation label is stamped on a full-resolution render (the captured samples are 1086–2048px wide, the smallest positive test image is 1086px), so the floor sits far below any genuine mark while killing the icon/thumbnail band (≤96px); `identify` falls back to "unknown" (the safe default) and removal, gated on detection, is suppressed too. Regression-guarded by `test_{doubao,jimeng,samsung}_engine.py::TestDetect::test_small_image_guarded_from_false_positive`.
`_text_mark_engine.py` — **shared base for the three text-mark engines (Doubao/Jimeng/Samsung), extracted 2026-06-09** (they were ~90% byte-identical clones). `TextMarkEngine(config: TextMarkConfig)` owns the `locate → extract_mask → detect` detection pipeline plus the removal that localizes the glyph blob to a footprint mask and hands it to the shared `watermark_registry.fill` (+ the asset-keyed `load_alpha_template`/`glyph_silhouette`/`template_match_score` caches). Detection still matches the glyph silhouette (NCC against the captured template); the removal MASK is TEMPLATE-FREE — it is the bounding box of the top-hat glyph blob from `extract_mask`, filled solid + dilated, so a re-rendered or differently-placed mark is still masked. This dropped the fixed alpha-template placement; the captured alpha maps are now used only for the detection silhouette, not for removal. Each engine module is a thin subclass supplying only its `TextMarkConfig` (the tuned constants, the bundled asset, and the bounded structural deltas — `corner` br/bl, `margin_floor` 4/2, `morph_open_size` 5/3, `min_gw` 8/16, and since 2026-07-21 `ladder` — the scale rungs `_tophat_best` sweeps, per-mark because 千问's two size modes do not fit the shared 3-rung comb (default `(0.8, 1.0, 1.25)`, unchanged for every other mark; densifying the SHARED ladder was measured and rejected -- see the verification plan's B2). plus the test-facing module shims (`_alpha_template`/`_glyph_silhouette`/`_template_match_score` + the constants). Gemini stays a SEPARATE engine (its multi-size fixed-slot sparkle model is genuinely different). Add a new text mark = a new `TextMarkConfig` + a thin subclass + one registry `_text_mark(...)` row. The engine bullets below describe each mark's calibration history; the LOGIC lives here. **Small-image detection guard (`_MIN_DETECT_SHORT_SIDE` 200, added 2026-06-26):** `detect` returns not-detected when the image short side is below 200px. Below that the glyph template degrades to the `min_gw` floor (~8px) and `TM_CCOEFF_NORMED` on a few pixels is noise, so an unrelated small geometric shape can spuriously correlate with the CJK silhouette — a 48×48 app-icon chevron scored Doubao 0.41 / Jimeng 0.47 (both above their thresholds), a pure small-size artifact (the same icon upscaled collapses to ~0.06–0.10 NCC at ≥256px). A real AI-generation label is stamped on a full-resolution render (the captured samples are 1086–2048px wide, the smallest positive test image is 1086px), so the floor sits far below any genuine mark while killing the icon/thumbnail band (≤96px); `identify` falls back to "unknown" (the safe default) and removal, gated on detection, is suppressed too. Regression-guarded by `test_{doubao,jimeng,samsung}_engine.py::TestDetect::test_small_image_guarded_from_false_positive`.
**Removal is localize -> fill.** The engine localizes the glyph blob (`extract_mask` over the located box) into a solid, dilated footprint mask and hands it to the shared `watermark_registry.fill` (cv2 / MI-GAN / big-LaMa). The template-free mask (bounding box of the glyph blob, not the fixed alpha template) means a re-rendered or moved mark is still covered, and the fill reconstructs the box from its surroundings. On corpus images doubao and jimeng localize + remove at ~100% with clean footprints (the filled region blends into its surroundings within a few LAB levels, no color shift, no dark pit); clean images with no vendor signature had 0% false removal.
@@ -253,6 +253,24 @@ The cost (mislabel ~8-33% of non-Gemini content as Gemini) outweighs the benefit
**No committed real sample** (only the flat calibration captures are committed) — `tests/test_samsung_engine.py` synthesizes a mark from the bundled template (bottom-left geometry), with `test_recovers_shifted_mark_on_texture` guarding the localize-on-shift path. Samsung Galaxy AI edits are independently caught by C2PA + the `genAIType` marker in `metadata`/`identify`, so this engine is the visible-mark *removal* path; it also feeds `identify` as the medium-confidence `visible_samsung` signal via the registry (the stripped-metadata fallback).
## `qwen_engine.py`
`qwen_engine.py` — **a thin `TextMarkEngine` subclass (config only), registered 2026-07-21.** visible Qwen (Alibaba Tongyi Qianwen) "千问AI生成" detector + localizer (cv2/numpy, no GPU), bottom-right, the same GB 45438-2025 6-glyph house style as Doubao (2-glyph vendor prefix + mandated `AI生成` tail; the vendor's tri-lobe logo precedes the text and is deliberately NOT in the silhouette -- logos vary between releases, the CJK run is what discriminates). The detection silhouette `assets/qwen_alpha.png` is font-rendered synthetic (`scripts/render_vendor_silhouettes.py`), never cut from an upload. Feeds `identify` as the medium-confidence `visible_qwen` signal via the registry.
**Why this registration took two attempts:** the 2026-07-18 attempt died at n=1 positives; the unlock was the TC260 label's `ContentProducer` field, whose USCC names the signing entity and partitions carriers into per-vendor cohorts from metadata alone (`scripts/vendor_cohort_harvest.py` -- 117 labelled Qwen frames, owing nothing to any pixel detector). Every constant was then MEASURED on that cohort against 286 hand-labelled clean frames (`scripts/vendor_mark_calibrate.py`), not inherited from Doubao:
* **Basis `short`** (frac_short CV 0.189 vs width 0.273).
* **Per-mark 2-rung `ladder=(0.78, 1.27)`** against `alpha_width_frac` 0.160: the mark sits in TWO size modes (~0.124 and ~0.203 of the short side, ratio 1.64, both clusters tight) -- wider than the shared 3-rung ladder's 1.5625 span, so the best single fraction covers only 74.5% and the small mode lands in the comb's collapse zone (real marks at the exact rung score ~0.94 vs ~0.67 on the shared ladder). A 4-rung variant scored strictly worse (its big-mode rung sits 4.6% off the mode). The shared default is untouched for every other mark.
* **Fitted locate box** (`width_frac` 0.231, `height_frac` 0.074, margins ~0.021): the real mark sits ~0.025 of the short side off the right edge, and Doubao's box (0.004 margin) clipped the first glyph -- an exact-size template collapsed 0.73 -> 0.26 on a real frame.
* **`alpha_height_frac` 0.0416 from the aspect fit** (p50 aspect 0.260 at the winning width) -- not the silhouette's own aspect (0.2219) and not Doubao's ratio.
* **Gate 0.45**: clean p99 0.301 / max 0.316 (286 frames), and every cohort frame >= 0.45 carries a visible mark (83 of ~96 eyeballed visible marks fire = 86% recall of visible marks; the misses are white-on-near-white contrast losses). 0.45 was picked over 0.32 for margin against unseen clean content at zero measured recall cost.
* **STRICT ONLY (`provenance_ncc_factor` 1.0):** the score band just below the gate is dominated by non-Qwen banners on same-cohort frames (a 夸克 anti-forgery strip at 0.274, a 造点 mark at 0.253), so a provenance-relaxed arm would be mostly false fills. There is no provenance mapping for qwen and no relaxed arm.
* **No rival margin:** 0 cross-fires on 400 Doubao-marked / 298 Jimeng-marked / 286 clean frames at the gate, while a 0.10 margin would have cost ~10% of genuine Qwen detections. A confident Qwen detection suppresses the Jimeng pill exactly like Doubao's does (`_keep_pill` -- a Qwen frame is TC260 too but is not Jimeng-basic).
**Parity:** detect -> cv2 fill -> re-detect is clean on **83/83** real cohort marks, no empty masks (the `tophat` faint-mask fallback rides the same `_tophat_best`); the e2e suite drives one live cohort positive through the real CLI (`scripts/real_examples_e2e.py`, qwen bucket = symlinks under the gitignored `_visible_datasets/`). Regression: `tests/test_qwen_engine.py` (pins the ladder, the strict-only factor, the box anchor, and both size modes at a score floor that discriminates the shared-ladder and Doubao-margin mutations).
**The clean-arm contamination trap (load-bearing for any future calibration):** the 2026-07-18 `present: []` labels are in the vocabulary of the REGISTERED marks only, so 146 of the 432 "clean" frames sit in a TC260 cohort -- including Qwen-cohort frames visibly carrying 千问AI生成. They made up the clean arm's entire top tail (clean p99 0.37 -> 0.69). `vendor_mark_calibrate.load_sets` now excludes every frame in ANY TC260 cohort from the clean arm; a gate read off the unguarded arm is meaningless.
## `region_eraser.py`
`region_eraser.py` — universal region eraser (`erase` CLI) AND the shared fill backend behind `watermark_registry.fill` for the visible localize -> fill removal. `erase(image, boxes=|mask=, backend=)` accepts grayscale (2D) and RGBA (4-channel) inputs on **all** backends (each splits off any alpha plane and re-attaches it unchanged, and promotes grayscale to BGR): `boxes_to_mask` → one of three backends.
@@ -350,7 +368,7 @@ Diffusion SynthID removal. The `--tile/--no-tile` knob is the *lossless* alterna
### `visible`
Known-visible-mark removal by **localize -> fill**: each detected mark is localized to a binary full-frame footprint mask, then one shared, swappable fill inpaints that mask. `--backend auto|cv2|migan|lama` (default `auto`) picks the fill: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN ONNX, the memory-tight pick where LaMa will not fit), `lama` (big-LaMa ONNX, best quality, heavier, auto-preferred when a learned backend is available); `auto` = LaMa > MI-GAN > cv2, best available (LaMa is auto-preferred when a learned backend is present; a memory-tight deploy pins migan). `--sensitivity auto|strict|assume-ai` (default `auto`) controls how hard a borderline mark is trusted (see the registry section: the visual detectors are metadata-independent; `auto` relaxes a mark only on same-product evidence, `assume-ai` relaxes every mark on the caller's AI assertion, subject to the assumed-trust confidence floor where the vendor is unconfirmed — the only path to higher recall on a metadata-stripped screenshot). `--backend` and `--sensitivity` are shared across `visible`/`all`/`batch`. Detection keys on each mark's own shape, and under `auto` the trust gate is relaxed when local metadata confirms the vendor (a Google/Gemini C2PA issuer relaxes gemini, a China-AIGC label relaxes doubao/jimeng, `samsung_genai` relaxes samsung), so a moved or re-rendered mark is still caught. `--mark auto` (default) removes EVERY detected mark in one pass (`registry.remove_auto_marks`, not the single strongest -- a Jimeng-basic image carries both the top-left pill and the bottom-right wordmark) from: the Gemini sparkle, the Doubao "豆包AI生成" text strip, the Jimeng "★ 即梦AI" wordmark, the Samsung Galaxy AI "✦ Contenuti generati dall'AI" strip (bottom-LEFT, Italian-locale detection), and the capture-less Jimeng "AI生成" pill (top-left, `pill_engine`). The pill's weak edge-NCC detector is gated in `remove_auto_marks` via `_keep_pill` (32k real-upload corpus validation 2026-07): never on Doubao, and two confirmation arms since metadata confirms the platform, not pill presence. (1) The bottom-right wordmark fired — ~94% precise and survives metadata-STRIPPED uploads (screenshots / re-saves) — removes the pill unrestricted. (2) TC260 metadata confirms Jimeng (`"jimeng" in provenance`, from `cli._visible_provenance`) OR the caller asserts AI (`sensitivity == "assume_ai"`), no wordmark — **re-measured 2026-07-18 on 149 blind-labelled pill fires: 21% precise raw (CI 16-29%), 29% (CI 20-40%) among the flat footprints the guard actually PASSES, 14% among those it blocks** — its false fires are textured ceilings/walls that the fill visibly SMEARS — removes the pill ONLY when the top-left footprint is flat enough for an invisible fill (`pill_engine.footprint_is_flat`, median-Sobel ≤ `_FLAT_TEXTURE_MAX`; the flatness guard holds even under `assume_ai`). No confirmation → never removed. `--mark gemini|doubao|jimeng|samsung|jimeng_pill` forces one (choices come from the registry). Corpus validation: doubao and jimeng localize + remove at ~100% with clean footprints (the filled region blends into its surroundings within a few LAB levels, no color shift, no dark pit); clean images with no vendor signature had 0% false removal. For arbitrary logos/objects use `erase`. **When `--mark auto` finds no known mark (the common case — ~74% of real uploads carry no registered visible mark), the command does NOT silently re-serve the input as a finished result.** It runs a cheap metadata-only `identify`, prints actionable guidance (if the image carries an invisible/metadata mark, e.g. an OpenAI/Gemini C2PA image, it points to `all`; otherwise it does NOT imply the image is clean -- it warns that an invisible pixel watermark like SynthID cannot be detected once the metadata proxy is gone and routes to both `all` and `erase --region`), writes NO output file, and exits **`EXIT_NO_VISIBLE_MARK` (2)** — distinct from success (0) and a hard error (1) so a wrapping service (raiw.cc) can surface the message instead of treating the unchanged image as done (the production "it didn't work" / score-0 trap). Same handling for an explicit `--mark <name>` that is not detected. Helper `cli._no_visible_mark_exit`; regression-guarded by `tests/test_cli.py::TestVisibleCommand::test_visible_auto_no_mark_exits_two_with_eraser_hint` and `test_visible_auto_no_mark_routes_to_all_when_metadata`. `--no-detect` still forces the gemini fallback and proceeds (exit 0).
Known-visible-mark removal by **localize -> fill**: each detected mark is localized to a binary full-frame footprint mask, then one shared, swappable fill inpaints that mask. `--backend auto|cv2|migan|lama` (default `auto`) picks the fill: `cv2` (classical inpaint, no deps, the floor), `migan` (MI-GAN ONNX, the memory-tight pick where LaMa will not fit), `lama` (big-LaMa ONNX, best quality, heavier, auto-preferred when a learned backend is available); `auto` = LaMa > MI-GAN > cv2, best available (LaMa is auto-preferred when a learned backend is present; a memory-tight deploy pins migan). `--sensitivity auto|strict|assume-ai` (default `auto`) controls how hard a borderline mark is trusted (see the registry section: the visual detectors are metadata-independent; `auto` relaxes a mark only on same-product evidence, `assume-ai` relaxes every mark on the caller's AI assertion, subject to the assumed-trust confidence floor where the vendor is unconfirmed — the only path to higher recall on a metadata-stripped screenshot). `--backend` and `--sensitivity` are shared across `visible`/`all`/`batch`. Detection keys on each mark's own shape, and under `auto` the trust gate is relaxed when local metadata confirms the vendor (a Google/Gemini C2PA issuer relaxes gemini, a China-AIGC label relaxes doubao/jimeng, `samsung_genai` relaxes samsung), so a moved or re-rendered mark is still caught. `--mark auto` (default) removes EVERY detected mark in one pass (`registry.remove_auto_marks`, not the single strongest -- a Jimeng-basic image carries both the top-left pill and the bottom-right wordmark) from: the Gemini sparkle, the Doubao "豆包AI生成" text strip, the Jimeng "★ 即梦AI" wordmark, the Qwen "千问AI生成" text strip, the Samsung Galaxy AI "✦ Contenuti generati dall'AI" strip (bottom-LEFT, Italian-locale detection), and the capture-less Jimeng "AI生成" pill (top-left, `pill_engine`). The pill's weak edge-NCC detector is gated in `remove_auto_marks` via `_keep_pill` (32k real-upload corpus validation 2026-07): never on Doubao or Qwen, and two confirmation arms since metadata confirms the platform, not pill presence. (1) The bottom-right wordmark fired — ~94% precise and survives metadata-STRIPPED uploads (screenshots / re-saves) — removes the pill unrestricted. (2) TC260 metadata confirms Jimeng (`"jimeng" in provenance`, from `cli._visible_provenance`) OR the caller asserts AI (`sensitivity == "assume_ai"`), no wordmark — **re-measured 2026-07-18 on 149 blind-labelled pill fires: 21% precise raw (CI 16-29%), 29% (CI 20-40%) among the flat footprints the guard actually PASSES, 14% among those it blocks** — its false fires are textured ceilings/walls that the fill visibly SMEARS — removes the pill ONLY when the top-left footprint is flat enough for an invisible fill (`pill_engine.footprint_is_flat`, median-Sobel ≤ `_FLAT_TEXTURE_MAX`; the flatness guard holds even under `assume_ai`). No confirmation → never removed. `--mark gemini|doubao|jimeng|qwen|samsung|jimeng_pill` forces one (choices come from the registry). Corpus validation: doubao and jimeng localize + remove at ~100% with clean footprints (the filled region blends into its surroundings within a few LAB levels, no color shift, no dark pit); clean images with no vendor signature had 0% false removal. For arbitrary logos/objects use `erase`. **When `--mark auto` finds no known mark (the common case — ~74% of real uploads carry no registered visible mark), the command does NOT silently re-serve the input as a finished result.** It runs a cheap metadata-only `identify`, prints actionable guidance (if the image carries an invisible/metadata mark, e.g. an OpenAI/Gemini C2PA image, it points to `all`; otherwise it does NOT imply the image is clean -- it warns that an invisible pixel watermark like SynthID cannot be detected once the metadata proxy is gone and routes to both `all` and `erase --region`), writes NO output file, and exits **`EXIT_NO_VISIBLE_MARK` (2)** — distinct from success (0) and a hard error (1) so a wrapping service (raiw.cc) can surface the message instead of treating the unchanged image as done (the production "it didn't work" / score-0 trap). Same handling for an explicit `--mark <name>` that is not detected. Helper `cli._no_visible_mark_exit`; regression-guarded by `tests/test_cli.py::TestVisibleCommand::test_visible_auto_no_mark_exits_two_with_eraser_hint` and `test_visible_auto_no_mark_routes_to_all_when_metadata`. `--no-detect` still forces the gemini fallback and proceeds (exit 0).
### `batch`
docs/verification-plan.md
+138 -24
View File
@@ -444,7 +444,8 @@ The machinery exists: `visible_recall_sample.py` -> `visible_sheets.py` ->
- **Precision** re-runs over the existing 779-cell ground truth; benchmark every detector
change with `--vs <snapshot>`.
- **Coverage**, the largest known gap: ~6% of sampled images carry an uncovered vendor's
mark (千问 / 百度 / 星绘 / 抖音-class) that no registered detector can fire on. This is a
mark (百度 / 星绘 / 抖音-class; 千问 was the first of this class and is registered since
2026-07-21) that no registered detector can fire on. This is a
coverage problem, not a tuning problem, and no threshold work will move it.
Three harness rules are load-bearing and must not be relaxed: score a mark only within its
@@ -550,15 +551,17 @@ Per mark, what actually goes away when metadata is stripped:
detections corpus-wide (12.5% of the 1,256 lost). That headroom exists but the
precision trade behind it was deliberate.
- **The largest gap is metadata-independent by nature**: ~6% of sampled images carry an
uncovered vendor's mark (千问 / 百度 / 星绘 / 抖音-class) that no registered detector
can fire on at all.
uncovered vendor's mark (百度 / 星绘 / 抖音-class; 千问 is registered since 2026-07-21)
that no registered detector can fire on at all.
### Where the evidence points
1. **A generic CJK AI-mark detector.** GB 45438-2025 mandates the shared `AI生成` tail,
and 千問 is already measured as non-separable from Doubao (AUC ~0.5) precisely because
of it. The right shape is to detect the mark CLASS and treat vendor attribution as
optional metadata. Closes the 6% coverage gap and is metadata-free by construction.
1. ~~A generic CJK AI-mark detector~~ **superseded for 千问 (2026-07-21).** The AUC ~0.5
non-separability from Doubao was measured at the WRONG size; at the fitted geometry an
exact-size 6-glyph template separates 千问 from 400 doubao-marked frames with ZERO
cross-fire at the gate, so per-vendor registration won and shipped. The generic
class-detector shape may still be right for the long tail of compliant vendors, but
its motivating measurement is gone.
2. **Port the `tophat` front-end to the remaining marks.** It took Doubao from 89% to 92%
recall at unchanged 99% precision. But the gate is front-end specific and **must be
recalibrated, never ported**: a naive 0.40 produced 8 false fires instead of 1 and
@@ -725,21 +728,22 @@ would take, so none of it has to be rediscovered.
### START HERE next session
The verification campaign is finished for everything that does not need new labelled data.
In priority order, with the reason each sits where it does:
Item 1 from the previous session is **DONE (2026-07-21)**: 千问 is registered --
see "The 千问 harvest (2026-07-21)" below for the decision and the numbers. In
priority order:
1. **Harvest labelled positives for the uncovered vendors** (`千问`, `百度`, and the
`星绘`/`抖音` class). This is the only thing that unblocks anything else. Every cheap
detector lever was measured to exhaustion this session and all are dead ends, so the
remaining questions -- can 千问 be registered, at what gate, does samsung's residual
generalize, is jimeng's 71% real -- all reduce to "we have too few examples to tell".
Tool: `scripts/cjk_tail_probe.py` (a harvesting aid, NOT a detector: measured 0.407 for
a bold 千问 positive against a clean p99 of 0.298). Target 30+ per vendor, then calibrate.
2. **Decide the exit-code split** (open defect 2). It is a deliberate product call, not
1. **Decide the exit-code split** (open defect 2). It is a deliberate product call, not
research: it is breaking for existing wrappers, so it needs a yes/no rather than more
measurement.
3. **The two small correctness items** (open defects 3 and 4) -- both are contained, both
2. **The two small correctness items** (open defects 3 and 4) -- both are contained, both
have the fix written out below.
3. **The bonus vendors from the harvest** (元宝 n=50, 可灵 n=30, cat-logo n=19) repeat
the 千问 playbook each: font-rendered silhouette, `--fit-geometry`, gate calibration
against the contamination-guarded clean arm, crossfire against doubao/jimeng. 可灵
additionally stamps a second mark bottom-LEFT, which no current text-mark config
expresses (the pill is top-left; a bottom-left CJK mark needs a `corner="bl"` CJK
config -- samsung is `bl` but Latin-script and width-based). 星绘/百度 are NOT in the
corpus in labelable quantity -- verified, do not hunt them again.
Do NOT restart the sweeps to "check". Their artifacts are on disk and listed under
"Completed full runs" below; re-running costs hours and answers nothing new. The fast way
@@ -747,6 +751,115 @@ to confirm the whole surface still works after a change is
`uv run python scripts/real_examples_e2e.py` (~2 min, real corpus examples through the real
CLI) plus `uv run python scripts/robustness_suite.py` (~3 min, adversarial inputs).
### The 千问 harvest (2026-07-21) -- RESOLVED, registered the same day
**The unlock: the TC260 label is not anonymous.** Its `ContentProducer` field carries the
producer's Chinese Unified Social Credit Code (`001191110102MACQD9K64010000` -> USCC
`91110102MACQD9K640`), which names a legal entity. So carriers partition into per-VENDOR
cohorts from METADATA ALONE, owing nothing to any pixel detector -- which is exactly what
broke the previous attempt, whose only way to find 千问 frames was to eyeball the misses of
a detector that cannot see them. A cohort is a LABEL: eyeball one frame, and every frame in
it is a labelled example. (CLAUDE.md's "the generic TC260 label names no specific vendor" is
about the label MARKER; the producer FIELD inside the block is a different thing.)
New tools, both lint-clean, both **uncommitted**:
- `scripts/vendor_cohort_harvest.py` -- full-corpus metadata scan -> cohorts. Joins which
detectors fired from the completed `_visible_positives.jsonl` rather than re-running the
pixel pass. Artifact `data/spaces/_vendor_cohorts.jsonl` (**4441 carriers, 46 entities**).
`--sheets N` writes full-width top/bottom band crops per cohort for eyeballing.
- `scripts/vendor_mark_calibrate.py` -- scores a cohort against the 432 hand-labelled
`present: []` negatives from the 2026-07-18 round, and writes score-SORTED corner crops so
mark presence and the gate are read in one visual pass.
- `src/.../assets/qwen_alpha.png` + `xinghui_alpha.png` regenerated (they were listed in
`render_vendor_silhouettes.py` but had never been committed).
**What the corpus actually contains** (this corrects the previous list of targets):
| Cohort USCC | n | quiet | Visible mark | Verdict |
|---|---|---|---|---|
| 91440101MA9Y9T4H7A | 117 | 112 | `千问AI生成` bottom-right | **the target, confirmed by eye** |
| 91340100MAEB4N8H76 | 73 | 70 | mostly none; one `RunningHub AI生成` | metadata-mostly |
| 913502007378955153 | 113 | 109 | none seen | metadata-only |
| 91440300708461136T | 50 | 46 | `元宝AI生成` (Tencent Yuanbao) | bonus vendor, bold |
| 91441900557262083U | 49 | 45 | none seen | metadata-only |
| 91110108335469089C | 30 | 28 | `可灵AI 3.0` (Kling) + an `AI生成` pill bottom-LEFT | bonus vendor |
| 91110108562144110X | 19 | 19 | cat-logo + `AI生成`, in **19/19** | bonus vendor, very clean |
**`百度` and the `星绘`/`抖音` class are NOT in this corpus in labelable quantity.** No brand
token for them appears in any AIGC label field, and every remaining cohort is <= 16 frames.
Do not spend another session hunting them here; the previous "one confirmed positive each"
is all there is. The corpus offers 千问 richly plus three DIFFERENT vendors instead.
Also worth knowing: a cohort is a strong grouping key but names the SIGNING ENTITY, not
always the consumer brand -- the 千问 cohort contains one `造点AI生成` frame. And TC260
provenance does NOT imply a visible mark, which is why four large cohorts above are
metadata-only. The cohort is the candidate pool; the eye settles mark presence.
**RESOLVED 2026-07-21: 千问 is registered** (`qwen_engine.py`, strict-only, no rival
margin). The ladder trade-off that was the open decision is settled in favor of a
**per-mark ladder**, not the shared one and not a wider shared one -- and the path there
found two more geometry defects the "single fraction on the shipped ladder" framing had
missed.
What the final calibration measured, in the order it happened:
1. **The locate box was clipping the mark.** Scoring with the fitted fractions still
collapsed the cohort (p50 0.209). Frame-level diff against the wide-ladder fit showed
why: the real mark sits ~0.025 of the short side off the right edge, while doubao's
inherited box anchors at 0.004 -- the box's left edge cut into the 千 glyph, and an
exact-size template scored 0.26 where the fit's wider box scored 0.73. So the locate
fractions are as mark-specific as the template size; `--fit-geometry` now records the
absolute match rects and fits the box too (margins ~0.021, width 0.231, height 0.074).
2. **The size distribution is cleanly BIMODAL.** With the box fixed, frac_short clusters
at ~0.124 (13 frames) and ~0.203 (38 frames), nothing between -- two stamp sizes,
ratio 1.64, just over the shared ladder's 1.5625 span. That is why no single fraction
covers the mark.
3. **A per-mark 2-rung ladder beats both alternatives.** Candidates, both arms scored on
the shipped code path: (A) shipped 3-rung @ frac 0.167 -- cohort p50 0.538; (B1)
2-rung (0.78, 1.27) @ frac 0.160, one rung centred on each mode -- cohort p50
**0.662**; (B2) 4-rung (0.8, 1.0, 1.25, 1.5625) @ frac 0.155 -- p50 0.449, strictly
worse (its big-mode rung sits 4.6% off the mode, and the extra rungs cover nothing).
B1 also costs one matchTemplate LESS than the shipped 3. `TextMarkConfig.ladder` was
added with the default `(0.8, 1.0, 1.25)`, so every other mark's computation is
byte-identical (the full 876-test suite plus e2e + robustness confirm); a shared
densification was already ruled out by B2's false-fire measurement. The doubao
false-fire check the plan asked for reduces to that equivalence-by-construction --
doubao's ladder never changed.
4. **`alpha_height_frac` measured, not inherited:** aspect fit at the winning width, p50
0.260 (tight, p10-p90 0.250-0.270) -> 0.0416. The silhouette's own aspect (0.2219)
and doubao's ratio were both measurably off.
5. **The clean arm was contaminated, and fixing it flipped the verdict.** The 2026-07-18
`present: []` labels are in the vocabulary of the REGISTERED marks only -- 146 of the
432 "clean" frames sit in a TC260 cohort, including 15 qwen-cohort frames VISIBLY
carrying 千问AI生成, and they were the clean arm's entire top tail (clean p99 0.37 ->
0.69 with the fitted geometry). `load_sets` now drops every frame in ANY cohort.
Final arm: 286 frames, clean p99 0.301 / max 0.316.
6. **Gate 0.45, strict-only, no rival margin.** Every cohort frame >= 0.45 carries a
visible mark (83 of ~96 eyeballed visible marks fire = **86% recall of visible
marks**; the misses are white-on-near-white contrast losses); 0/286 clean fires;
crossfire at the gate: 0/400 on doubao-marked frames, 0/298 on jimeng-marked frames
(the shared `AI生成` tail correlates at p50 0.224, far below gate -- the AUC-0.5
attribution fear from 2026-07-18 was a mis-sizing artifact). A 0.10 rival margin
would have cost ~10% of genuine qwen detections, so `rivals=()`. The band just below
the gate is dominated by non-qwen banners (夸克 anti-forgery strip 0.274, 造点 mark
0.253), so a provenance-relaxed arm would be mostly false fills:
`provenance_ncc_factor` is pinned at 1.0 and qwen has NO provenance mapping.
7. **Parity confirmed end to end:** detect -> cv2 fill -> re-detect clean on **83/83**
real cohort marks, no empty masks; `real_examples_e2e.py` now drives a live qwen
positive through the real CLI (qwen bucket = symlinks under the gitignored
`_visible_datasets/`); a confident qwen detection suppresses the jimeng pill exactly
like doubao's does.
Method note worth keeping: **the wide ladder flattered the clean arm exactly as
predicted, but the trap that actually bit was the LABEL vocabulary.** "present: []"
meant "no registered mark", not "no mark" -- and a calibration clean arm has to be
re-filtered per candidate, or the gate is read off frames that carry the very mark being
calibrated.
The bonus vendors (元宝, 可灵, cat-logo) need their own font-rendered silhouettes before
any of this repeats for them; 可灵 additionally puts a second mark bottom-LEFT, which no
current text-mark config expresses.
### Open defects
| # | Defect | Measured impact | What the fix takes |
@@ -838,12 +951,13 @@ evidence supports:
fire at 1.7:1 because the recoveries and the false fires are the same landscape size
shift. Moving the landscape width fraction is also out -- detected landscape marks
already sit at the nominal, so it would break more than it fixes. Do not spend here.
2. **Coverage of uncovered vendors is the largest lever** and is blocked on EVIDENCE, not
architecture. `千问` and `百度` marks sit in the same corner we already scan, and the
front-end that `render_vendor_silhouettes.py` said was missing now exists. But this
session found exactly one confirmed positive per vendor, and the 14 千问 positives that
note quotes are not reachable from any current script. Nothing may be registered off a
single frame. Harvest 30+ per vendor with `scripts/cjk_tail_probe.py`, then calibrate.
2. **Coverage of uncovered vendors is the largest lever.** 千问 was the head of this item
and is now CLOSED (registered 2026-07-21, see the harvest section above): the blocker
turned out to be evidence, and the TC260 producer-USCC cohort trick removed it. The
remaining named vendors are 元宝 (n=50), 可灵 (n=30) and cat-logo (n=19) -- each needs
a font-rendered silhouette, then the same calibrate-and-crossfire chain. `百度` and the
星绘/抖音 class are NOT in the corpus in labelable quantity (verified twice; do not
hunt them again). Nothing may be registered off a single frame.
3. **A generic shared-tail template is not a shortcut.** `AI生成` is guaranteed across
compliant vendors by GB 45438-2025, so one template covering all of them is the obvious
idea -- and measured on the tophat front-end it separates a bold 千问 positive from clean
scripts/real_examples_e2e.py
+1 -1
View File
@@ -180,7 +180,7 @@ def check_visible(res: Results, tmp: Path) -> None:
from remove_ai_watermarks.watermark_registry import detect_marks, get_mark, remove_auto_marks
print("\nvisible --mark auto -- real marked image per mark, product decision then re-detect")
for mark in ("doubao", "jimeng", "gemini", "samsung", "jimeng_pill"):
for mark in ("doubao", "jimeng", "qwen", "gemini", "samsung", "jimeng_pill"):
src = find_visible_positive(mark)
if src is None:
res.add("visible", mark, True, "no live positive in bucket (skipped, not a failure)")
scripts/render_vendor_silhouettes.py
+30 -19
View File
@@ -16,10 +16,17 @@ what does NOT discriminate -- see the rival-margin mechanism in _text_mark_engin
Regenerate with: uv run python scripts/render_vendor_silhouettes.py
STATUS 2026-07-18: these two marks are NOT registered, and this script is kept as the
method + the record of why. Measured on 14 hand-verified 千问 positives from the corpus,
the current detect architecture (top-hat glyph blob -> binary TM_CCOEFF_NORMED) cannot
see this mark AT ALL:
STATUS 2026-07-21: `qwen_alpha.png` IS registered (`qwen_engine.py`) -- the 2026-07-18
blocker quoted below turned out to be mis-sized GEOMETRY (two size modes + a locate box
that clipped the first glyph), not segmentation, and was solved by the TC260-producer
cohort harvest + `vendor_mark_calibrate.py` (117 labelled frames; full record in
`docs/verification-plan.md`). `xinghui_alpha.png` is still NOT registered: one confirmed
corpus example is nothing to calibrate a gate against.
--- the 2026-07-18 record, kept as the history of the failed first attempt ---
Measured on 14 hand-verified 千问 positives from the corpus,
the then-current detect architecture (top-hat glyph blob -> binary TM_CCOEFF_NORMED)
could not see this mark AT ALL:
same pipeline, each mark scored with its OWN template, on real positives
doubao n=40 mean NCC 0.723 median 0.835 >= 0.40 gate: 82%
@@ -35,13 +42,12 @@ Three checks ruled out the obvious explanations, in order:
3. NOT the appearance thresholds. Sweeping tophat_delta / logo_min_luma / kernel
reached at best mean 0.35 with 4/14 over the gate.
The blocker is SEGMENTATION on a faint mark: Doubao is stamped bold and opaque, so the
white top-hat returns a clean glyph blob; the Qwen mark is a thin translucent overlay
that shatters into specks, and no template can match a blob that is not there. Adding
it therefore needs a detection front-end that does not depend on binarizing the glyph
(grayscale/edge correlation on the raw top-hat, or a learned patch classifier) -- not a
new silhouette. Shipping it on the current front-end would mean a detector that finds
almost nothing and, at any threshold low enough to fire, fires on arbitrary corner text.
The blocker was named SEGMENTATION on a faint mark: Doubao is stamped bold and opaque,
so the white top-hat returns a clean glyph blob; the Qwen mark is a thin translucent
overlay that shatters into specks, and no template can match a blob that is not there.
The `tophat` front-end (built later, for doubao) removed that blocker -- and 千问 STILL
did not register, because the real residual was geometry. See the 2026-07-21 status
above.
星绘 additionally has only ONE confirmed example in the corpus, so even a working
front-end could not have its threshold calibrated yet.
@@ -66,14 +72,19 @@ neither settles the question:
The blocker is now EVIDENCE, not architecture: this session found exactly one 千问 and one
百度 positive (both by eyeballing doubao-provenance misses), and the 14 positives quoted
above were not preserved anywhere the current scripts can reach. Nothing should be
registered off a single frame. What it takes: harvest 30+ confirmed positives per vendor
-- `scripts/cjk_tail_probe.py` exists for exactly this, scoring TC260-provenance frames
that no detector fires on and writing a contact sheet of the top scorers to label -- then
calibrate a gate against the clean arm. Two traps worth knowing before repeating this
measurement: score with `alpha_height_frac`, not the silhouette's own aspect ratio (the
latter inflated the clean p99 from 0.30 to 0.58 and made every comparison meaningless),
and keep the ladder at the shipped 3 rungs, since a wide sweep hands clean corners many
extra chances to match.
registered off a single frame.
UPDATE 2026-07-21 (the resolution): the evidence arrived via the TC260 producer-USCC
cohort trick (`scripts/vendor_cohort_harvest.py` -- 117 labelled 千问 frames from metadata
alone), and the registration shipped the same day (`qwen_engine.py`). The "no separation
at all" reading above was the MIS-SIZED geometry, not the mark: at the fitted geometry the
full template separates the cohort from clean corners 0.662 vs 0.134 (p50). The traps
below still bind any NEXT vendor: score with `alpha_height_frac`, not the silhouette's own
aspect ratio (the latter inflated the clean p99 from 0.30 to 0.58 and made every
comparison meaningless); keep the ladder at the shipped rungs for gate-setting, since a
wide sweep hands clean corners many extra chances to match; and re-filter the clean arm
per candidate -- the 2026-07-18 `present: []` labels mean "no REGISTERED mark", so qwen
-cohort frames visibly carrying 千问AI生成 sat in it (see `vendor_mark_calibrate.load_sets`).
"""
from __future__ import annotations
scripts/vendor_cohort_harvest.py
+241
View File
@@ -0,0 +1,241 @@
"""Partition China-AIGC carriers into VENDOR COHORTS by their TC260 producer code.
THE PROBLEM THIS SOLVES
Coverage of uncovered vendors is the largest remaining detection lever
(`docs/verification-plan.md`, "Where detection work should go next"), and it is
blocked on EVIDENCE: nothing may be registered off a single frame, and the
previous session found exactly one confirmed positive each for `千问` and `百度`.
Harvesting more by PIXELS is circular -- a detector is what we are trying to
build -- and the generic shared-tail probe is too weak to label with (0.407 on a
bold positive against a clean p99 of 0.298; see `cjk_tail_probe.py`).
THE KEY
The TC260 label is not anonymous. Its `ContentProducer` field carries the
producer's Chinese Unified Social Credit Code (USCC), e.g.
``001191110102MACQD9K64010000`` -> USCC ``91110102MACQD9K640``, which names a
specific legal entity. So the metadata partitions carriers into per-ENTITY
cohorts without looking at a single pixel. A cohort is a LABEL: once one frame
in it is eyeballed, every frame in it is a labelled example of that vendor's
mark. That is what turns "one confirmed positive" into "30+ per vendor".
CLAUDE.md's "the generic TC260 label names no specific vendor" is about the
label MARKER (the bare presence of `TC260:AIGC`), which indeed names nobody.
The producer FIELD inside the block is a different thing and does name one.
Caveat kept in view: the code names the SIGNING ENTITY, which is not always the
consumer brand (an aggregator or a cloud host signs for several apps, and one
vendor can hold several codes). So a cohort is a strong grouping key and a
hypothesis about the brand -- the brand itself is settled by reading the crop,
which is what `--sheets` is for.
WHAT IT COSTS
Metadata only. The expensive pixel pass is NOT re-run: which detectors fired is
joined from `_visible_positives.jsonl` (the completed full-corpus artifact), per
the standing rule against relaunching finished sweeps to re-check them.
DATA SAFETY
Corpus images are real user uploads: read-only, local analysis, gitignored
output. Contact sheets stay under `data/spaces/`; nothing here is committed.
uv run python scripts/vendor_cohort_harvest.py
uv run python scripts/vendor_cohort_harvest.py --report-only --sheets 12
"""
from __future__ import annotations
import argparse
import glob
import json
import os
import sys
from collections import Counter
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).parent.parent))
sys.path.insert(0, str(Path(__file__).parent))
REPO = Path(__file__).resolve().parents[1]
CORPUS = REPO / "data" / "spaces" / "originals"
OUT = REPO / "data" / "spaces" / "_vendor_cohorts.jsonl"
FIRED = REPO / "data" / "spaces" / "_visible_positives.jsonl"
SHEET_DIR = REPO / "data" / "spaces" / "_vendor_cohort_sheets"
# A producer code is `001` + `1` + USCC(18) + a 5-digit app/product suffix, so two
# codes sharing the USCC are the same legal entity registering different products.
# Slicing is defensive: anything not matching the layout is grouped by its raw value.
_USCC_START, _USCC_END = 4, 22
def uscc_of(code: str) -> str:
"""The 18-char Unified Social Credit Code embedded in a TC260 producer code."""
if len(code) >= _USCC_END and code[:3] == "001":
return code[_USCC_START:_USCC_END]
return code
def _one(path_str: str) -> dict[str, Any] | None:
from remove_ai_watermarks.metadata import aigc_label
try:
label = aigc_label(Path(path_str))
except Exception:
return None
if not label:
return None
producer = str(label.get("ContentProducer") or "")
return {
"path": path_str,
"producer": producer,
"uscc": uscc_of(producer),
"propagator": str(label.get("ContentPropagator") or ""),
"service_provider": str(label.get("ServiceProvider") or ""),
}
def load_fired() -> dict[str, list[str]]:
"""path -> detector keys that fired, from the completed full-corpus artifact."""
if not FIRED.exists():
print(f"WARNING: {FIRED.name} missing; cohorts will show no detector state")
return {}
out: dict[str, list[str]] = {}
for line in FIRED.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
rec = json.loads(line)
out[rec["path"]] = rec.get("keys") or []
return out
def scan(limit: int, workers: int, out_path: Path) -> list[dict[str, Any]]:
pool = sorted(glob.glob(str(CORPUS / "*" / "*")))
if limit:
pool = pool[:limit]
print(f"scanning {len(pool)} corpus files for TC260 labels workers={workers}", flush=True)
rows: list[dict[str, Any]] = []
out_path.parent.mkdir(parents=True, exist_ok=True)
with open(out_path, "w", encoding="utf-8") as fh, ProcessPoolExecutor(max_workers=workers) as ex:
futures = [ex.submit(_one, p) for p in pool]
for i, fut in enumerate(as_completed(futures), 1):
try:
rec = fut.result()
except Exception: # noqa: S112 -- one bad file must not kill the scan
continue
if rec is not None:
fh.write(json.dumps(rec) + "\n")
rows.append(rec)
if i % 5000 == 0:
fh.flush()
print(f" {i}/{len(pool)} carriers={len(rows)}", flush=True)
return rows
def report(rows: list[dict[str, Any]], fired: dict[str, list[str]], min_size: int) -> None:
by_uscc: dict[str, list[dict[str, Any]]] = {}
for r in rows:
by_uscc.setdefault(r["uscc"], []).append(r)
print(f"\n{'=' * 92}\nVENDOR COHORTS ({len(rows)} TC260 carriers, {len(by_uscc)} distinct entities)\n{'=' * 92}")
print("\n`fires` = share of the cohort where SOME registered detector fires.")
print("A large cohort with a low fire rate is an uncovered vendor -- the harvest target.\n")
print(f"{'entity (USCC)':22s} {'n':>6s} {'fires':>7s} {'detectors seen':38s} {'products':>8s}")
print("-" * 92)
cohorts = sorted(by_uscc.items(), key=lambda kv: -len(kv[1]))
for uscc, members in cohorts:
if len(members) < min_size:
continue
keys: Counter[str] = Counter()
hit = 0
for m in members:
ks = fired.get(m["path"], [])
if ks:
hit += 1
keys.update(ks)
seen = ", ".join(f"{k}:{c}" for k, c in keys.most_common(4)) or "-- none --"
products = len({m["producer"] for m in members})
print(f"{uscc:22s} {len(members):6d} {100 * hit / len(members):6.1f}% {seen:38s} {products:8d}")
small = sum(1 for _, m in cohorts if len(m) < min_size)
if small:
print(f"\n({small} cohorts below --min-size {min_size} not shown)")
def _bands(img: Any, width: int, band: int) -> list[Any]:
"""Full-width top and bottom bands, scaled to a readable common width.
An unregistered vendor's placement is unknown, so cropping the bottom-RIGHT
corner (where the marks we already cover happen to sit) would beg the
question. Full-width bands catch any horizontal position, and the two bands
together cover every corner the standard's implementers actually use.
"""
import cv2
h = img.shape[0]
strip = max(24, int(h * band))
out = []
for piece in (img[:strip], img[h - strip :]):
scale = width / max(1, piece.shape[1])
out.append(cv2.resize(piece, (width, max(12, int(piece.shape[0] * scale))), interpolation=cv2.INTER_AREA))
return out
def sheets(rows: list[dict[str, Any]], fired: dict[str, list[str]], per: int, min_size: int) -> None:
"""Top/bottom bands per uncovered cohort, so the vendor and mark can be read off."""
import cv2
import numpy as np
from remove_ai_watermarks.image_io import imread
by_uscc: dict[str, list[dict[str, Any]]] = {}
for r in rows:
by_uscc.setdefault(r["uscc"], []).append(r)
SHEET_DIR.mkdir(parents=True, exist_ok=True)
print(f"\nwriting contact sheets -> {SHEET_DIR}")
for uscc, members in sorted(by_uscc.items(), key=lambda kv: -len(kv[1])):
if len(members) < min_size:
continue
quiet = [m for m in members if not fired.get(m["path"])]
if not quiet:
continue
width = 900
tiles: list[Any] = []
for m in quiet[:per]:
img = imread(m["path"])
if img is None:
continue
for b in _bands(img, width, 0.10):
tiles.append(b)
tiles.append(np.full((3, width, 3), 60, np.uint8))
if tiles:
dest = SHEET_DIR / f"{uscc}_n{len(members)}_quiet{len(quiet)}.png"
cv2.imwrite(str(dest), np.vstack(tiles))
print(f" {dest.name} ({len(tiles) // 4} frames)")
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--limit", type=int, default=0, help="cap files scanned (0 = whole corpus)")
ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 4) - 2))
ap.add_argument("--out", type=Path, default=OUT)
ap.add_argument("--report-only", action="store_true")
ap.add_argument("--min-size", type=int, default=5, help="hide cohorts smaller than this")
ap.add_argument("--sheets", type=int, default=0, help="crops per cohort contact sheet")
a = ap.parse_args()
if a.report_only:
rows = [json.loads(x) for x in a.out.read_text(encoding="utf-8").splitlines() if x.strip()]
else:
rows = scan(a.limit, a.workers, a.out)
fired = load_fired()
report(rows, fired, a.min_size)
if a.sheets:
sheets(rows, fired, a.sheets, a.min_size)
if __name__ == "__main__":
main()
scripts/vendor_mark_calibrate.py
+598
View File
@@ -0,0 +1,598 @@
"""Calibrate a candidate text-mark detector for an UNCOVERED vendor, on real positives.
WHERE THE POSITIVES COME FROM
`vendor_cohort_harvest.py` partitions China-AIGC carriers into per-entity cohorts by
their TC260 producer code, so a cohort is a vendor LABEL that owes nothing to any
pixel detector. That is what makes this calibration non-circular: the previous attempt
(`render_vendor_silhouettes.py`, 2026-07-18) died at n=1 because the only way it knew
to find 千问 frames was to eyeball the misses of a detector that cannot see them.
A cohort is NOT automatically a set of visible-mark positives: TC260 provenance is
metadata, and a vendor may label a frame without stamping it. So the cohort is the
CANDIDATE pool, and mark presence is settled by eye -- `--sheets` writes the corner
crops sorted by score, which makes that pass cheap and makes the separation (or its
absence) visible directly.
NEGATIVES
The 432 frames hand-labelled `present: []` in the 2026-07-18 round -- already-adjudicated
no-visible-mark images, so the false-fire arm rests on human labels rather than on the
absence of a detection.
THE TRAPS, INHERITED FROM THE 2026-07-18 MEASUREMENT
Both are encoded below rather than left to the caller:
* size the template with `alpha_height_frac`, NOT the silhouette's own aspect ratio
(the latter inflated the clean p99 from 0.30 to 0.58 and made comparison meaningless)
* keep the ladder at the shipped 3 rungs -- a wide sweep hands clean corners extra
chances to match, which flatters the positives and the negatives alike
DATA SAFETY
Corpus images are real user uploads: read-only, local, gitignored output. The template
is font-rendered synthetic (`render_vendor_silhouettes.py`), never cut from an upload.
uv run python scripts/vendor_mark_calibrate.py --cohort 91440101MA9Y9T4H7A \\
--asset qwen_alpha.png --sheets
"""
from __future__ import annotations
import argparse
import json
import os
import sys
from concurrent.futures import ProcessPoolExecutor, as_completed
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).parent.parent))
sys.path.insert(0, str(Path(__file__).parent))
REPO = Path(__file__).resolve().parents[1]
COHORTS = REPO / "data" / "spaces" / "_vendor_cohorts.jsonl"
SHEET_DIR = REPO / "data" / "spaces" / "_vendor_calib_sheets"
OUT = REPO / "data" / "spaces" / "_vendor_calibration.jsonl"
def build_config(
asset: str,
name: str,
scale_basis: str = "short",
overrides: dict[str, Any] | None = None,
) -> Any:
"""A candidate config: doubao's tuned geometry with this vendor's silhouette.
Transferring doubao's numbers is justified by LAYOUT, not by hope: every one of these
marks is the same GB 45438-2025 house style -- a 2-glyph vendor prefix, then the
mandated `AI生成` tail, set in a semibold CJK sans in the bottom-right corner. So
`豆包AI生成` and `千问AI生成` are the same 6 glyph cells at the same scale, and the
width/height fractions carry over. The NCC gate does NOT carry over and is what this
script exists to measure. Any tuned value can be overridden with what
`--fit-geometry` measured -- inheriting the locate box blindly clipped the big-mode
qwen mark, which is exactly the trap this tool exists to avoid.
"""
import dataclasses
from remove_ai_watermarks._text_mark_engine import TextMarkConfig
from remove_ai_watermarks.doubao_engine import _CONFIG
return dataclasses.replace(
TextMarkConfig(**dataclasses.asdict(_CONFIG)),
name=name,
asset_name=asset,
scale_basis=scale_basis,
**(overrides or {}),
)
ScoreArgs = tuple[str, str, str, str, "dict[str, Any]"]
def _score(args: ScoreArgs) -> dict[str, Any] | None:
path_str, asset, name, basis, overrides = args
from remove_ai_watermarks._text_mark_engine import TextMarkEngine
from remove_ai_watermarks.image_io import imread
img = imread(path_str)
if img is None or min(img.shape[:2]) < 64:
return None
eng = TextMarkEngine(build_config(asset, name, basis, overrides))
loc = eng.locate(img)
try:
score, box = eng._tophat_best(img, loc)
except Exception:
return None
return {"path": path_str, "score": round(float(score), 4), "box": box}
NEGATIVES = REPO / "data" / "spaces" / "_research_20260718_textmark_relaxation" / "groundtruth.jsonl"
def load_sets(cohort: str) -> tuple[list[str], list[str]]:
pos = [
json.loads(x)["path"]
for x in COHORTS.read_text(encoding="utf-8").splitlines()
if x.strip() and json.loads(x)["uscc"] == cohort
]
# The 2026-07-18 labels are in the vocabulary of the REGISTERED marks only
# (gemini/doubao/jimeng/jimeng_pill): `present: []` means "no registered mark", NOT
# "no mark at all" -- 146 of the 432 sit in a TC260 cohort, and qwen-cohort frames
# visibly carrying 千问AI生成 are labelled `present: []` there (measured 2026-07-21:
# they made up the clean arm's whole top tail, clean p99 0.37 -> 0.69). A gate read
# off that arm is meaningless, so the clean arm excludes every frame in ANY TC260
# cohort -- cohort membership is the cheap proxy for "may carry a CJK AI label".
in_any_cohort = {
str(Path(json.loads(x)["path"]).resolve())
for x in COHORTS.read_text(encoding="utf-8").splitlines()
if x.strip()
}
neg: list[str] = []
dropped = 0
for line in NEGATIVES.read_text(encoding="utf-8").splitlines():
if not line.strip():
continue
rec = json.loads(line)
if rec.get("present"):
continue
p = str((REPO / rec["path"]).resolve())
if p in in_any_cohort:
dropped += 1
continue
neg.append(p)
if dropped:
print(f"clean arm: dropped {dropped} negatives that sit in a TC260 cohort (contamination guard)")
return pos, neg
def run(
paths: list[str],
asset: str,
name: str,
workers: int,
basis: str = "short",
overrides: dict[str, Any] | None = None,
) -> list[dict[str, Any]]:
out: list[dict[str, Any]] = []
with ProcessPoolExecutor(max_workers=workers) as ex:
futs = [ex.submit(_score, (p, asset, name, basis, overrides or {})) for p in paths]
for f in as_completed(futs):
try:
r = f.result()
except Exception: # noqa: S112 -- one bad file must not kill the sweep
continue
if r:
out.append(r)
return out
def report(pos: list[dict[str, Any]], neg: list[dict[str, Any]], name: str) -> None:
import numpy as np
p = np.array([r["score"] for r in pos])
n = np.array([r["score"] for r in neg])
print(f"\n{'=' * 78}\n{name}: candidate-cohort vs hand-labelled clean\n{'=' * 78}")
print(f"\n{'arm':10s} {'n':>5s} {'p10':>7s} {'p50':>7s} {'p90':>7s} {'p95':>7s} {'p99':>7s} {'max':>7s}")
for label, arr in (("cohort", p), ("clean", n)):
if not len(arr):
continue
qs = [np.percentile(arr, q) for q in (10, 50, 90, 95, 99)]
print(f"{label:10s} {len(arr):5d} " + " ".join(f"{q:7.3f}" for q in qs) + f" {arr.max():7.3f}")
if len(p) and len(n):
print("\n\nOPERATING POINTS -- gate set on the CLEAN arm")
print("`cohort fire` is an UPPER BOUND on recall: the cohort also holds")
print("metadata-only frames that carry no visible mark to find.\n")
print(f"{'gate':>7s} {'clean fire':>12s} {'cohort fire':>13s} {'cohort n':>10s}")
for q in (90, 95, 99, 99.5, 100):
t = float(np.percentile(n, q))
cf, pf = 100 * float((n >= t).mean()), 100 * float((p >= t).mean())
print(f"{t:7.3f} {cf:11.2f}% {pf:12.1f}% {int((p >= t).sum()):10d}")
def sheets(pos: list[dict[str, Any]], name: str, per_sheet: int = 24) -> None:
"""Corner crops sorted by score, so mark presence and the gate are read in one pass."""
import cv2
import numpy as np
from remove_ai_watermarks._text_mark_engine import TextMarkEngine
from remove_ai_watermarks.image_io import imread
eng = TextMarkEngine(build_config("doubao_alpha.png", "roi"))
SHEET_DIR.mkdir(parents=True, exist_ok=True)
ranked = sorted(pos, key=lambda r: -r["score"])
width = 660
for start in range(0, len(ranked), per_sheet):
chunk = ranked[start : start + per_sheet]
tiles: list[Any] = []
for rank, r in enumerate(chunk, start + 1):
img = imread(r["path"])
if img is None:
continue
loc = eng.locate(img)
crop = img[loc.y : loc.y + loc.h, loc.x : loc.x + loc.w]
if not crop.size:
continue
tile = cv2.resize(crop, (width, 96), interpolation=cv2.INTER_AREA)
cv2.rectangle(tile, (0, 0), (118, 22), (0, 0, 0), -1)
cv2.putText(tile, f"#{rank} {r['score']:.3f}", (4, 16), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (255, 255, 255), 1)
tiles.append(tile)
tiles.append(np.full((2, width, 3), 70, np.uint8))
if tiles:
dest = SHEET_DIR / f"{name}_ranked_{start // per_sheet:02d}.png"
cv2.imwrite(str(dest), np.vstack(tiles))
print(f" {dest.name} (#{start + 1}..#{start + len(chunk)})")
# Wide and dense, for the geometry FIT only. An unregistered vendor's glyph size is
# genuinely unknown, which is the one case a dense ladder earns its cost -- but it also
# hands clean corners extra chances to match, so it must never set a gate.
_FIT_SCALES = tuple(round(0.4 * (1.03**i), 4) for i in range(80)) # 0.40 .. ~4.1
# The ladder the product actually ships (`_text_mark_engine._tophat_best`).
_SHIPPED_LADDER = (0.8, 1.0, 1.25)
def _fit_one(args: tuple[str, str]) -> dict[str, Any] | None:
"""Best match over the WIDE ladder, reported as a mark width in pixels.
Also measures the template ASPECT at the winning width: the mark's true height is
fitted by sweeping gh at the winning gw and reading the argmax, because
`alpha_height_frac` must be measured, not taken from the silhouette's own aspect
(that inflated the clean p99 from 0.30 to 0.58 on the 2026-07-18 attempt) and not
inherited from doubao.
"""
path_str, asset = args
import cv2
import numpy as np
from remove_ai_watermarks._text_mark_engine import TextMarkEngine
from remove_ai_watermarks.image_io import imread
cfg = build_config(asset, "fit", "width")
eng = TextMarkEngine(cfg)
img = imread(path_str)
if img is None:
return None
loc = eng.locate(img)
resp = eng.tophat_response(img, loc)
sil = eng._glyph_silhouette()
if resp is None or sil is None:
return None
w = img.shape[1]
best, best_gw = 0.0, 0
best_tl = (0, 0)
for s in _FIT_SCALES:
gw = max(cfg.min_gw, int(cfg.alpha_width_frac * w * s))
gh = max(4, int(cfg.alpha_height_frac * w * s))
if gw >= resp.shape[1] or gh >= resp.shape[0]:
continue
t = cv2.resize(sil, (gw, gh), interpolation=cv2.INTER_AREA)
res = cv2.matchTemplate(resp, t, cv2.TM_CCOEFF_NORMED)
_, v, _, tl = cv2.minMaxLoc(res)
if v > best:
best, best_gw, best_tl = v, gw, (int(tl[0]), int(tl[1]))
# Aspect fit at the winning width: sweep gh/gw and keep the argmax. Range covers
# everything between samsung's 0.12 and jimeng's 0.29 house styles, plus slack.
best_aspect = 0.0
if best_gw > 0:
best_gh_score = -1.0
for ratio in np.arange(0.12, 0.42, 0.01):
gh = max(4, int(best_gw * float(ratio)))
if gh >= resp.shape[0]:
continue
t = cv2.resize(sil, (best_gw, gh), interpolation=cv2.INTER_AREA)
v = float(cv2.matchTemplate(resp, t, cv2.TM_CCOEFF_NORMED).max())
if v > best_gh_score:
best_gh_score, best_aspect = v, float(ratio)
# The ABSOLUTE mark rect, so the LOCATE box fractions can be fitted too: inheriting
# doubao's corner box clipped the big-mode qwen mark's first glyph (the qwen mark
# sits ~0.025 of the short side off the right edge, doubao's box assumes ~0.004),
# which collapsed an exact-size template to 0.26.
ax = loc.x + best_tl[0]
ay = loc.y + best_tl[1]
return {
"path": path_str,
"best": round(best, 4),
"mark_w": best_gw,
"aspect": round(best_aspect, 3),
"x": ax,
"y": ay,
"w": w,
"h": img.shape[0],
}
def fit_geometry(paths: list[str], asset: str, workers: int, floor: float = 0.50, paths_name: str = "cohort") -> None:
"""Which basis and fraction does this vendor's mark actually scale with?
Only frames matching above ``floor`` are used: below it the winning size is the
ladder's best fit to background texture, not a measurement of the mark.
"""
import numpy as np
rows: list[dict[str, Any]] = []
with ProcessPoolExecutor(max_workers=workers) as ex:
for f in as_completed([ex.submit(_fit_one, (p, asset)) for p in paths]):
try:
r = f.result()
except Exception: # noqa: S112 -- one bad file must not kill the fit
continue
if r:
rows.append(r)
strong = [r for r in rows if r["best"] >= floor]
fit_out = REPO / "data" / "spaces" / f"_vendor_fit_{paths_name}.jsonl"
fit_out.write_text("\n".join(json.dumps(r) for r in rows), encoding="utf-8")
print(f"\n{'=' * 78}\nGEOMETRY FIT (n={len(rows)}, usable best>={floor}: {len(strong)})\n{'=' * 78}")
print(f"rows -> {fit_out}")
if len(strong) < 20:
print("too few strong frames to fit a basis -- do not ship a fraction off this")
return
mw = np.array([r["mark_w"] for r in strong], float)
w = np.array([r["w"] for r in strong], float)
h = np.array([r["h"] for r in strong], float)
bases = {
"width": w,
"height": h,
"short": np.minimum(w, h),
"long": np.maximum(w, h),
"sqrt(w*h)": np.sqrt(w * h),
"diagonal": np.hypot(w, h),
}
print(f"\n{'basis':12s} {'mean frac':>10s} {'CV':>8s} {'p10':>8s} {'p90':>8s}")
print("-" * 52)
for nm, b in sorted(bases.items(), key=lambda kv: float(np.std(mw / kv[1]) / np.mean(mw / kv[1]))):
r = mw / b
print(
f"{nm:12s} {np.mean(r):10.4f} {float(np.std(r) / np.mean(r)):8.3f} "
f"{np.percentile(r, 10):8.4f} {np.percentile(r, 90):8.4f}"
)
lo_l, hi_l = _SHIPPED_LADDER[0], _SHIPPED_LADDER[-1]
print(f"\nCoverage by a single fraction on the SHIPPED ladder ({lo_l} .. {hi_l}, span {hi_l / lo_l:.3f}x):")
print(f"{'basis':12s} {'frac':>7s} {'window':>16s} {'covered':>9s}")
for nm in ("short", "sqrt(w*h)", "width"):
fs = mw / bases[nm]
best_f, best_cov = 0.0, -1.0
for f in np.arange(float(fs.min()) * 0.9, float(fs.max()) * 1.1, 0.002):
cov = float(((fs >= f * lo_l) & (fs <= f * hi_l)).mean())
if cov > best_cov:
best_f, best_cov = float(f), cov
print(f"{nm:12s} {best_f:7.3f} {best_f * lo_l:7.3f}-{best_f * hi_l:.3f} {100 * best_cov:8.1f}%")
# The raw distribution behind the coverage number: where the mark actually sits,
# so the mode structure (and what a 4th rung would recover) is visible directly.
fs = mw / bases["short"]
qs = [np.percentile(fs, q) for q in (5, 25, 50, 75, 95)]
print(f"\nfrac_short distribution: p5 {qs[0]:.3f} p25 {qs[1]:.3f} p50 {qs[2]:.3f} p75 {qs[3]:.3f} p95 {qs[4]:.3f}")
hist, edges = np.histogram(fs, bins=16)
for c, e0, e1 in zip(hist, edges[:-1], edges[1:], strict=True):
print(f" {e0:.3f}-{e1:.3f} {'#' * c}")
# Template aspect at the winning width -> the alpha_height_frac recommendation.
# Measured, per the standing rule: not the silhouette's own aspect, not doubao's.
aspects = np.array([r["aspect"] for r in strong if r["aspect"] > 0], float)
if len(aspects) >= 20:
med = float(np.median(aspects))
print(
f"\nASPECT FIT (n={len(aspects)}): p10 {np.percentile(aspects, 10):.3f} "
f"p50 {med:.3f} p90 {np.percentile(aspects, 90):.3f}"
)
print("alpha_height_frac = alpha_width_frac * p50(aspect), per basis:")
for nm in ("short", "sqrt(w*h)", "width"):
fxs = mw / bases[nm]
best_f = max(
np.arange(float(fxs.min()) * 0.9, float(fxs.max()) * 1.1, 0.002),
key=lambda f: float(((fxs >= f * lo_l) & (fxs <= f * hi_l)).mean()),
)
print(f" {nm:12s} width {best_f:.3f} -> height {best_f * med:.4f}")
# LOCATE-box fit. The box fractions are as mark-specific as the template size:
# doubao's box clipped qwen's big-mode mark (see _fit_one). Derive the box from the
# measured absolute mark rects: margins must not exceed the mark's own (else the
# mark exits the anchored box), and the box must cover the mark plus NCC slack.
if len(aspects) >= 20:
short = np.minimum(w, h).astype(float)
mark_h = np.array([r["mark_w"] * r["aspect"] for r in strong], float)
ax = np.array([r["x"] for r in strong], float)
ay = np.array([r["y"] for r in strong], float)
right = (w - (ax + mw)) / short # frame right edge to mark right edge
bottom = (h - (ay + mark_h)) / short
print(f"\nLOCATE FIT (basis=short, n={len(strong)}):")
print(f" right-margin frac p5 {np.percentile(right, 5):.4f} p50 {np.percentile(right, 50):.4f}")
print(f" bottom-margin frac p5 {np.percentile(bottom, 5):.4f} p50 {np.percentile(bottom, 50):.4f}")
print(
f" mark height frac p50 {np.percentile(mark_h / short, 50):.4f} "
f"p95 {np.percentile(mark_h / short, 95):.4f}"
)
mx = max(0.002, float(np.percentile(right, 5)) - 0.004)
mb = max(0.002, float(np.percentile(bottom, 5)) - 0.004)
need_w = float(np.percentile(mw / short + right, 95)) - mx + 0.02
need_h = float(np.percentile(mark_h / short + bottom, 95)) - mb + 0.015
print(f" recommended: margin_x_frac={mx:.4f} margin_bottom_frac={mb:.4f}")
print(f" width_frac={need_w:.3f} height_frac={need_h:.3f}")
print("\nThese are DIAGNOSTIC. Re-score both arms on the shipped ladder with the")
print("fitted geometry before reading any gate off the clean arm.")
FIRED = REPO / "data" / "spaces" / "_visible_positives.jsonl"
def _fired_pool(mark: str, limit: int, seed: int = 7) -> list[str]:
"""Paths where ``mark`` fired, from the COMPLETED full-corpus artifact -- the
standing rule: detector firings are joined, never re-run."""
import random
pool = [
json.loads(x)["path"]
for x in FIRED.read_text(encoding="utf-8").splitlines()
if x.strip() and mark in (json.loads(x).get("keys") or [])
]
rng = random.Random(seed) # noqa: S311 -- reproducible sampling, not crypto
rng.shuffle(pool)
return pool[:limit]
def _cross_score(args: tuple[str, Any, Any]) -> dict[str, Any] | None:
"""One frame scored by BOTH the candidate and the doubao production configs."""
path_str, cand_cfg, db_cfg = args
from remove_ai_watermarks._text_mark_engine import TextMarkEngine
from remove_ai_watermarks.image_io import imread
img = imread(path_str)
if img is None or min(img.shape[:2]) < 200:
return None
out: dict[str, Any] = {"path": path_str}
for key, cfg in (("cand", cand_cfg), ("doubao", db_cfg)):
eng = TextMarkEngine(cfg)
loc = eng.locate(img)
try:
score, _ = eng._tophat_best(img, loc)
except Exception:
return None
out[key] = round(float(score), 4)
return out
def crossfire(
pools: dict[str, list[str]],
cand_cfg: Any,
workers: int,
gate: float,
margin: float = 0.10,
) -> None:
"""Score the candidate AND doubao's production template on the same frames.
The registration question a cohort-vs-clean run cannot answer: the candidate shares
the mandated `AI生成` tail with doubao (4 of 6 glyph cells), so its template will
correlate with doubao marks too. If the candidate fires on the doubao pool at the
candidate gate, registering it double-fills every doubao frame and mislabels it --
unless the rival margin suppresses it, which then has to be shown NOT to kill the
candidate on its own marks. Measured here in the tophat domain (the gate's domain);
production's `_rival_margin_ok` runs the same comparison on the binary blob.
"""
import numpy as np
from remove_ai_watermarks.doubao_engine import _CONFIG as db_cfg
print(f"\n{'=' * 78}\nCROSSFIRE -- candidate vs doubao, same frames, tophat domain\n{'=' * 78}")
print(f"candidate gate {gate:.3f} | rival margin {margin:.2f}\n")
print(
f"{'pool':8s} {'n':>5s} {'cand p50':>9s} {'cand p90':>9s} {'db p50':>7s} {'db p90':>7s} "
f"{'m-d p10':>8s} {'m-d p50':>8s} {'fire':>7s} {'fire+m':>7s}"
)
with ProcessPoolExecutor(max_workers=workers) as ex:
for pool_name, paths in pools.items():
rows: list[dict[str, Any]] = []
futs = [ex.submit(_cross_score, (p, cand_cfg, db_cfg)) for p in paths]
for f in as_completed(futs):
try:
r = f.result()
except Exception: # noqa: S112 -- one bad file must not kill the pool
continue
if r:
rows.append(r)
if not rows:
continue
c = np.array([r["cand"] for r in rows])
d = np.array([r["doubao"] for r in rows])
m = c - d
fire = c >= gate
fire_m = fire & (m >= margin)
print(
f"{pool_name:8s} {len(rows):5d} {np.percentile(c, 50):9.3f} {np.percentile(c, 90):9.3f} "
f"{np.percentile(d, 50):7.3f} {np.percentile(d, 90):7.3f} "
f"{np.percentile(m, 10):8.3f} {np.percentile(m, 50):8.3f} "
f"{100 * float(fire.mean()):6.1f}% {100 * float(fire_m.mean()):6.1f}%"
)
print("\nReading: on `qwen` the margin column must stay high (the candidate keeps its")
print("own marks); on `doubao` fire+m must sit near zero (the candidate stays off")
print("doubao marks). If fire is high on `doubao` and fire+m is not, the rival margin")
print("is load-bearing for registration; if both are high, the mark cannot be")
print("registered on this front-end at all.")
def _parse_ladder(raw: str) -> tuple[float, ...] | None:
if not raw:
return None
return tuple(float(x) for x in raw.split(","))
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--cohort", required=True, help="cohort USCC from vendor_cohort_harvest.py")
ap.add_argument("--asset", required=True, help="silhouette asset name, e.g. qwen_alpha.png")
ap.add_argument("--name", default="", help="label for output files (defaults to the asset stem)")
ap.add_argument("--workers", type=int, default=max(1, (os.cpu_count() or 4) - 2))
ap.add_argument("--scale-basis", choices=("short", "width"), default="short")
ap.add_argument("--sheets", action="store_true")
ap.add_argument(
"--fit-geometry",
action="store_true",
help="fit the basis + fraction + template aspect the mark scales with",
)
ap.add_argument("--width-frac", type=float, default=None, help="fitted alpha_width_frac (default: inherit doubao)")
ap.add_argument(
"--height-frac", type=float, default=None, help="fitted alpha_height_frac (default: inherit doubao)"
)
ap.add_argument("--ladder", default="", help="comma scale rungs, e.g. 0.8,1.0,1.25,1.6 (default: shipped 3)")
ap.add_argument("--gate", type=float, default=0.45, help="candidate gate for the crossfire fire rates")
ap.add_argument("--box-width-frac", type=float, default=None, help="fitted locate width_frac")
ap.add_argument("--box-height-frac", type=float, default=None, help="fitted locate height_frac")
ap.add_argument("--margin-x-frac", type=float, default=None, help="fitted locate margin_x_frac")
ap.add_argument("--margin-bottom-frac", type=float, default=None, help="fitted locate margin_bottom_frac")
ap.add_argument(
"--crossfire",
action="store_true",
help="score the candidate AND doubao on the cohort, the doubao/jimeng pools and the clean arm",
)
a = ap.parse_args()
name = a.name or a.asset.split("_")[0]
ladder = _parse_ladder(a.ladder)
overrides: dict[str, Any] = {}
for arg, field in (
(a.width_frac, "alpha_width_frac"),
(a.height_frac, "alpha_height_frac"),
(a.box_width_frac, "width_frac"),
(a.box_height_frac, "height_frac"),
(a.margin_x_frac, "margin_x_frac"),
(a.margin_bottom_frac, "margin_bottom_frac"),
):
if arg is not None:
overrides[field] = arg
if ladder is not None:
overrides["ladder"] = ladder
pos_paths, neg_paths = load_sets(a.cohort)
if a.fit_geometry:
print(f"cohort {a.cohort}: {len(pos_paths)} candidates")
fit_geometry(pos_paths, a.asset, a.workers, paths_name=name)
return
if a.crossfire:
cand = build_config(a.asset, name, a.scale_basis, overrides)
pools = {
"qwen": pos_paths,
"doubao": _fired_pool("doubao", 400),
"jimeng": _fired_pool("jimeng", 300),
"clean": neg_paths,
}
print("pools: " + ", ".join(f"{k}={len(v)}" for k, v in pools.items()))
crossfire(pools, cand, a.workers, a.gate)
return
print(f"cohort {a.cohort}: {len(pos_paths)} candidates | clean: {len(neg_paths)} hand-labelled")
print(f"scale_basis={a.scale_basis} overrides={overrides}")
pos = run(pos_paths, a.asset, name, a.workers, a.scale_basis, overrides)
neg = run(neg_paths, a.asset, name, a.workers, a.scale_basis, overrides)
OUT.write_text(
"\n".join(json.dumps({**r, "arm": arm}) for arm, rows in (("cohort", pos), ("clean", neg)) for r in rows),
encoding="utf-8",
)
report(pos, neg, name)
if a.sheets:
print(f"\nsheets -> {SHEET_DIR}")
sheets(pos, name)
if __name__ == "__main__":
main()
src/remove_ai_watermarks/_text_mark_engine.py
+12 -3
View File
@@ -131,6 +131,14 @@ class TextMarkConfig:
# Which image dimension the mark's size and margins scale with. VENDOR-SPECIFIC,
# measured, not assumed -- see TextMarkEngine.scale_base. "short" = min(h, w), "width" = w.
scale_basis: Literal["short", "width"] = "width"
# Scale rungs ``_tophat_best`` sweeps (the detection comb). PER-MARK: a vendor
# whose stamp sizes do not land on the shared 3-rung comb carries its own ladder
# (measured for 千问, whose marks sit in two size modes ~1.6x apart -- one fraction
# on 3 rungs covers only ~75% of them). Densifying the SHARED ladder for everyone
# was measured and rejected (false fire 2.52% -> 3.05%; see docs/verification-plan.md
# B2), so the default stays the shipped 3 rungs and a deviation must be calibrated
# per mark on real positives, never ported.
ladder: tuple[float, ...] = (0.8, 1.0, 1.25)
rivals: tuple[str, ...] = ()
rival_margin: float = 0.10
# Multiplier applied to detect_ncc_threshold when provenance confirms the vendor.
@@ -345,9 +353,10 @@ class TextMarkEngine:
"""Best TM_CCOEFF_NORMED of a soft template against the continuous response, and
the ROI-local box (x0, y0, x1, y1) where that best match sits.
Sweeps a small scale band: the nominal glyph size is derived from the mark's
Sweeps the mark's scale ladder: the nominal glyph size is derived from the mark's
geometry, but a vendor re-rasterization shifts it by a few percent and the
continuous response is sharp enough that an exact-size template would miss.
continuous response is sharp enough that an exact-size template would miss. The
ladder is per-mark (``TextMarkConfig.ladder``), defaulting to the shipped 3 rungs.
Detection and the removal mask BOTH read this one method -- the score gates
detection, the box bounds the fill. Sharing it is deliberate: the standing rule is
@@ -363,7 +372,7 @@ class TextMarkEngine:
base = self.scale_base(image)
best_score = 0.0
best_box: tuple[int, int, int, int] | None = None
for scale in (0.8, 1.0, 1.25):
for scale in c.ladder:
gw = max(c.min_gw, int(c.alpha_width_frac * base * scale))
gh = max(4, int(c.alpha_height_frac * base * scale))
if gw >= resp.shape[1] or gh >= resp.shape[0]:
src/remove_ai_watermarks/assets/qwen_alpha.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 3.4 KiB

src/remove_ai_watermarks/assets/xinghui_alpha.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 4.1 KiB

src/remove_ai_watermarks/cli.py
+2 -2
View File
@@ -430,7 +430,7 @@ def _no_visible_mark_exit(source: Path) -> NoReturn:
"""Explain why no visible watermark was removed, then exit non-zero.
The visible registry handles only known visual marks (the Gemini sparkle and
the Doubao/Jimeng/Samsung text strips). Most real uploads carry no such mark
the Doubao/Jimeng/Qwen/Samsung text strips). Most real uploads carry no such mark
-- frequently an invisible/metadata watermark instead (e.g. an OpenAI or
Gemini image whose only signal is C2PA + SynthID). Returning the input
unchanged with exit 0 reads as success to a caller and re-serves the
@@ -692,7 +692,7 @@ def cmd_visible(
) -> None:
"""Remove a known visible AI watermark from an image.
Finds a known mark in its usual place (Gemini sparkle / Doubao-Jimeng-Samsung
Finds a known mark in its usual place (Gemini sparkle / Doubao-Jimeng-Qwen-Samsung
text) via the watermark registry and removes it by LOCALIZING the mark to a mask
and filling that mask with the chosen ``--backend`` (auto: best available, LaMa >
MI-GAN > cv2). ``--mark auto`` removes every detected mark in one
src/remove_ai_watermarks/identify.py
+2 -1
View File
@@ -446,12 +446,13 @@ def _visible_sparkle(image_path: Path, *, image: NDArray[Any] | None = None) ->
_VISIBLE_MARK_PLATFORM = {
"doubao": "ByteDance Doubao (visible 豆包AI生成 mark detected)",
"jimeng": "ByteDance Jimeng / Dreamina (visible 即梦AI mark detected)",
"qwen": "Alibaba Tongyi Qianwen (visible 千问AI生成 mark detected)",
"samsung": "Samsung Galaxy AI (visible 'Contenuti generati dall'AI' mark detected)",
}
def _visible_text_marks(image_path: Path, *, image: NDArray[Any] | None = None) -> list[MarkDetection]:
"""Detected visible Doubao/Jimeng marks (registry ``MarkDetection`` list).
"""Detected visible text marks (registry ``MarkDetection`` list).
The Gemini sparkle keeps its own ``_visible_sparkle`` path (file-level
confidence); these two text marks reuse the registry detectors, which apply
src/remove_ai_watermarks/qwen_engine.py
+157
View File
@@ -0,0 +1,157 @@
"""Qwen (Tongyi Qianwen, Alibaba) visible watermark detector/localizer.
Qwen stamps its generations with a visible "千问AI生成" text strip in the
bottom-right corner -- the explicit AIGC label mandated by China's GB 45438-2025
(the same 6-glyph house style as Doubao's "豆包AI生成": a 2-glyph vendor prefix
plus the shared `AI生成` tail), preceded by the vendor's tri-lobe logo (not part
of the detection silhouette -- logos vary between releases, the CJK run is what
discriminates).
Detection matches the bundled glyph silhouette against the corner; removal is the
shared **localize -> fill** (the glyph-bbox :meth:`footprint_mask` feeds
``region_eraser``), NOT reverse-alpha. This module supplies only Qwen's tuned
:class:`TextMarkConfig` (``assets/qwen_alpha.png`` -- a font-rendered synthetic
silhouette from ``scripts/render_vendor_silhouettes.py``, never cut from an
upload). It also feeds ``identify`` as the medium-confidence ``visible_qwen``
signal via the registry.
EVERY tuned number below was measured on the vendor cohort (117 TC260 carriers
whose producer USCC 91440101MA9Y9T4H7A names the entity, 2026-07-21; harness
``scripts/vendor_mark_calibrate.py``), NOT inherited from Doubao:
* The mark sits in TWO size modes (frac of the short side ~0.124 and ~0.203,
ratio 1.64 -- wider than the shared 3-rung ladder's 1.5625 span), so a single
fraction on the shared ladder covers ~75% of marks and the rest land in the
comb's collapse zone. Qwen therefore carries its OWN 2-rung ladder
(``TextMarkConfig.ladder``), one rung centred on each mode; the shared
default is untouched for every other mark.
* The mark also sits FARTHER off the corner than Doubao's box assumes (right
margin ~0.025 vs 0.004 of the short side), so Doubao's locate box clipped the
first glyph and collapsed an exact-size template to 0.26; the box fractions
below are fitted from the measured absolute mark rects.
* ``alpha_height_frac`` comes from the aspect fit at the winning width (p50
aspect 0.26), not from the silhouette's own aspect (0.2219) and not from
Doubao's ratio.
* STRICT ONLY (``provenance_ncc_factor`` 1.0): the score band just below the
gate is dominated by non-Qwen banners on same-cohort frames (a 夸克
anti-forgery strip at 0.274, a 造点 mark at 0.253), so a provenance-relaxed
arm would be mostly false fills. No provenance relaxation exists for this
mark.
* No rival margin: at the shipped gate the template fires on 0 of 400
Doubao-marked frames, 0 of 298 Jimeng-marked frames and 0 of 286 hand-labelled
clean frames (the shared tail correlates at ~0.22, far below the gate), while
a 0.10 rival margin would have suppressed ~10% of genuine Qwen detections.
"""
# The module-level _alpha_template / _glyph_silhouette / _template_match_score below
# are thin test-facing shims (imported by tests/), so pyright's src-only pass sees them
# as unused; the use is cross-module.
# pyright: reportUnusedFunction=false
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from remove_ai_watermarks import _text_mark_engine
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine
if TYPE_CHECKING:
from pathlib import Path
from numpy.typing import NDArray
# Locate geometry as a fraction of the image SHORT side (measured basis -- see
# scale_base). The box is fitted to the measured mark rects: the mark's right
# margin is ~0.025 of the short side (not Doubao's 0.004), so the box anchor is
# wider off the corner; width/height cover the big size mode plus NCC slack.
WM_WIDTH_FRAC = 0.231
WM_HEIGHT_FRAC = 0.074
MARGIN_RIGHT_FRAC = 0.0203
MARGIN_BOTTOM_FRAC = 0.0218
# Glyph appearance: a light, low-saturation gray rendered brighter than the local
# background (white top-hat), same overlay class as Doubao -- inherited, and
# harmless because the tophat front-end turns these gates into weights.
MAX_SATURATION = 55
LOGO_MIN_LUMA = 150
TOPHAT_DELTA = 12
DETECT_MIN_COVERAGE = 0.04 # unused by the tophat front-end (kept for config parity)
# Calibrated 2026-07-21 on the vendor cohort vs 286 hand-labelled clean frames
# (cohort-contamination-guarded): clean p99 0.301 / max 0.316, and every cohort
# frame scoring >= 0.45 carries a visible 千问AI生成 mark (86% of the eyeballed
# visible marks fire, the misses being white-on-near-white contrast losses).
# 0.45 was picked over 0.32 (identical clean fire) for margin against unseen
# clean content at zero measured recall cost.
DETECT_NCC_THRESHOLD = 0.45
# Detection-silhouette geometry (fraction of the short side), fitted on the
# cohort: the mark's width modes and its aspect (0.26) at the winning width.
_ALPHA_WIDTH_FRAC = 0.160
_ALPHA_HEIGHT_FRAC = 0.0416
# The two measured size modes as scale rungs: 0.124 and 0.203 of the short side,
# expressed against the 0.160 nominal. Measured, not rounded: off-mode rungs drop
# NCC from ~0.73 to ~0.37 on real marks (the comb), and a 4-rung variant scored
# strictly worse (the extra rungs cover nothing and the big mode lands 4.6% off
# its nearest rung).
_LADDER = (0.78, 1.27)
_CONFIG = TextMarkConfig(
name="Qwen",
asset_name="qwen_alpha.png",
corner="br",
margin_floor=4,
width_frac=WM_WIDTH_FRAC,
height_frac=WM_HEIGHT_FRAC,
margin_x_frac=MARGIN_RIGHT_FRAC,
margin_bottom_frac=MARGIN_BOTTOM_FRAC,
max_saturation=MAX_SATURATION,
logo_min_luma=LOGO_MIN_LUMA,
tophat_delta=TOPHAT_DELTA,
morph_open_size=5,
detect_min_coverage=DETECT_MIN_COVERAGE,
detect_ncc_threshold=DETECT_NCC_THRESHOLD,
detect_frontend="tophat",
scale_basis="short", # measured: frac_short CV 0.189 vs width 0.273
ladder=_LADDER,
alpha_width_frac=_ALPHA_WIDTH_FRAC,
alpha_height_frac=_ALPHA_HEIGHT_FRAC,
min_gw=8,
# STRICT ONLY: the sub-gate band is dominated by non-Qwen banners, so
# provenance relaxation is disabled outright (factor 1.0 = never relaxed).
provenance_ncc_factor=1.0,
)
QwenDetection = TextMarkDetection
def _alpha_template() -> NDArray[Any] | None:
"""The bundled Qwen alpha template (float [0,1]), or None."""
return _text_mark_engine.load_alpha_template(_CONFIG.asset_name)
def _glyph_silhouette() -> NDArray[Any] | None:
"""Binary "千问AI生成" silhouette (255 = glyph) from the alpha map, or None."""
return _text_mark_engine.glyph_silhouette(_CONFIG.asset_name)
def _template_match_score(box_mask: NDArray[Any], scale_base: int) -> float:
"""TM_CCOEFF_NORMED of the Qwen glyph silhouette against ``box_mask``."""
return _text_mark_engine.template_match_score(box_mask, scale_base, _CONFIG)
class QwenEngine(TextMarkEngine):
"""Detect/localize the visible Qwen "千问AI生成" watermark (locate -> mask; mask feeds the fill)."""
def __init__(self) -> None:
super().__init__(_CONFIG)
def load_image_bgr(path: str | Path) -> NDArray[Any]:
"""Read an image as BGR ndarray (helper for scripts/tests)."""
from remove_ai_watermarks import image_io
img = image_io.imread(path)
if img is None:
raise FileNotFoundError(f"Failed to read image: {path}")
return img
src/remove_ai_watermarks/watermark_registry.py
+11 -3
View File
@@ -20,6 +20,7 @@ Entries:
- ``gemini`` -- Google Gemini / Nano Banana sparkle, bottom-right.
- ``doubao`` -- ByteDance Doubao "豆包AI生成" text strip, bottom-right.
- ``jimeng`` -- ByteDance Jimeng / Dreamina "★ 即梦AI" wordmark, bottom-right.
- ``qwen`` -- Alibaba Tongyi Qianwen "千问AI生成" text strip, bottom-right.
- ``samsung`` -- Samsung Galaxy AI "Contenuti generati dall'AI" strip, bottom-left.
- ``jimeng_pill`` -- Jimeng-basic "AI生成" pill, top-left (capture-less).
"""
@@ -83,6 +84,7 @@ _PRODUCT_OF: dict[str, str] = {
"doubao": "doubao",
"jimeng": "jimeng",
"jimeng_pill": "jimeng", # same product as the Jimeng wordmark
"qwen": "qwen",
"samsung": "samsung",
}
@@ -353,6 +355,10 @@ def _engine(key: str) -> Any:
from remove_ai_watermarks.jimeng_engine import JimengEngine
_engines[key] = JimengEngine()
elif key == "qwen":
from remove_ai_watermarks.qwen_engine import QwenEngine
_engines[key] = QwenEngine()
elif key == "samsung":
from remove_ai_watermarks.samsung_engine import SamsungEngine
@@ -502,6 +508,7 @@ _REGISTRY: tuple[KnownMark, ...] = (
KnownMark("gemini", "Google Gemini sparkle", "bottom-right", True, _gemini_detect, _gemini_mask),
_text_mark("doubao", "Doubao 豆包AI生成 text", "bottom-right"),
_text_mark("jimeng", "Jimeng 即梦AI wordmark", "bottom-right"),
_text_mark("qwen", "Qwen 千问AI生成 text", "bottom-right"),
_text_mark("samsung", "Samsung Galaxy AI text", "bottom-left"),
KnownMark("jimeng_pill", "Jimeng AI生成 pill", "top-left", True, _pill_detect, _pill_mask, _pill_features),
)
@@ -580,9 +587,10 @@ def _keep_pill(keys: set[str], *, provenance: frozenset[str], footprint_flat: bo
so real flat-scene pills (and harmless flat false fires) are cleaned while the
damaging textured false fires are left untouched.
A Doubao image is TC260 too but is not Jimeng-basic, so the pill never rides on a
Doubao detection. No confirmation at all -> never remove (blocks false fires on
non-Jimeng content)."""
if "doubao" in keys:
Doubao detection; a Qwen image likewise (another vendor's bottom-right mark naming
its own product), so a confident Qwen detection suppresses the pill the same way.
No confirmation at all -> never remove (blocks false fires on non-Jimeng content)."""
if "doubao" in keys or "qwen" in keys:
return False
if "jimeng" in keys:
return True
tests/test_qwen_engine.py
+155
View File
@@ -0,0 +1,155 @@
"""Tests for the Qwen (千问AI生成) visible-watermark engine (localize -> fill).
Every tuned constant in ``qwen_engine`` was measured on the 117-frame vendor
cohort (2026-07-21, ``scripts/vendor_mark_calibrate.py``); these tests pin the
load-bearing ones so a later "cleanup" cannot silently re-inherit Doubao's
geometry (the exact failure the calibration had to fix).
"""
from __future__ import annotations
import cv2
import numpy as np
import pytest
from remove_ai_watermarks import watermark_registry as registry
from remove_ai_watermarks.qwen_engine import (
_ALPHA_HEIGHT_FRAC,
_ALPHA_WIDTH_FRAC,
_LADDER,
QwenEngine,
_alpha_template,
_glyph_silhouette,
)
# The two measured size modes (fraction of the short side): a single fraction on
# the shared 3-rung ladder covers only ~75% of marks; the per-mark 2-rung ladder
# centres one rung on each mode.
_BIG_MODE, _SMALL_MODE = 0.203, 0.124
_MARGIN = 0.025 # measured right/bottom margin of the real mark
def _compose(w: int, h: int, mode: float = _BIG_MODE, bg: float = 100.0):
"""Composite the Qwen silhouette at a measured size mode onto a flat bg."""
img = np.full((h, w, 3), bg, np.float32)
at = _alpha_template()
short = min(w, h)
gw = int(mode * short)
gh = max(4, int(mode * (_ALPHA_HEIGHT_FRAC / _ALPHA_WIDTH_FRAC) * short))
margin = int(_MARGIN * short)
ax = w - margin - gw
ay = h - margin - gh
amap = np.zeros((h, w), np.float32)
amap[ay : ay + gh, ax : ax + gw] = cv2.resize(at, (gw, gh))
a3 = amap[:, :, None]
wm = (a3 * 255.0 + (1 - a3) * img).clip(0, 255).astype(np.uint8)
return wm, amap > 0.2
class TestLocate:
def test_box_anchored_bottom_right_off_the_corner(self):
# The measured right margin (~0.025 of short) is wider than Doubao's 0.004;
# inheriting Doubao's anchor clipped the first glyph (0.73 -> 0.26 NCC).
eng = QwenEngine()
img = np.zeros((2048, 2048, 3), np.uint8)
loc = eng.locate(img)
assert 2048 - (loc.x + loc.w) == pytest.approx(2048 * 0.0203, rel=0.15)
assert 2048 - (loc.y + loc.h) == pytest.approx(2048 * 0.0218, rel=0.15)
def test_box_scales_with_short_side_not_width(self):
# scale_basis="short" (measured: frac_short CV 0.189 vs width 0.273).
eng = QwenEngine()
landscape = eng.locate(np.zeros((640, 1280, 3), np.uint8))
wider = eng.locate(np.zeros((640, 2560, 3), np.uint8))
assert wider.w == landscape.w # same short side -> same box
bigger = eng.locate(np.zeros((1280, 1920, 3), np.uint8)) # 2x the short side
assert bigger.w == pytest.approx(landscape.w * 2, rel=0.05)
class TestConfig:
def test_per_mark_ladder_and_shared_default_untouched(self):
# Qwen's two size modes need their own 2-rung ladder; every other mark must
# keep the shipped 3-rung default (the field's whole point is per-mark).
assert _LADDER == (0.78, 1.27)
assert QwenEngine().config.ladder == (0.78, 1.27)
from remove_ai_watermarks.doubao_engine import _CONFIG as db
from remove_ai_watermarks.jimeng_engine import _CONFIG as jm
from remove_ai_watermarks.samsung_engine import _CONFIG as ss
assert db.ladder == jm.ladder == ss.ladder == (0.8, 1.0, 1.25)
def test_strict_only_no_provenance_relaxation(self):
# The sub-gate band is dominated by non-Qwen banners on same-cohort frames,
# so the relaxed arm was measured to be mostly false fills: factor pinned 1.0.
assert QwenEngine().config.provenance_ncc_factor == 1.0
def test_registry_row(self):
mark = registry.get_mark("qwen")
assert mark.location == "bottom-right"
assert "千问AI生成" in mark.label
assert mark.in_auto
class TestDetect:
def test_clean_gradient_not_detected(self):
eng = QwenEngine()
ramp = np.tile(np.linspace(0, 255, 1024, dtype=np.uint8), (1024, 1))
img = cv2.cvtColor(ramp, cv2.COLOR_GRAY2BGR)
assert not eng.detect(img).detected
def test_solid_blob_corner_not_detected(self):
eng = QwenEngine()
img = np.zeros((1024, 1024, 3), np.uint8)
x, y, bw, bh = eng.locate(img).bbox
img[y + bh // 4 : y + bh * 3 // 4, x : x + bw // 2] = 200
assert not eng.detect(img).detected
def test_silhouette_loads(self):
sil = _glyph_silhouette()
assert sil is not None
assert set(np.unique(sil)).issubset({0, 255})
@pytest.mark.parametrize("mode", [_BIG_MODE, _SMALL_MODE])
def test_both_size_modes_detected(self, mode):
# The registration's core claim: a mark at EITHER measured mode scores over
# the gate (a single fraction on the shared ladder lost the small mode).
# The floor is deliberately far above the gate: the synthetic mark is clean,
# so it scores ~0.88/~0.94 when the geometry is right, but ~0.49 with Doubao's
# box margins (the first glyph is clipped) and ~0.67 on the shared 3-rung
# ladder -- the floor is what makes this test discriminate both regressions
# (every variant passes a bare gate check on the synthetic).
wm, _ = _compose(853, 640, mode=mode)
det = QwenEngine().detect(wm)
assert det.detected
assert det.confidence >= 0.80
def test_small_image_guarded(self):
wm, _ = _compose(853, 640)
eng = QwenEngine()
assert eng.detect(wm).detected
assert not eng.detect(cv2.resize(wm, (150, 112))).detected
class TestFootprintMaskAndRemoval:
@pytest.mark.parametrize("mode", [_BIG_MODE, _SMALL_MODE])
def test_removes_composed_mark_at_both_modes(self, mode):
wm, mark = _compose(853, 640, mode=mode)
assert float(np.abs(wm.astype(np.float32)[mark] - 100.0).mean()) > 15 # mark visible
assert QwenEngine().detect(wm).detected
out, region = registry.get_mark("qwen").remove(wm, backend="cv2")
assert region is not None
assert not QwenEngine().detect(out).detected
h, w = wm.shape[:2]
assert np.array_equal(out[: h // 2, : w // 2], wm[: h // 2, : w // 2]) # far region exact
def test_footprint_mask_in_bottom_right(self):
wm, _ = _compose(853, 640)
mask = QwenEngine().footprint_mask(wm)
assert mask is not None
ys, xs = np.where(mask > 0)
assert ys.mean() > wm.shape[0] / 2
assert xs.mean() > wm.shape[1] / 2
def test_clean_frame_produces_no_mask(self):
clean = cv2.GaussianBlur(np.full((640, 853, 3), 120, np.uint8), (5, 5), 0)
assert QwenEngine().footprint_mask(clean, force=False) is None
tests/test_watermark_registry.py
+14 -3
View File
@@ -14,7 +14,7 @@ DOUBAO_SAMPLE = Path(__file__).resolve().parents[1] / "data" / "samples" / "doub
class TestCatalog:
def test_keys(self):
assert reg.mark_keys() == ["gemini", "doubao", "jimeng", "samsung", "jimeng_pill"]
assert reg.mark_keys() == ["gemini", "doubao", "jimeng", "qwen", "samsung", "jimeng_pill"]
def test_all_in_auto(self):
assert all(m.in_auto for m in reg.known_marks())
@@ -43,7 +43,7 @@ class TestScan:
def test_detect_marks_scans_all(self):
img = np.zeros((256, 256, 3), np.uint8)
keys = {d.key for d in reg.detect_marks(img)}
assert keys == {"gemini", "doubao", "jimeng", "samsung", "jimeng_pill"}
assert keys == {"gemini", "doubao", "jimeng", "qwen", "samsung", "jimeng_pill"}
def test_blank_image_no_auto_mark(self):
dets = reg.detect_marks(np.zeros((256, 256, 3), np.uint8), include_explicit=False)
@@ -67,7 +67,7 @@ class TestScan:
forced remove on a zero-size ndarray crashed (cv2.error on an empty Mat). detect
already guarded this; footprint_mask must too. Covers the text + gemini engines."""
empty = np.zeros(shape, np.uint8)
for key in ("doubao", "jimeng", "samsung", "gemini"):
for key in ("doubao", "jimeng", "qwen", "samsung", "gemini"):
_result, mask = reg.get_mark(key).remove(empty, force=True)
assert mask is None
@@ -321,6 +321,17 @@ class TestArbiter:
assert "doubao" in keys
assert "jimeng_pill" not in keys
def test_pill_dropped_on_qwen(self):
# A Qwen frame is TC260 too but is not Jimeng-basic either: a confident
# bottom-right 千问AI生成 detection suppresses the pill exactly like Doubao's.
cands = [
self._c("qwen", strict=True, relaxed=True),
self._c("jimeng_pill", strict=True, relaxed=True, flat=True),
]
keys = self._keys(cands, reg.Context(provenance=frozenset({"jimeng"})))
assert "qwen" in keys
assert "jimeng_pill" not in keys
def test_pill_metadata_arm_gated_on_flatness(self):
ctx = reg.Context(provenance=frozenset({"jimeng"}))
assert self._keys([self._c("jimeng_pill", strict=True, relaxed=True, flat=True)], ctx) == {"jimeng_pill"}