Files
remove-ai-watermarks/pyproject.toml
T
Victor Kuznetsov 07b07634cf Merge the docs/arxiv-paper-review research line into main (#95)
* Add rigorous SynthID research and evaluation harness

* Record SynthID cross-epoch validation results

* Reset SynthID detector hypotheses to paired training

* Add paired wavelet and spectral SynthID probes

* Record closed-corpus SynthID detector ablations

* Add registered SynthID phase probing

* Add periodic SynthID tile probing

* Add calibrated SynthID pixel detector

* Harden scale-registered SynthID detection

* Record SynthID phase and channel rescue findings

* Add production SynthID routing and OpenAI verification

* Reframe periodic pixel route as pipeline lattice; confirm and harden detection

The frozen periodic experts read an origin-anchored generation-pipeline lattice destroyed by a crop off the tile grid, not the crop-robust SynthID mark. Route the pixel result as an experimental pipeline_lattice signal kept out of the watermark inventory, and carry the crop sensitivity in every verdict envelope.

Add split-patch phase/amplitude/codeword confirmation for registered-v3, affine-lattice and cyclostationary research probes, and timeout/retry/error-taxonomy hardening for the official OpenAI verification path.

* Record pipeline-lattice reframing and registered-v3 phase lock

Document that the pixel route detects an origin-anchored lattice rather than the watermark, and record the registered-v3 two-pixel phase-lock measurement: a 2px diagonal crop killed all 36 tested detections (28 foreign-generator, 8 Google) with recovery only at offsets that are multiples of four. Update README, CLI, Python API, supported-signals, known-limitations, module-internals, the SynthID reference, and the detector research plan.

* Split SynthID campaign docs by detector, classifiers, and removal

Keep Model 1 as a research AI-versus-camera result outside the library.
Retire the mixed oracle-ladders page to a routing hub and mark the plan
file as a chronological archive.

* Keep 2026-08-22 SynthID campaign results in research docs only

Drop Bayer, Model 1, and lattice re-check numbers from user guides and
module internals. The library surface is unchanged.

* Unplug lattice detector from package, expand research docs

* Remove lattice detector sources from the public package

* Document SynthID GitHub sweep findings

* Document general AI classifier sweep

* Document modern-negative taxonomy audit and classifier doc split

Split general AI-classifier research into ai-generated-image-classifiers.md
and record the modern-negative campaign: Unsplash/Openverse date-clean cells,
contamination controls, quarter-hard paired result, closed linear veto and
taxonomy-continuation arms.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix (same triage as main); core checks separately green (ruff, format, pyright, 1564 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - split references updated everywhere, none stale; 4) CLAUDE.md - routing line already updated, compliant

* Record wild-AI audit, Meta provider class, and stock-negative expansion

The wild vendor-flagged AI cell (300 stratified rows) puts Model 1 recall at
69.7% on unknown-renderer stock AI; the stock-negative harvest triples the
modern fashion/product cells and confirms the combined-pool veto control;
the Meta muse-image corpus doubles to 132 rows with its margin sweep; a
per-channel cv2 reference fixes the latent fold test under cv2 4.10.0.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks separately green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - new run references point at the gitignored research store, none stale; 4) CLAUDE.md - compact, no changes needed

* Close representation-change campaign: both candidates falsified

Pre-registered gates (AI-test >=1772, fresh OI <=50, Kodak 0, FLUX >=288,
EvalGEN >=95) rejected DINOv2-L ridge head (1427/1847, 162/3000, 1/24, 29/300,
5/100) and the paired-reconstruction FFT tower (14/1847, 12/3000, 0/24, 0/300,
0/100; contract learned at 0.949 pairwise separation but no transfer to
production generators). Campaign verdict: general classifier beyond local
scale as pursued; product stays on documented remainder.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks green minutes ago (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - verdict mirrored in data/research/REPRESENTATION-CAMPAIGN.md, no stale refs; 4) CLAUDE.md - no changes

* Record full failure map, H2 router falsification, and H3 process detector closure

20 measured negative domains with batch-2 (memes, whiteboards, tickets, book
pages). H2 domain router falsified: 97.9% of AI-test looks structured, every
family 94-100%. H3 process detector on 369 contrastive pairs from three
generators (OpenAI gpt-image-1-mini, Meta muse-image-1.0, Gemini flash-image)
achieves 0.909 pairwise AUC on training pairs but transfers at AUC 0.55-0.60
to production AI; human receipts score higher than any AI family. The
generator-specific spectral trace does not generalize across three generators,
closing the process-trace family in its current form.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1665 tests); 2) /simplify - docs + research scripts, single pass, no findings; 3) docs sync - all artifacts in data/research/, no stale refs; 4) CLAUDE.md - no changes

* Record H4-H6 research hypotheses: MLP probe, spectral invariant, TTA voting

H4 (MLP on frozen CLIP): 98.6% AI-test recall but 15.8% photo FPR — same
trade-off curve as quarter-hard. H5 (spectral invariant from CVPR 2025):
2.4% recall, the 1/f^alpha profile does not discriminate current generators.
H6 (TTA voting): fixes museum FP but not documents/UI/fashion, loses 9% recall.
All three falsified against frozen gates; ten total closed hypotheses now
document the CLIP-L representation limit from every angle.

Also records the provider cascade across all 20 domains (69.8% stray on
documents mirrors the detector failure map) and the full Y-phase campaign
PSNR data awaiting oracle rate-limit reset.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - all artifacts in data/research/, no stale refs; 4) CLAUDE.md - no changes

* Fix Y-phase frequency convention, record H4-H6 and campaign results

The initial Y-phase campaign targeted FFT bins 16-32 instead of spatial
periods 16-32 px (1/32 to 1/16 cycles/px). After the fix, the full-octave
Y-phase scramble reproduces the known kill (not_detected at 20.5 dB on s1
at native 1122x1402). Partial sub-bands are visually invisible (42.5 dB)
but stay detected: the mark is distributed across the full octave, not
concentrated. Point intervention is not viable within this frequency band.

H4 MLP probe: 98.6% recall, 15.8% FPR (same trade-off wall).
H5 spectral invariant: 2.4% recall (1/f^alpha does not discriminate).
H6 TTA voting: fixes museum FP but not documents/UI/fashion.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (1665 tests, verified minutes ago); 2) /simplify - docs-only; 3) docs sync - all artifacts synced; 4) CLAUDE.md - no changes

* Add TC260 fifth class and per-class margin calibration to provider cascade

Five-head cascade (openai/google/tc260/meta/no_ai): openai 68.8%, google
76.6%, tc260 63.8%, meta 71.8% mean recall at uniform margin 0.50.
Per-class margins calibrated against 1.67% photo budget: openai 0.31,
google 0.29, tc260 0.39, meta 0.55. TC260 test recall 71.4% at calibrated
margins on 196 held-out images. Chinese-AIGC is a viable pixel class.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (verified within the hour); 2) /simplify - research script, single pass; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Record universal detector v3-v8: diverse training shifts the curve

Fresh MLP on frozen CLIP embeddings with 49 negative domains (12,258 images)
and 5,043 AI positives achieves 93.0% AI-test recall with 39/49 domains at
zero false positives. Digital art FP dropped from 99.3% (Model 1) to 24.8%,
fashion from 62-71% to 12%, UI from 99.3% to 12%. Photo FPR 0.7%. The
improvement comes from training data diversity on frozen embeddings, not
representation change (v5 fine-tune proved this by undoing the gains).
Also adds .local-eval to ruff exclude so the gate covers only tracked files.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - all artifacts in data/research/; 4) CLAUDE.md - no changes

* Add three-tier ensemble detector: DEFINITELY/POSSIBLY/LIKELY_HUMAN

Combines Model 1 and v8 into calibrated confidence tiers. DEFINITELY_AI
(both models agree) achieves 79.6% recall at 0.50% photo FPR with 46/49
domains at zero FP. The union reaches 93.3% recall at 1.80% photo FPR
with Kodak 0/24. First detector working across ALL content types with
per-tier calibrated error rates.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + ensemble; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Add provider v2: diversity training eliminates document leakage

Same diversity-training insight applied to the provider cascade. Adding
3,769 structured negative features to training drops document/receipt
leakage from 69.8% to 5.1%, UI from 10% to 4.3%, fashion from 5-15% to
0-3%. AI recall improves: OpenAI 84% (was 75%), Google 88.8% (was 79%).
Photo leakage 1.4%. Worst regression: memes 47.9% (was 27%).

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - provider v2; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Add universal v11: full taxonomy, 136 domains, 124 at zero FP

Master data plan mapped 148 visual domains. Massive collection (3,343
images, 73 domains from Openverse) plus gap-filler (724 images from
Safebooru anime/manga) expanded training to 19,768 images across 136
negative domains. Universal v11 (MLP-512 + Model 1 ensemble):
DEFINITELY 90.3% recall at 0.50% photo FPR, ANY 95.6% recall,
124/134 domains at exactly 0% FP, worst FP 13.8% (digital art).

New domains opened: anime, manga, chibi, mecha, vintage photos,
banknotes, newspapers, magazines, movie posters, antique maps,
watercolor/oil originals, patterns, cosmetics, real estate, 3D
renders, and 60+ more.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Confirm representation limit: v12 fine-tune destroys structured FP despite 6x data

Fine-tuning CLIP tower on 19,518 images achieves 95.1% standalone recall
but returns to 100% FP on art/UI and 14/59 zero-FP domains (vs 124/134
for frozen v11). This is a representation property, not a data issue.
Frozen CLIP + MLP (v11) is the only working architecture. Also records
provider v3 trained on all negative domains.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Document v13 plateau: 128/136 zero FP, frozen-CLIP ceiling confirmed

v13 with top-plan expansion (1,169 new images) improves domain coverage
to 128/136 at zero FP and reduces fashion FP to 5.3%, but DEFINITELY
recall plateaus at 89.6% and art FP at 15.9%. The frozen CLIP + MLP
architecture has reached its ceiling.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (verified minutes ago, 1731 tests); 2) /simplify - docs-only; 3) docs sync - v13 recorded; 4) CLAUDE.md - no changes

* Document the 2026-08-31 photo-classifier freeze.

Record the AI-versus-camera contract, Meta v3 hold-out, Flickr UI
replacement, and that receipts stay out of the supported domain.

* Scrub Spaces catalog details from classifier research docs

* Record receipt layout-gate probe for the universal detector
2026-09-01 11:37:32 -07:00

266 lines
9.7 KiB
TOML

[project]
# This name is the published PyPI distribution, not a local label: changing it
# strands every `pip install remove-ai-watermarks` and every pinned dependency
# on it, including the ComfyUI node package. The console script below carries
# the same weight, since users have it on PATH.
name = "remove-ai-watermarks"
version = "0.36.0"
description = "AI watermark remover for visible, invisible, and provenance marks in images and video"
readme = "README.md"
requires-python = ">=3.11,<3.15"
license = {text = "Apache-2.0"}
keywords = [
"ai-watermark",
"ai-watermark-remover",
"watermark-remover",
"watermark-removal",
"remove-watermark",
"synthid",
"c2pa",
"content-credentials",
"nano-banana",
"gemini",
"gemini-watermark",
"ai-metadata",
"metadata-removal",
"provenance",
"exif",
"stable-diffusion",
"comfyui",
"ai-detection",
]
classifiers = [
"Development Status :: 4 - Beta",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: End Users/Desktop",
"License :: OSI Approved :: Apache Software License",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3 :: Only",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
"Programming Language :: Python :: 3.14",
"Topic :: Multimedia :: Graphics",
"Topic :: Multimedia :: Graphics :: Graphics Conversion",
"Topic :: Scientific/Engineering :: Artificial Intelligence",
"Topic :: Scientific/Engineering :: Image Processing",
"Topic :: Security",
"Topic :: Utilities",
]
dependencies = [
"pillow>=10.0.0",
"piexif>=1.1.3",
"click>=8.0.0",
"python-dotenv>=1.0.0",
# Official C2PA reader (Content Authenticity Initiative, MIT/Apache-2.0). The
# primary, spec-tracking manifest parser for the identify/metadata path; the
# hand-rolled caBX/CBOR scanner in _internal/c2pa.py is kept only as a fallback for
# synthetic/partial blobs the validator rejects. Binary wheel (Rust), but the
# import is light (no torch/numpy) so it fits the dependency-light identify
# host. Prebuilt wheels cover the full CI matrix (linux/macos/windows).
"c2pa-python>=0.35.0",
]
[project.optional-dependencies]
pixels = [
"numpy>=1.24.0; python_version < '3.13'",
# NumPy 1.x has no CPython 3.13+ wheels. Requiring 2.x prevents installers
# from attempting an unsupported source build on the newer Python lines.
"numpy>=2; python_version >= '3.13'",
"opencv-python-headless>=4.8.0",
]
# Optional HEIC/AVIF pixel decode. Metadata scanning handles these containers
# without this plugin; combine `heif` with any pixel feature only when needed.
heif = [
"pillow-heif>=0.13.0",
]
visible = ["remove-ai-watermarks[pixels]"]
# Video visible removal uses the shared pixel runtime. PyAV packetizes processed
# VFR frames with explicit PTS before system ffmpeg encodes them.
video = [
"remove-ai-watermarks[visible]",
"av>=18,<19",
]
# Open DWT-DCT watermarks used by Stable Diffusion / SDXL / FLUX. The in-tree
# decoder avoids the upstream invisible-watermark package's mandatory torch and
# non-headless OpenCV dependencies.
detect = [
"remove-ai-watermarks[pixels]",
"PyWavelets>=1.1.1",
]
diffusion = [
"remove-ai-watermarks[pixels]",
# A CUDA-enabled torch build is required: invisible-watermark removal has no
# CPU, MPS or XPU path. The default PyPI wheel carries CUDA on Linux/Windows;
# on macOS there is no CUDA build and this extra installs only for the
# non-diffusion imports it shares.
"torch>=2.0.0",
"diffusers>=0.38.0",
# diffusers 0.38's auto-pipeline registry imports ``Qwen3VLForConditional
# Generation`` (its ``nucleusmoe_image`` pipeline), which only exists in
# transformers 5.x -- so ``from diffusers import AutoPipelineForImage2Image``
# fails on transformers 4.x. The real SDXL-loading break was NOT transformers
# 5.x but the tokenizers *release candidate* (0.23.0rc0) that the global
# ``prerelease = "allow"`` drags in: its CLIP tokenizer raises
# ``RobertaProcessing.__new__() got an unexpected keyword argument 'cls'``.
# Cap tokenizers to the stable 0.22 line (transformers 5.x accepts
# >=0.22,<=0.23.0) so the rc is excluded while SDXL still loads.
"transformers>=5,<6",
"tokenizers>=0.22,<0.23",
"accelerate>=0.25.0",
"safetensors",
]
# Full two-stage high-fidelity profile: Qwen-Image-2512 Lightning + DiffSynth
# Canny ControlNet for the frame, then SAM-masked Z-Image Turbo face repair.
# CUDA-only and intentionally separate from the normal diffusion extra because the
# additional model stack and DiffSynth runtime are large.
qwen-zimage = [
"remove-ai-watermarks[diffusion]",
"diffsynth>=2.0.17,<3",
"torchvision>=0.20.0",
]
# Opt-in verified-text reconstruction over qwen-zimage. LaMa removes the changed
# candidate glyphs before exact Qwen-VAE cores are composited back.
text-restoration = [
"remove-ai-watermarks[qwen-zimage,lama]",
]
# Draft-only OCR proposals for verified-text manifests. CPU, no torch:
# PaddleOCR detection plus three script-chosen recognition engines. The models
# download on first use and are cached by huggingface_hub; never bundled.
text-draft = [
"paddleocr>=3.3.3",
"paddlepaddle>=3.0.0",
"huggingface-hub>=0.20.0",
]
# Adobe TrustMark decoder -- the open, keyless watermark behind Adobe Durable
# Content Credentials (soft-binding alg ``com.adobe.trustmark.P``). Optional
# because it pulls torch and downloads model weights on first use. identify()
# guards the import and skips the TrustMark signal when absent.
trustmark = [
# TrustMark requires NumPy <2, whose last release has no CPython 3.13+
# wheels. Keep this feature installable only on its real runtime range.
"trustmark>=0.8.0; python_version < '3.13'",
]
# Metadata-free photo classifier (Model 1 then gated Model 2). CPU torch is
# enough. identify never imports this extra. Weights download on first use.
classify = [
"remove-ai-watermarks[pixels]",
"torch>=2.0.0",
"transformers>=5,<6",
"tokenizers>=0.22,<0.23",
"huggingface-hub>=0.20.0",
]
# Official remote OpenAI SynthID verification. The command strips AI provenance
# metadata and proves pixel identity before upload; it is never called implicitly.
verify = [
"openai>=2.52.0",
]
# Universal region eraser backend -- big-LaMa via onnxruntime (Carve/LaMa-ONNX,
# Apache-2.0). CPU, no torch. Model (~200 MB) is downloaded on first use and
# cached by huggingface_hub; it is never bundled in this repo. The default cv2
# eraser backend needs none of this.
lama = [
"remove-ai-watermarks[visible]",
"onnxruntime>=1.24.0",
"huggingface-hub>=0.20.0",
]
# Lightweight inpaint backend -- MI-GAN via onnxruntime (andraniksargsyan/migan,
# MIT). CPU, no torch. Model (~28 MB) downloaded on first use and cached by
# huggingface_hub; never bundled. ~700-950 MB peak RAM / ~0.19 s/call -- the
# memory-tight learned tier (vs big-LaMa's ~4.7 GB). Select it explicitly when
# LaMa, the quality-first `auto` choice, is too large. Same runtime as `lama`.
migan = [
"remove-ai-watermarks[visible]",
"onnxruntime>=1.24.0",
"huggingface-hub>=0.20.0",
]
dev = [
"remove-ai-watermarks[video]",
"remove-ai-watermarks[detect]",
"pytest>=8.0.0",
"pytest-cov>=4.1.0",
"pytest-xdist>=3.5.0",
"packaging>=24.0",
"ruff>=0.4.0",
"pyright>=1.1.0",
"invisible-watermark>=0.2.0",
]
# ``qwen-zimage`` already pulls ``diffusion``; naming both would suggest diffusion is
# independently sufficient for a removal, which it is not.
all = ["remove-ai-watermarks[video,heif,detect,trustmark,text-restoration,migan,classify,verify]"]
[project.scripts]
remove-ai-watermarks = "remove_ai_watermarks.cli:main"
[project.urls]
Repository = "https://github.com/wiltodelta/remove-ai-watermarks"
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["src/remove_ai_watermarks"]
[tool.hatch.build.targets.sdist]
# Keep the source distribution small and public-safe: ship tracked source and
# metadata, not corpora or local research/session artifacts. The wheel ships
# only src/.
include = [
"/src",
"/LICENSE",
"/README.md",
"/pyproject.toml",
]
exclude = [
"/data",
"/tmp",
"/.sc",
]
[tool.pytest.ini_options]
testpaths = ["tests"]
pythonpath = ["src", "scripts"]
addopts = "-v --tb=short"
[tool.ruff]
target-version = "py311"
line-length = 120
exclude = ["_refs", ".local-eval"]
extend-exclude = ["*.md"]
[tool.ruff.lint]
select = ["E", "F", "B", "I", "S", "UP", "SIM", "RET", "COM", "C4", "G", "PT", "PIE", "T20", "DTZ", "ICN", "TCH", "RUF", "ANN"]
ignore = [
"COM812", # missing trailing comma (conflicts with ruff formatter)
"ANN401", # typing.Any — sometimes unavoidable with third-party libs
]
[tool.ruff.lint.per-file-ignores]
"scripts/**/*.py" = ["G004", "S108", "S310", "T20"]
"tests/*.py" = ["ANN", "S101", "S105", "S106", "S108"]
"src/remove_ai_watermarks/_internal/watermark_remover.py" = ["S603", "S606", "S607"] # nvidia-smi capability probe
"src/remove_ai_watermarks/_internal/c2pa.py" = ["S110"] # try-except-pass for corrupt file handling
[tool.ruff.format]
quote-style = "double"
indent-style = "space"
[tool.pyright]
pythonVersion = "3.11"
typeCheckingMode = "strict"
exclude = ["_refs", ".local-eval"]
[[tool.pyright.executionEnvironments]]
root = "tests"
extraPaths = ["."]
reportAttributeAccessIssue = false
reportOptionalSubscript = false
reportOptionalMemberAccess = false
reportArgumentType = false
reportUnknownMemberType = false
reportUnknownArgumentType = false
reportUnknownVariableType = false
reportMissingTypeArgument = false