Three gaps found while measuring the record path against the file path, each one
a signal the library could not see:
WebP stores `XMP ` after the pixels, so on any WebP above the scan window a fixed
read stops short of the label. `_riff_late_metadata` steps over the coded image to
reach it, the RIFF analogue of the existing PNG and ISOBMFF readers. Three corpus
files hid an IPTC "Made with AI" tag and a C2PA `trainedAlgorithmicMedia` there.
The decoder-backed fallback now covers only what it is actually for -- metadata the
raw bytes do not spell, such as a compressed PNG `zTXt` packet.
A C2PA reader failure returned the same `None` as a file with no manifest, so a
verdict could fall back to the raw byte scan with no trace anywhere. Failures now
log at warning and only genuine ones do: a file without credentials never reaches
that branch, and an unsupported container is demoted to debug through the reader's
own `C2paError.NotSupported`. The first corpus run with it found a truncated PNG.
`scan_dataset.py` never registered the pillow-heif opener it declares as a
dependency, so every HEIC was scanned as unreadable -- no EXIF, and a pixel layer
that was 397 of 406 features NaN instead of 136.
`_riff_late_metadata` caps its total like `isobmff.scan_c2pa_region` does. Clamping
each chunk to the bytes remaining is not enough on its own: one chunk can declare a
length spanning most of the file, and this runs on the memoized verdict path over
images from arbitrary sources.
Also lands `identify_metadata_record` and `ProvenanceReport.to_dict()`, the
one-call entry point and the versioned JSON contract for the record path.
Record-vs-file equality holds over 3,478 corpus images, and the eight files these
fixes recovered still report AI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`collect_metadata_record` returns a JSON-safe record carrying an image's
provenance metadata regions -- never its pixels -- and the existing
`evidence_from_metadata_record` + `identify_from_evidence` build the verdict
from it without opening the file. The contract is equality with
`identify(path, metadata only)`, verified over the tracked fixtures and over a
local corpus of 3,478 images (every file carrying a rare signal, plus a random
slice): zero differences.
Three placements defeated earlier drafts and each is now a rule with a test:
the `scan_head` buffer is the head CONCATENATED with late metadata, so a
structural walk must read the raw head instead; Samsung splits its evidence
between a post-EOI trailer and the coded scan; and PIL's info keys must be
emitted in the file path's candidate order, since the first token match wins.
Also fix a real detection gap found while establishing that equality: a label
the decoder can read but a raw byte scan cannot -- a compressed PNG `zTXt`
packet, or a WebP XMP chunk past the scan window -- was invisible to
`identify`. Eight corpus files carrying a China TC260 AIGC label or an IPTC
"Made with AI" tag were reported as no signal at all.
`scripts/detection_timing.py` and its report script measure the metadata path
per method; they write outside the repository and are read-only over a dataset.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`InvisibleOptions` promises in its docstring that every default mirrors
`InvisibleEngine`. Two fields made that promise cost something to keep: `force` is
not an engine parameter at all, and `controlnet_scale` was a third spelling of the
engine's `controlnet_conditioning_scale`. The mirror test carried an exception
table for each. This removes both, so the comparison needs no exceptions -- a field
that needs one is a field that belongs somewhere else.
`force` decides WHETHER the engine runs, which is settled before it is built, so it
joins `backend` and `sensitivity` as a parameter of `remove_all` and `remove_batch`
and is threaded to `_run_invisible` as its own argument. `controlnet_scale` takes
the engine's own name; the click option stays `--controlnet-scale` and is now
translated exactly once instead of at three forwarding sites.
Safe to do today: both symbols landed after 0.25.0 and have never been published.
The forwarding turned out to be the weaker half. A defaults comparison cannot see a
hardcoded literal at the seam, and `_run_invisible` passed the entire suite with
`controlnet_conditioning_scale` pinned to a constant. Each of the two knobs also
reaches the engine through TWO paths -- `remove_all` versus `remove_batch(mode="all")`
for `force`, `_run_invisible` versus `_batch_engine` for the scale -- and guarding one
left the other free to hardcode with a green suite. So:
* `test_every_field_arrives_at_the_engine_with_the_caller_s_value` drives the real
seam with all 13 fields set off their defaults; mutating any one of them to its
default fails it.
* `test_force_reaches_the_scrub_gate_in_every_scrubbing_mode` and
`test_batch_controlnet_scale_flows_to_the_cached_engine` are parametrized over
both modes, so neither path can be pinned alone.
Also fixes an order-dependent test surfaced by the added tests reshuffling the xdist
shards. `test_visible_path_decodes_file_once` counted every `image_io.imread` in the
process, but the Gemini engine loads its own bundled capture assets on first
construction, so the count was 3 on a cold engine and 1 on a warm one and the test
passed only when an earlier test happened to build the engine first. It now counts
decodes of the SOURCE, which is the invariant it exists for, and still fails when the
shared decode is broken. The production path was never wrong: the source bitmap is
decoded exactly once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The visible-mark path had grown three copies of one ladder sweep, four
near-identical `detect` arms, and four hand-rolled `footprint_mask` overrides;
mark knowledge sat in five hand-maintained tables across three modules; and the
flagship `all`/`batch` pipeline existed only in cli.py, written twice with
divergent behavior.
Detection is now one measurement. `_ladder_best` replaces the three sweeps,
`_scan`/`_verdict` replace the four arms, and the winning box travels to the
mask on `TextMarkDetection.match_box` instead of being swept a second time.
`detect_both` returns the strict and relaxed verdicts from one scan, which
halves the arbiter's perception cost (260 -> 130 matchTemplate calls on a 2048²
image, verdicts identical field for field). A per-mark demotion goes in the new
`_post_gate` hook, never in a `detect` override -- an override is invisible to
the single-pass path, which is how the RunningHub and Yuanbao anchor gates
briefly stopped applying.
Everything about a mark is now one registry row: product, label regime, the
platform sentence `identify` reports, the metadata signals that confirm it, and
its TC260 producer codes. `identify._VISIBLE_MARK_PLATFORM`, the signal mapping
in `api.visible_provenance`, `_PRODUCT_OF` and the pill veto are derived from
those rows.
`api.remove_all` / `api.remove_batch` are the library form of the `all` and
`batch` commands; the CLI is a wrapper that owns console text and exit codes.
Progress is a `(stage, detail)` pair of stable tokens, so the CLI keys its
wording off structure rather than parsing the library's prose back.
Two intentional behavior changes, both verified against a recorded 811-image
sample of detector verdicts, removal-mask hashes, arbiter decisions and
`identify` reports:
* A TC260 label now relaxes the vendor its `ContentProducer` names rather than
ByteDance's pair on every China-AIGC image. 333 of 811 samples move; on 185
of them the previously relaxed pair was simply the wrong vendor, and the
mark actually present never reached the relaxed gate its own
`provenance_ncc_factor` was calibrated for.
* A confident LibLibAI detection suppresses the Jimeng pill, like every other
TC260 product's mark. It was registered alongside RunningHub and Baidu, both
of which were added to the hand-written veto list, and it was not. 1 sample
moves, and it is exactly the co-firing case.
Nothing else in that record changes: detector verdicts, mask hashes and
`identify` verdicts are byte-identical, and all 200 calibration constants are
untouched.
Also: `aigc_label` and friends plus `extract_c2pa_info` are memoized on
(path, mtime_ns, size) -- size because this package rewrites in place; the
native TC260 container readers route on magic bytes instead of the file
extension, so a mislabeled AVI or FLV is no longer invisible; `identify` shares
one pixel decode between the DWT-DCT and visible stages (TrustMark keeps its own
Pillow decode, which is not substitutable); and the six `stabilize_*` video
wrappers collapse into one policy table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An adversarial review of 52b2c11 (five independent audits, each finding put to
two skeptics, plus a completeness critic) found four defects that commit
introduced and several stale claims it should have caught.
The install hint no longer installs -- again. Folding five hints into one
INVISIBLE_EXTRA constant dropped the shell quoting the originals had, so the
printed remediation was `pip install remove-ai-watermarks[qwen-zimage]`. Bare
brackets are a glob in zsh, the macOS default shell: it dies with "no matches
found" before pip runs. That is the exact failure 52b2c11 existed to stop
producing, reintroduced in a different form by a bulk replace. The constant is
quoted now, and a test asserts the quotes rather than the bare substring -- the
old assertions passed either way, which is why nothing caught it.
Three tests were not guarding what they claimed:
- The commit's headline behaviour change, per-profile polish resolution inside
the engine, had no test at all. Rebinding resolve_adaptive_polish to the
pre-commit `bool(value)` left the full suite green. Now covered by a test that
drives the real engine and observes whether humanizer.adaptive_polish ran;
that mutation now fails it.
- TestAvailability still asserted the pre-commit (torch, diffusers) contract, so
in a diffusion-only environment it was simply wrong, and comparing each gate to
a tuple copied from itself could never catch the two gates disagreeing -- the
drift the shared REMOVAL_MODULES was introduced to prevent. Replaced with a
test that simulates each module's absence and requires BOTH gates to close.
- Both CUDA-refusal guards skipped in every environment, including CI: they were
gated on the diffusion stack, which no CI job installs. The refusal fires
before any torch attribute is read, so they now run everywhere; only the dtype
assertion keeps its skip.
Also: the retired-knob test covered `invisible` but not `all` or `batch`, though
all three declared those options separately; and smoke_matrix.py still called
remove_watermark(region=...), a parameter 52b2c11 deleted, with the resulting
TypeError swallowed into a skip by a broad except.
Stale documentation the previous sweep missed: known-limitations still described
an MPS out-of-memory fallback and a lighter-pipeline escape that no code can
produce; module-internals declared Canny thresholds of 100/200 as a compatibility
contract while the code uses 13/64, attributed enable_model_cpu_offload to
deleted profiles, and still warned that the engine and CLI defaults differ (this
commit's predecessor made them identical); cli.md gated `all` on the `diffusion`
extra; python-api claimed "cuda" was the only accepted explicit device when
"auto" is too. The claim that `device` is not a parameter was wrong in both
module-internals and .claude/rules/development.md -- it is one, deliberately, and
now says so. `--cpu-offload` help and the pipeline's CUDA guard both still
pointed at MPS.
Not fixed here, reported instead -- both are outside this repo:
- ComfyUI-remove-ai-watermarks nodes.py:332 passes num_inference_steps and
guidance_scale (plus min_resolution/upscaler from bf4bfc1). distribute.yml's
comfyui job runs on every release and fails the release if the node sync fails,
so 0.25.0 needs that node updated first.
- raiw-app modal_app.py:422-425 forwards the same two kwargs into
remove_watermark. Latent: it is pinned to 1a77e24 and nothing supplies a value
today, so it fires on the next pin bump.
pre-commit: 1) maintain.sh - exit 0 (1093 tests, Pyright 0 errors, no
vulnerabilities); 2) /simplify - not re-run, this commit is the applied output of
a five-dimension adversarial review; 3) docs sync - grepped MPS/mps, the extras
names and every symbol touched across README, docs/, scripts/, .claude/; updated
6 docs; 4) CLAUDE.md - corrected the device claim in .claude/rules/development.md
and added the shell-quoting rule
Verified by execution, not assertion: smoke_matrix --quick 51 pass / 0 fail,
_knob_rows driven directly 10 pass / 0 fail / 7 skip (no CUDA), the install hint
rendered and round-tripped through zsh, and each new test confirmed to fail
under the mutation it is meant to catch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The CLI still advertised --model, --steps, --guidance-scale, --device and a
deprecated --auto. Each pinned a value the two surviving profiles fix -- the
model stack, the per-stage distilled schedule, CFG 1.0, CUDA -- so the only
outcome any of them had was an error raised several frames below the caller,
under a message naming an internal profile. A flag whose sole result is a
refusal is worse than no flag: it advertises a capability that does not exist,
and it lets a wrapper thread a value that will silently do nothing. They are
gone from the parser, from InvisibleEngine, and from WatermarkRemover, so the
failure is now a TypeError or a Click "No such option" at the point the caller
can act on.
The install hint was wrong in the same way. is_available() checked torch and
diffusers, then told the user to install [diffusion] -- which contains neither
DiffSynth nor the Z-Image face stage both profiles run. Following the advice
produced a second, different failure. The module list and the extra name now
live once in watermark_profiles (REMOVAL_MODULES, INVISIBLE_EXTRA) and are read
by both the CLI gate and the remover's precondition, which cannot drift apart
because they are the same tuple.
The adaptive-polish default moved out of the argument parser. It was resolved by
reading Click's parameter source, which put per-profile data in the CLI layer,
left the engine declaring the opposite default (False vs True) so a library
caller and a CLI caller on one profile got different output, and lost the polish
entirely for anything that supplies the flag non-interactively. The flag is now
tri-state (default=None) and resolve_adaptive_polish owns the per-profile
answer. The seed follows the same rule: the CLI stopped pre-resolving it.
Dead code removed with it: six scan_*_video wrappers and the _scan_video helper
none of them had a caller for, PNG_METADATA_KEYS, feather_region_composite and
the remover region path that was only reachable from a no-caller convenience
wrapper, remove_watermark_batch on both layers, try_empty_device_cache, the
_generate/_run_qwen_zimage pass-through pair, self.model_id, and the _internal
PEP 562 shim that no caller ever went through. get_device now answers cuda or
cpu only: mps and xpu travelled one frame to the same CUDA-only refusal while
costing a device probe each, and that refusal now names the resolved device, so
device=None on a CUDA-less host says 'cpu' rather than 'None'. The XPU wheel
index went with them.
Docs: README, cli, installation, python-api, supported-signals,
known-limitations and module-internals all still described the removed profiles,
the CPU/MPS/XPU ladder, a `default`->`sdxl` alias, and the wrong extra.
known-limitations still listed the retired SDXL strength ladder as current.
scripts/smoke_matrix.py and real_examples_e2e.py drove --device mps.
Next release is 0.25.0, not a patch: this removes public parameters and
narrows a published extra on top of the released 0.24.0.
pre-commit: 1) maintain.sh - exit 0 (1091 tests, Pyright 0 errors, no
vulnerabilities); 2) /simplify - 4 agents, 11 findings applied, 2 skipped
(dropping the `device` parameter entirely, which raiw-app pins; folding
diffsynth into the `diffusion` extra, which video-only callers do not need);
3) docs sync - grepped every removed identifier across README, docs/, scripts/,
.claude/; updated 9 docs; 4) CLAUDE.md - added the no-error-only-knobs rule to
.claude/rules/development.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The min-resolution floor lifted small inputs toward SDXL's ~1024 training size,
and Real-ESRGAN was an optional way to do that lifting. Both surviving profiles
run at native geometry, so the engine forced the floor to 0 on every path; the
floor never fired, `upscaling` was never true, and nothing downstream of it could
execute. Gone: upscaler.py, _esrgan_upscale, the min_resolution and upscaler
parameters, _target_size's floor branch, --min-resolution, --upscaler,
_warn_if_esrgan_unavailable and the `esrgan` extra. max_resolution stays and is
now the only lever on geometry; it can only scale down.
scripts/smoke_matrix.py was the one live consumer and neither gate saw it -
Pyright is scoped to src/ and Ruff cannot resolve its function-local import - so
`--diffusion` would have died at import. Its knob rows were written for the
removed profiles besides (--pipeline sdxl, --steps 20, --guidance-scale 5.0,
--device mps), so they are rewritten rather than patched: most now assert a knob
is REJECTED, which is the coverage worth having when the CLI accepts a value the
library refuses several layers down. Accepted-knob rows skip without CUDA, so the
row count is host-dependent and verification-plan.md no longer claims a fixed 68.
This removes a public module, a CLI option and a published extra, so the next
release is 0.25.0, not a patch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
qwen-zimage becomes the default and sdxl-zimage the only alternative. The
controlnet, sdxl, qwen and default profiles are gone, and with them the CPU and
MPS paths for invisible-watermark removal: neither matched the two-stage
recipe's face preservation, so keeping them advertised a quality this library no
longer delivers. Visible-mark removal and every identify command still run
anywhere.
Retired names are rejected rather than remapped. Silently routing --pipeline
sdxl onward would run an old script at a different strength, on a different
model, at a different quality, and report success.
CUDA is now checked when the remover is constructed instead of when the model
loads. Auto-detection cheerfully returned mps on a Mac, so the failure arrived
several layers down, after the dependency check and the pipeline import, in a
message naming whichever internal pipeline happened to raise. _DEVICES collapses
to {"cuda"} and the cpu/mps float32 branch goes with it.
resolve_strength stays total. It briefly returned None for qwen-zimage, meaning
"ask the resolution curve", which pushed a branch onto both callers and left one
of the two strength policies outside the strength module; the CLI copy had
already grown an `or 0.0` guarding a path its own comment called unreachable. It
now takes the image size and answers for both profiles, so the displayed value
cannot drift from the executed one.
Deletion fallout removed with it: img2img_runner and progress.py (the MPS
recovery path and its progress monitor had no callers left), viable_steps, the
fp16 degenerate-output retry, the fp16 VAE fix, and the Qwen img2img call
builders. try_empty_device_cache moved into watermark_remover rather than
leaving a module whose docstring outlived its code. _HAS_DIFFUSERS routes
through optional_deps.module_available, which is what the rest of the library
uses and what correctly rejects a pruned namespace remnant.
--steps, --guidance-scale and --model now have exactly one legal value each and
are still accepted at parse time, then rejected in remove(). Their help text
says so, but validating them beside the option would be better.
Not addressed, and worth its own decision: invisible_engine forces
min_resolution to 0 for both profiles, so the --min-resolution floor, --upscaler,
_esrgan_upscale, upscaler.py and the esrgan extra are all unreachable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Victor Kuznetsov