mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-09 23:50:40 +02:00
8fe0b0110f55a8bbb42eec8069f8bf8ff978dcee
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Victor KuznetsovandClaude Opus 5
|
8fe0b0110f |
Make the video SynthID operating point measurable and hard to move silently
The shipped profile was certified by one oracle row, but only noise_std was pinned: long_side and fps -- two thirds of what the verifier was actually shown -- could move with a green suite. The test now derives the pin from data/evaluations/video-synthid-oracle.csv, so a default without a certifying row fails. The certified profile is a perturbation-to-signal ratio, not a bare noise_std. sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the AutoencoderKL class default under an upper-unbounded diffusers pin. The loader now gates that value, carries it on VideoVaeRuntime, and passes it into encode and decode so the validated value is the applied value. video_synthid_sweep.py loads through the same function: the harness producing the certified rows was the one path exempt from the gate it exists to feed. psnr_db is measured against the already-resized frame and before the encoder, so it cannot see the downscale, the decimation, or the codec, and no in-loop metric can. scripts/video_fidelity_probe.py scores the delivered file end to end, streaming the way the engine does and sharing its frame-selection rule rather than copying it -- a frame-count check cannot catch a rule that reorders frames without changing how many. The manifest gains source geometry, vae, track, verbatim verdict and session fields. The two 2026-07-31 rows keep them empty: they were never recorded and are not recoverable. Verdicts now have four states, because the verifier's unclear reading logged as not_detected is the silent regression the manifest exists to prevent. docs/video-synthid-quality-research.md records the research behind this: the noise axis is worth about 2 dB and is nearly exhausted, resolution is the real prize but is an uncertified destruction axis rather than a free win, and every proposed autoencoder swap was refuted. First local measurements included. Verified: engine output is byte-identical before and after the refactor on a locally built clip, at noise_std 0.00 and 0.15. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 5
|
4a896cd4b5 |
Make InvisibleOptions engine knobs only and pin the forwarding
`InvisibleOptions` promises in its docstring that every default mirrors
`InvisibleEngine`. Two fields made that promise cost something to keep: `force` is
not an engine parameter at all, and `controlnet_scale` was a third spelling of the
engine's `controlnet_conditioning_scale`. The mirror test carried an exception
table for each. This removes both, so the comparison needs no exceptions -- a field
that needs one is a field that belongs somewhere else.
`force` decides WHETHER the engine runs, which is settled before it is built, so it
joins `backend` and `sensitivity` as a parameter of `remove_all` and `remove_batch`
and is threaded to `_run_invisible` as its own argument. `controlnet_scale` takes
the engine's own name; the click option stays `--controlnet-scale` and is now
translated exactly once instead of at three forwarding sites.
Safe to do today: both symbols landed after 0.25.0 and have never been published.
The forwarding turned out to be the weaker half. A defaults comparison cannot see a
hardcoded literal at the seam, and `_run_invisible` passed the entire suite with
`controlnet_conditioning_scale` pinned to a constant. Each of the two knobs also
reaches the engine through TWO paths -- `remove_all` versus `remove_batch(mode="all")`
for `force`, `_run_invisible` versus `_batch_engine` for the scale -- and guarding one
left the other free to hardcode with a green suite. So:
* `test_every_field_arrives_at_the_engine_with_the_caller_s_value` drives the real
seam with all 13 fields set off their defaults; mutating any one of them to its
default fails it.
* `test_force_reaches_the_scrub_gate_in_every_scrubbing_mode` and
`test_batch_controlnet_scale_flows_to_the_cached_engine` are parametrized over
both modes, so neither path can be pinned alone.
Also fixes an order-dependent test surfaced by the added tests reshuffling the xdist
shards. `test_visible_path_decodes_file_once` counted every `image_io.imread` in the
process, but the Gemini engine loads its own bundled capture assets on first
construction, so the count was 3 on a cold engine and 1 on a warm one and the test
passed only when an earlier test happened to build the engine first. It now counts
decodes of the SOURCE, which is the invariant it exists for, and still fails when the
shared decode is broken. The production path was never wrong: the source bitmap is
decoded exactly once.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Victor KuznetsovandClaude Opus 5
|
13095fb45c |
Verify every doc claim against the source and fix what drifted
Every code-referencing claim in the docs, the README and the rules files was checked against src/, and each finding was re-derived independently before it was applied. 35 held, 5 were false positives. Two of them were code, not text. `InvisibleOptions` promises in its docstring to mirror `InvisibleEngine`, and two defaults had silently stopped: `max_resolution=None` reached `_target_size`'s `max_resolution > 0` and raised `TypeError` on every library call that left the options alone, and `cpu_offload=True` made a library run slower than the identical CLI run. Both are fixed, and `TestInvisibleOptionsMirrorTheEngine` compares the two signatures field by field rather than pinning the two values that happen to be known. A companion assertion in `TestTargetSize` reads the engine's own declared default, so a drift on the engine side -- which the mirror check alone would accept, because both sides would still agree -- fails too. The user-facing docs: README called `invisible` GPU-optional where it raises without CUDA, and gave the image `metadata` command `video metadata`'s output rule, promising the source survives a command that overwrites it. Yuanbao was missing from the supported-mark list. `veo` was listed among the video policies that require a run anchor, though its row sets no `anchor_iou`. `known-limitations` called ControlNet the default profile and contradicted itself ninety lines below. An unescaped pipe truncated the `hailuo` table row. The `dev` extra, the CI shape, ffmpeg's role, the sdist boundary and the strength-curve range were corrected, and `remove_all`/`remove_batch`, the pill gate, `erase --keep-metadata` and `all`'s CUDA failure mode were documented. Research notes that described removed modules, extras and flags in the present tense now say so once in the page banner instead of sentence by sentence, which covers the whole page rather than the lines that happened to be noticed, and one fixture is referred to by role rather than by name. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 5
|
78d9e81d0f |
Collapse the duplicated detection path and lift the image pipeline into the library
The visible-mark path had grown three copies of one ladder sweep, four
near-identical `detect` arms, and four hand-rolled `footprint_mask` overrides;
mark knowledge sat in five hand-maintained tables across three modules; and the
flagship `all`/`batch` pipeline existed only in cli.py, written twice with
divergent behavior.
Detection is now one measurement. `_ladder_best` replaces the three sweeps,
`_scan`/`_verdict` replace the four arms, and the winning box travels to the
mask on `TextMarkDetection.match_box` instead of being swept a second time.
`detect_both` returns the strict and relaxed verdicts from one scan, which
halves the arbiter's perception cost (260 -> 130 matchTemplate calls on a 2048²
image, verdicts identical field for field). A per-mark demotion goes in the new
`_post_gate` hook, never in a `detect` override -- an override is invisible to
the single-pass path, which is how the RunningHub and Yuanbao anchor gates
briefly stopped applying.
Everything about a mark is now one registry row: product, label regime, the
platform sentence `identify` reports, the metadata signals that confirm it, and
its TC260 producer codes. `identify._VISIBLE_MARK_PLATFORM`, the signal mapping
in `api.visible_provenance`, `_PRODUCT_OF` and the pill veto are derived from
those rows.
`api.remove_all` / `api.remove_batch` are the library form of the `all` and
`batch` commands; the CLI is a wrapper that owns console text and exit codes.
Progress is a `(stage, detail)` pair of stable tokens, so the CLI keys its
wording off structure rather than parsing the library's prose back.
Two intentional behavior changes, both verified against a recorded 811-image
sample of detector verdicts, removal-mask hashes, arbiter decisions and
`identify` reports:
* A TC260 label now relaxes the vendor its `ContentProducer` names rather than
ByteDance's pair on every China-AIGC image. 333 of 811 samples move; on 185
of them the previously relaxed pair was simply the wrong vendor, and the
mark actually present never reached the relaxed gate its own
`provenance_ncc_factor` was calibrated for.
* A confident LibLibAI detection suppresses the Jimeng pill, like every other
TC260 product's mark. It was registered alongside RunningHub and Baidu, both
of which were added to the hand-written veto list, and it was not. 1 sample
moves, and it is exactly the co-firing case.
Nothing else in that record changes: detector verdicts, mask hashes and
`identify` verdicts are byte-identical, and all 200 calibration constants are
untouched.
Also: `aigc_label` and friends plus `extract_c2pa_info` are memoized on
(path, mtime_ns, size) -- size because this package rewrites in place; the
native TC260 container readers route on magic bytes instead of the file
extension, so a mislabeled AVI or FLV is no longer invisible; `identify` shares
one pixel decode between the DWT-DCT and visible stages (TrustMark keeps its own
Pillow decode, which is not substitutable); and the six `stabilize_*` video
wrappers collapse into one policy table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Victor KuznetsovandClaude Opus 5
|
2f72257996 |
Fix what the verification pass found in the knob removal
An adversarial review of |
||
|
Victor KuznetsovandClaude Opus 5
|
52b2c115e8 |
Delete every knob the fixed profiles cannot honor
The CLI still advertised --model, --steps, --guidance-scale, --device and a deprecated --auto. Each pinned a value the two surviving profiles fix -- the model stack, the per-stage distilled schedule, CFG 1.0, CUDA -- so the only outcome any of them had was an error raised several frames below the caller, under a message naming an internal profile. A flag whose sole result is a refusal is worse than no flag: it advertises a capability that does not exist, and it lets a wrapper thread a value that will silently do nothing. They are gone from the parser, from InvisibleEngine, and from WatermarkRemover, so the failure is now a TypeError or a Click "No such option" at the point the caller can act on. The install hint was wrong in the same way. is_available() checked torch and diffusers, then told the user to install [diffusion] -- which contains neither DiffSynth nor the Z-Image face stage both profiles run. Following the advice produced a second, different failure. The module list and the extra name now live once in watermark_profiles (REMOVAL_MODULES, INVISIBLE_EXTRA) and are read by both the CLI gate and the remover's precondition, which cannot drift apart because they are the same tuple. The adaptive-polish default moved out of the argument parser. It was resolved by reading Click's parameter source, which put per-profile data in the CLI layer, left the engine declaring the opposite default (False vs True) so a library caller and a CLI caller on one profile got different output, and lost the polish entirely for anything that supplies the flag non-interactively. The flag is now tri-state (default=None) and resolve_adaptive_polish owns the per-profile answer. The seed follows the same rule: the CLI stopped pre-resolving it. Dead code removed with it: six scan_*_video wrappers and the _scan_video helper none of them had a caller for, PNG_METADATA_KEYS, feather_region_composite and the remover region path that was only reachable from a no-caller convenience wrapper, remove_watermark_batch on both layers, try_empty_device_cache, the _generate/_run_qwen_zimage pass-through pair, self.model_id, and the _internal PEP 562 shim that no caller ever went through. get_device now answers cuda or cpu only: mps and xpu travelled one frame to the same CUDA-only refusal while costing a device probe each, and that refusal now names the resolved device, so device=None on a CUDA-less host says 'cpu' rather than 'None'. The XPU wheel index went with them. Docs: README, cli, installation, python-api, supported-signals, known-limitations and module-internals all still described the removed profiles, the CPU/MPS/XPU ladder, a `default`->`sdxl` alias, and the wrong extra. known-limitations still listed the retired SDXL strength ladder as current. scripts/smoke_matrix.py and real_examples_e2e.py drove --device mps. Next release is 0.25.0, not a patch: this removes public parameters and narrows a published extra on top of the released 0.24.0. pre-commit: 1) maintain.sh - exit 0 (1091 tests, Pyright 0 errors, no vulnerabilities); 2) /simplify - 4 agents, 11 findings applied, 2 skipped (dropping the `device` parameter entirely, which raiw-app pins; folding diffsynth into the `diffusion` extra, which video-only callers do not need); 3) docs sync - grepped every removed identifier across README, docs/, scripts/, .claude/; updated 9 docs; 4) CLAUDE.md - added the no-error-only-knobs rule to .claude/rules/development.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 5
|
b0ca2054f6 |
Keep only the two-stage profiles and make CUDA a precondition
qwen-zimage becomes the default and sdxl-zimage the only alternative. The
controlnet, sdxl, qwen and default profiles are gone, and with them the CPU and
MPS paths for invisible-watermark removal: neither matched the two-stage
recipe's face preservation, so keeping them advertised a quality this library no
longer delivers. Visible-mark removal and every identify command still run
anywhere.
Retired names are rejected rather than remapped. Silently routing --pipeline
sdxl onward would run an old script at a different strength, on a different
model, at a different quality, and report success.
CUDA is now checked when the remover is constructed instead of when the model
loads. Auto-detection cheerfully returned mps on a Mac, so the failure arrived
several layers down, after the dependency check and the pipeline import, in a
message naming whichever internal pipeline happened to raise. _DEVICES collapses
to {"cuda"} and the cpu/mps float32 branch goes with it.
resolve_strength stays total. It briefly returned None for qwen-zimage, meaning
"ask the resolution curve", which pushed a branch onto both callers and left one
of the two strength policies outside the strength module; the CLI copy had
already grown an `or 0.0` guarding a path its own comment called unreachable. It
now takes the image size and answers for both profiles, so the displayed value
cannot drift from the executed one.
Deletion fallout removed with it: img2img_runner and progress.py (the MPS
recovery path and its progress monitor had no callers left), viable_steps, the
fp16 degenerate-output retry, the fp16 VAE fix, and the Qwen img2img call
builders. try_empty_device_cache moved into watermark_remover rather than
leaving a module whose docstring outlived its code. _HAS_DIFFUSERS routes
through optional_deps.module_available, which is what the rest of the library
uses and what correctly rejects a pruned namespace remnant.
--steps, --guidance-scale and --model now have exactly one legal value each and
are still accepted at parse time, then rejected in remove(). Their help text
says so, but validating them beside the option would be better.
Not addressed, and worth its own decision: invisible_engine forces
min_resolution to 0 for both profiles, so the --min-resolution floor, --upscaler,
_esrgan_upscale, upscaler.py and the esrgan extra are all unreachable.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Victor KuznetsovandClaude Opus 5
|
3d43bac6a5 |
Load the face stage in its own dtype, not the global stage's
The sdxl-zimage profile crashed on every image containing a face. The remover gives it torch.float16, because SDXL ships fp16 weights and an fp16-safe VAE, and that dtype reached the inherited _load_zimage while _zimage_vram_config hardcodes bfloat16 for its offload, onload and computation dtypes. Z-Image was therefore built bf16 and handed fp16 latents, dying in the VAE with "Input type (c10::Half) and bias type (c10::BFloat16) should be the same". Zero-face inputs never enter _run_faces, so the profile passed every timing run it was given, and its tests avoid model downloads, so nothing exercised the loader. Every face-stage loader now reads _face_stage_dtype(), the computation dtype of the VRAM config it is paired with. SAM is routed through it too: it never crashed, since it casts its own inputs and leaves through .float(), but it read the same field and would have re-landed the bug for the next profile with a different global dtype. That field was never the global dtype on this profile anyway - _load_sdxl hardcodes fp16 for its own ControlNet, VAE and pipeline - so its only readers were face-stage code. This also fixes a second instance transitively: the persisted prompt-embedding cache restores payloads at the DiffSynth pipe's dtype, which was fp16 into a bf16 stack before this change. For qwen-zimage the whole change is a strict no-op. The remover already hands it bfloat16, the same value _face_stage_dtype() returns, so production is untouched; verified on an H100 against the deployed pin. The guard asserts the dtype the Z-Image and SAM loaders actually receive rather than comparing the accessor to the config it derives from, which would restate the implementation and pass for any consistently wrong value. Both assertions were mutation-tested against the pre-fix line. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 5
|
39523aba09 |
Serve the static prompt embeddings from a persistent cache
Both qwen-zimage stages prompt with module constants, and at CFG 1.0 DiffSynth's PipelineUnitRunner reuses the positive embedding for the negative side rather than encoding it, so exactly one embedding per stage is ever computed. Persist it and neither text encoder has to be loaded at all. Measured on an H100 volume: this drops 15.45 GiB (Qwen2.5-VL) and 7.49 GiB (Z-Image) of an 87.6 GiB per-request read, worth a median 11.76 s and 4.10 s of load time paired within five containers. A nine-face fixture returned sha256 c8567e11077de32a both with and without the cache, so the output is byte-identical and the provider-oracle clearance is untouched. The cache key carries the cache version, model id, pipeline output params and the exact prompt, so a model bump or a prompt edit recomputes instead of reading a stale embedding. The write is atomic because a torn file must never read back as a hit, and a miss after the text encoder was already dropped raises rather than calling a model that is not loaded. _model_cache_dir now prefers HF_HOME: on a scale-to-zero runner that is the only persistently mounted path, so anything below it is re-derived every request. The YuNet download follows the same root. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
9c9e81c756 | Compact CLAUDE.md and route development guidance |