mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-21 04:57:12 +02:00
05f415b5673c33fc7931d356a6201e27c20b1bf9
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Victor KuznetsovandClaude Opus 5
|
78d9e81d0f |
Collapse the duplicated detection path and lift the image pipeline into the library
The visible-mark path had grown three copies of one ladder sweep, four
near-identical `detect` arms, and four hand-rolled `footprint_mask` overrides;
mark knowledge sat in five hand-maintained tables across three modules; and the
flagship `all`/`batch` pipeline existed only in cli.py, written twice with
divergent behavior.
Detection is now one measurement. `_ladder_best` replaces the three sweeps,
`_scan`/`_verdict` replace the four arms, and the winning box travels to the
mask on `TextMarkDetection.match_box` instead of being swept a second time.
`detect_both` returns the strict and relaxed verdicts from one scan, which
halves the arbiter's perception cost (260 -> 130 matchTemplate calls on a 2048²
image, verdicts identical field for field). A per-mark demotion goes in the new
`_post_gate` hook, never in a `detect` override -- an override is invisible to
the single-pass path, which is how the RunningHub and Yuanbao anchor gates
briefly stopped applying.
Everything about a mark is now one registry row: product, label regime, the
platform sentence `identify` reports, the metadata signals that confirm it, and
its TC260 producer codes. `identify._VISIBLE_MARK_PLATFORM`, the signal mapping
in `api.visible_provenance`, `_PRODUCT_OF` and the pill veto are derived from
those rows.
`api.remove_all` / `api.remove_batch` are the library form of the `all` and
`batch` commands; the CLI is a wrapper that owns console text and exit codes.
Progress is a `(stage, detail)` pair of stable tokens, so the CLI keys its
wording off structure rather than parsing the library's prose back.
Two intentional behavior changes, both verified against a recorded 811-image
sample of detector verdicts, removal-mask hashes, arbiter decisions and
`identify` reports:
* A TC260 label now relaxes the vendor its `ContentProducer` names rather than
ByteDance's pair on every China-AIGC image. 333 of 811 samples move; on 185
of them the previously relaxed pair was simply the wrong vendor, and the
mark actually present never reached the relaxed gate its own
`provenance_ncc_factor` was calibrated for.
* A confident LibLibAI detection suppresses the Jimeng pill, like every other
TC260 product's mark. It was registered alongside RunningHub and Baidu, both
of which were added to the hand-written veto list, and it was not. 1 sample
moves, and it is exactly the co-firing case.
Nothing else in that record changes: detector verdicts, mask hashes and
`identify` verdicts are byte-identical, and all 200 calibration constants are
untouched.
Also: `aigc_label` and friends plus `extract_c2pa_info` are memoized on
(path, mtime_ns, size) -- size because this package rewrites in place; the
native TC260 container readers route on magic bytes instead of the file
extension, so a mislabeled AVI or FLV is no longer invisible; `identify` shares
one pixel decode between the DWT-DCT and visible stages (TrustMark keeps its own
Pillow decode, which is not substitutable); and the six `stabilize_*` video
wrappers collapse into one policy table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Victor KuznetsovandClaude Opus 4.8
|
633fc3aa52 |
Report failed writes instead of crashing, and reject directories at parse time
The Tier E adversarial sweep (new, scripts/robustness_suite.py) drove the real CLI over truncated, corrupt, zero-byte, absurdly-shaped and bomb inputs, unicode and RTL paths, hostile output directories and concurrent runs. It found two crashes; the /simplify review then reproduced a third and worse one. 1. A FAILED WRITE CRASHED ON THE SIZE REPORT. image_io.imwrite is contractually non-raising and returns False, but write_bgr_with_alpha discarded that bool and returned None, so no caller could tell a failed write from a successful one. Every write site then ran output.stat() to print the size, so a read-only destination died with a bare FileNotFoundError pointing at the stat rather than the write. The fix is deliberately NOT uniform: single-image commands exit via the new cli._write_output_or_exit; api._write_visible_result RAISES so a library caller gets an accurate error instead of a confusing FileNotFoundError from the downstream metadata strip; and the batch sites raise but never SystemExit, because the batch loop counts per-image exceptions and aborting would kill the whole run. 2. BATCH LOST DATA SILENTLY. Into a read-only output directory it wrote ZERO files for 2 inputs and exited 0 -- no traceback, no error, an empty output directory a wrapping service would read as a completed run. The robustness harness could not see this class at all, since it scored exit codes and traceback markers and this failure has neither; it now asserts on the artifacts written. 3. A DIRECTORY PASSED AS THE IMAGE crashed the metadata scanner with IsADirectoryError, because click.Path(exists=True) accepts directories. Fixed with dir_okay=False on all six source arguments, so argument parsing refuses it. Also adds Tier B4 (scripts/resource_ceilings.py): peak RSS per fill backend from 1 MP to 25 MP, one fresh process per cell. migan 603->775 MB and lama 4679->4779 MB are flat in input size, confirming the crop-around-the-mask design and both documented figures; cv2 is the only backend that grows (74->440 MB, 5.9x). The harness's own no-op check originally allocated a full-frame temp before reading peak RSS and inflated the numbers with input size -- it now compares only the mask box, and the conclusion survived re-measurement. And scripts/real_examples_e2e.py, which drives every command over real corpus examples and checks the outcome rather than the exit code: 6/6 provenance classes identified, 10/10 metadata strips re-scan clean, all three fill backends write, diffusion on MPS writes genuinely changed images. It records samsung as a real partial (the faintest mark, 0.431 -> 0.404 against a 0.40 gate on the weakest of its 3 corpus positives) and treats the gated pill's refusal to act as correct. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 4.8
|
a4c901ff39 |
fix: metadata-strip parity, input robustness, and detection/clash coverage
Bug fixes (each with a regression test): - metadata strip parity across every marker placement: IPTC digitalSourceType in XMP, the Samsung post-EOI trailer, the China TC260 AIGC block in EXIF UserComment, a bare AIGC block in a non-standard APP segment, and the ISOBMFF EXIF path (AIGC + xAI) are all now stripped -- anything a scanner flags, the strip reaches - Samsung genAIType detected when its trailer sits past the 512 KB scan window (file-tail read on large photos) - crashes on edge inputs: Gemini detector on images with a short side < 16px, footprint_mask on a zero-size ndarray, the humanizer on chromatic_shift >= width, and the CLI on unreadable/corrupt/empty input (clean error, not a traceback) - WebP written losslessly (cv2 quality 101), not lossy at 100 - the IPTC digitalSourceType algorithmicMedia (procedural, not trained on sampled data) is no longer flagged as AI-generated, so clean procedural content is not scrubbed - c2pa source-type: compositeWithTrainedAlgorithmicMedia is checked before the bare algorithmicMedia token, so an AI-enhanced composite is not misclassified Detection: - integrity-clash coverage now normalizes ByteDance / Canva / ElevenLabs / Black Forest Labs, so a transplanted manifest next to an independent conflicting stamp is caught; the generic China TC260 AIGC label is attributed to a co-present TC260 vendor, so a legit Doubao image (its own C2PA + TC260 label) does not clash (corpus-validated: 0 new clashes on 5000 carriers) CLI: - batch exits non-zero (with a warning) when any image errors or a GPU-missing SynthID scrub is skipped, and copies the input through so the output dir stays complete -- it used to always exit 0 and could silently drop files Perf: - GeminiEngine reused as a process-wide singleton with a precomputed template ladder: -24% on the identify sparkle path, detection byte-identical Internal: one shared _ai_exif_targets rule set feeds both EXIF scrubbers so their coverage cannot drift; docs synced; maintain.sh hardened so the uv-secure internal teardown crash no longer aborts the gate (still fails on a real finding). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 4.8
|
1a955b096a |
feat(visible): localize->fill rewrite, sensitivity/backend + api, HEIC + lossless IO
- Replace reverse-alpha removal with localize -> fill (template-free mask + one shared cv2/MI-GAN/big-LaMa fill) for every mark; drops the colour-shift / dark-pit failure modes, version-robust to a moved or re-rendered mark - Separate perception/decision/action: engines report Candidates, a pure decide(candidates, Context) arbiter owns all policy (sensitivity + provenance + pill gate), remove_auto_marks orchestrates -- behavior-preserving (corpus 46/46/92) - Three orthogonal knobs replace --method: --backend cv2|migan|lama, --sensitivity auto|strict|assume-ai, provenance (auto from metadata) - Add high-level api.remove_visible / visible_provenance (lazy top-level re-export); visible --mark auto delegates to it so CLI and library share ONE path - Read+write HEIC/AVIF on the pixel path via pillow-heif; imwrite preserves the input format at max quality (JPEG q100/4:4:4); a no-op copies the original bytes verbatim - Lossless byte-level JPEG metadata strip (no DCT re-encode); consolidate the two remove_ai_metadata into one, delete legacy noai/cleaner + best_auto_mark - Bump 0.13.0 -> 0.14.0 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 4.8
|
2fcd00ced0 |
fix: address whole-project code review (visible all/batch, engine consolidation, I/O)
Nine findings from a high-effort project-wide review, fixed and verified (571 passed, ruff/pyright clean): Correctness: - all/batch now remove Doubao/Jimeng/Samsung visible text marks: the visible step routes through the registry (new cli._remove_visible_auto) instead of a hardcoded GeminiEngine, so they no longer leave the wordmark intact. - batch always reads the original source (dropped the out_path-reuse that re-processed already-cleaned outputs on a re-run). - img2img_runner only retries the diffusion call on the deprecated-callback TypeError; any other TypeError now propagates instead of double-running. - gemini detect/remove and the reverse-alpha engines normalize channels via a new image_io.to_bgr, fixing a grayscale/BGRA crash in the FP-gate path. - _png_late_metadata advances its cursor by the clamped length, so a malformed chunk length no longer aborts the late AI-label scan. Cleanup / efficiency: - Consolidate the ~90%-identical Doubao/Jimeng/Samsung engines into a shared config-driven _text_mark_engine.TextMarkEngine base; each engine is now a thin subclass (TextMarkConfig + test shims). Behavior is byte-exact (the three engine test suites pass unchanged). Registry adapters collapse to one _text_mark(...) row each. Gemini stays a separate engine. - scan_head is memoized per (path, size, mtime), so identify() reads the file head once instead of ~8 times. - invisible_engine post-processing decodes/encodes the output once (chained in memory) instead of 2-4 times across stages. - Remove the orphaned get_model_id_for_profile (+ CONTROLNET_PROFILE); derive the --strength help from the strength constants (strength_default_help) so it cannot drift; share the --pipeline/--strength click options; simplify the retired --auto resolver. Net -835 lines. Tests added for the registry-routed visible pass, to_bgr, the polish/model/guidance wiring, and strength_default_help. CLAUDE.md updated for the new base module, the engine/registry changes, image_io.to_bgr, and the scan_head cache. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor KuznetsovandClaude Opus 4.8
|
5d0e6c3a65 |
fix: harden metadata parsers and engines; sync docs (full-repo review)
Apply fixes from a full-repo review (code, tests, docs). Security / correctness: - Clamp attacker-controlled PNG/caBX chunk lengths to the remaining file size in metadata.py and noai/c2pa.py (a malformed length no longer drives a multi-GB read); skipped chunks seek instead of read. - noai/isobmff.strip_c2pa_boxes is now fail-safe on a malformed box: return the original bytes with a warning instead of silently truncating the tail, so metadata --remove can no longer emit a corrupt file. - doubao_engine._fixed_alpha_map clamps the glyph box to the image (no crash on degenerate width-vs-height). - watermark_remover._run_region_hires gates the phaseCorrelate offset on response and magnitude (a spurious shift no longer garbles text) and drops the generator after a CPU fallback (no MPS/CPU device mismatch). Robustness: - gemini_engine, doubao_engine, region_eraser normalize grayscale and RGBA inputs to BGR at the engine entry points. - image_io.imwrite returns False on an unwritable path (matches cv2). - invisible_engine guards a None imread result before use. - trustmark_detector._decoder uses a double-checked threading lock. - ctrlregen.tiling.tile_positions raises on overlap >= tile. - humanizer chromatic shift no longer wraps opposite-edge pixels. - identify OpenAI caveat keyed on the normalized vendor, not a substring. - Remove the dead "visible --detect-threshold" CLI option. - publish.yml verifies the release tag matches the package version. Docs: - README strength 0.05 to 0.10; .env.example HF_TOKEN marked optional; doubao_capture README updated to reverse-alpha-only; CLAUDE.md synced with the new behaviors and the batch command. Tests: new test_security_clamp.py for the read clamp and isobmff fail-safe; erase CLI coverage; integrity-clash rule 2 end-to-end; multi-tag EXIF survival and cross-format strip guards; channel/size, tiling, humanizer, and imwrite regressions. Full suite 493 passed, 2 skipped; ruff and pyright src/ clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
test-userandClaude Opus 4.7
|
7b47fa9f6a |
fix(io): Unicode-safe cv2 image IO + un-eat the [gpu] install hint (v0.6.6)
Two CLI/IO robustness bugs surfaced by issues #17 and #19. #17 -- non-ASCII image paths (Chinese/Cyrillic/accented) failed on Windows: cv2.imread/imwrite use the platform ANSI code-page API, so the decode came back empty with a "can't open/read file" warning. New image_io.imread/imwrite route through np.fromfile+cv2.imdecode / cv2.imencode+tofile (Unicode-safe, byte- identical output, cv2.imread None-semantics preserved); all 8 cv2 read/write call sites now go through it. Behavior-neutral on macOS/Linux (already accept UTF-8 paths), so the fix is correct-by-construction for the Windows-only bug. #19 (incidental) -- rich parsed the "[gpu]" in the GPU-extra install hint as a style tag and dropped it, so the printed command was the un-installable "pip install 'remove-ai-watermarks'". Escaped as \[gpu] at both call sites. Tests: test_image_io.py (non-ASCII round-trip, alpha, missing/empty/garbage semantics); test_cli.py::TestGpuHintMarkup (install hint keeps the extra). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |