mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-07-05 07:57:50 +02:00
c1971a3e8d9be7a89afdf8ca67d5bd8a586449fc
38 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Victor Kuznetsov
|
373b910a60 |
docs: fix the qwen-vs-controlnet face comparison to oracle-confirmed scrub floors
The face fidelity numbers cited an equal-strength compare (both 0.15), but Qwen at 0.15 does NOT clear Gemini SynthID -- so that output is un-scrubbed and the compare is invalid. Per the methodology rule (compare fidelity only between outputs where SynthID is removed in BOTH), restate faces at each pipeline's scrub floor (controlnet 0.15 / Qwen 0.30): ArcFace identity 0.546 vs 0.331, lapvar 0.62 vs 0.40, face LPIPS 0.09 vs 0.19 -- controlnet still wins faces, conclusion unchanged. Drop the "equal strength" framing in CLAUDE.md / module-internals / known-limitations. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
2d5b26ed18 |
test(eval): vision-transcribed ground truth for qwen_in + clean text-CER numbers
data/qwen_in/ground_truth.json is transcribed by vision (PaddleOCR mangled the stylized Cyrillic), so the text metric scores variants against an accurate reference instead of noisy OCR-vs-OCR. Re-measured text CER (controlnet vs qwen) with this ground truth confirms qwen wins text across EN/RU/ZH: openai_1 0.385 vs 0.241, openai_2 0.341 vs 0.290, gemini_1 (ZH) 0.037 vs 0.000 (perfect Chinese even at the higher 0.30 strength). Faces still favor controlnet. Refresh the numbers in docs/known-limitations.md to this cleaner methodology. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
e29c156279 |
test(eval): fix the qwen_in pipeline-fidelity eval set + PaddleOCR ground-truth flow
- data/qwen_in/: a stable, committed set of 4 AI-generated images (OpenAI +
Google, carrying SynthID/C2PA -- same class as data/samples fixtures) used to
compare the controlnet/sdxl/qwen pipelines for fidelity. Two text-multi-script
(incl. RU/CJK), one EN poster, one face grid. README documents the set + the
ground-truth workflow. data/ is sdist-excluded so the wheel is unaffected.
- scripts/fidelity_metrics.py: switch text OCR from EasyOCR to PaddleOCR
(PP-OCRv6, higher accuracy esp. CJK, single multilingual stack); split into
`ocr` (seed a {basename: text} ground truth) and `compare` (--ground-truth for
a clean CER vs the hand-verified reference instead of noisy OCR-vs-OCR). Spatial
IoU-NMS keeps the best-scoring read per line so wrong-script models don't inject
garbage over Cyrillic/CJK.
- Oracle methodology: validate the OpenAI arm FIRST (openai.com/verify is more
accessible and the strongest Playwright/Chrome-MCP automation candidate; the
Gemini app is more manual). Recorded in CLAUDE.md + docs/synthid.md.
Ground-truth JSON (data/qwen_in/ground_truth.json) lands in a follow-up once
hand-verified.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Victor Kuznetsov
|
a2c33af284 |
feat(scripts): fidelity_metrics.py + correct the qwen-vs-controlnet claim
Add scripts/fidelity_metrics.py: an objective eval harness comparing watermark-removal outputs against the original (reference) across four groups -- OCR character error rate (EasyOCR), ArcFace identity cosine (insightface), face texture (LPIPS + Laplacian-variance ratio), and whole-image LPIPS/SSIM/ PSNR. PEP 723 inline deps so it stays out of the package / uv.lock; metrics self-gate (faces only where faces, text only where text). The metrics overturned an eyeball conclusion: at EQUAL strength Qwen beats controlnet on TEXT (OpenAI typography 0.10: OCR CER 0.25 vs 0.37) but controlnet beats Qwen on FACES (gemini_3, 18 faces, 0.15 each: Laplacian-variance retention 0.62 vs 0.41, face LPIPS 0.09 vs 0.13 -- Qwen smooths faces MORE; ArcFace identity ~tied). So Qwen is the better TEXT-preserving remover, not a universal fidelity win. Correct the earlier "qwen keeps faces faithful where controlnet plasticizes" claim in CLAUDE.md, module-internals.md, known-limitations.md, README. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
76e3d4154c |
feat(invisible): add Qwen-Image img2img pipeline (--pipeline qwen)
A third diffusion pipeline alongside sdxl/controlnet: Qwen-Image (20B MMDiT, Apache-2.0 code AND weights) img2img. The scrub still comes from the img2img strength; Qwen preserves text (incl. CJK) and structure markedly better than SDXL at the scrub floor, so it over-regenerates real photos far less (directly targets the controlnet over-regeneration that degrades real uploads). - watermark_profiles: QWEN_MODEL_ID, normalize_profile accepts "qwen". - WatermarkRemover: _load_qwen_pipeline (bf16, loads Qwen base unless --model overridden, clear ImportError if diffusers lacks the class), _run_qwen (no MPS fallback -- 20B is CUDA/cloud-class), dispatch in _generate_one/preload, pure _build_qwen_kwargs (true_cfg_scale, not guidance_scale). - Shared _base_load_kwargs() across all three loaders (dtype + token). - CLI --pipeline gains "qwen"; invisible_engine threads it through. - scripts/qwen_scrub_prototype.py: standalone PEP 723 GPU experiment. Prototype oracle floors (Modal A100-80GB, single seed, controls SynthID-positive, PENDING seed-repeat cert): OpenAI clears at strength ~0.10, Gemini at ~0.30 (0.20 still detected), with CJK text + faces faithful where controlnet plasticizes. The Gemini floor is higher than the shared default ladder, so pass an explicit --strength for Gemini on this pipeline until a Qwen-specific ladder is certified. The model-running path is CUDA-only (untestable locally); unit tests cover the pure call-shape (_build_qwen_kwargs) and profile normalization without torch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
0c0c6c6b03 |
feat(invisible): sliding-window tiled diffusion for large inputs (--tile)
Add a lossless alternative to the --max-resolution downscale for large images that OOM on MPS/GPU: regenerate in overlapping, feather-blended tiles at native resolution. - noai/tiling.py: pure plan_tiles (uniform tiles, last flush to edge) + feather_weights (strictly-positive separable taper -> partition-of-unity blend) + run_tiled (per-tile generate callable, decoupled from the pipeline). Unit-tested without the model. - WatermarkRemover.remove_watermark: refactor _generate into _generate_one + a tiled branch that engages only when --tile is set and the long side exceeds tile_size (ControlNet canny is rebuilt per tile). - Thread tile/tile_size/tile_overlap through InvisibleEngine and the invisible/all/batch CLI commands via a shared _tile_options decorator. Verified end-to-end on the real SDXL pipeline (forced 2x2 tiling on a 1024px sample, MPS): non-degenerate output, no gross seam at tile borders. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
d5845a72f3 |
feat(metadata): blank AI-generator tokens in AVIF/HEIF Exif meta-box items
Closes a documented coverage gap (P2#9): an AI Software/Make/Artist/ImageDescription token in an EXIF item (its TIFF bytes live in mdat/idat) survived remove_ai_metadata because the top-level box stripper and (absent pillow-heif) the PIL EXIF reader can't reach it. New isobmff.blank_ai_exif_tokens finds EXIF TIFF blocks by their II/MM byte-order header, validates each with piexif (a coincidental II/MM run in pixels won't parse as a TIFF IFD, so it's ignored), and overwrites any AI_GENERATOR_TOKENS- bearing value with same-length spaces -- so box sizes and iloc offsets stay valid and the coded image is untouched (mirrors blank_ai_xmp_packets; no iinf/iloc surgery, no exiftool dep). Camera/editor EXIF without an AI token is preserved. Wired into remove_ai_metadata's ISOBMFF path. Covers the realistic AI-generator-token case; xAI- signature-in-meta-box-EXIF (Grok is JPEG-only) stays out. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
3f5d6a0af1 |
docs(landscape): back the DWT-DCT positive-only limitation with researched root cause + citations
Deep-research (2026-06-19, adversarially verified) confirms the open imwatermark dwtDct mark is fragile by scheme, not by our usage: maintainers admit no 100% clean-decode guarantee; measured ~0.79 bit accuracy clean (~38/48, below our 44 gate). Root causes (code-verified + locally reproduced): per-block max-coefficient bit read (content flips bits) and YUV chroma 8-bit clamping on bright pixels (the bright-flat / all-ones failure). No maintained fork or detector does this scheme reliably (WAVES relegates it to an appendix; learned schemes are a different class; dwtDctSvd cannot decode SDXL's dwtDct). Conclusion: keep it positive-only, rely on C2PA. Sources: imwatermark READMEs, arXiv:2406.08337 (WMAdapter), arXiv:2401.08573 (WAVES), diffusers SDXL watermark.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
4c8a57ec7b |
docs: dwtDct detector is carrier-fragile (all-ones = artifact), FLUX open-mark unresolvable
Final characterization after a positive-control sweep. The imwatermark dwtDct round-trip fails (28-39/48, below the 44 gate) not on "high texture" as a prior note claimed, but on a broad carrier class: the FLUX fox, doubao, a minimalist-FLAT FLUX generation, AND a clean synthetic bright-flat fill with NO watermark all fail identically. The degenerate all-ones decode is therefore a CARRIER ARTIFACT, not a watermark (the no-watermark synthetic image reproduces it; a double-embed test shows no interference). detect_invisible_watermark is positive-only: trust a hit, treat a None as inconclusive unless a same-carrier positive control first recovers >=44. Consequence: whether BFL hosted FLUX embeds the open DWT-DCT is unresolvable with this detector on the available carriers (textured AND flat FLUX both fail the control). C2PA stays the reliable FLUX signal. Low priority to chase further. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
a0a349cc66 |
docs: correct overstated FLUX open-watermark claim; record detector content-fragility
Earlier notes asserted BFL hosted output has no open DWT-DCT watermark. That was overstated: the test carriers were high-texture fox images where a clean encode->decode round-trip of a KNOWN-embedded watermark recovers only 28-35/48 bits (below the safe 44 gate), so the detector would miss a present mark there -- the None is inconclusive, not proof of absence. Verified positive-control (2026-06-19): imwatermark dwtDct round-trips 48/48 on synthetic carriers and on chatgpt-1.png (48/48) / firefly-1.png (45/48), but FAILS on flux-1.png (28/48) and doubao-1.png (39/48). So invisible_watermark detection is a positive-only signal: trust a hit, treat a miss on busy content as inconclusive. Affects all open SD/SDXL/FLUX DWT-DCT detection. C2PA stays the reliable FLUX identifier; whether BFL hosted embeds the open mark is unresolved (needs a low-texture hosted sample). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
99e57c872f |
perf(text-mark): footprint-sized arrays in reverse-alpha CPU path
The reverse-alpha text-mark engine (Doubao/Jimeng/Samsung) allocated
full-frame arrays where only the glyph footprint is ever read:
- _fixed_alpha_map / _aligned_alpha_map each built a full (h, w) float32
alpha map non-zero only inside the glyph box, and two were held at once
during removal (~96 MB of mostly-zeros on a 12 MP frame);
- extract_mask built a full (h, w) uint8 mask that every caller cropped to
the located box (~12 MB, rebuilt per text-mark detector on the
memory-tight identify path).
Both now return footprint-sized arrays: the alpha helpers return the
glyph-sized block plus its placement (ax, ay, gw, gh), and extract_mask
returns the box-sized mask. _apply_reverse_alpha consumes the block
directly; the residual inpaint embeds it into one full-frame uint8 mask only
at cv2.inpaint time (which needs a full-frame mask). remove_watermark_
reverse_alpha tracks the winning region alongside best_amap to place it.
Peak allocation drops from O(image*4)x2 + O(image) to O(footprint)x2 +
one gated O(image*1) uint8 mask -- a win every consumer gets, motivated by
the 512 MB raiw.cc worker that OOMs on large decodes. GPU path untouched.
Byte-identical to the old full-frame path (verified: 17 output hashes
across the three engines, inpaint/no-inpaint, detect, and the real
doubao-1.png fixture, unchanged before/after). tests/test_text_mark_memory.py
guards it by reconstructing the old full-frame path inline and asserting
equality, so the proof survives a cv2/asset bump, and pins the O(footprint)
shape so a regression to full-frame fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
||
|
Victor Kuznetsov
|
9614615001 |
docs(landscape): confirm BFL hosted = C2PA-only on FLUX.1 [dev] too
Lossless-PNG check across both BFL Playground model lines (FLUX.2 [pro] and FLUX.1 [dev]) confirms the open DWT-DCT pixel watermark is absent on hosted output regardless of model or container; only the signed C2PA manifest is present. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
9e307d020e |
test(c2pa): add real FLUX.2 BFL C2PA fixtures (PNG + JPEG)
flux-1.png / flux-1.jpg are real Black Forest Labs FLUX.2 [pro] Playground outputs (signed C2PA, issuer "Black Forest Labs" + trainedAlgorithmicMedia, manifests verified to contain no personal data). flux-1.jpg is the first committed JPEG-with-C2PA fixture, exercising the c2pa-python non-PNG reader path end to end. Regression tests assert both attribute to "Black Forest Labs (FLUX)". Also documents the verified finding (n=2, 2026-06-19): BFL's hosted output carries the signed C2PA manifest but NOT the open invisible-watermark DWT-DCT (decodes to degenerate all-ones, chance-level vs the FLUX reference) -- the open pixel mark is dev-inference-code-optional only. So a hosted FLUX.2 image is identified by C2PA alone, with no open-pixel fallback once C2PA is stripped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
d4d9429328 |
feat(identify): attribute Canva and BytePlus C2PA; fix BytePlus->Adobe mislabel
Mining the local production corpus (25,725 imgs) surfaced two AI vendors signing C2PA that the registry missed: - Canva (Magic Media) signed "Canva" + trainedAlgorithmicMedia -> detected AI but no platform attributed (disproves the old "Canva exports strip C2PA" assumption). - BytePlus (ByteDance international: Seedream/Seededit) signs "Byteplus Pte. Ltd."; the bare volcengine needle missed it, so its output was mis-attributed to "Adobe Firefly" via an incidental "Adobe XMP" string the fallback byte-scan picked up. Adding both to C2PA_AI_VENDORS lets the clean manifest issuer attribute them directly. Corpus re-run: 16 platform changes, all improvements (3 Adobe->ByteDance fixes, 4 None/TC260->ByteDance, 9 None->Canva), 0 regressions. An attempted signer-based attribution fallback was measured and dropped: it regressed 18 images (friendly ByteDance label -> raw Chinese cert org; IPTC tool name pre-empted). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
41f67973ce |
fix(visible): inpaint mid-tone Gemini sparkle instead of a dark diamond
The free `visible` path over-subtracted a faint Gemini sparkle on a mid-tone background into a darker-than-background brown diamond instead of removing it (2026-06-18 prod NPS report, "the watermark was not removed, just its color changed"). The existing over-subtraction guard only tripped when reverse-alpha drove a footprint pixel fully negative (the issue #30 dark-background black-pit case); on a mid-tone background the over-subtraction darkens the core well below the background without any pixel crossing zero, so the gate missed it and shipped the dark mark. Add a second over-subtraction signal to `_reverse_alpha_oversubtracts`: predict the reverse-alpha output at the bright core, (core - a*logo)/(1-a), and route to the footprint inpaint when it lands more than `_OVERSUB_DARK_MARGIN` (25) gray levels below the local background ring. Calibrated wide: clean removals predict within ~12 of background (demo_banana ~-1), the prod regression ~-40, the issue #30 dark case ~-82. Corpus-validated on the 479 detected Gemini images: 10 switch reverse-alpha to inpaint, all of them dark-diamond cases that improve or match; the other 469 stay byte-identical. demo_banana stays on the reverse-alpha path (byte-identical). Also crop both reverse-alpha helpers to the region they actually touch, a pure O(image) -> O(mark) win that is byte-identical to the full-frame math (a uint8<->float32 round-trip is exact): - `GeminiEngine._core_and_bg` converts only the footprint+ring crop to gray, not the whole frame (~70 ms -> 0.1 ms on a 12 MP image; it runs for both the alpha-gain estimate and the new gate). Verified identical across 479 images; detector confidence unchanged. - `TextMarkEngine._apply_reverse_alpha` computes the blend on the glyph crop only (`amap` is zero outside it, so the math is a no-op there): ~275 ms -> ~2 ms per placement on a 12 MP frame, up to 2 placements per removal. Verified identical across 142 Doubao/Jimeng placements. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
4c6b56f888 |
lower(strength): drop vendor-adaptive floor to OpenAI 0.10 / Google 0.15
A 2026-06-14 oracle re-test on the deployed Modal controlnet worker (v0.10.0) cleared SynthID at OpenAI 0.10 (2 photoreal) and Google 0.15 (2 native 2816x1536, retiring the "native >= 0.30" guess), while a pixel sweep showed the 2026-06-04 cert floors (0.20/0.30) over-regenerated for no efficacy gain (Google MAE -20% at 0.15). Lowers OPENAI_STRENGTH 0.20->0.10, GEMINI_STRENGTH and UNKNOWN_STRENGTH 0.30->0.15. Caveats documented in watermark_profiles.py + docs: removal near this floor is seed-non-deterministic (a service must pin a verified seed), and the n=2 re-test did not cover flat-graphic hard cases. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
28569bd05d |
fix(gemini): recover sub-0.85 corner sparkles via top-K fusion selection
The 256->512 detection-search widening (v0.8) let a large, low-gradient shape match outrank a genuine mid-size corner sparkle whose raw NCC sits below the 0.85 corner-promote gate, so `identify` read `unknown` on Gemini images that v0.7.2 caught (reporter osachub: scale-48 sparkle on light bedding -- true sparkle spatial 0.775 / grad 0.960 / fusion 0.676, but the size-weighted argmax locked onto a decoy at spatial 0.628 / grad 0.036). detect_watermark now keeps the top-K (_SELECT_TOPK=3) size-weighted candidates (NMS-deduped) plus the corner-promote candidate, scores each by full fusion (spatial+gradient+variance) via the extracted _grad_var_scores helper, and selects the highest -- the gradient term lifts the true sparkle over the decoy. Ranking by the SIZE-WEIGHTED score (not a raw-NCC argmax) preserves tiny-patch suppression: a raw-NCC argmax re-admitted 16-18px content false positives (14/65 doubao + 4/11 jimeng visible images). Top-K adds zero flips on the doubao/jimeng corpora and leaves the 495-image Gemini set unchanged (479 detected) while recovering the reporter's image at 0.676. - _grad_var_scores: gradient/variance scoring factored out of detect_watermark - confidence = best_fused (drop the duplicated fusion recompute) - tests: rename test_promotion_is_what_rescues_it -> test_size_weighted_search_alone_traps_on_the_decoy (corner-promote is no longer the sole rescue path); add a deterministic regression test mirroring the real spatial/grad signature - docs: module-internals.md detector section + CLAUDE.md mechanism map Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
9feea4ac1e |
Slim CLAUDE.md: move module internals, limitations, landscape research to docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
b1189549b8 |
feat(invisible): controlnet default, unified strength, retire --auto, add --model/--guidance-scale
Overhaul the diffusion-removal surface around a single robust default and a complete, consistent CLI. Pipeline + strength: - controlnet is now the DEFAULT pipeline (CLI --pipeline + both engine ctors). With the certified higher strength it clears both photoreal and flat-graphic content, whereas plain SDXL left SynthID on flat graphics. - Rename the plain-SDXL profile default -> sdxl; "default" stays as a back-compat alias (normalize_profile + a click callback that warns). - Unify the strength ladder: resolve_strength applies ONE vendor-adaptive ladder (the certified controlnet floors OpenAI 0.20 / Google 0.30 / unknown 0.30) to both pipelines. sdxl is the weaker remover on its own hard case (flat fills), so the certified floor is the right floor for it too. CLI completeness: - Add --model (HF model id) to invisible + batch (was only on all) and --guidance-scale (CFG) to all three diffusion commands; both were library knobs the CLI did not expose. - Flip --adaptive-polish to ON by default (it self-gates to a no-op where there is no detail deficit, so default-on is safe). - Share --pipeline / --strength / --model / --guidance-scale as single decorators so invisible/all/batch keep an identical surface; the --strength help is derived from the strength constants (strength_default_help) so it can never drift from the ladder. Removals: - Delete the auto_config content-detection planner + its YuNet/DBNet assets (~2.6 MB): with controlnet always the pipeline and the polish self-gating, the face/text/edge detection no longer changed behavior. --auto is now a deprecated no-op that only warns (the polish it enabled is the default). Docs (README, CLAUDE.md, docs/synthid.md) updated throughout; added an InvisibleEngine Python API example. Tests cover the alias warnings, the polish default, and the --model/--guidance-scale wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
20d7eda96a |
remove: drop all face-restore code (regeneration, not preservation)
Empirical conclusion from the 2026-06-04 - 2026-06-08 Modal cert sweeps: every face-restore approach we built (GFPGAN-on-cleaned, PhotoMaker-V2, InstantID txt2img, InstantID img2img-on-cleaned at three parameter settings) regenerates the face via SDXL diffusion rather than preserves it. Output face pixels are diffusion-fresh, so the regenerated face inherits SDXL "clean skin" aesthetic and loses original identity precision -- it looks MORE AI-generated than the cleaned image, not less. The cleaned image from the main controlnet 0.20 removal pass is the least-AI face state we can reach without re-introducing SynthID. Nothing in the restore family achieves the actual goal (preserve the original person's face). Keeping them around as opt-in invites users to ship something that defeats the point. Removing entirely. Library changes: - Deleted src/remove_ai_watermarks/instantid_restore.py - Deleted src/remove_ai_watermarks/photomaker_restore.py - Deleted tests/test_instantid_restore.py - Deleted tests/test_photomaker_restore.py - Removed `instantid` and `photomaker` extras from pyproject.toml - Removed `[tool.hatch.metadata] allow-direct-references = true` (was only needed for the photomaker git+ URL) - InvisibleEngine.remove_watermark: dropped `restore_faces` + `restore_faces_method` params, removed both `_restore_faces_instantid` and `_restore_faces_photomaker` private methods, removed dispatch - CLI: dropped `_restore_faces_options` decorator, all four cmd_* signatures lose `restore_faces` + `restore_faces_method`, kwarg passes to remove_watermark dropped - _apply_auto: dropped `restore_faces` from tuple shape (was unused after the engine no longer takes it) - auto_config.AutoConfig: dropped `restore_faces` field; `plan()` no longer sets it; `reason` no longer mentions it - Tests updated accordingly (test_auto_config.TestReason no longer asserts "face-restore on" in the reason string) Docs updated: - CLAUDE.md: removed the photomaker extras bullet, the Face restore trade-off bullet, the instantid_restore.py + photomaker_restore.py module bullets; replaced restore mentions in watermark_remover and controlnet bullets and prod recipe with the empirical conclusion - README.md: removed both `--restore-faces` callouts and the install snippet; the feature bullet and auto-mode comment updated - docs/synthid-robust-identity-research.md: added Status-retired notice at the top pointing at the 2026-06-08 followup raiw-app: - modal_cert.py: dropped `--restore-faces` flag entirely; sweep() no longer takes restore_faces; pinned _LIB_SPEC to `[gpu]` extras (no `photomaker` / `instantid` extras), points at main ruff + strict pyright clean; 569 tests pass; 18 restore-specific tests gone. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
567f3ae729 |
docs(restore): document that restore methods REGENERATE, not preserve
Empirical conclusion from the 2026-06-04 - 2026-06-08 cert sweeps: every shipped face-restore method (GFPGAN-on-cleaned, PhotoMaker-V2, InstantID txt2img, InstantID img2img-on-cleaned at three parameter settings) regenerates the face from an ArcFace embedding via SDXL diffusion. Output face pixels are diffusion-fresh, which makes the regenerated face look MORE AI-generated than the cleaned image (gloss, symmetric pores, SDXL "clean skin" aesthetic) regardless of license. The cleaned image from the main controlnet 0.20 removal pass is the LEAST-AI state we can reach without re-introducing SynthID; any restore on top trades original-look for embedding-driven regeneration. The fundamental issue is structural: ArcFace encodes "general look" at 512 dimensions, SDXL decodes that into pixels with the inherent SDXL aesthetic. Stronger identity push (higher strength + IP-Adapter scale) makes the face closer to the embedding but more AI-looking; weaker push leaves identity to drift further. No parameter setting recovers original identity AND looks less AI than cleaned. Production conclusion: do not ship `--restore-faces` in any monetized deployment. The extras (`instantid`, `photomaker`) stay in the library for research / personal use where users explicitly want regeneration. Documented at every entry point: - CLAUDE.md: new "Face restore trade-off" bullet + every restore mention rewritten to "REGENERATES, does NOT recover"; controlnet bullet updated - README.md: feature bullet + callout + secondary mention all updated - docs/synthid-robust-identity-research-2026-06-08.md: appended "Empirical follow-up" section documenting the InstantID sweep phases (Phase 1 txt2img v1/v2/v3, Phase 2 img2img defaults + stronger params) - docs/controlnet-removal-pipeline-research.md: updated restore-faces bullet to reflect the empirical conclusion - CLI help: `_restore_faces_options` docstring + `--restore-faces` / `--restore-faces-method` help text all updated Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
70e8b3a517 |
feat(face-restore): add InstantID as the default non-commercial restore path
Per the 2026-06-08 deep-research synthesis (docs/synthid-robust-identity- research-2026-06-08.md), the entire ArcFace-class identity-adapter ecosystem for SDXL is blocked from commercial use by InsightFace's non-commercial model packs (antelopev2 / buffalo_l). No commercial-safe ArcFace-grade identity stack exists today. The user explicitly opted into shipping a non-commercial restore path (research / personal use; raiw.cc must NOT install the extra). Architectural choice: InstantID over PhotoMaker-V2 as the default. - PhotoMaker-V2 (CLIP+ArcFace dual encoder, txt2img only): documented upstream identity drift on Asian male faces, visually confirmed in our cert sweep (tatsunari rendered as a generic woman; group photo collapsed into a patchwork). - InstantID (ArcFace cross-attention + landmark ControlNet): semantic identity branch + spatial weak landmark control, decoupled. Per InstantID paper (arXiv:2401.07519) and the research report, stronger identity fidelity on single portraits. Critically: NO original face pixels enter the diffusion (ArcFace embedding is semantic, landmark stick figure is pure geometry), so SynthID is not transported. Implementation: - New `src/remove_ai_watermarks/instantid_restore.py` mirrors the `photomaker_restore.py` shape (lazy singletons for pipeline + FaceAnalysis, per-face crop + _composite_faces from photomaker_restore). Loads the InstantID community pipeline via `DiffusionPipeline.from_pretrained( custom_pipeline="pipeline_stable_diffusion_xl_instantid")` -- no upstream Python package needed; diffusers fetches the file from its community examples. - New `instantid` extra in pyproject (insightface + onnxruntime + huggingface-hub). NON-COMMERCIAL block in the comment explains why. - CLI: `--restore-faces-method [instantid|photomaker]`, default `instantid`. Both methods explicitly labeled NON-COMMERCIAL in the help text. - Engine: dispatch on `restore_faces_method` to either `_restore_faces_instantid` or `_restore_faces_photomaker`. - 9 control-flow tests for InstantID without model download (mirror the photomaker_restore.py test pattern + draw_kps helper checks). 587/587 pass. Diffusers-0.38 compat verified by upstream code inspection: the InstantID pipeline inherits from `StableDiffusionXLControlNetPipeline`, uses only public diffusers APIs (`encode_prompt`, `prepare_image`, `prepare_latents`, `get_guidance_scale_embedding`), uses legacy attention processor API which diffusers preserves for backward compat. No PhotoMaker-V1-style internal text_encoder access. End-to-end execution will be validated by the Modal cert sweep in the next step. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
65de8df5c5 |
refactor(face-restore): drop GFPGAN, ship PhotoMaker-V2 as the sole restore (non-commercial)
Visual review of the GFPGAN-on-cleaned output (9-face grid, 1448x1086) showed it only polished the already-drifted face without restoring identity — useless for the "restore who is in the photo" intent. Dropping it. The shipped restore path is now PhotoMaker-V2, which delivers true identity-from- embedding face regeneration via a CLIP+ArcFace dual encoder. The ArcFace branch pulls InsightFace antelopev2/buffalo_l model packs at runtime, which InsightFace releases under a research-only license, so the whole extra is **NON-COMMERCIAL**. raiw.cc and any monetized deployment must NOT install the `photomaker` extra. This is called out at every entry point: CLI flag help, module docstring, pyproject extra block, CLAUDE.md extras bullet, README install snippet. Changes: - Deleted `src/remove_ai_watermarks/face_restore.py` and its tests. - Deleted the `restore` extra (gfpgan/facexlib/basicsr + scipy<1.18 / numba<0.60 pins) and the basicsr setuptools<69 build pin from pyproject.toml. - Restored `src/remove_ai_watermarks/photomaker_restore.py` (V2 this time: `TencentARC/PhotoMaker-V2`, `photomaker-v2.bin`, no `pm_version='v1'` override). - Restored the `photomaker` extra in pyproject with all the upstream-compat pins (einops, peft, onnxruntime, insightface) and the `allow-direct-references` hatch metadata block. - `InvisibleEngine` swapped `_restore_faces` -> `_restore_faces_photomaker`; `--restore-faces-method` removed (only one method, no choice). - CLI flag help, CLAUDE.md, README, docs/synthid.md, and docs/controlnet-removal-pipeline-research.md all updated. - docs/synthid-robust-identity-research.md status notice rewritten to list both abandoned commercial-safe attempts (V1 + GFPGAN-on-cleaned) and the non-commercial trade-off we accepted. ruff + strict pyright(src/) clean; 578 tests pass (the 9 GFPGAN tests are gone, the 11 PhotoMaker tests stay green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
01fe98bf54 |
refactor(face-restore): rollback PhotoMaker, restore GFPGAN on the CLEANED image
After 7 cascading upstream-compat fixes (insightface dep, peft dep, pm_version, device, etc.), the PhotoMaker V1 cert sweep still hit a CFG batch-dim mismatch inside the denoising loop. The upstream PhotoMaker `pipeline.py` is forked from diffusers v0.29.1 and our env runs 0.38; SDXL prompt-encoder handling changed significantly between those versions, so making PhotoMaker work end-to-end needs a proper fork or a diffusers downgrade — both expensive. Not worth shipping today. Pivot: restore `face_restore.py` (GFPGAN) with a single-line fix that makes it SynthID-safe by construction. The previous design ran GFPGAN.enhance on the ORIGINAL watermarked image and was oracle-confirmed to re-add SynthID via the weight-0.5 pixel blend. The fix is to run GFPGAN on the diffusion-CLEANED image — whatever pixels GFPGAN derives from are already SynthID-free, so the partial blend cannot transport the watermark. Identity fidelity is lower than a true identity-as-embedding stack would deliver, but it ships and works. Changes: - `src/remove_ai_watermarks/face_restore.py` restored from pre-wipe state with one line changed: `restorer.enhance(cleaned_bgr, ...)` instead of `restorer.enhance(original_bgr, ...)`. `original_bgr` is kept as an unused positional argument for API stability. - `src/remove_ai_watermarks/photomaker_restore.py` and its tests REMOVED. The research note (`docs/synthid-robust-identity-research.md`) keeps a "status notice" documenting why PhotoMaker is parked for now and what the path back in would look like. - `pyproject.toml` `restore` extra restored (gfpgan/facexlib/basicsr + scipy<1.18 + numba<0.60 pins + the basicsr setuptools<69 build pin), plus `photomaker` extra (with its einops/insightface/peft pile) and the `[tool.hatch.metadata] allow-direct-references = true` block REMOVED. - `InvisibleEngine._restore_faces_photomaker` removed; `_restore_faces` restored. The `--restore-faces` CLI flag and its plumbing through cmd_* signatures are unchanged. - CLAUDE.md, README.md, docs/synthid.md, docs/controlnet-removal-pipeline- research.md updated to describe the shipped GFPGAN-on-cleaned design and to reference PhotoMaker only as the parked alternative. ruff + strict pyright(src/) clean; 578 tests pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
dfa5181309 |
fix(photomaker): switch to V1 — V2 actually requires InsightFace (non-commercial)
A Modal cert sweep caught what the research doc missed: PhotoMaker-V2 fails at
import without InsightFace ("No module named 'insightface'"). Reading the upstream
source confirms it: `photomaker/__init__.py` imports `FaceAnalysis2` (an InsightFace
wrapper) at module load, V2's encoder is named
`PhotoMakerIDEncoder_CLIPInsightfaceExtendtoken`, and `model_v2.py`'s forward
takes an `id_embeds` argument that the pipeline computes via
`insightface.app.FaceAnalysis(name='antelopev2', ...)`. So V2 is a DUAL encoder
(CLIP + ArcFace), not CLIP-only as the model card line "id_encoder includes
finetuned OpenCLIP-ViT-H-14 and a few fuse layers" implied.
InsightFace's pretrained model packs (antelopev2, buffalo_l) are research/
non-commercial only per their own README:
"The pretrained models we provided with this library are available for
non-commercial research purposes only."
So V2 is blocked for a paid service like raiw.cc.
PhotoMaker-V1 is the commercial-safe alternative — its `PhotoMakerIDEncoder`
(model.py) forward takes only `(id_pixel_values, prompt_embeds, class_tokens_mask)`,
no ArcFace branch. Identity is CLIP-only, license is Apache-2.0, no InsightFace.
Code change: swap the repo + filename constants in `photomaker_restore.py`
(TencentARC/PhotoMaker, photomaker-v1.bin). Tests still pass (the 9 PhotoMaker
tests use a fake pipeline, so the model swap is transparent to them).
Doc correction: rewrote the verdict / license table / section 5 of
`docs/synthid-robust-identity-research.md` to lead with V1 and add a correction
notice explaining the V2 misread. Bulk-renamed `PhotoMaker-V2` to `PhotoMaker-V1`
across CLAUDE.md, README.md, docs/synthid.md, and
docs/controlnet-removal-pipeline-research.md (kept V2 only in the correction
notice, the license table, and the anchor reference).
ruff clean; 578 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Victor Kuznetsov
|
439eeadc07 |
refactor(face-restore): wipe GFPGAN path, --restore-faces is PhotoMaker-only
The GFPGAN `restore` extra and its `face_restore.py` module are gone. They were oracle-confirmed to re-introduce SynthID by blending watermarked original face pixels at fidelity weight 0.5 (clean A/B: gemini_3 controlnet 0.20 detected WITH GFPGAN, clean WITHOUT). Keeping them as the default restore method was a footgun for the removal pipeline. PhotoMaker-V2 (added in the previous commit) is the single shipped restore path now -- identity-as-embedding, SynthID-safe by construction. Removed: - src/remove_ai_watermarks/face_restore.py + tests/test_face_restore.py - pyproject.toml `restore` extra (gfpgan/facexlib/basicsr + scipy/numba pins) - pyproject.toml `[tool.uv.extra-build-dependencies] basicsr = [...]` build pin - CLI: `--restore-faces-method` and `--restore-faces-weight` (no method choice to make, no GFPGAN weight knob to expose) - InvisibleEngine._restore_faces method (only _restore_faces_photomaker remains) - All restore-faces-method / restore-faces-weight threading through cmd_* signatures and _process_batch_image Kept: - `--restore-faces / --no-restore-faces`: now binds to PhotoMaker-V2. - All adopted oracle findings about GFPGAN re-introducing SynthID (kept in the research docs as historical context that explains why the path was removed). Docs updated: CLAUDE.md (restore extras bullet collapsed to photomaker, removed face_restore Key-modules bullet, several inline GFPGAN refs scrubbed), README.md (face-identity callout + install section now point to the photomaker extra), docs/synthid.md 5.5 (net recipe), docs/controlnet-removal-pipeline-research.md (recommendations). ruff + strict pyright (src/) clean; 578 tests pass (the 9 GFPGAN tests are gone, the 9 PhotoMaker tests stay green). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
f8f247308b |
docs(identity): smoke test confirms OpenCLIP embedding is invariant to SynthID-magnitude noise
Empirical confirmation of the load-bearing assumption in the PhotoMaker-V2 path: the identity embedding cannot transport an invisible pixel watermark. Tested OpenCLIP-ViT-H/14 (laion2B-s32B-b79K — the same encoder PhotoMaker-V2 fine-tunes) on 31 face crops from gemini_3/gemini_4/openai_3 grid. cosine similarity between embed(orig) and embed(perturbed): - synthid_proxy (±2 LSB low-frequency noise, the regime SynthID actually lives in): mean 0.9977, min 0.9937. Embedding moves by 0.002 — an order of magnitude less than JPEG90 (mean 0.928), which SynthID survives at >=99% TPR by design. - noise3 / jpeg70 / blur1: 0.89-0.95, all clearly above the SynthID floor. - self check: 1.0000 (pipeline sane). So the embedder discards exactly the dimensions SynthID hides in. PhotoMaker-V2 conditioned on a watermarked face will see the same identity vector as a clean face of that person, so the generated face inherits identity, not the watermark. This unblocks step 2 of the research plan: prototype PhotoMaker-V2 in the controlnet pipeline. The previously logged ad-hoc "cos(orig, SDXL-cleaned)" numbers (0.56-0.93) measured diffusion drift, not watermark invariance, and are not relevant to the hypothesis. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
310ce912ba |
docs: SynthID-robust identity research — PhotoMaker-V2 is the only commercial-safe SDXL stack
After GFPGAN restore was oracle-confirmed to RE-INTRODUCE SynthID (it is a fidelity- restoration net conditioned on the watermarked input), the only identity path that will not transport the watermark is identity-by-EMBEDDING: a semantic vector that conditions a fresh generation. That requires a face-recognition / ArcFace-class or CLIP-image embedder. Verified the license stack of every credible 2025-2026 SDXL identity adapter by fetching primary sources directly (HuggingFace model cards, insightface.ai): - IP-Adapter FaceID family, InstantID, PuLID, Arc2Face -> all blocked. Each depends at runtime on InsightFace's antelopev2/buffalo_l ArcFace packs, and insightface.ai explicitly states "Code is MIT licensed; models require separate commercial licensing." IP-Adapter FaceID's own model card flags itself non- commercial for the same reason. - PhotoMaker-V2 is the single commercial-safe end-to-end stack today: Apache-2.0 adapter weights with identity encoded as a fine-tuned OpenCLIP-ViT-H/14 (the model card's exact phrase: "id_encoder includes finetuned OpenCLIP-ViT-H-14 and a few fuse layers"). No InsightFace. Mechanistic argument that an identity embedding cannot transport SynthID: the embedder is trained to be invariant to low-amplitude pixel changes (JPEG, resize, brightness, noise), which is exactly the regime SynthID hides in by design. So the embedding extracted from a watermarked face should be ~identical to the embedding from the cleaned face, and the embedding cannot carry the watermark into a freshly generated face. Flagged explicitly as not-yet-measured -- the first integration step is a cosine-similarity smoke test (no codegen) before investing in a PhotoMaker prototype. Process note: the deep-research harness was run but its verifier subagents failed to call StructuredOutput (same harness bug as a prior session), so its synthesis was unusable; the license claims here are direct quotes from the primary sources, fetched and verified, not from the workflow synthesis. Docs only. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
be14eca207 |
docs: certified controlnet strength floors from the Modal GPU oracle sweep
Ran the isolated raiw-controlnet-cert Modal app (raiw-app/modal_cert.py) over a
strength x seed grid, restore OFF, --max-resolution 1536, each vendor checked on its
OWN oracle (OpenAI -> openai.com/verify, Gemini -> the Gemini app). Certified
controlnet SynthID-removal floors:
- OpenAI 0.20: 2 photoreal images (9-face grid + bracelet) x seed {1,2,3} = 6/6 clean;
the bracelet that flipped at 0.15 is seed-robust at 0.20. Transfers to prod (OpenAI
removal is resolution-independent).
- Gemini 0.30: 0.20 detected -> 0.30 clean on 2/2 seeds (hardest face). Holds only at
<= 1536; Gemini is resolution-sensitive and raiw.cc runs NATIVE, so cap Gemini
<= 1536 + use 0.30, or native-calibrate (~0.35+).
Prod recipe recorded: controlnet + a controlnet-specific per-vendor schedule in
resolve_strength (OpenAI 0.20 / Gemini 0.30, NOT the default 0.10/0.15 ladder) +
FIXED prod seed (kills the near-threshold non-determinism) + restore reworked/off.
Added to docs/controlnet-removal-pipeline-research.md (certified floors table),
docs/synthid.md 5.5, and the CLAUDE.md controlnet bullet. Docs only.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Victor Kuznetsov
|
d38b9a6122 |
docs: correct controlnet/restore SynthID-removal claims from the 2026-06-04 oracle pass
Oracle validation (openai.com/verify + the Gemini app) overturned three claims that were on main, and consolidates the controlnet findings into one authoritative place. - controlnet does NOT reliably remove SynthID at the low vendor-adaptive strength: removal is content x pipeline dependent and the survivors FLIP by content type (photoreal survives controlnet / clears default; flat graphic survives default / clears controlnet; flat text clears both). Root cause is insufficient strength, not the pipeline; controlnet needs a higher, per-vendor floor than default. - removal near the threshold is SEED-non-deterministic (same image+pipeline+strength can pass or fail run-to-run); a single clean run does not certify a strength. - `--restore-faces` RE-INTRODUCES SynthID: GFPGAN runs on the ORIGINAL watermarked face at weight 0.5 and composites it back over the cleaned result (clean A/B: a Gemini face stayed detected through controlnet 0.15/0.20/0.25 WITH restore, cleared at 0.20 with --no-restore-faces). The old "GFPGAN scrubs SynthID" claim was wrong. Corrected in CLAUDE.md (watermark_remover controlnet bullet, controlnet Known-limitations bullet, face_restore bullet, vendor-adaptive strength bullet) and docs/synthid.md (5.1 controlnet/face-identity, 5.2 strength floors, new 5.5 oracle validation log). docs/controlnet-removal-pipeline-research.md gains an authoritative "Oracle validation 2026-06-04" section that the others point to as the single source. Docs only; no code change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
8523f48fb6 |
data(corpus): archive June 2026 SynthID strength-study subjects
Back docs/synthid.md section 2.2 with the actual test set: the per-image oracle-verified subjects were only in a local working dir, while the doc claimed they were recorded in data/synthid_corpus/. Ingest the key pos+cleaned pairs so the claim holds. - pos: openai_1/2/3 originals (gpt-image, openai-verify) + gemini_1/2/3/4 originals (Gemini app, gemini-app); all probe as C2PA-SynthID present. - cleaned: OpenAI at strength 0.05 (openai_2 only s010 captured) + Gemini at 0.15 --max-resolution 1536; oracle: SynthID NOT detected. Metadata stripped, so no C2PA on the cleaned rows. - Excluded the third-party issue #14 image (pic3): oracle-verified but not committed to the public corpus. - docs/synthid.md 2.2: state OpenAI n=4 = 3 archived + 1 external-only. - CLAUDE.md: drop the drift-prone "~65 MB" corpus size from the sdist note. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
5ec8269949 |
chore: mark controlnet pipeline + GFPGAN restore-faces as experimental
Both content-preservation features are now flagged EXPERIMENTAL and opt-in. --pipeline controlnet was already opt-in (default=default); --restore-faces flips from on-by-default to OFF by default, matching the repo's prior pattern for experimental preservation passes (the removed protect_text/protect_faces). - cli.py: --restore-faces/--no-restore-faces default False; EXPERIMENTAL in the --restore-faces / --controlnet-scale / --pipeline help; batch default False. - invisible_engine.py: remove_watermark restore_faces default False + docstring. - CLAUDE.md / README.md / docs/synthid.md: label both experimental/opt-in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
411ef16ec3 |
feat: GFPGAN face-identity restoration post-pass
Add an optional, commercial-safe face-restoration post-pass that recovers face identity the diffusion removal pass drifts (canny holds structure, not likeness) while still scrubbing the pixel watermark in the face regions. - face_restore.py: GFPGANer singleton (CPU unless CUDA), the basicsr torchvision.transforms.functional_tensor shim, and the pure feather _composite_faces helper (unit-tested without the model). GFPGAN re-synthesizes each face from a StyleGAN2 prior, so composited face pixels are GAN-generated (no watermark, no pixel-copy) -- oracle-clean at weight 0.5 with identity preserved. - InvisibleEngine.remove_watermark: restore_faces / restore_faces_weight, best-effort, auto-skips when the extra is absent or no face is detected. - CLI --restore-faces/--no-restore-faces + --restore-faces-weight on invisible/all/batch (on by default). - restore extra (gfpgan/facexlib/basicsr), numpy<2-pinned (scipy<1.18, numba<0.60) and kept out of `all`; basicsr needs Python <3.13 + setuptools<69 to build, so pin .python-version 3.12. Commercial-safe: GFPGAN Apache-2.0, RetinaFace MIT. The CodeFormer alternative is non-commercial and is not shipped. The earlier IP-Adapter FaceID layer was removed (footgun: needs high strength, corrupts faces at the low removal strength). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
d90d5d886a |
feat: controlnet pipeline for text/face-structure preservation
Add `--pipeline controlnet` (SDXL base + xinsir canny ControlNet via StableDiffusionXLControlNetImg2ImgPipeline): the canny edge map conditions the img2img regeneration so text and face STRUCTURE stay sharp, while the watermark is still removed by the regeneration (`strength`) -- no original pixels are copied or frozen, so SynthID does not survive. Oracle-verified clean on OpenAI with better text/structure fidelity than plain img2img at equal strength. `--controlnet-scale` tunes structure preservation; fp32 on mps/cpu (fp16-fixed VAE on cuda/xpu). Shares the img2img runner (live progress + MPS->CPU fallback) and the fp16-VAE-fix / device-move helpers with the default pipeline. Remove the superseded subsystems -- ctrlregen (SD1.5 clean-noise), text-protection (differential / region-hires) and face-protection: they either destroyed real content or shielded the watermark by re-using original pixels. controlnet replaces them by regenerating everything under edge conditioning. Canny preserves face structure but not identity; face IDENTITY is a separate face-restoration post-pass (CodeFormer/GFPGAN), researched + prototyped but not yet shipped. An IP-Adapter FaceID attempt was built and removed (footgun: needs high strength, corrupts faces at removal strength). Docs: docs/controlnet-removal-pipeline-research.md, scripts/controlnet_sweep.py. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
96038f960f |
feat(invisible): vendor-adaptive default strength (OpenAI 0.10 / Google 0.15)
The default img2img strength is now chosen from the detected SynthID vendor (C2PA issuer) instead of a single fixed 0.30: OpenAI gpt-image -> 0.10, Google Gemini -> 0.15, unknown source -> 0.15. Explicit --strength always wins. Basis: an oracle-verified June 2026 controlled study (clean v0.8.6, text/face protection OFF, per-image openai.com/verify or Gemini-app verdict). OpenAI's SynthID clears at 0.05 across 1024-1600 px (n=4, resolution-independent); Google's is ~3x more robust and needs 0.15 on the capped-1536 path (n=4). The dominant factor is the VENDOR, not resolution. The earlier single 0.30 default and the "resolution dependence" lore came from contaminated tests run with the protect-text bug ON (issue #14) -- re-running those same 1600x1600 images clean removes SynthID at 0.05. `vendor_for_strength(path)` reads metadata.synthid_source on the ORIGINAL input and is threaded through cli (invisible/all/batch) -> invisible_engine -> watermark_remover -> resolve_strength(strength, profile, vendor), so display and execution use the same vendor (the engine sees a temp path whose C2PA the visible pass already stripped, so detection must happen in the CLI on the pristine source). Caveat: Google's 0.15 was validated only on --max-resolution 1536; native 2816 Gemini was not locally measurable (OOM on Apple Silicon) and is pending GPU validation on raiw.cc. Docs: docs/synthid.md sections 2.2/4.4/5.2 corrected (the contaminated resolution-dependence findings replaced with the clean oracle-verified table); README and CLAUDE.md updated; CLI --strength help reflects the adaptive default. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
4b0b370ac0 |
fix(invisible): disable protect-text/protect-faces by default; add docs/synthid.md
Both text and face protection were shielding SynthID from removal. The text-protection high-res re-scrub regenerates pixels at an upscaled resolution where the per-region pass may not be strong enough to re-destroy the SynthID payload, allowing it to survive in text areas. Face protection has an even more direct mechanism: it pastes back the original (pre-diffusion, watermarked) face pixels after the global pass, guaranteeing SynthID survives in face regions regardless of strength. Both --protect-text and --protect-faces are now off by default and opt-in. Rename from --no-protect-text / --no-protect-faces to --protect-text / --protect-faces. Extract shared click.option decorators to module-level constants (_protect_text_option, _protect_faces_option) to eliminate copy-paste between cmd_invisible and cmd_all. Add docs/synthid.md: primary-source-cited technical reference for SynthID-Image covering mechanism (post-hoc encoder/decoder, 136-bit payload, pixel-space, no model-weight modification), robustness numbers (arXiv:2510.09263: ~99.98% TPR at 0.1% FPR across 30 transforms), removal attacks and forensic detectability (arXiv:2605.09203: all 6 attacks detectable >98% TPR@1%FPR), detectability limits, oracle scope, adoption landscape, and practical implications including the protect-text/faces SynthID-preservation finding. Verified June 2026 on gpt-image 1600x1600 via openai.com/verify: with --protect-text SynthID detected; without, SynthID removed. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
4b4049a6f1 |
docs(text-protection): update stale strength note (~0.05 -> ~0.30 SynthID threshold)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Victor Kuznetsov
|
58bdf51c59 |
Visible-watermark registry: reverse-alpha-only Doubao + Gemini, exact native recovery (#28)
* fix(trustmark): gate detection on re-encode durability to kill false positives TrustMark's wm_present flag is a BCH validity check that spuriously validates on a content-correlated fraction of un-watermarked images (AI textures trip it more than camera photos). On a 1343-image set all 20 raw detections were false, several on Gemini/OpenAI/Doubao output that cannot carry Adobe's watermark, with random-bytes secrets. A genuine TrustMark is a durable soft binding that survives re-encoding, so detect_trustmark now re-decodes after a mild JPEG round-trip and requires the same schema both times. Every observed false positive collapsed under this gate; the second decode runs only on the rare hit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(identify): Samsung Galaxy AI, FLUX, ByteDance C2PA; fix C2PA substring FP Detection extensions verified on real signed files (2026-05-29): - Samsung Galaxy AI: signer attribution via a new _SIGNER_C2PA_PLATFORM (Samsung Galaxy / ASUS Gallery) kept separate from the capture-camera _DEVICE_C2PA_PLATFORM so a Galaxy AI edit (device cert + AI source type) does not trip the camera-vs-AI integrity clash. Plus metadata.samsung_genai: the proprietary genAIType marker in PhotoEditor_Re_Edit_Data, a medium- confidence AI-editing signal (samsung_only branch). - Black Forest Labs (FLUX) and ByteDance Volcano Engine (Doubao/Jimeng) added as C2PA issuers + issuer->platform mappings. - fix: C2PA presence required only the bare 4-byte 'c2pa' substring, which false-positives on compressed pixel data (a recompressed PNG IDAT re-flagged C2PA after its manifest was correctly stripped). New c2pa_marker_in() requires the JUMBF wrapper (jumb+c2pa) or the C2PA uuid box; applied in identify + metadata. Verified: all 535 real C2PA files carry jumb. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): gate detection on text structure to cut ~95% of false positives (#23) Coverage alone over-fired: any textured bottom-right corner cleared the threshold, so the detector false-positived on ~28% of arbitrary images. The real '豆包AI生成' mark is six glyphs in one row, so detect now also requires the text-structure signature (_glyph_structure): many connected components, no single dominant blob, concentration in a thin horizontal band. False positives dropped 343 -> 17 across the corpus while keeping real-mark recall and the doubao-1.png sample. Also accept a no-op force kwarg for remover-interface symmetry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(samsung): add Samsung Galaxy AI visible-badge remover New samsung_engine.py removes the bottom-left sparkle + localized 'AI-generated content' badge that Galaxy AI tools stamp. Mirrors the Doubao locate->mask->inpaint pattern but bottom-left, with a dual-polarity top-hat mask (the badge is light-on-dark or dark-on-light). Detection gates on a band + left-anchor signature (the Doubao CJK-component gate does not transfer: Latin badge letters connect into few blobs). Explicit-only -- tuned on few real badges with a ~4% FP floor, so it is not used in auto. Synthetic byte-blob fixtures (real badges are user content, not shipped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(visible): unified known-watermark registry + LaMa inpaint backend watermark_registry.py is a single catalog of known visible marks, each tying {usual location, in_auto flag, recovery strategy, detect adapter, remove adapter}: gemini (reverse-alpha, exact), doubao, samsung. cmd_visible is now registry-driven (best_auto_mark for --mark auto; mark_keys() feeds the CLI choices) -- the per-mark _run_doubao/_run_samsung helper branches are gone. Cross-engine confidences are not comparable, so the gemini adapter applies the corpus-validated 0.5 sparkle threshold for auto arbitration (its engine flag is loose and weakly fired ~0.36 on Doubao text, hijacking auto). --backend auto|cv2|lama chooses background reconstruction for the mask-based marks; auto = LaMa when onnxruntime is present, else cv2. For LaMa the mask is the FILLED glyph bounding box (sparse glyph masks leave anti-aliased edges behind). cv2 stays the zero-dependency fallback. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs: watermark registry, Samsung/FLUX/ByteDance detection, LaMa backend, trustmark gate Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(doubao): exact reverse-alpha removal from captured alpha map The Doubao '豆包AI生成' mark is a fixed semi-transparent white overlay, so given its alpha map the original pixels are recovered exactly: original = (wm - a*logo)/(1-a) -- no inpaint hallucination. The alpha map + logo colour were solved from real black+gray Doubao captures on a controlled background: on black captured = a*logo, and the black/gray pair solves a per-pixel without assuming the logo colour (a_max~0.65, logo near-white); the white capture cross-validates (mark vanishes to a flat fill). Bundled as assets/doubao_alpha.png + geometry constants. remove_watermark_reverse_alpha applies it scaled to image width; exact at the captured width, so the registry routes doubao through it only when reverse_alpha_available (width within the calibrated band) and the mark is detected, falling back to mask inpaint (cv2/LaMa) otherwise. A light residual inpaint cleans the sub-pixel rescaling error. Add captures at more resolutions to widen exact coverage. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(visible): reverse-alpha only -- drop inpaint removal + heuristic detection Per the principle that we only remove/detect what we can do exactly, the visible-mark path is now reverse-alpha only: - Doubao detect is reverse-alpha-consistent: match the bundled alpha glyph silhouette against the corner via TM_CCOEFF_NORMED (DETECT_NCC_THRESHOLD 0.4) -- keys on the '豆包AI生成' SHAPE, not coverage/structure heuristics. FP 7/1243 (0.6%). Removes the cv2 inpaint path + the _glyph_structure gate. - Registry is reverse-alpha only: dropped the cv2/LaMa backend (_glyph_remove, _lama_box_inpaint, default_backend, --backend) and the Samsung entry. Doubao outside the alpha resolution band is skipped, never inpainted. - Removed samsung_engine.py + tests + --mark samsung (no alpha map captured; Samsung C2PA/genAIType metadata detection in identify is unaffected). - The universal erase --region (cv2/LaMa) is unchanged -- arbitrary-region inpainting stays a user-directed tool, separate from the known-mark registry. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(doubao): NCC sub-pixel alignment -> reverse-alpha at any resolution A pure width-scale of the captured alpha map is only sub-pixel-accurate at the captured width and leaves a faint ghost elsewhere. remove_watermark_reverse_alpha now registers the alpha glyph to the actual mark via a TM_CCOEFF_NORMED scale+position search (_aligned_alpha_map) before inverting the blend, so the single 2048 capture works at any resolution -- verified clean on the 1773x2364 (3:4) corpus size, the biggest coverage gap (23 files). reverse_alpha_available is now just 'asset present' (no width band); the registry still gates removal on detect so a clean corner is never touched. Drops the _ALPHA_WIDTH_TOLERANCE gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): keep native recovery exact -- fixed geometry at captured width Integer-pixel NCC alignment landed ~1px off at the captured width, degrading the otherwise-exact native reverse-alpha (synthetic recovery error 0.94 -> 1.39). remove_watermark_reverse_alpha now uses exact width-relative geometry within _ALPHA_NATIVE_BAND of the captured width and the NCC search only off it -- best of both: native back to 0.94, other resolutions still aligned. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(doubao): harden alignment -- try fixed+aligned, keep least residual (56/56) On a faint/busy-background mark the NCC alignment peak can wander a few px off the true mark and leave a residual (2/56 real corpus files). Off the captured width, remove_watermark_reverse_alpha now builds BOTH the fixed-geometry and the NCC-aligned alpha map, applies each, and keeps whichever leaves the least residual mark (re-detect confidence on the bare reverse-alpha) -- geometry wins on faint marks, alignment on clear ones, no magic threshold. Real-file round-trip now removes 56/56 detected Doubao clean across every corpus resolution (was 54). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * perf(doubao): skip residual inpaint at native width for exact recovery At the captured width the fixed-geometry reverse-alpha is pixel-exact, so inpainting over it only replaced exactly-recovered interior pixels with a cv2 hallucination -- measured worse on a textured background (native error vs true bg 1.6 reverse-alpha-only vs 2.6 with the old always-on full-footprint inpaint). Native now returns the bare recovery untouched; off-native, where NCC alignment is only sub-pixel-approximate, the footprint inpaint stays to clean the seam. Real round-trip still 56/56 across all corpus resolutions; negatives 0/60, Gemini unaffected. Add test_native_returns_exact_reverse_alpha_no_inpaint as the regression guard. Sync CLAUDE.md + README (the table cell and prose described the pre-NCC "skipped off native / cv2-LaMa" behavior, now stale). Gitignore the session scheduled_tasks.lock, and add the text-protection research note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |