The invisible/SynthID diffusion pass loads the whole SDXL fp16 pipeline
into VRAM via `pipeline.to("cuda")`. On an 8 GB card the weights alone
(~7 GB) leave no room for activations, so the run OOMs and there is no
in-tool way to recover short of falling back to CPU (~9 min/image).
Add an opt-in `--cpu-offload` flag (default off) that calls diffusers'
`enable_model_cpu_offload()` instead: submodules are streamed to the GPU
on demand, dropping peak VRAM to roughly the largest single submodule at
the cost of per-step transfers. CUDA-only; a no-op on cpu/mps. Threaded
through `invisible`, `all`, and `batch` to keep the knob set identical
across the three, mirroring the existing `--device`/`--pipeline` options.
Measured on a GTX 1070 Ti (8 GB): `invisible --pipeline sdxl --cpu-offload`
runs the SynthID scrub on-GPU in ~2.5 min vs ~9 min on CPU, where the
default full-VRAM path OOMs.
Test drives the placement decision with a mock pipeline (no model/GPU),
gated on torch so it runs under the `gpu` extra and skips the core CI
matrix, consistent with the model-running test policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolve 10 code-review findings on the v0.14.0 localize->fill path, several
release-blocking:
- gemini: build the removal mask from the decision's provenance-aware region
instead of a strict internal re-detect. A relaxed/assume_ai sparkle was
re-demoted by the FP gate into a None mask and reported removed while left in
the image; this also drops the redundant double-detect.
- registry: report a mark removed only when a fill actually happened (remove()
returns a None region for an empty mask), so a no-op is never claimed.
- api/cli: add write_noop so the CLI `visible` no-mark path writes nothing and
cannot clobber a pre-existing -o file (was write-then-unlink -> data loss);
create output.parent; skip the same-file copy (SameFileError on in-place).
- cli: catch the missing migan/lama backend RuntimeError on the visible/all
paths (matches `erase`); route the single-mark relaxation through the shared
resolve_relax instead of an inline copy.
- metadata: keep_standard=False no longer takes the AI-only lossless JPEG
short-circuit (it left standard metadata); defer a malformed-marker JPEG to
the PIL fallback instead of reporting a partial strip as complete.
- invisible: register the HEIF opener before Image.open (HEIC --force) and
RGB-convert before the PNG temp (CMYK JPEG).
- pill: normalize via to_bgr so a 4-channel BGRA array cannot crash cvtColor.
Regression tests for each; docs synced (resolve_relax, write_noop,
best_auto_mark -> detect_marks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace reverse-alpha removal with localize -> fill (template-free mask + one
shared cv2/MI-GAN/big-LaMa fill) for every mark; drops the colour-shift / dark-pit
failure modes, version-robust to a moved or re-rendered mark
- Separate perception/decision/action: engines report Candidates, a pure
decide(candidates, Context) arbiter owns all policy (sensitivity + provenance +
pill gate), remove_auto_marks orchestrates -- behavior-preserving (corpus 46/46/92)
- Three orthogonal knobs replace --method: --backend cv2|migan|lama,
--sensitivity auto|strict|assume-ai, provenance (auto from metadata)
- Add high-level api.remove_visible / visible_provenance (lazy top-level re-export);
visible --mark auto delegates to it so CLI and library share ONE path
- Read+write HEIC/AVIF on the pixel path via pillow-heif; imwrite preserves the input
format at max quality (JPEG q100/4:4:4); a no-op copies the original bytes verbatim
- Lossless byte-level JPEG metadata strip (no DCT re-encode); consolidate the two
remove_ai_metadata into one, delete legacy noai/cleaner + best_auto_mark
- Bump 0.13.0 -> 0.14.0
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A third diffusion pipeline alongside sdxl/controlnet: Qwen-Image (20B MMDiT,
Apache-2.0 code AND weights) img2img. The scrub still comes from the img2img
strength; Qwen preserves text (incl. CJK) and structure markedly better than
SDXL at the scrub floor, so it over-regenerates real photos far less (directly
targets the controlnet over-regeneration that degrades real uploads).
- watermark_profiles: QWEN_MODEL_ID, normalize_profile accepts "qwen".
- WatermarkRemover: _load_qwen_pipeline (bf16, loads Qwen base unless --model
overridden, clear ImportError if diffusers lacks the class), _run_qwen (no
MPS fallback -- 20B is CUDA/cloud-class), dispatch in _generate_one/preload,
pure _build_qwen_kwargs (true_cfg_scale, not guidance_scale).
- Shared _base_load_kwargs() across all three loaders (dtype + token).
- CLI --pipeline gains "qwen"; invisible_engine threads it through.
- scripts/qwen_scrub_prototype.py: standalone PEP 723 GPU experiment.
Prototype oracle floors (Modal A100-80GB, single seed, controls SynthID-positive,
PENDING seed-repeat cert): OpenAI clears at strength ~0.10, Gemini at ~0.30 (0.20
still detected), with CJK text + faces faithful where controlnet plasticizes. The
Gemini floor is higher than the shared default ladder, so pass an explicit
--strength for Gemini on this pipeline until a Qwen-specific ladder is certified.
The model-running path is CUDA-only (untestable locally); unit tests cover the
pure call-shape (_build_qwen_kwargs) and profile normalization without torch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a lossless alternative to the --max-resolution downscale for large
images that OOM on MPS/GPU: regenerate in overlapping, feather-blended
tiles at native resolution.
- noai/tiling.py: pure plan_tiles (uniform tiles, last flush to edge) +
feather_weights (strictly-positive separable taper -> partition-of-unity
blend) + run_tiled (per-tile generate callable, decoupled from the
pipeline). Unit-tested without the model.
- WatermarkRemover.remove_watermark: refactor _generate into _generate_one
+ a tiled branch that engages only when --tile is set and the long side
exceeds tile_size (ControlNet canny is rebuilt per tile).
- Thread tile/tile_size/tile_overlap through InvisibleEngine and the
invisible/all/batch CLI commands via a shared _tile_options decorator.
Verified end-to-end on the real SDXL pipeline (forced 2x2 tiling on a
1024px sample, MPS): non-degenerate output, no gross seam at tile borders.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nine findings from a high-effort project-wide review, fixed and verified
(571 passed, ruff/pyright clean):
Correctness:
- all/batch now remove Doubao/Jimeng/Samsung visible text marks: the visible
step routes through the registry (new cli._remove_visible_auto) instead of a
hardcoded GeminiEngine, so they no longer leave the wordmark intact.
- batch always reads the original source (dropped the out_path-reuse that
re-processed already-cleaned outputs on a re-run).
- img2img_runner only retries the diffusion call on the deprecated-callback
TypeError; any other TypeError now propagates instead of double-running.
- gemini detect/remove and the reverse-alpha engines normalize channels via a
new image_io.to_bgr, fixing a grayscale/BGRA crash in the FP-gate path.
- _png_late_metadata advances its cursor by the clamped length, so a malformed
chunk length no longer aborts the late AI-label scan.
Cleanup / efficiency:
- Consolidate the ~90%-identical Doubao/Jimeng/Samsung engines into a shared
config-driven _text_mark_engine.TextMarkEngine base; each engine is now a thin
subclass (TextMarkConfig + test shims). Behavior is byte-exact (the three
engine test suites pass unchanged). Registry adapters collapse to one
_text_mark(...) row each. Gemini stays a separate engine.
- scan_head is memoized per (path, size, mtime), so identify() reads the file
head once instead of ~8 times.
- invisible_engine post-processing decodes/encodes the output once (chained in
memory) instead of 2-4 times across stages.
- Remove the orphaned get_model_id_for_profile (+ CONTROLNET_PROFILE); derive
the --strength help from the strength constants (strength_default_help) so it
cannot drift; share the --pipeline/--strength click options; simplify the
retired --auto resolver.
Net -835 lines. Tests added for the registry-routed visible pass, to_bgr,
the polish/model/guidance wiring, and strength_default_help. CLAUDE.md updated
for the new base module, the engine/registry changes, image_io.to_bgr, and the
scan_head cache.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Overhaul the diffusion-removal surface around a single robust default and a
complete, consistent CLI.
Pipeline + strength:
- controlnet is now the DEFAULT pipeline (CLI --pipeline + both engine ctors).
With the certified higher strength it clears both photoreal and flat-graphic
content, whereas plain SDXL left SynthID on flat graphics.
- Rename the plain-SDXL profile default -> sdxl; "default" stays as a back-compat
alias (normalize_profile + a click callback that warns).
- Unify the strength ladder: resolve_strength applies ONE vendor-adaptive ladder
(the certified controlnet floors OpenAI 0.20 / Google 0.30 / unknown 0.30) to
both pipelines. sdxl is the weaker remover on its own hard case (flat fills),
so the certified floor is the right floor for it too.
CLI completeness:
- Add --model (HF model id) to invisible + batch (was only on all) and
--guidance-scale (CFG) to all three diffusion commands; both were library
knobs the CLI did not expose.
- Flip --adaptive-polish to ON by default (it self-gates to a no-op where there
is no detail deficit, so default-on is safe).
- Share --pipeline / --strength / --model / --guidance-scale as single
decorators so invisible/all/batch keep an identical surface; the --strength
help is derived from the strength constants (strength_default_help) so it can
never drift from the ladder.
Removals:
- Delete the auto_config content-detection planner + its YuNet/DBNet assets
(~2.6 MB): with controlnet always the pipeline and the polish self-gating, the
face/text/edge detection no longer changed behavior. --auto is now a deprecated
no-op that only warns (the polish it enabled is the default).
Docs (README, CLAUDE.md, docs/synthid.md) updated throughout; added an
InvisibleEngine Python API example. Tests cover the alias warnings, the
polish default, and the --model/--guidance-scale wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Empirical conclusion from the 2026-06-04 - 2026-06-08 Modal cert sweeps:
every face-restore approach we built (GFPGAN-on-cleaned, PhotoMaker-V2,
InstantID txt2img, InstantID img2img-on-cleaned at three parameter
settings) regenerates the face via SDXL diffusion rather than preserves
it. Output face pixels are diffusion-fresh, so the regenerated face
inherits SDXL "clean skin" aesthetic and loses original identity
precision -- it looks MORE AI-generated than the cleaned image, not
less. The cleaned image from the main controlnet 0.20 removal pass is
the least-AI face state we can reach without re-introducing SynthID.
Nothing in the restore family achieves the actual goal (preserve the
original person's face). Keeping them around as opt-in invites users to
ship something that defeats the point. Removing entirely.
Library changes:
- Deleted src/remove_ai_watermarks/instantid_restore.py
- Deleted src/remove_ai_watermarks/photomaker_restore.py
- Deleted tests/test_instantid_restore.py
- Deleted tests/test_photomaker_restore.py
- Removed `instantid` and `photomaker` extras from pyproject.toml
- Removed `[tool.hatch.metadata] allow-direct-references = true` (was
only needed for the photomaker git+ URL)
- InvisibleEngine.remove_watermark: dropped `restore_faces` +
`restore_faces_method` params, removed both `_restore_faces_instantid`
and `_restore_faces_photomaker` private methods, removed dispatch
- CLI: dropped `_restore_faces_options` decorator, all four cmd_*
signatures lose `restore_faces` + `restore_faces_method`, kwarg passes
to remove_watermark dropped
- _apply_auto: dropped `restore_faces` from tuple shape (was unused after
the engine no longer takes it)
- auto_config.AutoConfig: dropped `restore_faces` field; `plan()` no
longer sets it; `reason` no longer mentions it
- Tests updated accordingly (test_auto_config.TestReason no longer asserts
"face-restore on" in the reason string)
Docs updated:
- CLAUDE.md: removed the photomaker extras bullet, the Face restore
trade-off bullet, the instantid_restore.py + photomaker_restore.py
module bullets; replaced restore mentions in watermark_remover and
controlnet bullets and prod recipe with the empirical conclusion
- README.md: removed both `--restore-faces` callouts and the install
snippet; the feature bullet and auto-mode comment updated
- docs/synthid-robust-identity-research.md: added Status-retired notice
at the top pointing at the 2026-06-08 followup
raiw-app:
- modal_cert.py: dropped `--restore-faces` flag entirely; sweep() no
longer takes restore_faces; pinned _LIB_SPEC to `[gpu]` extras (no
`photomaker` / `instantid` extras), points at main
ruff + strict pyright clean; 569 tests pass; 18 restore-specific tests
gone.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The InstantID cert sweep emitted `restore_faces post-pass failed ()` -- the
exception's str() was empty so the log line told us nothing about what
actually failed. Adding `exc_info=True` plus `type(e).__name__` so the
full traceback and exception class land in the log even when the message
is empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Per the 2026-06-08 deep-research synthesis (docs/synthid-robust-identity-
research-2026-06-08.md), the entire ArcFace-class identity-adapter ecosystem
for SDXL is blocked from commercial use by InsightFace's non-commercial model
packs (antelopev2 / buffalo_l). No commercial-safe ArcFace-grade identity
stack exists today. The user explicitly opted into shipping a non-commercial
restore path (research / personal use; raiw.cc must NOT install the extra).
Architectural choice: InstantID over PhotoMaker-V2 as the default.
- PhotoMaker-V2 (CLIP+ArcFace dual encoder, txt2img only): documented upstream
identity drift on Asian male faces, visually confirmed in our cert sweep
(tatsunari rendered as a generic woman; group photo collapsed into a
patchwork).
- InstantID (ArcFace cross-attention + landmark ControlNet): semantic
identity branch + spatial weak landmark control, decoupled. Per InstantID
paper (arXiv:2401.07519) and the research report, stronger identity fidelity
on single portraits. Critically: NO original face pixels enter the diffusion
(ArcFace embedding is semantic, landmark stick figure is pure geometry), so
SynthID is not transported.
Implementation:
- New `src/remove_ai_watermarks/instantid_restore.py` mirrors the
`photomaker_restore.py` shape (lazy singletons for pipeline + FaceAnalysis,
per-face crop + _composite_faces from photomaker_restore). Loads the
InstantID community pipeline via `DiffusionPipeline.from_pretrained(
custom_pipeline="pipeline_stable_diffusion_xl_instantid")` -- no upstream
Python package needed; diffusers fetches the file from its community
examples.
- New `instantid` extra in pyproject (insightface + onnxruntime +
huggingface-hub). NON-COMMERCIAL block in the comment explains why.
- CLI: `--restore-faces-method [instantid|photomaker]`, default `instantid`.
Both methods explicitly labeled NON-COMMERCIAL in the help text.
- Engine: dispatch on `restore_faces_method` to either
`_restore_faces_instantid` or `_restore_faces_photomaker`.
- 9 control-flow tests for InstantID without model download (mirror the
photomaker_restore.py test pattern + draw_kps helper checks). 587/587 pass.
Diffusers-0.38 compat verified by upstream code inspection: the InstantID
pipeline inherits from `StableDiffusionXLControlNetPipeline`, uses only
public diffusers APIs (`encode_prompt`, `prepare_image`, `prepare_latents`,
`get_guidance_scale_embedding`), uses legacy attention processor API which
diffusers preserves for backward compat. No PhotoMaker-V1-style internal
text_encoder access. End-to-end execution will be validated by the Modal
cert sweep in the next step.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Visual review of the GFPGAN-on-cleaned output (9-face grid, 1448x1086) showed it
only polished the already-drifted face without restoring identity — useless for the
"restore who is in the photo" intent. Dropping it.
The shipped restore path is now PhotoMaker-V2, which delivers true identity-from-
embedding face regeneration via a CLIP+ArcFace dual encoder. The ArcFace branch
pulls InsightFace antelopev2/buffalo_l model packs at runtime, which InsightFace
releases under a research-only license, so the whole extra is **NON-COMMERCIAL**.
raiw.cc and any monetized deployment must NOT install the `photomaker` extra.
This is called out at every entry point: CLI flag help, module docstring,
pyproject extra block, CLAUDE.md extras bullet, README install snippet.
Changes:
- Deleted `src/remove_ai_watermarks/face_restore.py` and its tests.
- Deleted the `restore` extra (gfpgan/facexlib/basicsr + scipy<1.18 / numba<0.60
pins) and the basicsr setuptools<69 build pin from pyproject.toml.
- Restored `src/remove_ai_watermarks/photomaker_restore.py` (V2 this time:
`TencentARC/PhotoMaker-V2`, `photomaker-v2.bin`, no `pm_version='v1'` override).
- Restored the `photomaker` extra in pyproject with all the upstream-compat
pins (einops, peft, onnxruntime, insightface) and the `allow-direct-references`
hatch metadata block.
- `InvisibleEngine` swapped `_restore_faces` -> `_restore_faces_photomaker`;
`--restore-faces-method` removed (only one method, no choice).
- CLI flag help, CLAUDE.md, README, docs/synthid.md, and
docs/controlnet-removal-pipeline-research.md all updated.
- docs/synthid-robust-identity-research.md status notice rewritten to list both
abandoned commercial-safe attempts (V1 + GFPGAN-on-cleaned) and the
non-commercial trade-off we accepted.
ruff + strict pyright(src/) clean; 578 tests pass (the 9 GFPGAN tests are gone,
the 11 PhotoMaker tests stay green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After 7 cascading upstream-compat fixes (insightface dep, peft dep, pm_version,
device, etc.), the PhotoMaker V1 cert sweep still hit a CFG batch-dim mismatch
inside the denoising loop. The upstream PhotoMaker `pipeline.py` is forked from
diffusers v0.29.1 and our env runs 0.38; SDXL prompt-encoder handling changed
significantly between those versions, so making PhotoMaker work end-to-end
needs a proper fork or a diffusers downgrade — both expensive. Not worth
shipping today.
Pivot: restore `face_restore.py` (GFPGAN) with a single-line fix that makes it
SynthID-safe by construction. The previous design ran GFPGAN.enhance on the
ORIGINAL watermarked image and was oracle-confirmed to re-add SynthID via the
weight-0.5 pixel blend. The fix is to run GFPGAN on the diffusion-CLEANED
image — whatever pixels GFPGAN derives from are already SynthID-free, so the
partial blend cannot transport the watermark. Identity fidelity is lower than
a true identity-as-embedding stack would deliver, but it ships and works.
Changes:
- `src/remove_ai_watermarks/face_restore.py` restored from pre-wipe state with
one line changed: `restorer.enhance(cleaned_bgr, ...)` instead of
`restorer.enhance(original_bgr, ...)`. `original_bgr` is kept as an unused
positional argument for API stability.
- `src/remove_ai_watermarks/photomaker_restore.py` and its tests REMOVED. The
research note (`docs/synthid-robust-identity-research.md`) keeps a "status
notice" documenting why PhotoMaker is parked for now and what the path back
in would look like.
- `pyproject.toml` `restore` extra restored (gfpgan/facexlib/basicsr +
scipy<1.18 + numba<0.60 pins + the basicsr setuptools<69 build pin), plus
`photomaker` extra (with its einops/insightface/peft pile) and the
`[tool.hatch.metadata] allow-direct-references = true` block REMOVED.
- `InvisibleEngine._restore_faces_photomaker` removed; `_restore_faces`
restored. The `--restore-faces` CLI flag and its plumbing through cmd_*
signatures are unchanged.
- CLAUDE.md, README.md, docs/synthid.md, docs/controlnet-removal-pipeline-
research.md updated to describe the shipped GFPGAN-on-cleaned design and to
reference PhotoMaker only as the parked alternative.
ruff + strict pyright(src/) clean; 578 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The GFPGAN `restore` extra and its `face_restore.py` module are gone. They were
oracle-confirmed to re-introduce SynthID by blending watermarked original face
pixels at fidelity weight 0.5 (clean A/B: gemini_3 controlnet 0.20 detected WITH
GFPGAN, clean WITHOUT). Keeping them as the default restore method was a footgun
for the removal pipeline. PhotoMaker-V2 (added in the previous commit) is the
single shipped restore path now -- identity-as-embedding, SynthID-safe by
construction.
Removed:
- src/remove_ai_watermarks/face_restore.py + tests/test_face_restore.py
- pyproject.toml `restore` extra (gfpgan/facexlib/basicsr + scipy/numba pins)
- pyproject.toml `[tool.uv.extra-build-dependencies] basicsr = [...]` build pin
- CLI: `--restore-faces-method` and `--restore-faces-weight` (no method choice
to make, no GFPGAN weight knob to expose)
- InvisibleEngine._restore_faces method (only _restore_faces_photomaker remains)
- All restore-faces-method / restore-faces-weight threading through cmd_*
signatures and _process_batch_image
Kept:
- `--restore-faces / --no-restore-faces`: now binds to PhotoMaker-V2.
- All adopted oracle findings about GFPGAN re-introducing SynthID (kept in the
research docs as historical context that explains why the path was removed).
Docs updated: CLAUDE.md (restore extras bullet collapsed to photomaker, removed
face_restore Key-modules bullet, several inline GFPGAN refs scrubbed), README.md
(face-identity callout + install section now point to the photomaker extra),
docs/synthid.md 5.5 (net recipe), docs/controlnet-removal-pipeline-research.md
(recommendations).
ruff + strict pyright (src/) clean; 578 tests pass (the 9 GFPGAN tests are gone,
the 9 PhotoMaker tests stay green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the second face-restore mechanism, selectable via the new CLI option
`--restore-faces-method=photomaker`. Unlike the existing GFPGAN path (which runs on
the watermarked ORIGINAL and was oracle-confirmed to re-introduce SynthID by partial
pixel blending), PhotoMaker carries identity in a SynthID-invariant OpenCLIP
embedding and regenerates fresh face pixels conditioned on it — the pixels in the
output are diffusion-fresh, so the watermark cannot be transported.
The load-bearing assumption (embedding invariance to SynthID-magnitude pixel noise)
was empirically validated in the prior commit (smoke test): cosine drift 0.002
under a ±2 LSB low-freq carrier, an order of magnitude less than JPEG90 drift
which SynthID survives at >=99% TPR.
End-to-end commercial-safe:
- PhotoMaker-V2 weights: Apache-2.0 (TencentARC)
- ID encoder: OpenCLIP-ViT-H/14 (MIT)
- SDXL base: shared with the main pipeline
- NO InsightFace (the non-commercial blocker for IP-Adapter FaceID / InstantID /
PuLID / Arc2Face)
Two-pass architecture (PhotoMaker has no ControlNetImg2img class in diffusers):
1) main controlnet/default removal pass cleans SynthID + drifts faces
2) PhotoMaker txt2img regenerates each face from its embedding, feather-composited
back into the cleaned image
New module `photomaker_restore.py` mirrors `face_restore.py`: lazy pipeline
singleton (double-checked lock), `is_available()` gate, pure `_face_crop_square` and
`_composite_faces` helpers, all unit-tested without the model (9 new tests). New
`InvisibleEngine._restore_faces_photomaker` runs after the diffusion pass, mirroring
`_restore_faces`. CLI flag `--restore-faces-method=[gfpgan|photomaker]` threaded
through `cmd_invisible`/`cmd_all`/`cmd_batch` + `_process_batch_image`.
New optional `photomaker` extra (Apache-2.0 + Apache-2.0/MIT deps, no basicsr).
`[tool.hatch.metadata] allow-direct-references = true` is required because the
upstream PhotoMaker package lives only on GitHub.
The next step (separate work) is oracle validation: run a 6-image cert sweep
through the new pipeline (default/controlnet at the certified strength +
--restore-faces-method=photomaker) and confirm SynthID stays clean while face
identity is recovered. The required infrastructure (`raiw-app/modal_cert.py`) is
already in place.
ruff + strict pyright(src/) clean; 586 tests pass (+ 9 new in
tests/test_photomaker_restore.py).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three content-quality features for the invisible/all/batch pipeline.
DBNet text detector (auto_config): replace the MSER text heuristic with
PP-OCRv3 differentiable-binarization via cv2.dnn.TextDetectionModel_DB,
using a bundled 2.4 MB Apache-2.0 model (en/cn detection nets are
byte-identical, so it ships language-neutral). cv2.dnn is core OpenCV, so
no new pip dep. MSER stays as the fallback when the model can't load.
Validated on real images: matches MSER everywhere and additionally catches
the Doubao CJK mark MSER missed; routing decisions unchanged otherwise.
Real-ESRGAN upscaler (new upscaler.py, esrgan extra): optional
pre-diffusion super-resolution for the min-resolution floor upscale, loaded
via spandrel (MIT, no basicsr) with BSD-3-Clause weights downloaded on
first use. New --upscaler {lanczos,esrgan} on invisible/all/batch; default
stays lanczos and the engine falls back to lanczos when the extra is absent
or the model errors (never breaks removal). It is a manual opt-in knob (the
auto plan never selects it) -- as a generic GAN it sharpens photo/texture
content strongly but can degrade faces (the diffusion pass regenerates
them) and thin text, documented accordingly.
batch --auto: wire the content-adaptive --auto (+ --adaptive-polish) into
cmd_batch. The plan is recomputed per image and the invisible engine is
cached per resolved pipeline (default/controlnet), so a mixed directory
builds at most one engine of each kind. Verified end-to-end: 3 mixed
images routed correctly with only 2 pipeline loads (controlnet reused).
ruff + strict pyright(src/) clean; 558 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The fixed mild auto polish (unsharp 0.5 / grain 2.0) under-corrected soft
photo/face output (gemini_3 stayed at lap-var 84 vs its 592 original) and its
grain speckled small text. Replace it with humanizer.adaptive_polish: target the
input's Laplacian variance with a capped unsharp scaled to the deficit + edge-
masked grain (smooth regions only), calibrated by a short sigma search. Self-
limiting on text/graphics -- already high-frequency, so almost no polish lands
and text edges are masked out. Validated on the spaces corpus (gemini_3 84 -> 334
end-to-end; openai_1 text near-untouched).
Interface: every --auto decision is now independently overridable -- add
--adaptive-polish/--no-adaptive-polish (matching --restore-faces; works without
--auto too) so the polish can be disabled or used manually. _apply_auto overrides
exactly the three content-adaptive modes (pipeline, restore-faces, adaptive-
polish); --unsharp/--humanize stay independent fixed filters.
cv2-only, no new deps. Threaded through invisible/all (not batch).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two quality knobs for the SDXL invisible pass:
- min_resolution floor (default 1024, --min-resolution): small inputs are
upscaled to a 1024px long-side floor before diffusion, since SDXL img2img
distorts on a tiny latent (a 381x512 portrait wrecks at native). The output
is restored to the original input size, so it is a transparent quality boost;
it adds time/memory on small inputs. 0 disables. Extends the pure _target_size
helper (now cap-or-floor-or-native, min skipped on a min>max misconfig),
unit-tested without a model.
- unsharp post-filter (humanizer.unsharp_mask, --unsharp, opt-in default 0):
applied LAST, after the GFPGAN face pass (a pre-GFPGAN sharpen would be
smoothed back over), to counter the soft/over-smoothed look that diffusion +
restoration leave behind (an AI tell). Pairs with --humanize (grain).
Both threaded through invisible/all/batch + the module-level helper. Verified
end-to-end on a 381x512 portrait: upscaled to 1024, sharpened, restored to
381x512.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both content-preservation features are now flagged EXPERIMENTAL and opt-in.
--pipeline controlnet was already opt-in (default=default); --restore-faces
flips from on-by-default to OFF by default, matching the repo's prior pattern
for experimental preservation passes (the removed protect_text/protect_faces).
- cli.py: --restore-faces/--no-restore-faces default False; EXPERIMENTAL in the
--restore-faces / --controlnet-scale / --pipeline help; batch default False.
- invisible_engine.py: remove_watermark restore_faces default False + docstring.
- CLAUDE.md / README.md / docs/synthid.md: label both experimental/opt-in.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add an optional, commercial-safe face-restoration post-pass that recovers
face identity the diffusion removal pass drifts (canny holds structure, not
likeness) while still scrubbing the pixel watermark in the face regions.
- face_restore.py: GFPGANer singleton (CPU unless CUDA), the basicsr
torchvision.transforms.functional_tensor shim, and the pure feather
_composite_faces helper (unit-tested without the model). GFPGAN
re-synthesizes each face from a StyleGAN2 prior, so composited face pixels
are GAN-generated (no watermark, no pixel-copy) -- oracle-clean at weight 0.5
with identity preserved.
- InvisibleEngine.remove_watermark: restore_faces / restore_faces_weight,
best-effort, auto-skips when the extra is absent or no face is detected.
- CLI --restore-faces/--no-restore-faces + --restore-faces-weight on
invisible/all/batch (on by default).
- restore extra (gfpgan/facexlib/basicsr), numpy<2-pinned (scipy<1.18,
numba<0.60) and kept out of `all`; basicsr needs Python <3.13 + setuptools<69
to build, so pin .python-version 3.12.
Commercial-safe: GFPGAN Apache-2.0, RetinaFace MIT. The CodeFormer alternative
is non-commercial and is not shipped. The earlier IP-Adapter FaceID layer was
removed (footgun: needs high strength, corrupts faces at the low removal
strength).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `--pipeline controlnet` (SDXL base + xinsir canny ControlNet via
StableDiffusionXLControlNetImg2ImgPipeline): the canny edge map conditions the
img2img regeneration so text and face STRUCTURE stay sharp, while the watermark
is still removed by the regeneration (`strength`) -- no original pixels are
copied or frozen, so SynthID does not survive. Oracle-verified clean on OpenAI
with better text/structure fidelity than plain img2img at equal strength.
`--controlnet-scale` tunes structure preservation; fp32 on mps/cpu (fp16-fixed
VAE on cuda/xpu). Shares the img2img runner (live progress + MPS->CPU fallback)
and the fp16-VAE-fix / device-move helpers with the default pipeline.
Remove the superseded subsystems -- ctrlregen (SD1.5 clean-noise),
text-protection (differential / region-hires) and face-protection: they either
destroyed real content or shielded the watermark by re-using original pixels.
controlnet replaces them by regenerating everything under edge conditioning.
Canny preserves face structure but not identity; face IDENTITY is a separate
face-restoration post-pass (CodeFormer/GFPGAN), researched + prototyped but not
yet shipped. An IP-Adapter FaceID attempt was built and removed (footgun: needs
high strength, corrupts faces at removal strength).
Docs: docs/controlnet-removal-pipeline-research.md, scripts/controlnet_sweep.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The default img2img strength is now chosen from the detected SynthID vendor
(C2PA issuer) instead of a single fixed 0.30: OpenAI gpt-image -> 0.10, Google
Gemini -> 0.15, unknown source -> 0.15. Explicit --strength always wins.
Basis: an oracle-verified June 2026 controlled study (clean v0.8.6, text/face
protection OFF, per-image openai.com/verify or Gemini-app verdict). OpenAI's
SynthID clears at 0.05 across 1024-1600 px (n=4, resolution-independent);
Google's is ~3x more robust and needs 0.15 on the capped-1536 path (n=4). The
dominant factor is the VENDOR, not resolution. The earlier single 0.30 default
and the "resolution dependence" lore came from contaminated tests run with the
protect-text bug ON (issue #14) -- re-running those same 1600x1600 images clean
removes SynthID at 0.05.
`vendor_for_strength(path)` reads metadata.synthid_source on the ORIGINAL input
and is threaded through cli (invisible/all/batch) -> invisible_engine ->
watermark_remover -> resolve_strength(strength, profile, vendor), so display and
execution use the same vendor (the engine sees a temp path whose C2PA the visible
pass already stripped, so detection must happen in the CLI on the pristine
source). Caveat: Google's 0.15 was validated only on --max-resolution 1536;
native 2816 Gemini was not locally measurable (OOM on Apple Silicon) and is
pending GPU validation on raiw.cc.
Docs: docs/synthid.md sections 2.2/4.4/5.2 corrected (the contaminated
resolution-dependence findings replaced with the clean oracle-verified table);
README and CLAUDE.md updated; CLI --strength help reflects the adaptive default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Both text and face protection were shielding SynthID from removal. The
text-protection high-res re-scrub regenerates pixels at an upscaled resolution
where the per-region pass may not be strong enough to re-destroy the SynthID
payload, allowing it to survive in text areas. Face protection has an even more
direct mechanism: it pastes back the original (pre-diffusion, watermarked) face
pixels after the global pass, guaranteeing SynthID survives in face regions
regardless of strength.
Both --protect-text and --protect-faces are now off by default and opt-in.
Rename from --no-protect-text / --no-protect-faces to --protect-text /
--protect-faces. Extract shared click.option decorators to module-level
constants (_protect_text_option, _protect_faces_option) to eliminate
copy-paste between cmd_invisible and cmd_all.
Add docs/synthid.md: primary-source-cited technical reference for SynthID-Image
covering mechanism (post-hoc encoder/decoder, 136-bit payload, pixel-space, no
model-weight modification), robustness numbers (arXiv:2510.09263: ~99.98% TPR
at 0.1% FPR across 30 transforms), removal attacks and forensic detectability
(arXiv:2605.09203: all 6 attacks detectable >98% TPR@1%FPR), detectability
limits, oracle scope, adoption landscape, and practical implications including
the protect-text/faces SynthID-preservation finding.
Verified June 2026 on gpt-image 1600x1600 via openai.com/verify: with
--protect-text SynthID detected; without, SynthID removed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An oracle-verified GPU strength study (Modal A100, native res, Gemini-app
'Verify with SynthID', n=3 fresh Gemini images, protect_text/faces off) found the
current Google SynthID survives strength 0.10/0.15/0.2 and is removed only at 0.3.
The previous 0.10 default (set from an n=1 result) no longer clears it -- Google
hardened SynthID and the threshold has climbed 0.05 -> 0.10 -> ~0.3. Bump
DEFAULT_STRENGTH to 0.30; OpenAI/ChatGPT carry C2PA not SynthID, so 0.10 is plenty
there (pass --strength 0.10). Note protect_text shields the text regions SynthID
hides in (use --no-protect-text for full removal on text-heavy images).
The same study found ctrlregen at clean-noise strength DESTROYS real images
(hallucinated micro-text in smooth regions), with no usable middle setting, so the
literature's 'clean-noise is the lever' did not hold empirically. Flag ctrlregen
EXPERIMENTAL in the CLI --pipeline help, README, and watermark_profiles; SDXL
img2img at ~0.3 stays the shippable path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ctrlregen profile inherited the SDXL img2img --strength default (0.10), a
near-identity pass that loaded ControlNet + DINOv2-giant and barely changed the
image -- a no-op for removal. resolve_strength() now resolves an unset strength
per profile: 0.10 for the SDXL default, CTRLREGEN_DEFAULT_STRENGTH (1.0,
clean-noise) for ctrlregen. It checks `is None` rather than falsiness, so an
explicit 0.0 is respected (the old `strength or DEFAULT` swallowed it).
Research basis: CtrlRegen (ICLR 2025, arXiv:2410.05470) removes robust
watermarks by regenerating from clean Gaussian noise; partial-noise img2img
retains watermark info that diffuses back, so a high (clean-noise) strength is
the lever, not a knob on the light SDXL pass. CLI wiring (--strength default
None) lands with the cli refactor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply fixes from a full-repo review (code, tests, docs).
Security / correctness:
- Clamp attacker-controlled PNG/caBX chunk lengths to the remaining file
size in metadata.py and noai/c2pa.py (a malformed length no longer drives
a multi-GB read); skipped chunks seek instead of read.
- noai/isobmff.strip_c2pa_boxes is now fail-safe on a malformed box: return
the original bytes with a warning instead of silently truncating the tail,
so metadata --remove can no longer emit a corrupt file.
- doubao_engine._fixed_alpha_map clamps the glyph box to the image (no crash
on degenerate width-vs-height).
- watermark_remover._run_region_hires gates the phaseCorrelate offset on
response and magnitude (a spurious shift no longer garbles text) and drops
the generator after a CPU fallback (no MPS/CPU device mismatch).
Robustness:
- gemini_engine, doubao_engine, region_eraser normalize grayscale and RGBA
inputs to BGR at the engine entry points.
- image_io.imwrite returns False on an unwritable path (matches cv2).
- invisible_engine guards a None imread result before use.
- trustmark_detector._decoder uses a double-checked threading lock.
- ctrlregen.tiling.tile_positions raises on overlap >= tile.
- humanizer chromatic shift no longer wraps opposite-edge pixels.
- identify OpenAI caveat keyed on the normalized vendor, not a substring.
- Remove the dead "visible --detect-threshold" CLI option.
- publish.yml verifies the release tag matches the package version.
Docs:
- README strength 0.05 to 0.10; .env.example HF_TOKEN marked optional;
doubao_capture README updated to reverse-alpha-only; CLAUDE.md synced with
the new behaviors and the batch command.
Tests: new test_security_clamp.py for the read clamp and isobmff fail-safe;
erase CLI coverage; integrity-clash rule 2 end-to-end; multi-tag EXIF
survival and cross-format strip guards; channel/size, tiling, humanizer, and
imwrite regressions. Full suite 493 passed, 2 skipped; ruff and pyright src/
clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Raise default SynthID-removal strength 0.05 -> 0.10 (current Google SynthID)
The old default (0.04/0.05) no longer removes the CURRENT Google SynthID (Nano
Banana / Gemini 3): verified 2026-05-30 via the Gemini 'Verify with SynthID'
oracle on a real image -- 0.05 still detected, 0.10 not detected (OpenAI's was
already cleared at 0.05). Add DEFAULT_STRENGTH=0.10 in watermark_profiles, route
the engine + CLI defaults to it. At 0.10 small text deforms more, which is why
text protection (_run_region_hires) runs by default. CLAUDE.md SynthID note
corrected. CAVEAT: n=1 Google + n=1 OpenAI; broad corpus oracle validation
pending (task tracked).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop unused LOW/MEDIUM/HIGH strength profiles; CLI --strength defaults to DEFAULT_STRENGTH
The fixed strength presets (and get_recommended_strength) were dead -- nothing in
the pipeline used them, only tests. One knob now: DEFAULT_STRENGTH (0.10),
overridable per-call via the CLI --strength flag, which now defaults to that
constant (single source of truth). Removed the WatermarkRemover.LOW/MEDIUM/HIGH
class attrs and the get_recommended_strength tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(device): support xpu backend
* Fall back to CPU seed generator when device RNG unsupported (xpu)
Some torch-xpu builds have no device-side RNG, so torch.Generator(device="xpu")
raises when --seed is used. _make_seed_generator tries the device generator and
falls back to a backend-agnostic CPU generator. Adds a fallback unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Victor Kuznetsov <kuznetsov.va@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `pyright src/` strict-clean via a hybrid: pure-logic files are fully typed
(piexif gets a local typings/ stub; PIL info-dict loops guard isinstance(key, str);
progress returns Callable[..., None]; availability checks use importlib.util.find_spec
instead of unused imports), while the irreducibly-untyped cv2/torch/diffusers boundary
files carry a documented per-file `# pyright:` relax pragma (or a ctrlregen
executionEnvironment) that disables only the unknown-type rules. Public ndarray-returning
signatures on the relaxed engines are annotated NDArray[Any] so strict consumers (cli.py)
stay clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror protect_faces: protect_text defaults to True in invisible_engine and
watermark_remover, so the SDXL pipeline detects text per image and switches to
Differential Diffusion only when glyphs are found. Text-free inputs fall back to
plain img2img with no differential-pipeline load, so the autonomy is free. The
CLI now exposes a single off-switch --no-protect-text instead of the positive
flag, keeping the interface minimal.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SDXL img2img regenerates every pixel, so small text and CJK glyphs deform
at the strengths that defeat SynthID (issue #21). With --protect-text a
CJK-native PP-OCRv3 detector (2.4 MB ONNX, cv2.dnn, no torch, cached on
first use) locates text regions and the pass switches to the SDXL
Differential-Diffusion community pipeline: a per-pixel change map keeps
text regions largely intact while the background is regenerated to strip
the watermark. Gated to the SDXL default model; falls back to plain
img2img with a warning when unavailable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two CLI/IO robustness bugs surfaced by issues #17 and #19.
#17 -- non-ASCII image paths (Chinese/Cyrillic/accented) failed on Windows:
cv2.imread/imwrite use the platform ANSI code-page API, so the decode came back
empty with a "can't open/read file" warning. New image_io.imread/imwrite route
through np.fromfile+cv2.imdecode / cv2.imencode+tofile (Unicode-safe, byte-
identical output, cv2.imread None-semantics preserved); all 8 cv2 read/write
call sites now go through it. Behavior-neutral on macOS/Linux (already accept
UTF-8 paths), so the fix is correct-by-construction for the Windows-only bug.
#19 (incidental) -- rich parsed the "[gpu]" in the GPU-extra install hint as a
style tag and dropped it, so the printed command was the un-installable
"pip install 'remove-ai-watermarks'". Escaped as \[gpu] at both call sites.
Tests: test_image_io.py (non-ASCII round-trip, alpha, missing/empty/garbage
semantics); test_cli.py::TestGpuHintMarkup (install hint keeps the extra).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The native-vs-downscale decision in InvisibleEngine.remove_watermark (the
issue #10/#15 fix: max_resolution=0 must not pre-downscale, since any
downscale both loses quality and lets SynthID survive) had no test. Extract
it into a pure helper invisible_engine._target_size(w, h, max_resolution)
and cover it with tests/test_invisible_engine.py::TestTargetSize so a
re-introduced forced downscale fails CI instead of silently regressing #15.
Also:
- Clamp the short side to >=1 in _target_size: extreme aspect ratios (e.g.
5000x3 with --max-resolution 1024) truncated it to 0 and crashed
image.resize(). Pre-existing in the inline math; fixed now that it is a
named, tested function.
- Consolidate the two duplicated temp-file save blocks into one
unconditional save (behavior unchanged: the EXIF-transposed image is
still always persisted before WatermarkRemover reloads it by path), and
drop the now-redundant `_tmp_path is not None` guard in finally.
- Bump version 0.5.3 -> 0.5.4 (pyproject, __init__, uv.lock); document the
helper as the regression guard in CLAUDE.md.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The invisible pipeline force-downscaled inputs >1024px to 1024 before
diffusion, then upscaled the result back -- a lossy round-trip that was
the main cause of the quality loss reported in #10. The hosted raiw.cc
backend (fal fast-sdxl) does no pre-downscale, and at strength ~0.05
SDXL img2img doesn't need it.
Default is now native resolution (max_resolution=0). New --max-resolution
flag (invisible / all / batch) re-introduces an opt-in long-side cap only
to bound GPU/MPS memory on very large inputs.
Addresses #10. End-to-end quality/removal not re-verified locally (no GPU
here); matches raiw-app's proven production config.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
SD-1.5 dreamshaper at 768 px did not defeat SynthID v2 on Gemini 3 Pro
outputs (verified May 2026 via Gemini app's "Verify with SynthID"). Switch
the default invisible engine to SDXL at 1024 px, matching the raiw-app
production config (strength 0.05, steps 50). Drop the SD-1.5 pipeline.
Metadata layer: add C2PA UUID and IPTC AI marker byte-scan detection
across all formats, plus an ISOBMFF box walker (noai/isobmff.py) that
strips top-level C2PA uuid and JUMBF jumb boxes from AVIF/HEIF/JPEG-XL
containers without re-encoding.
README gets a Legal table and a Threat-model section about SynthID v2's
136-bit payload. CLAUDE.md tracks the SD-1.5 regression as historical
context.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Remove opencv-python from [gpu] extra (conflicts with headless in base deps)
- Add graceful fallback in 'invisible' and 'all' commands when GPU deps missing
- Cache InvisibleEngine in batch mode (avoid reloading model per image)
- Fix --humanize help text (was '0.0-1.0', actual range is 0-6.0+)
- Fix stale docstring referencing non-existent [invisible] extra
- Add [gpu] extra install instructions to README
- Fix broken NeuralBleach placeholder URL in Credits
- CLI with visible, invisible, all, metadata, and batch commands
- Gemini watermark removal via reverse alpha blending
- Invisible watermark removal via diffusion regeneration (SynthID, TreeRing)
- AI metadata stripping (EXIF, PNG text, C2PA)
- Face protection (YOLO/Haar) and analog humanizer
- 137 tests covering all CLI modes and core engines
- Ruff and Pyright clean
