Corpus-mined vendor gaps (42k-file metadata scan, 2026-07-23):
- Bria AI signs C2PA as "Bria Artificial Intelligence" with source type
empty (no trainedAlgorithmicMedia), so identify was completely blind;
registered with asserts_ai like Dreamina.
- fal.ai ("fal - Features & Labels Inc.", fal-ai/<model> generators)
was detected via the source type but never attributed; registered,
also asserts_ai as a pure generative platform.
- Apple Photos Clean Up (Apple Intelligence object removal) was detected
as a generic made-with-AI tag; now attributed as an AI edit via the
photoshop:Credit marker, and the credit value joins
AI_GENERATOR_TOKENS so removal strips it in parity.
Each new test was verified red without its fix.
The invisible/SynthID diffusion pass loads the whole SDXL fp16 pipeline
into VRAM via `pipeline.to("cuda")`. On an 8 GB card the weights alone
(~7 GB) leave no room for activations, so the run OOMs and there is no
in-tool way to recover short of falling back to CPU (~9 min/image).
Add an opt-in `--cpu-offload` flag (default off) that calls diffusers'
`enable_model_cpu_offload()` instead: submodules are streamed to the GPU
on demand, dropping peak VRAM to roughly the largest single submodule at
the cost of per-step transfers. CUDA-only; a no-op on cpu/mps. Threaded
through `invisible`, `all`, and `batch` to keep the knob set identical
across the three, mirroring the existing `--device`/`--pipeline` options.
Measured on a GTX 1070 Ti (8 GB): `invisible --pipeline sdxl --cpu-offload`
runs the SynthID scrub on-GPU in ~2.5 min vs ~9 min on CPU, where the
default full-VRAM path OOMs.
Test drives the placement decision with a mock pipeline (no model/GPU),
gated on torch so it runs under the `gpu` extra and skips the core CI
matrix, consistent with the model-running test policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The SDXL removal pipelines (sdxl + controlnet) were built without
add_watermarker=False, so diffusers embedded its default open "Stable
Diffusion XL" DWT-DCT invisible watermark on every output whenever
invisible-watermark is installed (the detect extra). A watermark REMOVER was
therefore replacing one detectable AI watermark (SynthID) with another: the
cleaned output re-read as AI (identify -> "Open invisible watermark: Stable
Diffusion XL"), observed on the SynthID validation sample.
Both SDXL loaders now call a shared _disable_sdxl_watermarker helper (mirrors
_maybe_add_fp16_vae; the ControlNetModel sub-model and the Qwen loader never
call it, since only the pipeline accepts the kwarg). Verified end to end: the
affected outputs re-run clean (is_ai=None, no open watermark).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bug fixes (each with a regression test):
- metadata strip parity across every marker placement: IPTC digitalSourceType
in XMP, the Samsung post-EOI trailer, the China TC260 AIGC block in EXIF
UserComment, a bare AIGC block in a non-standard APP segment, and the ISOBMFF
EXIF path (AIGC + xAI) are all now stripped -- anything a scanner flags, the
strip reaches
- Samsung genAIType detected when its trailer sits past the 512 KB scan window
(file-tail read on large photos)
- crashes on edge inputs: Gemini detector on images with a short side < 16px,
footprint_mask on a zero-size ndarray, the humanizer on chromatic_shift >=
width, and the CLI on unreadable/corrupt/empty input (clean error, not a
traceback)
- WebP written losslessly (cv2 quality 101), not lossy at 100
- the IPTC digitalSourceType algorithmicMedia (procedural, not trained on
sampled data) is no longer flagged as AI-generated, so clean procedural
content is not scrubbed
- c2pa source-type: compositeWithTrainedAlgorithmicMedia is checked before the
bare algorithmicMedia token, so an AI-enhanced composite is not misclassified
Detection:
- integrity-clash coverage now normalizes ByteDance / Canva / ElevenLabs /
Black Forest Labs, so a transplanted manifest next to an independent
conflicting stamp is caught; the generic China TC260 AIGC label is attributed
to a co-present TC260 vendor, so a legit Doubao image (its own C2PA + TC260
label) does not clash (corpus-validated: 0 new clashes on 5000 carriers)
CLI:
- batch exits non-zero (with a warning) when any image errors or a GPU-missing
SynthID scrub is skipped, and copies the input through so the output dir stays
complete -- it used to always exit 0 and could silently drop files
Perf:
- GeminiEngine reused as a process-wide singleton with a precomputed template
ladder: -24% on the identify sparkle path, detection byte-identical
Internal: one shared _ai_exif_targets rule set feeds both EXIF scrubbers so
their coverage cannot drift; docs synced; maintain.sh hardened so the uv-secure
internal teardown crash no longer aborts the gate (still fails on a real finding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two AI-provenance metadata types mined from the retained corpus that
identify previously read as no-signal:
- Dreamina (ByteDance's international Jimeng brand) signs C2PA as
"Bytedance Pte. Ltd." with a "Dreamina/x.y" claim generator and NO
digitalSourceType, so the generator name is the only AI signal. Add a
C2paAiVendor row with a new asserts_ai flag (identity-AI: presence
asserts AI without trainedAlgorithmicMedia) plus the derived
C2PA_IDENTITY_AI_ORGS view, folded into identify's c2pa_is_ai. Keyed on
the Dreamina generator token, not the "Bytedance Pte" issuer, so non-AI
CapCut edits signed by the same entity stay unattributed. 7/7 corpus
files now attribute to ByteDance.
- Tencent Cloud's TC260 AIGC variant uses a ServiceProvider/ServiceUser
schema (vs the producer-side ContentProducer schema), embedded in EXIF
ImageDescription; add those field names to _TC260_FIELDS so the generic
{"AIGC":{...}} gate accepts it. 11/11 corpus files now flagged.
Test-first: reproducing tests in test_identify.py / test_metadata.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Candidate carries only the fields the arbiter reads (key, label,
detected_strict, detected_relaxed, features); location/region/confidence were
vestigial from the removed best_auto_mark max-by-confidence path.
- resolve_backend returns preferred_inpaint_backend() directly (typed Literal)
instead of an identity ternary.
- colour/normalise/behaviour -> US spelling across code comments and docs.
No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace reverse-alpha removal with localize -> fill (template-free mask + one
shared cv2/MI-GAN/big-LaMa fill) for every mark; drops the colour-shift / dark-pit
failure modes, version-robust to a moved or re-rendered mark
- Separate perception/decision/action: engines report Candidates, a pure
decide(candidates, Context) arbiter owns all policy (sensitivity + provenance +
pill gate), remove_auto_marks orchestrates -- behavior-preserving (corpus 46/46/92)
- Three orthogonal knobs replace --method: --backend cv2|migan|lama,
--sensitivity auto|strict|assume-ai, provenance (auto from metadata)
- Add high-level api.remove_visible / visible_provenance (lazy top-level re-export);
visible --mark auto delegates to it so CLI and library share ONE path
- Read+write HEIC/AVIF on the pixel path via pillow-heif; imwrite preserves the input
format at max quality (JPEG q100/4:4:4); a no-op copies the original bytes verbatim
- Lossless byte-level JPEG metadata strip (no DCT re-encode); consolidate the two
remove_ai_metadata into one, delete legacy noai/cleaner + best_auto_mark
- Bump 0.13.0 -> 0.14.0
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
InvisibleEngine loads SDXL/ControlNet in fp16 on CUDA/XPU but called from_pretrained
without variant="fp16", so it read the full fp32 weight files (~7 GB) and downcast in
memory. _load_from_pretrained now passes variant="fp16" when torch_dtype is float16,
reading the half-precision files (~3.5 GB) instead - roughly halving the cold-start
weight read + host->device transfer (a phase-timed Modal run measured weight load as
~half of the ~25s cold start). Falls back to the default weights when a checkpoint ships
no fp16 variant (a custom --model), so the worst case is the prior behavior. fp32
(cpu/mps) and bf16 (qwen) never request the variant.
Tests: TestFp16WeightVariant (variant requested on fp16, fallback on missing, never on
fp32).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mined from the retained corpus 2026-06-22 (open-world EXIF/PNG-text/XMP scan,
minus the registry): three AI image generators that stamp a plain generator
name and no C2PA, so identify read them as no-signal -- and under the P0#5
no-signal skip would have skipped the scrub.
- NovelAI (anime SD): PNG tEXt Software/Source/Title. exif_generator now reads
PNG text chunks (via img.info), not only EXIF/XMP.
- Reve (reve.com): EXIF Software / XMP CreatorTool. Token is the full
"reve.com", not bare "reve" (would false-fire on "forever"/"reverie").
- Aphrodite AI: EXIF Make / Software.
Detection/removal parity: NovelAI stamps an AI-shaped VALUE under a non-AI KEY
(Title/Source), which _is_ai_key alone keeps. New _is_ai_value drops a text
chunk by value-token match on removal, mirroring exif_generator -- else the
cleaned file still read as NovelAI (verified on a real corpus file).
Tests: TestExifGenerator gains NovelAI PNG-text, Reve, Reve-not-overmatched,
Aphrodite, and a NovelAI detect/remove parity regression. Docs synced
(module-internals, watermarking-landscape, CLAUDE.md).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The qwen oracle floors are certified, not pending. Near-threshold scrub is
seed-non-deterministic, but the prod path pins one fixed seed, so a certified
floor reproduces run-to-run -- pinning the seed is the release gate, not a seed
sweep. Reword the docstring so it stops implying an open seed-repeat gate.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- watermark_remover: _build_qwen_kwargs now passes explicit height/width (via
_qwen_target_size, floored to /16). Without it QwenImageImg2ImgPipeline defaults to
1024x1024 and silently squishes non-square inputs, distorting the scene and garbling text.
- watermark_profiles: resolve_strength gains a `pipeline` arg + a Qwen strength ladder
(_QWEN_VENDOR_STRENGTH, Gemini 0.25), so `--pipeline qwen` gets its certified floor
automatically; retires the manual "pass --strength 0.25 for Gemini on qwen" workaround.
- fidelity_metrics: replace per-face nearest matching (collided on multi-face images when a
variant dropped a face, corrupting the identity metric) with a collision-free one-to-one
assignment (assign_faces_one_to_one). lapvar/LPIPS were always bbox-anchored and immune.
Regression-guarded by tests/test_fidelity_matching.py.
- docs: record the measured outcomes of the qwen-improvement arc. The Qwen ControlNet
face-fix is CLOSED (no permissive Qwen detail/tile ControlNet exists; canny carries edges,
not skin grain). The `--pipeline auto` router + faces+text mixed dual-pass were prototyped
and DROPPED (controlnet wins faces AND display text: abba CER 0.114 vs qwen 0.379).
Z-Image-Turbo was tried and dropped (same regeneration limits). qwen stays a manual opt-in;
controlnet is the default for everything.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For AI-enhanced composites (digitalSourceType compositeWithTrainedAlgorithmicMedia,
identify ai_source_kind == "enhanced"; roadmap P1#8): regenerate ONLY the AI
region and preserve the real photo elsewhere, instead of regenerating the whole
frame.
- noai.tiling.feather_region_composite(base, regenerated, box, *, feather): pure,
model-free compositor that blends the regenerated AI box back over the original
with a feathered seam, leaving pixels OUTSIDE the box exactly equal to base.
Fully unit-tested (outside-box exactness, interior == regenerated, hard paste at
feather 0, monotonic seam ramp, dtype/grayscale/clamp/empty-box/shape-mismatch).
- WatermarkRemover.remove_watermark(region=, region_feather=) and the module-level
convenience function thread it through: the remover regenerates (or tiles) the
frame, then composites only the AI box back over the original input. The box is
caller-supplied -- a C2PA composite manifest carries no reliable machine-readable
region, so none is fabricated. The no-model lossless region path stays
region_eraser.erase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retained-corpus mining (2026-06-20) surfaced three provenance gaps; all are
oracle-free and regression-guarded.
- C2PA vendor coverage (roadmap): register Volcano Engine under its Chinese
legal entity 北京火山引擎科技有限公司 (the latin "volcengine" needle misses
those certs) -> normalizes to the same ByteDance platform; register ElevenLabs
("Eleven Labs Inc.", pure generative-AI) as a generator. Document the
deliberate exclusion of TikTok Inc. and PixelBin.io/"Fynd" (provenance/transform
signers, not generators) so they are not re-added.
- AI-generated vs AI-enhanced (roadmap): ProvenanceReport.ai_source_kind splits
the C2PA digital-source-type into "generated" (trainedAlgorithmicMedia) vs
"enhanced" (compositeWithTrainedAlgorithmicMedia) so a caller branches a
full-frame scrub from a region-targeted clean. Parsed once in
noai.c2pa._populate_registry_fields (PNG + any c2pa-python-readable container),
with a raw head-scan fallback in identify for the non-PNG raw-blob path. CLI
verdict reads "AI-generated (fully synthetic)" vs "AI-enhanced (real content
with an AI-composited region)"; surfaced in --json.
- Detect-vs-remove threshold desync (P0#7): identify's sparkle threshold and the
removal arbitration gate were two independent 0.5 constants. Unify them into the
single GEMINI_SPARKLE_TRUST_CONF (identify imports it) so they can never drift.
Lowering the gate to recover faint sub-0.5 sparkles was evaluated and REJECTED:
a real Doubao text mark scores ~0.40-0.42 as a gemini match with a higher
core-ring brightness margin than a genuine faint sparkle, so neither confidence
nor the brightness gate separates them in [0.35, 0.5) -- lowering would trade a
rare miss for false-positive removals on clean images. Regression-guarded by
TestSparkleDetectRemoveAlignment (real demo sparkle at borderline opacities;
identify and best_auto_mark must agree on either side of the line).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Oracle seed-repeat + floor refinement (2026-06-20, data/qwen_in):
- OpenAI floor 0.10 is SEED-ROBUST: 0.05 and 0.075 still detected; 0.10 clean on
seeds 0-4 (5/5) -> a random seed is safe.
- Gemini floor lowered 0.30 -> 0.25 (0.20 still detected, 0.25 clean on both
images). Single-seed (seed 0): the Gemini oracle rate-limits volume seed-repeat,
so pin a seed in prod rather than relying on seed-robustness there.
Re-measured fidelity at the certified floors (controlnet 0.15 vs Qwen 0.25 for
Gemini): faces still favor controlnet (ArcFace 0.546 vs 0.382, lapvar 0.62 vs
0.40); the short-CJK text case is now a TIE (gemini_1 0.037 vs 0.037 -- the earlier
Qwen 0.000 was at 0.30, not the floor). Qwen's text win holds on substantial
Latin/mixed text (OpenAI 0.385 vs 0.241 / 0.341 vs 0.290). Update watermark_profiles
comment, CLAUDE.md, module-internals, known-limitations.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A third diffusion pipeline alongside sdxl/controlnet: Qwen-Image (20B MMDiT,
Apache-2.0 code AND weights) img2img. The scrub still comes from the img2img
strength; Qwen preserves text (incl. CJK) and structure markedly better than
SDXL at the scrub floor, so it over-regenerates real photos far less (directly
targets the controlnet over-regeneration that degrades real uploads).
- watermark_profiles: QWEN_MODEL_ID, normalize_profile accepts "qwen".
- WatermarkRemover: _load_qwen_pipeline (bf16, loads Qwen base unless --model
overridden, clear ImportError if diffusers lacks the class), _run_qwen (no
MPS fallback -- 20B is CUDA/cloud-class), dispatch in _generate_one/preload,
pure _build_qwen_kwargs (true_cfg_scale, not guidance_scale).
- Shared _base_load_kwargs() across all three loaders (dtype + token).
- CLI --pipeline gains "qwen"; invisible_engine threads it through.
- scripts/qwen_scrub_prototype.py: standalone PEP 723 GPU experiment.
Prototype oracle floors (Modal A100-80GB, single seed, controls SynthID-positive,
PENDING seed-repeat cert): OpenAI clears at strength ~0.10, Gemini at ~0.30 (0.20
still detected), with CJK text + faces faithful where controlnet plasticizes. The
Gemini floor is higher than the shared default ladder, so pass an explicit
--strength for Gemini on this pipeline until a Qwen-specific ladder is certified.
The model-running path is CUDA-only (untestable locally); unit tests cover the
pure call-shape (_build_qwen_kwargs) and profile normalization without torch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a lossless alternative to the --max-resolution downscale for large
images that OOM on MPS/GPU: regenerate in overlapping, feather-blended
tiles at native resolution.
- noai/tiling.py: pure plan_tiles (uniform tiles, last flush to edge) +
feather_weights (strictly-positive separable taper -> partition-of-unity
blend) + run_tiled (per-tile generate callable, decoupled from the
pipeline). Unit-tested without the model.
- WatermarkRemover.remove_watermark: refactor _generate into _generate_one
+ a tiled branch that engages only when --tile is set and the long side
exceeds tile_size (ControlNet canny is rebuilt per tile).
- Thread tile/tile_size/tile_overlap through InvisibleEngine and the
invisible/all/batch CLI commands via a shared _tile_options decorator.
Verified end-to-end on the real SDXL pipeline (forced 2x2 tiling on a
1024px sample, MPS): non-degenerate output, no gross seam at tile borders.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes a documented coverage gap (P2#9): an AI Software/Make/Artist/ImageDescription
token in an EXIF item (its TIFF bytes live in mdat/idat) survived remove_ai_metadata
because the top-level box stripper and (absent pillow-heif) the PIL EXIF reader can't
reach it. New isobmff.blank_ai_exif_tokens finds EXIF TIFF blocks by their II/MM
byte-order header, validates each with piexif (a coincidental II/MM run in pixels
won't parse as a TIFF IFD, so it's ignored), and overwrites any AI_GENERATOR_TOKENS-
bearing value with same-length spaces -- so box sizes and iloc offsets stay valid and
the coded image is untouched (mirrors blank_ai_xmp_packets; no iinf/iloc surgery, no
exiftool dep). Camera/editor EXIF without an AI token is preserved. Wired into
remove_ai_metadata's ISOBMFF path. Covers the realistic AI-generator-token case; xAI-
signature-in-meta-box-EXIF (Grok is JPEG-only) stays out.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mining the local production corpus (25,725 imgs) surfaced two AI vendors signing
C2PA that the registry missed:
- Canva (Magic Media) signed "Canva" + trainedAlgorithmicMedia -> detected AI but
no platform attributed (disproves the old "Canva exports strip C2PA" assumption).
- BytePlus (ByteDance international: Seedream/Seededit) signs "Byteplus Pte. Ltd.";
the bare volcengine needle missed it, so its output was mis-attributed to "Adobe
Firefly" via an incidental "Adobe XMP" string the fallback byte-scan picked up.
Adding both to C2PA_AI_VENDORS lets the clean manifest issuer attribute them
directly. Corpus re-run: 16 platform changes, all improvements (3 Adobe->ByteDance
fixes, 4 None/TC260->ByteDance, 9 None->Canva), 0 regressions. An attempted
signer-based attribution fallback was measured and dropped: it regressed 18 images
(friendly ByteDance label -> raw Chinese cert org; IPTC tool name pre-empted).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
extract_c2pa_info now uses the c2pa-python Reader first (any container, whole
manifest store incl. ingredient manifests), falling back to the hand-rolled caBX
parser for blobs the validator rejects (synthetic/partial, broken wheel). The
issuer/source-type/SynthID/soft-binding registry scan is shared by both paths
(_populate_registry_fields), so the return-dict contract is unchanged. Also
replaces the dead `from c2pa import has_c2pa_metadata` import in metadata.py with
a real Reader presence check. c2pa-python added as a core dep (MIT/Apache, ~+5MB
RSS, no torch; wheels cover the CI matrix).
Validated on the full local spaces corpus (25,725 imgs): 0 regressions; 384
manifests newly parsed (379 non-PNG JPEG/WebP + 2 PNGs the byte-scanner missed);
3 false Adobe/Microsoft->Google attributions fixed via real-manifest parsing.
The docs/module-internals.md section for this change already landed in 41f6797.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A 2026-06-14 oracle re-test on the deployed Modal controlnet worker (v0.10.0)
cleared SynthID at OpenAI 0.10 (2 photoreal) and Google 0.15 (2 native
2816x1536, retiring the "native >= 0.30" guess), while a pixel sweep showed the
2026-06-04 cert floors (0.20/0.30) over-regenerated for no efficacy gain
(Google MAE -20% at 0.15). Lowers OPENAI_STRENGTH 0.20->0.10, GEMINI_STRENGTH
and UNKNOWN_STRENGTH 0.30->0.15.
Caveats documented in watermark_profiles.py + docs: removal near this floor is
seed-non-deterministic (a service must pin a verified seed), and the n=2 re-test
did not cover flat-graphic hard cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The convenience wrapper's docstring still quoted the pre-2026-06 ladder
(0.10 OpenAI / 0.15 Google / 0.15 unknown). The live constants in
watermark_profiles.py are 0.20 / 0.30 / 0.30, applied to both the controlnet
and sdxl pipelines. Docstring only; behaviour was already correct via
vendor_for_strength + resolve_strength.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nine findings from a high-effort project-wide review, fixed and verified
(571 passed, ruff/pyright clean):
Correctness:
- all/batch now remove Doubao/Jimeng/Samsung visible text marks: the visible
step routes through the registry (new cli._remove_visible_auto) instead of a
hardcoded GeminiEngine, so they no longer leave the wordmark intact.
- batch always reads the original source (dropped the out_path-reuse that
re-processed already-cleaned outputs on a re-run).
- img2img_runner only retries the diffusion call on the deprecated-callback
TypeError; any other TypeError now propagates instead of double-running.
- gemini detect/remove and the reverse-alpha engines normalize channels via a
new image_io.to_bgr, fixing a grayscale/BGRA crash in the FP-gate path.
- _png_late_metadata advances its cursor by the clamped length, so a malformed
chunk length no longer aborts the late AI-label scan.
Cleanup / efficiency:
- Consolidate the ~90%-identical Doubao/Jimeng/Samsung engines into a shared
config-driven _text_mark_engine.TextMarkEngine base; each engine is now a thin
subclass (TextMarkConfig + test shims). Behavior is byte-exact (the three
engine test suites pass unchanged). Registry adapters collapse to one
_text_mark(...) row each. Gemini stays a separate engine.
- scan_head is memoized per (path, size, mtime), so identify() reads the file
head once instead of ~8 times.
- invisible_engine post-processing decodes/encodes the output once (chained in
memory) instead of 2-4 times across stages.
- Remove the orphaned get_model_id_for_profile (+ CONTROLNET_PROFILE); derive
the --strength help from the strength constants (strength_default_help) so it
cannot drift; share the --pipeline/--strength click options; simplify the
retired --auto resolver.
Net -835 lines. Tests added for the registry-routed visible pass, to_bgr,
the polish/model/guidance wiring, and strength_default_help. CLAUDE.md updated
for the new base module, the engine/registry changes, image_io.to_bgr, and the
scan_head cache.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Overhaul the diffusion-removal surface around a single robust default and a
complete, consistent CLI.
Pipeline + strength:
- controlnet is now the DEFAULT pipeline (CLI --pipeline + both engine ctors).
With the certified higher strength it clears both photoreal and flat-graphic
content, whereas plain SDXL left SynthID on flat graphics.
- Rename the plain-SDXL profile default -> sdxl; "default" stays as a back-compat
alias (normalize_profile + a click callback that warns).
- Unify the strength ladder: resolve_strength applies ONE vendor-adaptive ladder
(the certified controlnet floors OpenAI 0.20 / Google 0.30 / unknown 0.30) to
both pipelines. sdxl is the weaker remover on its own hard case (flat fills),
so the certified floor is the right floor for it too.
CLI completeness:
- Add --model (HF model id) to invisible + batch (was only on all) and
--guidance-scale (CFG) to all three diffusion commands; both were library
knobs the CLI did not expose.
- Flip --adaptive-polish to ON by default (it self-gates to a no-op where there
is no detail deficit, so default-on is safe).
- Share --pipeline / --strength / --model / --guidance-scale as single
decorators so invisible/all/batch keep an identical surface; the --strength
help is derived from the strength constants (strength_default_help) so it can
never drift from the ladder.
Removals:
- Delete the auto_config content-detection planner + its YuNet/DBNet assets
(~2.6 MB): with controlnet always the pipeline and the polish self-gating, the
face/text/edge detection no longer changed behavior. --auto is now a deprecated
no-op that only warns (the polish it enabled is the default).
Docs (README, CLAUDE.md, docs/synthid.md) updated throughout; added an
InvisibleEngine Python API example. Tests cover the alias warnings, the
polish default, and the --model/--guidance-scale wiring.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The fp16-fix VAE swap (#29) is gated to the default SDXL checkpoint, so a
custom model_id, a stale pre-fix install, or a fal/custom loader can still
decode to an all-black/NaN frame in fp16 (reporter: gpt-image 1448x1086,
the `image_processor.py invalid value encountered in cast` warning).
Add a model-agnostic backstop in remove_watermark: after generation, if the
run was fp16 and the output is degenerate (_is_degenerate_image: near-zero
mean and variance), rebuild the pipeline in fp32 on the same device and
re-run once. fp32 is the verified-clean path, so a black image is never
returned regardless of model_id or version. Mirrors the MPS->CPU fallback's
self-mutation pattern; batch inherits it. Verified e2e on MPS by forcing
fp16 with the swap disabled (first pass black, guard fired, retry clean).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three P2 cleanups from a library-wide review.
Detection -- single C2PA_AI_VENDORS registry (noai/constants.py):
- C2PA_ISSUERS, SYNTHID_C2PA_ISSUERS, and identify._ISSUER_PLATFORM now derive
from one C2paAiVendor table, so adding a C2PA vendor is one entry instead of
edits in three places across two files. Behavior-identical (262 detection
tests pass; the kept `needle` field is load-bearing -- it differs from `org`
for Google and ByteDance, with no mechanical derivation).
Code-health:
- region_eraser.erase_lama now accepts grayscale/BGRA like erase_cv2 (it
crashed on grayscale and silently dropped alpha on BGRA). +2 regression tests.
- batch frees the device cache between images via a shared try_empty_device_cache
helper (generalized from the MPS-only _try_clear_mps_cache, now reused by both
the MPS->CPU fallback and the batch loop).
- batch gained --controlnet-scale (parity with invisible/all).
CI / packaging:
- publish.yml uploads via `uv publish` (PyPI trusted publishing over OIDC),
replacing pypa/gh-action-pypi-publish so uploads no longer depend on that
action's bundled twine accepting the Metadata-Version. Workflow filename +
pypi environment unchanged, so PyPI's trusted-publisher entry still matches.
- hatchling pin relaxed <1.28 -> <1.31 (verified against hatch's changelog:
1.30.0 made Metadata 2.5 the default, 1.30.1 reverted to 2.4; 1.27-1.29 were
always 2.4). Kept as belt-and-suspenders so the first uv-publish release ships
2.4, isolating the uploader swap from the metadata-version bump.
Docs (CLAUDE.md, pyproject) synced; corrected the inaccurate "hatchling 1.28+
emits 2.5" note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add `--pipeline controlnet` (SDXL base + xinsir canny ControlNet via
StableDiffusionXLControlNetImg2ImgPipeline): the canny edge map conditions the
img2img regeneration so text and face STRUCTURE stay sharp, while the watermark
is still removed by the regeneration (`strength`) -- no original pixels are
copied or frozen, so SynthID does not survive. Oracle-verified clean on OpenAI
with better text/structure fidelity than plain img2img at equal strength.
`--controlnet-scale` tunes structure preservation; fp32 on mps/cpu (fp16-fixed
VAE on cuda/xpu). Shares the img2img runner (live progress + MPS->CPU fallback)
and the fp16-VAE-fix / device-move helpers with the default pipeline.
Remove the superseded subsystems -- ctrlregen (SD1.5 clean-noise),
text-protection (differential / region-hires) and face-protection: they either
destroyed real content or shielded the watermark by re-using original pixels.
controlnet replaces them by regenerating everything under edge conditioning.
Canny preserves face structure but not identity; face IDENTITY is a separate
face-restoration post-pass (CodeFormer/GFPGAN), researched + prototyped but not
yet shipped. An IP-Adapter FaceID attempt was built and removed (footgun: needs
high strength, corrupts faces at removal strength).
Docs: docs/controlnet-removal-pipeline-research.md, scripts/controlnet_sweep.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The default img2img strength is now chosen from the detected SynthID vendor
(C2PA issuer) instead of a single fixed 0.30: OpenAI gpt-image -> 0.10, Google
Gemini -> 0.15, unknown source -> 0.15. Explicit --strength always wins.
Basis: an oracle-verified June 2026 controlled study (clean v0.8.6, text/face
protection OFF, per-image openai.com/verify or Gemini-app verdict). OpenAI's
SynthID clears at 0.05 across 1024-1600 px (n=4, resolution-independent);
Google's is ~3x more robust and needs 0.15 on the capped-1536 path (n=4). The
dominant factor is the VENDOR, not resolution. The earlier single 0.30 default
and the "resolution dependence" lore came from contaminated tests run with the
protect-text bug ON (issue #14) -- re-running those same 1600x1600 images clean
removes SynthID at 0.05.
`vendor_for_strength(path)` reads metadata.synthid_source on the ORIGINAL input
and is threaded through cli (invisible/all/batch) -> invisible_engine ->
watermark_remover -> resolve_strength(strength, profile, vendor), so display and
execution use the same vendor (the engine sees a temp path whose C2PA the visible
pass already stripped, so detection must happen in the CLI on the pristine
source). Caveat: Google's 0.15 was validated only on --max-resolution 1536;
native 2816 Gemini was not locally measurable (OOM on Apple Silicon) and is
pending GPU validation on raiw.cc.
Docs: docs/synthid.md sections 2.2/4.4/5.2 corrected (the contaminated
resolution-dependence findings replaced with the clean oracle-verified table);
README and CLAUDE.md updated; CLI --strength help reflects the adaptive default.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
identify previously ran only the Gemini sparkle as a visible detector, so a
Doubao/Jimeng image with stripped TC260 metadata had no visible fallback. Add
`_visible_text_marks` (registry-backed) so the ByteDance Doubao 豆包AI生成 and
Jimeng 即梦AI marks are detected too, each gated by its own engine NCC threshold
via MarkDetection.detected. New signals `visible_doubao` / `visible_jimeng`
(medium), same stripped-metadata fallback role as the sparkle; excluded from
integrity-clash vendor claims; set platform only when no harder signal did.
Also make `noai/__init__` lazy (PEP 562 __getattr__): importing the light
`noai.c2pa` / `noai.constants` submodules (which identify needs) no longer
eagerly pulls `watermark_remover`, which imports torch + diffusers at module
top. `import remove_ai_watermarks.identify` drops from ~420 MB to ~21 MB in a
full gpu/detect install (torch not loaded), so it fits a 512 MB host; the
removal API resolves lazily on first access. Guarded by TestIdentifyImportIsLight.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An A/B at strength 0.3 on a real e-commerce infographic (updated GPU study)
reverses the earlier claim: SynthID is a GLOBAL watermark, so 0.3 removes it
whether protect_text is on or off, and protection SALVAGES text fidelity (medium
headings/body stay readable; off, they garble). The earlier 'protect_text shields
the watermark, use --no-protect-text' was wrong -- it mistook the 0.10 strength
failure for a protection effect. Recommended SynthID config: ~0.3 + protect_text ON
(the default). Also document the oracle scope: the Gemini app 'Verify with SynthID'
is the only valid SynthID oracle; openai.com/verify is provenance-scoped (C2PA) and
does NOT measure SynthID. Corrects CLAUDE.md + README + watermark_profiles comment
shipped in cddbaf6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An oracle-verified GPU strength study (Modal A100, native res, Gemini-app
'Verify with SynthID', n=3 fresh Gemini images, protect_text/faces off) found the
current Google SynthID survives strength 0.10/0.15/0.2 and is removed only at 0.3.
The previous 0.10 default (set from an n=1 result) no longer clears it -- Google
hardened SynthID and the threshold has climbed 0.05 -> 0.10 -> ~0.3. Bump
DEFAULT_STRENGTH to 0.30; OpenAI/ChatGPT carry C2PA not SynthID, so 0.10 is plenty
there (pass --strength 0.10). Note protect_text shields the text regions SynthID
hides in (use --no-protect-text for full removal on text-heavy images).
The same study found ctrlregen at clean-noise strength DESTROYS real images
(hallucinated micro-text in smooth regions), with no usable middle setting, so the
literature's 'clean-noise is the lever' did not hold empirically. Flag ctrlregen
EXPERIMENTAL in the CLI --pipeline help, README, and watermark_profiles; SDXL
img2img at ~0.3 stays the shippable path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The ctrlregen profile inherited the SDXL img2img --strength default (0.10), a
near-identity pass that loaded ControlNet + DINOv2-giant and barely changed the
image -- a no-op for removal. resolve_strength() now resolves an unset strength
per profile: 0.10 for the SDXL default, CTRLREGEN_DEFAULT_STRENGTH (1.0,
clean-noise) for ctrlregen. It checks `is None` rather than falsiness, so an
explicit 0.0 is respected (the old `strength or DEFAULT` swallowed it).
Research basis: CtrlRegen (ICLR 2025, arXiv:2410.05470) removes robust
watermarks by regenerating from clean Gaussian noise; partial-noise img2img
retains watermark info that diffuses back, so a high (clean-noise) strength is
the lever, not a knob on the light SDXL pass. CLI wiring (--strength default
None) lands with the cli refactor.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply fixes from a full-repo review (code, tests, docs).
Security / correctness:
- Clamp attacker-controlled PNG/caBX chunk lengths to the remaining file
size in metadata.py and noai/c2pa.py (a malformed length no longer drives
a multi-GB read); skipped chunks seek instead of read.
- noai/isobmff.strip_c2pa_boxes is now fail-safe on a malformed box: return
the original bytes with a warning instead of silently truncating the tail,
so metadata --remove can no longer emit a corrupt file.
- doubao_engine._fixed_alpha_map clamps the glyph box to the image (no crash
on degenerate width-vs-height).
- watermark_remover._run_region_hires gates the phaseCorrelate offset on
response and magnitude (a spurious shift no longer garbles text) and drops
the generator after a CPU fallback (no MPS/CPU device mismatch).
Robustness:
- gemini_engine, doubao_engine, region_eraser normalize grayscale and RGBA
inputs to BGR at the engine entry points.
- image_io.imwrite returns False on an unwritable path (matches cv2).
- invisible_engine guards a None imread result before use.
- trustmark_detector._decoder uses a double-checked threading lock.
- ctrlregen.tiling.tile_positions raises on overlap >= tile.
- humanizer chromatic shift no longer wraps opposite-edge pixels.
- identify OpenAI caveat keyed on the normalized vendor, not a substring.
- Remove the dead "visible --detect-threshold" CLI option.
- publish.yml verifies the release tag matches the package version.
Docs:
- README strength 0.05 to 0.10; .env.example HF_TOKEN marked optional;
doubao_capture README updated to reverse-alpha-only; CLAUDE.md synced with
the new behaviors and the batch command.
Tests: new test_security_clamp.py for the read clamp and isobmff fail-safe;
erase CLI coverage; integrity-clash rule 2 end-to-end; multi-tag EXIF
survival and cross-format strip guards; channel/size, tiling, humanizer, and
imwrite regressions. Full suite 493 passed, 2 skipped; ruff and pyright src/
clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The stock SDXL VAE overflows to NaN in fp16, so the plain img2img path decodes
to an all-black image on a CUDA/XPU fp16 backend. This is the raiw.cc black
result HitaoLin reported (a 1086x1448 input came back uniformly black). cpu/mps
run fp32 and never hit it, and the differential / region-hires pipeline already
upcasts the VAE itself, so only the plain path on a fp16 GPU was exposed.
`_load_pipeline` now loads `madebyollin/sdxl-vae-fp16-fix` for the default SDXL
checkpoint when running fp16, gated by the pure helper `_needs_fp16_vae_fix`. A
custom non-SDXL model keeps its own VAE.
The decision logic is unit-tested without a download (TestFp16VaeFix). The
black->clean recovery itself needs a CUDA GPU and was not verifiable on this MPS
machine; it must be confirmed on the backend.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Raise default SynthID-removal strength 0.05 -> 0.10 (current Google SynthID)
The old default (0.04/0.05) no longer removes the CURRENT Google SynthID (Nano
Banana / Gemini 3): verified 2026-05-30 via the Gemini 'Verify with SynthID'
oracle on a real image -- 0.05 still detected, 0.10 not detected (OpenAI's was
already cleared at 0.05). Add DEFAULT_STRENGTH=0.10 in watermark_profiles, route
the engine + CLI defaults to it. At 0.10 small text deforms more, which is why
text protection (_run_region_hires) runs by default. CLAUDE.md SynthID note
corrected. CAVEAT: n=1 Google + n=1 OpenAI; broad corpus oracle validation
pending (task tracked).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Drop unused LOW/MEDIUM/HIGH strength profiles; CLI --strength defaults to DEFAULT_STRENGTH
The fixed strength presets (and get_recommended_strength) were dead -- nothing in
the pipeline used them, only tests. One knob now: DEFAULT_STRENGTH (0.10),
overridable per-call via the CLI --strength flag, which now defaults to that
constant (single source of truth). Removed the WatermarkRemover.LOW/MEDIUM/HIGH
class attrs and the get_recommended_strength tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the default text-protection path. Differential Diffusion froze text in
latent space, which left SynthID intact inside text (violating remove-everywhere)
and still softened sub-8px strokes (VAE latent limit). _run_region_hires instead
scrubs the whole image, then re-scrubs each detected text block at high resolution
and feather-composites it back: every pixel is regenerated (watermark removed
everywhere) while small text stays crisp (high-res strokes span >1 latent cell).
merge_text_regions + feather_paste are pure and unit-tested; each re-scrubbed
patch is phase-correlated back to the original crop to null the ~1-2px round-trip
offset. Synthetic 18px multilingual text: text-region SSIM 0.28 -> 0.48, visually
garbled -> readable across Latin/Cyrillic/CJK. Legacy _run_differential /
build_change_map remain but are no longer the default. Prod use still requires
confirming via the SynthID oracle that re-scrubbed text zones read watermark-free.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(trustmark): gate detection on re-encode durability to kill false positives
TrustMark's wm_present flag is a BCH validity check that spuriously
validates on a content-correlated fraction of un-watermarked images
(AI textures trip it more than camera photos). On a 1343-image set all
20 raw detections were false, several on Gemini/OpenAI/Doubao output that
cannot carry Adobe's watermark, with random-bytes secrets.
A genuine TrustMark is a durable soft binding that survives re-encoding,
so detect_trustmark now re-decodes after a mild JPEG round-trip and
requires the same schema both times. Every observed false positive
collapsed under this gate; the second decode runs only on the rare hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(identify): Samsung Galaxy AI, FLUX, ByteDance C2PA; fix C2PA substring FP
Detection extensions verified on real signed files (2026-05-29):
- Samsung Galaxy AI: signer attribution via a new _SIGNER_C2PA_PLATFORM
(Samsung Galaxy / ASUS Gallery) kept separate from the capture-camera
_DEVICE_C2PA_PLATFORM so a Galaxy AI edit (device cert + AI source type)
does not trip the camera-vs-AI integrity clash. Plus metadata.samsung_genai:
the proprietary genAIType marker in PhotoEditor_Re_Edit_Data, a medium-
confidence AI-editing signal (samsung_only branch).
- Black Forest Labs (FLUX) and ByteDance Volcano Engine (Doubao/Jimeng)
added as C2PA issuers + issuer->platform mappings.
- fix: C2PA presence required only the bare 4-byte 'c2pa' substring, which
false-positives on compressed pixel data (a recompressed PNG IDAT re-flagged
C2PA after its manifest was correctly stripped). New c2pa_marker_in() requires
the JUMBF wrapper (jumb+c2pa) or the C2PA uuid box; applied in identify +
metadata. Verified: all 535 real C2PA files carry jumb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): gate detection on text structure to cut ~95% of false positives (#23)
Coverage alone over-fired: any textured bottom-right corner cleared the
threshold, so the detector false-positived on ~28% of arbitrary images.
The real '豆包AI生成' mark is six glyphs in one row, so detect now also
requires the text-structure signature (_glyph_structure): many connected
components, no single dominant blob, concentration in a thin horizontal
band. False positives dropped 343 -> 17 across the corpus while keeping
real-mark recall and the doubao-1.png sample. Also accept a no-op force
kwarg for remover-interface symmetry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(samsung): add Samsung Galaxy AI visible-badge remover
New samsung_engine.py removes the bottom-left sparkle + localized
'AI-generated content' badge that Galaxy AI tools stamp. Mirrors the
Doubao locate->mask->inpaint pattern but bottom-left, with a dual-polarity
top-hat mask (the badge is light-on-dark or dark-on-light). Detection gates
on a band + left-anchor signature (the Doubao CJK-component gate does not
transfer: Latin badge letters connect into few blobs). Explicit-only --
tuned on few real badges with a ~4% FP floor, so it is not used in auto.
Synthetic byte-blob fixtures (real badges are user content, not shipped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(visible): unified known-watermark registry + LaMa inpaint backend
watermark_registry.py is a single catalog of known visible marks, each
tying {usual location, in_auto flag, recovery strategy, detect adapter,
remove adapter}: gemini (reverse-alpha, exact), doubao, samsung. cmd_visible
is now registry-driven (best_auto_mark for --mark auto; mark_keys() feeds the
CLI choices) -- the per-mark _run_doubao/_run_samsung helper branches are gone.
Cross-engine confidences are not comparable, so the gemini adapter applies the
corpus-validated 0.5 sparkle threshold for auto arbitration (its engine flag is
loose and weakly fired ~0.36 on Doubao text, hijacking auto).
--backend auto|cv2|lama chooses background reconstruction for the mask-based
marks; auto = LaMa when onnxruntime is present, else cv2. For LaMa the mask is
the FILLED glyph bounding box (sparse glyph masks leave anti-aliased edges
behind). cv2 stays the zero-dependency fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: watermark registry, Samsung/FLUX/ByteDance detection, LaMa backend, trustmark gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(doubao): exact reverse-alpha removal from captured alpha map
The Doubao '豆包AI生成' mark is a fixed semi-transparent white overlay, so
given its alpha map the original pixels are recovered exactly:
original = (wm - a*logo)/(1-a) -- no inpaint hallucination.
The alpha map + logo colour were solved from real black+gray Doubao captures
on a controlled background: on black captured = a*logo, and the black/gray pair
solves a per-pixel without assuming the logo colour (a_max~0.65, logo near-white);
the white capture cross-validates (mark vanishes to a flat fill). Bundled as
assets/doubao_alpha.png + geometry constants.
remove_watermark_reverse_alpha applies it scaled to image width; exact at the
captured width, so the registry routes doubao through it only when
reverse_alpha_available (width within the calibrated band) and the mark is
detected, falling back to mask inpaint (cv2/LaMa) otherwise. A light residual
inpaint cleans the sub-pixel rescaling error. Add captures at more resolutions
to widen exact coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(visible): reverse-alpha only -- drop inpaint removal + heuristic detection
Per the principle that we only remove/detect what we can do exactly, the
visible-mark path is now reverse-alpha only:
- Doubao detect is reverse-alpha-consistent: match the bundled alpha glyph
silhouette against the corner via TM_CCOEFF_NORMED (DETECT_NCC_THRESHOLD 0.4)
-- keys on the '豆包AI生成' SHAPE, not coverage/structure heuristics. FP
7/1243 (0.6%). Removes the cv2 inpaint path + the _glyph_structure gate.
- Registry is reverse-alpha only: dropped the cv2/LaMa backend (_glyph_remove,
_lama_box_inpaint, default_backend, --backend) and the Samsung entry. Doubao
outside the alpha resolution band is skipped, never inpainted.
- Removed samsung_engine.py + tests + --mark samsung (no alpha map captured;
Samsung C2PA/genAIType metadata detection in identify is unaffected).
- The universal erase --region (cv2/LaMa) is unchanged -- arbitrary-region
inpainting stays a user-directed tool, separate from the known-mark registry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(doubao): NCC sub-pixel alignment -> reverse-alpha at any resolution
A pure width-scale of the captured alpha map is only sub-pixel-accurate at the
captured width and leaves a faint ghost elsewhere. remove_watermark_reverse_alpha
now registers the alpha glyph to the actual mark via a TM_CCOEFF_NORMED
scale+position search (_aligned_alpha_map) before inverting the blend, so the
single 2048 capture works at any resolution -- verified clean on the 1773x2364
(3:4) corpus size, the biggest coverage gap (23 files).
reverse_alpha_available is now just 'asset present' (no width band); the registry
still gates removal on detect so a clean corner is never touched. Drops the
_ALPHA_WIDTH_TOLERANCE gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): keep native recovery exact -- fixed geometry at captured width
Integer-pixel NCC alignment landed ~1px off at the captured width, degrading the
otherwise-exact native reverse-alpha (synthetic recovery error 0.94 -> 1.39).
remove_watermark_reverse_alpha now uses exact width-relative geometry within
_ALPHA_NATIVE_BAND of the captured width and the NCC search only off it -- best
of both: native back to 0.94, other resolutions still aligned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): harden alignment -- try fixed+aligned, keep least residual (56/56)
On a faint/busy-background mark the NCC alignment peak can wander a few px off
the true mark and leave a residual (2/56 real corpus files). Off the captured
width, remove_watermark_reverse_alpha now builds BOTH the fixed-geometry and the
NCC-aligned alpha map, applies each, and keeps whichever leaves the least
residual mark (re-detect confidence on the bare reverse-alpha) -- geometry wins
on faint marks, alignment on clear ones, no magic threshold. Real-file round-trip
now removes 56/56 detected Doubao clean across every corpus resolution (was 54).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(doubao): skip residual inpaint at native width for exact recovery
At the captured width the fixed-geometry reverse-alpha is pixel-exact, so
inpainting over it only replaced exactly-recovered interior pixels with a
cv2 hallucination -- measured worse on a textured background (native error
vs true bg 1.6 reverse-alpha-only vs 2.6 with the old always-on
full-footprint inpaint). Native now returns the bare recovery untouched;
off-native, where NCC alignment is only sub-pixel-approximate, the footprint
inpaint stays to clean the seam. Real round-trip still 56/56 across all
corpus resolutions; negatives 0/60, Gemini unaffected.
Add test_native_returns_exact_reverse_alpha_no_inpaint as the regression
guard. Sync CLAUDE.md + README (the table cell and prose described the
pre-NCC "skipped off native / cv2-LaMa" behavior, now stale). Gitignore the
session scheduled_tasks.lock, and add the text-protection research note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(device): support xpu backend
* Fall back to CPU seed generator when device RNG unsupported (xpu)
Some torch-xpu builds have no device-side RNG, so torch.Generator(device="xpu")
raises when --seed is used. _make_seed_generator tries the device generator and
falls back to a backend-agnostic CPU generator. Adds a fallback unit test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Victor Kuznetsov <kuznetsov.va@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make `pyright src/` strict-clean via a hybrid: pure-logic files are fully typed
(piexif gets a local typings/ stub; PIL info-dict loops guard isinstance(key, str);
progress returns Callable[..., None]; availability checks use importlib.util.find_spec
instead of unused imports), while the irreducibly-untyped cv2/torch/diffusers boundary
files carry a documented per-file `# pyright:` relax pragma (or a ctrlregen
executionEnvironment) that disables only the unknown-type rules. Public ndarray-returning
signatures on the relaxed engines are annotated NDArray[Any] so strict consumers (cli.py)
stay clean.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mirror protect_faces: protect_text defaults to True in invisible_engine and
watermark_remover, so the SDXL pipeline detects text per image and switches to
Differential Diffusion only when glyphs are found. Text-free inputs fall back to
plain img2img with no differential-pipeline load, so the autonomy is free. The
CLI now exposes a single off-switch --no-protect-text instead of the positive
flag, keeping the interface minimal.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
SDXL img2img regenerates every pixel, so small text and CJK glyphs deform
at the strengths that defeat SynthID (issue #21). With --protect-text a
CJK-native PP-OCRv3 detector (2.4 MB ONNX, cv2.dnn, no torch, cached on
first use) locates text regions and the pass switches to the SDXL
Differential-Diffusion community pipeline: a per-pixel change map keeps
text regions largely intact while the background is regenerated to strip
the watermark. Gated to the SDXL default model; falls back to plain
img2img with a warning when unavailable.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
HEIF/AVIF store XMP as a meta-box `mime` item whose bytes live in mdat/idat, out
of reach of the top-level uuid/jumb box stripper. An AI-label XMP packet there
(TC260 AIGC, IPTC "Made with AI", IPTC 2025.1) was therefore left in place.
isobmff.blank_ai_xmp_packets locates each XMP packet by its <?xpacket begin ...
end?> delimiters and, if it carries an AI marker (_AI_LABEL_MARKERS), overwrites
it with spaces of the SAME length. Equal length means no box size or iloc offset
shifts -- the coded image stays bit-for-bit intact, the item stays structurally
valid, only the AI label content is destroyed. Plain (non-AI) XMP is left alone,
mirroring the top-level XMP-uuid content match. Wired into remove_ai_metadata's
ISOBMFF branch after strip_c2pa_boxes.
Chosen over exiftool (a non-bundled binary dep) to stay pure-Python and
droplet-compatible; over full iinf/iloc surgery to avoid offset-rewrite
corruption risk. The AI labels we target are all XMP, so this closes the
practical gap. An Exif *item* inside the meta box (rare) still needs iinf/iloc
surgery or exiftool -- documented.
4 new tests (TestMetaBoxXmpBlanking): AI packet blanked (same length, marker
gone, surrounding image bytes intact), plain XMP preserved, no-packet no-op, and
end-to-end remove_ai_metadata on a .heic.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Provenance detection no longer relies on a fixed first-MB read. In a streaming /
non-faststart MP4 the C2PA manifest sits AFTER a multi-megabyte mdat, beyond the
1 MB scan window, so it was missed.
- isobmff.scan_c2pa_region(path): a file-seeking top-level box walker that
returns the payloads of uuid/jumb (provenance) boxes, seeking past mdat by
size without reading it -- works on multi-GB files. Returns b"" for
non-ISOBMFF or on read error. Mirrors the box-size encoding of the existing
in-memory _iter_top_level_boxes (largesize / size==0).
- metadata.scan_head(path, size): the shared input for every C2PA/AIGC/IPTC
byte scan -- first __TEXT __DATA __OBJC others dec hex bytes plus, for ISOBMFF, the late provenance-box
payloads. Behavior-neutral (f.read(size)) for non-ISOBMFF inputs.
- Routed all six metadata scan sites (has_ai_metadata, aigc_label,
iptc_ai_system, synthid_source, exif_generator XMP, get_ai_metadata
soft-binding) and identify's head read through scan_head.
6 new tests: late box found by scan_c2pa_region / scan_head, the fixed window
provably misses it, non-ISOBMFF -> b"", front-placed (faststart) regression.
The remaining gap stays documented: EXIF/XMP stored as items inside the meta
box (AVIF/HEIF stills) still needs meta-box surgery or exiftool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
