Train now dedups by sha256 (earliest date wins), splits by hash group,
and drops post-cutoff rows whose hash was seen in training, so neither
holdout reports memorized duplicates. The feature schema is explicit
(v1 = the original 97 structural features, v2 adds CFA peaks, DCT AC
histograms and JPEG quant/Huffman/scan stats), stored in the bundle and
read back at scoring time; legacy bundles default to v1. Vectors are
fixed-width with NaN padding, so a sparse record no longer shifts every
column. Scoring runs in batches instead of one predict_proba per record.
Co-Authored-By: Claude <noreply@anthropic.com>
Corpus-mined vendor gaps (42k-file metadata scan, 2026-07-23):
- Bria AI signs C2PA as "Bria Artificial Intelligence" with source type
empty (no trainedAlgorithmicMedia), so identify was completely blind;
registered with asserts_ai like Dreamina.
- fal.ai ("fal - Features & Labels Inc.", fal-ai/<model> generators)
was detected via the source type but never attributed; registered,
also asserts_ai as a pure generative platform.
- Apple Photos Clean Up (Apple Intelligence object removal) was detected
as a generic made-with-AI tag; now attributed as an AI edit via the
photoshop:Credit marker, and the credit value joins
AI_GENERATOR_TOKENS so removal strips it in parity.
Each new test was verified red without its fix.
The invisible/SynthID diffusion pass loads the whole SDXL fp16 pipeline
into VRAM via `pipeline.to("cuda")`. On an 8 GB card the weights alone
(~7 GB) leave no room for activations, so the run OOMs and there is no
in-tool way to recover short of falling back to CPU (~9 min/image).
Add an opt-in `--cpu-offload` flag (default off) that calls diffusers'
`enable_model_cpu_offload()` instead: submodules are streamed to the GPU
on demand, dropping peak VRAM to roughly the largest single submodule at
the cost of per-step transfers. CUDA-only; a no-op on cpu/mps. Threaded
through `invisible`, `all`, and `batch` to keep the knob set identical
across the three, mirroring the existing `--device`/`--pipeline` options.
Measured on a GTX 1070 Ti (8 GB): `invisible --pipeline sdxl --cpu-offload`
runs the SynthID scrub on-GPU in ~2.5 min vs ~9 min on CPU, where the
default full-VRAM path OOMs.
Test drives the placement decision with a mock pipeline (no model/GPU),
gated on torch so it runs under the `gpu` extra and skips the core CI
matrix, consistent with the model-running test policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New engines, each calibrated on its TC260 USCC cohort and validated by a
full-corpus sweep (42009 files):
- runninghub: top-left corner (new corner="tl"), faint mid-gray text via
the new raw-grayscale "gray" detection front-end, anchor-position gate
- baidu: text-run-only template (pill is a bright-blob magnet), load-bearing
Doubao+Qwen rival margins, corner-extended footprint for the white tag
- liblib: bottom-center (new corner="bc"), Arial silhouette (font is the
discriminative lever against latin UI text), logo-extended footprint
Qingyan parked (no clean-arm separation at any render/box), MiniMax/Hailuo
parked (1 visible frame, the xinghui rule); silhouettes kept as starting
points.
Kling (USCC cohort 91110108335469089C, n=30): kling_engine.py, gate 0.35
(clean p99 0.304 / max 0.320), strict-only, unimodal 0.12/short on the
shared ladder, fitted locate box, no rival margin (crossfire 1/400 doubao
below gate, 0 jimeng, 0 clean), parity 9/9 detect->fill->re-detect.
Suppresses the jimeng pill like doubao/qwen. identify gains visible_kling.
Yuanbao: measured negative -- the two-line italic block does not separate
from clean corners on either front-end at any render/box/font setting;
the fitted recipe stays in render_vendor_silhouettes.py MARK_OPTS.
cat-logo: cohort has only 2 unique carriers, parked on evidence; the
draw_catlogo silhouette already separates (0.50 vs clean max 0.333), so
registration is a gate pick once more uniques arrive.
vendor_mark_calibrate: --fit-geometry takes locate-box overrides (two-line
marks were clipped by the inherited box) and the aspect sweep reaches 0.62.
Calibrated on the 117-frame TC260-producer cohort (vendor_cohort_harvest +
vendor_mark_calibrate, both committed here): per-mark 2-rung ladder
(0.78, 1.27) for the two measured size modes, fitted locate box (the mark
sits ~0.025 of the short side off the edge; doubao's box clipped the first
glyph), measured template aspect 0.26, gate 0.45 (clean p99 0.301).
Strict-only (the sub-gate band is non-Qwen banners), no rival margin
(0 cross-fires on 400 doubao / 298 jimeng / 286 clean frames).
83/83 real marks detector-clean after cv2 fill.
TextMarkConfig gains a per-mark ladder field; the shipped 3-rung default
is unchanged for every other mark.
The Tier E adversarial sweep (new, scripts/robustness_suite.py) drove the real CLI
over truncated, corrupt, zero-byte, absurdly-shaped and bomb inputs, unicode and
RTL paths, hostile output directories and concurrent runs. It found two crashes;
the /simplify review then reproduced a third and worse one.
1. A FAILED WRITE CRASHED ON THE SIZE REPORT. image_io.imwrite is contractually
non-raising and returns False, but write_bgr_with_alpha discarded that bool and
returned None, so no caller could tell a failed write from a successful one.
Every write site then ran output.stat() to print the size, so a read-only
destination died with a bare FileNotFoundError pointing at the stat rather than
the write. The fix is deliberately NOT uniform: single-image commands exit via
the new cli._write_output_or_exit; api._write_visible_result RAISES so a library
caller gets an accurate error instead of a confusing FileNotFoundError from the
downstream metadata strip; and the batch sites raise but never SystemExit,
because the batch loop counts per-image exceptions and aborting would kill the
whole run.
2. BATCH LOST DATA SILENTLY. Into a read-only output directory it wrote ZERO files
for 2 inputs and exited 0 -- no traceback, no error, an empty output directory a
wrapping service would read as a completed run. The robustness harness could not
see this class at all, since it scored exit codes and traceback markers and this
failure has neither; it now asserts on the artifacts written.
3. A DIRECTORY PASSED AS THE IMAGE crashed the metadata scanner with
IsADirectoryError, because click.Path(exists=True) accepts directories. Fixed
with dir_okay=False on all six source arguments, so argument parsing refuses it.
Also adds Tier B4 (scripts/resource_ceilings.py): peak RSS per fill backend from
1 MP to 25 MP, one fresh process per cell. migan 603->775 MB and lama 4679->4779 MB
are flat in input size, confirming the crop-around-the-mask design and both
documented figures; cv2 is the only backend that grows (74->440 MB, 5.9x). The
harness's own no-op check originally allocated a full-frame temp before reading
peak RSS and inflated the numbers with input size -- it now compares only the mask
box, and the conclusion survived re-measurement.
And scripts/real_examples_e2e.py, which drives every command over real corpus
examples and checks the outcome rather than the exit code: 6/6 provenance classes
identified, 10/10 metadata strips re-scan clean, all three fill backends write,
diffusion on MPS writes genuinely changed images. It records samsung as a real
partial (the faintest mark, 0.431 -> 0.404 against a 0.40 gate on the weakest of
its 3 corpus positives) and treats the gated pill's refusal to act as correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The faint-mask fallback added for the tophat front-end thresholded the
max-normalized uint8 response at 0.5 -- which selects every non-zero pixel,
not "half the peak" as its comment claimed -- and filled ~120% of the corner
box on textured frames. Measured on 14 real faint-path frames (cv2 fill,
detector re-run after): the detector's own best-match box fills a 58.7%-median
corner box vs 120.9% for the threshold, both 100% detector-clean. Detection and
the mask now read one method, _tophat_best, whose score gates detection and
whose argmax box bounds the fill, so the two cannot drift by construction --
which is how the mismatch arose. The 0.5 constant is deleted.
Parity could not catch this (a mask that fills everything is trivially
detector-clean) and the regression test could not either: its flat fixture
gives every threshold the same box, so mutating the constant to 99.0 stayed
green. The fixture now carries texture and asserts the mask area is bounded,
not merely non-empty; it reproduces the corpus number (127% pre-fix).
Also lands the Tier B2 verification harnesses that found and bounded this:
- detector_response.py: response curves (detected AND maskable per cell); found
the size response is a comb, contrast is near-irrelevant, no unmaskable cells.
- ladder_headroom.py: measured that a denser scale ladder recovers 7.6% of
misses for a 2.52%->3.05% false-fire rise, and the one landscape rung that
helps is a geometry shift that helps and hurts equally (1.7:1) -- do not add.
- cjk_tail_probe.py: a generic shared-tail (AI生成) template does not separate
uncovered vendors from clean corners (0.407 vs clean p99 0.298).
Records the visible-parity re-run confirming the earlier front-end fix (doubao
91.8% -> 99.3%), and dedups the thrice-written stamp forward model into one
fill_quality.composite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The SDXL removal pipelines (sdxl + controlnet) were built without
add_watermarker=False, so diffusers embedded its default open "Stable
Diffusion XL" DWT-DCT invisible watermark on every output whenever
invisible-watermark is installed (the detect extra). A watermark REMOVER was
therefore replacing one detectable AI watermark (SynthID) with another: the
cleaned output re-read as AI (identify -> "Open invisible watermark: Stable
Diffusion XL"), observed on the SynthID validation sample.
Both SDXL loaders now call a shared _disable_sdxl_watermarker helper (mirrors
_maybe_add_fp16_vae; the ControlNetModel sub-model and the Qwen loader never
call it, since only the pipeline accepts the kwarg). Verified end to end: the
affected outputs re-run clean (is_ai=None, no open watermark).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
remove_ai_metadata chose its save format (and the lossless-JPEG fast path)
from the OUTPUT file extension. On the ~2% of real uploads whose extension
lies about their content (a PNG served as .jpg is the common case, ~0.9% of
the corpus), the default flow -- which inherits the source's own extension --
re-encoded a lossless PNG/WebP into a real JPEG, silently degrading the pixels
and breaking the "work with originals" invariant.
Sniff the actual container from magic bytes (_sniff_image_format, reusing the
12-byte head already read for the ISOBMFF check) and route on content: a
misnamed lossless source (source-extension format != content) is preserved in
its true format, while a correctly-named source still honors a deliberate
output-extension conversion (source.png -> output.jpg). The JPEG-lossless gate
is likewise content-gated.
Found by a new metadata-removal parity audit over the local corpus
(scripts/metadata_removal_audit.py): 18170/18173 carriers strip cleanly, and
this fix takes the 208 pixel-integrity failures (all misnamed PNGs) to 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-corpus strip audit surfaced 24 china_aigc survivors on real uploads:
- 19 JPEG carried the bare AIGC{...} blob in APP11 (0xEB). That marker's branch
in _jpeg_app_carries_ai only tested for a C2PA/JUMBF manifest and RETURNED, so
a bare AIGC there slipped past the generic AIGC check. The specific
C2PA(APP11)/XMP(APP1)/IPTC(APP13) checks now fall through to the generic
_is_aigc_exif_value drop, which runs for every APP marker they did not claim.
- PNG carried the {"AIGC":{...}} block in a STANDARD text chunk (Description).
_is_ai_key keeps that key, so removal now also drops a text value carrying an
AIGC block (_is_aigc_exif_value broadened to accept str; wired into the PNG
re-save value filter), in parity with aigc_label's detection.
Verified on the corpus: decodable china_aigc survivors 27 -> 0; the 3 remaining
are truncated files the strip fail-safe (v0.15.1) copies through by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PIL raises OSError decoding a truncated file, which crashed remove_ai_metadata
(the PNG/WebP PIL re-save path) -- a direct library caller like a web worker
500s on a partial upload. ~0.2% of the real upload corpus is truncated. The
strip now probes decodability first and, on failure, copies the input through
unchanged and returns rather than raising (we cannot strip what we cannot parse),
mirroring strip_c2pa_boxes' fail-safe. identify already handled these.
The CLI `metadata --remove` on an unreadable file therefore now exits 0 with the
input passed through, not a clean error (exit 1) -- `visible`, which must decode
to remove a mark, still exits 1. Test updated to the per-command contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bug fixes (each with a regression test):
- metadata strip parity across every marker placement: IPTC digitalSourceType
in XMP, the Samsung post-EOI trailer, the China TC260 AIGC block in EXIF
UserComment, a bare AIGC block in a non-standard APP segment, and the ISOBMFF
EXIF path (AIGC + xAI) are all now stripped -- anything a scanner flags, the
strip reaches
- Samsung genAIType detected when its trailer sits past the 512 KB scan window
(file-tail read on large photos)
- crashes on edge inputs: Gemini detector on images with a short side < 16px,
footprint_mask on a zero-size ndarray, the humanizer on chromatic_shift >=
width, and the CLI on unreadable/corrupt/empty input (clean error, not a
traceback)
- WebP written losslessly (cv2 quality 101), not lossy at 100
- the IPTC digitalSourceType algorithmicMedia (procedural, not trained on
sampled data) is no longer flagged as AI-generated, so clean procedural
content is not scrubbed
- c2pa source-type: compositeWithTrainedAlgorithmicMedia is checked before the
bare algorithmicMedia token, so an AI-enhanced composite is not misclassified
Detection:
- integrity-clash coverage now normalizes ByteDance / Canva / ElevenLabs /
Black Forest Labs, so a transplanted manifest next to an independent
conflicting stamp is caught; the generic China TC260 AIGC label is attributed
to a co-present TC260 vendor, so a legit Doubao image (its own C2PA + TC260
label) does not clash (corpus-validated: 0 new clashes on 5000 carriers)
CLI:
- batch exits non-zero (with a warning) when any image errors or a GPU-missing
SynthID scrub is skipped, and copies the input through so the output dir stays
complete -- it used to always exit 0 and could silently drop files
Perf:
- GeminiEngine reused as a process-wide singleton with a precomputed template
ladder: -24% on the identify sparkle path, detection byte-identical
Internal: one shared _ai_exif_targets rule set feeds both EXIF scrubbers so
their coverage cannot drift; docs synced; maintain.sh hardened so the uv-secure
internal teardown crash no longer aborts the gate (still fails on a real finding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
erase_migan fed the whole frame to the ONNX model, so peak RSS scaled with the
upload (~0.6 GB at 4 MP up to ~2.4 GB at 25 MP). Mirror erase_lama: crop a padded
region around the mask (pad = max(256, 2*bbox)), feed only that crop (at native
resolution -- MI-GAN accepts arbitrary dims, unlike LaMa's fixed 512 square), and
paste only masked pixels back. Peak RSS is now bounded by the mark size
(~0.6-0.9 GB), so a memory-tight host (a 1-2 GB web worker) can run MI-GAN on a
25 MP upload.
Fill quality is unchanged: verified by eye on real Gemini/Doubao marks plus a
ground-truth reconstruction sweep -- a tighter view if anything reduces the GAN's
hallucination of large background structure.
Extract the shared padded-crop-box math into _padded_crop_box (used by both
erase_lama and erase_migan).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two AI-provenance metadata types mined from the retained corpus that
identify previously read as no-signal:
- Dreamina (ByteDance's international Jimeng brand) signs C2PA as
"Bytedance Pte. Ltd." with a "Dreamina/x.y" claim generator and NO
digitalSourceType, so the generator name is the only AI signal. Add a
C2paAiVendor row with a new asserts_ai flag (identity-AI: presence
asserts AI without trainedAlgorithmicMedia) plus the derived
C2PA_IDENTITY_AI_ORGS view, folded into identify's c2pa_is_ai. Keyed on
the Dreamina generator token, not the "Bytedance Pte" issuer, so non-AI
CapCut edits signed by the same entity stay unattributed. 7/7 corpus
files now attribute to ByteDance.
- Tencent Cloud's TC260 AIGC variant uses a ServiceProvider/ServiceUser
schema (vs the producer-side ContentProducer schema), embedded in EXIF
ImageDescription; add those field names to _TC260_FIELDS so the generic
{"AIGC":{...}} gate accepts it. 11/11 corpus files now flagged.
Test-first: reproducing tests in test_identify.py / test_metadata.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The FP gate demotes a low-gradient match, but a real FAINT sparkle also has soft
edges, so metadata-stripped faint sparkles were dropped. Keep a low-grad match
that is a strong (conf >= 0.52), bright, near-WHITE-core sparkle: a real sparkle
core is white, a clean bright corner that shape-matches (sky/sun) is colored
(_core_saturation). Recovers ~14/20 stripped faint sparkles under the DEFAULT
strict/auto (no metadata, no flag) at ~1.25% clean false-fire (baseline 0.55%);
the ~0.51-scoring bright-background FPs stay demoted (below 0.52).
A learned classifier on the same features measured WORSE than the tuned gate
(tier-1: MLP 86.7% recall vs the gate's 90.8% at equal false-fire), so the
heuristic stays; a patch-CNN with richer features is roadmapped P2 with low
expected value -- the precision/recall wall is fundamental (deep-research +
tier-1 both confirm it).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The auto backend now resolves best-first: LaMa (highest quality, recovers the
textured/structured backgrounds the classical fill smears) > MI-GAN > cv2. Both
learned backends share the same onnxruntime availability check, so auto cannot
tell them apart and always prefers the better one; a memory-tight deployment
that cannot afford LaMa's ~4.7 GB peak pins MI-GAN explicitly via
`--backend migan` / `backend="migan"` (the deployment's call, not the library's).
cv2 stays the no-deps floor and now emits a one-time quality warning when auto
falls back to it, since it smears texture/structure.
Motivated by a v0.12.1 reverse-alpha vs 0.14 localize->fill head-to-head:
reverse-alpha recovered structured backgrounds more cleanly than any inpaint;
LaMa closes most of that gap, MI-GAN can ghost/hallucinate, cv2 is weakest.
doubao/jimeng removal is identical between versions; gemini strict coverage is
4pp lower (all recovered via assume_ai) with cleaner clearance and no
outside-box damage.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Candidate carries only the fields the arbiter reads (key, label,
detected_strict, detected_relaxed, features); location/region/confidence were
vestigial from the removed best_auto_mark max-by-confidence path.
- resolve_backend returns preferred_inpaint_backend() directly (typed Literal)
instead of an identity ternary.
- colour/normalise/behaviour -> US spelling across code comments and docs.
No behavior change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 10 code-review findings on the v0.14.0 localize->fill path, several
release-blocking:
- gemini: build the removal mask from the decision's provenance-aware region
instead of a strict internal re-detect. A relaxed/assume_ai sparkle was
re-demoted by the FP gate into a None mask and reported removed while left in
the image; this also drops the redundant double-detect.
- registry: report a mark removed only when a fill actually happened (remove()
returns a None region for an empty mask), so a no-op is never claimed.
- api/cli: add write_noop so the CLI `visible` no-mark path writes nothing and
cannot clobber a pre-existing -o file (was write-then-unlink -> data loss);
create output.parent; skip the same-file copy (SameFileError on in-place).
- cli: catch the missing migan/lama backend RuntimeError on the visible/all
paths (matches `erase`); route the single-mark relaxation through the shared
resolve_relax instead of an inline copy.
- metadata: keep_standard=False no longer takes the AI-only lossless JPEG
short-circuit (it left standard metadata); defer a malformed-marker JPEG to
the PIL fallback instead of reporting a partial strip as complete.
- invisible: register the HEIF opener before Image.open (HEIC --force) and
RGB-convert before the PNG temp (CMYK JPEG).
- pill: normalize via to_bgr so a 4-channel BGRA array cannot crash cvtColor.
Regression tests for each; docs synced (resolve_relax, write_noop,
best_auto_mark -> detect_marks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Replace reverse-alpha removal with localize -> fill (template-free mask + one
shared cv2/MI-GAN/big-LaMa fill) for every mark; drops the colour-shift / dark-pit
failure modes, version-robust to a moved or re-rendered mark
- Separate perception/decision/action: engines report Candidates, a pure
decide(candidates, Context) arbiter owns all policy (sensitivity + provenance +
pill gate), remove_auto_marks orchestrates -- behavior-preserving (corpus 46/46/92)
- Three orthogonal knobs replace --method: --backend cv2|migan|lama,
--sensitivity auto|strict|assume-ai, provenance (auto from metadata)
- Add high-level api.remove_visible / visible_provenance (lazy top-level re-export);
visible --mark auto delegates to it so CLI and library share ONE path
- Read+write HEIC/AVIF on the pixel path via pillow-heif; imwrite preserves the input
format at max quality (JPEG q100/4:4:4); a no-op copies the original bytes verbatim
- Lossless byte-level JPEG metadata strip (no DCT re-encode); consolidate the two
remove_ai_metadata into one, delete legacy noai/cleaner + best_auto_mark
- Bump 0.13.0 -> 0.14.0
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Verified 0.13.0 pill removal on a 32k real-upload corpus. The metadata-OR-wordmark
gate was only ~1/3 precise: TC260 metadata confirms Jimeng-class provenance, not pill
presence, so the weak edge-NCC detector's false fires (textured ceilings/walls, where
inpaint visibly smears) were admitted whenever metadata was present.
Split into two arms (_keep_pill): the reliable bottom-right wordmark (~94% precise,
survives metadata stripping) removes the pill unrestricted; the metadata-only arm
removes it ONLY when the top-left footprint is flat enough for an invisible inpaint
(PillEngine.footprint_is_flat, median-Sobel <= _FLAT_TEXTURE_MAX). Keeps real
flat-scene pills and harmless flat false fires; leaves the damaging textured false
fires untouched. Corpus: 270 -> 118 removals, ~90 true preserved, damaging FP -> ~0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add the Jimeng-basic top-left "AI生成" pill as a CAPTURE-LESS mark
(pill_engine.py): synthetic-silhouette edge-NCC detect + inpaint-only removal.
Gated in remove_auto_marks: kept only when Jimeng is confirmed (TC260 metadata
OR the bottom-right "★ 即梦AI" wordmark fired -- the wordmark keeps recall on
metadata-STRIPPED uploads) AND Doubao did not fire.
- Add an inpaint-fallback removal path + MI-GAN ONNX backend (migan extra, MIT,
~28 MB / ~1 GB peak -- droplet-friendly) alongside big-LaMa. New
--method auto|reverse-alpha|inpaint (shared across visible/all/batch) and
erase --backend migan; footprint_mask on each engine.
- auto is deterministic: reverse-alpha for capture marks (recovers exact pixels,
lighter -- measured cleaner than MI-GAN on structured backgrounds) and inpaint
only for the capture-less pill.
- --mark auto now removes EVERY detected mark in one pass (remove_auto_marks),
so a Jimeng-basic image's top-left pill AND bottom-right wordmark both clear.
- Bump 0.12.1 -> 0.13.0.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Bright-background photos/renders and a tiny app icon were flagged as
AI-generated by the visible detectors. Two failure modes:
- Gemini sparkle on a bright background (snow+sky photo, white product
render) scored ~0.51. The FP gate only demoted on a low core-ring
brightness margin, which a bright background makes high. Add a gradient
floor (_SPARKLE_FP_GRAD 0.55): a real sparkle is a crisp star (grad
~0.97-1.0), a smooth luminance blob that NCC-matches the diamond is not
(the two FPs measured grad 0.105 / 0.463). The OR is a strict superset
of the old margin-only demotion, so it cannot regress dark/mid (kept by
margin) or white-bg (kept by confidence) real sparkles.
- A 48x48 geometric icon matched the Doubao/Jimeng CJK silhouette at
0.41/0.47 NCC. Purely a small-size artifact (the same icon at >=256px
collapses to ~0.06-0.10). Guard text-mark detection below a 200px short
side (_MIN_DETECT_SHORT_SIDE); real marks ship on full-resolution
renders (smallest captured sample 1086px).
Corpus re-sweep flips only OpenAI content and already-cleaned outputs,
all sub-0.5, so no provenance verdict changes. Add synthetic regression
fixtures for both modes; docs/module-internals.md updated.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
InvisibleEngine loads SDXL/ControlNet in fp16 on CUDA/XPU but called from_pretrained
without variant="fp16", so it read the full fp32 weight files (~7 GB) and downcast in
memory. _load_from_pretrained now passes variant="fp16" when torch_dtype is float16,
reading the half-precision files (~3.5 GB) instead - roughly halving the cold-start
weight read + host->device transfer (a phase-timed Modal run measured weight load as
~half of the ~25s cold start). Falls back to the default weights when a checkpoint ships
no fp16 variant (a custom --model), so the worst case is the prior behavior. fp32
(cpu/mps) and bf16 (qwen) never request the variant.
Tests: TestFp16WeightVariant (variant requested on fp16, fallback on missing, never on
fp32).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mined from the retained corpus 2026-06-22 (open-world EXIF/PNG-text/XMP scan,
minus the registry): three AI image generators that stamp a plain generator
name and no C2PA, so identify read them as no-signal -- and under the P0#5
no-signal skip would have skipped the scrub.
- NovelAI (anime SD): PNG tEXt Software/Source/Title. exif_generator now reads
PNG text chunks (via img.info), not only EXIF/XMP.
- Reve (reve.com): EXIF Software / XMP CreatorTool. Token is the full
"reve.com", not bare "reve" (would false-fire on "forever"/"reverie").
- Aphrodite AI: EXIF Make / Software.
Detection/removal parity: NovelAI stamps an AI-shaped VALUE under a non-AI KEY
(Title/Source), which _is_ai_key alone keeps. New _is_ai_value drops a text
chunk by value-token match on removal, mirroring exif_generator -- else the
cleaned file still read as NovelAI (verified on a real corpus file).
Tests: TestExifGenerator gains NovelAI PNG-text, Reve, Reve-not-overmatched,
Aphrodite, and a NovelAI detect/remove parity regression. Docs synced
(module-internals, watermarking-landscape, CLAUDE.md).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerating pixels removes SynthID / open watermarks but degrades a real
photo, so running it on a clean image is the dominant paid score-0 cause on
no-watermark uploads. Gate invisible/all/batch on identify.has_invisible_target:
when no invisible AI signal is locally detectable and --force is unset, skip the
regeneration. Per-command semantics:
- invisible: write no output, exit EXIT_NO_INVISIBLE_SIGNAL (2)
- all: skip step 2 but keep visible-removed pixels + strip metadata, exit 0
- batch: skip the scrub; copy the input through in invisible mode
A skip never claims the image is clean (a pixel SynthID is undetectable once its
metadata proxy is gone); the message says so and routes to --force. The gate
fails safe (a detector error runs the removal).
has_invisible_target wraps identify(check_visible=False, check_invisible=True)
and returns the new ProvenanceReport.ai_from_metadata field (the confidence==high
union), so the raiw.cc worker can reuse the same gate. Gate placed before engine
construction so the skip path is cheap; shared via cli._should_skip_invisible_scrub.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- watermark_remover: _build_qwen_kwargs now passes explicit height/width (via
_qwen_target_size, floored to /16). Without it QwenImageImg2ImgPipeline defaults to
1024x1024 and silently squishes non-square inputs, distorting the scene and garbling text.
- watermark_profiles: resolve_strength gains a `pipeline` arg + a Qwen strength ladder
(_QWEN_VENDOR_STRENGTH, Gemini 0.25), so `--pipeline qwen` gets its certified floor
automatically; retires the manual "pass --strength 0.25 for Gemini on qwen" workaround.
- fidelity_metrics: replace per-face nearest matching (collided on multi-face images when a
variant dropped a face, corrupting the identity metric) with a collision-free one-to-one
assignment (assign_faces_one_to_one). lapvar/LPIPS were always bbox-anchored and immune.
Regression-guarded by tests/test_fidelity_matching.py.
- docs: record the measured outcomes of the qwen-improvement arc. The Qwen ControlNet
face-fix is CLOSED (no permissive Qwen detail/tile ControlNet exists; canny carries edges,
not skin grain). The `--pipeline auto` router + faces+text mixed dual-pass were prototyped
and DROPPED (controlnet wins faces AND display text: abba CER 0.114 vs qwen 0.379).
Z-Image-Turbo was tried and dropped (same regeneration limits). qwen stays a manual opt-in;
controlnet is the default for everything.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
For AI-enhanced composites (digitalSourceType compositeWithTrainedAlgorithmicMedia,
identify ai_source_kind == "enhanced"; roadmap P1#8): regenerate ONLY the AI
region and preserve the real photo elsewhere, instead of regenerating the whole
frame.
- noai.tiling.feather_region_composite(base, regenerated, box, *, feather): pure,
model-free compositor that blends the regenerated AI box back over the original
with a feathered seam, leaving pixels OUTSIDE the box exactly equal to base.
Fully unit-tested (outside-box exactness, interior == regenerated, hard paste at
feather 0, monotonic seam ramp, dtype/grayscale/clamp/empty-box/shape-mismatch).
- WatermarkRemover.remove_watermark(region=, region_feather=) and the module-level
convenience function thread it through: the remover regenerates (or tiles) the
frame, then composites only the AI box back over the original input. The box is
caller-supplied -- a C2PA composite manifest carries no reliable machine-readable
region, so none is fabricated. The no-model lossless region path stays
region_eraser.erase.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Port the Gemini sparkle dark-pit guard (commit 41f6797) to the shared
TextMarkEngine reverse-alpha base (roadmap P0#8): on a dark or mid-tone
background the captured alpha can over-estimate this image's mark opacity, and
reverse-alpha leaves a darker-than-background glyph ghost instead of recovering
the true pixels. The sparkle-only fix left the text marks unhandled.
_reverse_alpha_oversubtracts predicts the reverse-alpha output PER PIXEL over the
glyph body from the INPUT ((obs - a*logo)/(1-a), the remover's own math); when
the predicted body lands more than _OVERSUB_DARK_MARGIN (25) gray levels below
the local background ring it abandons the reverse-alpha output for the footprint
and inpaints it from the original surroundings (_inpaint_footprint, wider dilate/
radius than the thin residual pass). Predicting per-pixel from the input (not the
produced output, which depends on which placement the remover picked) keeps a
cleanly captured full-strength mark byte-identical -- it predicts back to the
background everywhere, so the guard never trips on it (verified across all three
engines on white/mid/dark/midgray backgrounds).
Regression-guarded by tests/test_text_mark_oversubtraction.py: predicate True on
faint / False on clean, end-to-end no-dark-pit acceptance, clean-mark byte
identity, and textured-background footprint recovery.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retained-corpus mining (2026-06-20) surfaced three provenance gaps; all are
oracle-free and regression-guarded.
- C2PA vendor coverage (roadmap): register Volcano Engine under its Chinese
legal entity 北京火山引擎科技有限公司 (the latin "volcengine" needle misses
those certs) -> normalizes to the same ByteDance platform; register ElevenLabs
("Eleven Labs Inc.", pure generative-AI) as a generator. Document the
deliberate exclusion of TikTok Inc. and PixelBin.io/"Fynd" (provenance/transform
signers, not generators) so they are not re-added.
- AI-generated vs AI-enhanced (roadmap): ProvenanceReport.ai_source_kind splits
the C2PA digital-source-type into "generated" (trainedAlgorithmicMedia) vs
"enhanced" (compositeWithTrainedAlgorithmicMedia) so a caller branches a
full-frame scrub from a region-targeted clean. Parsed once in
noai.c2pa._populate_registry_fields (PNG + any c2pa-python-readable container),
with a raw head-scan fallback in identify for the non-PNG raw-blob path. CLI
verdict reads "AI-generated (fully synthetic)" vs "AI-enhanced (real content
with an AI-composited region)"; surfaced in --json.
- Detect-vs-remove threshold desync (P0#7): identify's sparkle threshold and the
removal arbitration gate were two independent 0.5 constants. Unify them into the
single GEMINI_SPARKLE_TRUST_CONF (identify imports it) so they can never drift.
Lowering the gate to recover faint sub-0.5 sparkles was evaluated and REJECTED:
a real Doubao text mark scores ~0.40-0.42 as a gemini match with a higher
core-ring brightness margin than a genuine faint sparkle, so neither confidence
nor the brightness gate separates them in [0.35, 0.5) -- lowering would trade a
rare miss for false-positive removals on clean images. Regression-guarded by
TestSparkleDetectRemoveAlignment (real demo sparkle at borderline opacities;
identify and best_auto_mark must agree on either side of the line).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
