Corpus-mined vendor gaps (42k-file metadata scan, 2026-07-23):
- Bria AI signs C2PA as "Bria Artificial Intelligence" with source type
empty (no trainedAlgorithmicMedia), so identify was completely blind;
registered with asserts_ai like Dreamina.
- fal.ai ("fal - Features & Labels Inc.", fal-ai/<model> generators)
was detected via the source type but never attributed; registered,
also asserts_ai as a pure generative platform.
- Apple Photos Clean Up (Apple Intelligence object removal) was detected
as a generic made-with-AI tag; now attributed as an AI edit via the
photoshop:Credit marker, and the credit value joins
AI_GENERATOR_TOKENS so removal strips it in parity.
Each new test was verified red without its fix.
The invisible/SynthID diffusion pass loads the whole SDXL fp16 pipeline
into VRAM via `pipeline.to("cuda")`. On an 8 GB card the weights alone
(~7 GB) leave no room for activations, so the run OOMs and there is no
in-tool way to recover short of falling back to CPU (~9 min/image).
Add an opt-in `--cpu-offload` flag (default off) that calls diffusers'
`enable_model_cpu_offload()` instead: submodules are streamed to the GPU
on demand, dropping peak VRAM to roughly the largest single submodule at
the cost of per-step transfers. CUDA-only; a no-op on cpu/mps. Threaded
through `invisible`, `all`, and `batch` to keep the knob set identical
across the three, mirroring the existing `--device`/`--pipeline` options.
Measured on a GTX 1070 Ti (8 GB): `invisible --pipeline sdxl --cpu-offload`
runs the SynthID scrub on-GPU in ~2.5 min vs ~9 min on CPU, where the
default full-VRAM path OOMs.
Test drives the placement decision with a mock pipeline (no model/GPU),
gated on torch so it runs under the `gpu` extra and skips the core CI
matrix, consistent with the model-running test policy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New engines, each calibrated on its TC260 USCC cohort and validated by a
full-corpus sweep (42009 files):
- runninghub: top-left corner (new corner="tl"), faint mid-gray text via
the new raw-grayscale "gray" detection front-end, anchor-position gate
- baidu: text-run-only template (pill is a bright-blob magnet), load-bearing
Doubao+Qwen rival margins, corner-extended footprint for the white tag
- liblib: bottom-center (new corner="bc"), Arial silhouette (font is the
discriminative lever against latin UI text), logo-extended footprint
Qingyan parked (no clean-arm separation at any render/box), MiniMax/Hailuo
parked (1 visible frame, the xinghui rule); silhouettes kept as starting
points.
Kling (USCC cohort 91110108335469089C, n=30): kling_engine.py, gate 0.35
(clean p99 0.304 / max 0.320), strict-only, unimodal 0.12/short on the
shared ladder, fitted locate box, no rival margin (crossfire 1/400 doubao
below gate, 0 jimeng, 0 clean), parity 9/9 detect->fill->re-detect.
Suppresses the jimeng pill like doubao/qwen. identify gains visible_kling.
Yuanbao: measured negative -- the two-line italic block does not separate
from clean corners on either front-end at any render/box/font setting;
the fitted recipe stays in render_vendor_silhouettes.py MARK_OPTS.
cat-logo: cohort has only 2 unique carriers, parked on evidence; the
draw_catlogo silhouette already separates (0.50 vs clean max 0.333), so
registration is a gate pick once more uniques arrive.
vendor_mark_calibrate: --fit-geometry takes locate-box overrides (two-line
marks were clipped by the inherited box) and the aspect sweep reaches 0.62.
Calibrated on the 117-frame TC260-producer cohort (vendor_cohort_harvest +
vendor_mark_calibrate, both committed here): per-mark 2-rung ladder
(0.78, 1.27) for the two measured size modes, fitted locate box (the mark
sits ~0.025 of the short side off the edge; doubao's box clipped the first
glyph), measured template aspect 0.26, gate 0.45 (clean p99 0.301).
Strict-only (the sub-gate band is non-Qwen banners), no rival margin
(0 cross-fires on 400 doubao / 298 jimeng / 286 clean frames).
83/83 real marks detector-clean after cv2 fill.
TextMarkConfig gains a per-mark ladder field; the shipped 3-rung default
is unchanged for every other mark.
The Tier E adversarial sweep (new, scripts/robustness_suite.py) drove the real CLI
over truncated, corrupt, zero-byte, absurdly-shaped and bomb inputs, unicode and
RTL paths, hostile output directories and concurrent runs. It found two crashes;
the /simplify review then reproduced a third and worse one.
1. A FAILED WRITE CRASHED ON THE SIZE REPORT. image_io.imwrite is contractually
non-raising and returns False, but write_bgr_with_alpha discarded that bool and
returned None, so no caller could tell a failed write from a successful one.
Every write site then ran output.stat() to print the size, so a read-only
destination died with a bare FileNotFoundError pointing at the stat rather than
the write. The fix is deliberately NOT uniform: single-image commands exit via
the new cli._write_output_or_exit; api._write_visible_result RAISES so a library
caller gets an accurate error instead of a confusing FileNotFoundError from the
downstream metadata strip; and the batch sites raise but never SystemExit,
because the batch loop counts per-image exceptions and aborting would kill the
whole run.
2. BATCH LOST DATA SILENTLY. Into a read-only output directory it wrote ZERO files
for 2 inputs and exited 0 -- no traceback, no error, an empty output directory a
wrapping service would read as a completed run. The robustness harness could not
see this class at all, since it scored exit codes and traceback markers and this
failure has neither; it now asserts on the artifacts written.
3. A DIRECTORY PASSED AS THE IMAGE crashed the metadata scanner with
IsADirectoryError, because click.Path(exists=True) accepts directories. Fixed
with dir_okay=False on all six source arguments, so argument parsing refuses it.
Also adds Tier B4 (scripts/resource_ceilings.py): peak RSS per fill backend from
1 MP to 25 MP, one fresh process per cell. migan 603->775 MB and lama 4679->4779 MB
are flat in input size, confirming the crop-around-the-mask design and both
documented figures; cv2 is the only backend that grows (74->440 MB, 5.9x). The
harness's own no-op check originally allocated a full-frame temp before reading
peak RSS and inflated the numbers with input size -- it now compares only the mask
box, and the conclusion survived re-measurement.
And scripts/real_examples_e2e.py, which drives every command over real corpus
examples and checks the outcome rather than the exit code: 6/6 provenance classes
identified, 10/10 metadata strips re-scan clean, all three fill backends write,
diffusion on MPS writes genuinely changed images. It records samsung as a real
partial (the faintest mark, 0.431 -> 0.404 against a 0.40 gate on the weakest of
its 3 corpus positives) and treats the gated pill's refusal to act as correct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The faint-mask fallback added for the tophat front-end thresholded the
max-normalized uint8 response at 0.5 -- which selects every non-zero pixel,
not "half the peak" as its comment claimed -- and filled ~120% of the corner
box on textured frames. Measured on 14 real faint-path frames (cv2 fill,
detector re-run after): the detector's own best-match box fills a 58.7%-median
corner box vs 120.9% for the threshold, both 100% detector-clean. Detection and
the mask now read one method, _tophat_best, whose score gates detection and
whose argmax box bounds the fill, so the two cannot drift by construction --
which is how the mismatch arose. The 0.5 constant is deleted.
Parity could not catch this (a mask that fills everything is trivially
detector-clean) and the regression test could not either: its flat fixture
gives every threshold the same box, so mutating the constant to 99.0 stayed
green. The fixture now carries texture and asserts the mask area is bounded,
not merely non-empty; it reproduces the corpus number (127% pre-fix).
Also lands the Tier B2 verification harnesses that found and bounded this:
- detector_response.py: response curves (detected AND maskable per cell); found
the size response is a comb, contrast is near-irrelevant, no unmaskable cells.
- ladder_headroom.py: measured that a denser scale ladder recovers 7.6% of
misses for a 2.52%->3.05% false-fire rise, and the one landscape rung that
helps is a geometry shift that helps and hurts equally (1.7:1) -- do not add.
- cjk_tail_probe.py: a generic shared-tail (AI生成) template does not separate
uncovered vendors from clean corners (0.407 vs clean p99 0.298).
Records the visible-parity re-run confirming the earlier front-end fix (doubao
91.8% -> 99.3%), and dedups the thrice-written stamp forward model into one
fill_quality.composite.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The SDXL removal pipelines (sdxl + controlnet) were built without
add_watermarker=False, so diffusers embedded its default open "Stable
Diffusion XL" DWT-DCT invisible watermark on every output whenever
invisible-watermark is installed (the detect extra). A watermark REMOVER was
therefore replacing one detectable AI watermark (SynthID) with another: the
cleaned output re-read as AI (identify -> "Open invisible watermark: Stable
Diffusion XL"), observed on the SynthID validation sample.
Both SDXL loaders now call a shared _disable_sdxl_watermarker helper (mirrors
_maybe_add_fp16_vae; the ControlNetModel sub-model and the Qwen loader never
call it, since only the pipeline accepts the kwarg). Verified end to end: the
affected outputs re-run clean (is_ai=None, no open watermark).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
remove_ai_metadata chose its save format (and the lossless-JPEG fast path)
from the OUTPUT file extension. On the ~2% of real uploads whose extension
lies about their content (a PNG served as .jpg is the common case, ~0.9% of
the corpus), the default flow -- which inherits the source's own extension --
re-encoded a lossless PNG/WebP into a real JPEG, silently degrading the pixels
and breaking the "work with originals" invariant.
Sniff the actual container from magic bytes (_sniff_image_format, reusing the
12-byte head already read for the ISOBMFF check) and route on content: a
misnamed lossless source (source-extension format != content) is preserved in
its true format, while a correctly-named source still honors a deliberate
output-extension conversion (source.png -> output.jpg). The JPEG-lossless gate
is likewise content-gated.
Found by a new metadata-removal parity audit over the local corpus
(scripts/metadata_removal_audit.py): 18170/18173 carriers strip cleanly, and
this fix takes the 208 pixel-integrity failures (all misnamed PNGs) to 0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Full-corpus strip audit surfaced 24 china_aigc survivors on real uploads:
- 19 JPEG carried the bare AIGC{...} blob in APP11 (0xEB). That marker's branch
in _jpeg_app_carries_ai only tested for a C2PA/JUMBF manifest and RETURNED, so
a bare AIGC there slipped past the generic AIGC check. The specific
C2PA(APP11)/XMP(APP1)/IPTC(APP13) checks now fall through to the generic
_is_aigc_exif_value drop, which runs for every APP marker they did not claim.
- PNG carried the {"AIGC":{...}} block in a STANDARD text chunk (Description).
_is_ai_key keeps that key, so removal now also drops a text value carrying an
AIGC block (_is_aigc_exif_value broadened to accept str; wired into the PNG
re-save value filter), in parity with aigc_label's detection.
Verified on the corpus: decodable china_aigc survivors 27 -> 0; the 3 remaining
are truncated files the strip fail-safe (v0.15.1) copies through by design.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PIL raises OSError decoding a truncated file, which crashed remove_ai_metadata
(the PNG/WebP PIL re-save path) -- a direct library caller like a web worker
500s on a partial upload. ~0.2% of the real upload corpus is truncated. The
strip now probes decodability first and, on failure, copies the input through
unchanged and returns rather than raising (we cannot strip what we cannot parse),
mirroring strip_c2pa_boxes' fail-safe. identify already handled these.
The CLI `metadata --remove` on an unreadable file therefore now exits 0 with the
input passed through, not a clean error (exit 1) -- `visible`, which must decode
to remove a mark, still exits 1. Test updated to the per-command contract.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Bug fixes (each with a regression test):
- metadata strip parity across every marker placement: IPTC digitalSourceType
in XMP, the Samsung post-EOI trailer, the China TC260 AIGC block in EXIF
UserComment, a bare AIGC block in a non-standard APP segment, and the ISOBMFF
EXIF path (AIGC + xAI) are all now stripped -- anything a scanner flags, the
strip reaches
- Samsung genAIType detected when its trailer sits past the 512 KB scan window
(file-tail read on large photos)
- crashes on edge inputs: Gemini detector on images with a short side < 16px,
footprint_mask on a zero-size ndarray, the humanizer on chromatic_shift >=
width, and the CLI on unreadable/corrupt/empty input (clean error, not a
traceback)
- WebP written losslessly (cv2 quality 101), not lossy at 100
- the IPTC digitalSourceType algorithmicMedia (procedural, not trained on
sampled data) is no longer flagged as AI-generated, so clean procedural
content is not scrubbed
- c2pa source-type: compositeWithTrainedAlgorithmicMedia is checked before the
bare algorithmicMedia token, so an AI-enhanced composite is not misclassified
Detection:
- integrity-clash coverage now normalizes ByteDance / Canva / ElevenLabs /
Black Forest Labs, so a transplanted manifest next to an independent
conflicting stamp is caught; the generic China TC260 AIGC label is attributed
to a co-present TC260 vendor, so a legit Doubao image (its own C2PA + TC260
label) does not clash (corpus-validated: 0 new clashes on 5000 carriers)
CLI:
- batch exits non-zero (with a warning) when any image errors or a GPU-missing
SynthID scrub is skipped, and copies the input through so the output dir stays
complete -- it used to always exit 0 and could silently drop files
Perf:
- GeminiEngine reused as a process-wide singleton with a precomputed template
ladder: -24% on the identify sparkle path, detection byte-identical
Internal: one shared _ai_exif_targets rule set feeds both EXIF scrubbers so
their coverage cannot drift; docs synced; maintain.sh hardened so the uv-secure
internal teardown crash no longer aborts the gate (still fails on a real finding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
erase_migan fed the whole frame to the ONNX model, so peak RSS scaled with the
upload (~0.6 GB at 4 MP up to ~2.4 GB at 25 MP). Mirror erase_lama: crop a padded
region around the mask (pad = max(256, 2*bbox)), feed only that crop (at native
resolution -- MI-GAN accepts arbitrary dims, unlike LaMa's fixed 512 square), and
paste only masked pixels back. Peak RSS is now bounded by the mark size
(~0.6-0.9 GB), so a memory-tight host (a 1-2 GB web worker) can run MI-GAN on a
25 MP upload.
Fill quality is unchanged: verified by eye on real Gemini/Doubao marks plus a
ground-truth reconstruction sweep -- a tighter view if anything reduces the GAN's
hallucination of large background structure.
Extract the shared padded-crop-box math into _padded_crop_box (used by both
erase_lama and erase_migan).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Victor Kuznetsov