Tiled diffusion was never provider-oracle calibrated with verified text
restoration: the tiled VAE donor path ran anyway and produced results no
oracle had certified. The combination is now rejected at both the
pipeline and the engine seam (ValueError with the reason), and the CLI
help no longer implies support. The invisible help is generalized and
the metadata container list corrected (MKA/OGA/Opus/AAC).
scripts/contentseal_transforms.py reproduces the deterministic crop,
resize, and JPEG variants of the Content Seal corpus from manifest.csv,
hash-verifying every output; its README gains scripts/README.md context
and new data tests. The corpus README is honest about the one crop the
daily oracle limit left unchecked, and the eval CSVs carry the updated
verdicts. The byte-scan SynthID suppression hoists its soft-binding
lookup so the guard is computed once.
Staged on top of 0.33.1; no version bump in this commit.
A standalone-tag file showed 'platform not specified' while the panel
below priced the Muse Content Seal removal - the platform line lagged
the attribution the seal signal and the strength router already make.
The fallback now attributes 'Meta Muse Image (attributed by the
standalone AI digital-source tag)' when no more specific attribution
applies, so the source line and the priced target agree. Apple Photos
Clean Up keeps its own attribution; the hedge names the attribution
basis, not a detection.
The strength router already bets that a file whose only provenance is
the standalone AI digital-source tag is Meta Muse output (C2PA issuers
win first), and Muse stamps every output with the invisible Content
Seal. identify now surfaces that same bet as evidence: the additive
content_seal signal (medium confidence - an attribution, not a decode,
since no public decoder exists) plus the watermark string, emitted on
exactly the standalone_iptc condition that routes the cohort. Clients
select pixel removal from the signal list instead of parsing caveats,
the way InvisMark is additive over soft_binding. The API's invisible
gate already runs on ai_from_metadata, so all/invisible behavior is
unchanged; has_invisible_target needs no edit.
A manifest that names its own forensic soft-binding algorithm carries
that vendor's mark; the generic watermark-action vendor-token inference
must not add a second, differently-attributed invisible watermark from
the same bytes. Microsoft Designer manifests triggered exactly that:
signed by Microsoft, watermarked by InvisMark, with the generation
agent named "Azure OpenAI ImageGen" - the OpenAI issuer token inside
that service name plus the InvisMark watermarked action satisfied the
OpenAI SynthID-evidence rule, and identify reported one forensic mark
as two paid pixel watermarks.
Three changes, one rule at every inference site (the verdict-scan
comment's own lesson: a rule that lives in only one copy is a rule the
others silently lack):
- c2pa.py structured path: SynthID evidence now scopes to the
signer/generator identity strings only (signature issuer, claim
generator), never the raw chain, and is suppressed entirely when a
soft-binding algorithm is named.
- c2pa.py byte fallback and metadata.py synthid_source: suppressed when
the scan names a soft-binding algorithm.
- identify.py verdict scan: same suppression.
Gemini and ChatGPT originals keep their provenance-asserted SynthID
strings; the Designer regression is pinned by
test_designer_synthid_suppression.py (agent name alone is not the
vendor's provenance, and a named soft binding suppresses the
inference).
Full Meta Muse Image support in the invisible-removal path:
- QWEN_ZIMAGE_META_STRENGTH = 0.1: derived by the standard
worst-boundary-plus-cross-source-spread method over five oracle-bracketed
generations (data/contentseal/manifest.csv)
- Auto mode: vendor_for_strength routes a file whose only provenance is the
standalone AI IPTC trainedAlgorithmicMedia tag onto the meta cohort; C2PA
issuers win first, so Google/OpenAI/Microsoft routing is unchanged. Muse
WebP outputs place the XMP in a tail chunk, so the scan uses the shared
chunk-aware metadata.scan_head rather than a plain head read
- Explicit override: --vendor on invisible/all/batch and
InvisibleOptions.vendor name the cohort on stripped files; naming a cohort
asserts the watermark is present, so the no-signal gate treats it like
--force at both the CLI and API seams
- sdxl-zimage has no measured Meta rung: an explicit meta vendor falls to
the conservative unknown 0.25 rather than inventing one
- identify emits a Content Seal caveat pointing at the removal path
- The legacy visible 'Imagined with AI' mark stays unregistered: a dedicated
sample hunt (newsroom mockups, community posts, press screenshots, dead
imagine.meta.com, broken Wayback captures) found no pixel-verifiable
capture, and the registry rule forbids encoding a corner without one.
erase --region remains its removal path; outcome recorded in the landscape
Co-Authored-By: Claude Fable 4.5 <noreply@anthropic.com>
High-confidence C2PA attribution required signingCredential.trusted, a status code
the reader emits only when a trust anchor list is loaded. None ships, so from 0.27.0
through 0.30.0 the branch was unreachable in production for every vendor: an intact,
cryptographically bound manifest scored the same medium as a fallback parse that
validated nothing, which collapsed the one distinction the official reader exists to
draw. A hand-built info dict stamping that code kept the branch green in the suite.
Confidence now follows the binding. Signer trust and certificate expiry stay visible
as their own dimensions and as caveats, because a trust list that was never
configured is a missing input, not a finding against the credential. Every committed
provenance fixture with a reader result and an intact binding now reaches high
confidence, and test_no_committed_fixture_reports_a_trusted_signer guards the
reachability itself rather than a synthesized status set.
Revocation joins binding and signature failures as disqualifying. It arrives only on
signer_validity, so a check reading the other two returned a confident AI verdict off
a credential the issuer had disowned, with an empty integrity_clashes -- quieter than
a hash mismatch on the same file. Expiry stays non-disqualifying: it does not imply
the signed bytes changed, and a signature genuinely made outside validity already
arrives as claimSignature.outsideValidity.
The rule now lives in one place. _validation_fields maps status codes to the four
dimensions and names the failures that moved one; c2pa_info_has_invalid_credential
maps dimensions to disqualified. The ingredient-reachability walk and the
user-visible reason both consume that path instead of re-classifying raw codes, so
adding this one rule no longer means editing three layers in lockstep.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Reachable c2pa.soft-binding assertions now surface their exact alg and the
bounded printable block value next to the normalized vendor label; a value
without its algorithm stays hidden because it cannot be attributed.
com.microsoft.invismark.1 uses that value as the pixel-watermark identifier
in Microsoft Paint output, so identify names it, metadata carries it, and an
InvisMark soft binding keeps the invisible-removal gate fail-safe even after
C2PA asset binding goes invalid. Content fingerprints still do not trigger
pixel regeneration. Removal is verified against Microsoft's Content
Provenance Detection API, which reports Watermark and C2PA separately; the
protocol and the pixel-identical control requirement are documented.
Implemented in a parallel session; verified, gated, and committed by pi.
pre-commit: 1) maintain.sh - exit 1 on the known uv-secure lightning PYSEC-2026-3624 triage (no fix available, unchanged from 0.29.0); ruff, pyright src/, and 1391 tests passed separately; 2) /simplify - single-pass, clean; 3) docs sync - five docs updated by the author session, no remaining references found; 4) CLAUDE.md - invariants recorded in module-internals, no change needed
The registries are raw substrings and the shortest tokens are four and five bytes
(`Bria`, `Adobe`, `Canva`). Over a megabyte of compressed pixel data such a sequence
turns up by chance: `Bria` matched inside the entropy-coded scan of 4 of 14,707
corpus JPEGs, in none of which the manifest names Bria. The rate is what a four-byte
pattern predicts on that corpus, and the Bria entry asserts AI, so a chance match can
declare an image AI-generated rather than merely mislabel its signer.
`_metadata_region` gives the registry scans the container's metadata: JPEG marker
segments before the coded scan, PNG chunks other than IDAT, both trailers, and
whatever `scan_head` appended past the window. Every other check keeps the full
buffer -- their markers are long and distinctive. A container that does not parse is
returned whole, since dropping real evidence to avoid a chance match is the wrong
trade. `c2pa_marker_in` already refuses a bare `c2pa` substring for this reason;
this is the same defence for the registries.
Verified the way the rules require for a change that MOVES a verdict: over all 48,905
corpus images, exactly one file changed, the one named in advance, from
"C2PA Content Credentials (Bria Artificial Intelligence)" to "(unknown signer)".
Record-path parity is 0 disagreements, down from 75 when this work started.
The audit's own baseline comparison is fixed here too. It compared confidence and
signals only, and so reported "0 changed" for the run whose single intended
correction was a watermark line -- the change it exists to show.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A full-corpus audit of the record path against the file path found 75 of 48,905
images disagreeing, and 74 were one gap: the SynthID byte scan for containers whose
manifest no parser reaches lived in `get_ai_metadata`, an extractor the record path
does not run. The record silently reported no SynthID for images `identify` flagged.
Moving the scan into `identify_from_evidence` fixes it by construction rather than by
copying the rule into a second extractor -- the same shape `soft_binding` already
uses. Its byte checks mirror `metadata.synthid_source` literally instead of reusing
the broader `has_c2pa` / `c2pa_source_kind` derived above, so the file path's answers
do not move: verdicts over a 4,000-image sample are byte-identical.
`scripts/record_parity_audit.py` is the audit itself, now repeatable. It walks a
dataset, judges every image through both seams with the record round-tripped through
JSON, and reports disagreements by field and by signal. The rule in
`.claude/rules/development.md` says to re-run both sides of this seam after changing
either; this is what to run.
Both timing and audit scripts now put the package's OWN `src` on the path. From a
worktree an editable install resolves to the main checkout, so the audit imported a
different tree than the one under test -- the failure the same rules file warns about,
reproduced within an hour of writing it down.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three gaps found while measuring the record path against the file path, each one
a signal the library could not see:
WebP stores `XMP ` after the pixels, so on any WebP above the scan window a fixed
read stops short of the label. `_riff_late_metadata` steps over the coded image to
reach it, the RIFF analogue of the existing PNG and ISOBMFF readers. Three corpus
files hid an IPTC "Made with AI" tag and a C2PA `trainedAlgorithmicMedia` there.
The decoder-backed fallback now covers only what it is actually for -- metadata the
raw bytes do not spell, such as a compressed PNG `zTXt` packet.
A C2PA reader failure returned the same `None` as a file with no manifest, so a
verdict could fall back to the raw byte scan with no trace anywhere. Failures now
log at warning and only genuine ones do: a file without credentials never reaches
that branch, and an unsupported container is demoted to debug through the reader's
own `C2paError.NotSupported`. The first corpus run with it found a truncated PNG.
`scan_dataset.py` never registered the pillow-heif opener it declares as a
dependency, so every HEIC was scanned as unreadable -- no EXIF, and a pixel layer
that was 397 of 406 features NaN instead of 136.
`_riff_late_metadata` caps its total like `isobmff.scan_c2pa_region` does. Clamping
each chunk to the bytes remaining is not enough on its own: one chunk can declare a
length spanning most of the file, and this runs on the memoized verdict path over
images from arbitrary sources.
Also lands `identify_metadata_record` and `ProvenanceReport.to_dict()`, the
one-call entry point and the versioned JSON contract for the record path.
Record-vs-file equality holds over 3,478 corpus images, and the eight files these
fixes recovered still report AI.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The visible-mark path had grown three copies of one ladder sweep, four
near-identical `detect` arms, and four hand-rolled `footprint_mask` overrides;
mark knowledge sat in five hand-maintained tables across three modules; and the
flagship `all`/`batch` pipeline existed only in cli.py, written twice with
divergent behavior.
Detection is now one measurement. `_ladder_best` replaces the three sweeps,
`_scan`/`_verdict` replace the four arms, and the winning box travels to the
mask on `TextMarkDetection.match_box` instead of being swept a second time.
`detect_both` returns the strict and relaxed verdicts from one scan, which
halves the arbiter's perception cost (260 -> 130 matchTemplate calls on a 2048²
image, verdicts identical field for field). A per-mark demotion goes in the new
`_post_gate` hook, never in a `detect` override -- an override is invisible to
the single-pass path, which is how the RunningHub and Yuanbao anchor gates
briefly stopped applying.
Everything about a mark is now one registry row: product, label regime, the
platform sentence `identify` reports, the metadata signals that confirm it, and
its TC260 producer codes. `identify._VISIBLE_MARK_PLATFORM`, the signal mapping
in `api.visible_provenance`, `_PRODUCT_OF` and the pill veto are derived from
those rows.
`api.remove_all` / `api.remove_batch` are the library form of the `all` and
`batch` commands; the CLI is a wrapper that owns console text and exit codes.
Progress is a `(stage, detail)` pair of stable tokens, so the CLI keys its
wording off structure rather than parsing the library's prose back.
Two intentional behavior changes, both verified against a recorded 811-image
sample of detector verdicts, removal-mask hashes, arbiter decisions and
`identify` reports:
* A TC260 label now relaxes the vendor its `ContentProducer` names rather than
ByteDance's pair on every China-AIGC image. 333 of 811 samples move; on 185
of them the previously relaxed pair was simply the wrong vendor, and the
mark actually present never reached the relaxed gate its own
`provenance_ncc_factor` was calibrated for.
* A confident LibLibAI detection suppresses the Jimeng pill, like every other
TC260 product's mark. It was registered alongside RunningHub and Baidu, both
of which were added to the hand-written veto list, and it was not. 1 sample
moves, and it is exactly the co-firing case.
Nothing else in that record changes: detector verdicts, mask hashes and
`identify` verdicts are byte-identical, and all 200 calibration constants are
untouched.
Also: `aigc_label` and friends plus `extract_c2pa_info` are memoized on
(path, mtime_ns, size) -- size because this package rewrites in place; the
native TC260 container readers route on magic bytes instead of the file
extension, so a mislabeled AVI or FLV is no longer invisible; `identify` shares
one pixel decode between the DWT-DCT and visible stages (TrustMark keeps its own
Pillow decode, which is not substitutable); and the six `stabilize_*` video
wrappers collapse into one policy table.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Corpus-mined vendor gaps (42k-file metadata scan, 2026-07-23):
- Bria AI signs C2PA as "Bria Artificial Intelligence" with source type
empty (no trainedAlgorithmicMedia), so identify was completely blind;
registered with asserts_ai like Dreamina.
- fal.ai ("fal - Features & Labels Inc.", fal-ai/<model> generators)
was detected via the source type but never attributed; registered,
also asserts_ai as a pure generative platform.
- Apple Photos Clean Up (Apple Intelligence object removal) was detected
as a generic made-with-AI tag; now attributed as an AI edit via the
photoshop:Credit marker, and the credit value joins
AI_GENERATOR_TOKENS so removal strips it in parity.
Each new test was verified red without its fix.
New engines, each calibrated on its TC260 USCC cohort and validated by a
full-corpus sweep (42009 files):
- runninghub: top-left corner (new corner="tl"), faint mid-gray text via
the new raw-grayscale "gray" detection front-end, anchor-position gate
- baidu: text-run-only template (pill is a bright-blob magnet), load-bearing
Doubao+Qwen rival margins, corner-extended footprint for the white tag
- liblib: bottom-center (new corner="bc"), Arial silhouette (font is the
discriminative lever against latin UI text), logo-extended footprint
Qingyan parked (no clean-arm separation at any render/box), MiniMax/Hailuo
parked (1 visible frame, the xinghui rule); silhouettes kept as starting
points.
Kling (USCC cohort 91110108335469089C, n=30): kling_engine.py, gate 0.35
(clean p99 0.304 / max 0.320), strict-only, unimodal 0.12/short on the
shared ladder, fitted locate box, no rival margin (crossfire 1/400 doubao
below gate, 0 jimeng, 0 clean), parity 9/9 detect->fill->re-detect.
Suppresses the jimeng pill like doubao/qwen. identify gains visible_kling.
Yuanbao: measured negative -- the two-line italic block does not separate
from clean corners on either front-end at any render/box/font setting;
the fitted recipe stays in render_vendor_silhouettes.py MARK_OPTS.
cat-logo: cohort has only 2 unique carriers, parked on evidence; the
draw_catlogo silhouette already separates (0.50 vs clean max 0.333), so
registration is a gate pick once more uniques arrive.
vendor_mark_calibrate: --fit-geometry takes locate-box overrides (two-line
marks were clipped by the inherited box) and the aspect sweep reaches 0.62.
Calibrated on the 117-frame TC260-producer cohort (vendor_cohort_harvest +
vendor_mark_calibrate, both committed here): per-mark 2-rung ladder
(0.78, 1.27) for the two measured size modes, fitted locate box (the mark
sits ~0.025 of the short side off the edge; doubao's box clipped the first
glyph), measured template aspect 0.26, gate 0.45 (clean p99 0.301).
Strict-only (the sub-gate band is non-Qwen banners), no rival margin
(0 cross-fires on 400 doubao / 298 jimeng / 286 clean frames).
83/83 real marks detector-clean after cv2 fill.
TextMarkConfig gains a per-mark ladder field; the shipped 3-rung default
is unchanged for every other mark.
Bug fixes (each with a regression test):
- metadata strip parity across every marker placement: IPTC digitalSourceType
in XMP, the Samsung post-EOI trailer, the China TC260 AIGC block in EXIF
UserComment, a bare AIGC block in a non-standard APP segment, and the ISOBMFF
EXIF path (AIGC + xAI) are all now stripped -- anything a scanner flags, the
strip reaches
- Samsung genAIType detected when its trailer sits past the 512 KB scan window
(file-tail read on large photos)
- crashes on edge inputs: Gemini detector on images with a short side < 16px,
footprint_mask on a zero-size ndarray, the humanizer on chromatic_shift >=
width, and the CLI on unreadable/corrupt/empty input (clean error, not a
traceback)
- WebP written losslessly (cv2 quality 101), not lossy at 100
- the IPTC digitalSourceType algorithmicMedia (procedural, not trained on
sampled data) is no longer flagged as AI-generated, so clean procedural
content is not scrubbed
- c2pa source-type: compositeWithTrainedAlgorithmicMedia is checked before the
bare algorithmicMedia token, so an AI-enhanced composite is not misclassified
Detection:
- integrity-clash coverage now normalizes ByteDance / Canva / ElevenLabs /
Black Forest Labs, so a transplanted manifest next to an independent
conflicting stamp is caught; the generic China TC260 AIGC label is attributed
to a co-present TC260 vendor, so a legit Doubao image (its own C2PA + TC260
label) does not clash (corpus-validated: 0 new clashes on 5000 carriers)
CLI:
- batch exits non-zero (with a warning) when any image errors or a GPU-missing
SynthID scrub is skipped, and copies the input through so the output dir stays
complete -- it used to always exit 0 and could silently drop files
Perf:
- GeminiEngine reused as a process-wide singleton with a precomputed template
ladder: -24% on the identify sparkle path, detection byte-identical
Internal: one shared _ai_exif_targets rule set feeds both EXIF scrubbers so
their coverage cannot drift; docs synced; maintain.sh hardened so the uv-secure
internal teardown crash no longer aborts the gate (still fails on a real finding).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two AI-provenance metadata types mined from the retained corpus that
identify previously read as no-signal:
- Dreamina (ByteDance's international Jimeng brand) signs C2PA as
"Bytedance Pte. Ltd." with a "Dreamina/x.y" claim generator and NO
digitalSourceType, so the generator name is the only AI signal. Add a
C2paAiVendor row with a new asserts_ai flag (identity-AI: presence
asserts AI without trainedAlgorithmicMedia) plus the derived
C2PA_IDENTITY_AI_ORGS view, folded into identify's c2pa_is_ai. Keyed on
the Dreamina generator token, not the "Bytedance Pte" issuer, so non-AI
CapCut edits signed by the same entity stay unattributed. 7/7 corpus
files now attribute to ByteDance.
- Tencent Cloud's TC260 AIGC variant uses a ServiceProvider/ServiceUser
schema (vs the producer-side ContentProducer schema), embedded in EXIF
ImageDescription; add those field names to _TC260_FIELDS so the generic
{"AIGC":{...}} gate accepts it. 11/11 corpus files now flagged.
Test-first: reproducing tests in test_identify.py / test_metadata.py.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Mined from the retained corpus 2026-06-22 (open-world EXIF/PNG-text/XMP scan,
minus the registry): three AI image generators that stamp a plain generator
name and no C2PA, so identify read them as no-signal -- and under the P0#5
no-signal skip would have skipped the scrub.
- NovelAI (anime SD): PNG tEXt Software/Source/Title. exif_generator now reads
PNG text chunks (via img.info), not only EXIF/XMP.
- Reve (reve.com): EXIF Software / XMP CreatorTool. Token is the full
"reve.com", not bare "reve" (would false-fire on "forever"/"reverie").
- Aphrodite AI: EXIF Make / Software.
Detection/removal parity: NovelAI stamps an AI-shaped VALUE under a non-AI KEY
(Title/Source), which _is_ai_key alone keeps. New _is_ai_value drops a text
chunk by value-token match on removal, mirroring exif_generator -- else the
cleaned file still read as NovelAI (verified on a real corpus file).
Tests: TestExifGenerator gains NovelAI PNG-text, Reve, Reve-not-overmatched,
Aphrodite, and a NovelAI detect/remove parity regression. Docs synced
(module-internals, watermarking-landscape, CLAUDE.md).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Regenerating pixels removes SynthID / open watermarks but degrades a real
photo, so running it on a clean image is the dominant paid score-0 cause on
no-watermark uploads. Gate invisible/all/batch on identify.has_invisible_target:
when no invisible AI signal is locally detectable and --force is unset, skip the
regeneration. Per-command semantics:
- invisible: write no output, exit EXIT_NO_INVISIBLE_SIGNAL (2)
- all: skip step 2 but keep visible-removed pixels + strip metadata, exit 0
- batch: skip the scrub; copy the input through in invisible mode
A skip never claims the image is clean (a pixel SynthID is undetectable once its
metadata proxy is gone); the message says so and routes to --force. The gate
fails safe (a detector error runs the removal).
has_invisible_target wraps identify(check_visible=False, check_invisible=True)
and returns the new ProvenanceReport.ai_from_metadata field (the confidence==high
union), so the raiw.cc worker can reuse the same gate. Gate placed before engine
construction so the skip path is cheap; shared via cli._should_skip_invisible_scrub.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retained-corpus mining (2026-06-20) surfaced three provenance gaps; all are
oracle-free and regression-guarded.
- C2PA vendor coverage (roadmap): register Volcano Engine under its Chinese
legal entity 北京火山引擎科技有限公司 (the latin "volcengine" needle misses
those certs) -> normalizes to the same ByteDance platform; register ElevenLabs
("Eleven Labs Inc.", pure generative-AI) as a generator. Document the
deliberate exclusion of TikTok Inc. and PixelBin.io/"Fynd" (provenance/transform
signers, not generators) so they are not re-added.
- AI-generated vs AI-enhanced (roadmap): ProvenanceReport.ai_source_kind splits
the C2PA digital-source-type into "generated" (trainedAlgorithmicMedia) vs
"enhanced" (compositeWithTrainedAlgorithmicMedia) so a caller branches a
full-frame scrub from a region-targeted clean. Parsed once in
noai.c2pa._populate_registry_fields (PNG + any c2pa-python-readable container),
with a raw head-scan fallback in identify for the non-PNG raw-blob path. CLI
verdict reads "AI-generated (fully synthetic)" vs "AI-enhanced (real content
with an AI-composited region)"; surfaced in --json.
- Detect-vs-remove threshold desync (P0#7): identify's sparkle threshold and the
removal arbitration gate were two independent 0.5 constants. Unify them into the
single GEMINI_SPARKLE_TRUST_CONF (identify imports it) so they can never drift.
Lowering the gate to recover faint sub-0.5 sparkles was evaluated and REJECTED:
a real Doubao text mark scores ~0.40-0.42 as a gemini match with a higher
core-ring brightness margin than a genuine faint sparkle, so neither confidence
nor the brightness gate separates them in [0.35, 0.5) -- lowering would trade a
rare miss for false-positive removals on clean images. Regression-guarded by
TestSparkleDetectRemoveAlignment (real demo sparkle at borderline opacities;
identify and best_auto_mark must agree on either side of the line).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
identify(check_visible=True) ran the Gemini-sparkle detector and the
Doubao/Jimeng text-mark detector each with its own image_io.imread, so the
same bitmap was fully decoded twice. On a memory-constrained host (the raiw.cc
512 MB web worker, which runs identify on every upload) that doubled the peak
decode allocation and contributed to OOM restarts.
Decode once in identify() and pass the BGR array to both detectors. The detect
methods already accept an NDArray, so this only threads the pre-decoded array
through: detect_sparkle_confidence and the two _visible_* helpers gain an
optional image= param that, when None, preserves the old self-read behavior
(so direct callers and the cv2-missing/unreadable paths are unchanged).
Only the visible path is deduplicated; the optional check_invisible decoders
are unaffected (and off on the web hot path). Adds a test asserting
identify(check_visible=True, check_invisible=False) decodes exactly once.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- AIGC: parse the bare ``AIGC{...}`` blob form (label glued to its JSON in a
JPEG APP segment near the JFIF header), and scan both raw-JSON forms in one
fall-through loop so a quoted ``"AIGC"`` later in an XMP packet no longer
shadows a real bare label earlier in the file (3 files read unknown before).
- Integrity clash rule 2: a camera device + an AI marker from the SAME C2PA
manifest (Google Pixel Magic Editor / Pixel Studio edit chain) is a legitimate
edit chain, not a contradiction. Fire only when the AI marker's source is
independent of the camera's manifest; pure cameras (Leica/Sony/Nikon) are
unaffected (2 Pixel files mis-flagged before).
- New c2pa_cloud_manifest detector: surface a C2PA 2.4 Durable Content
Credentials cloud-manifest reference (Adobe cai-manifests.adobe.com) as a
medium provenance signal when the embedded manifest is stripped. Provenance
only, never asserts is_ai (2 files read fully unknown before).
identify reuses its already-loaded scan head for the cloud check (no second
read). +7 tests; CLAUDE.md + README synced.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New samsung_engine.py mirrors the jimeng engine but anchors bottom-left; wired
into watermark_registry, the CLI (--mark samsung / auto), and identify
(visible_samsung, medium). visible_alpha_solve.py gains a corner=bl mode;
samsung_alpha.png solved from @f-liva's flat captures. Calibrated for the
Italian "Contenuti generati dall'AI" variant. Flat black/gray/white captures
committed, real photos gitignored. Tests + docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Three P2 cleanups from a library-wide review.
Detection -- single C2PA_AI_VENDORS registry (noai/constants.py):
- C2PA_ISSUERS, SYNTHID_C2PA_ISSUERS, and identify._ISSUER_PLATFORM now derive
from one C2paAiVendor table, so adding a C2PA vendor is one entry instead of
edits in three places across two files. Behavior-identical (262 detection
tests pass; the kept `needle` field is load-bearing -- it differs from `org`
for Google and ByteDance, with no mechanical derivation).
Code-health:
- region_eraser.erase_lama now accepts grayscale/BGRA like erase_cv2 (it
crashed on grayscale and silently dropped alpha on BGRA). +2 regression tests.
- batch frees the device cache between images via a shared try_empty_device_cache
helper (generalized from the MPS-only _try_clear_mps_cache, now reused by both
the MPS->CPU fallback and the batch loop).
- batch gained --controlnet-scale (parity with invisible/all).
CI / packaging:
- publish.yml uploads via `uv publish` (PyPI trusted publishing over OIDC),
replacing pypa/gh-action-pypi-publish so uploads no longer depend on that
action's bundled twine accepting the Metadata-Version. Workflow filename +
pypi environment unchanged, so PyPI's trusted-publisher entry still matches.
- hatchling pin relaxed <1.28 -> <1.31 (verified against hatch's changelog:
1.30.0 made Metadata 2.5 the default, 1.30.1 reverted to 2.4; 1.27-1.29 were
always 2.4). Kept as belt-and-suspenders so the first uv-publish release ships
2.4, isolating the uploader swap from the metadata-version bump.
Docs (CLAUDE.md, pyproject) synced; corrected the inaccurate "hatchling 1.28+
emits 2.5" note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The C2PA issuer attribution (`c2pa`) and the SynthID proxy (`synthid`) are
derived from the same manifest, so treating them as independent signals made
rule 1 fire on legitimate multi-actor manifests where a product wraps another
vendor's engine (Microsoft Designer on OpenAI, Microsoft on Google) or an edit
chain re-signs (Adobe over a Gemini original). 19 such files in the
2026-06-01/02 spaces batches read as "likely spoofed/laundered" before this.
Group `c2pa` + `synthid` into one provenance source via `_CLASH_SOURCE`; rule 1
now requires two vendors from different sources. A manifest vendor still clashes
with a genuinely independent stamp (EXIF/XMP generator, IPTC AISystemUsed, AIGC,
xAI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
identify previously ran only the Gemini sparkle as a visible detector, so a
Doubao/Jimeng image with stripped TC260 metadata had no visible fallback. Add
`_visible_text_marks` (registry-backed) so the ByteDance Doubao 豆包AI生成 and
Jimeng 即梦AI marks are detected too, each gated by its own engine NCC threshold
via MarkDetection.detected. New signals `visible_doubao` / `visible_jimeng`
(medium), same stripped-metadata fallback role as the sparkle; excluded from
integrity-clash vendor claims; set platform only when no harder signal did.
Also make `noai/__init__` lazy (PEP 562 __getattr__): importing the light
`noai.c2pa` / `noai.constants` submodules (which identify needs) no longer
eagerly pulls `watermark_remover`, which imports torch + diffusers at module
top. `import remove_ai_watermarks.identify` drops from ~420 MB to ~21 MB in a
full gpu/detect install (torch not loaded), so it fits a 512 MB host; the
removal API resolves lazily on first access. Guarded by TestIdentifyImportIsLight.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Visible-watermark work across all three corner-mark engines plus a committed,
reproducible alpha-build pipeline (scripts/visible_alpha_solve.py) fed by committed
solid black/gray/white captures.
- jimeng: new "即梦AI" wordmark remover (reverse-alpha + thin residual inpaint,
always NCC-aligned -- the mark re-rasterizes/jitters per image). Detect via glyph
silhouette NCC (0.45 threshold; does not cross-fire with Doubao). Registered in the
visible-mark catalog; `visible --mark jimeng` / `--mark auto`.
- doubao: fix a real production defect -- the shipped remover left a READABLE
"豆包AI生成" outline on real samples while detect() returned conf 0.0 (fooled by a
thin outline), so the test passed and the "56/56 clean" claim was detector-measured,
not visual. Root cause: under-estimated alpha + fixed-geometry-no-inpaint + tight
locate box. Rebuilt alpha (careful gray-self solve), always-align, thin inpaint,
widened locate box -> readable outline becomes faint texture-level traces.
- gemini: rebuild gemini_bg_{96,48} from our own controlled captures (validated NCC
0.9998 vs the prior third-party asset); removal re-verified clean, no behaviour change.
- tests: add textured-shift regression to both engines (guards the align-on-shift path
the Doubao defect exposed; lesson: a detector-only removal test is insufficient,
assert visual residual).
- docs: CLAUDE.md, README, capture READMEs and docstrings synced; stale
"exact/pixel-exact/56-clean" claims removed.
Also includes a SynthID label-wording clarification in identify.py/cli.py
("SynthID pixel watermark" -> "SynthID watermark, inferred from C2PA metadata").
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Apply fixes from a full-repo review (code, tests, docs).
Security / correctness:
- Clamp attacker-controlled PNG/caBX chunk lengths to the remaining file
size in metadata.py and noai/c2pa.py (a malformed length no longer drives
a multi-GB read); skipped chunks seek instead of read.
- noai/isobmff.strip_c2pa_boxes is now fail-safe on a malformed box: return
the original bytes with a warning instead of silently truncating the tail,
so metadata --remove can no longer emit a corrupt file.
- doubao_engine._fixed_alpha_map clamps the glyph box to the image (no crash
on degenerate width-vs-height).
- watermark_remover._run_region_hires gates the phaseCorrelate offset on
response and magnitude (a spurious shift no longer garbles text) and drops
the generator after a CPU fallback (no MPS/CPU device mismatch).
Robustness:
- gemini_engine, doubao_engine, region_eraser normalize grayscale and RGBA
inputs to BGR at the engine entry points.
- image_io.imwrite returns False on an unwritable path (matches cv2).
- invisible_engine guards a None imread result before use.
- trustmark_detector._decoder uses a double-checked threading lock.
- ctrlregen.tiling.tile_positions raises on overlap >= tile.
- humanizer chromatic shift no longer wraps opposite-edge pixels.
- identify OpenAI caveat keyed on the normalized vendor, not a substring.
- Remove the dead "visible --detect-threshold" CLI option.
- publish.yml verifies the release tag matches the package version.
Docs:
- README strength 0.05 to 0.10; .env.example HF_TOKEN marked optional;
doubao_capture README updated to reverse-alpha-only; CLAUDE.md synced with
the new behaviors and the batch command.
Tests: new test_security_clamp.py for the read clamp and isobmff fail-safe;
erase CLI coverage; integrity-clash rule 2 end-to-end; multi-tag EXIF
survival and cross-format strip guards; channel/size, tiling, humanizer, and
imwrite regressions. Full suite 493 passed, 2 skipped; ruff and pyright src/
clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(trustmark): gate detection on re-encode durability to kill false positives
TrustMark's wm_present flag is a BCH validity check that spuriously
validates on a content-correlated fraction of un-watermarked images
(AI textures trip it more than camera photos). On a 1343-image set all
20 raw detections were false, several on Gemini/OpenAI/Doubao output that
cannot carry Adobe's watermark, with random-bytes secrets.
A genuine TrustMark is a durable soft binding that survives re-encoding,
so detect_trustmark now re-decodes after a mild JPEG round-trip and
requires the same schema both times. Every observed false positive
collapsed under this gate; the second decode runs only on the rare hit.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(identify): Samsung Galaxy AI, FLUX, ByteDance C2PA; fix C2PA substring FP
Detection extensions verified on real signed files (2026-05-29):
- Samsung Galaxy AI: signer attribution via a new _SIGNER_C2PA_PLATFORM
(Samsung Galaxy / ASUS Gallery) kept separate from the capture-camera
_DEVICE_C2PA_PLATFORM so a Galaxy AI edit (device cert + AI source type)
does not trip the camera-vs-AI integrity clash. Plus metadata.samsung_genai:
the proprietary genAIType marker in PhotoEditor_Re_Edit_Data, a medium-
confidence AI-editing signal (samsung_only branch).
- Black Forest Labs (FLUX) and ByteDance Volcano Engine (Doubao/Jimeng)
added as C2PA issuers + issuer->platform mappings.
- fix: C2PA presence required only the bare 4-byte 'c2pa' substring, which
false-positives on compressed pixel data (a recompressed PNG IDAT re-flagged
C2PA after its manifest was correctly stripped). New c2pa_marker_in() requires
the JUMBF wrapper (jumb+c2pa) or the C2PA uuid box; applied in identify +
metadata. Verified: all 535 real C2PA files carry jumb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): gate detection on text structure to cut ~95% of false positives (#23)
Coverage alone over-fired: any textured bottom-right corner cleared the
threshold, so the detector false-positived on ~28% of arbitrary images.
The real '豆包AI生成' mark is six glyphs in one row, so detect now also
requires the text-structure signature (_glyph_structure): many connected
components, no single dominant blob, concentration in a thin horizontal
band. False positives dropped 343 -> 17 across the corpus while keeping
real-mark recall and the doubao-1.png sample. Also accept a no-op force
kwarg for remover-interface symmetry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(samsung): add Samsung Galaxy AI visible-badge remover
New samsung_engine.py removes the bottom-left sparkle + localized
'AI-generated content' badge that Galaxy AI tools stamp. Mirrors the
Doubao locate->mask->inpaint pattern but bottom-left, with a dual-polarity
top-hat mask (the badge is light-on-dark or dark-on-light). Detection gates
on a band + left-anchor signature (the Doubao CJK-component gate does not
transfer: Latin badge letters connect into few blobs). Explicit-only --
tuned on few real badges with a ~4% FP floor, so it is not used in auto.
Synthetic byte-blob fixtures (real badges are user content, not shipped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(visible): unified known-watermark registry + LaMa inpaint backend
watermark_registry.py is a single catalog of known visible marks, each
tying {usual location, in_auto flag, recovery strategy, detect adapter,
remove adapter}: gemini (reverse-alpha, exact), doubao, samsung. cmd_visible
is now registry-driven (best_auto_mark for --mark auto; mark_keys() feeds the
CLI choices) -- the per-mark _run_doubao/_run_samsung helper branches are gone.
Cross-engine confidences are not comparable, so the gemini adapter applies the
corpus-validated 0.5 sparkle threshold for auto arbitration (its engine flag is
loose and weakly fired ~0.36 on Doubao text, hijacking auto).
--backend auto|cv2|lama chooses background reconstruction for the mask-based
marks; auto = LaMa when onnxruntime is present, else cv2. For LaMa the mask is
the FILLED glyph bounding box (sparse glyph masks leave anti-aliased edges
behind). cv2 stays the zero-dependency fallback.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs: watermark registry, Samsung/FLUX/ByteDance detection, LaMa backend, trustmark gate
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(doubao): exact reverse-alpha removal from captured alpha map
The Doubao '豆包AI生成' mark is a fixed semi-transparent white overlay, so
given its alpha map the original pixels are recovered exactly:
original = (wm - a*logo)/(1-a) -- no inpaint hallucination.
The alpha map + logo colour were solved from real black+gray Doubao captures
on a controlled background: on black captured = a*logo, and the black/gray pair
solves a per-pixel without assuming the logo colour (a_max~0.65, logo near-white);
the white capture cross-validates (mark vanishes to a flat fill). Bundled as
assets/doubao_alpha.png + geometry constants.
remove_watermark_reverse_alpha applies it scaled to image width; exact at the
captured width, so the registry routes doubao through it only when
reverse_alpha_available (width within the calibrated band) and the mark is
detected, falling back to mask inpaint (cv2/LaMa) otherwise. A light residual
inpaint cleans the sub-pixel rescaling error. Add captures at more resolutions
to widen exact coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor(visible): reverse-alpha only -- drop inpaint removal + heuristic detection
Per the principle that we only remove/detect what we can do exactly, the
visible-mark path is now reverse-alpha only:
- Doubao detect is reverse-alpha-consistent: match the bundled alpha glyph
silhouette against the corner via TM_CCOEFF_NORMED (DETECT_NCC_THRESHOLD 0.4)
-- keys on the '豆包AI生成' SHAPE, not coverage/structure heuristics. FP
7/1243 (0.6%). Removes the cv2 inpaint path + the _glyph_structure gate.
- Registry is reverse-alpha only: dropped the cv2/LaMa backend (_glyph_remove,
_lama_box_inpaint, default_backend, --backend) and the Samsung entry. Doubao
outside the alpha resolution band is skipped, never inpainted.
- Removed samsung_engine.py + tests + --mark samsung (no alpha map captured;
Samsung C2PA/genAIType metadata detection in identify is unaffected).
- The universal erase --region (cv2/LaMa) is unchanged -- arbitrary-region
inpainting stays a user-directed tool, separate from the known-mark registry.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(doubao): NCC sub-pixel alignment -> reverse-alpha at any resolution
A pure width-scale of the captured alpha map is only sub-pixel-accurate at the
captured width and leaves a faint ghost elsewhere. remove_watermark_reverse_alpha
now registers the alpha glyph to the actual mark via a TM_CCOEFF_NORMED
scale+position search (_aligned_alpha_map) before inverting the blend, so the
single 2048 capture works at any resolution -- verified clean on the 1773x2364
(3:4) corpus size, the biggest coverage gap (23 files).
reverse_alpha_available is now just 'asset present' (no width band); the registry
still gates removal on detect so a clean corner is never touched. Drops the
_ALPHA_WIDTH_TOLERANCE gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): keep native recovery exact -- fixed geometry at captured width
Integer-pixel NCC alignment landed ~1px off at the captured width, degrading the
otherwise-exact native reverse-alpha (synthetic recovery error 0.94 -> 1.39).
remove_watermark_reverse_alpha now uses exact width-relative geometry within
_ALPHA_NATIVE_BAND of the captured width and the NCC search only off it -- best
of both: native back to 0.94, other resolutions still aligned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(doubao): harden alignment -- try fixed+aligned, keep least residual (56/56)
On a faint/busy-background mark the NCC alignment peak can wander a few px off
the true mark and leave a residual (2/56 real corpus files). Off the captured
width, remove_watermark_reverse_alpha now builds BOTH the fixed-geometry and the
NCC-aligned alpha map, applies each, and keeps whichever leaves the least
residual mark (re-detect confidence on the bare reverse-alpha) -- geometry wins
on faint marks, alignment on clear ones, no magic threshold. Real-file round-trip
now removes 56/56 detected Doubao clean across every corpus resolution (was 54).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* perf(doubao): skip residual inpaint at native width for exact recovery
At the captured width the fixed-geometry reverse-alpha is pixel-exact, so
inpainting over it only replaced exactly-recovered interior pixels with a
cv2 hallucination -- measured worse on a textured background (native error
vs true bg 1.6 reverse-alpha-only vs 2.6 with the old always-on
full-footprint inpaint). Native now returns the bare recovery untouched;
off-native, where NCC alignment is only sub-pixel-approximate, the footprint
inpaint stays to clean the seam. Real round-trip still 56/56 across all
corpus resolutions; negatives 0/60, Gemini unaffected.
Add test_native_returns_exact_reverse_alpha_no_inpaint as the regression
guard. Sync CLAUDE.md + README (the table cell and prose described the
pre-NCC "skipped off native / cv2-LaMa" behavior, now stale). Gitignore the
session scheduled_tasks.lock, and add the text-protection research note.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aigc_label now reads the TC260 label from a raw-JSON `AIGC` PNG tEXt chunk
(as Doubao/ByteDance write it, with no namespaced XMP marker) in addition to
the `<TC260:AIGC>` XMP block, via a shared _parse helper gated on a TC260 field
so a generic AIGC key cannot false-positive. New huggingface_job() reads the
hf-job-id PNG chunk; identify surfaces it as a medium-confidence hf_job signal
(parallel to the visible sparkle, never overriding a hard metadata verdict).
Both wired into has_ai_metadata/get_ai_metadata; the PNG save whitelist already
strips them on removal. Found by auditing 646 corpus originals: 28 AIGC and 3
hf-job files the library previously reported as Unknown.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Provenance detection no longer relies on a fixed first-MB read. In a streaming /
non-faststart MP4 the C2PA manifest sits AFTER a multi-megabyte mdat, beyond the
1 MB scan window, so it was missed.
- isobmff.scan_c2pa_region(path): a file-seeking top-level box walker that
returns the payloads of uuid/jumb (provenance) boxes, seeking past mdat by
size without reading it -- works on multi-GB files. Returns b"" for
non-ISOBMFF or on read error. Mirrors the box-size encoding of the existing
in-memory _iter_top_level_boxes (largesize / size==0).
- metadata.scan_head(path, size): the shared input for every C2PA/AIGC/IPTC
byte scan -- first __TEXT __DATA __OBJC others dec hex bytes plus, for ISOBMFF, the late provenance-box
payloads. Behavior-neutral (f.read(size)) for non-ISOBMFF inputs.
- Routed all six metadata scan sites (has_ai_metadata, aigc_label,
iptc_ai_system, synthid_source, exif_generator XMP, get_ai_metadata
soft-binding) and identify's head read through scan_head.
6 new tests: late box found by scan_c2pa_region / scan_head, the fixed window
provably misses it, non-ISOBMFF -> b"", front-placed (faststart) regression.
The remaining gap stays documented: EXIF/XMP stored as items inside the meta
box (AVIF/HEIF stills) still needs meta-box surgery or exiftool.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
