Read WebP metadata past the scan window and surface C2PA reader failures

Three gaps found while measuring the record path against the file path, each one
a signal the library could not see:

WebP stores `XMP ` after the pixels, so on any WebP above the scan window a fixed
read stops short of the label. `_riff_late_metadata` steps over the coded image to
reach it, the RIFF analogue of the existing PNG and ISOBMFF readers. Three corpus
files hid an IPTC "Made with AI" tag and a C2PA `trainedAlgorithmicMedia` there.
The decoder-backed fallback now covers only what it is actually for -- metadata the
raw bytes do not spell, such as a compressed PNG `zTXt` packet.

A C2PA reader failure returned the same `None` as a file with no manifest, so a
verdict could fall back to the raw byte scan with no trace anywhere. Failures now
log at warning and only genuine ones do: a file without credentials never reaches
that branch, and an unsupported container is demoted to debug through the reader's
own `C2paError.NotSupported`. The first corpus run with it found a truncated PNG.

`scan_dataset.py` never registered the pillow-heif opener it declares as a
dependency, so every HEIC was scanned as unreadable -- no EXIF, and a pixel layer
that was 397 of 406 features NaN instead of 136.

`_riff_late_metadata` caps its total like `isobmff.scan_c2pa_region` does. Clamping
each chunk to the bytes remaining is not enough on its own: one chunk can declare a
length spanning most of the file, and this runs on the memoized verdict path over
images from arbitrary sources.

Also lands `identify_metadata_record` and `ProvenanceReport.to_dict()`, the
one-call entry point and the versioned JSON contract for the record path.

Record-vs-file equality holds over 3,478 corpus images, and the eight files these
fixes recovered still report AI.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
2026-08-05 21:10:38 -07:00
co-authored by Claude Opus 5
parent 0c5961a0ed
commit 2668f1302d
14 changed files with 300 additions and 55 deletions
+10 -10
View File
@@ -191,29 +191,29 @@ built from on another machine, in another process, or later.
```python
import json
from remove_ai_watermarks.identify import evidence_from_metadata_record, identify_from_evidence
from remove_ai_watermarks.identify import identify_metadata_record
from remove_ai_watermarks.metadata_record import collect_metadata_record
record = collect_metadata_record(Path("input.png")) # reads the file
blob = json.dumps(record) # ship it anywhere
evidence = evidence_from_metadata_record(json.loads(blob), path=Path("input.png"))
report = identify_from_evidence(evidence) # reads nothing
report = identify_metadata_record(json.loads(blob), path=Path("input.png")) # reads nothing
payload = report.to_dict() # versioned JSON contract
```
The verdict is the same one `identify(path, check_visible=False,
check_invisible=False)` returns for that file. That equality is the record's whole
contract, and it is verified two ways: over the tracked provenance fixtures in
`tests/test_metadata_record.py`, and over a local corpus, where 3,478 images
(every file carrying a rare signal, plus a random slice) produced identical
reports through both paths.
contract and is verified over the tracked provenance fixtures and a separate local
evaluation corpus. `ProvenanceReport.to_dict()` is the stable service boundary: it
adds a `schema_version`, contains only JSON-safe values, and deliberately omits the
local source path.
A record carries metadata regions, never pixels: marker segments before the JPEG
scan, every PNG chunk except `IDAT`, RIFF chunks except the coded image, the
ISOBMFF provenance boxes, the container's trailer, the parsed EXIF tags the
verdict reads by name, PIL's info mapping, and the C2PA manifest store. Typical
size is 13 kB (p90 93 kB); the tail belongs to images carrying a large embedded
manifest, where the store itself dominates.
verdict reads by name, PIL's info mapping, and the C2PA manifest store. Record size
is bounded by those metadata regions and trailers; images with large embedded
manifests naturally produce larger records.
The `path` argument is metadata: it labels the report and is never opened by
either function, so a record collected elsewhere can be judged against a path that