Merge remote-tracking branch 'origin/main' into research/video-synthid-quality-groundwork

docs/module-internals.md

@@ -368,7 +368,16 @@ metadata scanners and `remove_ai_metadata`.
 
 Key contracts:
 
-- `scan_head` is the shared cached input for bounded byte scans.
+- `scan_head` is the shared cached input for bounded byte scans. It fills the buffer
+  in two layers. Structural readers first, one per container, each seeking past the
+  pixel payload to reach metadata placed beyond the window: `isobmff.scan_c2pa_region`,
+  `_png_late_metadata`, `_riff_late_metadata`. A decoder-backed fallback last,
+  `_decoder_visible_text`, for metadata the raw bytes do not spell at all — a
+  zlib-compressed PNG `zTXt` packet is readable only after inflation. The layers are
+  ordered that way because the structural readers work on files no decoder can open.
+- A C2PA reader failure is logged at warning, not debug. It returns the same `None` as
+  a file with no manifest, so nothing downstream can distinguish "no credentials" from
+  "the credentials could not be read", and the second silently downgrades a verdict.
 - JPEG stripping walks metadata segments and preserves the entropy-coded image
   scan.
 - ISOBMFF containers use
@@ -407,13 +416,90 @@ metadata extraction from verdict logic:
 - `extract_provenance_evidence` reads the supported metadata signals into
   `ProvenanceEvidence`.
 - `evidence_from_metadata_record` normalizes an externally collected nested
-  metadata record into the same evidence type without file access. Diagnostic
-  values under `error` and `kind` are excluded from evidence while nested raw
-  bytes remain available through encoded binary fields.
-- `identify_from_evidence` evaluates that evidence without reopening the source.
+  metadata record into the same evidence type without file access. Versioned native
+  records accept only source-derived fields; filenames, hashes, timings, errors,
+  prior verdicts, and pixel results cannot become evidence. Unknown native schema
+  versions and other record types are rejected.
+- The vendor registries are matched over `_metadata_region(head)`, not the whole scan
+  buffer: they see the container's metadata and not its coded pixels. The tokens are
+  raw substrings and the shortest are four and five bytes, so over a megabyte of
+  compressed data one turns up by chance, and the entry it hits may assert AI. The
+  trim happens only when the container parses -- a malformed or unknown one is left
+  whole, because dropping real evidence to avoid a chance match is the wrong trade.
+- `identify_from_evidence` evaluates that evidence without reopening the source. Rules
+  that decide a verdict live here, not in extraction: extraction has two
+  implementations, and a rule in only one of them is a rule the other lacks. The
+  SynthID proxy is the worked example — its structured form comes from the manifest,
+  and the byte-scan fallback for containers no parser reaches runs in the verdict, so
+  both extractors reach the same answer. It did not, and the record path silently
+  reported no SynthID for images the file path flagged.
 - `identify` preserves the path-based API and adds the optional registered
   visible-mark and open invisible-watermark decoders after extraction.
 
+### Portable metadata record
+
+[`metadata_record.py`](../src/remove_ai_watermarks/metadata_record.py) produces the
+record `evidence_from_metadata_record` consumes, so collection and verdict can run
+on different machines. Its contract is equality with the file path, and the three
+defects found while establishing that equality are the reason each rule exists:
+
+- It walks the file's RAW head, never the `scan_head` buffer. That buffer is the
+  head concatenated with late metadata payloads, so a structural walk runs off the
+  end of the real head and parses appended bytes as chunks, inflating the record and
+  creating false signals.
+- Samsung Galaxy AI splits its evidence: the `PhotoEditor_Re_Edit_Data` marker sits
+  in the post-EOI trailer while the `genAIType` value it is gated on can sit inside
+  the entropy-coded scan. A marked file therefore keeps the whole tail window, not
+  just the trailer.
+- PIL's info keys are emitted in the file path's own candidate order
+  (`Software`, `Source`, `Title`, `Description`, then EXIF). `generator_from_metadata`
+  returns the FIRST candidate carrying a known token, so preserving candidate order
+  is part of verdict equivalence.
+
+The transport is independently versioned as `provenance_metadata` schema 1. Native
+records require the exact integer schema version and a `complete` status. Source
+read failures are explicit error records and cannot be judged. WebP walks the full
+declared RIFF container by seeking over `VP8`, `VP8L`, `ALPH`, and `ANMF`, so late
+XMP/C2PA remains visible without shipping coded frames or parsing appended trailer
+bytes as chunks.
+
+Pixel forensics are deliberately absent: the provenance path does not read them.
+Verdict equivalence is checked over tracked fixtures and a separate local evaluation
+corpus.
+
+### Broad forensic metadata
+
+[`forensic_metadata.py`](../src/remove_ai_watermarks/forensic_metadata.py) owns the
+wide metadata-only inspection record: hashes and timestamps, full EXIF/IPTC, C2PA,
+container inventories, bounded binary metadata, and embedded-thumbnail forensics.
+It is a separate `forensic_metadata` record type and is deliberately rejected by the
+provenance normalizer. Integration code publishes the strict
+`ProvenanceReport.to_dict()` alongside it rather than letting operational fields or
+derived results influence detection.
+
+### Pixel forensics
+
+[`pixel_evidence.py`](../src/remove_ai_watermarks/pixel_evidence.py) measures six
+families of scale-robust pixel statistics (block-DCT histograms and Benford
+deviation, FFT band energies and CFA peaks, high-pass residual, error level,
+gradient, color) in a single decode, sharing the intermediate maps between them.
+
+It remains independent of verdict and removal. `PixelEvidence.to_dict()` is the
+versioned service boundary: it omits the local path, exposes complete/partial/error
+status, keeps exception details in logs, and can include opt-in per-stage timings.
+
+The provenance metadata collector, broad forensic collector, provenance report,
+and pixel report all accept an explicit output `schema_version`. Package releases
+may add an output schema while retaining older serializers, so a rolling consumer
+can keep requesting the version it already understands. Within one schema, changes
+are additive; existing fields, types, meanings, signal names, and watermark labels
+remain stable. Unsupported selections raise before a different shape is returned.
+
+`artifacts=True` additionally returns the spatial layer: a perceptual hash, a 128px
+JPEG thumbnail, and coarse ELA, residual and phase maps. Those identify the source
+image rather than describe it, which is why they are opt-in and a separate field: a
+caller storing them is handling image content, not statistics about it.
+
 The DWT-DCT detector and the visible-mark stage share a single decode of the
 source, held by
 a per-call `_SharedDecode`. It exposes two accessors because the two arms need

docs/python-api.md

@@ -182,8 +182,61 @@ evidence = extract_provenance_evidence(Path("input.png"))
 report = identify_from_evidence(evidence)
 ```
 
-If metadata was collected by another component, normalize its nested record
-without reopening the original file:
+### Collect once, judge elsewhere
+
+`collect_metadata_record` splits the two halves apart: it is the only step that
+touches the file, and it returns a JSON-serializable record the verdict can be
+built from on another machine, in another process, or later.
+
+```python
+import json
+
+from remove_ai_watermarks.identify import identify_metadata_record
+from remove_ai_watermarks.metadata_record import collect_metadata_record
+
+record = collect_metadata_record(Path("input.png"), schema_version=1)   # reads the file
+blob = json.dumps(record)                             # ship it anywhere
+
+report = identify_metadata_record(json.loads(blob), path=Path("input.png"))  # reads nothing
+payload = report.to_dict(schema_version=1)            # versioned JSON contract
+```
+
+The collection record has `record_type="provenance_metadata"`,
+`schema_version=1`, and a `status`. A vanished or unreadable source produces an
+`error` record with structured `issues`; `identify_metadata_record` rejects that
+record instead of turning a collection failure into an unknown-image verdict.
+Unknown schema versions, non-integer aliases, and native records without a
+`complete` collection status are rejected explicitly.
+
+The verdict is the same one `identify(path, check_visible=False,
+check_invisible=False)` returns for that file. That equality is the record's whole
+contract and is verified over the tracked provenance fixtures and a separate local
+evaluation corpus. `ProvenanceReport.to_dict()` is the stable service boundary: it
+adds a `schema_version`, contains only JSON-safe values, and deliberately omits the
+local source path.
+
+Package and transport versions evolve independently. Long-lived consumers should
+request the schema they implement, as above, instead of assuming the installed
+package's latest schema. Within schema 1, existing fields, types, meanings,
+`signals[].name` values, and `watermarks[]` labels remain compatible; releases may
+add fields that consumers must ignore. A breaking change requires a new schema while
+the schema 1 serializer remains available for rolling upgrades. Asking a release for
+an unsupported schema raises `ValueError` rather than silently returning another
+shape.
+
+A record carries metadata regions, not the primary coded-pixel stream: marker
+segments before the JPEG scan, every PNG chunk except `IDAT`, RIFF chunks except the
+coded image, the ISOBMFF provenance boxes, the container's trailer, the parsed EXIF tags the
+verdict reads by name, PIL's info mapping, and the C2PA manifest store. Record size
+is bounded by those metadata regions and trailers; images with large embedded
+manifests naturally produce larger records.
+
+The `path` argument is metadata: it labels the report and is never opened by
+either function, so a record collected elsewhere can be judged against a path that
+does not exist locally.
+
+If metadata was collected by another component instead, normalize its nested
+record the same way:
 
 ```python
 from remove_ai_watermarks.identify import (
@@ -199,13 +252,54 @@ evidence = evidence_from_metadata_record(record, path=Path("input.png"))
 report = identify_from_evidence(evidence)
 ```
 
-The normalizer recursively preserves text and byte values. It also decodes
+Unversioned third-party records are normalized recursively for compatibility. The
+normalizer preserves text and byte values. It also decodes
 strings prefixed with `hex:` and fields named `base64` or ending in
-`_base64`. Diagnostic values under `error` and `kind` are ignored because they
-describe the collector rather than the source file. Pass a C2PA manifest-store
-dictionary in `record["c2pa_store"]`, or through the explicit
+`_base64`. Diagnostic, transport, timing, hash, provenance-result, and pixel-result
+subtrees are ignored because they describe the collector or a derived result rather
+than the source file. A versioned portable record is stricter still: only
+`metadata_base64`, `tail_base64`, `pil`, `exif`, and `c2pa_store` are accepted as
+source evidence. Other `record_type` values are rejected, so do not pass a broad
+forensic inspection record to this API. Pass a C2PA manifest-store dictionary in
+`record["c2pa_store"]`, or through the explicit
 `c2pa_manifest_store` argument.
 
+### Broad metadata inspection
+
+`collect_forensic_metadata` provides the wide metadata-only record used by forensic
+inspection and migration adapters. It preserves hashes and timestamps, full EXIF and
+IPTC, C2PA, container inventories, bounded raw metadata payloads, and embedded
+thumbnail forensics. It does not calculate a provenance verdict or pixel statistics.
+
+```python
+from remove_ai_watermarks.forensic_metadata import collect_forensic_metadata
+
+record = collect_forensic_metadata(Path("input.png"), schema_version=1)
+assert record["record_type"] == "forensic_metadata"
+```
+
+This record is intentionally not accepted by `identify_metadata_record`. Collect the
+small strict provenance record separately and publish the resulting
+`ProvenanceReport.to_dict()` as the detector contract.
+
+### Pixel evidence
+
+`extract_pixel_evidence` decodes once and calculates the DCT, FFT, residual, ELA,
+gradient, and color families. Its versioned `to_dict()` result has a semantic
+`status`: `complete`, `partial` when an individual family failed, or `error` when the
+source could not be decoded. Transported errors contain only the exception class, so
+local paths stay in the caller's logs rather than crossing the service boundary.
+
+```python
+from remove_ai_watermarks.pixel_evidence import extract_pixel_evidence
+
+pixels = extract_pixel_evidence(Path("input.png"), artifacts=False, timings=True)
+payload = pixels.to_dict(schema_version=1)
+```
+
+Timings and spatial artifacts are opt-in. Artifacts include image-identifying data
+such as a thumbnail and perceptual hash; aggregate feature families do not.
+
 `identify_from_evidence` does not reopen the source file by default: it evaluates
 metadata only, and registered visible marks and pixel-backed invisible watermarks
 remain in the path-based `identify` call.

docs/release-and-distribution.md

@@ -71,10 +71,10 @@ The source distribution uses an explicit allowlist for `/src`, `/LICENSE`,
 `[tool.hatch.build.targets.sdist]` in `pyproject.toml`. It also defensively
 excludes `/data`, `/tmp`, and `/.sc`. Keep both controls: calibration captures,
 test corpora, generated research outputs, and local session state do not belong
-in the published package archive. `.gitignore` covers `tmp/` and `.sc/`, so those
-never reach a commit, but ignore rules are not the build boundary -- hatchling may
-include untracked files, and `data/` is deliberately tracked, so the sdist exclude
-is the only control keeping it out of the archive.
+in the published package archive. Hatchling always adds the root `.gitignore` to
+the sdist, so keep its comments generic and free of local operational context.
+Ignore rules are not the build boundary: `data/` is deliberately tracked, while
+the sdist configuration keeps it and the other excluded paths out of the archive.
 
 ## Build backend
 
@@ -97,6 +97,15 @@ write access.
 
 ## Release verification
 
+Forensic transports are versioned independently from the package. Before publishing
+a change to provenance metadata, provenance reports, broad forensic metadata, or
+pixel evidence, run their schema 1 contract tests. Additive fields are compatible;
+renaming a field, changing its type or meaning, changing a signal name or watermark
+label, or removing a field requires a new output schema. Add the new serializer
+without removing schema 1 so long-lived consumers can update separately. A package
+release must never silently substitute its latest schema when a caller explicitly
+requests an older supported one.
+
 After publication, verify:
 
 - both wheel and source distribution exist on PyPI;

docs/watermarking-landscape.md

@@ -68,7 +68,7 @@ payloads. Removal remuxes either container through ffmpeg with stream copy.
   Metadata stripping for supported audio containers is a separate implemented
   path.
 
-**Box detection window — now handled (v0.6.8):** detection no longer relies on a fixed first-MB read. `metadata.scan_head(path, size)` reads the first `size` bytes and, for ISOBMFF, appends the payloads of late provenance boxes found by `isobmff.scan_c2pa_region` (a file-seeking top-level box walker that skips past `mdat` by size without reading it), so a C2PA/AIGC/IPTC manifest placed AFTER a large `mdat` in a streaming/non-faststart MP4 is now caught. Every C2PA/marker byte scan (`has_ai_metadata`, `aigc_label`, `iptc_ai_system`, `synthid_source`, `exif_generator` XMP, `get_ai_metadata` soft-binding, and `identify`) goes through `scan_head`; for PNG it likewise appends the payloads of `tEXt` / `iTXt` / `zTXt` / `eXIf` / `iCCP` chunks that start beyond the window (`_png_late_metadata`, seeking past `IDAT`), which is how a TC260 AIGC label appended after the pixel stream is caught; for every other input, and for any file that fits inside `size`, it is exactly `f.read(size)`.
+**Box detection window — now handled (v0.6.8):** detection no longer relies on a fixed first-MB read. `metadata.scan_head(path, size)` reads the first `size` bytes and, for ISOBMFF, appends the payloads of late provenance boxes found by `isobmff.scan_c2pa_region` (a file-seeking top-level box walker that skips past `mdat` by size without reading it), so a C2PA/AIGC/IPTC manifest placed AFTER a large `mdat` in a streaming/non-faststart MP4 is now caught. Every C2PA/marker byte scan (`has_ai_metadata`, `aigc_label`, `iptc_ai_system`, `synthid_source`, `exif_generator` XMP, `get_ai_metadata` soft-binding, and `identify`) goes through `scan_head`; for PNG it likewise appends the payloads of `tEXt` / `iTXt` / `zTXt` / `eXIf` / `iCCP` chunks that start beyond the window (`_png_late_metadata`, seeking past `IDAT`), which is how a TC260 AIGC label appended after the pixel stream is caught; for WebP it appends the `EXIF` / `XMP ` / `ICCP` / `C2PA` chunks past the window (`_riff_late_metadata`, stepping over the coded image), which is how an IPTC "Made with AI" tag stored after the pixels is caught; and for a file at least `size` bytes long it finally appends the metadata text the decoder reaches but a raw read cannot (`_decoder_visible_text`), which covers a compressed PNG `zTXt` packet no byte scan can spell; for any file that fits inside `size`, it is exactly `f.read(size)`.
 
 Native TC260 MP4/MOV tags do not live in those top-level provenance boxes.
 `tc260_aigc_payloads` separately seeks through `moov.udta.meta.keys/ilst`, so