Add versioned forensic metadata transports

docs/module-internals.md

@@ -388,9 +388,10 @@ metadata extraction from verdict logic:
 - `extract_provenance_evidence` reads the supported metadata signals into
   `ProvenanceEvidence`.
 - `evidence_from_metadata_record` normalizes an externally collected nested
-  metadata record into the same evidence type without file access. Diagnostic
-  values under `error` and `kind` are excluded from evidence while nested raw
-  bytes remain available through encoded binary fields.
+  metadata record into the same evidence type without file access. Versioned native
+  records accept only source-derived fields; filenames, hashes, timings, errors,
+  prior verdicts, and pixel results cannot become evidence. Unknown native schema
+  versions and other record types are rejected.
 - The vendor registries are matched over `_metadata_region(head)`, not the whole scan
   buffer: they see the container's metadata and not its coded pixels. The tokens are
   raw substrings and the shortest are four and five bytes, so over a megabyte of
@@ -427,19 +428,44 @@ defects found while establishing that equality are the reason each rule exists:
   returns the FIRST candidate carrying a known token, so preserving candidate order
   is part of verdict equivalence.
 
-Pixel forensics are deliberately absent: nothing in the provenance path reads them.
+The transport is independently versioned as `provenance_metadata` schema 1. Native
+records require the exact integer schema version and a `complete` status. Source
+read failures are explicit error records and cannot be judged. WebP walks the full
+declared RIFF container by seeking over `VP8`, `VP8L`, `ALPH`, and `ANMF`, so late
+XMP/C2PA remains visible without shipping coded frames or parsing appended trailer
+bytes as chunks.
+
+Pixel forensics are deliberately absent: the provenance path does not read them.
 Verdict equivalence is checked over tracked fixtures and a separate local evaluation
 corpus.
 
-### Experimental pixel forensics
+### Broad forensic metadata
+
+[`forensic_metadata.py`](../src/remove_ai_watermarks/forensic_metadata.py) owns the
+wide metadata-only inspection record: hashes and timestamps, full EXIF/IPTC, C2PA,
+container inventories, bounded binary metadata, and embedded-thumbnail forensics.
+It is a separate `forensic_metadata` record type and is deliberately rejected by the
+provenance normalizer. Integration code publishes the strict
+`ProvenanceReport.to_dict()` alongside it rather than letting operational fields or
+derived results influence detection.
+
+### Pixel forensics
 
 [`pixel_evidence.py`](../src/remove_ai_watermarks/pixel_evidence.py) measures six
 families of scale-robust pixel statistics (block-DCT histograms and Benford
 deviation, FFT band energies and CFA peaks, high-pass residual, error level,
-gradient, colour) in a single decode, sharing the intermediate maps between them.
+gradient, color) in a single decode, sharing the intermediate maps between them.
 
-It has no consumer. Nothing in the package reads it -- not the verdict, not removal,
-not the CLI -- and the shape is unstable until something does.
+It remains independent of verdict and removal. `PixelEvidence.to_dict()` is the
+versioned service boundary: it omits the local path, exposes complete/partial/error
+status, keeps exception details in logs, and can include opt-in per-stage timings.
+
+The provenance metadata collector, broad forensic collector, provenance report,
+and pixel report all accept an explicit output `schema_version`. Package releases
+may add an output schema while retaining older serializers, so a rolling consumer
+can keep requesting the version it already understands. Within one schema, changes
+are additive; existing fields, types, meanings, signal names, and watermark labels
+remain stable. Unsupported selections raise before a different shape is returned.
 
 `artifacts=True` additionally returns the spatial layer: a perceptual hash, a 128px
 JPEG thumbnail, and coarse ELA, residual and phase maps. Those identify the source

docs/python-api.md

@@ -194,13 +194,20 @@ import json
 from remove_ai_watermarks.identify import identify_metadata_record
 from remove_ai_watermarks.metadata_record import collect_metadata_record
 
-record = collect_metadata_record(Path("input.png"))   # reads the file
+record = collect_metadata_record(Path("input.png"), schema_version=1)   # reads the file
 blob = json.dumps(record)                             # ship it anywhere
 
 report = identify_metadata_record(json.loads(blob), path=Path("input.png"))  # reads nothing
-payload = report.to_dict()                            # versioned JSON contract
+payload = report.to_dict(schema_version=1)            # versioned JSON contract
 ```
 
+The collection record has `record_type="provenance_metadata"`,
+`schema_version=1`, and a `status`. A vanished or unreadable source produces an
+`error` record with structured `issues`; `identify_metadata_record` rejects that
+record instead of turning a collection failure into an unknown-image verdict.
+Unknown schema versions, non-integer aliases, and native records without a
+`complete` collection status are rejected explicitly.
+
 The verdict is the same one `identify(path, check_visible=False,
 check_invisible=False)` returns for that file. That equality is the record's whole
 contract and is verified over the tracked provenance fixtures and a separate local
@@ -208,9 +215,18 @@ evaluation corpus. `ProvenanceReport.to_dict()` is the stable service boundary:
 adds a `schema_version`, contains only JSON-safe values, and deliberately omits the
 local source path.
 
-A record carries metadata regions, never pixels: marker segments before the JPEG
-scan, every PNG chunk except `IDAT`, RIFF chunks except the coded image, the
-ISOBMFF provenance boxes, the container's trailer, the parsed EXIF tags the
+Package and transport versions evolve independently. Long-lived consumers should
+request the schema they implement, as above, instead of assuming the installed
+package's latest schema. Within schema 1, existing fields, types, meanings,
+`signals[].name` values, and `watermarks[]` labels remain compatible; releases may
+add fields that consumers must ignore. A breaking change requires a new schema while
+the schema 1 serializer remains available for rolling upgrades. Asking a release for
+an unsupported schema raises `ValueError` rather than silently returning another
+shape.
+
+A record carries metadata regions, not the primary coded-pixel stream: marker
+segments before the JPEG scan, every PNG chunk except `IDAT`, RIFF chunks except the
+coded image, the ISOBMFF provenance boxes, the container's trailer, the parsed EXIF tags the
 verdict reads by name, PIL's info mapping, and the C2PA manifest store. Record size
 is bounded by those metadata regions and trailers; images with large embedded
 manifests naturally produce larger records.
@@ -236,13 +252,54 @@ evidence = evidence_from_metadata_record(record, path=Path("input.png"))
 report = identify_from_evidence(evidence)
 ```
 
-The normalizer recursively preserves text and byte values. It also decodes
+Unversioned third-party records are normalized recursively for compatibility. The
+normalizer preserves text and byte values. It also decodes
 strings prefixed with `hex:` and fields named `base64` or ending in
-`_base64`. Diagnostic values under `error` and `kind` are ignored because they
-describe the collector rather than the source file. Pass a C2PA manifest-store
-dictionary in `record["c2pa_store"]`, or through the explicit
+`_base64`. Diagnostic, transport, timing, hash, provenance-result, and pixel-result
+subtrees are ignored because they describe the collector or a derived result rather
+than the source file. A versioned portable record is stricter still: only
+`metadata_base64`, `tail_base64`, `pil`, `exif`, and `c2pa_store` are accepted as
+source evidence. Other `record_type` values are rejected, so do not pass a broad
+forensic inspection record to this API. Pass a C2PA manifest-store dictionary in
+`record["c2pa_store"]`, or through the explicit
 `c2pa_manifest_store` argument.
 
+### Broad metadata inspection
+
+`collect_forensic_metadata` provides the wide metadata-only record used by forensic
+inspection and migration adapters. It preserves hashes and timestamps, full EXIF and
+IPTC, C2PA, container inventories, bounded raw metadata payloads, and embedded
+thumbnail forensics. It does not calculate a provenance verdict or pixel statistics.
+
+```python
+from remove_ai_watermarks.forensic_metadata import collect_forensic_metadata
+
+record = collect_forensic_metadata(Path("input.png"), schema_version=1)
+assert record["record_type"] == "forensic_metadata"
+```
+
+This record is intentionally not accepted by `identify_metadata_record`. Collect the
+small strict provenance record separately and publish the resulting
+`ProvenanceReport.to_dict()` as the detector contract.
+
+### Pixel evidence
+
+`extract_pixel_evidence` decodes once and calculates the DCT, FFT, residual, ELA,
+gradient, and color families. Its versioned `to_dict()` result has a semantic
+`status`: `complete`, `partial` when an individual family failed, or `error` when the
+source could not be decoded. Transported errors contain only the exception class, so
+local paths stay in the caller's logs rather than crossing the service boundary.
+
+```python
+from remove_ai_watermarks.pixel_evidence import extract_pixel_evidence
+
+pixels = extract_pixel_evidence(Path("input.png"), artifacts=False, timings=True)
+payload = pixels.to_dict(schema_version=1)
+```
+
+Timings and spatial artifacts are opt-in. Artifacts include image-identifying data
+such as a thumbnail and perceptual hash; aggregate feature families do not.
+
 `identify_from_evidence` does not reopen the source file by default: it evaluates
 metadata only, and registered visible marks and pixel-backed invisible watermarks
 remain in the path-based `identify` call.

docs/release-and-distribution.md

@@ -71,10 +71,10 @@ The source distribution uses an explicit allowlist for `/src`, `/LICENSE`,
 `[tool.hatch.build.targets.sdist]` in `pyproject.toml`. It also defensively
 excludes `/data`, `/tmp`, and `/.sc`. Keep both controls: calibration captures,
 test corpora, generated research outputs, and local session state do not belong
-in the published package archive. `.gitignore` covers `tmp/` and `.sc/`, so those
-never reach a commit, but ignore rules are not the build boundary -- hatchling may
-include untracked files, and `data/` is deliberately tracked, so the sdist exclude
-is the only control keeping it out of the archive.
+in the published package archive. Hatchling always adds the root `.gitignore` to
+the sdist, so keep its comments generic and free of local operational context.
+Ignore rules are not the build boundary: `data/` is deliberately tracked, while
+the sdist configuration keeps it and the other excluded paths out of the archive.
 
 ## Build backend
 
@@ -97,6 +97,15 @@ write access.
 
 ## Release verification
 
+Forensic transports are versioned independently from the package. Before publishing
+a change to provenance metadata, provenance reports, broad forensic metadata, or
+pixel evidence, run their schema 1 contract tests. Additive fields are compatible;
+renaming a field, changing its type or meaning, changing a signal name or watermark
+label, or removing a field requires a new output schema. Add the new serializer
+without removing schema 1 so long-lived consumers can update separately. A package
+release must never silently substitute its latest schema when a caller explicitly
+requests an older supported one.
+
 After publication, verify:
 
 - both wheel and source distribution exist on PyPI;