Decide the SynthID proxy in the verdict, where both extractors meet

A full-corpus audit of the record path against the file path found 75 of 48,905
images disagreeing, and 74 were one gap: the SynthID byte scan for containers whose
manifest no parser reaches lived in `get_ai_metadata`, an extractor the record path
does not run. The record silently reported no SynthID for images `identify` flagged.

Moving the scan into `identify_from_evidence` fixes it by construction rather than by
copying the rule into a second extractor -- the same shape `soft_binding` already
uses. Its byte checks mirror `metadata.synthid_source` literally instead of reusing
the broader `has_c2pa` / `c2pa_source_kind` derived above, so the file path's answers
do not move: verdicts over a 4,000-image sample are byte-identical.

`scripts/record_parity_audit.py` is the audit itself, now repeatable. It walks a
dataset, judges every image through both seams with the record round-tripped through
JSON, and reports disagreements by field and by signal. The rule in
`.claude/rules/development.md` says to re-run both sides of this seam after changing
either; this is what to run.

Both timing and audit scripts now put the package's OWN `src` on the path. From a
worktree an editable install resolves to the main checkout, so the audit imported a
different tree than the one under test -- the failure the same rules file warns about,
reproduced within an hour of writing it down.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
2026-08-05 21:10:38 -07:00
co-authored by Claude Opus 5
parent 2668f1302d
commit bebff368fc
5 changed files with 299 additions and 4 deletions
+53
View File
@@ -1353,3 +1353,56 @@ class TestSharedPixelDecode:
report = identify(self.SAMPLE, check_visible=True, check_invisible=False)
assert not any(s.name.startswith("visible_") for s in report.signals)
assert report.is_ai_generated is True # the C2PA verdict survives the decode failure
class TestSynthIdProxyIsDecidedInTheVerdict:
"""The SynthID byte scan belongs to the verdict, not to extraction.
Extraction has two implementations -- one reading a file, one reading a portable
record -- so a rule that lives in only one of them is a rule the other silently
lacks. This one did: 74 corpus images reported SynthID through ``identify`` and
not through the record."""
# A JUMBF-wrapped manifest from a SynthID-pairing signer on AI-generated content:
# the exact shape `synthid_source`'s byte scan is gated on. Spliced into a real
# JPEG as a well-formed APP11 segment, because a malformed one is skipped by the
# record's structural walk and the test would compare two different inputs.
MANIFEST = b"jumb c2pa Google LLC trainedAlgorithmicMedia"
def _jpeg_with_manifest(self, path: Path) -> Path:
import numpy as np
from PIL import Image
Image.fromarray(np.zeros((32, 32, 3), dtype=np.uint8)).save(path, "JPEG")
data = path.read_bytes()
segment = b"\xff\xeb" + (len(self.MANIFEST) + 2).to_bytes(2, "big") + self.MANIFEST
path.write_bytes(data[:2] + segment + data[2:])
return path
def test_both_paths_infer_it_from_the_same_bytes(self, tmp_path: Path):
from remove_ai_watermarks.identify import identify_metadata_record
from remove_ai_watermarks.metadata_record import collect_metadata_record
path = self._jpeg_with_manifest(tmp_path / "gemini.jpg")
via_file = identify(path, check_visible=False, check_invisible=False)
via_record = identify_metadata_record(collect_metadata_record(path), path=path)
assert any("SynthID" in mark for mark in via_file.watermarks)
assert via_record.watermarks == via_file.watermarks
def test_it_needs_a_manifest_and_an_ai_source_type(self, tmp_path: Path):
"""The vendor name alone is not evidence: an ordinary photo mentioning
"Google LLC" in EXIF must not acquire a SynthID verdict."""
import numpy as np
from PIL import Image
path = tmp_path / "photo.jpg"
Image.fromarray(np.zeros((32, 32, 3), dtype=np.uint8)).save(path, "JPEG")
data = path.read_bytes()
note = b"Google LLC Pixel"
path.write_bytes(data[:2] + b"\xff\xeb" + (len(note) + 2).to_bytes(2, "big") + note + data[2:])
report = identify(path, check_visible=False, check_invisible=False)
assert not any("SynthID" in mark for mark in report.watermarks)