Victor Kuznetsov
464fd142ea
Extend the development-only benchmark family from images to audio and video, and close four research directions from the watermark-landscape map with local, revision-pinned evidence. Benchmark schema: media_type audio and video rows, audioseal and videoseal adapters with media-checked adapter binding, sample-domain audio fidelity (SNR with unbounded_identical), frame-domain video fidelity (mean PSNR, changed-frame fraction), and per-frame temporal evidence in videoseal detection records. The report accepts all three media types and carries SNR columns. Audio: shared pinned AudioSeal oracle, synthetic and system-voice speech cohorts with matched negatives, codec, noise, removed, and forged arms, and the provenance experiment proving visual-only video cleaning leaves the audio bitstream and watermark verdict invariant. Video: TorchScript-only VideoSeal oracle (the pip package is unusable on macOS arm64 and its loader resolves paths relative to cwd), video cohort with crf/scale/fps attacks and four-state arms, temporal evaluation measuring the crf decay curve, the frame-aggregation matrix (computed locally after finding the TorchScript build ignores the aggregation argument), and real provider clips as carriers. Forgery: double-embedding study showing the last writer wins completely under both oracles, scoped remover-trace probe, and the forged/ removed cohort arms encoding each scheme's correct expected verdict (presence rule for audioseal, message rule for videoseal). C2PA resolution: measured the five vendors' resolver contracts and the C2PA 2.2 18.10.5 privacy gradient; product decision recorded that this tool never resolves over the network. A registry guard pins the published resolution-API set. Gate: ruff and pyright src/ clean, 2114 passed / 3 skipped; maintain.sh stops only on the known unfixed lightning 2.6.5 CVEs, pre-existing.
4.3 KiB
VideoSeal temporal evaluation
Development-only study, run on 2026-09-07 from
scripts/videoseal_temporal_study.py, answering the three temporal questions
the case-level benchmark verdict aggregates away: how decoded bit accuracy
decays across H.264 quality, whether the frame-aggregation choice changes the
outcome, and how per-frame accuracy moves in time on real provider clips
versus synthetic carriers.
Carriers: the two synthetic cohort clips (moving gradient, moving texture) plus the committed real provider videos for Veo and Sora, standardized to a 64-frame 256x256 prefix (first 64 frames, center-crop, INTER_AREA resize). Attacks: crf 18/23/28/32 re-encodes, 75 percent downscale, and frame-rate halving. Everything is local; the real clips are already-committed project fixtures with public test clearance.
The TorchScript aggregation trap
The first matrix run produced identical numbers under all four aggregations,
and that was a measurement artifact, not a finding: the standalone TorchScript
build ignores the aggregation argument of detect_video_and_aggregate and
returns the same thresholded bits for every choice (verified by checksum on a
single artifact). The oracle now computes the aggregations itself from the raw
per-frame logits using the exact upstream extract_message formulas - avg,
squared_avg, and l1/l2-norm-weighted averages - and the recorded matrix comes
from those. The formula port is pinned by a hand-computed unit test.
Matrix (bit accuracy of the aggregated 256-bit decode)
| Carrier | crf18 | crf23 | crf28 | crf32 | scale .75 | fps .5 |
|---|---|---|---|---|---|---|
| moving_gradient (clean 0.52) | 0.887 | 0.570 | 0.559 | 0.500 | 0.691 | 0.996 |
| moving_texture (clean 0.48) | 0.602 | 0.559 | 0.531 | 0.586 | 0.879 | 0.980 |
| real_veo (clean 0.52) | 0.801 | 0.723 | 0.621 | 0.574 | 1.000 | 0.855 |
| real_sora (clean 0.51) | 0.996 | 1.000 | 0.992 | 0.902 | 1.000 | 0.996 |
Readings, not verdicts: the benchmark adapter's rule is 0.9 under plain avg.
- Carrier content dominates. The real Sora clip keeps the message through every attack including crf 32; the real Veo clip decays smoothly with quality; the smooth synthetic gradient dies at crf 23. Any "survives H.264" claim must name the carrier and the crf.
- Downscale is not an attack on real content. 0.75 downscale plus re-encode scored a perfect 1.000 on both real clips - resize was a training augmentation, and the antialias resample appears to help the detector. On the synthetic gradient it still costs 0.31.
- Frame-rate halving is harmless. Every carrier kept the message after dropping half the frames: the 256-bit secret repeats across keyframes, so temporal redundancy, not frame count, carries it.
- Aggregation choice matters at the margin. Norm-weighted averaging recovered 2-6 points on damaged arms (moving_texture scale: 0.879 avg versus 0.941 l1/l2, crossing the 0.9 rule) but never rescued a collapsed case and never lifted a clean negative above chance. The kernel keeps plain avg as its canonical rule; the matrix is the evidence for when that choice is conservative.
Per-frame accuracy on the marked clips: real_sora sits at 0.988-1.000 on
every frame, real_veo spans 0.547-0.910, moving_texture 0.457-0.961 - the
temporal spread is itself carrier-dependent, and the kernel's
detection.temporal block now records it for every videoseal case.
Pins
Study script
sha256:682ccf14b21d6ac17bb6500a7efb364247eebae2d9e0e8895ffac33e6d294d4f,
oracle
sha256:5aa67ceaca1b0050727aa0a75440b1bf4c583435b3bf9e8ef2e97fadd46a035c,
case rows
sha256:5494c9bbe8a6085ecbd27fd9732dadf8d004dab7be159e905051e854dabe64aa.
Checkpoint: TorchScript y_256b_img.jit
sha256:5c7a4581c36fc6090aafdcfb3999123bae5172a4847f22e2da4e7fd1a39d1e1b
at upstream commit 870ca7f. The kernel re-run of the video cohort with
temporal records used kernel
sha256:657b0d7e35af75eb97b9bd9b81c15fcae754399f27999e08a5c28c494e69cf05
and produced case rows
sha256:435215e9a55589f6d83e34d62f34469c079db078a0a174ead64ffaa74af664c5
with unchanged verdicts.
Follow-ups
- A crf sweep on more real carriers (the other committed provider clips) to widen the carrier-dominated picture beyond two samples.
- Temporal inpainting comparison remains open from the original research map; the per-frame rows recorded by the kernel are the substrate for it.