diff --git a/data/evaluations/video-synthid-oracle.csv b/data/evaluations/video-synthid-oracle.csv index 74b90f8..a740419 100644 --- a/data/evaluations/video-synthid-oracle.csv +++ b/data/evaluations/video-synthid-oracle.csv @@ -1,3 +1,3 @@ date,source_url,source_sha256,source_width,source_height,source_fps,duration_seconds,source_verdict,vae,noise_std,long_side,fps,seed,output_sha256,output_verdict,output_verdict_text,output_detected_range,track,session_id,stratum,psnr_db,temporal_residual_ratio -2026-07-31,https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/media/veo__veo-3__off-road.mp4,79a552b9406a079682440c31f14d33a10ba8e1b8b2e96425f5de70f63350299d,,,,8,detected_all_frames,,0.10,512,12,0,079165105d4c56e1612091987c08c2627049423025f74c0d4e245fb47c2ff0e3,detected,,,,,,26.2932,1.0072 -2026-07-31,https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/media/veo__veo-3__off-road.mp4,79a552b9406a079682440c31f14d33a10ba8e1b8b2e96425f5de70f63350299d,,,,8,detected_all_frames,,0.15,512,12,0,1c4046bcfdead138353b4e2a73339ba227bb5e544878d80c5bc6cd8427c7b00e,not_detected,,,,,,25.3911,1.0578 +2026-07-31,https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/media/veo__veo-3__off-road.mp4,79a552b9406a079682440c31f14d33a10ba8e1b8b2e96425f5de70f63350299d,1920,1080,24,8,detected_all_frames,,0.10,512,12,0,079165105d4c56e1612091987c08c2627049423025f74c0d4e245fb47c2ff0e3,detected,,,,,,26.2932,1.0072 +2026-07-31,https://storage.googleapis.com/gdm-deepmind-com-prod-public/media/media/veo__veo-3__off-road.mp4,79a552b9406a079682440c31f14d33a10ba8e1b8b2e96425f5de70f63350299d,1920,1080,24,8,detected_all_frames,,0.15,512,12,0,1c4046bcfdead138353b4e2a73339ba227bb5e544878d80c5bc6cd8427c7b00e,not_detected,,,,,,25.3911,1.0578 diff --git a/docs/known-limitations.md b/docs/known-limitations.md index 8b7a73a..9bbdeba 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -112,6 +112,21 @@ measured 25.39 dB paired PSNR and a 1.058 motion-compensated temporal-residual ratio. This is one carrier, not a universal guarantee; hashes and exact verdicts are tracked in `data/evaluations/video-synthid-oracle.csv`. +### The invisible video path does not touch the audio track + +`remove_video_invisible` regenerates the video stream and copies the source audio +verbatim: the extracted audio bitstream of input and output has an identical +sha256, measured on two carriers. It also strips every metadata marker, so the +output can report clean from `get_ai_metadata` and `identify_video` while carrying +an untouched generated audio track. Google's verifier scores audio and visual +tracks separately, and a track this path never modifies is a track it cannot have +cleaned. + +Whether a given carrier's audio actually holds a mark the verifier reads has not +been established -- that needs a provider verdict, which has not been obtained. +Treat a clean local report on a clip with generated audio as unproven, not as a +guarantee, and check the audio separately when it matters. + The shipped engine streams sampled frames in bounded batches, computes its fidelity metrics incrementally, and pipes regenerated pixels directly to ffmpeg. Its frame and latent memory is therefore bounded by `--batch-size` diff --git a/docs/video-synthid-quality-research.md b/docs/video-synthid-quality-research.md index f1ac015..dc2e4c0 100644 --- a/docs/video-synthid-quality-research.md +++ b/docs/video-synthid-quality-research.md @@ -171,6 +171,120 @@ Three readings, all MEASURED, none of them about SynthID: so the `max(temporal_baseline, 1e-6)` denominator collapses and the ratio inflates. This is the predicted defect, now observed rather than argued. +## Measurements on the real carrier (2026-08-06) + +The public Veo 3 off-road sample was downloaded and its sha256 matched the manifest +exactly, so everything below is measured on the same file the 2026-07-31 +calibration used. Its properties, none of which the manifest recorded at the time: +H.264 1920x1080 at 24 fps, 8.000 s, 5,155,063 bps video, plus an **AAC 48 kHz +stereo 192,241 bps audio track of 7.914 s**. + +The shipped profile therefore delivers 512x288 at 12 fps: **7.1% of the frame area +and 3.6% of the pixels per second.** + +### The geometry prize, measured end to end + +Scored with `scripts/video_fidelity_probe.py` against the untouched 1920x1080 +source, all at `noise_std=0.15`: + +| Geometry | Frame area | PSNR | SSIM | Temporal | Bitrate | +| --- | --- | --- | --- | --- | --- | +| 512x288 @12 (shipped) | 7.1% | 25.3971 | 0.8215 | 1.0407 | 2187 kbps | +| 768x432 @12 | 16.0% | 27.3796 | 0.8556 | 1.0323 | 3782 kbps | +| 1024x576 @12 | 28.4% | 28.8542 | 0.8819 | 1.0262 | 5604 kbps | +| 512x288 @24 | 7.1% | 25.6142 | 0.8270 | 1.0995 | 3181 kbps | + +**512 to 1024 is worth +3.46 dB, against 1.92 dB for the entire `noise_std` axis.** +The geometry lever is roughly twice the noise lever, which is the first hard number +this investigation has for the size of the prize. What it still costs in removal is +unknown: none of these was ever submitted. + +Two readings that are not obvious from the table: + +- **Temporal residual improves as resolution rises** (1.0407 to 1.0262). The shared + noise field lives on the latent grid, so at 1920 it is four times finer relative + to the frame than at 512, where it stretches into low-frequency blotches. Higher + resolution plausibly makes the perturbation *less* visible, not more. +- **The frame-rate arm looks bad here and the metric is the wrong judge.** 24 fps + buys +0.22 dB and worsens the temporal ratio to 1.0995, but what 24 fps actually + buys is smoothness, which neither PSNR nor SSIM measures at all. The metric sees + only the cost. Do not decide the fps question from this table. + +Encoder cost on an M-series MPS device, 8 seconds of source: 51 s at 512/12, 180 s +at 768/12, 380 s at 1024/12, 190 s at 512/24. Native 1920x1080 at 24 fps +extrapolates to roughly 46 minutes for the same 8 seconds, around 350x real time. A +native run was started and killed before producing anything, so that figure remains +an extrapolation. + +### The certified row does not reproduce + +Re-running the certified configuration (512/12/0.15, seed 0) on this machine gave +25.5002 dB and 1.0675 against the manifest's 25.3911 and 1.0578, and a different +output sha256. The manifest records neither device nor dtype, and the original was +almost certainly CUDA fp16 while this was MPS fp32. **The oracle certified a +specific file that cannot be rebuilt**, so any re-run produces an artifact the +verifier never saw. Device and dtype belong in the schema. + +### Audio passes through untouched, and the output still reads clean + +Established on two carriers, by comparing the extracted audio bitstream sha256 of +input and output: + +- Veo carrier: `674063df...` in, `674063df...` out. +- A Lyria 3 track generated for this purpose: `6ee4c041...` in, `6ee4c041...` out. + +The Lyria case is the sharper one because its provenance is known. Google's own +generator stamps it: `get_ai_metadata` reads a C2PA manifest naming +`Google C2PA Core Generator Library`, issuer `Google LLC`, `trainedAlgorithmicMedia`. +After the shipped path runs, that output reports `{}` -- every metadata marker +stripped, `remaining_metadata` empty, visuals regenerated -- while the audio the +generator produced is bit-for-bit intact. + +So the pipeline can produce a file that **our own detector calls clean while an +untouched Google-generated audio track sits inside it**. The mechanism is proven; +whether that audio carries an audible-to-the-verifier SynthID mark is not, and only +the oracle can say. If it does, this is a shipped defect rather than a quality +question, and it outranks everything else in this document. + +Worth recording alongside it: the Veo carrier itself returns `{}` from +`get_ai_metadata`. The public sample carries no C2PA at all, which is exactly why it +needed an oracle verdict in the first place. + +## State when this work was parked (2026-08-06) + +Landed in `main` through PR #78: the manifest-backed default pin, the +scaling-factor gate and its single seam, the sweep loading through the engine's +loader, `scripts/video_fidelity_probe.py` with its tests, the extended manifest +schema, and this document. **The shipped operating point did not move**: it is +still 512 px / 12 fps / `noise_std=0.15`. + +A product requirement arrived late and reframes the target: output resolution, +frame rate and bitrate must not be reduced. Resolution, frame rate and crf are +parameters and can be preserved, but zero quality loss is not achievable by this +mechanism at all -- removal works by rebuilding pixels, so a regeneration can never +return its input. The honest goal is to preserve container properties and make the +perceptual loss imperceptible, judged by DISTS and VMAF rather than PSNR. Two +consequences follow: crf 18 on this path against crf 14 on the visible path is an +unexplained asymmetry and a free quality win, and the oracle's 100 MB limit means a +higher-quality profile is *harder* to certify than the one it replaces. + +Nothing is blocked on analysis. Six oracle submissions were prepared and never run, +because file upload to the verifier failed at the tool level: + +1. A Lyria carrier and the pipeline's output from it, answering the audio question. +2. A multiplexed control-plus-candidate file at the shipped geometry, validating + whether the verifier reads a concatenated pair by time range. Verified locally + frame by frame: half one is bit-identical to the control, half two to the + candidate. +3. Three more multiplexed pairs for 768, 1024 and 24 fps, worth three queries + instead of six if the instrument validates. + +Every generated artifact for those submissions has since been lost with its +scratch directory, which is the intended lifecycle for generated media. Rebuilding +costs one download and roughly 25 minutes of encoding; the Lyria track is the +exception, since regenerating it would produce a different track, and the copies +that survive are the only ones. + ## The single most important unknown **How much the 512 px downscale contributes to removal.** Every quality gain routes