Make the video SynthID operating point measurable and hard to move silently

The shipped profile was certified by one oracle row, but only noise_std was
pinned: long_side and fps -- two thirds of what the verifier was actually shown
-- could move with a green suite. The test now derives the pin from
data/evaluations/video-synthid-oracle.csv, so a default without a certifying row
fails.

The certified profile is a perturbation-to-signal ratio, not a bare noise_std.
sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the
AutoencoderKL class default under an upper-unbounded diffusers pin. The loader
now gates that value, carries it on VideoVaeRuntime, and passes it into encode
and decode so the validated value is the applied value. video_synthid_sweep.py
loads through the same function: the harness producing the certified rows was
the one path exempt from the gate it exists to feed.

psnr_db is measured against the already-resized frame and before the encoder, so
it cannot see the downscale, the decimation, or the codec, and no in-loop metric
can. scripts/video_fidelity_probe.py scores the delivered file end to end,
streaming the way the engine does and sharing its frame-selection rule rather
than copying it -- a frame-count check cannot catch a rule that reorders frames
without changing how many.

The manifest gains source geometry, vae, track, verbatim verdict and session
fields. The two 2026-07-31 rows keep them empty: they were never recorded and
are not recoverable. Verdicts now have four states, because the verifier's
unclear reading logged as not_detected is the silent regression the manifest
exists to prevent.

docs/video-synthid-quality-research.md records the research behind this: the
noise axis is worth about 2 dB and is nearly exhausted, resolution is the real
prize but is an uncertified destruction axis rather than a free win, and every
proposed autoencoder swap was refuted. First local measurements included.

Verified: engine output is byte-identical before and after the refactor on a
locally built clip, at noise_std 0.00 and 0.15.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
2026-08-05 11:17:38 -07:00
co-authored by Claude Opus 5
parent f481e6f944
commit 8fe0b0110f
14 changed files with 927 additions and 27 deletions
+29 -1
View File
@@ -252,7 +252,35 @@ quality measurements. Neither is a watermark detector. The high-level result
reports completed removal without a separate verification-status flag. The companion
`scripts/video_synthid_sweep.py` imports the same engine helpers to build a
matched control and candidate grid, preventing research and shipped
regeneration paths from drifting. The full-clip oracle floor is
regeneration paths from drifting.
The engine's `psnr_db` is measured against the already-resized frame and before
the encoder, so it scores the VAE round trip plus latent noise and cannot see
the downscale, the decimation, or the codec. No in-loop metric can: the
candidate frame is captured before it reaches the encoder pipe.
`scripts/video_fidelity_probe.py` covers the rest by decoding the delivered file
after muxing, upscaling it back to the source geometry, and scoring it against
the untouched source frames. It also reports the delivered file's bitrate, so a
fixed-crf bitrate rise cannot read as unchanged quality; because the mux copies
source audio verbatim, that figure is a container bitrate, not a video one. The
probe streams and accumulates the same way the engine does, so its peak memory
does not grow with clip length. It drives the source through the engine's own
`_iter_sampled_frames` at the source geometry rather than repeating the
selection rule: a frame-count check cannot catch a rule that reorders frames
without changing how many, so the rule itself has to be shared.
`load_video_vae_runtime` asserts the default model's latent scaling factor
against `VIDEO_SYNTHID_VAE_SCALING_FACTOR` and warns that no certified profile
exists for any other model. The published `sd-vae-ft-mse` config carries no
`scaling_factor` key, so the value is a `diffusers` class default under an
upper-unbounded pin, and the certified profile is a perturbation-to-signal ratio
rather than a bare `noise_std`. A library bump that moved that default would
otherwise rescale every perturbation with a green suite. The validated factor is
carried on `VideoVaeRuntime` and passed into encode and decode, so the gated
value and the applied value are one measurement rather than three independent
reads. `scripts/video_synthid_sweep.py` loads through the same function: the
harness that produces the certified rows is the last place that should be exempt
from the gate. The full-clip oracle floor is
`noise_std=0.15`: on the public eight-second Veo carrier, `0.10` remained
detected while `0.15` did not.