Make the video SynthID operating point measurable and hard to move silently

The shipped profile was certified by one oracle row, but only noise_std was
pinned: long_side and fps -- two thirds of what the verifier was actually shown
-- could move with a green suite. The test now derives the pin from
data/evaluations/video-synthid-oracle.csv, so a default without a certifying row
fails.

The certified profile is a perturbation-to-signal ratio, not a bare noise_std.
sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the
AutoencoderKL class default under an upper-unbounded diffusers pin. The loader
now gates that value, carries it on VideoVaeRuntime, and passes it into encode
and decode so the validated value is the applied value. video_synthid_sweep.py
loads through the same function: the harness producing the certified rows was
the one path exempt from the gate it exists to feed.

psnr_db is measured against the already-resized frame and before the encoder, so
it cannot see the downscale, the decimation, or the codec, and no in-loop metric
can. scripts/video_fidelity_probe.py scores the delivered file end to end,
streaming the way the engine does and sharing its frame-selection rule rather
than copying it -- a frame-count check cannot catch a rule that reorders frames
without changing how many.

The manifest gains source geometry, vae, track, verbatim verdict and session
fields. The two 2026-07-31 rows keep them empty: they were never recorded and
are not recoverable. Verdicts now have four states, because the verifier's
unclear reading logged as not_detected is the silent regression the manifest
exists to prevent.

docs/video-synthid-quality-research.md records the research behind this: the
noise axis is worth about 2 dB and is nearly exhausted, resolution is the real
prize but is an uncertified destruction axis rather than a free win, and every
proposed autoencoder swap was refuted. First local measurements included.

Verified: engine output is byte-identical before and after the refactor on a
locally built clip, at noise_std 0.00 and 0.15.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
2026-08-05 11:17:38 -07:00
co-authored by Claude Opus 5
parent f481e6f944
commit 8fe0b0110f
14 changed files with 927 additions and 27 deletions
+27 -2
View File
@@ -33,6 +33,7 @@ from remove_ai_watermarks.video_synthid import (
DEFAULT_VIDEO_SYNTHID_NOISE_STD,
DEFAULT_VIDEO_SYNTHID_VAE,
VIDEO_SYNTHID_LATENT_MULTIPLE,
VIDEO_SYNTHID_VAE_SCALING_FACTOR,
)
from remove_ai_watermarks.video_temporal import (
_backward_map,
@@ -70,6 +71,9 @@ class VideoVaeRuntime:
requested_device: str
resolved_device: str
vae: Any
# The gate that validates this factor and the encode/decode calls that apply it
# must read one value, not three independent reads of the same attribute.
scaling_factor: float
def is_available() -> bool:
@@ -133,11 +137,22 @@ def load_video_vae_runtime(
vae = AutoencoderKL.from_pretrained(model, torch_dtype=dtype).to(resolved_device)
vae.eval()
vae.enable_slicing()
scaling_factor = float(vae.config.scaling_factor)
log.info("Latent scaling factor %.5f", scaling_factor)
if model != DEFAULT_VIDEO_SYNTHID_VAE:
log.warning("No oracle-certified profile exists for %s; the shipped noise_std is not calibrated for it", model)
elif scaling_factor != VIDEO_SYNTHID_VAE_SCALING_FACTOR:
raise RuntimeError(
f"{model} loaded with latent scaling factor {scaling_factor}, but the certified "
f"profile is defined against {VIDEO_SYNTHID_VAE_SCALING_FACTOR}. The perturbation "
"would be rescaled and the output would no longer match any certified row."
)
return VideoVaeRuntime(
model=model,
requested_device=device,
resolved_device=resolved_device,
vae=vae,
scaling_factor=scaling_factor,
)
@@ -262,12 +277,12 @@ def _encode_frame_latents(
vae: Any,
device: str,
batch_size: int,
scaling_factor: float,
) -> list[Any]:
"""Encode source frames once so every candidate can reuse identical latents."""
import torch
latent_batches: list[Any] = []
scaling_factor = float(vae.config.scaling_factor)
with torch.inference_mode():
for batch in _frame_batches(frames, batch_size):
rgb = np.stack([frame[:, :, ::-1] for frame in batch])
@@ -284,12 +299,12 @@ def _decode_frame_latents(
vae: Any,
noise_std: float,
shared_noise: Any,
scaling_factor: float,
) -> list[np.ndarray]:
"""Decode cached latents with one perturbation shared across time."""
import torch
output: list[np.ndarray] = []
scaling_factor = float(vae.config.scaling_factor)
with torch.inference_mode():
for latents in latent_batches:
perturbed = latents + noise_std * shared_noise.expand(latents.shape[0], -1, -1, -1)
@@ -411,9 +426,18 @@ def regenerate_video_candidate(
vae=vae,
device=resolved_device,
batch_size=batch_size,
scaling_factor=runtime.scaling_factor,
)
latents = latent_batches[0]
if shared_noise is None:
# Removal strength is the ratio of the perturbation to this spread,
# not noise_std alone: it is the only local quantity that makes two
# models' doses comparable.
log.info(
"First latent batch spread %.4f against noise_std %.4f",
float(latents.float().std()),
noise_std,
)
shared_noise = _shared_latent_noise(
latents.shape[1:],
seed=seed,
@@ -425,6 +449,7 @@ def regenerate_video_candidate(
vae=vae,
noise_std=noise_std,
shared_noise=shared_noise,
scaling_factor=runtime.scaling_factor,
)
for reference, candidate in zip(frames, regenerated, strict=True):
frame_pipe.write(candidate.tobytes())
@@ -1,6 +1,10 @@
"""Shared configuration for oracle-certified video SynthID removal."""
DEFAULT_VIDEO_SYNTHID_VAE = "stabilityai/sd-vae-ft-mse"
# The certified profile is a perturbation-to-signal ratio, so it is pinned against
# this latent scaling factor as much as against noise_std. Rationale and the drift
# it guards against: docs/module-internals.md.
VIDEO_SYNTHID_VAE_SCALING_FACTOR = 0.18215
DEFAULT_VIDEO_SYNTHID_NOISE_STD = 0.15
DEFAULT_VIDEO_SYNTHID_LONG_SIDE = 512
DEFAULT_VIDEO_SYNTHID_FPS = 12.0