Files
remove-ai-watermarks/docs/video-synthid-quality-research.md
T
Victor KuznetsovandClaude Opus 5 8fe0b0110f Make the video SynthID operating point measurable and hard to move silently
The shipped profile was certified by one oracle row, but only noise_std was
pinned: long_side and fps -- two thirds of what the verifier was actually shown
-- could move with a green suite. The test now derives the pin from
data/evaluations/video-synthid-oracle.csv, so a default without a certifying row
fails.

The certified profile is a perturbation-to-signal ratio, not a bare noise_std.
sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the
AutoencoderKL class default under an upper-unbounded diffusers pin. The loader
now gates that value, carries it on VideoVaeRuntime, and passes it into encode
and decode so the validated value is the applied value. video_synthid_sweep.py
loads through the same function: the harness producing the certified rows was
the one path exempt from the gate it exists to feed.

psnr_db is measured against the already-resized frame and before the encoder, so
it cannot see the downscale, the decimation, or the codec, and no in-loop metric
can. scripts/video_fidelity_probe.py scores the delivered file end to end,
streaming the way the engine does and sharing its frame-selection rule rather
than copying it -- a frame-count check cannot catch a rule that reorders frames
without changing how many.

The manifest gains source geometry, vae, track, verbatim verdict and session
fields. The two 2026-07-31 rows keep them empty: they were never recorded and
are not recoverable. Verdicts now have four states, because the verifier's
unclear reading logged as not_detected is the silent regression the manifest
exists to prevent.

docs/video-synthid-quality-research.md records the research behind this: the
noise axis is worth about 2 dB and is nearly exhausted, resolution is the real
prize but is an uncertified destruction axis rather than a free win, and every
proposed autoencoder swap was refuted. First local measurements included.

Verified: engine output is byte-identical before and after the refactor on a
locally built clip, at noise_std 0.00 and 0.15.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-05 11:17:38 -07:00

494 lines
29 KiB
Markdown

# Video SynthID quality research (2026-08-05)
> Research archive. This page records experiments and decisions from the date
> above. It may mention prototypes or defaults that were later changed. Use the
> user guides and current source code for the supported interface.
Cited research behind the question **"can the video SynthID path keep removing the
mark while giving up far less quality than the shipped 512 px / 12 fps profile?"**
Produced by a 13-agent workflow: 6 parallel scouts (pipeline ablation, SynthID
internals, attack literature, autoencoder landscape, perceptual masking, experiment
design), 11 change proposals from 3 independent design angles, and 3 adversarial
critics (signal theory, repository engineering, verifiability). No experiment was
run against the provider oracle for this page; every claim is labeled MEASURED,
REPORTED, or INFERRED.
Repository claims below were re-verified against source after the workflow
returned: the PSNR reference frame, the audio stream copy, the missing `_fit_size`
clamp, the absent `enable_tiling` call, the crf asymmetry, the test pins, and the
absent `scaling_factor` key in the cached checkpoint config.
## Context
`remove_video_invisible` regenerates video pixels through `sd-vae-ft-mse`: frames
are decimated to 12 fps, resized to a 512 px long side, encoded to latents with
`latent_dist.mode()`, perturbed by one seeded spatial noise field shared across
every frame, decoded, and streamed to an H.264 encoder at crf 18. A separate
stream-copy mux re-adds source audio and strips metadata. The shipped
`noise_std=0.15` is certified by one oracle row on one carrier.
The quality complaint is real: a 1080p source is delivered at roughly a quarter of
its linear resolution and half its frame rate. This page asks what part of that
cost is buying removal and what part is buying nothing.
## What the two oracle rows do and do not prove
`data/evaluations/video-synthid-oracle.csv`, one carrier (Veo 3 off-road,
sha256 `79a552b9...`), one seed, one geometry:
| noise_std | long_side | fps | verdict | psnr_db | temporal_residual_ratio |
| --- | --- | --- | --- | --- | --- |
| 0.10 | 512 | 12 | detected | 26.2932 | 1.0072 |
| 0.15 | 512 | 12 | not_detected | 25.3911 | 1.0578 |
They prove exactly one thing: the envelope of 512 px, 12 fps, crf 18, and a full
VAE round trip does **not** silence the oracle on its own. The verdict flip is
bought by the last 0.05 of latent noise.
They cannot decompose that conjunction. No row varies `long_side` or `fps`, so the
contribution of the downscale is unmeasured. The frequent reading "the 512 px
downscale was probably doing the removal work" is not supported, and neither is its
opposite.
## Measured facts from the repository
- **The fidelity metric is blind to the expensive steps.** `_iter_sampled_frames`
applies `cv2.resize` at [`video_invisible.py:236`](../src/remove_ai_watermarks/video_invisible.py);
the same generator feeds the loop at `:401-407`; the accumulator at `:429-435`
zips those already-resized frames against `regenerated` captured at `:423`, which
is before `frame_pipe.write` at `:430`. `psnr_db` therefore excludes the
downscale, the decimation, and the encoder. It measures the VAE round trip plus
latent noise at the working geometry, nothing else.
- **Frame decimation cannot destroy the carrier.** `_iter_sampled_frames`
(`:224-238`) is pure subset selection: the only statement inside the threshold
test is `yield cv2.resize(...)`, with no arithmetic across frames. The
perturbation is equally time-blind: `shared_noise.expand(latents.shape[0], -1, -1, -1)`
at `:295` broadcasts one CHW field across the batch and never mixes across time.
Any fps effect on the verdict is a detection-probability effect, not carrier
destruction.
- **`_fit_size` has no clamp at 1.0.** `scale = long_side / max(width, height)` at
`:88` upscales any source whose long side is below 512, and both axes are floored
independently to a multiple of 8 at `:89-96`, which introduces an anamorphic
shift at scale 1.0 (1366x768 becomes 1360x768).
- **`enable_tiling` is never called** anywhere in `src/`, `scripts/`, or `tests/`.
`enable_slicing()` at `:135` already splits both encode and decode to single
frames, so activation VRAM is set by one frame and does not scale with
`--batch-size`.
- **crf asymmetry:** 18 on the invisible path (`:321`, `:386`) against 14 on the
visible path ([`video_visible.py:1343`](../src/remove_ai_watermarks/video_visible.py))
through the same encoder.
- **No HDR guard on the invisible path.** `_HDR_TRANSFERS` (`video_visible.py:97`)
and the `component_depth > 8` rejection (`:1322`) exist only for visible removal.
- **Audio is byte-copied.** `-map 0:v:0`, `-map 1:a?`, `-c copy` at
[`video_encoding.py:413`](../src/remove_ai_watermarks/video_encoding.py).
- **Only `noise_std` is pinned.** `tests/test_video_invisible.py:285` asserts
`DEFAULT_VIDEO_SYNTHID_NOISE_STD == 0.15`. Neither `long_side` nor `fps` is
pinned anywhere, so changing either breaks no test.
- **`scaling_factor` 0.18215 is a class default, not a checkpoint fact.** The
cached `config.json` for `stabilityai/sd-vae-ft-mse` has no `scaling_factor` key
at all (verified locally: `_class_name`, `latent_channels`, `block_out_channels`,
`sample_size`, and block types only). The value comes from the `AutoencoderKL`
class default under `diffusers>=0.38.0` with no upper bound (`pyproject.toml:94`,
`uv.lock` resolves 0.39.0), while `maintain.sh` runs `uv-outdated`. A dependency
bump can move the certified operating point with a fully green test suite.
- **The manifest schema cannot record what a real program needs:** no source
geometry, no control row, no track column, no verbatim verdict, no indeterminate
state.
## Measured facts from external sources
- Gemini video verification quota: 10 checks per rolling 24 hours, up to 5 minutes
of video total, under 90 seconds and 100 MB per file. The verifier reports which
parts of the video carry the mark, and it has a **third** state beyond detected
and not detected: unclear, with documented causes including "not enough details
to watermark".
- The verifier scores **audio and visual tracks separately**. Google's published
example verdict reads as SynthID detected in the audio over a time range with no
SynthID detected in the visuals.
- SynthID-Image (arXiv:2510.09263) is a post-hoc, model-independent pixel-space
watermark: a separate encoder network stamps an already-decoded image rather than
being injected into the generator's latents. Its published payload figure is 136
bits within a 512x512 image, and its product setup runs at 1536x1536.
## Inferred, with the reasoning that makes them weak
- **The noise axis is nearly exhausted.** Fitting `MSE = A + B * noise_std^2` to the
two measured rows gives A = 124.5 and B = 2815, so a pure round trip
(`noise_std = 0`) lands near 27.2 dB. The entire noise budget is worth at most
**+1.79 dB**; everything else is the autoencoder. This is a two-parameter fit to
two points with zero degrees of freedom, and the assumptions that MSE is
quadratic in `noise_std` and that reconstruction and noise errors are orthogonal
are untested. One local run at `noise_std=0` replaces it with a measurement.
- **Statistical power of the current certification.** With zero failures at n = 1,
the exact one-sided Clopper-Pearson bound `1 - 0.05^(1/n)` is 95%: the data are
compatible with removal failing almost always. n = 15 gives 18.1%, n = 30 gives
9.5%.
- **Size of the geometry prize.** At a 1280x720 source, `_fit_size(1280, 720, 512)`
is `(512, 288)` (pinned in `tests/test_video_synthid_sweep.py:30`), so 6.25x of
the pixels are discarded by the downscale and another 2x by decimation from a
24 fps source. Ladder rungs: 768 gives 2.25x the current area, 1024 gives 4.0x.
The carrier's own geometry is recorded nowhere, so this is conditional.
- **The direction of the resolution effect is disputed, and both sides are
inference.** Against raising it: the absolute frequency ceiling below which an f8
VAE reconstructs faithfully is tied to the latent pitch, so 512x288 (a 64x36
latent) preserves roughly up to 32 cycles per frame width while 1920 (a 240x135
latent) preserves up to about 120. Raising resolution moves the carrier band out
of the regime the decoder synthesizes and into the regime it reproduces
faithfully, handing the detector more evidence. For raising it: Google's product
operating point is 1536x1536, so native is closer to the distribution the
watermark encoder targets. Note that the second argument cuts against the
proposal rather than for it.
## First local measurements (2026-08-05)
Run without the oracle on a locally built carrier: a 6-second 1280x720 24 fps
clip panning across `data/fixtures/provenance/doubao-1.png`, processed by the
shipped path at 512 px / 12 fps on MPS. Generated media stayed outside the
repository.
| noise_std | engine `psnr_db` | end-to-end PSNR | end-to-end SSIM | bitrate |
| --- | --- | --- | --- | --- |
| 0.00 | 27.8049 | 27.6935 | 0.6955 | 2564 kbps |
| 0.15 | 25.8853 | 25.8541 | 0.6703 | 2716 kbps |
Three readings, all MEASURED, none of them about SynthID:
1. **The two-point fit's shape survives contact with independent content.** The
pure round trip lands at 27.80 dB against the 27.2 dB the carrier fit
predicted, and the whole noise budget costs 1.92 dB against the predicted
1.79 dB. These are not the carrier's numbers, but the decomposition holds:
the autoencoder is the floor and the entire `noise_std` axis is worth about
2 dB.
2. **On this content the downscale is nearly free in PSNR terms.** End-to-end
PSNR sits within 0.11 dB of the in-loop number, meaning the 512 px geometry
cost almost nothing next to the VAE damage. This clip is a pan over a smooth
generated image with little high-frequency detail, so it is the friendly
case: real camera texture should widen that gap. Run the probe across content
types before trusting any estimate of the geometry prize.
3. **`temporal_residual_ratio` is not meaningful on a near-static shot.** It read
1.82 at `noise_std=0` and 2.47 at 0.15, far outside the [1.0072, 1.0578] band
ever observed before. A slow pan gives the source almost no motion residual,
so the `max(temporal_baseline, 1e-6)` denominator collapses and the ratio
inflates. This is the predicted defect, now observed rather than argued.
## The single most important unknown
**How much the 512 px downscale contributes to removal.** Every quality gain routes
through this question, and the critics moved it from "probably free" to "direction
unknown", which is exactly what makes it the highest-information experiment
available.
## Ranked experiment program
Ranked by information per oracle query. The budget is roughly 10 checks per 24
hours (MEASURED), so the program is paced by calendar, not by money.
### E0. The zero-oracle tier (0 submissions) - do this first
Not an experiment on the oracle; the precondition that makes everything else
interpretable.
- `ffprobe` the carrier and record `source_width`, `source_height`, `source_fps`.
The actual downscale factor of the existing rows is currently unrecoverable.
- Extend the manifest schema: variant/control, source geometry, `vae`, `track`
(audio|visual), verdict state in {DETECTED, NOT_DETECTED, INDETERMINATE,
REFUSED}, verbatim verdict text, detected time range for the output,
`session_id`, content stratum.
- **Measure the `noise_std = 0.0` round trip locally** on the same carrier. This
converts the inferred 27.2 dB ceiling into a measurement and bounds the whole
noise axis for one GPU pass and zero oracle cost.
- Freeze the current defaults' metrics on a fixed clip set, matching the
record-then-diff rule in `.claude/rules/development.md`.
- Determine whether the carrier has an audio track and whether it carries SynthID.
### E1. Instrument validation: multiplexing and a session anchor (1-2 submissions)
The verifier reports time ranges, so one file can carry several doses. A 24-second
file of `[control 8 s | certified 0.15 8 s | control 8 s]` should read detected on
the outer segments and not detected on the middle one, both halves already known
from the manifest.
`encode_video_frames` requires matching dimensions and one frame rate, so
multiplexing works **only along the `noise_std` axis**, not across geometries: one
multiplexed file per geometric envelope.
Separately, submitting the existing detected file first in each session costs 10%
of the quota, turns "the oracle may have changed" into a per-session gate, and
measures the flip rate on a byte-identical file, a quantity every plan silently
assumes is zero.
### E2. Resolution ladder at fixed dose - the decisive axis (2 submissions per rung)
A fixed value, not a ceiling. Rung 1 is `long_side = 768` with fps held at 12 so
exactly one destruction axis moves. Jumping straight to 1920 is a bad first rung:
a 3.75x jump makes a detected verdict uninformative about where the boundary lies.
- **Matched control:** 768 / 12 / `noise_std = 0.10`, expected DETECTED. It is
strictly stronger than a plain re-encode control because it discharges the
envelope, the VAE round trip, and a nonzero dose at once, and it diffs directly
against the existing 512 row.
- **Candidate:** 768 / 12 / 0.15.
- **Positive:** the resolution axis is open; next rungs 1024 and then native with a
clamp, 2 submissions each.
- **Negative:** the downscale contributes to removal. This is the most valuable
available negative: it kills the native-resolution proposals outright and turns
the question into how much dose must be spent to buy resolution back, at the
known exchange rate B = 2815.
- Watch for a monotonicity violation: 768 / 0.10 reading NOT_DETECTED would
overturn the critics' spectral argument.
### E3. Frame rate at fixed geometry and dose (2 submissions)
Control 512 / 24 / 0.10 expected DETECTED; candidate 512 / 24 / 0.15 expected
not detected. The mechanism is provable from source, so the prior is high, but the
only counter-mechanism - per-frame count aggregation - is probabilistic and n = 1
on an 8-second clip does not measure it. Certify on the longest clip that fits the
90 s / 100 MB limits, because the risk compounds with frame count.
Ranked below E2 because the outcome is nearly predetermined and the prize (judder)
is smaller than the geometry prize. It is the safest bet if a guaranteed win is
wanted.
### E4. `noise_std = 0.0` as an oracle probe (1 submission) - deliberately low rank
The local half of this probe is the cheapest high-information measurement in the
program and already sits in E0 at zero oracle cost. The **oracle** half ranks low:
it buys no quality by construction, and its likely DETECTED outcome is nearly
deducible from the 0.10 row under monotonicity. Its one real value is that
NOT_DETECTED here would be non-monotone against 0.10 and would refute the
assumption underneath every ladder and bisection in this program. Worth one query,
after E2 and E3.
### E5. Stratified certification of the surviving operating point (15-30 submissions)
Only after E2 and E3 produce a winner. Zero detections across at least 15
stratified carriers (at least 3 per stratum: face closeup, fast motion, flat sky,
dark night, text overlay) bounds the failure rate at 18.1%; 30 carriers bound it at
9.5%. Add the project's 1.5x margin convention, at least 2 seeds, at least 2
sessions, and per-track verdict recording.
For the flat-sky stratum, check the source reads detected first: Google documents a
"not enough details to watermark" state, so a clean reading there may say nothing
about removal.
## Surviving change candidates
**S1. Fix the metric's reference frame (0 oracle submissions).** Add
`source_psnr_db` (candidate upscaled back to native against the untouched source
frame) **and** a separate post-mux pass that decodes the delivered file. The
critics killed the original claim that this would expose crf 18: the candidate is
captured at `:423`, before `frame_pipe.write` at `:430`, so no in-loop metric can
see the codec. Only the post-mux pass covers resize, decimation, crf, and mux
together. Use `INTER_AREA` or a fixed analytic size for the upscale so the CPU
resize does not dominate the loop, and update both other consumers of the
generator in the same change (`read_sampled_frames` at `:188-209`, and
`scripts/video_synthid_sweep.py:147` where `np.stack(frames)` must keep receiving
resized frames). Do not redefine the existing `psnr_db`: the two manifest rows stay
comparable to each other only while that field keeps its exact current meaning.
**S2. Frame rate as a fixed certified value (after E3).**
`DEFAULT_VIDEO_SYNTHID_FPS` 12.0 to 24.0, keeping `min(fps, source_fps)`. A
ceiling of 60.0 was **rejected**: it makes the delivered operating point a function
of the user's source, so one sample from a family gets certified while the rest
ship uncertified, and the risk direction is unfavorable. Port the VFR/PTS bridge
(`probe_video_timestamps`, `timestamped_input`, currently only in
`video_visible.py`): at native frame rate the output stops reading as a proxy, and
silent CFR-ification becomes a master-quality defect. Add `fps` and `long_side`
pins next to `tests/test_video_invisible.py:285`, tied to the certifying manifest
row.
**S3. Clamp and align `_fit_size` (0 oracle submissions, with a caveat).**
`scale = min(1.0, long_side / max(width, height))` at `:88`; align the long side
and derive the short side from the true aspect, rounding to the nearest multiple of
8. The pinned `_fit_size(1280, 720, 512) == (512, 288)` survives. The caveat: the
clamp is an **uncertified operator change for a whole class of users**, since a
320x240 source is upscaled to 512x384 today and would run at 320x240 afterwards,
and neither has been tested. Ship it as its own documented change, not as a
drive-by fix.
**S4. Extend the manifest schema and oracle protocol (0 submissions).** A
precondition for the whole program. Record the verbatim verdict text, because an
unclear state logged as not detected is exactly the silent regression the protocol
exists to prevent.
**S5. A sigma normalization contract (0 submissions, applies to the current path).**
No VAE swap survived, but two elements are real risks today: `vae.config.scaling_factor`
cannot be trusted, so log `latents.std()` at `:276` and assert the effective
scaling factor at load; and any cross-model sigma transfer must match on the **RMS
of the decoded pixel perturbation**, not on latent units.
**S6. Additive texture masking - contingency on an E2 negative only.** Not energy
preserving: the certified sigma stays as a floor everywhere and the mask only adds
on top in textured regions. The original energy-preserving form was refuted - the
mechanism normalizes and then clips, which breaks the claimed invariant by a
content-dependent amount, and the proposed unit test asserted the invariant before
the clip and would have stayed green. Energy preservation is also unsafe in
principle: the detector does not need a full frame, and the verifier reports
per-segment, so a flat region perturbed only at the floor is a crop carrying nearly
the full carrier. The salvaged version has **no standalone PSNR win**; its only
value is freeing distortion budget to spend on resolution if E2 shows the dose must
rise. It additionally needs motion compensation for the mask (flow is computed at
`:442`, after the decode at `:423`, so the loop must be restructured) and one
deliberately bad candidate so `temporal_residual_ratio` acquires a known failing
value: it has only ever been observed in [1.0072, 1.0578], so "the ratio looks
fine" is currently an unfalsified claim.
## Refuted proposals - do not resurrect
- **Carrier-scale, resolution-invariant noise field.** "Carrier scale" is set by
our VAE's latent pitch, not the carrier's. The motivating arithmetic treats a
field on the latent grid as white at pixel resolution and is wrong by exactly the
square of the scale factor: white noise on a 240-wide latent already spans
0-120 cycles per frame, entirely inside the band the decoder reproduces. The
construction actually band-limits the perturbation from 0-120 to 0-32 cycles,
making it smoother - the first thing any spread-spectrum extractor's content
suppression removes - while moving its energy toward the peak of the contrast
sensitivity function, making it more visible. Bilinear upsampling is also
heteroscedastic, and global renormalization by `noise.std()` stamps a visible
amplitude lattice rather than fixing it.
- **Native resolution plus tiling plus a carrier-scale field.** Inherits the
arithmetic above, concedes that native processing destroys less carrier without
proposing a replacement mechanism, and bundles four destruction axes into one
candidate on a 4-query budget so a detected verdict is undecomposable. Its cost
model was also wrong by roughly 10x. The salvageable parts moved into S1 and S3.
- **`AutoencoderKLTemporalDecoder`.** Refuted independently by three critics.
Removal is a property of the encode-decode composition, so a strictly more
faithful decoder preserves more carrier at the same sigma. The class implements
neither slicing nor tiling, so `enable_slicing()` at `:135` raises
`NotImplementedError` and the documented bounded-memory property dies. Its
temporal receptive field needs windows of tens of frames, its license carries a
revenue restriction incompatible with an Apache-2.0 default, and its own
published table shows a **worse** FID (9.17 against 7.61).
- **Wan or another temporal video VAE.** Hard refusal in code: `AutoencoderKLWan._encode`
uses `iter_ = 1 + (num_frame - 1) // 4`, so at the shipped `batch_size = 4` only
frame 0 is encoded and a 4-frame batch decodes back to 1, which makes
`zip(..., strict=True)` at `:429` raise. The proposed remedy of carrying the
causal feature cache across chunks is impossible through the public API, since
`clear_cache()` runs on both entry and exit of `_encode` and `_decode`.
`float(vae.config.scaling_factor)` at `:270` and `:292` also raises, because Wan
exposes `latents_mean`/`latents_std` instead. Logically, feeding 4:1 temporal
compression frames 83 ms apart either reconstructs well (carrier preserved, no
removal gain) or hallucinates (no quality prize); the two claims are mutually
exclusive.
- **A 16-channel f8 VAE.** Self-refuted and confirmed by the critics. Quadrupling
latent channels roughly doubles L in Zhao's theorem, so sigma must roughly double
to hold removal, and the VAE frontier in the UnMarker results is monotone with no
published point where a more faithful autoencoder removes **more**. Predicted net
loss of 2-7.5 dB by its own arithmetic. Its config also carries
`scaling_factor = 0.2614`, so an unchanged `noise_std = 0.15` is about 30% weaker
in raw latent units. Only the normalization contract survived, as S5.
- **A per-channel luma/chroma probe.** The arms match in absolute latent units but
not in relative dose per channel, since `scaling_factor` normalizes the aggregate
latent, so the very reading the probe exists for is confounded with dose. The
chroma arm is additionally subsampled by the yuv420p encode, which no current
metric sees. Maximum prize under 1 dB by its own fit.
- **Post-processing with unsharp plus grain, as a shipped stage.** The
carrier-reimport argument is sound and was verified line by line: `unsharp_mask`
reads only its own argument ([`humanizer.py:79`](../src/remove_ai_watermarks/humanizer.py)),
and `adaptive_polish` touches the source only through one float. But reimport was
never the binding risk:
`cv2.addWeighted(img_f, 1.0 + amount, blurred, -amount, 0.0)` at
`_ADAPTIVE_MAX_UNSHARP = 1.0` multiplies the **surviving carrier residue** by up
to 2x as the terminal operation before encoding, and whether that re-arms a given
file depends on that file's unobservable margin. With no local decoder and only a
sampled oracle the stage is structurally uncertifiable, not merely expensive to
certify. The only rescue is moving sharpening **above** the VAE stage so the
removal operator stays terminal. Grain, which monotonically lowers detector SNR,
is the safe half and can be separated.
## Measuring quality properly
The current `psnr_db` cannot show the improvement this work exists to produce.
Minimum upgrade:
1. **`source_psnr_db`** - candidate upscaled back to native against the untouched
source frame. Good for ranking configurations against a common reference,
dominated by unrecoverable high-frequency content, so not a measure of what the
VAE costs.
2. **A post-mux end-to-end pass** - decode the delivered file after
`mux_encoded_video` (`:456`) and compare against the source. The only measure
covering resize, decimation, crf, and mux together. Keep it out of the streaming
loop so `test_stream_batches_consumes_only_one_batch_ahead` stays valid.
Implemented as `scripts/video_fidelity_probe.py`, which streams, reports the
delivered file's bitrate, and shares the engine's frame-selection rule rather
than copying it.
3. **DISTS** (arXiv:2004.07728), built to tolerate texture resampling - exactly
what the VAE does to foliage, skin, and grass, and exactly what PSNR and LPIPS
punish even when the result is perceptually equivalent. It separates "the VAE
resampled the grass" from "the VAE destroyed an edge".
4. **VMAF** through `ffmpeg` `libvmaf`, cheapest to add, whose ADM/DLM feature
names this exact complaint. Its temporal term is only a mean absolute luma
difference between neighboring frames, so it is not a flicker detector: keep the
motion-compensated ratio.
5. **Encoded file size as a third axis.** Every current metric is taken before the
pipe, so a bitrate explosion currently reads as "quality did not suffer". This
matters most for any perturbation that varies frame to frame: at fixed crf it
raises bitrate rather than lowering quality.
6. **`temporal_residual_ratio` repairs**: add a p95 across frames next to the mean
so localized flicker stops being averaged away, add a long-horizon term against
the first frame, and replace `max(temporal_baseline, 1e-6)` with an explicit
undefined result on static shots.
7. **A no-reference metric** (DOVER or FAST-VQA) for the case where no
full-reference measure can compare a 512/12 output against a 1024/24 output on
one scale. Relative ranking within a sweep only; absolute values are
uncalibrated for this artifact class.
Nothing local measures removal. `remove_video_invisible` checks `get_ai_metadata`
on the output, which is metadata, not pixels. Only a manifest row counts.
## Risks
**Error asymmetry is the governing constraint.** Shipping a leak means a user
receives a watermarked file believing it is clean, with no local pixel decoder to
catch it and no feedback path that would surface it. Staying conservative means a
user receives 512 px / 12 fps, a cost that is visible, bounded, and reversible with
an explicit flag. A symmetric test is therefore inadmissible, and the burden of
proof sits entirely on the new default.
**n = 1.** The current default is certified by one success on one carrier with one
seed. Any quality change inherits that weakness and must not deepen it.
**Content and seed dependence** is MEASURED on this project's image branch:
survivors switch by content type, and near the threshold the same input flipped
between runs on seed alone.
**Oracle instability.** Three verdict states, not two. Per-segment reporting.
Separate audio and visual tracks - and the pipeline byte-copies audio, so a Veo 3
clip with generated sound leaves with its audio SynthID intact, and the manifest
cannot even record which track the 2026-07-31 negative referred to. That is
potentially a shipped product hole, not only an experimental confound. Session
drift is currently unfalsifiable.
**Observability narrows exactly where quality rises.** At fixed crf the bitrate
grows with pixels per second, so the maximum duration fitting under 100 MB falls as
fps and resolution rise. The configuration users actually receive becomes **less**
observable than the one it replaces. That is a permanent property, not an
inconvenience.
**Ceilings instead of fixed values.** Any ceiling makes the delivered operating
point a function of the user's source, and with it the perturbation's cycles per
frame - the quantity carrying the certified margin. Ship fixed values.
**Documentation and test surface.** Only `DEFAULT_VIDEO_SYNTHID_NOISE_STD` is
pinned. Moving `long_side` or `fps` touches hardcoded numbers in `README.md`,
`docs/known-limitations.md`, `docs/cli.md`, `docs/python-api.md`,
`docs/module-internals.md`, `docs/synthid.md`, and `docs/verification-plan.md`.
Drift-prone operational numbers should not live in seven places of prose; their
source of truth is the manifest.
**Silent configuration drift.** `sd-vae-ft-mse/config.json` carries no
`scaling_factor`, so 0.18215 is a class default under an upper-unbounded
`diffusers>=0.38.0` while `maintain.sh` runs `uv-outdated`. A library bump can move
the certified operating point with a green suite.
**Missing guards that get worse at native resolution.** There is no HDR or >8-bit
rejection and no VFR/PTS bridge on the invisible path. A 512 px output plainly
reads as a proxy; a native-resolution output reads as a master, and the cost of
silently flattening a 10-bit PQ source to 8-bit SDR rises accordingly.
**Do not bundle axes.** Raising `long_side`, raising `fps`, and lowering crf all
reduce total destruction and all require recertification. The metric reference-frame
fix is the only exception, because it changes the measurement rather than the
pixels. Do not put a crf 18 to 14 change in the same candidate as a resolution
change.