Victor KuznetsovandClaude Opus 5
8fe0b0110f
The shipped profile was certified by one oracle row, but only noise_std was pinned: long_side and fps -- two thirds of what the verifier was actually shown -- could move with a green suite. The test now derives the pin from data/evaluations/video-synthid-oracle.csv, so a default without a certifying row fails. The certified profile is a perturbation-to-signal ratio, not a bare noise_std. sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the AutoencoderKL class default under an upper-unbounded diffusers pin. The loader now gates that value, carries it on VideoVaeRuntime, and passes it into encode and decode so the validated value is the applied value. video_synthid_sweep.py loads through the same function: the harness producing the certified rows was the one path exempt from the gate it exists to feed. psnr_db is measured against the already-resized frame and before the encoder, so it cannot see the downscale, the decimation, or the codec, and no in-loop metric can. scripts/video_fidelity_probe.py scores the delivered file end to end, streaming the way the engine does and sharing its frame-selection rule rather than copying it -- a frame-count check cannot catch a rule that reorders frames without changing how many. The manifest gains source geometry, vae, track, verbatim verdict and session fields. The two 2026-07-31 rows keep them empty: they were never recorded and are not recoverable. Verdicts now have four states, because the verifier's unclear reading logged as not_detected is the silent regression the manifest exists to prevent. docs/video-synthid-quality-research.md records the research behind this: the noise axis is worth about 2 dB and is nearly exhausted, resolution is the real prize but is an uncertified destruction axis rather than a free win, and every proposed autoencoder swap was refuted. First local measurements included. Verified: engine output is byte-identical before and after the refactor on a locally built clip, at noise_std 0.00 and 0.15. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
3.9 KiB
Repository data
Tracked data is organized by purpose:
data/
fixtures/
provenance/ Real format and provenance fixtures used by tests
calibration/
<vendor>/ Minimal controlled inputs needed to rebuild detector assets
synthid/
originals/ Canonical provider-oracle fixtures, stored once
manifest.csv Provenance and verification record for each original
full-pipeline-quality.csv
Reusable full-pipeline evaluation selection
evaluations/
fidelity/ Evaluation instructions and hand-verified ground truth
video-synthid-oracle.csv
Reproducible full-clip Gemini SynthID verdicts
Storage rules
- Store each binary image once. Evaluation manifests and documentation point to its canonical location.
- Put executable test fixtures in
fixtures/. - Put only the minimal reproducible detector inputs in
calibration/. - Put externally verified SynthID originals in
synthid/originals/and keep both CSV files synchronized. - Keep evaluation outputs outside the repository. Record reproducible
commands, hashes, and oracle verdicts instead of committing another corpus
copy. A small curated before-and-after example may live in
docs/images/when it is part of the public documentation. - Runtime detector assets belong in
src/remove_ai_watermarks/assets/. Unregistered research candidates belong inscripts/assets/visible-mark-candidates/so they are not shipped in the wheel.
The source distribution excludes data/; the wheel contains only package
runtime assets.
Video SynthID oracle manifest
evaluations/video-synthid-oracle.csv is the only evidence that the shipped
video removal profile works, so it is also the source of truth for three shipped
defaults: tests/test_video_invisible.py asserts that noise_std, long_side,
and fps together match a row this manifest records as certified. Changing one
of those three without adding the row that certifies it fails the suite. vae is
deliberately outside that check because neither tracked row records one; add it
to the assertion in the same commit as the first row that does.
| Column | Meaning |
|---|---|
date, source_url, source_sha256 |
Identify the carrier. |
source_width, source_height, source_fps |
Carrier geometry. Without it the actual downscale factor of a row cannot be recovered later. |
duration_seconds, source_verdict |
Clip length submitted and the verifier's reading of the untouched carrier. |
vae, noise_std, long_side, fps, seed |
The full run configuration. |
output_sha256 |
Identifies the exact submitted file. |
output_verdict |
One of detected, not_detected, indeterminate, refused. |
output_verdict_text |
The verifier's wording, verbatim. |
output_detected_range |
Time range the verifier reported for the output. |
track |
visual, audio, both, or empty. The verifier scores tracks separately and this path copies source audio unchanged. |
session_id |
Groups rows submitted in one oracle session, so per-session drift stays visible. |
stratum |
Content class of the carrier, for stratified certification. |
psnr_db, temporal_residual_ratio |
Fidelity measurements. Neither is a watermark verdict. |
Record indeterminate when the verifier answers with its unclear state rather
than a negative: an unclear reading logged as not_detected is exactly the
silent regression this manifest exists to prevent. Leave a field empty when it
was not recorded, and never backfill it with a plausible value.
psnr_db is measured against the already-resized frame and before the encode,
so it excludes the downscale, the decimation, and the codec. Rows stay
comparable to each other only while that definition holds.
The two 2026-07-31 rows predate this schema; their empty fields were never recorded and are not recoverable from the row.