mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-09 15:36:01 +02:00
Victor KuznetsovandClaude Opus 5
8fe0b0110f
The shipped profile was certified by one oracle row, but only noise_std was pinned: long_side and fps -- two thirds of what the verifier was actually shown -- could move with a green suite. The test now derives the pin from data/evaluations/video-synthid-oracle.csv, so a default without a certifying row fails. The certified profile is a perturbation-to-signal ratio, not a bare noise_std. sd-vae-ft-mse publishes no scaling_factor key, so 0.18215 comes from the AutoencoderKL class default under an upper-unbounded diffusers pin. The loader now gates that value, carries it on VideoVaeRuntime, and passes it into encode and decode so the validated value is the applied value. video_synthid_sweep.py loads through the same function: the harness producing the certified rows was the one path exempt from the gate it exists to feed. psnr_db is measured against the already-resized frame and before the encoder, so it cannot see the downscale, the decimation, or the codec, and no in-loop metric can. scripts/video_fidelity_probe.py scores the delivered file end to end, streaming the way the engine does and sharing its frame-selection rule rather than copying it -- a frame-count check cannot catch a rule that reorders frames without changing how many. The manifest gains source geometry, vae, track, verbatim verdict and session fields. The two 2026-07-31 rows keep them empty: they were never recorded and are not recoverable. Verdicts now have four states, because the verifier's unclear reading logged as not_detected is the silent regression the manifest exists to prevent. docs/video-synthid-quality-research.md records the research behind this: the noise axis is worth about 2 dB and is nearly exhausted, resolution is the real prize but is an uncertified destruction axis rather than a free win, and every proposed autoencoder swap was refuted. First local measurements included. Verified: engine output is byte-identical before and after the refactor on a locally built clip, at noise_std 0.00 and 0.15. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
78 lines
3.9 KiB
Markdown
78 lines
3.9 KiB
Markdown
# Repository data
|
|
|
|
Tracked data is organized by purpose:
|
|
|
|
```text
|
|
data/
|
|
fixtures/
|
|
provenance/ Real format and provenance fixtures used by tests
|
|
calibration/
|
|
<vendor>/ Minimal controlled inputs needed to rebuild detector assets
|
|
synthid/
|
|
originals/ Canonical provider-oracle fixtures, stored once
|
|
manifest.csv Provenance and verification record for each original
|
|
full-pipeline-quality.csv
|
|
Reusable full-pipeline evaluation selection
|
|
evaluations/
|
|
fidelity/ Evaluation instructions and hand-verified ground truth
|
|
video-synthid-oracle.csv
|
|
Reproducible full-clip Gemini SynthID verdicts
|
|
```
|
|
|
|
## Storage rules
|
|
|
|
1. Store each binary image once. Evaluation manifests and documentation point
|
|
to its canonical location.
|
|
2. Put executable test fixtures in `fixtures/`.
|
|
3. Put only the minimal reproducible detector inputs in `calibration/`.
|
|
4. Put externally verified SynthID originals in `synthid/originals/` and keep
|
|
both CSV files synchronized.
|
|
5. Keep evaluation outputs outside the repository. Record reproducible
|
|
commands, hashes, and oracle verdicts instead of committing another corpus
|
|
copy. A small curated before-and-after example may live in `docs/images/`
|
|
when it is part of the public documentation.
|
|
6. Runtime detector assets belong in `src/remove_ai_watermarks/assets/`.
|
|
Unregistered research candidates belong in
|
|
`scripts/assets/visible-mark-candidates/` so they are not shipped in the
|
|
wheel.
|
|
|
|
The source distribution excludes `data/`; the wheel contains only package
|
|
runtime assets.
|
|
|
|
## Video SynthID oracle manifest
|
|
|
|
`evaluations/video-synthid-oracle.csv` is the only evidence that the shipped
|
|
video removal profile works, so it is also the source of truth for three shipped
|
|
defaults: `tests/test_video_invisible.py` asserts that `noise_std`, `long_side`,
|
|
and `fps` together match a row this manifest records as certified. Changing one
|
|
of those three without adding the row that certifies it fails the suite. `vae` is
|
|
deliberately outside that check because neither tracked row records one; add it
|
|
to the assertion in the same commit as the first row that does.
|
|
|
|
| Column | Meaning |
|
|
| --- | --- |
|
|
| `date`, `source_url`, `source_sha256` | Identify the carrier. |
|
|
| `source_width`, `source_height`, `source_fps` | Carrier geometry. Without it the actual downscale factor of a row cannot be recovered later. |
|
|
| `duration_seconds`, `source_verdict` | Clip length submitted and the verifier's reading of the untouched carrier. |
|
|
| `vae`, `noise_std`, `long_side`, `fps`, `seed` | The full run configuration. |
|
|
| `output_sha256` | Identifies the exact submitted file. |
|
|
| `output_verdict` | One of `detected`, `not_detected`, `indeterminate`, `refused`. |
|
|
| `output_verdict_text` | The verifier's wording, verbatim. |
|
|
| `output_detected_range` | Time range the verifier reported for the output. |
|
|
| `track` | `visual`, `audio`, `both`, or empty. The verifier scores tracks separately and this path copies source audio unchanged. |
|
|
| `session_id` | Groups rows submitted in one oracle session, so per-session drift stays visible. |
|
|
| `stratum` | Content class of the carrier, for stratified certification. |
|
|
| `psnr_db`, `temporal_residual_ratio` | Fidelity measurements. Neither is a watermark verdict. |
|
|
|
|
Record `indeterminate` when the verifier answers with its unclear state rather
|
|
than a negative: an unclear reading logged as `not_detected` is exactly the
|
|
silent regression this manifest exists to prevent. Leave a field empty when it
|
|
was not recorded, and never backfill it with a plausible value.
|
|
|
|
`psnr_db` is measured against the already-resized frame and before the encode,
|
|
so it excludes the downscale, the decimation, and the codec. Rows stay
|
|
comparable to each other only while that definition holds.
|
|
|
|
The two 2026-07-31 rows predate this schema; their empty fields were never
|
|
recorded and are not recoverable from the row.
|