0 Fork 0

mirror of https://github.com/wiltodelta/remove-ai-watermarks.git synced 2026-10-15 05:50:45 +02:00

Files

T

Victor Kuznetsov 464fd142ea Add audio and video watermark benchmark oracles, cohorts, and research studies

Extend the development-only benchmark family from images to audio and
video, and close four research directions from the watermark-landscape
map with local, revision-pinned evidence.

Benchmark schema: media_type audio and video rows, audioseal and
videoseal adapters with media-checked adapter binding, sample-domain
audio fidelity (SNR with unbounded_identical), frame-domain video
fidelity (mean PSNR, changed-frame fraction), and per-frame temporal
evidence in videoseal detection records. The report accepts all three
media types and carries SNR columns.

Audio: shared pinned AudioSeal oracle, synthetic and system-voice
speech cohorts with matched negatives, codec, noise, removed, and
forged arms, and the provenance experiment proving visual-only video
cleaning leaves the audio bitstream and watermark verdict invariant.

Video: TorchScript-only VideoSeal oracle (the pip package is unusable
on macOS arm64 and its loader resolves paths relative to cwd), video
cohort with crf/scale/fps attacks and four-state arms, temporal
evaluation measuring the crf decay curve, the frame-aggregation matrix
(computed locally after finding the TorchScript build ignores the
aggregation argument), and real provider clips as carriers.

Forgery: double-embedding study showing the last writer wins completely
under both oracles, scoped remover-trace probe, and the forged/
removed cohort arms encoding each scheme's correct expected verdict
(presence rule for audioseal, message rule for videoseal).

C2PA resolution: measured the five vendors' resolver contracts and the
C2PA 2.2 18.10.5 privacy gradient; product decision recorded that this
tool never resolves over the network. A registry guard pins the
published resolution-API set.

Gate: ruff and pyright src/ clean, 2114 passed / 3 skipped; maintain.sh
stops only on the known unfixed lightning 2.6.5 CVEs, pre-existing.

2026-09-07 18:23:18 -07:00

3.9 KiB

Raw Blame History

Watermark forgery study

Development-only study, run on 2026-09-07 from scripts/watermark_forgery_study.py, opening the forgery/remover-forensics direction from the original research map. It measures the four-state model (clean / marked / removed / forged) and the double-embedding question for both pinned oracles: does embedding message B over an A-marked artifact leave A, B, or mutual interference?

Carriers: the tone-stack audio carrier, the synthetic moving-gradient video clip, and the real committed Sora clip. Message B is the oracle's message generator at seed 8. Everything local; artifacts under .local-eval/.

The double-embedding answer: the last writer wins, completely

Cell	AudioSeal acc vs A	vs B	VideoSeal (gradient) vs A	vs B	VideoSeal (real Sora) vs A	vs B
marked A	1.000	0.44	1.000	0.47	1.000	0.47
removed	0.50 (presence 0.28)	0.44	0.57	0.48	1.000	0.47
forged B on clean	0.44	1.000	0.47	1.000	0.47	1.000
forged B on marked A	0.44	1.000	0.47	1.000	0.48	0.99
forged B on removed	0.56	0.875	0.47	1.000	0.47	1.00

Embedding B over marked A reads as pure B: the original message decays to coin-flip accuracy under both schemes, on synthetic and real content alike. There is no mutual interference and no residue of A. With an unkeyed, openly published embedder this is the expected outcome - and now it is a measurement, not an assumption. The forensic consequence: payload attribution without key control is meaningless against an attacker who can run the same open embedder; only presence detection survives, and even that carries the attacker's payload. Provider schemes that rely on secret keys are a different question this study cannot answer.

Two boundary observations travel with the table:

The removal transform is carrier-dependent, as the cohorts already showed: 10 dB noise removes the tone stack (presence 0.28) and crf 23 removes the gradient clip (0.57), but the real Sora clip keeps a perfect 1.000 decode through the same crf 23 - the same number the temporal study recorded.
A forged row reads detected under the AudioSeal presence rule (the label carries the foreign payload) and not_detected under the VideoSeal message-accuracy rule. Both cohort fixtures encode their scheme's correct expected value, and the benchmark docs state the two rules side by side.

Remover-trace probe (scoped first evidence)

High-to-low frequency band energy ratio on clean versus noise-removed audio: the tone stack jumps from 8.6e-05 to 2.3e-02 (a 269x trace), the pinkish carrier 0.016 to 0.039, and the white-noise carrier does not separate at all (0.330 to 0.332) - a noise carrier absorbs a noise remover. This detects the trace of THIS study's gaussian-noise removal only; it is not a general remover detector, and the probe JSON says so in its scope field.

Cohort arms

The audio cohort (v2, 20 cases) and video cohort (v2, 15 cases) now carry the four states as strict manifest rows: removed rows where the transform measurably destroys the mark (audio: noise on tone stack; video: crf 23 on both synthetic carriers), and forged rows embedding message B with the scheme-correct expected verdict. Zero expected-result mismatches in both recorded runs.

Pins

Study script sha256:3985522c862ac44a0c7ef8aa02946a11701439a94d67f5867f00819f638c032b, case rows sha256:03d446c08a5daed623a4be2980e097bb8bbc0cd7a6d3ab3f931d25aa58570a16. Oracles and checkpoints are pinned as in the kernel document.

Follow-ups

A keyed-scheme perspective needs a scheme with actual key control; WmForger and WMCopier diffusion-attack baselines remain the map's heavyweight item.
A general remover-trace detector would need features beyond one band ratio and validation against removals this project did not produce.

3.9 KiB Raw Blame History