diff --git a/.claude/rules/development.md b/.claude/rules/development.md index 49c1455..aec51c4 100644 --- a/.claude/rules/development.md +++ b/.claude/rules/development.md @@ -186,7 +186,11 @@ path exempt from the gate it exists to feed, which is exactly backwards. Prove a video-path refactor the same way the detection path is proven, and without needing an oracle carrier: build a clip from a tracked fixture with ffmpeg, run the engine before and after, and require an identical output sha256. Keep the generated -media outside the repository. +media outside the repository. Bind that comparison to one synced environment: a +full-extras `uv sync` was observed to change re-encoded H.264 packet bytes with +no code change while every detection stayed identical +(`docs/audio-provenance-experiment.md`), so a moved video hash across an +environment change is not by itself a regression signal. Frame sampling is compared as `timestamp + 1e-9 >= next_sample_time`, so mutating that `>=` to `>` changes nothing and a green suite proves nothing. Mutate the phase diff --git a/CLAUDE.md b/CLAUDE.md index b62f72e..18ac9b4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -43,7 +43,7 @@ The published Agent Skill is [`skills/remove-ai-watermarks/`](skills/remove-ai-w [`docs/module-internals.md`](docs/module-internals.md) is the canonical per-module map, including design decisions, thresholds, calibration history, incident records, and regression guards. Read the relevant section before changing a subsystem. -Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md). Pixel photo classification is [`docs/photo-classify.md`](docs/photo-classify.md); it is not `identify`. Classifier research is split between general [`AI-generated image classifiers`](docs/ai-generated-image-classifiers.md) and [`SynthID source classifiers`](docs/synthid-classifiers.md). Other SynthID campaign logs are [`docs/synthid-detector-research.md`](docs/synthid-detector-research.md) and [`docs/synthid-removal-research.md`](docs/synthid-removal-research.md). The pre-split chronological archive is [`docs/synthid-detector-removal-plan.md`](docs/synthid-detector-removal-plan.md). +Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md). Pixel photo classification is [`docs/photo-classify.md`](docs/photo-classify.md); it is not `identify`. Classifier research is split between general [`AI-generated image classifiers`](docs/ai-generated-image-classifiers.md) and [`SynthID source classifiers`](docs/synthid-classifiers.md). Other SynthID campaign logs are [`docs/synthid-detector-research.md`](docs/synthid-detector-research.md) and [`docs/synthid-removal-research.md`](docs/synthid-removal-research.md). The pre-split chronological archive is [`docs/synthid-detector-removal-plan.md`](docs/synthid-detector-removal-plan.md). The benchmark kernel and its pinned local oracles (AudioSeal, VideoSeal) cover the audio/video watermark cohorts and research studies: [`docs/benchmark-kernel.md`](docs/benchmark-kernel.md). ## Data safety diff --git a/docs/audio-provenance-experiment.md b/docs/audio-provenance-experiment.md new file mode 100644 index 0000000..c4f6ad5 --- /dev/null +++ b/docs/audio-provenance-experiment.md @@ -0,0 +1,133 @@ +# Audio provenance experiment (local AudioSeal) + +Development-only study, run on 2026-09-06 from `scripts/audioseal_experiment.py`. +It answers the first open item of the original watermark-landscape research: +does a video remain provenance-positive through its untouched audio track +after a visual-only cleaning pass? AudioSeal (Meta, MIT license) is used as a +local matched oracle: the same package embeds the watermark and reads it back, +so no provider API is contacted and no external upload happens. + +This page records the design, the pinned inputs, and the measured outcome. It +is not a detector-quality claim: carriers are synthetic non-speech signals, +and AudioSeal is not any provider's production audio watermark. + +## Design + +The experiment separates three questions and never merges them: + +1. `bitstream_identity`: are the cleaned video's audio packets byte-identical + to the source video's audio packets? +2. `cleaning_invariance`: does the detector verdict on the decoded audio + change across the visual cleaning pass? +3. `audio_attack`: how does the watermark respond to audio-path processing + only, with no video involved? + +Stages: + +- Three deterministic 8-second 16 kHz mono synthetic carriers (AM tone stack, + box-smoothed "pinkish" noise, white noise), a fixed 16-bit message, clean + and marked WAVs. Watermark SNR 17.9-34.6 dB. +- Three system-voice speech carriers rendered by the local macOS `say` tool + (Samantha, Daniel, Milena; no network, byte-deterministic per voice, pinned + by voice, text digest, and macOS version), each with clean and marked rows + and per-second interval readings on both the marked audio and a variant + whose seconds two through four were overwritten with 10 dB-SNR noise while + every other sample stays bit-identical. +- Audio attacks on the marked tone-stack WAV: AAC-LC 128k at 16 kHz and 48 + kHz, MP3 128k and 64k, 44.1 kHz resample, 8 kHz lowpass, additive noise at + 10 dB SNR, 0.5 gain. +- Three video arms muxing the same marked WAV under a stamped Sora-like mark + (24 frames, 12 fps, 840x480): PCM in MOV (bitstream-preserving), AAC 128k at + 16 kHz, AAC 128k at 48 kHz. Each arm runs the production + `remove_video_visible` path, which transcodes video and stream-copies + audio. Audio packets are extracted container-independently and hashed before + and after cleaning. + +Weights are pinned to Hugging Face revision +`3c19eba53390776cf2cc9ed5f6c9ac67ce72ecba` and verified against pinned SHA-256 +digests before loading; the loader mirrors `AudioSeal.parse_model` with the +checkpoint bytes pinned and now lives in the shared `scripts/audioseal_oracle.py`. +Dependency versions at run time: audioseal 0.2.0, torch 2.13.0, Python +3.12.13, ffmpeg 9.0.1, macOS 26.6.2 for the system voices. The executed code is +bound by hash: experiment script +`sha256:cc1f8360403ca1e4f04d72717b4b18ff84ca0718fa01bc1a8316dc189359b5b9`, +oracle +`sha256:d765e31b4c439fda82536b72bb602b55dac666d13e954809b42f48acfad1d2e8`, +case rows +`sha256:99b56a68fdb7789f4adc9f321eadf86277354a383270fe6d689b2c87eba83883`. +Within one fixed environment two independent runs produced byte-identical +case files (verified twice, before and after the oracle refactor). A +full-extras `uv sync` between two recorded runs changed the cleaned-video +H.264 packet bytes - every detector reading, every audio packet digest, and +all source artifacts stayed identical, only the re-encoded video stream +moved - so cross-environment comparisons must bind on the audio evidence and +readings, not on cleaned-container hashes. + +## Measured outcome, 2026-09-06 + +| Stage | Reading | +| --- | --- | +| Clean carriers (matched negatives) | mean detect 0.003-0.008, frac above threshold 0 | +| Marked carriers | mean detect >= 0.9997, bit accuracy 0.94-1.0 | +| All video arms, source audio | mean detect >= 0.9999, bit accuracy 1.0 | +| All video arms, after visual cleaning | identical readings; audio packets byte-identical | +| MP3 128k/64k, AAC 128k (16 and 48 kHz), 44.1 kHz resample, 8 kHz lowpass, 0.5 gain | mean detect >= 0.9999, bit accuracy 1.0 | +| Raw ADTS AAC decode | mean detect 0.990, bit accuracy 0.69 | +| Additive noise, 10 dB SNR | mean detect 0.30, bit accuracy 0.5 | + +The thesis holds in this model: `remove_video_visible` leaves the audio +bitstream untouched in every arm, so the audio watermark verdict is invariant +across visual cleaning, and a cleaned video stays provenance-positive through +its audio track. The result is scoped to AudioSeal as the mark; it proves the +copy semantics of the cleaning path, not that any specific provider embeds a +comparable audio mark. + +The speech carriers close the out-of-domain gap the synthetic trio exposed: +all three system voices decoded the full 16-bit message (the pinkish synthetic +carrier had flipped one bit), with watermark SNR of 27.9-32.2 dB and clean +speech far below the decision rule. Per-second interval readings localize the +signal in time: overwriting seconds two through four with 10 dB noise killed +exactly those two windows (0.19 falling to 0.0009 mean probability) while the +untouched windows kept their original readings to the sixth decimal. The +final window of a clip is a short remainder (0.1 s here) and its readings are +accordingly less stable - visible in the tables, never dropped. + +Three findings qualify the naive readings: + +- **Codecs are not the attack.** At these rates every codec and resample round + trip preserved both detection and the full message, on synthetic and speech + carriers alike. Full-clip additive noise at 10 dB SNR killed every speech + detection while the white and pinkish synthetic carriers survived it: speech + has quiet moments where the noise dominates, so the effective removal lever + on audio is loud additive noise, and its power depends on the carrier. A + removal pipeline that wants an audio path clean must not rely on + transcoding alone. +- **The decode path is part of the measurement.** Decoding raw ADTS keeps the + encoder's 1024-sample priming head in the stream, which alone moved the + verdict from 0.9999/full message to 0.990/partial message; a gapless-aware + container (MP4/M4A edit list, or MP3 gapless info) decodes sample-identical + audio and restores the reading. The experiment carries the raw-ADTS arm + (`aac_128k_adts_raw`) explicitly so the artifact stays a named measurement + instead of contaminating codec rows. +- **Voice names are not carrier identity.** macOS 26.6.2 renders `Alex` and + `Samantha` byte-identically, so a naive three-voice set was one carrier + counted twice. The shipped script verifies distinct content hashes across + configured voices and refuses an aliased set; diversity claims rest on the + hash, never the name. + +Detector cost on this host (Apple silicon, CPU): cold first call 176-465 ms +across runs, warm median 158-160 ms for 8-second clips. These are diagnostic +host timings, not a certified profile. + +An earlier iteration derived per-carrier seeds from Python's `hash()`, which is +salted per process; it produced different carrier bytes on every run. The +shipped script derives seeds from `zlib.crc32`, and the two recorded final runs +are byte-identical. + +## Follow-ups + +- The speech and interval layers are complete; the remaining audio-side + upgrade is folding per-second windows into the benchmark schema itself, if a + case-level question ever needs them as rows. The forgery/remover-forensics + and VideoSeal temporal items remain separate open directions from the + original research map. diff --git a/docs/benchmark-kernel.md b/docs/benchmark-kernel.md index 6b7c129..f3b3cd7 100644 --- a/docs/benchmark-kernel.md +++ b/docs/benchmark-kernel.md @@ -1,11 +1,15 @@ # Watermark benchmark kernel `scripts/watermark_benchmark.py` is a development-only runner for reproducible -image-watermark experiments. It calls the existing local DWT-DCT and TrustMark -detectors through thin adapters. It does not add a runtime command, download a -corpus, or contact a provenance oracle or provider API. The optional TrustMark -package can fetch its official Adobe model weights when its local package cache -is incomplete; prepare that dependency before an offline run. +image-, audio-, and video-watermark experiments. It calls the existing local +DWT-DCT and TrustMark detectors plus the revision-pinned AudioSeal and +VideoSeal oracles through thin adapters. It does not add a runtime command, +download a corpus, or contact a provenance oracle or provider API. The +optional TrustMark package can fetch its official Adobe model weights when its +local package cache is incomplete, the audioseal adapter fetches its pinned +checkpoints the same way, and the videoseal adapter downloads one pinned +TorchScript file into a user cache; prepare those dependencies before an +offline run. The kernel answers three different questions and never merges their answers: @@ -36,10 +40,10 @@ The fixed fields are: | `schema_version` | Integer `1`. | | `case_id` | Unique case identity. | | `pair_id` | Groups related clean, marked, attacked, and removed cases. | -| `media_type` | `image` in schema v1. | -| `adapter` | `dwt-dct` or `trustmark`. | +| `media_type` | `image`, `audio`, or `video` in schema v1. | +| `adapter` | `dwt-dct` or `trustmark` for `image`; `audioseal` for `audio`; `videoseal` for `video`. The loader rejects a known adapter named against the wrong media type. | | `arm` | `positive`, `matched_negative`, `wrong_key`, or `hard_negative`. | -| `state` | `clean`, `marked`, `attacked`, or `removed`. | +| `state` | `clean`, `marked`, `attacked`, `removed`, or `forged`. `forged` names an artifact carrying a watermark with a message different from the adapter oracle's fixed one; the matched verifier's correct answer there is `not_detected` with the forensic detail carried by the study layer, not the verdict. | | `path`, `sha256` | Artifact path and pinned content digest. | | `reference_path`, `reference_sha256` | Both strings or both `null`; the fidelity reference. | | `source_revision` | Corpus, generator, or acquisition revision. | @@ -85,10 +89,110 @@ pixels produce `psnr_db: null` with `psnr_status: unbounded_identical`, never a misleading numeric zero. Missing or incomparable references remain explicit states rather than fabricated metrics. -Schema v1 deliberately has no aggregate score, confidence interval, video -adapter, automatic attack generator, removal runner, or provider oracle. Add -those only after the case-level evidence is large enough to define and test the -corresponding statistical or media contract. +Schema v1 deliberately has no aggregate score, confidence interval, automatic +attack generator, removal runner, or provider oracle. Add those only after the +case-level evidence is large enough to define and test the corresponding +statistical or media contract. + +## Audio rows + +`media_type: audio` rows decode every artifact and reference to mono 16 kHz +float32 through the system `ffmpeg` before any adapter runs, mirroring the +image path where decoding also stays outside the detector interval. Audio rows +therefore require `ffmpeg` on `PATH`; a missing or failing decode is the same +explicit `error` status as an undecodable image. + +The `audioseal` adapter reads the shared revision-pinned oracle in +`scripts/audioseal_oracle.py` (Meta AudioSeal, MIT license, Hugging Face +revision and checkpoint SHA-256 pinned and verified at load). Its `detected` +status is the detector's own documented rule: the fraction of per-sample +probabilities above 0.5 reaching 0.5. The `label` is the decoded 16-bit message +as a binary string, so a case whose payload differs from the embedded one is +still `detected` with a different label, never `not_detected`. A `wrong_key` +arm for this adapter is therefore a different-message row, not a clean row. + +Audio fidelity compares the decoded sample arrays against the explicit +reference: identical decoded samples produce `snr_db: null` with +`snr_status: unbounded_identical`, never a misleading numeric zero; a silent +reference reports `zero_reference_power`; a length mismatch stays an explicit +`incomparable` state. SNR here is reference signal power over error power, the +sample-domain analogue of the image PSNR contract. + +## Audio cohort v1 + +`scripts/watermark_benchmark_audio_cohort.py` builds the audio analogue of the +synthetic image cohort: three deterministic synthetic carriers (AM tone stack, +smoothed noise, white noise), the pinned AudioSeal message, matched clean +negatives, MP3 128k, gapless M4A AAC 128k, and deterministic 10 dB additive +noise attacks, plus one never-embedded white-noise hard negative. Since v2 +the audio cohort also carries the four-state arms: a `removed` row where the +noise transform measurably destroys the mark (the tone stack only - the +synthetic noise carriers survive 10 dB noise, so their noise rows stay open +`attacked` measurements), and `forged` rows embedding a foreign message with +`expected: detected`, because the audioseal rule is watermark presence and +the decoded label is what exposes the foreign payload. 20 cases in total, +everything hashed into a strict manifest validated through the +benchmark loader. + +```bash +uv run --extra dev python scripts/watermark_benchmark_audio_cohort.py \ + --output-dir .local-eval/watermark-benchmark-audio-cohort-v1 +uv run --extra dev python scripts/watermark_benchmark.py \ + .local-eval/watermark-benchmark-audio-cohort-v1/manifest.jsonl \ + --output .local-eval/audio-cohort-v1-results.jsonl +``` + +### Development baseline, 2026-09-06 + +The first local run used kernel +`sha256:2216169ab7c33f7439720f0934e8ac6ba8379e510bfaeabe008f820c7658701d`, +oracle `sha256:bd67e2b1be5c486384ab1f64709f661241451ab1ea7592cc56f0eefb91c415ab`, +and recipe `sha256:f361163b4fbf23b885c026751c544e97be0bf054906c1aad455276992a2768d2`, +producing manifest `sha256:b6c48684f0c85e882159f6d5c4fbeea2f43c3b24d3a2c3a034209c6f531ba73d` +and case rows `sha256:66ac23bd926925dc5baf7216c03f8a7246eeb473315a17701358a3803ab7b0d1`. +Dependencies were audioseal 0.2.0, torch 2.13.0, and ffmpeg 9.0.1. + +| State and transform | Cases | Detected | Not detected | +| --- | ---: | ---: | ---: | +| clean matched negatives + hard negative | 4 | 0 | 4 | +| marked, audioseal embed | 3 | 3 | 0 | +| attacked, MP3 128k | 3 | 3 | 0 | +| attacked, AAC 128k (gapless M4A) | 3 | 3 | 0 | +| attacked, 10 dB additive noise | 3 | 2 | 1 | + +Every clean and never-embedded negative stayed below the decision rule. All +codecs retained detection with the full message on the tone-stack and +white-noise carriers; the pinkish-noise carrier decoded one flipped payload bit +while remaining detected, matching the experiment's per-carrier bit-accuracy +observation. The 10 dB noise arm destroyed the tone-stack detection but not +white or pinkish noise: carrier-dependent robustness is exactly what the +matched arms exist to expose. Nearest-rank adapter timing: cold first call +405.1 ms, warm median 124.2 ms, warm max 157.4 ms for 8-second clips on this +host. As with the image baselines, these are diagnostic host rows, not a +certified profile. + +### Speech extension + +`--with-speech` adds fifteen more rows: three system-voice speech carriers +(Samantha, Daniel, Milena) rendered by the local macOS `say` tool - no +network, byte-deterministic per voice on a fixed macOS version - each with a +matched clean negative, a marked positive, and the same three attacks. Every +speech row's transform parameters carry the full TTS provenance (tool, macOS +version, voice, text digest, duration). The builder verifies the configured +voices render distinct bytes and refuses an aliased set, because macOS 26.6.2 +renders `Alex` and `Samantha` byte-identically and a name is not a carrier +identity. + +The first `--with-speech` run (2026-09-06, 31 cases total, recipe +`sha256:3f291f49ab9ef2499085708fb614e3723ebee6d35f0b509fc36f017ec5a86c3b`, +manifest +`sha256:b9baa79066deb002402cc20166835887d67a5158ff9ce7bbc1963e5d603db3a5`, +case rows +`sha256:abc68590f59fbddab1c086371d2854622b41ea756ba61a7a87b76cb6df64c386`) +kept every synthetic verdict and added: speech clean 0/3 detected, speech +marked 3/3 with the full message, codecs 3/3, and full-clip 10 dB noise 0/3 - +unlike the synthetic white and pinkish carriers, speech has quiet moments +where the noise dominates. Speech marked fidelity sat at 27.9-32.2 dB SNR. ## Synthetic cohort v1 @@ -275,6 +379,116 @@ uncommitted. The exact inputs and implementations are therefore bound by hash: The local timing rows were host-contended and are not used for a new latency claim. The controlled resource profile below remains the performance evidence. +## Video rows + +`media_type: video` rows decode every artifact and reference to float RGB +frames in [0, 1] as (T, H, W, 3) through the system `ffmpeg`, staying outside +the detector interval like the image and audio decodes. Video fidelity +compares decoded frames against the explicit reference clip: mean PSNR over +frames, a changed-frame fraction, `unbounded_identical` for bit-identical +pixel sequences, and an explicit `shape_mismatch` state when frame counts or +geometry differ - which is why geometry-changing attack arms carry no +reference instead of a fabricated comparison. + +The `videoseal` adapter loads Meta's standalone TorchScript release of +VideoSeal (MIT license) from `scripts/videoseal_oracle.py` - one pinned file, +no `videoseal` package, whose wheel pulls a research dependency chain without +macOS arm64 wheels and whose loader resolves cards relative to the working +directory. The upstream video evaluation ignores the model's detection head +(it prepends a constant) and scores the aggregated 256-bit message decode, so +the adapter's `detected` is its own explicit rule: bit accuracy of the +averaged decode against the oracle's fixed message at or above 0.9. The +`label` is the decoded message as 64 hex characters. Because the rule is +message accuracy, a different-message row reads `not_detected` - the matched +verifier rejects the foreign payload - unlike the audioseal presence rule +above. Every videoseal detection record +carries a `temporal` block - frame count, min/mean/max and the full per-frame +bit-accuracy series - and the aggregation is computed in the oracle from raw +per-frame logits, because the TorchScript build ignores the aggregation +argument of `detect_video_and_aggregate`. The aggregation matrix itself +(avg versus norm-weighted) is evaluated in +[VideoSeal temporal evaluation](videoseal-temporal-evaluation.md), not in the +kernel, which keeps plain avg as its canonical rule. + +## Video cohort v1 + +`scripts/watermark_benchmark_video_cohort.py` builds the video analogue of +the other cohorts: two deterministic synthetic 64-frame 256x256 clips (moving +gradient, moving texture), the pinned 256-bit message, matched clean +negatives, one never-embedded hard negative, and three attacks - a realistic +H.264 re-encode at crf 23, a 75 percent downscale, and a frame-rate halving +that drops half the keyframes. Since v2 the video cohort also carries the +four-state arms: `removed` rows reusing the crf 23 re-encode that measurably +takes both synthetic carriers below the rule, and `forged` rows embedding a +foreign message with `expected: not_detected`, because the videoseal rule is +message accuracy and the matched verifier rejects a foreign payload. 15 +cases in total. + +```bash +uv run --extra dev python scripts/watermark_benchmark_video_cohort.py \ + --output-dir .local-eval/watermark-benchmark-video-cohort-v1 +uv run --extra dev python scripts/watermark_benchmark.py \ + .local-eval/watermark-benchmark-video-cohort-v1/manifest.jsonl \ + --output .local-eval/video-cohort-v1-results.jsonl +``` + +### Development baseline, 2026-09-07 + +The first local run used kernel +`sha256:7a3906cebc3c7cacbac7e204c014d1d6281615e7acbbc5b3a8fcecc939ca68c7`, +oracle `sha256:6c889bf399e8e3bada439eee4a1e9c0b04177be78dfee756d5daa79e73db71b3`, +recipe `sha256:dd06ae1c40ef7d7f6fd2b258b20444a3250254a5ef258542a4eaf43cc501342b`, +manifest +`sha256:bb6a7dda6b3897794f2fbfe47291329bb7b9ff7ee57c0634ebb386e58acd5e7a`, +and case rows +`sha256:5793b8ba33c55e135e5bef43f33fd2699bb57bae6ca8ead38b1dab0123d1dd12`. +The checkpoint is the TorchScript build `y_256b_img.jit` +`sha256:5c7a4581c36fc6090aafdcfb3999123bae5172a4847f22e2da4e7fd1a39d1e1b` +pinned to upstream commit `870ca7f`. + +| State and transform | Cases | Detected | Not detected | +| --- | ---: | ---: | ---: | +| clean matched negatives + hard negative | 3 | 0 | 3 | +| marked, videoseal embed (crf 8 artifact) | 2 | 2 | 0 | +| attacked, H.264 crf 23 | 2 | 0 | 2 | +| attacked, 75 percent downscale + crf 18 | 2 | 0 | 2 | +| attacked, frame rate halved | 2 | 2 | 0 | + +Direct oracle readings fill in what the threshold hides: the marked clips +decode at 0.992-0.996 bit accuracy (48.5 and 41.4 dB mean PSNR against their +clean carriers), the crf 23 re-encode collapses to 0.56-0.57, the downscale +to 0.69, and the halved frame rate keeps 0.996. Two cautions travel with +these numbers: the watermark survives H.264 only at generous quality - a +realistic crf 23 re-encode erases it on these smooth synthetic carriers, +while a high-bitrate uniform-noise carrier survived crf 18 in the isolated +probe, so retention is carrier- and quality-dependent - and dropping half +the frames does not touch the message because the secret repeats across +keyframes. These are first-baseline observations on synthetic carriers, not +robustness curves; a sweep across crf values and real-content carriers is +the natural follow-up. + +### Four-state baseline, 2026-09-07 (v2 cohorts) + +The v2 runs used kernel +`sha256:22f2c1dd5c3b7aeb96090c4c578c6d1841ee62d90958dd1c5c7ad7db71835eec` +with audio recipe +`sha256:31c021144212a43565fb3746fadc835bfc37ea8ac98998fad9bba9e72410e82c` +(manifest +`sha256:43c7f77f1d4537c00d4b0dd5128e9becc6c860cdd196a7ba2f7541f4276784ec`, +case rows +`sha256:c73d7dddfdb90262e96be4d67ec1567d9046812661928cc816cb9f97f4a01a7e`) +and video recipe +`sha256:8f25929304ef4f94721667968ee4bb89ebf584a2108de7e02db36c855dc2ddc8` +(manifest +`sha256:b5659848c0d05e205f80ab36480e70fdf7e5f9df4d8ada6ffadec42c97b88998`, +case rows +`sha256:c9ad56848bb7efe200aaf1189a8eb03f7571567e6702edd195a62f812fa05815`). +Every v1 verdict was unchanged; the new arms read: audio removed 0/1 +detected, audio forged 3/3 detected under the presence rule, video removed +0/2, video forged 0/2 under the message rule - zero expected mismatches. +The mechanism underneath these rows, including double embedding, is measured +in [Watermark forgery study](watermark-forgery-study.md). + ## Aggregate repeated runs `scripts/watermark_benchmark_report.py` turns one or more result JSONL files diff --git a/docs/c2pa-resolution-research.md b/docs/c2pa-resolution-research.md new file mode 100644 index 0000000..31e200b --- /dev/null +++ b/docs/c2pa-resolution-research.md @@ -0,0 +1,83 @@ +# C2PA soft-binding resolution research + +Development-only investigation, 2026-09-07, completing the privacy-tiered C2PA +resolution item from the original research map. The outcome is a product +decision rather than a feature: this tool resolves nothing over the network, +and this page records what the resolution ecosystem would learn if it did. + +## The normative shape + +C2PA 2.2 specification section 18.10.5 defines the soft binding resolution +API: a standard way of retrieving manifest stores from a resolution endpoint +given **a soft binding value, a manifest identifier, or an asset** - three +input modes that are exactly three privacy exposure levels. The algorithm +registry (`c2pa-org/softbinding-algorithm-list`) carries the endpoint URIs in +`softBindingResolutionApis`. As of the packaged revision `c0218cd628c9`, +8 of 53 entries publish them. + +## Measured contracts + +| Algorithm | Kind | Endpoint | Input | Content leaves | Content digest leaves | +| --- | --- | --- | --- | --- | --- | +| `com.aiwatermark.audioseal.1` / `videoseal` / `pixelseal` | watermark | `aiwatermark.com/api/v1/resolve` | `GET ?payload={extracted payload}&alg=` | no | no | +| `com.joinmonolith.sha256` | fingerprint | `api.joinmonolith.com/api/c2pa/matches/byBinding` | `GET ?alg&value=base64(SHA-256 of asset)` | no | yes | +| `me.deepmark.audio.vigil.128` | watermark | `resolution-api.deepmark.me` | payload-based per vendor site | no | no | +| `ai.trufo.pawprint.watermark` / `.fingerprint` | both | `c2pa.trufo.ai/v1` | not published publicly | ? | ? | +| `io.blockfact.audio.watermark.32` | watermark | `api.blockfact.io/api/soft-binding/v1` | not published publicly | ? | ? | + +Sources, verified 2026-09-07 against the vendors' own pages: the Monolith +soft-binding documentation (lookup endpoints public by design, "a verifier is +usually a stranger"; only manifest minting is authenticated), the AIWatermark +algorithm page ("after extracting the watermark payload from an audio file, +validators can resolve the payload"; model source Meta FAIR AudioSeal), the +Deepmark product description ("the signal is detected and its payload +resolved"), the Trufo developer documentation and `trufo-py` repository +(signing-side only), and the BlockFact API reference (its public verify +endpoint takes a Starknet transaction hash, not a soft binding). + +The privacy gradient, concretely: + +- **Payload-only** resolution (AIWatermark, Deepmark) sends a locally + extracted watermark payload - opaque bits that identify the mark, not the + file. This is the mode our pinned local oracles could serve without ever + touching the network with content or a digest. +- **Digest** resolution (Monolith) sends base64(SHA-256) of the whole asset. + No bytes leave, but the digest is linkable to the exact content: the + registry learns you are inspecting that precise file. +- **Undocumented** (Trufo, BlockFact): no public contract; any integration + would be guesswork and is not attempted. + +## The product decision + +This tool does not resolve anything over the network, at any tier, and no +`local_only` / `binding_only` / `full_content` modes ship. The reasoning: + +- The tool's user processes their own generated or edited content and wants + to know which provenance signals it carries and to remove them. Local + payload extraction - already available through the pinned development + oracles - answers the identification question without a network. +- Querying a vendor's resolver tells that vendor someone is inspecting the + specific content (directly for digest-based lookups, via the payload for + payload-based ones). For a removal tool that is a leak in the opposite + direction of its purpose, and it contradicts the offline runtime boundary + the project already guarantees. +- No user scenario of this tool requires manifest recovery from a registry: + that is a verifier-of-third-party-content function, which is outside the + intended use. + +The runtime boundary stays: identification reports locally decoded evidence +only, and `docs/known-limitations.md` states the resolution boundary in user +terms. A guard test pins the packaged registry's published-API set, so an +upstream ecosystem change surfaces as a conscious decision recorded against +this page rather than a silent regeneration. + +## Investigation note + +The first reconnaissance pass concluded "upstream publishes zero resolution +APIs" because the probe searched the registry JSON for guessed key names +instead of the `softBindingResolutionApis` key the sync actually reads. The +drift-guard test written from that wrong premise failed immediately against +the packaged snapshot and exposed the error - eight live endpoints, including +resolution hosting for the exact AudioSeal and VideoSeal schemes this project +benchmarks. The corrected guard now pins the real set. Field names are +verified against the consumer's schema, never guessed. diff --git a/docs/index.md b/docs/index.md index a5fb154..cbdcc38 100644 --- a/docs/index.md +++ b/docs/index.md @@ -50,4 +50,8 @@ The current behavior is defined by the code, tests, README, and user guides. - [OpenAI SynthID oracle ladders](synthid-oracle-ladders.md) (routing hub) - [SynthID detector and removal plan](synthid-detector-removal-plan.md) (chronological mixed archive) - [Text protection research](text-protection-research.md) +- [Audio provenance experiment](audio-provenance-experiment.md) (local AudioSeal matched-oracle study of video audio tracks across visual cleaning) +- [VideoSeal temporal evaluation](videoseal-temporal-evaluation.md) (per-frame scores, aggregation matrix, and crf decay on synthetic and real clips) +- [Watermark forgery study](watermark-forgery-study.md) (four-state model and double-embedding: the last writer wins under both oracles) +- [C2PA resolution research](c2pa-resolution-research.md) (measured resolver contracts, the privacy gradient, and the decision to never resolve over the network) - [Chroma1 engine research](chroma1-engine-research.md) diff --git a/docs/known-limitations.md b/docs/known-limitations.md index bd38a3a..1e5cbf3 100644 --- a/docs/known-limitations.md +++ b/docs/known-limitations.md @@ -207,6 +207,15 @@ The public video results make that boundary explicit without decoding audio: the nested audio status reports `copied_if_present` and `unverified`. CLI output uses the same terms. These fields add no media pass or model inference. +A local matched-oracle study has now measured the boundary concretely: +[Audio provenance experiment](audio-provenance-experiment.md) embeds AudioSeal +into synthetic carriers, runs the visible video-cleaning path, and shows the +audio packets byte-identical and the watermark verdict invariant across +cleaning, while additive noise on the audio path is the effective lever. The +boundary above still stands for provider marks: this tool still does not +decode them, and a `copied_if_present` audio track remains `unverified` in +runtime output. + The shipped engine streams sampled frames in bounded batches, computes its fidelity metrics incrementally, and pipes regenerated pixels directly to ffmpeg. Its frame and latent memory is therefore bounded by `--batch-size` @@ -346,6 +355,18 @@ Memory needs depend on the profile, input size, dtype, and card. ## Metadata and formats +### Identification never resolves anything over the network + +C2PA soft bindings can be resolved to manifests through vendor endpoints +(the registry carries eight of them), and querying one always tells the +vendor that someone is inspecting the specific content - directly through +the asset digest for fingerprint lookups, through the watermark payload for +payload-based ones. This tool deliberately has no resolution mode, tier, or +opt-in: identification reports only locally decoded evidence, and the +watermark decoders in this repository run offline against pinned weights. +The contracts, the privacy gradient, and the reasoning are recorded in +[the resolution research](c2pa-resolution-research.md). + ### Missing metadata does not mean clean Screenshots, social platforms, and re-encoding can remove metadata while a diff --git a/docs/verification-plan.md b/docs/verification-plan.md index d3d0add..ac9f540 100644 --- a/docs/verification-plan.md +++ b/docs/verification-plan.md @@ -432,6 +432,32 @@ resource tier; its CPU throughput makes a six-provider online matrix counterproductive. Store generated outputs and the detailed CSV only under `.local-eval/`. +The audio side of that contract now has a local matched-oracle study: +[Audio provenance experiment](audio-provenance-experiment.md) embeds AudioSeal +into synthetic carriers, verifies the copied-audio bitstream identity and a +watermark verdict invariant across `remove_video_visible`, and measures the +audio-path attacks separately. Its carriers are synthetic non-speech audio +and its oracle is AudioSeal, so it validates the copy semantics, not provider +audio marks. The audio and video arms are now first-class benchmark rows: +[benchmark kernel](benchmark-kernel.md) accepts `media_type: audio` and +`media_type: video` with the revision-pinned `audioseal` and `videoseal` +adapters, and both cohorts carry matched negatives, codec, and degradation +arms as strict manifest cases. The temporal layer is closed too: videoseal +detection records carry per-frame bit accuracy, and +[VideoSeal temporal evaluation](videoseal-temporal-evaluation.md) measures +the aggregation matrix and the crf decay curve on synthetic and real clips. +What remains open here is a wider real-content sweep and the learned temporal +inpainting comparison. + +The privacy-tiered C2PA resolution item is closed as a product decision: +[the resolution research](c2pa-resolution-research.md) measured the five +vendors' resolver contracts (payload-only, asset digest, or unpublished), +grounded them in C2PA 2.2 section 18.10.5, and concluded that this tool +resolves nothing over the network at any tier - querying a resolver tells +the vendor someone is inspecting the specific content, which is a leak in +the opposite direction of a removal tool's purpose. A registry guard test +pins the published-API set so ecosystem changes surface consciously. + The same full-clip gate runs the public metadata-only path against its real MP4 and verifies unchanged file size, decoded frames, stream properties, and AAC packets. A separate synthetic large-`mdat` test rejects full-source diff --git a/docs/videoseal-temporal-evaluation.md b/docs/videoseal-temporal-evaluation.md new file mode 100644 index 0000000..4d5bd9c --- /dev/null +++ b/docs/videoseal-temporal-evaluation.md @@ -0,0 +1,84 @@ +# VideoSeal temporal evaluation + +Development-only study, run on 2026-09-07 from +`scripts/videoseal_temporal_study.py`, answering the three temporal questions +the case-level benchmark verdict aggregates away: how decoded bit accuracy +decays across H.264 quality, whether the frame-aggregation choice changes the +outcome, and how per-frame accuracy moves in time on real provider clips +versus synthetic carriers. + +Carriers: the two synthetic cohort clips (moving gradient, moving texture) +plus the committed real provider videos for Veo and Sora, standardized to a +64-frame 256x256 prefix (first 64 frames, center-crop, INTER_AREA resize). +Attacks: crf 18/23/28/32 re-encodes, 75 percent downscale, and frame-rate +halving. Everything is local; the real clips are already-committed project +fixtures with public test clearance. + +## The TorchScript aggregation trap + +The first matrix run produced identical numbers under all four aggregations, +and that was a measurement artifact, not a finding: the standalone TorchScript +build ignores the `aggregation` argument of `detect_video_and_aggregate` and +returns the same thresholded bits for every choice (verified by checksum on a +single artifact). The oracle now computes the aggregations itself from the raw +per-frame logits using the exact upstream `extract_message` formulas - avg, +squared_avg, and l1/l2-norm-weighted averages - and the recorded matrix comes +from those. The formula port is pinned by a hand-computed unit test. + +## Matrix (bit accuracy of the aggregated 256-bit decode) + +| Carrier | crf18 | crf23 | crf28 | crf32 | scale .75 | fps .5 | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| moving_gradient (clean 0.52) | 0.887 | 0.570 | 0.559 | 0.500 | 0.691 | 0.996 | +| moving_texture (clean 0.48) | 0.602 | 0.559 | 0.531 | 0.586 | 0.879 | 0.980 | +| real_veo (clean 0.52) | 0.801 | 0.723 | 0.621 | 0.574 | 1.000 | 0.855 | +| real_sora (clean 0.51) | 0.996 | 1.000 | 0.992 | 0.902 | 1.000 | 0.996 | + +Readings, not verdicts: the benchmark adapter's rule is 0.9 under plain avg. + +- **Carrier content dominates.** The real Sora clip keeps the message through + every attack including crf 32; the real Veo clip decays smoothly with + quality; the smooth synthetic gradient dies at crf 23. Any "survives H.264" + claim must name the carrier and the crf. +- **Downscale is not an attack on real content.** 0.75 downscale plus re-encode + scored a perfect 1.000 on both real clips - resize was a training + augmentation, and the antialias resample appears to help the detector. On + the synthetic gradient it still costs 0.31. +- **Frame-rate halving is harmless.** Every carrier kept the message after + dropping half the frames: the 256-bit secret repeats across keyframes, so + temporal redundancy, not frame count, carries it. +- **Aggregation choice matters at the margin.** Norm-weighted averaging + recovered 2-6 points on damaged arms (moving_texture scale: 0.879 avg + versus 0.941 l1/l2, crossing the 0.9 rule) but never rescued a collapsed + case and never lifted a clean negative above chance. The kernel keeps plain + avg as its canonical rule; the matrix is the evidence for when that choice + is conservative. + +Per-frame accuracy on the marked clips: real_sora sits at 0.988-1.000 on +every frame, real_veo spans 0.547-0.910, moving_texture 0.457-0.961 - the +temporal spread is itself carrier-dependent, and the kernel's +`detection.temporal` block now records it for every videoseal case. + +## Pins + +Study script +`sha256:682ccf14b21d6ac17bb6500a7efb364247eebae2d9e0e8895ffac33e6d294d4f`, +oracle +`sha256:5aa67ceaca1b0050727aa0a75440b1bf4c583435b3bf9e8ef2e97fadd46a035c`, +case rows +`sha256:5494c9bbe8a6085ecbd27fd9732dadf8d004dab7be159e905051e854dabe64aa`. +Checkpoint: TorchScript `y_256b_img.jit` +`sha256:5c7a4581c36fc6090aafdcfb3999123bae5172a4847f22e2da4e7fd1a39d1e1b` +at upstream commit `870ca7f`. The kernel re-run of the video cohort with +temporal records used kernel +`sha256:657b0d7e35af75eb97b9bd9b81c15fcae754399f27999e08a5c28c494e69cf05` +and produced case rows +`sha256:435215e9a55589f6d83e34d62f34469c079db078a0a174ead64ffaa74af664c5` +with unchanged verdicts. + +## Follow-ups + +- A crf sweep on more real carriers (the other committed provider clips) to + widen the carrier-dominated picture beyond two samples. +- Temporal inpainting comparison remains open from the original research map; + the per-frame rows recorded by the kernel are the substrate for it. diff --git a/docs/watermark-forgery-study.md b/docs/watermark-forgery-study.md new file mode 100644 index 0000000..d4267f5 --- /dev/null +++ b/docs/watermark-forgery-study.md @@ -0,0 +1,76 @@ +# Watermark forgery study + +Development-only study, run on 2026-09-07 from +`scripts/watermark_forgery_study.py`, opening the forgery/remover-forensics +direction from the original research map. It measures the four-state model +(clean / marked / removed / forged) and the double-embedding question for both +pinned oracles: does embedding message B over an A-marked artifact leave A, +B, or mutual interference? + +Carriers: the tone-stack audio carrier, the synthetic moving-gradient video +clip, and the real committed Sora clip. Message B is the oracle's message +generator at seed 8. Everything local; artifacts under `.local-eval/`. + +## The double-embedding answer: the last writer wins, completely + +| Cell | AudioSeal acc vs A | vs B | VideoSeal (gradient) vs A | vs B | VideoSeal (real Sora) vs A | vs B | +| --- | ---: | ---: | ---: | ---: | ---: | ---: | +| marked A | 1.000 | 0.44 | 1.000 | 0.47 | 1.000 | 0.47 | +| removed | 0.50 (presence 0.28) | 0.44 | 0.57 | 0.48 | 1.000 | 0.47 | +| forged B on clean | 0.44 | 1.000 | 0.47 | 1.000 | 0.47 | 1.000 | +| **forged B on marked A** | **0.44** | **1.000** | **0.47** | **1.000** | **0.48** | **0.99** | +| forged B on removed | 0.56 | 0.875 | 0.47 | 1.000 | 0.47 | 1.00 | + +Embedding B over marked A reads as pure B: the original message decays to +coin-flip accuracy under both schemes, on synthetic and real content alike. +There is no mutual interference and no residue of A. With an unkeyed, openly +published embedder this is the expected outcome - and now it is a measurement, +not an assumption. The forensic consequence: payload attribution without key +control is meaningless against an attacker who can run the same open embedder; +only presence detection survives, and even that carries the attacker's +payload. Provider schemes that rely on secret keys are a different question +this study cannot answer. + +Two boundary observations travel with the table: + +- The removal transform is carrier-dependent, as the cohorts already showed: + 10 dB noise removes the tone stack (presence 0.28) and crf 23 removes the + gradient clip (0.57), but the real Sora clip keeps a perfect 1.000 decode + through the same crf 23 - the same number the temporal study recorded. +- A forged row reads `detected` under the AudioSeal presence rule (the label + carries the foreign payload) and `not_detected` under the VideoSeal + message-accuracy rule. Both cohort fixtures encode their scheme's correct + expected value, and the benchmark docs state the two rules side by side. + +## Remover-trace probe (scoped first evidence) + +High-to-low frequency band energy ratio on clean versus noise-removed audio: +the tone stack jumps from 8.6e-05 to 2.3e-02 (a 269x trace), the pinkish +carrier 0.016 to 0.039, and the white-noise carrier does not separate at all +(0.330 to 0.332) - a noise carrier absorbs a noise remover. This detects the +trace of THIS study's gaussian-noise removal only; it is not a general +remover detector, and the probe JSON says so in its scope field. + +## Cohort arms + +The audio cohort (v2, 20 cases) and video cohort (v2, 15 cases) now carry the +four states as strict manifest rows: `removed` rows where the transform +measurably destroys the mark (audio: noise on tone stack; video: crf 23 on +both synthetic carriers), and `forged` rows embedding message B with the +scheme-correct expected verdict. Zero expected-result mismatches in both +recorded runs. + +## Pins + +Study script +`sha256:3985522c862ac44a0c7ef8aa02946a11701439a94d67f5867f00819f638c032b`, +case rows +`sha256:03d446c08a5daed623a4be2980e097bb8bbc0cd7a6d3ab3f931d25aa58570a16`. +Oracles and checkpoints are pinned as in the kernel document. + +## Follow-ups + +- A keyed-scheme perspective needs a scheme with actual key control; WmForger + and WMCopier diffusion-attack baselines remain the map's heavyweight item. +- A general remover-trace detector would need features beyond one band ratio + and validation against removals this project did not produce. diff --git a/pyproject.toml b/pyproject.toml index f31e5e1..70c9bac 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -179,6 +179,11 @@ dev = [ # Development-only OpenAI Content Provenance oracle. It is not exposed by # the installed CLI or the public Python API. "openai>=2.52.0", + # Development-only local AudioSeal audio watermark oracle for the + # scripts/audioseal_experiment.py provenance study. Not part of the + # installed CLI, the public Python API, or any runtime pipeline. + "audioseal>=0.2.0", + "huggingface-hub>=0.20.0", "pytest>=8.0.0", "pytest-cov>=4.1.0", "pytest-xdist>=3.5.0", diff --git a/scripts/audioseal_experiment.py b/scripts/audioseal_experiment.py new file mode 100644 index 0000000..207cbe2 --- /dev/null +++ b/scripts/audioseal_experiment.py @@ -0,0 +1,800 @@ +#!/usr/bin/env python3 +"""Local AudioSeal audio-provenance experiment (development-only). + +The study answers one question the image benchmark cannot: does a video stay +provenance-positive through its untouched audio track after a visual-only +cleaning pass? AudioSeal (Meta, MIT license) is embedded locally into +synthetic carriers, muxed into videos next to a visible Sora-like mark, and the +production ``remove_video_visible`` path is run on the result. That path +transcodes the video stream and stream-copies audio, so the experiment measures +three separable things and never merges them: + +- ``bitstream_identity``: whether the cleaned video's audio packets are + byte-identical to the source video's audio packets. +- ``cleaning_invariance``: whether the detector verdict on the decoded audio is + unchanged by the visual cleaning pass. +- ``audio_attack``: how the watermark responds to audio-path processing only + (codecs, resampling, noise, gain, lowpass) with no video involved. + +Everything runs locally. No provider oracle or API is contacted, the package +runtime is untouched, and generated artifacts stay outside the repository under +``.local-eval/``. Weights are pinned to an exact Hugging Face revision and +verified against pinned SHA-256 digests before use. + + uv run python scripts/audioseal_experiment.py \ + --output-dir .local-eval/audioseal-experiment-v1 +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import logging +import shutil +import struct +import subprocess +import sys +import zlib +from dataclasses import dataclass +from pathlib import Path +from time import perf_counter_ns +from typing import TYPE_CHECKING + +import numpy as np +from PIL import Image, ImageDraw + +if TYPE_CHECKING: + from collections.abc import Sequence + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) +sys.path.insert(0, str(REPO / "scripts")) + +import audioseal_oracle as oracle # noqa: E402 +from audioseal_oracle import SAMPLE_RATE # noqa: E402 + +log = logging.getLogger("audioseal_experiment") + +DURATION_S = 8.0 +FRAME_COUNT = 24 +FRAME_RATE = 12.0 +FRAME_WIDTH = 840 +FRAME_HEIGHT = 480 +CARRIER_SEED = 20260907 +MESSAGE_SEED = 7 + + +@dataclass(frozen=True) +class Detection: + """AudioSeal detector readings on one decoded audio tensor.""" + + mean_detect_prob: float + frac_above_threshold: float + bit_accuracy: float | None + decoded_bits: list[int] + + def as_dict(self) -> dict[str, object]: + return { + "mean_detect_prob": round(self.mean_detect_prob, 6), + "frac_above_threshold": round(self.frac_above_threshold, 6), + "bit_accuracy": None if self.bit_accuracy is None else round(self.bit_accuracy, 6), + "decoded_bits": self.decoded_bits, + } + + +def sha256_bytes(data: bytes) -> str: + return hashlib.sha256(data).hexdigest() + + +def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as fh: + for chunk in iter(lambda: fh.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +# --- Pure carrier and WAV helpers (unit-tested without torch) -------------- + + +def carrier_seed(name: str) -> int: + """Stable per-carrier seed; Python's hash() is salted per process.""" + return CARRIER_SEED + zlib.crc32(name.encode()) % 1000 + + +def synth_carrier(name: str, seconds: float) -> np.ndarray: + """Synthesize one deterministic carrier by registered name.""" + factory = CARRIERS.get(name) + if factory is None: + raise KeyError(f"unknown carrier {name!r}; expected one of {sorted(CARRIERS)}") + master = np.random.default_rng(carrier_seed(name)) + return np.asarray(factory(int(seconds * SAMPLE_RATE), master), dtype=np.float32) + + +def tone_stack_carrier(n: int, rng: np.random.Generator) -> np.ndarray: + t = np.arange(n, dtype=np.float64) / SAMPLE_RATE + am = 0.5 + 0.5 * np.sin(2 * np.pi * 1.1 * t) + carrier = ( + 0.5 * am * np.sin(2 * np.pi * 220.0 * t) + + 0.3 * am * np.sin(2 * np.pi * 440.0 * t) + + 0.2 * am * np.sin(2 * np.pi * 880.0 * t) + ) + return carrier + rng.normal(0.0, 0.005, n) + + +def pinkish_carrier(n: int, rng: np.random.Generator) -> np.ndarray: + white = rng.normal(0.0, 1.0, n) + kernel = np.ones(9) / 9.0 + smoothed = np.convolve(white, kernel, mode="same") + return smoothed / np.std(smoothed) * 0.1 + + +def white_carrier(n: int, rng: np.random.Generator) -> np.ndarray: + return rng.normal(0.0, 0.1, n) + + +CARRIERS: dict[str, object] = { + "tone_stack": tone_stack_carrier, + "pinkish_noise": pinkish_carrier, + "white_noise": white_carrier, +} + + +def message_bits(seed: int = MESSAGE_SEED) -> list[int]: + rng = np.random.default_rng(seed) + return [int(b) for b in rng.integers(0, 2, oracle.N_BITS)] + + +def wav_pcm16_bytes(samples: np.ndarray, sample_rate: int) -> bytes: + """Serialize mono float samples in [-1, 1) as a 16-bit PCM WAV file.""" + clipped = np.clip(samples, -1.0, 1.0) + pcm = (clipped * 32767.0).astype(" float: + power = float(np.sum(signal.astype(np.float64) ** 2)) + noise = float(np.sum(residual.astype(np.float64) ** 2)) + return 10.0 * np.log10(power / max(noise, 1e-12)) + + +# --- ffmpeg helpers -------------------------------------------------------- + + +def require_tools() -> tuple[str, str]: + ffmpeg = shutil.which("ffmpeg") + ffprobe = shutil.which("ffprobe") + if ffmpeg is None or ffprobe is None: + raise SystemExit("audioseal_experiment requires ffmpeg and ffprobe on PATH") + return ffmpeg, ffprobe + + +def run_ffmpeg(ffmpeg: str, args: Sequence[str], *, stdin: bytes | None = None) -> bytes: + result = subprocess.run( # noqa: S603 + [ffmpeg, "-y", "-loglevel", "error", *args], + input=stdin, + capture_output=True, + check=False, + ) + if result.returncode != 0: + raise RuntimeError(f"ffmpeg failed ({result.returncode}): {result.stderr.decode(errors='replace')[:400]}") + return result.stdout + + +def ffmpeg_version(ffmpeg: str) -> str: + result = subprocess.run([ffmpeg, "-version"], capture_output=True, check=False) # noqa: S603 + return result.stdout.decode(errors="replace").splitlines()[0].strip() + + +def decode_audio_f32(ffmpeg: str, path: Path) -> np.ndarray: + raw = run_ffmpeg( + ffmpeg, + ["-i", str(path), "-ac", "1", "-ar", str(SAMPLE_RATE), "-f", "f32le", "pipe:1"], + ) + return np.frombuffer(raw, dtype=" bytes: + return run_ffmpeg(ffmpeg, ["-i", str(path), "-map", "0:a:0", "-c:a", "copy", "-f", "adts", "pipe:1"]) + + +def pcm_packet_bytes(ffmpeg: str, path: Path) -> bytes: + return run_ffmpeg(ffmpeg, ["-i", str(path), "-map", "0:a:0", "-c:a", "copy", "-f", "s16le", "pipe:1"]) + + +# --- Synthetic video (independent Sora-like mark, moving background) ------- + + +def moving_background(frame_index: int) -> np.ndarray: + x = np.arange(FRAME_WIDTH, dtype=np.float32)[None, :] + y = np.arange(FRAME_HEIGHT, dtype=np.float32)[:, None] + luma = 42 + 10 * np.sin((x + frame_index * 5) / 38) + 5 * np.cos((y - frame_index * 2) / 54) + frame = np.stack( + ( + np.clip(luma - 5, 0, 255), + np.clip(luma + 1, 0, 255), + np.clip(luma + 7, 0, 255), + ), + axis=2, + ).astype(np.uint8) + moving_x = 30 + frame_index * 8 + frame[moving_x : moving_x + 72, 70:132] = (80, 140, 210) + frame[220 + frame_index : 223 + frame_index, :] = (110, 70, 45) + return frame + + +def sora_like_mark() -> tuple[np.ndarray, int, int]: + """Render a Sora-like mark without reusing the detector's template.""" + mark = Image.new("L", (180, 64), 0) + draw = ImageDraw.Draw(mark) + draw.ellipse((1, 14, 32, 54), fill=255) + draw.ellipse((25, 8, 62, 58), fill=255) + draw.ellipse((15, 20, 28, 44), fill=0) + draw.ellipse((37, 18, 50, 43), fill=0) + from PIL import ImageFont + + try: + font = ImageFont.load_default(size=49) + except TypeError: + font = ImageFont.load_default() + draw.text((68, 1), "Sora", font=font, fill=255, stroke_width=1) + resized = mark.resize((124, 44), Image.Resampling.BOX) + return np.asarray(resized, dtype=np.uint8), 620, 398 + + +def stamped_frame(frame_index: int, mark: np.ndarray, mark_x: int, mark_y: int) -> np.ndarray: + frame = moving_background(frame_index) + height, width = mark.shape + crop = frame[mark_y : mark_y + height, mark_x : mark_x + width].astype(np.float32) + alpha = mark[:, :, None].astype(np.float32) / 255.0 * 0.78 + frame[mark_y : mark_y + height, mark_x : mark_x + width] = np.clip(crop * (1 - alpha) + 255 * alpha, 0, 255).astype( + np.uint8 + ) + return frame + + +def encode_video( + ffmpeg: str, + path: Path, + frames: list[np.ndarray], + audio: Path, + *, + audio_codec: str, + audio_rate: int, +) -> None: + """Encode raw BGR frames plus a WAV audio input into one container.""" + height, width = frames[0].shape[:2] + duration = len(frames) / FRAME_RATE + args = [ + "-f", + "rawvideo", + "-pix_fmt", + "bgr24", + "-s:v", + f"{width}x{height}", + "-r", + f"{FRAME_RATE:.12g}", + "-i", + "pipe:0", + "-i", + str(audio), + "-map", + "0:v:0", + "-map", + "1:a:0", + "-c:v", + "libx264", + "-preset", + "ultrafast", + "-crf", + "8", + "-pix_fmt", + "yuv420p", + ] + if audio_codec == "pcm_s16le": + args += ["-c:a", "pcm_s16le"] + else: + args += [ + "-c:a", + audio_codec, + "-b:a", + "128k", + "-ar", + str(audio_rate), + ] + args += ["-t", f"{duration:.12g}", str(path)] + payload = b"".join(frame.astype(np.uint8).tobytes() for frame in frames) + run_ffmpeg(ffmpeg, args, stdin=payload) + + +# --- AudioSeal oracle (shared, revision-pinned) ----------------------------- + + +def load_pinned_models() -> tuple[object, object, dict[str, str]]: + """Load the shared pinned oracle models and record their provenance.""" + import importlib.metadata as md + import inspect + + import torch + from audioseal_oracle import load_pinned_models as shared_load + + generator, detector = shared_load() + versions = { + "audioseal": md.version("audioseal"), + "torch": torch.__version__, + "huggingface_revision": oracle.HF_REVISION, + "generator_sha256": oracle.GENERATOR_SHA256, + "detector_sha256": oracle.DETECTOR_SHA256, + "oracle_source_sha256": sha256_file(Path(inspect.getfile(shared_load))), + } + return generator, detector, versions + + +class DetectorClock: + """Records the first (cold) detector call separately from warm calls.""" + + def __init__(self) -> None: + self.cold_ms: float | None = None + self.warm_ms: list[float] = [] + + def measure(self, call: object, wav: object) -> object: + started = perf_counter_ns() + result = call(wav) + elapsed = (perf_counter_ns() - started) / 1e6 + if self.cold_ms is None: + self.cold_ms = elapsed + else: + self.warm_ms.append(elapsed) + return result + + +def detect_audio(detector: object, clock: DetectorClock, samples: np.ndarray) -> Detection: + """Read one detection under the clock and score it against the message.""" + reading = clock.measure(lambda wav: oracle.read(detector, wav), samples) + expected = message_bits() + bits = list(reading.decoded_bits) + accuracy = sum(int(b == e) for b, e in zip(bits, expected, strict=True)) / oracle.N_BITS + return Detection( + mean_detect_prob=reading.mean_detect_prob, + frac_above_threshold=reading.frac_above_threshold, + bit_accuracy=accuracy, + decoded_bits=bits, + ) + + +def embed_audio(generator: object, samples: np.ndarray, message: list[int]) -> np.ndarray: + """Embed the fixed message through the shared pinned oracle.""" + return np.asarray(oracle.embed(generator, samples, message), dtype=np.float32) + + +# --- Attack transforms ----------------------------------------------------- + + +def numpy_attacks(samples: np.ndarray) -> dict[str, np.ndarray]: + rng = np.random.default_rng(CARRIER_SEED + 1) + out: dict[str, np.ndarray] = {} + power = float(np.sum(samples.astype(np.float64) ** 2)) + noise = rng.normal(0.0, 1.0, samples.shape[0]).astype(np.float32) + noise *= np.sqrt(power / max(np.sum(noise.astype(np.float64) ** 2), 1e-12) / 10.0) + out["noise_snr10"] = samples + noise + out["gain_0.5"] = samples * 0.5 + return out + + +def partial_noise_attack(samples: np.ndarray, start_s: float, end_s: float) -> np.ndarray: + """Mix 10 dB-SNR gaussian noise into one time window, leaving the rest bit-identical.""" + rng = np.random.default_rng(CARRIER_SEED + 2) + out = samples.copy() + start = int(start_s * SAMPLE_RATE) + end = int(end_s * SAMPLE_RATE) + window = out[start:end] + power = float(np.sum(window.astype(np.float64) ** 2)) + noise = rng.normal(0.0, 1.0, window.shape[0]).astype(np.float32) + noise *= np.sqrt(power / max(np.sum(noise.astype(np.float64) ** 2), 1e-12) / 10.0) + out[start:end] = window + noise + return out + + +# --- Local speech carriers (system TTS, no network) -------------------------- + +SPEECH_TEXT = ( + "Provenance through audio remains measurable after visual cleaning. " + "The second sentence extends this carrier past three seconds." +) +SPEECH_VOICES: tuple[str, ...] = ("Samantha", "Daniel", "Milena") + + +def assert_distinct_voices(artifacts: dict[str, bytes]) -> None: + """Reject voice aliasing: two names rendering byte-identical audio. + + macOS 26.6.2 aliases ``Alex`` to ``Samantha``; a carrier-diversity claim + built on aliased voices is one carrier counted twice. Verified by content + hash, never by voice name. + """ + seen: dict[str, str] = {} + for voice, payload in artifacts.items(): + digest = sha256_bytes(payload) + duplicate = seen.get(digest) + if duplicate is not None: + raise SystemExit( + f"system voices {duplicate!r} and {voice!r} render byte-identical audio; pick genuinely distinct voices" + ) + seen[digest] = voice + + +def speech_provenance() -> dict[str, str]: + """Record everything that pins system-synthesized speech bytes.""" + import platform + + product = "" + sw_vers = shutil.which("sw_vers") + if sw_vers is not None: + result = subprocess.run( # noqa: S603 - resolved sw_vers with fixed arguments + [sw_vers, "-productVersion"], capture_output=True, text=True, check=False + ) + product = result.stdout.strip() + return { + "tts": "say", + "macos": product or platform.mac_ver()[0], + "text_sha256": sha256_bytes(SPEECH_TEXT.encode()), + } + + +def synthesize_speech(voice: str, workdir: Path) -> Path: + """Render one deterministic utterance through the local system voice.""" + say = shutil.which("say") + if say is None: + raise SystemExit("speech carriers require the macOS say tool") + target = workdir / f"speech-{voice}.aiff" + result = subprocess.run( # noqa: S603 - resolved say with fixed arguments + [say, "-o", str(target), "-v", voice, SPEECH_TEXT], + capture_output=True, + check=False, + ) + if result.returncode != 0 or not target.is_file(): + raise SystemExit(f"system voice {voice!r} is unavailable: {result.stderr.decode(errors='replace')[:200]}") + return target + + +def speech_samples(voice: str, workdir: Path, ffmpeg: str) -> np.ndarray: + """Decode one system-voice utterance to mono 16 kHz float32.""" + return decode_audio_f32(ffmpeg, synthesize_speech(voice, workdir)) + + +def ffmpeg_attack( + ffmpeg: str, name: str, suffix: str, args: Sequence[str], source: Path, workdir: Path, *, stem: str = "attack" +) -> Path: + target = workdir / f"{stem}-{name}.{suffix}" + run_ffmpeg(ffmpeg, ["-i", str(source), *args, str(target)]) + return target + + +FFMPEG_ATTACKS: dict[str, tuple[str, list[str]]] = { + # AAC attacks use the MP4/M4A container so the decode honors the encoder's + # priming edit list. A raw ADTS decode keeps the 1024-sample encoder-delay + # head in the stream, which measures the no-gapless decode path rather + # than the codec itself; that path is kept as its own explicit arm below. + "aac_128k": ("m4a", ["-c:a", "aac", "-b:a", "128k"]), + "aac_128k_48k": ("m4a", ["-c:a", "aac", "-b:a", "128k", "-ar", "48000"]), + "aac_128k_adts_raw": ("adts", ["-c:a", "aac", "-b:a", "128k"]), + "mp3_128k": ("mp3", ["-c:a", "libmp3lame", "-b:a", "128k"]), + "mp3_64k": ("mp3", ["-c:a", "libmp3lame", "-b:a", "64k"]), + "resample_44k_wav": ("wav", ["-c:a", "pcm_s16le", "-ar", "44100"]), + "lowpass_8k_wav": ("wav", ["-c:a", "pcm_s16le", "-af", "lowpass=f=8000"]), +} + + +# --- Experiment ------------------------------------------------------------ + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--skip-video", action="store_true", help="audio stages only") + parser.add_argument( + "--skip-speech", action="store_true", help="skip system-voice speech carriers and interval readings" + ) + args = parser.parse_args() + + out_dir: Path = args.output_dir.resolve() + if out_dir.exists(): + raise SystemExit(f"refusing to overwrite existing output directory {out_dir}") + out_dir.mkdir(parents=True) + + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + ffmpeg, _ffprobe = require_tools() + + generator, detector, versions = load_pinned_models() + clock = DetectorClock() + rows: list[dict[str, object]] = [] + + def record(stage: str, **fields: object) -> None: + row = {"stage": stage, **fields} + rows.append(row) + log.info("%s", json.dumps(row, sort_keys=True, default=str)[:400]) + + def detect_and_record(stage: str, label: str, samples: np.ndarray, path: Path | None, **extra: object) -> Detection: + det = detect_audio(detector, clock, samples) + record( + stage, + label=label, + detection=det.as_dict(), + sha256=None if path is None else sha256_file(path), + **extra, + ) + return det + + message = message_bits() + record("pins", message=message, **versions, ffmpeg=ffmpeg_version(ffmpeg)) + + # Stage 1: carriers, clean vs marked. + marked_wavs: dict[str, Path] = {} + for name in CARRIERS: + clean = synth_carrier(name, DURATION_S) + clean_path = out_dir / f"{name}-clean.wav" + clean_path.write_bytes(wav_pcm16_bytes(clean, SAMPLE_RATE)) + detect_and_record("carrier_clean", f"{name}/clean", clean, clean_path) + + marked = embed_audio(generator, clean, message) + marked_path = out_dir / f"{name}-marked.wav" + marked_path.write_bytes(wav_pcm16_bytes(marked, SAMPLE_RATE)) + detect_and_record( + "carrier_marked", + f"{name}/marked", + marked, + marked_path, + watermark_snr_db=round(snr_db(clean, marked - clean), 3), + ) + marked_wavs[name] = marked_path + + # Stage 2: audio attacks on the tone_stack marked wav only; the codec arms + # double as the audio tracks used for the video arms below. + primary = marked_wavs["tone_stack"] + attack_paths: dict[str, Path] = {} + for name, (suffix, ffargs) in FFMPEG_ATTACKS.items(): + target = ffmpeg_attack(ffmpeg, name, suffix, ffargs, primary, out_dir) + attack_paths[name] = target + detect_and_record("audio_attack", name, decode_audio_f32(ffmpeg, target), target) + for name, samples in numpy_attacks(decode_audio_f32(ffmpeg, primary)).items(): + target = out_dir / f"attack-{name}.wav" + target.write_bytes(wav_pcm16_bytes(samples, SAMPLE_RATE)) + attack_paths[name] = target + detect_and_record("audio_attack", name, samples, target) + + # Stage 2b: system-voice speech carriers with per-second interval readings. + # Speech is the detector's training domain, and the interval report shows + # WHERE in time the signal lives - including its localized death under an + # attack confined to seconds two through four. + if not args.skip_speech: + from audioseal_oracle import read_intervals + + provenance = speech_provenance() + raw_speech: dict[str, Path] = {} + for voice in SPEECH_VOICES: + raw_speech[voice] = synthesize_speech(voice, out_dir) + assert_distinct_voices({voice: path.read_bytes() for voice, path in raw_speech.items()}) + for voice in SPEECH_VOICES: + label = f"speech_{voice}" + clean_speech = decode_audio_f32(ffmpeg, raw_speech[voice]) + clean_path = out_dir / f"{label}-clean.wav" + clean_path.write_bytes(wav_pcm16_bytes(clean_speech, SAMPLE_RATE)) + detect_and_record( + "speech_carrier", + f"{label}/clean", + clean_speech, + clean_path, + tts={**provenance, "voice": voice}, + duration_s=round(clean_speech.shape[0] / SAMPLE_RATE, 3), + ) + + marked_speech = embed_audio(generator, clean_speech, message) + marked_path = out_dir / f"{label}-marked.wav" + marked_path.write_bytes(wav_pcm16_bytes(marked_speech, SAMPLE_RATE)) + detect_and_record( + "speech_carrier", + f"{label}/marked", + marked_speech, + marked_path, + tts={**provenance, "voice": voice}, + watermark_snr_db=round(snr_db(clean_speech, marked_speech - clean_speech), 3), + ) + partial = partial_noise_attack(marked_speech, 2.0, 4.0) + partial_path = out_dir / f"{label}-partial_noise_2s_4s.wav" + partial_path.write_bytes(wav_pcm16_bytes(partial, SAMPLE_RATE)) + for tag, samples, artifact_path in ( + ("marked", marked_speech, marked_path), + ("partial_noise_2s_4s", partial, partial_path), + ): + intervals = read_intervals(detector, samples) + record( + "speech_intervals", + label=f"{label}/{tag}", + per_second=[ + { + "start_s": index, + "mean_detect_prob": round(reading.mean_detect_prob, 6), + "frac_above_threshold": round(reading.frac_above_threshold, 6), + "detected": reading.detected, + } + for index, reading in enumerate(intervals) + ], + sha256=sha256_file(artifact_path), + ) + + # Stage 3: video arms. Each arm muxes the SAME marked wav next to a + # stamped Sora-like mark, differing only in how audio is carried. + if not args.skip_video: + from remove_ai_watermarks.video import remove_video_visible + + mark, mark_x, mark_y = sora_like_mark() + frames = [stamped_frame(i, mark, mark_x, mark_y) for i in range(FRAME_COUNT)] + arms: dict[str, tuple[str, int | None]] = { + "pcm_mov": ("pcm_s16le", None), + "aac_16k": ("aac", SAMPLE_RATE), + "aac_48k": ("aac", 48_000), + } + for arm, (codec, rate) in arms.items(): + suffix = "mov" if codec == "pcm_s16le" else "mp4" + source = out_dir / f"video-{arm}-source.{suffix}" + encode_video(ffmpeg, source, frames, primary, audio_codec=codec, audio_rate=rate or SAMPLE_RATE) + packetize = pcm_packet_bytes if codec == "pcm_s16le" else aac_packet_bytes + source_packets = packetize(ffmpeg, source) + source_digest = sha256_bytes(source_packets) + + cleaned = out_dir / f"video-{arm}-cleaned.{suffix}" + result = remove_video_visible(source, cleaned, mark="sora", backend="cv2") + if result.output is None: + raise SystemExit(f"visual cleaning wrote no output for arm {arm}") + cleaned_packets = packetize(ffmpeg, cleaned) + cleaned_digest = sha256_bytes(cleaned_packets) + identical = source_digest == cleaned_digest + + det_source = detect_and_record( + "video_audio", + f"{arm}/source", + decode_audio_f32(ffmpeg, source), + source, + audio_codec=codec, + ) + det_cleaned = detect_and_record( + "video_audio_cleaned", + f"{arm}/cleaned", + decode_audio_f32(ffmpeg, cleaned), + cleaned, + audio_codec=codec, + bitstream_identical=identical, + source_packet_sha256=source_digest, + cleaned_packet_sha256=cleaned_digest, + visual_removed_frames=result.removed_frames, + ) + invariance = ( + det_source.mean_detect_prob == det_cleaned.mean_detect_prob + and det_source.frac_above_threshold == det_cleaned.frac_above_threshold + and det_source.decoded_bits == det_cleaned.decoded_bits + ) + record( + "cleaning_invariance", + arm=arm, + bitstream_identical=identical, + verdict_invariant=invariance, + ) + + # Timing summary and reports. + warm = sorted(clock.warm_ms) + timing = { + "detector_cold_ms": None if clock.cold_ms is None else round(clock.cold_ms, 3), + "detector_warm_n": len(warm), + "detector_warm_median_ms": None if not warm else round(warm[len(warm) // 2], 3), + "detector_warm_max_ms": None if not warm else round(warm[-1], 3), + } + cases_path = out_dir / "cases.jsonl" + cases_path.write_text("".join(json.dumps(r, default=str) + "\n" for r in rows)) + summary_path = out_dir / "summary.md" + summary_path.write_text(render_summary(rows, timing, versions, ffmpeg, out_dir)) + print(f"wrote {cases_path}") + print(f"wrote {summary_path}") + return 0 + + +def render_summary( + rows: list[dict[str, object]], + timing: dict[str, object], + versions: dict[str, str], + ffmpeg: str, + out_dir: Path, +) -> str: + def table(stage: str, columns: list[str]) -> list[str]: + selected = [r for r in rows if r["stage"] == stage] + lines = [ + f"### {stage}", + "", + "| " + " | ".join(columns) + " |", + "| " + " | ".join(["---"] * len(columns)) + " |", + ] + for r in selected: + det = r.get("detection", {}) if isinstance(r.get("detection"), dict) else {} + cells = [] + for col in columns: + if col == "mean_detect": + cells.append(f"{det.get('mean_detect_prob', '')}") + elif col == "frac>0.5": + cells.append(f"{det.get('frac_above_threshold', '')}") + elif col == "bit_acc": + cells.append(f"{det.get('bit_accuracy', '')}") + elif col == "label": + cells.append(str(r.get("label", r.get("arm", "")))) + else: + cells.append(str(r.get(col, ""))) + lines.append("| " + " | ".join(cells) + " |") + lines.append("") + return lines + + def interval_table() -> list[str]: + selected = [r for r in rows if r["stage"] == "speech_intervals"] + lines = [ + "### speech_intervals", + "", + "| label | second | mean_detect | frac>0.5 | detected |", + "| --- | --- | --- | --- | --- |", + ] + for r in selected: + for window in r.get("per_second", []): + if not isinstance(window, dict): + continue + lines.append( + "| " + + " | ".join( + ( + str(r.get("label", "")), + str(window.get("start_s", "")), + str(window.get("mean_detect_prob", "")), + str(window.get("frac_above_threshold", "")), + str(window.get("detected", "")), + ) + ) + + " |" + ) + lines.append("") + return lines + + lines = [ + "# AudioSeal audio-provenance experiment", + "", + f"- output: `{out_dir.name}`", + f"- revision: `{versions['huggingface_revision']}`", + f"- audioseal {versions['audioseal']}, torch {versions['torch']}", + f"- ffmpeg: `{ffmpeg_version(ffmpeg)}`", + f"- detector timing: {json.dumps(timing)}", + "", + *table("carrier_clean", ["label", "mean_detect", "frac>0.5"]), + *table("speech_carrier", ["label", "mean_detect", "frac>0.5", "bit_acc", "watermark_snr_db"]), + *interval_table(), + *table("carrier_marked", ["label", "mean_detect", "frac>0.5", "bit_acc", "watermark_snr_db"]), + *table("audio_attack", ["label", "mean_detect", "frac>0.5", "bit_acc"]), + *table("video_audio", ["label", "mean_detect", "frac>0.5", "bit_acc"]), + *table("video_audio_cleaned", ["label", "mean_detect", "frac>0.5", "bit_acc", "bitstream_identical"]), + *table("cleaning_invariance", ["arm", "bitstream_identical", "verdict_invariant"]), + "", + "`bitstream_identical` compares audio packet digests of the source and", + "visually cleaned videos. `verdict_invariant` additionally requires the", + "detector readings to match. Rows are measurements on synthetic", + "carriers, not detector-quality claims.", + "", + ] + return "\n".join(lines) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/audioseal_oracle.py b/scripts/audioseal_oracle.py new file mode 100644 index 0000000..1bbb569 --- /dev/null +++ b/scripts/audioseal_oracle.py @@ -0,0 +1,190 @@ +#!/usr/bin/env python3 +"""Revision-pinned local AudioSeal oracle for development-only benchmarks. + +One shared seam for the benchmark kernel adapter, the audio cohort builder, +and the audio-provenance experiment. Weights are pinned to an exact Hugging +Face revision of ``facebook/audioseal`` and verified against pinned SHA-256 +digests before loading; the loader mirrors ``AudioSeal.parse_model`` with the +checkpoint bytes pinned, so card values override the checkpoint's embedded +``xp.cfg`` exactly like the package loader does for card loads. + +The first load may download the pinned checkpoints into the local Hugging +Face cache when it is incomplete, like the optional TrustMark weights. Prepare +the dependency before an offline run. Nothing here contacts a provenance +oracle or provider API, and none of this ships in the installed CLI or the +public Python API. +""" + +from __future__ import annotations + +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Sequence + +HF_REPO = "facebook/audioseal" +HF_REVISION = "3c19eba53390776cf2cc9ed5f6c9ac67ce72ecba" +GENERATOR_SHA256 = "7a845b5fbe9364a63a3909d8ab3fe064d13a76ae4c2e983573e08c69b7b51748" +DETECTOR_SHA256 = "8a78e8a83584113523e161fc599fcab10fd0e94c04d2eb9d2fa1e9ec91ab69d9" +SAMPLE_RATE = 16_000 +N_BITS = 16 +DETECTION_THRESHOLD = 0.5 +MESSAGE_THRESHOLD = 0.5 +GENERATOR_CARD = "audioseal_wm_16bits" +DETECTOR_CARD = "audioseal_detector_16bits" + + +@dataclass(frozen=True) +class AudioSealReading: + """Detector readings on one decoded mono sample array.""" + + mean_detect_prob: float + frac_above_threshold: float + decoded_bits: tuple[int, ...] + + @property + def detected(self) -> bool: + """The detector's own document decision rule at the default threshold.""" + return self.frac_above_threshold >= DETECTION_THRESHOLD + + @property + def label(self) -> str: + return "".join(str(bit) for bit in self.decoded_bits) + + +def import_failure() -> ImportError | None: + """Return the missing-dependency error, or None when the stack is present.""" + try: + import audioseal # noqa: F401 + import huggingface_hub # noqa: F401 + import omegaconf # noqa: F401 + import torch # noqa: F401 + except ImportError as exc: + return exc + return None + + +def available() -> bool: + """Whether the audioseal stack is importable in this environment.""" + return import_failure() is None + + +def load_pinned_models() -> tuple[object, object]: + """Load generator and detector from the pinned, digest-verified revision.""" + failure = import_failure() + if failure is not None: + raise RuntimeError( + "audioseal oracle requires the dev extra: uv sync --extra dev (audioseal, torch, huggingface-hub)" + ) from failure + + import hashlib + + import audioseal as audioseal_pkg + import torch + from audioseal import AudioSeal + from audioseal.builder import create_detector, create_generator + from audioseal.loader import ( + AudioSealDetectorConfig, + AudioSealWMConfig, + convert_state_dict_for_scriptable_model, + load_model_checkpoint, + load_state_dict, + ) + from huggingface_hub import hf_hub_download + from omegaconf import OmegaConf + + cards_dir = Path(audioseal_pkg.__file__).parent / "cards" + + def sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + def load(card_name: str, filename: str, cfg_type: object, create: object) -> object: + local = Path(hf_hub_download(HF_REPO, filename, revision=HF_REVISION)) + digest = sha256_file(local) + expected = GENERATOR_SHA256 if "generator" in filename else DETECTOR_SHA256 + if digest != expected: + raise RuntimeError(f"pinned checkpoint digest mismatch for {filename}: {digest} != {expected}") + config_dict = OmegaConf.to_container(OmegaConf.load(cards_dir / f"{card_name}.yaml")) + if not isinstance(config_dict, dict): + raise RuntimeError(f"audioseal card {card_name} did not parse into a mapping") + config_dict.pop("checkpoint") + checkpoint = load_model_checkpoint(str(local)) + merged = {**checkpoint.get("xp.cfg", {}), **config_dict} + config = AudioSeal.parse_config(merged, cfg_type) # type: ignore[arg-type] + raw = checkpoint.get("best_state", checkpoint.get("model", checkpoint)) + raw = convert_state_dict_for_scriptable_model(raw) + model = create(config, device="cpu") # type: ignore[call-arg] + load_state_dict(model, state_dict=raw) + model.eval() + return model + + generator = load(GENERATOR_CARD, "generator_base.pth", AudioSealWMConfig, create_generator) + detector = load(DETECTOR_CARD, "detector_base.pth", AudioSealDetectorConfig, create_detector) + if torch is None: # pragma: no cover - defensive, import kept for symmetry + raise RuntimeError("torch import disappeared") + return generator, detector + + +def embed(generator: object, samples: object, message: Sequence[int]) -> object: + """Embed one fixed bit message into a float32 mono sample array.""" + import torch + + wav = torch.from_numpy(samples).view(1, 1, -1) + msg = torch.tensor([list(message)], dtype=torch.float32) + with torch.no_grad(): + marked = generator(wav, sample_rate=SAMPLE_RATE, alpha=1.0, message=msg) + return marked.view(-1).numpy() + + +def read(detector: object, samples: object) -> AudioSealReading: + """Read detection probabilities and the decoded message from samples.""" + import torch + + wav = torch.from_numpy(samples).view(1, 1, -1) + with torch.no_grad(): + result, message = detector(wav, sample_rate=SAMPLE_RATE) + probs = result[0, 1, :].numpy() + bits = tuple(int(bit) for bit in (message.view(-1).numpy() > MESSAGE_THRESHOLD).astype(int)) + return AudioSealReading( + mean_detect_prob=float(probs.mean()), + frac_above_threshold=float((probs > DETECTION_THRESHOLD).mean()), + decoded_bits=bits, + ) + + +def window_bounds(total_samples: int, window_s: float, sample_rate: int) -> list[tuple[int, int]]: + """Half-open sample bounds of fixed seconds-long windows over a signal. + + A trailing remainder shorter than ``window_s`` is its own final window, so + interval reporting never silently drops a partial second. + """ + if total_samples < 0: + raise ValueError("total_samples must be non-negative") + if window_s <= 0: + raise ValueError("window_s must be positive") + if sample_rate <= 0: + raise ValueError("sample_rate must be positive") + size = int(window_s * sample_rate) + if size <= 0: + raise ValueError("window_s is shorter than one sample") + return [(start, min(start + size, total_samples)) for start in range(0, total_samples, size)] + + +def read_intervals(detector: object, samples: object, *, window_s: float = 1.0) -> list[AudioSealReading]: + """Read one detection per fixed time window, oldest window first.""" + import numpy as np + + array = np.asarray(samples, dtype=np.float32).reshape(-1) + return [read(detector, array[start:end]) for start, end in window_bounds(array.shape[0], window_s, SAMPLE_RATE)] + + +def detect_sample_array(detector: object, samples: object) -> str | None: + """Return the decoded message label when the detector fires, else None.""" + reading = read(detector, samples) + return reading.label if reading.detected else None diff --git a/scripts/videoseal_oracle.py b/scripts/videoseal_oracle.py new file mode 100644 index 0000000..96fc8d3 --- /dev/null +++ b/scripts/videoseal_oracle.py @@ -0,0 +1,231 @@ +#!/usr/bin/env python3 +"""Revision-pinned local VideoSeal oracle for development-only benchmarks. + +Loads Meta's standalone TorchScript release of VideoSeal (MIT license, 256-bit +message) with nothing but PyTorch: no ``videoseal`` package, whose wheel pulls +a research dependency chain (decord without macOS arm64 wheels, timm, +pycocotools) and whose loader resolves model cards and attenuation configs +relative to the working directory. The TorchScript build published in the +project's ``docs/torchscript.md`` guide exposes the same embed/detect surface +as one self-contained file. + +The checkpoint is pinned by URL, source commit, and SHA-256 digest, downloaded +once into a user-cache directory outside any repository, and verified before +every load. The detector's own "detection bit" column is unused by the +upstream video evaluation (it prepends a constant); the practical detection +signal is the aggregated message decode, so this oracle's matched decision +rule is bit accuracy of the 256-bit aggregated decode against the oracle's +fixed message. The 0.9 threshold is this adapter's explicit rule, not an +upstream constant; upstream reports accuracy curves rather than a verdict. + +Nothing here contacts a provenance oracle or provider API, and none of this +ships in the installed CLI or the public Python API. +""" + +from __future__ import annotations + +import hashlib +import os +import urllib.request +from dataclasses import dataclass +from pathlib import Path +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Sequence + +JIT_URL = "https://dl.fbaipublicfiles.com/videoseal/y_256b_img.jit" +JIT_SHA256 = "5c7a4581c36fc6090aafdcfb3999123bae5172a4847f22e2da4e7fd1a39d1e1b" +SOURCE_COMMIT = "870ca7fb33578b90f14c602016b6c2788096226e" +N_BITS = 256 +MESSAGE_SEED = 7 +DETECTION_BIT_ACCURACY_THRESHOLD = 0.9 +AGGREGATION = "avg" +AGGREGATIONS: tuple[str, ...] = ("avg", "squared_avg", "l1norm_avg", "l2norm_avg") +CACHE_DIR = Path( + os.environ.get( + "VIDEOSEAL_CACHE_DIR", + str(Path.home() / ".cache" / "remove-ai-watermarks-dev" / "videoseal"), + ) +) + + +@dataclass(frozen=True) +class VideoSealReading: + """Aggregated decode of one video against the oracle's fixed message.""" + + bit_accuracy: float + decoded_bits: tuple[int, ...] + per_frame_bit_accuracy: tuple[float, ...] + + @property + def detected(self) -> bool: + """This adapter's explicit matched-detection rule.""" + return self.bit_accuracy >= DETECTION_BIT_ACCURACY_THRESHOLD + + @property + def label(self) -> str: + return "".join(f"{byte:02x}" for byte in _bytes(self.decoded_bits)) + + +def _bytes(bits: Sequence[int]) -> bytes: + return bytes(sum(bit << (7 - index) for index, bit in enumerate(bits[i : i + 8])) for i in range(0, len(bits), 8)) + + +def message_bits(seed: int = MESSAGE_SEED) -> tuple[int, ...]: + """The oracle's fixed 256-bit message, deterministic across processes.""" + import numpy as np + + rng = np.random.default_rng(seed) + return tuple(int(bit) for bit in rng.integers(0, 2, N_BITS)) + + +def import_failure() -> ImportError | None: + """Return the missing-dependency error, or None when torch is present.""" + try: + import torch # noqa: F401 + except ImportError as exc: + return exc + return None + + +def available() -> bool: + """Whether the TorchScript stack is importable in this environment.""" + return import_failure() is None + + +def _sha256_file(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as stream: + for chunk in iter(lambda: stream.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def checkpoint_path() -> Path: + """Return the pinned local checkpoint, downloading and verifying it once.""" + target = CACHE_DIR / "y_256b_img.jit" + if target.is_file() and _sha256_file(target) == JIT_SHA256: + return target + CACHE_DIR.mkdir(parents=True, exist_ok=True) + temporary = CACHE_DIR / "y_256b_img.jit.part" + print(f"downloading pinned VideoSeal TorchScript checkpoint from {JIT_URL}") + urllib.request.urlretrieve(JIT_URL, temporary) + digest = _sha256_file(temporary) + if digest != JIT_SHA256: + temporary.unlink(missing_ok=True) + raise RuntimeError(f"pinned VideoSeal checkpoint digest mismatch: {digest} != {JIT_SHA256}") + temporary.replace(target) + return target + + +def load_model() -> object: + """Load the pinned TorchScript marker/detector on the CPU.""" + failure = import_failure() + if failure is not None: + raise RuntimeError("videoseal oracle requires the dev extra: uv sync --extra dev (torch)") from failure + import torch + + model = torch.jit.load(str(checkpoint_path()), map_location="cpu") + model.eval() + return model + + +def embed(model: object, frames: object, message: Sequence[int]) -> object: + """Embed the fixed message into float RGB frames in [0, 1] (T, H, W, 3).""" + import numpy as np + import torch + + array = torch.from_numpy(np.ascontiguousarray(frames, dtype=np.float32)) + tensor = array.permute(0, 3, 1, 2) + msg = torch.tensor([list(message)], dtype=torch.float32) + with torch.no_grad(): + marked = model.embed(tensor, msg, is_video=True) + return marked.permute(0, 2, 3, 1).numpy() + + +def _aggregate_bit_preds(bit_preds: object, aggregation: str) -> object: + """Aggregate per-frame bit logits exactly as upstream ``extract_message``. + + The TorchScript build ignores the aggregation argument of + ``detect_video_and_aggregate`` (identical bool output for every choice), + so the aggregation is computed here from the raw per-frame logits using + the upstream formulas. + """ + import torch + + if aggregation == "avg": + return bit_preds.mean(dim=0) + if aggregation == "squared_avg": + return (bit_preds * bit_preds.abs()).mean(dim=0) + if aggregation == "l1norm_avg": + weights = torch.norm(bit_preds, p=1, dim=1).unsqueeze(1) + return (bit_preds * weights).mean(dim=0) + if aggregation == "l2norm_avg": + weights = torch.norm(bit_preds, p=2, dim=1).unsqueeze(1) + return (bit_preds * weights).mean(dim=0) + raise ValueError(f"unknown aggregation {aggregation!r}; expected one of {AGGREGATIONS}") + + +def _raw_preds(model: object, tensor: object) -> object: + """One detector pass; the TorchScript build returns the tensor directly.""" + preds = model.detect(tensor, is_video=True) + if isinstance(preds, dict): + preds = preds["preds"] + return preds + + +def read( + model: object, + frames: object, + *, + message: Sequence[int] | None = None, + aggregation: str = AGGREGATION, +) -> VideoSealReading: + """Decode one video: aggregated bits plus per-frame bit accuracy.""" + if aggregation not in AGGREGATIONS: + raise ValueError(f"unknown aggregation {aggregation!r}; expected one of {AGGREGATIONS}") + import numpy as np + import torch + + expected = message_bits() if message is None else tuple(message) + array = torch.from_numpy(np.ascontiguousarray(frames, dtype=np.float32)) + tensor = array.permute(0, 3, 1, 2) + with torch.no_grad(): + preds = _raw_preds(model, tensor) + bit_preds = preds[:, 1:] + aggregated = _aggregate_bit_preds(bit_preds, aggregation) + decoded = tuple(int(bit) for bit in (aggregated.view(-1) > 0).to(torch.int64).tolist()) + accuracy = sum(int(bit == want) for bit, want in zip(decoded, expected, strict=True)) / N_BITS + frame_bits = (bit_preds > 0).to(torch.int64) + want = torch.tensor(expected, dtype=torch.int64).view(1, -1) + per_frame = (frame_bits == want).float().mean(dim=1).tolist() + return VideoSealReading( + bit_accuracy=accuracy, + decoded_bits=decoded, + per_frame_bit_accuracy=tuple(float(value) for value in per_frame), + ) + + +def read_aggregation_matrix(model: object, frames: object) -> dict[str, float]: + """Bit accuracy under every supported frame aggregation, one detect pass.""" + import numpy as np + import torch + + expected = message_bits() + array = torch.from_numpy(np.ascontiguousarray(frames, dtype=np.float32)) + tensor = array.permute(0, 3, 1, 2) + with torch.no_grad(): + bit_preds = _raw_preds(model, tensor)[:, 1:] + matrix: dict[str, float] = {} + for aggregation in AGGREGATIONS: + aggregated = _aggregate_bit_preds(bit_preds, aggregation) + decoded = (aggregated.view(-1) > 0).to(torch.int64).tolist() + matrix[aggregation] = sum(int(b == w) for b, w in zip(decoded, expected, strict=True)) / N_BITS + return matrix + + +def detect_sample_array(model: object, frames: object) -> str | None: + """Benchmark-kernel seam: the decoded message label, or None when absent.""" + reading = read(model, frames) + return reading.label if reading.detected else None diff --git a/scripts/videoseal_temporal_study.py b/scripts/videoseal_temporal_study.py new file mode 100644 index 0000000..465ef0d --- /dev/null +++ b/scripts/videoseal_temporal_study.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +"""Development-only VideoSeal temporal evaluation study. + +Three questions the case-level benchmark verdict aggregates away: + +- how the decoded bit accuracy decays across H.264 quality (crf 18-32), + downscaling, and frame-rate halving; +- whether the frame-aggregation choice (avg, squared_avg, l1norm_avg, + l2norm_avg) changes the outcome on the same artifact; +- how the per-frame bit accuracy moves in time on real provider clips versus + synthetic carriers. + +Carriers are the two synthetic cohort clips plus committed real provider +videos (their bytes already carry the project's public test clearance). No +provider API is contacted; artifacts stay outside the repository under +``.local-eval/``. + + uv run --extra dev python scripts/videoseal_temporal_study.py \ + --output-dir .local-eval/videoseal-temporal-v1 +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import logging +import shutil +import subprocess +import sys +from pathlib import Path + +import numpy as np + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) +sys.path.insert(0, str(REPO / "scripts")) + +import videoseal_oracle # noqa: E402 +from audioseal_experiment import ffmpeg_version # noqa: E402 +from videoseal_oracle import read, read_aggregation_matrix # noqa: E402 +from watermark_benchmark import _decode_video, sha256_file # noqa: E402 +from watermark_benchmark_video_cohort import ( # noqa: E402 + FRAME_RATE, + encode_clip, + synth_carrier, +) + +log = logging.getLogger("videoseal_temporal_study") + +CRF_SWEEP: tuple[int, ...] = (18, 23, 28, 32) +ATTACKS: tuple[str, ...] = ("scale_075", "fps_half") +REAL_CARRIERS: tuple[str, ...] = ("veo", "sora") +STUDY_FRAMES = 64 + + +def require_tools() -> str: + ffmpeg = shutil.which("ffmpeg") + if ffmpeg is None: + raise SystemExit("videoseal temporal study requires ffmpeg on PATH") + return ffmpeg + + +def real_carrier_frames(name: str) -> tuple[np.ndarray, dict[str, object]]: + """Decode a committed provider clip to a square 256x256 64-frame prefix.""" + import cv2 + + source = REPO / "data" / "fixtures" / "visible" / name / "provider-original.mp4" + if not source.is_file(): + raise SystemExit(f"provider fixture missing: {source}") + frames = _decode_video(source) + if frames is None or frames.shape[0] < STUDY_FRAMES: + raise SystemExit(f"provider fixture {name} did not decode to {STUDY_FRAMES} frames") + square = frames[:STUDY_FRAMES] + height, width = square.shape[1:3] + side = min(height, width) + top = (height - side) // 2 + left = (width - side) // 2 + cropped = square[:, top : top + side, left : left + side, :] + resized = np.stack([cv2.resize(frame, (256, 256), interpolation=cv2.INTER_AREA) for frame in cropped]).astype( + np.float32 + ) + provenance = { + "source": f"data/fixtures/visible/{name}/provider-original.mp4", + "source_sha256": sha256_file(source), + "source_geometry": [int(height), int(width)], + "recipe": "first-64-frames, center-crop to square, INTER_AREA resize to 256x256", + } + return resized, provenance + + +def apply_crf(ffmpeg: str, marked: Path, crf: int, workdir: Path) -> Path: + target = workdir / f"{marked.stem}-crf{crf}.mp4" + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + ffmpeg, + "-y", + "-loglevel", + "error", + "-i", + str(marked), + "-c:v", + "libx264", + "-crf", + str(crf), + "-pix_fmt", + "yuv420p", + str(target), + ], + check=True, + ) + return target + + +def apply_scale(ffmpeg: str, marked: Path, workdir: Path) -> Path: + target = workdir / f"{marked.stem}-scale_075.mp4" + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + ffmpeg, + "-y", + "-loglevel", + "error", + "-i", + str(marked), + "-vf", + "scale=192:192", + "-c:v", + "libx264", + "-crf", + "18", + "-pix_fmt", + "yuv420p", + str(target), + ], + check=True, + ) + return target + + +def apply_fps_half(ffmpeg: str, marked: Path, workdir: Path) -> Path: + target = workdir / f"{marked.stem}-fps_half.mp4" + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + ffmpeg, + "-y", + "-loglevel", + "error", + "-i", + str(marked), + "-vf", + f"fps={FRAME_RATE / 2:.12g}", + "-c:v", + "libx264", + "-crf", + "8", + "-pix_fmt", + "yuv420p", + str(target), + ], + check=True, + ) + return target + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + out_dir: Path = args.output_dir.resolve() + if out_dir.exists(): + raise SystemExit(f"refusing to overwrite existing output directory {out_dir}") + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + + ffmpeg = require_tools() + out_dir.mkdir(parents=True) + model = videoseal_oracle.load_model() + message = videoseal_oracle.message_bits() + + rows: list[dict[str, object]] = [] + + def record(stage: str, **fields: object) -> None: + row = {"stage": stage, **fields} + rows.append(row) + log.info("%s", json.dumps(row, sort_keys=True, default=str)[:400]) + + import importlib.metadata as md + + record( + "pins", + torch=md.version("torch"), + videoseal_jit_sha256=videoseal_oracle.JIT_SHA256, + study_source_sha256=sha256_file(Path(__file__)), + ffmpeg=ffmpeg_version(ffmpeg), + message_sha256=hashlib.sha256(bytes(message)).hexdigest(), + ) + + carriers: dict[str, tuple[np.ndarray, dict[str, object]]] = { + name: (synth_carrier(name)[:STUDY_FRAMES], {"recipe": "synthetic cohort carrier"}) + for name in ("moving_gradient", "moving_texture") + } + for name in REAL_CARRIERS: + carriers[f"real_{name}"] = real_carrier_frames(name) + + for name, (clean_frames, provenance) in carriers.items(): + workdir = out_dir / name + workdir.mkdir() + clean_path = encode_clip(ffmpeg, workdir / f"{name}-clean.mp4", clean_frames) + record( + "carrier_clean", + carrier=name, + sha256=sha256_file(clean_path), + aggregation_matrix=read_aggregation_matrix(model, clean_frames), + **provenance, + ) + + marked_frames = np.asarray(videoseal_oracle.embed(model, clean_frames, message), dtype=np.float32) + marked_path = encode_clip(ffmpeg, workdir / f"{name}-marked.mp4", marked_frames) + marked_read = read(model, marked_frames) + record( + "carrier_marked", + carrier=name, + sha256=sha256_file(marked_path), + bit_accuracy_avg=marked_read.bit_accuracy, + aggregation_matrix=read_aggregation_matrix(model, marked_frames), + per_frame_bit_accuracy=[round(v, 4) for v in marked_read.per_frame_bit_accuracy], + wm_snr_db=round( + float( + 10 + * np.log10( + np.sum(clean_frames.astype(np.float64) ** 2) + / max(np.sum((marked_frames - clean_frames).astype(np.float64) ** 2), 1e-12) + ) + ), + 3, + ), + ) + + attacked: list[tuple[str, Path]] = [ + (f"crf{crf}", apply_crf(ffmpeg, marked_path, crf, workdir)) for crf in CRF_SWEEP + ] + attacked += [ + ("scale_075", apply_scale(ffmpeg, marked_path, workdir)), + ("fps_half", apply_fps_half(ffmpeg, marked_path, workdir)), + ] + for attack, path in attacked: + frames = _decode_video(path) + if frames is None: + raise SystemExit(f"attack artifact failed to decode: {path}") + reading = read(model, frames) + record( + "attack", + carrier=name, + attack=attack, + sha256=sha256_file(path), + frames=frames.shape[0], + bit_accuracy_avg=reading.bit_accuracy, + aggregation_matrix=read_aggregation_matrix(model, frames), + per_frame_bit_accuracy=[round(v, 4) for v in reading.per_frame_bit_accuracy], + ) + + cases_path = out_dir / "cases.jsonl" + cases_path.write_text("".join(json.dumps(r, default=str) + "\n" for r in rows)) + summary_path = out_dir / "matrix.md" + summary_path.write_text(render_matrix(rows)) + print(f"wrote {cases_path}") + print(f"wrote {summary_path}") + return 0 + + +def cast_to_matrix(value: object) -> dict[str, float]: + if not isinstance(value, dict): + raise TypeError(f"aggregation matrix must be a dict, got {type(value).__name__}") + return value # type: ignore[no-any-return] + + +def render_matrix(rows: list[dict[str, object]]) -> str: + aggregations = videoseal_oracle.AGGREGATIONS + lines = [ + "# VideoSeal temporal evaluation study", + "", + "Bit accuracy of the aggregated 256-bit decode per carrier, attack, and", + "frame aggregation. The benchmark threshold (0.9) is the adapter's", + "verdict rule; this matrix shows the readings underneath it.", + "", + "| Carrier | Attack | " + " | ".join(aggregations) + " |", + "| --- | --- | " + " | ".join(["---"] * len(aggregations)) + " |", + ] + clean_rows = {str(r["carrier"]): r for r in rows if r["stage"] == "carrier_clean"} + for carrier, row in sorted(clean_rows.items()): + matrix = cast_to_matrix(row["aggregation_matrix"]) + lines.append(f"| {carrier} | (clean) | " + " | ".join(f"{float(matrix[a]):.3f}" for a in aggregations) + " |") + for r in rows: + if r["stage"] != "attack": + continue + matrix = cast_to_matrix(r["aggregation_matrix"]) + lines.append( + f"| {r['carrier']} | {r['attack']} | " + " | ".join(f"{float(matrix[a]):.3f}" for a in aggregations) + " |" + ) + lines += [ + "", + "Per-frame bit accuracy ranges (avg aggregation) for marked clips:", + "", + "| Carrier | min | mean | max |", + "| --- | --- | --- | --- |", + ] + for r in rows: + if r["stage"] != "carrier_marked": + continue + per_frame = [float(v) for v in r["per_frame_bit_accuracy"]] # type: ignore[arg-type] + lines.append( + f"| {r['carrier']} | {min(per_frame):.3f} | {sum(per_frame) / len(per_frame):.3f} | {max(per_frame):.3f} |" + ) + lines.append("") + return "\n".join(lines) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/scripts/watermark_benchmark.py b/scripts/watermark_benchmark.py index 1a40933..cfdc21f 100644 --- a/scripts/watermark_benchmark.py +++ b/scripts/watermark_benchmark.py @@ -3,19 +3,24 @@ The input is a strict JSONL manifest. Every row names one immutable artifact, its evidence arm and processing state, the detector adapter, source and -transform revisions, an explicit seed, and an optional decoded-pixel reference. -Artifacts and references are hash-checked before any detector runs. +transform revisions, an explicit seed, and an optional decoded-media +reference. Artifacts and references are hash-checked before any detector +runs. Image rows decode through the image pipeline; audio rows decode to mono +16 kHz float32 through the system ffmpeg and are measured in the sample +domain. The output keeps three questions separate: * ``detection`` records only what one named adapter recognized; * ``removal`` records the post-removal observation without certifying erasure; -* ``fidelity`` measures decoded-pixel distance from an explicit reference. +* ``fidelity`` measures the decoded-media distance from an explicit reference. This is a maintainer tool, not an installed command. It calls no provenance oracle or provider API, does not download corpora, and refuses to overwrite an existing report. The optional TrustMark dependency may fetch its official model -weights when its local package cache is incomplete. +weights when its local package cache is incomplete; the optional audioseal +adapter loads revision-pinned weights through the shared local oracle the same +way. uv run python scripts/watermark_benchmark.py manifest.jsonl --output report.jsonl """ @@ -52,15 +57,23 @@ from remove_ai_watermarks._internal.schema import require_schema_version # noqa log = logging.getLogger(__name__) SCHEMA_VERSION = 1 +Media = Literal["image", "audio", "video"] Arm = Literal["positive", "matched_negative", "wrong_key", "hard_negative"] -State = Literal["clean", "marked", "attacked", "removed"] +State = Literal["clean", "marked", "attacked", "removed", "forged"] ExpectedDetection = Literal["detected", "not_detected", "unresolved"] DetectorStatus = Literal["detected", "not_detected", "unavailable", "error"] _ARMS: tuple[Arm, ...] = ("positive", "matched_negative", "wrong_key", "hard_negative") -_STATES: tuple[State, ...] = ("clean", "marked", "attacked", "removed") +_STATES: tuple[State, ...] = ("clean", "marked", "attacked", "removed", "forged") _EXPECTED: tuple[ExpectedDetection, ...] = ("detected", "not_detected", "unresolved") DETECTOR_STATUSES: tuple[DetectorStatus, ...] = ("detected", "not_detected", "unavailable", "error") +MEDIA_TYPES: tuple[Media, ...] = ("image", "audio", "video") +ADAPTER_MEDIA: dict[str, Media] = { + "dwt-dct": "image", + "trustmark": "image", + "audioseal": "audio", + "videoseal": "video", +} _FIELDS = { "schema_version", "case_id", @@ -96,7 +109,7 @@ class BenchmarkCase: case_id: str pair_id: str - media_type: Literal["image"] + media_type: Media adapter: str arm: Arm state: State @@ -117,6 +130,7 @@ class DetectorOutcome: status: DetectorStatus label: str | None error: str | None = None + temporal: tuple[float, ...] | None = None class DetectorAdapter(Protocol): @@ -131,7 +145,7 @@ class DetectorAdapter(Protocol): @property def available(self) -> bool: ... - def detect(self, path: Path, image: NDArray[Any]) -> DetectorOutcome: ... + def detect(self, path: Path, artifact: NDArray[Any]) -> DetectorOutcome: ... @dataclass(frozen=True) @@ -143,11 +157,30 @@ class FunctionAdapter: available: bool detector: Callable[[Path, NDArray[Any]], str | None] - def detect(self, path: Path, image: NDArray[Any]) -> DetectorOutcome: - label = self.detector(path, image) + def detect(self, path: Path, artifact: NDArray[Any]) -> DetectorOutcome: + label = self.detector(path, artifact) return DetectorOutcome(status="detected" if label is not None else "not_detected", label=label) +@dataclass(frozen=True) +class VideoSealAdapter: + """Adapter whose outcome also carries per-frame temporal evidence.""" + + name: str + source_file: Path + available: bool + + def detect(self, path: Path, frames: NDArray[Any]) -> DetectorOutcome: + from videoseal_oracle import read + + reading = read(_videoseal_model(), frames) + return DetectorOutcome( + status="detected" if reading.detected else "not_detected", + label=reading.label if reading.detected else None, + temporal=reading.per_frame_bit_accuracy, + ) + + def sha256_file(path: Path) -> str: """Return the lowercase SHA-256 digest of one file.""" digest = hashlib.sha256() @@ -264,8 +297,12 @@ def _case( raise ValueError(f"{location}: {exc}") from exc media_type = values["media_type"] - if media_type != "image": - raise ValueError(f"{location}: media_type must be image in schema v{SCHEMA_VERSION}") + if media_type not in MEDIA_TYPES: + raise ValueError(f"{location}: media_type must be one of {', '.join(MEDIA_TYPES)} in schema v{SCHEMA_VERSION}") + adapter = require_nonempty_string(values["adapter"], field="adapter", location=location) + expected_media = ADAPTER_MEDIA.get(adapter) + if expected_media is not None and expected_media != media_type: + raise ValueError(f"{location}: adapter {adapter!r} measures {expected_media} artifacts, not {media_type}") arm = values["arm"] if arm not in _ARMS: raise ValueError(f"{location}: arm must be one of {', '.join(_ARMS)}") @@ -306,8 +343,8 @@ def _case( return BenchmarkCase( case_id=require_nonempty_string(values["case_id"], field="case_id", location=location), pair_id=require_nonempty_string(values["pair_id"], field="pair_id", location=location), - media_type="image", - adapter=require_nonempty_string(values["adapter"], field="adapter", location=location), + media_type=media_type, + adapter=adapter, arm=arm, state=state, path=path, @@ -365,11 +402,19 @@ def _detect_trustmark(path: Path, image: NDArray[Any]) -> str | None: def default_adapters() -> dict[str, DetectorAdapter]: - """Return the first two local, open image-watermark adapters.""" + """Return the local, open watermark adapters for both media types.""" from remove_ai_watermarks import invisible_watermark, trustmark_detector + scripts_dir = Path(__file__).resolve().parent + if str(scripts_dir) not in sys.path: + sys.path.insert(0, str(scripts_dir)) + import audioseal_oracle + import videoseal_oracle + dwt_source = Path(invisible_watermark.__file__).resolve() trustmark_source = Path(trustmark_detector.__file__).resolve() + audio_oracle_source = Path(audioseal_oracle.__file__).resolve() + video_oracle_source = Path(videoseal_oracle.__file__).resolve() return { "dwt-dct": FunctionAdapter( name="dwt-dct", @@ -383,6 +428,17 @@ def default_adapters() -> dict[str, DetectorAdapter]: available=trustmark_detector.is_available(), detector=_detect_trustmark, ), + "audioseal": FunctionAdapter( + name="audioseal", + source_file=audio_oracle_source, + available=audioseal_oracle.available(), + detector=_detect_audioseal, + ), + "videoseal": VideoSealAdapter( + name="videoseal", + source_file=video_oracle_source, + available=videoseal_oracle.available(), + ), } @@ -418,6 +474,113 @@ def _decode_image(path: Path) -> NDArray[Any] | None: return imread(path, cv2.IMREAD_UNCHANGED) +def _decode_audio(path: Path) -> NDArray[Any] | None: + """Decode any audio artifact to mono 16 kHz float32 via system ffmpeg.""" + import subprocess + + import numpy as np + + ffmpeg = shutil.which("ffmpeg") + if ffmpeg is None: + return None + result = subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ffmpeg, "-loglevel", "error", "-i", str(path), "-ac", "1", "-ar", "16000", "-f", "f32le", "pipe:1"], + capture_output=True, + check=False, + ) + if result.returncode != 0 or not result.stdout: + return None + return np.frombuffer(result.stdout, dtype=" NDArray[Any] | None: + """Decode a video artifact to float RGB frames in [0, 1] as (T, H, W, 3).""" + import subprocess + + import numpy as np + + ffmpeg = shutil.which("ffmpeg") + ffprobe = shutil.which("ffprobe") + if ffmpeg is None or ffprobe is None: + return None + probe = subprocess.run( # noqa: S603 - resolved ffprobe with fixed arguments + [ + ffprobe, + "-v", + "error", + "-select_streams", + "v:0", + "-show_entries", + "stream=width,height", + "-of", + "csv=p=0", + str(path), + ], + capture_output=True, + text=True, + check=False, + ) + if probe.returncode != 0: + return None + try: + height, width = (int(value) for value in probe.stdout.strip().split(",")) + except ValueError: + return None + result = subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + ffmpeg, + "-loglevel", + "error", + "-i", + str(path), + "-map", + "0:v:0", + "-f", + "rawvideo", + "-pix_fmt", + "rgb24", + "pipe:1", + ], + capture_output=True, + check=False, + ) + if result.returncode != 0 or not result.stdout: + return None + frame_size = height * width * 3 + if len(result.stdout) % frame_size != 0: + return None + frames = np.frombuffer(result.stdout, dtype=np.uint8).reshape(-1, height, width, 3) + return frames.astype(np.float32) / 255.0 + + +def _detect_audioseal(path: Path, samples: NDArray[Any]) -> str | None: + """Read the pinned AudioSeal oracle; the label is the decoded message.""" + from audioseal_oracle import detect_sample_array + + return detect_sample_array(_audioseal_detector(), samples) + + +@cache +def _audioseal_detector() -> object: + scripts_dir = Path(__file__).resolve().parent + if str(scripts_dir) not in sys.path: + sys.path.insert(0, str(scripts_dir)) + from audioseal_oracle import load_pinned_models + + _generator, detector = load_pinned_models() + return detector + + +@cache +def _videoseal_model() -> object: + scripts_dir = Path(__file__).resolve().parent + if str(scripts_dir) not in sys.path: + sys.path.insert(0, str(scripts_dir)) + from videoseal_oracle import load_model + + return load_model() + + def _image_fidelity( case: BenchmarkCase, artifact: NDArray[Any] | None, @@ -463,9 +626,84 @@ def _image_fidelity( } -def _detect(adapter: DetectorAdapter, path: Path, image: NDArray[Any]) -> DetectorOutcome: +def _audio_fidelity( + case: BenchmarkCase, + artifact: NDArray[Any] | None, + reference_decoder: Callable[[Path], NDArray[Any] | None], +) -> dict[str, Any]: + """Compare decoded mono 16 kHz samples against an explicit reference.""" + if case.reference_path is None: + return {"status": "not_measured", "reason": "reference_not_provided"} + + import numpy as np + + reference = artifact if case.reference_path == case.path else reference_decoder(case.reference_path) + if artifact is None or reference is None: + return {"status": "not_measured", "reason": "decoded_samples_unavailable"} + artifact_samples = np.asarray(artifact, dtype=np.float32).reshape(-1) + reference_samples = np.asarray(reference, dtype=np.float32).reshape(-1) + if artifact_samples.shape != reference_samples.shape: + return { + "status": "incomparable", + "reason": "length_mismatch", + "artifact_samples": int(artifact_samples.shape[0]), + "reference_samples": int(reference_samples.shape[0]), + } + delta = artifact_samples.astype(np.float64) - reference_samples.astype(np.float64) + reference_power = float(np.sum(reference_samples.astype(np.float64) ** 2)) + error_power = float(np.sum(delta**2)) + identical = error_power == 0.0 + return { + "status": "measured", + "identical": identical, + "max_abs_error": float(np.max(np.abs(delta))), + "snr_db": None if identical or reference_power == 0.0 else 10.0 * math.log10(reference_power / error_power), + "snr_status": ( + "unbounded_identical" if identical else ("zero_reference_power" if reference_power == 0.0 else "measured") + ), + } + + +def _video_fidelity( + case: BenchmarkCase, + artifact: NDArray[Any] | None, + reference_decoder: Callable[[Path], NDArray[Any] | None], +) -> dict[str, Any]: + """Compare decoded RGB frames against an explicit reference clip.""" + if case.reference_path is None: + return {"status": "not_measured", "reason": "reference_not_provided"} + + import numpy as np + + reference = artifact if case.reference_path == case.path else reference_decoder(case.reference_path) + if artifact is None or reference is None: + return {"status": "not_measured", "reason": "decoded_frames_unavailable"} + artifact_frames = np.asarray(artifact, dtype=np.float32) + reference_frames = np.asarray(reference, dtype=np.float32) + if artifact_frames.shape != reference_frames.shape: + return { + "status": "incomparable", + "reason": "shape_mismatch", + "artifact_shape": list(artifact_frames.shape), + "reference_shape": list(reference_frames.shape), + } + delta = artifact_frames.astype(np.float64) - reference_frames.astype(np.float64) + per_frame_mse = np.mean(np.square(delta), axis=(1, 2, 3)) + identical = bool(np.all(per_frame_mse == 0.0)) + changed_frames = np.asarray(per_frame_mse) > 0.0 + finite = per_frame_mse[per_frame_mse > 0.0] + return { + "status": "measured", + "identical": identical, + "changed_frame_fraction": float(np.mean(changed_frames)), + "mean_psnr_db": None if identical or finite.size == 0 else float(np.mean(10.0 * np.log10(1.0 / finite))), + "psnr_status": "unbounded_identical" if identical else "measured", + } + + +def _detect(adapter: DetectorAdapter, path: Path, artifact: NDArray[Any]) -> DetectorOutcome: try: - return adapter.detect(path, image) + return adapter.detect(path, artifact) except Exception as exc: log.warning("Benchmark adapter %s failed for %s: %s", adapter.name, path, exc) return DetectorOutcome(status="error", label=None, error=f"{type(exc).__name__}: {exc}") @@ -485,6 +723,15 @@ def _detection_record( "positive_evidence": outcome.status == "detected", "adapter_elapsed_ms": adapter_elapsed_ms, } + if outcome.temporal is not None: + values = [float(value) for value in outcome.temporal] + record["temporal"] = { + "frames": len(values), + "min_bit_accuracy": min(values) if values else None, + "mean_bit_accuracy": sum(values) / len(values) if values else None, + "max_bit_accuracy": max(values) if values else None, + "per_frame_bit_accuracy": [round(value, 6) for value in values], + } if outcome.error is not None: record["error"] = outcome.error return record @@ -511,8 +758,8 @@ def evaluate_case( *, adapters: Mapping[str, DetectorAdapter], repository: Mapping[str, str | bool], - artifact_decoder: Callable[[Path], NDArray[Any] | None] = _decode_image, - reference_decoder: Callable[[Path], NDArray[Any] | None] = _decode_image, + artifact_decoder: Callable[[Path], NDArray[Any] | None] | None = None, + reference_decoder: Callable[[Path], NDArray[Any] | None] | None = None, clock_ns: Callable[[], int] = time.perf_counter_ns, ) -> dict[str, Any]: """Evaluate one case while keeping detector and fidelity claims distinct.""" @@ -520,6 +767,10 @@ def evaluate_case( adapter = adapters[case.adapter] except KeyError as exc: raise ValueError(f"unknown adapter {case.adapter!r}") from exc + if artifact_decoder is None: + artifact_decoder = _default_decoder(case.media_type) + if reference_decoder is None: + reference_decoder = _default_decoder(case.media_type) artifact = artifact_decoder(case.path) adapter_elapsed_ms: float | None = None if artifact is None: @@ -556,10 +807,32 @@ def evaluate_case( }, "detection": _detection_record(outcome, case.expected, adapter_elapsed_ms), "removal": _removal_record(case.state, outcome), - "fidelity": _image_fidelity(case, artifact, reference_decoder), + "fidelity": _fidelity_for(case, artifact, reference_decoder), } +def _default_decoder(media: Media) -> Callable[[Path], NDArray[Any] | None]: + """The per-media artifact decoder used when a caller injects none.""" + if media == "image": + return _decode_image + if media == "audio": + return _decode_audio + return _decode_video + + +def _fidelity_for( + case: BenchmarkCase, + artifact: NDArray[Any] | None, + reference_decoder: Callable[[Path], NDArray[Any] | None], +) -> dict[str, Any]: + """Dispatch the fidelity contract by media type.""" + if case.media_type == "image": + return _image_fidelity(case, artifact, reference_decoder) + if case.media_type == "audio": + return _audio_fidelity(case, artifact, reference_decoder) + return _video_fidelity(case, artifact, reference_decoder) + + def write_jsonl(path: Path, records: Iterable[Mapping[str, Any]]) -> None: """Publish strict JSONL atomically and refuse to replace an existing report.""" if path.exists(): @@ -594,23 +867,29 @@ def run_benchmark(manifest: Path, output: Path) -> Counter[str]: counts: Counter[str] = Counter() path_counts = Counter(case.path for case in cases) path_counts.update(case.reference_path for case in cases if case.reference_path is not None) - recurring_images: dict[Path, NDArray[Any] | None] = {} + recurring_artifacts: dict[Path, NDArray[Any] | None] = {} - def decode_image(path: Path) -> NDArray[Any] | None: - if path_counts[path] < 2: - return _decode_image(path) - if path not in recurring_images: - recurring_images[path] = _decode_image(path) - return recurring_images[path] + def decoder_for(media: Media) -> Callable[[Path], NDArray[Any] | None]: + base = _default_decoder(media) + + def decode(path: Path) -> NDArray[Any] | None: + if path_counts[path] < 2: + return base(path) + if path not in recurring_artifacts: + recurring_artifacts[path] = base(path) + return recurring_artifacts[path] + + return decode def records() -> Iterable[dict[str, Any]]: for case in cases: + decoder = decoder_for(case.media_type) record = evaluate_case( case, adapters=adapters, repository=repository, - artifact_decoder=decode_image, - reference_decoder=decode_image, + artifact_decoder=decoder, + reference_decoder=decoder, ) counts[record["detection"]["status"]] += 1 yield record diff --git a/scripts/watermark_benchmark_audio_cohort.py b/scripts/watermark_benchmark_audio_cohort.py new file mode 100755 index 0000000..4b36f8e --- /dev/null +++ b/scripts/watermark_benchmark_audio_cohort.py @@ -0,0 +1,478 @@ +#!/usr/bin/env python3 +"""Build a local, synthetic audio-watermark benchmark cohort. + +The builder creates deterministic synthetic carriers, embeds the pinned +AudioSeal message through the shared local oracle, applies fixed audio attacks, +and writes every generated artifact once under a new output directory. It emits +the strict JSONL manifest consumed by ``watermark_benchmark.py`` with +``media_type: audio`` rows for the ``audioseal`` adapter. + +Generated artifacts and reports belong under ``.local-eval/`` and are never +committed. No provider oracle or API is used; the first build may download the +revision-pinned AudioSeal checkpoints when the local cache is incomplete. + + uv run --extra dev python scripts/watermark_benchmark_audio_cohort.py \ + --output-dir .local-eval/watermark-benchmark-audio-cohort-v1 +""" + +from __future__ import annotations + +import argparse +import importlib.metadata +import logging +import shutil +import sys +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np + +if TYPE_CHECKING: + from collections.abc import Sequence + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) +sys.path.insert(0, str(REPO / "scripts")) + +import audioseal_oracle # noqa: E402 +from audioseal_experiment import ( # noqa: E402 + FFMPEG_ATTACKS, + SAMPLE_RATE, + decode_audio_f32, + ffmpeg_attack, + ffmpeg_version, + message_bits, + numpy_attacks, + synth_carrier, + wav_pcm16_bytes, +) +from watermark_benchmark import SCHEMA_VERSION, load_manifest, sha256_file, write_jsonl # noqa: E402 + +log = logging.getLogger("watermark_benchmark_audio_cohort") + +RECIPE_VERSION = "watermark-benchmark-audio-cohort-v2" +FORGED_MESSAGE_SEED = 8 +# Carriers the 10 dB additive-noise transform measurably takes below the +# audioseal decision rule; the synthetic noise carriers survive it. +REMOVED_BY_NOISE: tuple[str, ...] = ("tone_stack",) +CARRIER_NAMES: tuple[str, ...] = ("tone_stack", "pinkish_noise", "white_noise") +HARD_NEGATIVE_CARRIER = "white_noise_hard_negative" +ATTACKS: tuple[str, ...] = ("mp3_128k", "aac_128k", "noise_snr10") +DEFAULT_DURATION_S = 8.0 + + +def require_tools() -> str: + ffmpeg = shutil.which("ffmpeg") + if ffmpeg is None: + raise SystemExit("audio cohort builder requires ffmpeg on PATH") + return ffmpeg + + +def oracle_dependencies() -> dict[str, str]: + """Return versions of everything that can affect cohort artifacts.""" + + def package(name: str) -> str: + try: + return importlib.metadata.version(name) + except importlib.metadata.PackageNotFoundError: + return "unavailable" + + return { + "numpy": package("numpy"), + "audioseal": package("audioseal"), + "torch": package("torch"), + "huggingface-revision": audioseal_oracle.HF_REVISION, + "ffmpeg": ffmpeg_version(require_tools()), + } + + +def _source_revision() -> str: + return f"{RECIPE_VERSION}@sha256:{sha256_file(Path(__file__))}" + + +def benchmark_row( + *, + root: Path, + case_id: str, + pair_id: str, + arm: str, + state: str, + path: Path, + reference: Path | None, + transform_name: str, + transform_revision: str, + parameters: dict[str, object], + seed: int, + expected: str, +) -> dict[str, object]: + """Build one strict benchmark-manifest row for a generated artifact.""" + return { + "schema_version": SCHEMA_VERSION, + "case_id": case_id, + "pair_id": pair_id, + "media_type": "audio", + "adapter": "audioseal", + "arm": arm, + "state": state, + "path": path.relative_to(root).as_posix(), + "sha256": sha256_file(path), + "reference_path": reference.relative_to(root).as_posix() if reference is not None else None, + "reference_sha256": sha256_file(reference) if reference is not None else None, + "source_revision": _source_revision(), + "transform": { + "name": transform_name, + "revision": transform_revision, + "parameters": parameters, + }, + "seed": seed, + "expected": expected, + } + + +def apply_attack(ffmpeg: str, attack: str, marked: Path, artifacts: Path) -> Path: + """Apply one named attack to the marked WAV and return its artifact path.""" + if attack in FFMPEG_ATTACKS: + suffix, ffargs = FFMPEG_ATTACKS[attack] + return ffmpeg_attack(ffmpeg, attack, suffix, ffargs, marked, artifacts, stem=marked.stem) + if attack == "noise_snr10": + samples = numpy_attacks(decode_audio_f32(ffmpeg, marked))[attack] + target = artifacts / f"{marked.stem}-{attack}.wav" + target.write_bytes(wav_pcm16_bytes(samples, SAMPLE_RATE)) + return target + raise ValueError(f"unknown attack {attack!r}") + + +def build_cohort( + output_dir: Path, + *, + carriers: Sequence[str] = CARRIER_NAMES, + attacks: Sequence[str] = ATTACKS, + duration_s: float = DEFAULT_DURATION_S, + include_speech: bool = False, +) -> Path: + """Build and validate a new audio cohort directory, returning its manifest.""" + if output_dir.exists(): + raise FileExistsError(f"output directory already exists: {output_dir}") + if not audioseal_oracle.available(): + raise SystemExit( + "audio cohort builder requires the dev extra: uv sync --extra dev (audioseal, torch, huggingface-hub)" + ) + unknown = sorted(set(carriers) - set(CARRIER_NAMES)) + if unknown: + raise ValueError(f"unknown carriers: {', '.join(unknown)}") + unknown_attacks = sorted(set(attacks) - set(ATTACKS)) + if unknown_attacks: + raise ValueError(f"unknown attacks: {', '.join(unknown_attacks)}") + + from audioseal_oracle import embed, load_pinned_models + + ffmpeg = require_tools() + output_dir.mkdir(parents=True) + artifacts = output_dir / "artifacts" + artifacts.mkdir() + dependencies = oracle_dependencies() + message = message_bits() + generator, _detector = load_pinned_models() + rows: list[dict[str, object]] = [] + + for name in carriers: + clean = synth_carrier(name, duration_s) + clean_path = artifacts / f"{name}-clean.wav" + clean_path.write_bytes(wav_pcm16_bytes(clean, SAMPLE_RATE)) + + marked = np.asarray(embed(generator, clean, message), dtype=np.float32) + marked_path = artifacts / f"{name}-marked.wav" + marked_path.write_bytes(wav_pcm16_bytes(marked, SAMPLE_RATE)) + + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-clean", + pair_id=name, + arm="matched_negative", + state="clean", + path=clean_path, + reference=None, + transform_name="synthesize-carrier", + transform_revision=RECIPE_VERSION, + parameters={ + "carrier": name, + "duration_s": duration_s, + "sample_rate": SAMPLE_RATE, + "dependencies": dependencies, + }, + seed=20260907, + expected="not_detected", + ) + ) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-marked", + pair_id=name, + arm="positive", + state="marked", + path=marked_path, + reference=clean_path, + transform_name="audioseal-embed", + transform_revision=f"audioseal@{dependencies['audioseal']}+hf@{audioseal_oracle.HF_REVISION}", + parameters={ + "message": message, + "sample_rate": SAMPLE_RATE, + "alpha": 1.0, + "dependencies": dependencies, + }, + seed=7, + expected="detected", + ) + ) + + for attack in attacks: + attacked_path = apply_attack(ffmpeg, attack, marked_path, artifacts) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-{attack}", + pair_id=name, + arm="positive", + state="attacked", + path=attacked_path, + reference=clean_path, + transform_name=f"attack-{attack}", + transform_revision=RECIPE_VERSION, + parameters=attack_parameters(attack, dependencies), + seed=7, + expected="unresolved", + ) + ) + + # The removed row asserts a removal outcome; only carriers the + # transform measurably clears qualify (see REMOVED_BY_NOISE). + removed_path = artifacts / f"{name}-marked-noise_snr10.wav" + if removed_path.is_file() and name in REMOVED_BY_NOISE: + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-removed", + pair_id=name, + arm="positive", + state="removed", + path=removed_path, + reference=clean_path, + transform_name="remove-noise-snr10", + transform_revision=RECIPE_VERSION, + parameters=attack_parameters("noise_snr10", dependencies), + seed=7, + expected="not_detected", + ) + ) + + # The forged row embeds a different message into the same clean + # carrier: a watermark is present, just not the oracle's, and the + # matched verifier's correct answer is not_detected. + forged_message = message_bits(FORGED_MESSAGE_SEED) + forged_samples = np.asarray(audioseal_oracle.embed(generator, clean, forged_message), dtype=np.float32) + forged_path = artifacts / f"{name}-forged.wav" + forged_path.write_bytes(wav_pcm16_bytes(forged_samples, SAMPLE_RATE)) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-forged", + pair_id=name, + arm="wrong_key", + state="forged", + path=forged_path, + reference=clean_path, + transform_name="audioseal-embed-foreign-message", + transform_revision=f"audioseal@{dependencies['audioseal']}+hf@{audioseal_oracle.HF_REVISION}", + parameters={ + "message": list(forged_message), + "message_seed": FORGED_MESSAGE_SEED, + "sample_rate": SAMPLE_RATE, + "alpha": 1.0, + "dependencies": dependencies, + }, + seed=FORGED_MESSAGE_SEED, + # The audioseal decision rule is watermark PRESENCE, so a + # foreign payload still reads detected; the decoded label is + # what distinguishes it from the oracle's message. + expected="detected", + ) + ) + + hard_negative = synth_carrier_hard_negative(duration_s) + hard_path = artifacts / f"{HARD_NEGATIVE_CARRIER}.wav" + hard_path.write_bytes(wav_pcm16_bytes(hard_negative, SAMPLE_RATE)) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{HARD_NEGATIVE_CARRIER}-clean", + pair_id=HARD_NEGATIVE_CARRIER, + arm="hard_negative", + state="clean", + path=hard_path, + reference=None, + transform_name="synthesize-carrier", + transform_revision=RECIPE_VERSION, + parameters={ + "carrier": "white_noise", + "seed_offset": 991, + "duration_s": duration_s, + "sample_rate": SAMPLE_RATE, + "dependencies": dependencies, + }, + seed=20260907 + 991, + expected="not_detected", + ) + ) + + if include_speech: + _append_speech_rows( + rows, + root=output_dir, + artifacts=artifacts, + ffmpeg=ffmpeg, + attacks=attacks, + dependencies=dependencies, + generator=generator, + message=message, + ) + + manifest = output_dir / "manifest.jsonl" + write_jsonl(manifest, rows) + load_manifest(manifest) + return manifest + + +def _append_speech_rows( + rows: list[dict[str, object]], + *, + root: Path, + artifacts: Path, + ffmpeg: str, + attacks: Sequence[str], + dependencies: dict[str, str], + generator: object, + message: list[int], +) -> None: + """Add system-voice speech carriers with their own TTS provenance.""" + from audioseal_experiment import ( + SPEECH_VOICES, + assert_distinct_voices, + decode_audio_f32, + speech_provenance, + synthesize_speech, + ) + + raw = {voice: synthesize_speech(voice, artifacts) for voice in SPEECH_VOICES} + assert_distinct_voices({voice: path.read_bytes() for voice, path in raw.items()}) + provenance = speech_provenance() + for voice, aiff in raw.items(): + pair = f"speech_{voice}" + clean = decode_audio_f32(ffmpeg, aiff) + clean_path = artifacts / f"{pair}-clean.wav" + clean_path.write_bytes(wav_pcm16_bytes(clean, SAMPLE_RATE)) + tts = {**provenance, "voice": voice, "duration_s": round(clean.shape[0] / SAMPLE_RATE, 3)} + + marked = np.asarray(audioseal_oracle.embed(generator, clean, message), dtype=np.float32) + marked_path = artifacts / f"{pair}-marked.wav" + marked_path.write_bytes(wav_pcm16_bytes(marked, SAMPLE_RATE)) + + rows.append( + benchmark_row( + root=root, + case_id=f"{pair}-clean", + pair_id=pair, + arm="matched_negative", + state="clean", + path=clean_path, + reference=None, + transform_name="synthesize-speech", + transform_revision=RECIPE_VERSION, + parameters={"tts": tts, "sample_rate": SAMPLE_RATE, "dependencies": dependencies}, + seed=20260907, + expected="not_detected", + ) + ) + rows.append( + benchmark_row( + root=root, + case_id=f"{pair}-marked", + pair_id=pair, + arm="positive", + state="marked", + path=marked_path, + reference=clean_path, + transform_name="audioseal-embed", + transform_revision=f"audioseal@{dependencies['audioseal']}+hf@{audioseal_oracle.HF_REVISION}", + parameters={ + "message": message, + "sample_rate": SAMPLE_RATE, + "alpha": 1.0, + "tts": tts, + "dependencies": dependencies, + }, + seed=7, + expected="detected", + ) + ) + for attack in attacks: + attacked_path = apply_attack(ffmpeg, attack, marked_path, artifacts) + rows.append( + benchmark_row( + root=root, + case_id=f"{pair}-{attack}", + pair_id=pair, + arm="positive", + state="attacked", + path=attacked_path, + reference=clean_path, + transform_name=f"attack-{attack}", + transform_revision=RECIPE_VERSION, + parameters={**attack_parameters(attack, dependencies), "tts": tts}, + seed=7, + expected="unresolved", + ) + ) + + +def synth_carrier_hard_negative(duration_s: float) -> np.ndarray: + """A never-embedded white-noise carrier keyed off a distinct seed.""" + rng = np.random.default_rng(20260907 + 991) + n = int(duration_s * SAMPLE_RATE) + return (rng.normal(0.0, 0.1, n)).astype(np.float32) + + +def attack_parameters(attack: str, dependencies: dict[str, str]) -> dict[str, object]: + """Return the exact parameters represented by an attack name.""" + if attack == "mp3_128k": + values: dict[str, object] = {"codec": "libmp3lame", "bitrate": "128k"} + elif attack == "aac_128k": + values = {"codec": "aac", "bitrate": "128k", "container": "m4a"} + elif attack == "noise_snr10": + values = {"additive_gaussian": True, "snr_db": 10.0} + else: + raise ValueError(f"unknown attack {attack!r}") + return {**values, "dependencies": dependencies} + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + parser.add_argument("--duration", type=float, default=DEFAULT_DURATION_S) + parser.add_argument( + "--with-speech", + action="store_true", + help="add deterministic system-voice speech carriers (requires the macOS say tool)", + ) + args = parser.parse_args() + logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") + try: + manifest = build_cohort(args.output_dir.resolve(), duration_s=args.duration, include_speech=args.with_speech) + except (FileExistsError, ValueError) as exc: + parser.error(str(exc)) + log.info("Wrote validated audio benchmark cohort to %s", manifest) + + +if __name__ == "__main__": + main() diff --git a/scripts/watermark_benchmark_report.py b/scripts/watermark_benchmark_report.py index 86363a8..80f654b 100755 --- a/scripts/watermark_benchmark_report.py +++ b/scripts/watermark_benchmark_report.py @@ -85,8 +85,8 @@ def _validate_result(raw: object, *, location: str) -> dict[str, Any]: raise ValueError(f"{location}: unsupported benchmark schema {values['schema_version']!r}") for field in ("case_id", "pair_id", "adapter", "arm", "state"): require_nonempty_string(values[field], field=field, location=location) - if values["media_type"] != "image": - raise ValueError(f"{location}: media_type must be image") + if values["media_type"] not in ("image", "audio", "video"): + raise ValueError(f"{location}: media_type must be image, audio, or video") artifact = _object(values["artifact"], field="artifact", location=location) require_sha256(artifact.get("sha256"), field="artifact.sha256", location=location) @@ -411,6 +411,7 @@ def _fidelity_groups(records: Sequence[ResultRecord]) -> list[dict[str, Any]]: counts = {"measured": 0, "incomparable": 0, "not_measured": 0, "inconsistent": 0} identical = 0 psnr_values: list[float] = [] + snr_values: list[float] = [] for key, (_, value) in pairs.items(): if key in inconsistent_pairs: counts["inconsistent"] += 1 @@ -425,6 +426,9 @@ def _fidelity_groups(records: Sequence[ResultRecord]) -> list[dict[str, Any]]: psnr = value.get("psnr_db") if isinstance(psnr, (int, float)) and not isinstance(psnr, bool) and math.isfinite(psnr): psnr_values.append(float(psnr)) + snr = value.get("snr_db") + if isinstance(snr, (int, float)) and not isinstance(snr, bool) and math.isfinite(snr): + snr_values.append(float(snr)) summaries.append( { "adapter": adapter, @@ -434,6 +438,8 @@ def _fidelity_groups(records: Sequence[ResultRecord]) -> list[dict[str, Any]]: "identical": identical, "finite_psnr": len(psnr_values), "p50_psnr_db": nearest_rank(sorted(psnr_values), 0.50) if psnr_values else None, + "finite_snr": len(snr_values), + "p50_snr_db": nearest_rank(sorted(snr_values), 0.50) if snr_values else None, } ) return summaries @@ -737,6 +743,8 @@ def render_markdown(summary: Mapping[str, Any], *, title: str = "Watermark bench ("inconsistent", "Inconsistent"), ("finite_psnr", "Finite PSNR"), ("p50_psnr_db", "p50 PSNR dB"), + ("finite_snr", "Finite SNR"), + ("p50_snr_db", "p50 SNR dB"), ), cast("list[dict[str, Any]]", summary["fidelity"]), ), diff --git a/scripts/watermark_benchmark_video_cohort.py b/scripts/watermark_benchmark_video_cohort.py new file mode 100755 index 0000000..db036c9 --- /dev/null +++ b/scripts/watermark_benchmark_video_cohort.py @@ -0,0 +1,392 @@ +#!/usr/bin/env python3 +"""Build a local, synthetic video-watermark benchmark cohort. + +The builder renders deterministic synthetic clips, embeds the pinned VideoSeal +message through the TorchScript oracle, applies fixed ffmpeg attacks, and +writes every generated artifact once under a new output directory. It emits +the strict JSONL manifest consumed by ``watermark_benchmark.py`` with +``media_type: video`` rows for the ``videoseal`` adapter. + +Generated artifacts and reports belong under ``.local-eval/`` and are never +committed. No provider oracle or API is used. The first build downloads the +pinned TorchScript checkpoint into the user cache when it is incomplete. + + uv run --extra dev python scripts/watermark_benchmark_video_cohort.py \ + --output-dir .local-eval/watermark-benchmark-video-cohort-v1 +""" + +from __future__ import annotations + +import argparse +import hashlib +import importlib.metadata +import logging +import shutil +import sys +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np + +if TYPE_CHECKING: + from collections.abc import Sequence + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) +sys.path.insert(0, str(REPO / "scripts")) + +import videoseal_oracle # noqa: E402 +from audioseal_experiment import ffmpeg_version # noqa: E402 +from watermark_benchmark import SCHEMA_VERSION, load_manifest, sha256_file, write_jsonl # noqa: E402 + +log = logging.getLogger("watermark_benchmark_video_cohort") + +RECIPE_VERSION = "watermark-benchmark-video-cohort-v2" +FORGED_MESSAGE_SEED = 8 +CARRIER_NAMES: tuple[str, ...] = ("moving_gradient", "moving_texture") +HARD_NEGATIVE_CARRIER = "moving_texture_hard_negative" +ATTACKS: tuple[str, ...] = ("h264_crf23", "scale_075", "fps_half") +FRAME_COUNT = 64 +FRAME_RATE = 12.0 +WIDTH = 256 +HEIGHT = 256 +CARRIER_SEED = 20260908 +# Attacks that keep the artifact geometry comparable with the reference clip. +REFERENCE_BACKED_ATTACKS: tuple[str, ...] = ("h264_crf23",) + + +def require_tools() -> str: + ffmpeg = shutil.which("ffmpeg") + if ffmpeg is None: + raise SystemExit("video cohort builder requires ffmpeg on PATH") + return ffmpeg + + +def oracle_dependencies() -> dict[str, str]: + """Return versions of everything that can affect cohort artifacts.""" + + def package(name: str) -> str: + try: + return importlib.metadata.version(name) + except importlib.metadata.PackageNotFoundError: + return "unavailable" + + return { + "numpy": package("numpy"), + "torch": package("torch"), + "videoseal-jit-sha256": videoseal_oracle.JIT_SHA256, + "ffmpeg": ffmpeg_version(require_tools()), + } + + +def _source_revision() -> str: + return f"{RECIPE_VERSION}@sha256:{sha256_file(Path(__file__))}" + + +def _carrier_seed(name: str) -> int: + import zlib + + return CARRIER_SEED + zlib.crc32(name.encode()) % 1000 + + +def synth_carrier(name: str, *, seed_offset: int = 0) -> np.ndarray: + """Render one deterministic clip as float RGB frames in [0, 1].""" + rng = np.random.default_rng(_carrier_seed(name) + seed_offset) + frames = np.empty((FRAME_COUNT, HEIGHT, WIDTH, 3), dtype=np.float32) + axis = np.arange(WIDTH, dtype=np.float32)[None, :] + rows = np.arange(HEIGHT, dtype=np.float32)[:, None] + for index in range(FRAME_COUNT): + if name == "moving_gradient": + luma = 0.35 + 0.25 * np.sin((axis + index * 4) / 23.0) + 0.15 * np.cos((rows - index * 2) / 31.0) + frame = np.stack((luma, np.clip(luma + 0.08, 0, 1), np.clip(luma - 0.06, 0, 1)), axis=2) + elif name == "moving_texture": + noise = rng.normal(0.0, 1.0, (HEIGHT // 4, WIDTH // 4)).astype(np.float32) + texture = np.asarray(np.kron(noise, np.ones((4, 4), dtype=np.float32)) * 0.12 + 0.5, dtype=np.float32) + frame = np.stack((texture, texture, texture), axis=2) + block_x = (index * 5) % (WIDTH - 64) + frame[:, block_x : block_x + 64, 1] = np.clip(frame[:, block_x : block_x + 64, 1] + 0.25, 0, 1) + else: + raise ValueError(f"unknown carrier {name!r}") + frames[index] = np.clip(frame, 0.0, 1.0) + return frames + + +def encode_clip(ffmpeg: str, path: Path, frames: np.ndarray, *, crf: int = 8) -> Path: + """Encode float frames in [0, 1] to an MP4 clip.""" + import subprocess + + height, width = frames.shape[1:3] + payload = (frames * 255.0).astype(np.uint8).tobytes() + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + ffmpeg, + "-y", + "-loglevel", + "error", + "-f", + "rawvideo", + "-pix_fmt", + "rgb24", + "-s:v", + f"{width}x{height}", + "-r", + f"{FRAME_RATE:.12g}", + "-i", + "pipe:0", + "-c:v", + "libx264", + "-preset", + "medium", + "-crf", + str(crf), + "-pix_fmt", + "yuv420p", + str(path), + ], + input=payload, + check=True, + ) + return path + + +ATTACK_ARGS: dict[str, list[str]] = { + "h264_crf23": ["-c:v", "libx264", "-crf", "23", "-pix_fmt", "yuv420p"], + "scale_075": ["-vf", "scale=192:192", "-c:v", "libx264", "-crf", "18", "-pix_fmt", "yuv420p"], + "fps_half": ["-vf", f"fps={FRAME_RATE / 2:.12g}", "-c:v", "libx264", "-crf", "8", "-pix_fmt", "yuv420p"], +} + + +def attack_parameters(attack: str, dependencies: dict[str, str]) -> dict[str, object]: + values: dict[str, object] = dict.fromkeys((), "") + if attack == "h264_crf23": + values = {"codec": "libx264", "crf": 23} + elif attack == "scale_075": + values = {"scale": [192, 192], "codec": "libx264", "crf": 18} + elif attack == "fps_half": + values = {"fps": FRAME_RATE / 2, "codec": "libx264", "crf": 8} + else: + raise ValueError(f"unknown attack {attack!r}") + return {**values, "dependencies": dependencies} + + +def apply_attack(ffmpeg: str, attack: str, marked: Path, artifacts: Path) -> Path: + import subprocess + + target = artifacts / f"{marked.stem}-{attack}.mp4" + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ffmpeg, "-y", "-loglevel", "error", "-i", str(marked), *ATTACK_ARGS[attack], str(target)], + check=True, + ) + return target + + +def benchmark_row( + *, + root: Path, + case_id: str, + pair_id: str, + arm: str, + state: str, + path: Path, + reference: Path | None, + transform_name: str, + transform_revision: str, + parameters: dict[str, object], + seed: int, + expected: str, +) -> dict[str, object]: + return { + "schema_version": SCHEMA_VERSION, + "case_id": case_id, + "pair_id": pair_id, + "media_type": "video", + "adapter": "videoseal", + "arm": arm, + "state": state, + "path": path.relative_to(root).as_posix(), + "sha256": sha256_file(path), + "reference_path": reference.relative_to(root).as_posix() if reference is not None else None, + "reference_sha256": sha256_file(reference) if reference is not None else None, + "source_revision": _source_revision(), + "transform": {"name": transform_name, "revision": transform_revision, "parameters": parameters}, + "seed": seed, + "expected": expected, + } + + +def build_cohort( + output_dir: Path, + *, + carriers: Sequence[str] = CARRIER_NAMES, + attacks: Sequence[str] = ATTACKS, +) -> Path: + """Build and validate a new video cohort directory, returning its manifest.""" + if output_dir.exists(): + raise FileExistsError(f"output directory already exists: {output_dir}") + if not videoseal_oracle.available(): + raise SystemExit("video cohort builder requires the dev extra: uv sync --extra dev (torch)") + unknown = sorted(set(carriers) - set(CARRIER_NAMES)) + if unknown: + raise ValueError(f"unknown carriers: {', '.join(unknown)}") + unknown_attacks = sorted(set(attacks) - set(ATTACKS)) + if unknown_attacks: + raise ValueError(f"unknown attacks: {', '.join(unknown_attacks)}") + + ffmpeg = require_tools() + model = videoseal_oracle.load_model() + message = videoseal_oracle.message_bits() + output_dir.mkdir(parents=True) + artifacts = output_dir / "artifacts" + artifacts.mkdir() + dependencies = oracle_dependencies() + rows: list[dict[str, object]] = [] + + for name in carriers: + clean = synth_carrier(name) + clean_path = encode_clip(ffmpeg, artifacts / f"{name}-clean.mp4", clean) + marked_frames = np.asarray(videoseal_oracle.embed(model, clean, message), dtype=np.float32) + marked_path = encode_clip(ffmpeg, artifacts / f"{name}-marked.mp4", marked_frames) + geometry = {"frames": FRAME_COUNT, "width": WIDTH, "height": HEIGHT, "fps": FRAME_RATE} + + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-clean", + pair_id=name, + arm="matched_negative", + state="clean", + path=clean_path, + reference=None, + transform_name="synthesize-clip", + transform_revision=RECIPE_VERSION, + parameters={"carrier": name, **geometry, "dependencies": dependencies}, + seed=_carrier_seed(name), + expected="not_detected", + ) + ) + message_digest = hashlib.sha256(bytes(message)).hexdigest() + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-marked", + pair_id=name, + arm="positive", + state="marked", + path=marked_path, + reference=clean_path, + transform_name="videoseal-embed", + transform_revision=f"torch@{dependencies['torch']}+jit@{videoseal_oracle.JIT_SHA256}", + parameters={ + "message_sha256": message_digest, + **geometry, + "dependencies": dependencies, + }, + seed=7, + expected="detected", + ) + ) + for attack in attacks: + attacked = apply_attack(ffmpeg, attack, marked_path, artifacts) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-{attack}", + pair_id=name, + arm="positive", + state="attacked", + path=attacked, + reference=clean_path if attack in REFERENCE_BACKED_ATTACKS else None, + transform_name=f"attack-{attack}", + transform_revision=RECIPE_VERSION, + parameters=attack_parameters(attack, dependencies), + seed=7, + expected="unresolved", + ) + ) + + # The removed row reuses the crf 23 re-encode that measurably takes + # both carriers below the decision rule. + removed_path = artifacts / f"{name}-marked-h264_crf23.mp4" + if removed_path.is_file(): + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-removed", + pair_id=name, + arm="positive", + state="removed", + path=removed_path, + reference=clean_path, + transform_name="remove-h264-crf23", + transform_revision=RECIPE_VERSION, + parameters=attack_parameters("h264_crf23", dependencies), + seed=7, + expected="not_detected", + ) + ) + + # The forged row embeds a different message into the same clean clip. + forged_message = videoseal_oracle.message_bits(FORGED_MESSAGE_SEED) + forged_frames = np.asarray(videoseal_oracle.embed(model, clean, forged_message), dtype=np.float32) + forged_path = encode_clip(ffmpeg, artifacts / f"{name}-forged.mp4", forged_frames) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{name}-forged", + pair_id=name, + arm="wrong_key", + state="forged", + path=forged_path, + reference=clean_path, + transform_name="videoseal-embed-foreign-message", + transform_revision=f"torch@{dependencies['torch']}+jit@{videoseal_oracle.JIT_SHA256}", + parameters={ + "message_sha256": hashlib.sha256(bytes(forged_message)).hexdigest(), + "message_seed": FORGED_MESSAGE_SEED, + **geometry, + "dependencies": dependencies, + }, + seed=FORGED_MESSAGE_SEED, + expected="not_detected", + ) + ) + + hard = synth_carrier("moving_texture", seed_offset=991) + hard_path = encode_clip(ffmpeg, artifacts / f"{HARD_NEGATIVE_CARRIER}.mp4", hard) + rows.append( + benchmark_row( + root=output_dir, + case_id=f"{HARD_NEGATIVE_CARRIER}-clean", + pair_id=HARD_NEGATIVE_CARRIER, + arm="hard_negative", + state="clean", + path=hard_path, + reference=None, + transform_name="synthesize-clip", + transform_revision=RECIPE_VERSION, + parameters={"carrier": "moving_texture", "seed_offset": 991, "dependencies": dependencies}, + seed=_carrier_seed(HARD_NEGATIVE_CARRIER) + 991, + expected="not_detected", + ) + ) + + manifest = output_dir / "manifest.jsonl" + write_jsonl(manifest, rows) + load_manifest(manifest) + return manifest + + +def main() -> None: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s") + try: + manifest = build_cohort(args.output_dir.resolve()) + except (FileExistsError, ValueError) as exc: + parser.error(str(exc)) + log.info("Wrote validated video benchmark cohort to %s", manifest) + + +if __name__ == "__main__": + main() diff --git a/scripts/watermark_forgery_study.py b/scripts/watermark_forgery_study.py new file mode 100644 index 0000000..f1339fc --- /dev/null +++ b/scripts/watermark_forgery_study.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +"""Development-only watermark forgery study (four states plus double embedding). + +The benchmark's removed and forged rows assert outcomes; this study measures +the mechanism underneath them, for both pinned oracles: + +- ``clean`` baseline; ``marked`` the oracle's message A; +- ``removed`` the transform that measurably destroys A; +- ``forged on clean`` message B embedded into unmarked media (false + attribution); +- ``forged on marked`` message B embedded OVER an A-marked artifact - the + double-embedding question: does the decoder see B (forgery wins), A (the + original survives), or neither (mutual interference)? +- ``forged on removed`` B embedded after the removal attempt. + +Everything runs locally against the pinned oracles. Artifacts stay outside +the repository under ``.local-eval/``. + + uv run --extra dev python scripts/watermark_forgery_study.py \ + --output-dir .local-eval/watermark-forgery-v1 +""" + +from __future__ import annotations + +import argparse +import json +import logging +import shutil +import sys +from pathlib import Path +from typing import TYPE_CHECKING + +import numpy as np + +if TYPE_CHECKING: + from collections.abc import Sequence + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "src")) +sys.path.insert(0, str(REPO / "scripts")) + +import audioseal_oracle # noqa: E402 +import videoseal_oracle # noqa: E402 +from audioseal_experiment import numpy_attacks # noqa: E402 +from audioseal_experiment import synth_carrier as synth_audio_carrier # noqa: E402 +from videoseal_temporal_study import apply_crf, real_carrier_frames # noqa: E402 +from watermark_benchmark import sha256_file # noqa: E402 +from watermark_benchmark_video_cohort import synth_carrier as synth_video_carrier # noqa: E402 + +log = logging.getLogger("watermark_forgery_study") + +FORGED_SEED = 8 +CLIP_FRAMES = 64 + + +def require_ffmpeg() -> str: + ffmpeg = shutil.which("ffmpeg") + if ffmpeg is None: + raise SystemExit("forgery study requires ffmpeg on PATH") + return ffmpeg + + +def bit_accuracy(decoded: Sequence[int], message: Sequence[int]) -> float: + return sum(int(a == b) for a, b in zip(decoded, message, strict=True)) / len(message) + + +def audio_cells(out_dir: Path) -> list[dict[str, object]]: + import importlib.metadata as md + + from audioseal_experiment import SAMPLE_RATE, message_bits, wav_pcm16_bytes + + generator, detector = audioseal_oracle.load_pinned_models() + message_a = message_bits() + message_b = message_bits(FORGED_SEED) + rows: list[dict[str, object]] = [] + + def measure(label: str, samples: np.ndarray, path: Path | None = None) -> dict[str, object]: + reading = audioseal_oracle.read(detector, samples) + row: dict[str, object] = { + "scheme": "audioseal", + "cell": label, + "presence_frac": round(reading.frac_above_threshold, 6), + "bit_accuracy_vs_a": round(bit_accuracy(reading.decoded_bits, message_a), 6), + "bit_accuracy_vs_b": round(bit_accuracy(reading.decoded_bits, message_b), 6), + } + if path is not None: + row["sha256"] = sha256_file(path) + rows.append(row) + log.info("%s", json.dumps(row, sort_keys=True)) + return row + + carrier_name = "tone_stack" + clean = synth_audio_carrier(carrier_name, 8.0) + workdir = out_dir / "audioseal" + workdir.mkdir() + clean_path = workdir / "clean.wav" + clean_path.write_bytes(wav_pcm16_bytes(clean, SAMPLE_RATE)) + measure("clean", clean, clean_path) + + marked_a = np.asarray(audioseal_oracle.embed(generator, clean, message_a), dtype=np.float32) + marked_a_path = workdir / "marked_a.wav" + marked_a_path.write_bytes(wav_pcm16_bytes(marked_a, SAMPLE_RATE)) + measure("marked_a", marked_a, marked_a_path) + + removed = numpy_attacks(marked_a)["noise_snr10"] + removed_path = workdir / "removed.wav" + removed_path.write_bytes(wav_pcm16_bytes(removed, SAMPLE_RATE)) + measure("removed_noise", removed, removed_path) + + forged_on_clean = np.asarray(audioseal_oracle.embed(generator, clean, message_b), dtype=np.float32) + path = workdir / "forged_on_clean.wav" + path.write_bytes(wav_pcm16_bytes(forged_on_clean, SAMPLE_RATE)) + measure("forged_b_on_clean", forged_on_clean, path) + + forged_on_marked = np.asarray(audioseal_oracle.embed(generator, marked_a, message_b), dtype=np.float32) + path = workdir / "forged_b_on_marked.wav" + path.write_bytes(wav_pcm16_bytes(forged_on_marked, SAMPLE_RATE)) + measure("forged_b_on_marked_a", forged_on_marked, path) + + forged_on_removed = np.asarray(audioseal_oracle.embed(generator, removed, message_b), dtype=np.float32) + path = workdir / "forged_on_removed.wav" + path.write_bytes(wav_pcm16_bytes(forged_on_removed, SAMPLE_RATE)) + measure("forged_b_on_removed_a", forged_on_removed, path) + + rows.append( + { + "scheme": "audioseal", + "cell": "pins", + "audioseal": md.version("audioseal"), + "huggingface_revision": audioseal_oracle.HF_REVISION, + } + ) + return rows + + +def video_cells(out_dir: Path, ffmpeg: str) -> list[dict[str, object]]: + from watermark_benchmark_video_cohort import encode_clip + + model = videoseal_oracle.load_model() + message_a = videoseal_oracle.message_bits() + message_b = videoseal_oracle.message_bits(FORGED_SEED) + rows: list[dict[str, object]] = [] + workdir = out_dir / "videoseal" + workdir.mkdir() + + def measure(label: str, frames: np.ndarray, path: Path) -> dict[str, object]: + reading = videoseal_oracle.read(model, frames, message=message_a) + row: dict[str, object] = { + "scheme": "videoseal", + "cell": label, + "bit_accuracy_vs_a": round(reading.bit_accuracy, 6), + "bit_accuracy_vs_b": round(bit_accuracy(reading.decoded_bits, message_b), 6), + "detected_by_matched_rule": reading.detected, + "sha256": sha256_file(path), + } + rows.append(row) + log.info("%s", json.dumps(row, sort_keys=True)) + return row + + for carrier_name, clean in ( + ("moving_gradient", synth_video_carrier("moving_gradient")[:CLIP_FRAMES]), + *((f"real_{name}", real_carrier_frames(name)[0]) for name in ("sora",)), + ): + prefix = f"{carrier_name}-" + clean_path = encode_clip(ffmpeg, workdir / f"{prefix}clean.mp4", clean) + measure(f"{carrier_name}/clean", clean, clean_path) + + marked_a = np.asarray(videoseal_oracle.embed(model, clean, message_a), dtype=np.float32) + marked_a_path = encode_clip(ffmpeg, workdir / f"{prefix}marked_a.mp4", marked_a) + measure(f"{carrier_name}/marked_a", marked_a, marked_a_path) + + removed_path = apply_crf(ffmpeg, marked_a_path, 23, workdir) + from watermark_benchmark import _decode_video + + removed_frames = _decode_video(removed_path) + if removed_frames is None: + raise SystemExit(f"removed artifact failed to decode: {removed_path}") + measure(f"{carrier_name}/removed_crf23", removed_frames, removed_path) + + forged_on_clean = np.asarray(videoseal_oracle.embed(model, clean, message_b), dtype=np.float32) + path = encode_clip(ffmpeg, workdir / f"{prefix}forged_b_on_clean.mp4", forged_on_clean) + measure(f"{carrier_name}/forged_b_on_clean", forged_on_clean, path) + + forged_on_marked = np.asarray(videoseal_oracle.embed(model, marked_a, message_b), dtype=np.float32) + path = encode_clip(ffmpeg, workdir / f"{prefix}forged_b_on_marked.mp4", forged_on_marked) + measure(f"{carrier_name}/forged_b_on_marked_a", forged_on_marked, path) + + if removed_frames.shape[0] == marked_a.shape[0]: + forged_on_removed = np.asarray(videoseal_oracle.embed(model, removed_frames, message_b), dtype=np.float32) + path = encode_clip(ffmpeg, workdir / f"{prefix}forged_b_on_removed.mp4", forged_on_removed) + measure(f"{carrier_name}/forged_b_on_removed_a", forged_on_removed, path) + + rows.append( + { + "scheme": "videoseal", + "cell": "pins", + "videoseal_jit_sha256": videoseal_oracle.JIT_SHA256, + } + ) + return rows + + +def remover_trace_probe(out_dir: Path) -> dict[str, object]: + """Scoped remover-trace probe: does our noise 'removal' leave a trace? + + Measures the high-to-low frequency band energy ratio of clean versus + noise-removed audio. This detects the trace of OUR gaussian-noise removal, + not a general remover detector; it is first evidence, not a product. + """ + from audioseal_experiment import SAMPLE_RATE, message_bits, numpy_attacks, synth_carrier + + generator, _detector = audioseal_oracle.load_pinned_models() + message_a = message_bits() + stats: dict[str, list[float]] = {"clean": [], "removed": []} + for name in ("tone_stack", "pinkish_noise", "white_noise"): + clean = synth_carrier(name, 8.0) + marked = np.asarray(audioseal_oracle.embed(generator, clean, message_a), dtype=np.float32) + removed = numpy_attacks(marked)["noise_snr10"] + for label, samples in (("clean", clean), ("removed", removed)): + spectrum = np.abs(np.fft.rfft(samples.astype(np.float64))) ** 2 + freqs = np.fft.rfftfreq(samples.shape[0], 1.0 / SAMPLE_RATE) + high = spectrum[freqs > 6000].sum() + low = spectrum[freqs <= 6000].sum() + stats[label].append(float(high / max(low, 1e-12))) + probe = { + "scope": "traces of this study's gaussian-noise removal only", + "hf_lf_ratio_clean": [round(v, 6) for v in stats["clean"]], + "hf_lf_ratio_removed": [round(v, 6) for v in stats["removed"]], + } + (out_dir / "remover_trace_probe.json").write_text(json.dumps(probe, indent=2) + "\n") + return probe + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--output-dir", type=Path, required=True) + args = parser.parse_args() + out_dir: Path = args.output_dir.resolve() + if out_dir.exists(): + raise SystemExit(f"refusing to overwrite existing output directory {out_dir}") + logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") + ffmpeg = require_ffmpeg() + out_dir.mkdir(parents=True) + + rows = audio_cells(out_dir) + rows += video_cells(out_dir, ffmpeg) + probe = remover_trace_probe(out_dir) + rows.append({"scheme": "probe", "cell": "remover_trace", **probe}) + + cases_path = out_dir / "cases.jsonl" + cases_path.write_text("".join(json.dumps(r, default=str) + "\n" for r in rows)) + print(f"wrote {cases_path}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/tests/test_audioseal_experiment.py b/tests/test_audioseal_experiment.py new file mode 100644 index 0000000..84dc8af --- /dev/null +++ b/tests/test_audioseal_experiment.py @@ -0,0 +1,151 @@ +"""Tests for the pure helpers of the AudioSeal experiment script. + +The oracle-dependent stages (embedding, detection, ffmpeg arms) stay +development-only and are exercised by the recorded local runs; these tests +cover the deterministic helpers that must not drift. +""" + +from __future__ import annotations + +import shutil +import struct +import sys +import wave +from pathlib import Path + +import numpy as np +import pytest + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO / "scripts")) + +from audioseal_experiment import ( # noqa: E402 + CARRIERS, + SAMPLE_RATE, + SPEECH_VOICES, + assert_distinct_voices, + message_bits, + moving_background, + partial_noise_attack, + snr_db, + sora_like_mark, + stamped_frame, + synthesize_speech, + tone_stack_carrier, + wav_pcm16_bytes, +) + + +class TestCarriers: + def test_carriers_are_deterministic_per_seed(self) -> None: + n = 4000 + first = tone_stack_carrier(n, np.random.default_rng(11)) + second = tone_stack_carrier(n, np.random.default_rng(11)) + np.testing.assert_array_equal(first, second) + + def test_different_seeds_produce_different_carriers(self) -> None: + n = 4000 + first = tone_stack_carrier(n, np.random.default_rng(11)) + second = tone_stack_carrier(n, np.random.default_rng(12)) + assert not np.array_equal(first, second) + + def test_registered_carriers_return_expected_lengths(self) -> None: + n = 1000 + for name, factory in CARRIERS.items(): + samples = np.asarray(factory(n, np.random.default_rng(3)), dtype=np.float32) + assert samples.shape == (n,), name + assert np.isfinite(samples).all(), name + + +class TestMessage: + def test_message_is_stable_and_binary(self) -> None: + bits = message_bits() + assert bits == message_bits() + assert len(bits) == 16 + assert set(bits) <= {0, 1} + + +class TestWavWriter: + def test_wav_bytes_round_trip_through_wave_module(self, tmp_path: Path) -> None: + samples = np.linspace(-0.9, 0.9, 256, dtype=np.float32) + path = tmp_path / "out.wav" + path.write_bytes(wav_pcm16_bytes(samples, SAMPLE_RATE)) + with wave.open(str(path)) as reader: + assert reader.getnchannels() == 1 + assert reader.getsampwidth() == 2 + assert reader.getframerate() == SAMPLE_RATE + decoded = np.frombuffer(reader.readframes(reader.getnframes()), dtype=" None: + samples = np.zeros(10, dtype=np.float32) + data = wav_pcm16_bytes(samples, SAMPLE_RATE) + assert data[:4] == b"RIFF" + assert struct.unpack(" None: + signal = np.ones(100, dtype=np.float32) + assert snr_db(signal, np.zeros(100, dtype=np.float32)) > 100.0 + + def test_known_ratio(self) -> None: + signal = np.ones(1000, dtype=np.float32) + residual = np.full(1000, 0.1, dtype=np.float32) + assert snr_db(signal, residual) == pytest.approx(20.0, abs=0.01) + + +class TestVideoFrames: + def test_mark_stamps_only_its_region(self) -> None: + clean = moving_background(5) + mark, mark_x, mark_y = sora_like_mark() + stamped = stamped_frame(5, mark, mark_x, mark_y) + height, width = mark.shape + outside = ( + slice(0, mark_y), + slice(0, mark_x), + ) + np.testing.assert_array_equal(stamped[outside], clean[outside]) + assert not np.array_equal( + stamped[mark_y : mark_y + height, mark_x : mark_x + width], + clean[mark_y : mark_y + height, mark_x : mark_x + width], + ) + + def test_mark_geometry_matches_the_test_clip_layout(self) -> None: + mark, mark_x, mark_y = sora_like_mark() + assert mark.shape == (44, 124) + assert mark_x + 124 <= 840 + assert mark_y + 44 <= 480 + + +class TestSpeechCarriers: + def test_assert_distinct_voices_rejects_aliased_bytes(self) -> None: + with pytest.raises(SystemExit, match="byte-identical"): + assert_distinct_voices({"Samantha": b"abc", "Alex": b"abc"}) + + def test_assert_distinct_voices_accepts_distinct_bytes(self) -> None: + assert_distinct_voices({"Samantha": b"abc", "Daniel": b"abd"}) + + def test_partial_noise_attack_touches_only_its_window(self) -> None: + samples = np.ones(4 * SAMPLE_RATE, dtype=np.float32) + attacked = partial_noise_attack(samples, 1.0, 2.0) + + np.testing.assert_array_equal(attacked[:SAMPLE_RATE], samples[:SAMPLE_RATE]) + np.testing.assert_array_equal(attacked[2 * SAMPLE_RATE :], samples[2 * SAMPLE_RATE :]) + assert not np.array_equal(attacked[SAMPLE_RATE : 2 * SAMPLE_RATE], samples[SAMPLE_RATE : 2 * SAMPLE_RATE]) + + @pytest.mark.skipif(shutil.which("say") is None, reason="requires the macOS say tool") + def test_system_speech_is_deterministic_per_voice(self, tmp_path: Path) -> None: + first = synthesize_speech("Samantha", tmp_path) + (tmp_path / "again").mkdir() + synthesize_speech("Samantha", tmp_path / "again") + + assert first.read_bytes() == (tmp_path / "again" / "speech-Samantha.aiff").read_bytes() + + @pytest.mark.skipif(shutil.which("say") is None, reason="requires the macOS say tool") + def test_configured_voices_render_distinct_bytes(self, tmp_path: Path) -> None: + rendered = {voice: synthesize_speech(voice, tmp_path).read_bytes() for voice in SPEECH_VOICES} + + assert_distinct_voices(rendered) diff --git a/tests/test_audioseal_oracle.py b/tests/test_audioseal_oracle.py new file mode 100644 index 0000000..ef073bb --- /dev/null +++ b/tests/test_audioseal_oracle.py @@ -0,0 +1,41 @@ +"""Tests for the shared AudioSeal oracle's pure helpers.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +SCRIPTS = Path(__file__).resolve().parents[1] / "scripts" +if str(SCRIPTS) not in sys.path: + sys.path.insert(0, str(SCRIPTS)) + +from audioseal_oracle import SAMPLE_RATE, window_bounds # noqa: E402 + + +class TestWindowBounds: + def test_exact_multiple_covers_every_sample_once(self) -> None: + bounds = window_bounds(32_000, 1.0, SAMPLE_RATE) + + assert bounds == [(0, 16_000), (16_000, 32_000)] + + def test_trailing_remainder_keeps_its_own_window(self) -> None: + bounds = window_bounds(32_500, 1.0, SAMPLE_RATE) + + assert bounds[-1] == (32_000, 32_500) + assert len(bounds) == 3 + + def test_short_signal_is_one_partial_window(self) -> None: + assert window_bounds(100, 1.0, SAMPLE_RATE) == [(0, 100)] + + def test_empty_signal_has_no_windows(self) -> None: + assert window_bounds(0, 1.0, SAMPLE_RATE) == [] + + @pytest.mark.parametrize( + ("total", "window_s", "rate"), + [(-1, 1.0, 16_000), (100, 0.0, 16_000), (100, -1.0, 16_000), (100, 1.0, 0), (100, 0.00001, 16_000)], + ) + def test_rejects_degenerate_arguments(self, total: int, window_s: float, rate: int) -> None: + with pytest.raises(ValueError, match=r"total_samples|window_s|sample_rate"): + window_bounds(total, window_s, rate) diff --git a/tests/test_c2pa_registry_sync.py b/tests/test_c2pa_registry_sync.py index 7428c1e..1b586e6 100644 --- a/tests/test_c2pa_registry_sync.py +++ b/tests/test_c2pa_registry_sync.py @@ -78,3 +78,37 @@ def test_official_source_is_fetched_through_its_resolved_revision(monkeypatch: p assert payload == b"[]" assert resolved == revision assert requested == [sync.PINNED_SOURCE_URL.format(revision=revision)] + + +def test_packaged_snapshot_preserves_published_resolution_apis(): + """Guard the resolution-API subset the runtime deliberately never calls. + + Eight algorithms publish ``softBindingResolutionApis`` today. The runtime + resolves nothing over the network by design (see + docs/c2pa-resolution-research.md for the contracts and the product + decision); this guard keeps the packaged snapshot honest about what the + ecosystem publishes, so an upstream change surfaces here consciously + instead of passing silently through a registry regeneration. + """ + documented = { + "ai.trufo.pawprint.watermark": ("https://c2pa.trufo.ai/v1",), + "ai.trufo.pawprint.fingerprint": ("https://c2pa.trufo.ai/v1",), + "com.aiwatermark.videoseal.1": ("https://aiwatermark.com/api/v1",), + "com.aiwatermark.pixelseal.1": ("https://aiwatermark.com/api/v1",), + "com.aiwatermark.audioseal.1": ("https://aiwatermark.com/api/v1",), + "me.deepmark.audio.vigil.128": ("https://resolution-api.deepmark.me",), + "io.blockfact.audio.watermark.32": ("https://api.blockfact.io/api/soft-binding/v1",), + "com.joinmonolith.sha256": ("https://api.joinmonolith.com/api/c2pa",), + } + packaged = {entry.algorithm: entry.resolution_apis for entry in C2PA_SOFT_BINDING_REGISTRY} + + for algorithm, apis in documented.items(): + assert packaged.get(algorithm) == apis, ( + f"{algorithm} resolution APIs changed: {packaged.get(algorithm)} != {apis}; " + "update docs/c2pa-resolution-research.md with the new contract" + ) + with_apis = {algorithm for algorithm, apis in packaged.items() if apis} + assert with_apis == set(documented), ( + f"upstream softbinding-algorithm-list resolution-API set changed: {sorted(with_apis)}; " + "revisit docs/c2pa-resolution-research.md before regenerating" + ) diff --git a/tests/test_videoseal_oracle.py b/tests/test_videoseal_oracle.py new file mode 100644 index 0000000..d6197f3 --- /dev/null +++ b/tests/test_videoseal_oracle.py @@ -0,0 +1,97 @@ +"""Tests for the pinned VideoSeal TorchScript oracle's pure helpers.""" + +from __future__ import annotations + +import sys +from pathlib import Path + +import pytest + +SCRIPTS = Path(__file__).resolve().parents[1] / "scripts" +if str(SCRIPTS) not in sys.path: + sys.path.insert(0, str(SCRIPTS)) + +import videoseal_oracle # noqa: E402 + + +class TestMessage: + def test_message_is_deterministic_and_256_bits(self) -> None: + first = videoseal_oracle.message_bits() + second = videoseal_oracle.message_bits() + + assert first == second + assert len(first) == 256 + assert set(first) <= {0, 1} + + def test_seed_changes_the_message(self) -> None: + assert videoseal_oracle.message_bits(seed=8) != videoseal_oracle.message_bits(seed=7) + + +class TestLabel: + def test_label_is_hex_of_the_decoded_bits(self) -> None: + bits = tuple([1] * 8 + [0] * 8) + reading = videoseal_oracle.VideoSealReading( + bit_accuracy=1.0, + decoded_bits=bits, + per_frame_bit_accuracy=(1.0,), + ) + + assert reading.label == "ff00" + + def test_detection_rule_is_the_documented_threshold(self) -> None: + below = videoseal_oracle.VideoSealReading(0.89, (0,), ()) + exact = videoseal_oracle.VideoSealReading(0.9, (0,), ()) + + assert below.detected is False + assert exact.detected is True + + +class TestPins: + def test_checkpoint_pin_is_a_sha256_digest(self) -> None: + digest = videoseal_oracle.JIT_SHA256 + + assert len(digest) == 64 + assert all(char in "0123456789abcdef" for char in digest) + + def test_source_commit_is_pinned(self) -> None: + assert len(videoseal_oracle.SOURCE_COMMIT) == 40 + + +class TestCacheOverride: + def test_cache_dir_env_override_is_honored(self, monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: + monkeypatch.setenv("VIDEOSEAL_CACHE_DIR", str(tmp_path / "cache")) + + import importlib + + importlib.reload(videoseal_oracle) + + try: + assert tmp_path / "cache" == videoseal_oracle.CACHE_DIR + finally: + importlib.reload(videoseal_oracle) + + +class TestAggregationFormulas: + def test_aggregations_match_the_upstream_formulas(self) -> None: + torch = pytest.importorskip("torch") + # Two frames, four bits; hand-computed from the upstream extract_message + # formulas so a silent mutation of the port fails here. + bit_preds = torch.tensor([[2.0, -1.0, 0.5, -3.0], [0.1, 0.2, -0.4, 0.3]]) + + avg = videoseal_oracle._aggregate_bit_preds(bit_preds, "avg") + squared = videoseal_oracle._aggregate_bit_preds(bit_preds, "squared_avg") + l1 = videoseal_oracle._aggregate_bit_preds(bit_preds, "l1norm_avg") + l2 = videoseal_oracle._aggregate_bit_preds(bit_preds, "l2norm_avg") + + torch.testing.assert_close(avg, bit_preds.mean(dim=0)) + torch.testing.assert_close(squared, (bit_preds * bit_preds.abs()).mean(dim=0)) + w1 = torch.norm(bit_preds, p=1, dim=1).unsqueeze(1) + torch.testing.assert_close(l1, (bit_preds * w1).mean(dim=0)) + w2 = torch.norm(bit_preds, p=2, dim=1).unsqueeze(1) + torch.testing.assert_close(l2, (bit_preds * w2).mean(dim=0)) + + def test_unknown_aggregation_is_rejected(self) -> None: + torch = pytest.importorskip("torch") + + with pytest.raises(ValueError, match="unknown aggregation"): + videoseal_oracle._aggregate_bit_preds(torch.zeros(2, 4), "median") diff --git a/tests/test_watermark_benchmark.py b/tests/test_watermark_benchmark.py index 9fbd5c2..0f3d7c5 100644 --- a/tests/test_watermark_benchmark.py +++ b/tests/test_watermark_benchmark.py @@ -5,6 +5,7 @@ from __future__ import annotations import hashlib import importlib.util import json +import shutil import sys from pathlib import Path from typing import Any @@ -392,9 +393,11 @@ def test_jsonl_writer_refuses_overwrite_and_emits_strict_json(tmp_path: Path) -> def test_builtin_adapters_name_their_exact_source_files() -> None: adapters = MODULE.default_adapters() - assert set(adapters) == {"dwt-dct", "trustmark"} + assert set(adapters) == {"dwt-dct", "trustmark", "audioseal", "videoseal"} assert adapters["dwt-dct"].source_file.name == "invisible_watermark.py" assert adapters["trustmark"].source_file.name == "trustmark_detector.py" + assert adapters["audioseal"].source_file.name == "audioseal_oracle.py" + assert adapters["videoseal"].source_file.name == "videoseal_oracle.py" def test_run_benchmark_caches_only_recurring_images(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: @@ -446,3 +449,314 @@ def test_run_benchmark_writes_official_trustmark_result(tmp_path: Path) -> None: assert record["detection"]["status"] == "detected" assert record["detection"]["label"] == "Adobe TrustMark (variant P, schema 1)" assert record["case_id"] == "case-1" + + +# --- Audio rows (schema v1, media_type "audio", adapter "audioseal") -------- + + +def _wav(path: Path, samples: np.ndarray) -> Path: + import sys + + scripts = Path(__file__).resolve().parents[1] / "scripts" + if str(scripts) not in sys.path: + sys.path.insert(0, str(scripts)) + from audioseal_experiment import wav_pcm16_bytes + + path.write_bytes(wav_pcm16_bytes(samples, 16_000)) + return path + + +def _audio_row(artifact: Path, *, reference: Path | None = None, adapter: str = "audioseal") -> dict[str, Any]: + return { + "schema_version": 1, + "case_id": "audio-case-1", + "pair_id": "audio-pair-1", + "media_type": "audio", + "adapter": adapter, + "arm": "positive", + "state": "marked", + "path": artifact.name, + "sha256": _sha256(artifact), + "reference_path": reference.name if reference is not None else None, + "reference_sha256": _sha256(reference) if reference is not None else None, + "source_revision": "fixture-set@abc123", + "transform": {"name": "audioseal-embed", "revision": "builtin-v1", "parameters": {}}, + "seed": 7, + "expected": "detected", + } + + +def test_manifest_accepts_audio_rows(tmp_path: Path) -> None: + artifact = _wav(tmp_path / "marked.wav", np.zeros(1600, dtype=np.float32)) + manifest = _write_manifest(tmp_path / "manifest.jsonl", [_audio_row(artifact)]) + + row = MODULE.load_manifest(manifest)[0] + + assert row.media_type == "audio" + assert row.adapter == "audioseal" + + +def test_manifest_rejects_unknown_media_type(tmp_path: Path) -> None: + artifact = _image(tmp_path / "marked.png", 100) + row = _row(artifact) + row["media_type"] = "document" + manifest = _write_manifest(tmp_path / "manifest.jsonl", [row]) + + with pytest.raises(ValueError, match="media_type must be one of image, audio, video"): + MODULE.load_manifest(manifest) + + +def test_manifest_rejects_adapter_measuring_another_media_type(tmp_path: Path) -> None: + artifact = _wav(tmp_path / "marked.wav", np.zeros(1600, dtype=np.float32)) + row = _audio_row(artifact) + row["adapter"] = "dwt-dct" + manifest = _write_manifest(tmp_path / "manifest.jsonl", [row]) + + with pytest.raises(ValueError, match="'dwt-dct' measures image artifacts, not audio"): + MODULE.load_manifest(manifest) + + +def test_manifest_still_accepts_custom_adapter_names_on_audio_rows(tmp_path: Path) -> None: + artifact = _wav(tmp_path / "marked.wav", np.zeros(1600, dtype=np.float32)) + row = _audio_row(artifact, adapter="custom-audio-detector") + manifest = _write_manifest(tmp_path / "manifest.jsonl", [row]) + + assert MODULE.load_manifest(manifest)[0].adapter == "custom-audio-detector" + + +def test_audio_fidelity_reports_unbounded_snr_for_identical_samples( + tmp_path: Path, monkeypatch: pytest.MonkeyPatch +) -> None: + samples = np.linspace(-0.5, 0.5, 4096, dtype=np.float32) + artifact = _wav(tmp_path / "marked.wav", samples) + reference = _wav(tmp_path / "clean.wav", samples.copy()) + row = _audio_row(artifact, reference=reference, adapter="fake-audio") + manifest = _write_manifest(tmp_path / "manifest.jsonl", [row]) + case = MODULE.load_manifest(manifest)[0] + decode_calls: list[Path] = [] + + def decode(path: Path) -> Any: + decode_calls.append(path) + return samples + + fidelity = MODULE._audio_fidelity(case, samples, decode) + + assert fidelity["status"] == "measured" + assert fidelity["identical"] is True + assert fidelity["snr_db"] is None + assert fidelity["snr_status"] == "unbounded_identical" + assert decode_calls == [reference.resolve()] + + +def test_audio_fidelity_marks_length_mismatch_incomparable(tmp_path: Path) -> None: + short = np.zeros(1000, dtype=np.float32) + long = np.zeros(2000, dtype=np.float32) + artifact = _wav(tmp_path / "marked.wav", short) + reference = _wav(tmp_path / "clean.wav", long) + row = _audio_row(artifact, reference=reference, adapter="fake-audio") + case = MODULE.load_manifest(_write_manifest(tmp_path / "manifest.jsonl", [row]))[0] + + fidelity = MODULE._audio_fidelity(case, short, lambda _path: long) + + assert fidelity["status"] == "incomparable" + assert fidelity["reason"] == "length_mismatch" + assert fidelity["artifact_samples"] == 1000 + assert fidelity["reference_samples"] == 2000 + + +def test_run_benchmark_evaluates_audio_rows_end_to_end(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("audio rows decode through the system ffmpeg") + samples = np.linspace(-0.5, 0.5, 16000, dtype=np.float32) + artifact = _wav(tmp_path / "marked.wav", samples) + reference = _wav(tmp_path / "clean.wav", samples * 0.99) + manifest = _write_manifest( + tmp_path / "manifest.jsonl", + [_audio_row(artifact, reference=reference)], + ) + adapter = _FakeAdapter(status="detected", label="1111111000011001") + monkeypatch.setattr(MODULE, "default_adapters", lambda: {"audioseal": adapter}) + output = tmp_path / "result.jsonl" + + counts = MODULE.run_benchmark(manifest, output) + + record = json.loads(output.read_text(encoding="utf-8")) + assert counts == {"detected": 1} + assert record["media_type"] == "audio" + assert record["detection"]["label"] == "1111111000011001" + assert record["fidelity"]["status"] == "measured" + assert record["fidelity"]["snr_status"] == "measured" + assert record["fidelity"]["snr_db"] > 30.0 + assert adapter.calls == [artifact.resolve()] + + +# --- Video rows (schema v1, media_type "video", adapter "videoseal") -------- + + +def _video(path: Path, frames: int = 4, width: int = 64, height: int = 48) -> Path: + import subprocess + + payload = b"".join(bytes([frame_index * 12 % 256]) * (height * width * 3) for frame_index in range(frames)) + subprocess.run( # noqa: S603 - resolved ffmpeg with fixed arguments + [ + shutil.which("ffmpeg"), + "-y", + "-loglevel", + "error", + "-f", + "rawvideo", + "-pix_fmt", + "rgb24", + "-s:v", + f"{width}x{height}", + "-r", + "12", + "-i", + "pipe:0", + "-c:v", + "libx264", + "-preset", + "ultrafast", + "-crf", + "8", + "-pix_fmt", + "yuv420p", + str(path), + ], + input=payload, + check=True, + ) + return path + + +def _video_row(artifact: Path, *, reference: Path | None = None, adapter: str = "videoseal") -> dict[str, Any]: + return { + "schema_version": 1, + "case_id": "video-case-1", + "pair_id": "video-pair-1", + "media_type": "video", + "adapter": adapter, + "arm": "positive", + "state": "marked", + "path": artifact.name, + "sha256": _sha256(artifact), + "reference_path": reference.name if reference is not None else None, + "reference_sha256": _sha256(reference) if reference is not None else None, + "source_revision": "fixture-set@abc123", + "transform": {"name": "videoseal-embed", "revision": "builtin-v1", "parameters": {}}, + "seed": 7, + "expected": "detected", + } + + +def test_manifest_accepts_video_rows(tmp_path: Path) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + artifact = _video(tmp_path / "marked.mp4") + manifest = _write_manifest(tmp_path / "manifest.jsonl", [_video_row(artifact)]) + + row = MODULE.load_manifest(manifest)[0] + + assert row.media_type == "video" + assert row.adapter == "videoseal" + + +def test_manifest_rejects_image_adapter_on_video_rows(tmp_path: Path) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + artifact = _video(tmp_path / "marked.mp4") + row = _video_row(artifact) + row["adapter"] = "trustmark" + manifest = _write_manifest(tmp_path / "manifest.jsonl", [row]) + + with pytest.raises(ValueError, match="'trustmark' measures image artifacts, not video"): + MODULE.load_manifest(manifest) + + +def test_video_fidelity_marks_shape_mismatch_incomparable(tmp_path: Path) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + artifact = _video(tmp_path / "marked.mp4", frames=4) + reference = _video(tmp_path / "clean.mp4", frames=3) + row = _video_row(artifact, reference=reference, adapter="fake-video") + case = MODULE.load_manifest(_write_manifest(tmp_path / "manifest.jsonl", [row]))[0] + + fidelity = MODULE._video_fidelity(case, np.zeros((4, 48, 64, 3)), lambda _path: np.zeros((3, 48, 64, 3))) + + assert fidelity["status"] == "incomparable" + assert fidelity["reason"] == "shape_mismatch" + + +def test_video_fidelity_reports_unbounded_psnr_for_identical_frames(tmp_path: Path) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + frames = np.full((4, 48, 64, 3), 0.5, dtype=np.float32) + artifact = _video(tmp_path / "marked.mp4") + reference = _video(tmp_path / "clean.mp4") + row = _video_row(artifact, reference=reference, adapter="fake-video") + case = MODULE.load_manifest(_write_manifest(tmp_path / "manifest.jsonl", [row]))[0] + + fidelity = MODULE._video_fidelity(case, frames, lambda _path: frames) + + assert fidelity["status"] == "measured" + assert fidelity["identical"] is True + assert fidelity["mean_psnr_db"] is None + assert fidelity["psnr_status"] == "unbounded_identical" + + +def test_run_benchmark_evaluates_video_rows_end_to_end(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + artifact = _video(tmp_path / "marked.mp4") + manifest = _write_manifest(tmp_path / "manifest.jsonl", [_video_row(artifact)]) + adapter = _FakeAdapter(status="detected", label="a1b2") + monkeypatch.setattr(MODULE, "default_adapters", lambda: {"videoseal": adapter}) + output = tmp_path / "result.jsonl" + + counts = MODULE.run_benchmark(manifest, output) + + record = json.loads(output.read_text(encoding="utf-8")) + assert counts == {"detected": 1} + assert record["media_type"] == "video" + assert record["detection"]["label"] == "a1b2" + assert record["fidelity"]["status"] == "not_measured" + + +def test_detection_record_carries_temporal_evidence_when_present() -> None: + outcome = MODULE.DetectorOutcome(status="detected", label="ff00", temporal=(0.8, 0.9, 1.0)) + + record = MODULE._detection_record(outcome, "detected", 1.5) + + temporal = record["temporal"] + assert temporal["frames"] == 3 + assert temporal["min_bit_accuracy"] == 0.8 + assert temporal["mean_bit_accuracy"] == pytest.approx(0.9) + assert temporal["max_bit_accuracy"] == 1.0 + assert temporal["per_frame_bit_accuracy"] == [0.8, 0.9, 1.0] + + +def test_detection_record_omits_temporal_for_non_temporal_adapters() -> None: + outcome = MODULE.DetectorOutcome(status="not_detected", label=None) + + assert "temporal" not in MODULE._detection_record(outcome, "not_detected", None) + + +def test_run_benchmark_records_videoseal_temporal_evidence(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("video rows decode through the system ffmpeg") + artifact = _video(tmp_path / "marked.mp4") + manifest = _write_manifest(tmp_path / "manifest.jsonl", [_video_row(artifact)]) + + class TemporalAdapter(_FakeAdapter): + def detect(self, path: Path, frames: Any) -> Any: + self.calls.append(path) + return MODULE.DetectorOutcome(status=self.status, label=self.label, temporal=(0.5, 0.75, 1.0)) + + adapter = TemporalAdapter(status="detected", label="ff00") + monkeypatch.setattr(MODULE, "default_adapters", lambda: {"videoseal": adapter}) + output = tmp_path / "result.jsonl" + + MODULE.run_benchmark(manifest, output) + + record = json.loads(output.read_text(encoding="utf-8")) + assert record["detection"]["temporal"]["frames"] == 3 diff --git a/tests/test_watermark_benchmark_audio_cohort.py b/tests/test_watermark_benchmark_audio_cohort.py new file mode 100644 index 0000000..bc64be7 --- /dev/null +++ b/tests/test_watermark_benchmark_audio_cohort.py @@ -0,0 +1,126 @@ +"""Contracts for the audio benchmark cohort builder.""" + +from __future__ import annotations + +import importlib.util +import json +import shutil +import sys +from pathlib import Path + +import numpy as np +import pytest + +SCRIPTS = Path(__file__).resolve().parents[1] / "scripts" + + +def _load(name: str, filename: str) -> object: + spec = importlib.util.spec_from_file_location(name, SCRIPTS / filename) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +COHORT = _load("watermark_benchmark_audio_cohort", "watermark_benchmark_audio_cohort.py") +EXPERIMENT = _load("audioseal_experiment_cohort_test", "audioseal_experiment.py") +BENCHMARK = sys.modules.get("watermark_benchmark") or _load("watermark_benchmark", "watermark_benchmark.py") +sys.path.insert(0, str(SCRIPTS)) +import audioseal_oracle # noqa: E402 + +pytestmark = pytest.mark.skipif( + shutil.which("ffmpeg") is None or not audioseal_oracle.available(), + reason="audio cohort builder requires ffmpeg and the dev extra (audioseal, torch)", +) + + +def test_builder_refuses_existing_output_directory(tmp_path: Path) -> None: + (tmp_path / "cohort").mkdir() + + with pytest.raises(FileExistsError): + COHORT.build_cohort(tmp_path / "cohort", duration_s=0.2) + + +def test_builder_rejects_unknown_carriers_and_attacks(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="unknown carriers"): + COHORT.build_cohort(tmp_path / "a", carriers=("silence",), duration_s=0.2) + with pytest.raises(ValueError, match="unknown attacks"): + COHORT.build_cohort(tmp_path / "b", attacks=("reverb",), duration_s=0.2) + + +def test_built_cohort_manifest_is_strict_and_complete(tmp_path: Path) -> None: + manifest = COHORT.build_cohort(tmp_path / "cohort", duration_s=0.5) + + cases = BENCHMARK.load_manifest(manifest) + assert len(cases) == 20 + by_case = {case.case_id: case for case in cases} + assert by_case["tone_stack-removed"].state == "removed" + assert by_case["tone_stack-removed"].expected == "not_detected" + assert by_case["tone_stack-forged"].state == "forged" + assert by_case["tone_stack-forged"].arm == "wrong_key" + assert by_case["tone_stack-forged"].expected == "detected" + assert by_case["tone_stack-clean"].arm == "matched_negative" + assert by_case["tone_stack-clean"].expected == "not_detected" + assert by_case["tone_stack-marked"].arm == "positive" + assert by_case["tone_stack-mp3_128k"].state == "attacked" + assert by_case["tone_stack-mp3_128k"].expected == "unresolved" + assert by_case["white_noise_hard_negative-clean"].arm == "hard_negative" + assert all(case.media_type == "audio" for case in cases) + assert all(case.adapter == "audioseal" for case in cases) + assert all(case.path.is_file() for case in cases) + marked = by_case["tone_stack-marked"] + assert marked.reference_path == by_case["tone_stack-clean"].path + + rows = [json.loads(line) for line in manifest.read_text(encoding="utf-8").splitlines()] + embed_parameters = next(row for row in rows if row["case_id"] == "tone_stack-marked") + assert embed_parameters["transform"]["parameters"]["message"] == EXPERIMENT.message_bits() + + +def test_carrier_artifacts_are_deterministic_across_rebuilds(tmp_path: Path) -> None: + manifest = COHORT.build_cohort(tmp_path / "cohort", duration_s=0.5) + + cases = {case.case_id: case for case in BENCHMARK.load_manifest(manifest)} + first = (tmp_path / "cohort" / "artifacts" / "tone_stack-clean.wav").read_bytes() + second = EXPERIMENT.wav_pcm16_bytes(EXPERIMENT.synth_carrier("tone_stack", 0.5), 16_000) + assert first == second + rebuilt = COHORT.build_cohort(tmp_path / "cohort-2", duration_s=0.5) + cases_2 = {case.case_id: case for case in BENCHMARK.load_manifest(rebuilt)} + for case_id, case in cases.items(): + if case.state == "clean": + assert case.sha256 == cases_2[case_id].sha256, case_id + + +def test_hard_negative_is_distinct_from_every_marked_carrier(tmp_path: Path) -> None: + hard = COHORT.synth_carrier_hard_negative(0.5) + carrier = EXPERIMENT.synth_carrier("white_noise", 0.5) + + assert hard.shape == carrier.shape + assert not np.array_equal(hard, carrier) + + +@pytest.mark.skipif(shutil.which("say") is None, reason="speech carriers require the macOS say tool") +def test_speech_extension_adds_provenance_backed_rows(tmp_path: Path) -> None: + manifest = COHORT.build_cohort(tmp_path / "cohort", duration_s=0.5, include_speech=True) + + cases = BENCHMARK.load_manifest(manifest) + assert len(cases) == 35 + speech = {case.case_id: case for case in cases if case.pair_id.startswith("speech_")} + assert len(speech) == 15 + assert {case.pair_id for case in speech.values()} == { + "speech_Samantha", + "speech_Daniel", + "speech_Milena", + } + marked = next(case for case in speech.values() if case.state == "marked") + assert marked.reference_path is not None + digests = {case.sha256 for case in speech.values() if case.state == "clean"} + assert len(digests) == 3, "aliased system voices would collapse clean-carrier digests" + + rows = {json.loads(line)["case_id"]: json.loads(line) for line in manifest.read_text(encoding="utf-8").splitlines()} + tts = rows["speech_Samantha-clean"]["transform"]["parameters"]["tts"] + assert tts["tts"] == "say" + assert tts["macos"] + assert tts["voice"] == "Samantha" + assert tts["text_sha256"] diff --git a/tests/test_watermark_benchmark_report.py b/tests/test_watermark_benchmark_report.py index 2a85e8d..e383b80 100644 --- a/tests/test_watermark_benchmark_report.py +++ b/tests/test_watermark_benchmark_report.py @@ -2,8 +2,10 @@ from __future__ import annotations +import hashlib import importlib.util import json +import shutil import sys from pathlib import Path from typing import Any @@ -325,3 +327,73 @@ def test_report_refuses_overwrite_and_preserves_interpretation_caveat(tmp_path: assert "not evidence that a watermark is absent" in report with pytest.raises(FileExistsError, match="refusing to overwrite"): MODULE.write_report(output, report) + + +def test_report_accepts_audio_rows_and_reports_snr(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: + if shutil.which("ffmpeg") is None: + pytest.skip("audio rows decode through the system ffmpeg") + import sys + + scripts = Path(__file__).resolve().parents[1] / "scripts" + if str(scripts) not in sys.path: + sys.path.insert(0, str(scripts)) + from audioseal_experiment import wav_pcm16_bytes + + samples = np.linspace(-0.5, 0.5, 16000, dtype=np.float32) + artifact = tmp_path / "marked.wav" + reference = tmp_path / "clean.wav" + artifact.write_bytes(wav_pcm16_bytes(samples, 16_000)) + reference.write_bytes(wav_pcm16_bytes(samples * 0.9, 16_000)) + + def sha(path: Path) -> str: + return hashlib.sha256(path.read_bytes()).hexdigest() + + manifest = tmp_path / "manifest.jsonl" + manifest.write_text( + json.dumps( + { + "schema_version": 1, + "case_id": "audio-1", + "pair_id": "audio-1", + "media_type": "audio", + "adapter": "audioseal", + "arm": "positive", + "state": "marked", + "path": artifact.name, + "sha256": sha(artifact), + "reference_path": reference.name, + "reference_sha256": sha(reference), + "source_revision": "fixture@1", + "transform": {"name": "embed", "revision": "v1", "parameters": {}}, + "seed": 1, + "expected": "detected", + } + ) + + "\n", + encoding="utf-8", + ) + case = BENCHMARK.load_manifest(manifest)[0] + monkeypatch.setattr( + BENCHMARK, + "default_adapters", + lambda: {"audioseal": _FakeAdapter("detected")}, + ) + record = BENCHMARK.evaluate_case( + case, + adapters=BENCHMARK.default_adapters(), + repository={"commit": "abc", "dirty": False}, + ) + + result_path = tmp_path / "run.jsonl" + BENCHMARK.write_jsonl(result_path, [record]) + + summary = MODULE.summarize(MODULE.load_results([result_path])) + fidelity = next(row for row in summary["fidelity"] if row["adapter"] == "audioseal") + + assert fidelity["measured"] == 1 + assert fidelity["finite_snr"] == 1 + assert fidelity["p50_snr_db"] is not None + assert fidelity["finite_psnr"] == 0 + + markdown = MODULE.render_markdown(summary) + assert "p50 SNR dB" in markdown diff --git a/tests/test_watermark_benchmark_video_cohort.py b/tests/test_watermark_benchmark_video_cohort.py new file mode 100644 index 0000000..afeb6d5 --- /dev/null +++ b/tests/test_watermark_benchmark_video_cohort.py @@ -0,0 +1,98 @@ +"""Contracts for the video benchmark cohort builder.""" + +from __future__ import annotations + +import importlib.util +import json +import shutil +import sys +from pathlib import Path + +import numpy as np +import pytest + +SCRIPTS = Path(__file__).resolve().parents[1] / "scripts" + + +def _load(name: str, filename: str) -> object: + spec = importlib.util.spec_from_file_location(name, SCRIPTS / filename) + assert spec is not None + assert spec.loader is not None + module = importlib.util.module_from_spec(spec) + sys.modules[name] = module + spec.loader.exec_module(module) + return module + + +COHORT = _load("watermark_benchmark_video_cohort", "watermark_benchmark_video_cohort.py") +BENCHMARK = sys.modules.get("watermark_benchmark") or _load("watermark_benchmark", "watermark_benchmark.py") +sys.path.insert(0, str(SCRIPTS)) +import videoseal_oracle # noqa: E402 + +pytestmark = pytest.mark.skipif( + shutil.which("ffmpeg") is None or not videoseal_oracle.available(), + reason="video cohort builder requires ffmpeg and the dev extra (torch)", +) + + +def test_builder_refuses_existing_output_directory(tmp_path: Path) -> None: + (tmp_path / "cohort").mkdir() + + with pytest.raises(FileExistsError): + COHORT.build_cohort(tmp_path / "cohort") + + +def test_builder_rejects_unknown_carriers_and_attacks(tmp_path: Path) -> None: + with pytest.raises(ValueError, match="unknown carriers"): + COHORT.build_cohort(tmp_path / "a", carriers=("black",)) + with pytest.raises(ValueError, match="unknown attacks"): + COHORT.build_cohort(tmp_path / "b", attacks=("rotate",)) + + +def test_carriers_are_deterministic_and_distinct() -> None: + first = COHORT.synth_carrier("moving_gradient") + second = COHORT.synth_carrier("moving_gradient") + texture = COHORT.synth_carrier("moving_texture") + hard = COHORT.synth_carrier("moving_texture", seed_offset=991) + + assert first.shape == (COHORT.FRAME_COUNT, COHORT.HEIGHT, COHORT.WIDTH, 3) + np.testing.assert_array_equal(first, second) + assert not np.array_equal(first, texture) + assert not np.array_equal(texture, hard) + + +def test_built_cohort_manifest_is_strict_and_complete(tmp_path: Path) -> None: + manifest = COHORT.build_cohort(tmp_path / "cohort") + + cases = BENCHMARK.load_manifest(manifest) + assert len(cases) == 15 + by_case = {case.case_id: case for case in cases} + assert by_case["moving_gradient-removed"].state == "removed" + assert by_case["moving_gradient-removed"].expected == "not_detected" + assert by_case["moving_gradient-forged"].state == "forged" + assert by_case["moving_gradient-forged"].arm == "wrong_key" + assert by_case["moving_gradient-forged"].expected == "not_detected" + assert by_case["moving_gradient-clean"].arm == "matched_negative" + assert by_case["moving_gradient-marked"].arm == "positive" + assert by_case["moving_gradient-marked"].reference_path == by_case["moving_gradient-clean"].path + assert by_case["moving_gradient-h264_crf23"].state == "attacked" + assert by_case["moving_gradient-h264_crf23"].expected == "unresolved" + assert by_case["moving_texture_hard_negative-clean"].arm == "hard_negative" + assert all(case.media_type == "video" for case in cases) + assert all(case.adapter == "videoseal" for case in cases) + assert all(case.path.is_file() for case in cases) + + rows = {json.loads(line)["case_id"]: json.loads(line) for line in manifest.read_text(encoding="utf-8").splitlines()} + marked = rows["moving_gradient-marked"]["transform"]["parameters"] + assert marked["frames"] == COHORT.FRAME_COUNT + assert marked["message_sha256"] + assert rows["moving_gradient-h264_crf23"]["transform"]["parameters"]["crf"] == 23 + + +def test_scale_and_fps_attacks_carry_no_reference(tmp_path: Path) -> None: + manifest = COHORT.build_cohort(tmp_path / "cohort") + + cases = {case.case_id: case for case in BENCHMARK.load_manifest(manifest)} + assert cases["moving_gradient-h264_crf23"].reference_path is not None + assert cases["moving_gradient-scale_075"].reference_path is None + assert cases["moving_gradient-fps_half"].reference_path is None diff --git a/uv.lock b/uv.lock index 536e539..d92292e 100644 --- a/uv.lock +++ b/uv.lock @@ -245,6 +245,22 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/64/b4/17d4b0b2a2dc85a6df63d1157e028ed19f90d4cd97c36717afef2bc2f395/attrs-26.1.0-py3-none-any.whl", hash = "sha256:c647aa4a12dfbad9333ca4e71fe62ddc36f4e63b2d260a37a8b83d2f043ac309", size = 67548, upload-time = "2026-03-19T14:22:23.645Z" }, ] +[[package]] +name = "audioseal" +version = "0.2.0" +source = { registry = "https://pypi.org/simple" } +dependencies = [ + { name = "einops" }, + { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, + { name = "numpy", version = "2.3.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, + { name = "omegaconf" }, + { name = "torch" }, +] +sdist = { url = "https://files.pythonhosted.org/packages/ba/63/7012f074ee592184c5dd2fd119b64b25233bbcc466c803e9d4bf95c64086/audioseal-0.2.0.tar.gz", hash = "sha256:569a16eb898748ea7f3f5d485e094e21ddcebd68314ad873051a79e9e5b708ae", size = 2572957, upload-time = "2025-12-17T13:43:00.651Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/06/6c/c8875d5e2e8b6b9cb06f483df35b81a2644e793dadd050b33b2d8a8e941d/audioseal-0.2.0-py3-none-any.whl", hash = "sha256:a5989d1e831177e09ba94ec6789ad28ffbd75a2b64e4463a44d92b81f72faf75", size = 63304, upload-time = "2025-12-17T13:42:57.525Z" }, +] + [[package]] name = "av" version = "18.1.0" @@ -1979,8 +1995,8 @@ name = "omegaconf" version = "2.3.1" source = { registry = "https://pypi.org/simple" } dependencies = [ - { name = "antlr4-python3-runtime", marker = "python_full_version < '3.13'" }, - { name = "pyyaml", marker = "python_full_version < '3.13'" }, + { name = "antlr4-python3-runtime" }, + { name = "pyyaml" }, ] sdist = { url = "https://files.pythonhosted.org/packages/ce/3d/e4b57b8d9008c6ebe0d5eff901f91d5700cf7bdb8c8863df817463a7fd5e/omegaconf-2.3.1.tar.gz", hash = "sha256:e5e7de64aeebeddaf8e6d3f7a783b32ac2a01c0fbd9c878012caecb891a1f42a", size = 3298472, upload-time = "2026-06-11T05:05:12.885Z" } wheels = [ @@ -3257,7 +3273,9 @@ detect = [ { name = "pywavelets" }, ] dev = [ + { name = "audioseal" }, { name = "av" }, + { name = "huggingface-hub" }, { name = "invisible-watermark" }, { name = "numpy", version = "1.26.4", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version < '3.13'" }, { name = "numpy", version = "2.3.5", source = { registry = "https://pypi.org/simple" }, marker = "python_full_version >= '3.13'" }, @@ -3357,12 +3375,14 @@ visible = [ [package.metadata] requires-dist = [ { name = "accelerate", marker = "extra == 'diffusion'", specifier = ">=0.25.0" }, + { name = "audioseal", marker = "extra == 'dev'", specifier = ">=0.2.0" }, { name = "av", marker = "extra == 'video'", specifier = ">=18,<19" }, { name = "c2pa-python", specifier = ">=0.35.0" }, { name = "click", specifier = ">=8.0.0" }, { name = "diffsynth", marker = "extra == 'qwen-zimage'", specifier = ">=2.0.17,<3" }, { name = "diffusers", marker = "extra == 'diffusion'", specifier = ">=0.38.0" }, { name = "huggingface-hub", marker = "extra == 'classify'", specifier = ">=0.20.0" }, + { name = "huggingface-hub", marker = "extra == 'dev'", specifier = ">=0.20.0" }, { name = "huggingface-hub", marker = "extra == 'lama'", specifier = ">=0.20.0" }, { name = "huggingface-hub", marker = "extra == 'migan'", specifier = ">=0.20.0" }, { name = "huggingface-hub", marker = "extra == 'text-draft'", specifier = ">=0.20.0" },