remove-ai-watermarks/docs/benchmark-kernel.md

Watermark benchmark kernel

scripts/watermark_benchmark.py is a development-only runner for reproducible image-, audio-, and video-watermark experiments. It calls the existing local DWT-DCT and TrustMark detectors plus the revision-pinned AudioSeal, Perth, PixelSeal, and VideoSeal oracles through thin adapters. It does not add a runtime command, download a corpus, or contact a provenance oracle or provider API. The optional TrustMark package can fetch its official Adobe model weights when its local package cache is incomplete, the audioseal adapter fetches its pinned checkpoints the same way, and the PixelSeal and VideoSeal adapters download pinned weights into a user cache; prepare those dependencies before an offline run. Perth's checkpoint is bundled in its pinned source revision.

The kernel answers three different questions and never merges their answers:

• detection records whether one named adapter recognized its own signal.
• removal records that a removed-state artifact no longer produced a signal, or that the signal remained. It always sets certifies_erasure to false.
• fidelity compares decoded pixels with an explicit reference. It says nothing about whether an invisible watermark remains.

In particular, not_detected means only that the selected detector did not recognize its signal. It is not a general clean verdict.

Manifest

Input is strict JSONL, one case per line. Relative paths resolve from the manifest directory. The runner verifies every named file against its lowercase SHA-256 digest before invoking any detector, rejects duplicate case_id values, and rejects missing or extra fields.

{"schema_version":1,"case_id":"sample-removed","pair_id":"sample","media_type":"image","adapter":"dwt-dct","arm":"positive","state":"removed","path":"removed.png","sha256":"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","reference_path":"clean.png","reference_sha256":"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789","source_revision":"local-corpus@2026-09-05","transform":{"name":"remove","revision":"remove-ai-watermarks@abc123","parameters":{"method":"auto"}},"seed":7,"expected":"not_detected"}

The fixed fields are:

Field	Contract
schema_version	Integer 1.
case_id	Unique case identity.
pair_id	Groups related clean, marked, attacked, and removed cases.
media_type	image, audio, or video in schema v1.
adapter	dwt-dct, trustmark, or pixelseal for image; audioseal or perth for audio; videoseal for video. The loader rejects a known adapter named against the wrong media type.
arm	positive, matched_negative, wrong_key, or hard_negative.
state	clean, marked, attacked, removed, or forged. forged names an artifact carrying a watermark with a message different from the adapter oracle's fixed one. Expected detection depends on the adapter: PixelSeal and VideoSeal check the fixed message, while AudioSeal's presence rule can still report detected for a foreign message. Perth has an implicit fixed mark and no message payload, so it has no meaningful forged arm. The study records message accuracy separately where the scheme exposes a message.
path, sha256	Artifact path and pinned content digest.
reference_path, reference_sha256	Both strings or both null; the fidelity reference.
source_revision	Corpus, generator, or acquisition revision.
transform	Exact name, revision, and JSON parameters.
seed	Integer or null; never an implicit random seed.
expected	detected, not_detected, or unresolved.

The arms prevent a positive-only benchmark from being mistaken for a detector evaluation. A useful detector run needs known positives plus matched negatives, wrong-key cases where the method supports keys, and difficult watermark-free images. Transformations such as resize, crop, recompression, or removal should be separate rows with exact parameters, not edits made outside the manifest.

Run and output

Keep the report outside the repository, and choose a new output path for every run:

uv run python scripts/watermark_benchmark.py \
  /path/to/manifest.jsonl \
  --output /path/to/results/run-2026-09-05.jsonl

The runner refuses to overwrite an existing report. Every output row repeats the case identity and records the repository commit, whether tracked files were dirty, and SHA-256 digests of the benchmark kernel and adapter source. Detector dependency absence is unavailable; an undecodable artifact or adapter exception is error. Neither is counted as not_detected.

Message-matched adapters record detection.message_bit_accuracy. Score-based presence adapters record detection.score. A missing field means the adapter does not expose that quantity; it is not a numeric zero.

detection.adapter_elapsed_ms measures the wall time of the named detector adapter call only. Image decoding and fidelity calculation are outside that interval. The value is null when the adapter was unavailable or the artifact could not be decoded, because no detector call occurred. Preserve the first call separately when aggregating: it can include lazy imports and model loading, while subsequent rows describe warm validation cost.

Fidelity is calculated only when a reference is present, both decoded images are 8-bit, and their dimensions match. Other bit depths remain explicitly incomparable until their sample-range contract is defined. The result records 8-bit MAE, the fraction of changed pixel positions, and PSNR. Identical decoded pixels produce psnr_db: null with psnr_status: unbounded_identical, never a misleading numeric zero. Missing or incomparable references remain explicit states rather than fabricated metrics.

Schema v1 deliberately has no aggregate score, confidence interval, automatic attack generator, removal runner, or provider oracle. Add those only after the case-level evidence is large enough to define and test the corresponding statistical or media contract.

Audio rows

media_type: audio rows decode every artifact and reference to mono 16 kHz float32 through the system ffmpeg before any adapter runs, mirroring the image path where decoding also stays outside the detector interval. Audio rows therefore require ffmpeg on PATH; a missing or failing decode is the same explicit error status as an undecodable image.

The audioseal adapter reads the shared revision-pinned oracle in scripts/audioseal_oracle.py (Meta AudioSeal, MIT license, Hugging Face revision and checkpoint SHA-256 pinned and verified at load). Its detected status is the detector's own documented rule: the fraction of per-sample probabilities above 0.5 reaching 0.5. The label is the decoded 16-bit message as a binary string, so a case whose payload differs from the embedded one is still detected with a different label, never not_detected. A wrong_key arm for this adapter is therefore a different-message row, not a clean row.

The perth adapter reads scripts/perth_oracle.py, which pins Resemble AI's Perth source at commit ff1c8ac and verifies the bundled perth_net_250000.pth.tar checkpoint as sha256:a15bce457ebc53ce5e6c9c3f11df78cf7ee2bf9cdab0a798902135b4c4027670. The upstream package name is resemble-perth, the pinned source declares version 1.1.0, and its license is MIT. That permits a public, development-only oracle. The published 1.0.1 wheel pins an obsolete Torch stack, so the benchmark installs the exact source revision without adding it to project or runtime dependencies. Perth v1 is an implicit presence mark: the detector exposes one confidence and upstream's strictly-above-0.5 decision rule, but no user message. The kernel therefore records score, uses label: perth-v1 only above threshold, and leaves message accuracy not applicable.

Audio fidelity compares the decoded sample arrays against the explicit reference: identical decoded samples produce snr_db: null with snr_status: unbounded_identical, never a misleading numeric zero; a silent reference reports zero_reference_power; a length mismatch stays an explicit incomparable state. SNR here is reference signal power over error power, the sample-domain analogue of the image PSNR contract.

Perth and PixelSeal local oracle study

scripts/perth_pixelseal_study.py builds a local-only study from three deterministic synthetic audio carriers, three macOS system-voice speech carriers, and two publication-cleared tracked image carriers. It never calls a provider oracle or uploads an artifact. Generated media, the strict manifest, and results stay outside git.

PixelSeal comes from Meta's VideoSeal source tree, package videoseal, at the same exact 870ca7f revision as the VideoSeal oracle. The project and weights are MIT licensed. The adapter verifies the official PixelSeal model card as sha256:96c4a448f7d81810780e5bde5a51b039db544b7ec2baf65a605302c1b17362c9 and its 1,237,429,197-byte checkpoint.pth as sha256:0c5665cff20eb6ce1b5aaa7d91c19dafb418bfee32d02dd3344e4ed60d9d75bd. The source metadata declares package version 1.0. The 1.0.1 PyPI wheel omits the PixelSeal card, so this development oracle likewise requires the exact source revision and remains outside installed CLI and library paths. Its explicit matched-message decision is 256-bit accuracy at or above 0.9, aligned with the VideoSeal adapter; this is the benchmark's rule, not a general watermark-presence classifier.

uv run python scripts/perth_pixelseal_study.py \
  --output-dir .local-eval/perth-pixelseal-2026-09-28-v4

Development baseline, 2026-09-28

The final local run used recipe sha256:b43953678294d97a72bb323623797ec8d7b8e476b101485e9ecfefbe02353c0e, kernel sha256:26dce20f42f7131daa4885eed073baf7372ebb07c0e8ccb4ec50cd603da082bf, Perth oracle sha256:a99b9b64a7667b2ee46ac8b57e7c29eec6b13646e3297856e67816984f1a3a9c, and PixelSeal oracle sha256:fc15c444e21404c1413d94251e7746964a347d9596faecfdd94b5e45ab2d0949. Its manifest is sha256:ecdcca929a73b127db3682636f8a2322fc128abb632f3b779e415a7130e0150d, case rows are sha256:ec0a794f5a6580e7bc50e122e1ebae0ea193c9b35b69c7408126d00304c45b38, and study metadata is sha256:5906d8ef48132f0fb371e052870a67a682d4b85c7c5c0e23730a951dca970cc1. All asserted rows matched their expected result; unresolved synthetic Perth positives remain measurements rather than hidden failures.

Adapter and state	Cases	Detected	Not detected	Message accuracy or score
PixelSeal clean	2	0	2	0.473-0.504 fixed-message accuracy
PixelSeal marked	2	2	0	1.000 on both
PixelSeal JPEG q90	2	2	0	0.996-1.000
PixelSeal foreign-message forged	2	0	2	0.469-0.473 against the fixed message
Perth speech clean	3	0	3	0-0.016 confidence
Perth speech marked	3	3	0	1.000 confidence
Perth synthetic clean	3	0	3	0-0.000083 confidence
Perth synthetic marked	3	1	2	0.190, 0.246, and 1.000 confidence

PixelSeal embedding measured 46.15-49.46 dB PSNR against the two clean carriers. JPEG q90 retained the mark at 37.19-39.87 dB PSNR. The fixed-message verifier rejected both images carrying the foreign message; that proves only matched-key rejection, not absence of a PixelSeal payload.

Perth was reliable on the in-domain speech sample but carrier-dependent on the deliberately out-of-domain synthetic set: only white noise crossed the rule, while the tone stack and pinkish noise remained at 0.190 and 0.246. The three synthetic marked rows measured 12.91-14.98 dB SNR. Perth's internal 16 to 32 to 16 kHz resampling shortened the three marked speech outputs by 53-104 samples, so the kernel correctly records their SNR as incomparable rather than silently trimming them.

The production visible-video cleaning path removed the visible mark from all 24 frames while copying the marked AAC packets byte-for-byte. Both the source and cleaned audio scored 1.000, and the source and output packet streams share sha256:d6be870baea63ef8dceea97b2c04b27a1a35ba3bd1500cd022ccb167fd0795c6. This directly confirms that the current video pipeline does not affect Perth.

PixelSeal removal was not run on this Apple Silicon host. The applicable qwen-zimage, sdxl-zimage, and chroma-zimage invisible-removal profiles require an NVIDIA CUDA device; MPS is not a supported execution path. The missing removed-state rows are therefore an explicit hardware gap, not a clean or failed result.

Audio cohort v1

scripts/watermark_benchmark_audio_cohort.py builds the audio analogue of the synthetic image cohort: three deterministic synthetic carriers (AM tone stack, smoothed noise, white noise), the pinned AudioSeal message, matched clean negatives, MP3 128k, gapless M4A AAC 128k, and deterministic 10 dB additive noise attacks, plus one never-embedded white-noise hard negative. Since v2 the audio cohort also carries the four-state arms: a removed row where the noise transform measurably destroys the mark (the tone stack only - the synthetic noise carriers survive 10 dB noise, so their noise rows stay open attacked measurements), and forged rows embedding a foreign message with expected: detected, because the audioseal rule is watermark presence and the decoded label is what exposes the foreign payload. 20 cases in total, everything hashed into a strict manifest validated through the benchmark loader.

uv run --extra dev python scripts/watermark_benchmark_audio_cohort.py \
  --output-dir .local-eval/watermark-benchmark-audio-cohort-v1
uv run --extra dev python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-audio-cohort-v1/manifest.jsonl \
  --output .local-eval/audio-cohort-v1-results.jsonl

Development baseline, 2026-09-06

The first local run used kernel sha256:2216169ab7c33f7439720f0934e8ac6ba8379e510bfaeabe008f820c7658701d, oracle sha256:bd67e2b1be5c486384ab1f64709f661241451ab1ea7592cc56f0eefb91c415ab, and recipe sha256:f361163b4fbf23b885c026751c544e97be0bf054906c1aad455276992a2768d2, producing manifest sha256:b6c48684f0c85e882159f6d5c4fbeea2f43c3b24d3a2c3a034209c6f531ba73d and case rows sha256:66ac23bd926925dc5baf7216c03f8a7246eeb473315a17701358a3803ab7b0d1. Dependencies were audioseal 0.2.0, torch 2.13.0, and ffmpeg 9.0.1.

State and transform	Cases	Detected	Not detected
clean matched negatives + hard negative	4	0	4
marked, audioseal embed	3	3	0
attacked, MP3 128k	3	3	0
attacked, AAC 128k (gapless M4A)	3	3	0
attacked, 10 dB additive noise	3	2	1

Every clean and never-embedded negative stayed below the decision rule. All codecs retained detection with the full message on the tone-stack and white-noise carriers; the pinkish-noise carrier decoded one flipped payload bit while remaining detected, matching the experiment's per-carrier bit-accuracy observation. The 10 dB noise arm destroyed the tone-stack detection but not white or pinkish noise: carrier-dependent robustness is exactly what the matched arms exist to expose. Nearest-rank adapter timing: cold first call 405.1 ms, warm median 124.2 ms, warm max 157.4 ms for 8-second clips on this host. As with the image baselines, these are diagnostic host rows, not a certified profile.

Speech extension

--with-speech adds fifteen more rows: three system-voice speech carriers (Samantha, Daniel, Milena) rendered by the local macOS say tool - no network, byte-deterministic per voice on a fixed macOS version - each with a matched clean negative, a marked positive, and the same three attacks. Every speech row's transform parameters carry the full TTS provenance (tool, macOS version, voice, text digest, duration). The builder verifies the configured voices render distinct bytes and refuses an aliased set, because macOS 26.6.2 renders Alex and Samantha byte-identically and a name is not a carrier identity.

The first --with-speech run (2026-09-06, 31 cases total, recipe sha256:3f291f49ab9ef2499085708fb614e3723ebee6d35f0b509fc36f017ec5a86c3b, manifest sha256:b9baa79066deb002402cc20166835887d67a5158ff9ce7bbc1963e5d603db3a5, case rows sha256:abc68590f59fbddab1c086371d2854622b41ea756ba61a7a87b76cb6df64c386) kept every synthetic verdict and added: speech clean 0/3 detected, speech marked 3/3 with the full message, codecs 3/3, and full-clip 10 dB noise 0/3 - unlike the synthetic white and pinkish carriers, speech has quiet moments where the noise dominates. Speech marked fidelity sat at 27.9-32.2 dB SNR.

Synthetic cohort v1

scripts/watermark_benchmark_cohort.py builds a reproducible local smoke cohort without a provider API or provenance oracle. Its default 65 cases cover three deterministic 512-pixel carriers, SDXL, FLUX, and Stable Diffusion 1.x DWT-DCT patterns, TrustMark Variant P with schema 1, and JPEG quality 90, 75% resize round-trip, and 5% crop-per-edge round-trip attacks. Each adapter also receives a high-frequency hard negative. TrustMark cases include the package remover's output as a separate removed state.

uv run python scripts/watermark_benchmark_cohort.py \
  --output-dir .local-eval/watermark-benchmark-cohort-v1 \
  --size 512
uv run python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-cohort-v1/manifest.jsonl \
  --output .local-eval/watermark-benchmark-cohort-v1-results.jsonl

The builder refuses an existing destination, hashes every generated file, pins its own recipe digest and dependency versions, and validates the finished manifest through the benchmark's strict loader. It reads the DWT-DCT patterns from the production detector so the generated positive cannot silently drift from the matcher. TrustMark's optional package and model weights must already be available.

--size accepts square carrier sides of at least 256 pixels, divisible by 16. The size is part of every case identity and transform record, so resource sweeps at multiple boundaries can be aggregated without merging unlike artifacts.

Attacked and removed rows intentionally use expected: unresolved: they are measurements, not assertions smuggled into fixtures. The cohort is too small and too synthetic for prevalence, confidence, or release-quality claims. It is a deterministic first baseline that proves the harness and exposes gross detector behavior before the real-image cohort below is run.

Development baseline, 2026-09-05

The first local v1 run used cohort recipe sha256:d98fd6402bcc149833005ec03aab77bd0074ee105696f9e1ac5da01ef9c666f7 and kernel sha256:37e552ba53c3d2aff2c5f9e160a33ad47b0d32c877b0a8fbba39c90ca0ab0179. Dependencies were invisible-watermark 0.2.0, TrustMark 0.9.1, NumPy 1.26.4, and Pillow 12.3.0. The tree was based on commit ad53790 with the cohort and timing changes still uncommitted, so the content hashes, not that commit alone, identify the executed code. A second build in a different directory produced a byte-identical manifest and artifact-hash inventory.

Adapter and state	Unique artifacts	Detected	Not detected
DWT-DCT clean	4	0	4
DWT-DCT marked	9	6	3
DWT-DCT attacked	27	13	14
TrustMark clean	4	0	4
TrustMark marked	3	2	1
TrustMark attacked	9	4	5
TrustMark removed	3	0	3

All three plain-gradient DWT-DCT positives were missed, while all texture and low-detail positives were recognized. DWT-DCT retained 7/9 positives through JPEG, 6/9 through resize, and 0/9 through crop. TrustMark retained 2/3 through JPEG and 1/3 through each resize and crop. Its three remover outputs produced no recognized signal and measured 46.17-49.24 dB PSNR against their clean carriers. Those three negatives are a post-removal observation, not proof that the signal is absent beyond this decoder.

Ten independent sequential benchmark processes reproduced every case status. Nearest-rank adapter timings were:

Adapter call	n	p50	p90	max
DWT-DCT cold, first call per process	10	14.96 ms	19.86 ms	27.88 ms
DWT-DCT warm	450	0.65 ms	1.64 ms	13.53 ms
TrustMark cold, first call per process	10	1,989.47 ms	2,085.62 ms	2,086.02 ms
TrustMark warm	180	17.96 ms	34.20 ms	39.95 ms
TrustMark warm, removed state only	30	16.80 ms	17.69 ms	18.56 ms

These intervals include only the detector adapter call. They exclude image decoding, fidelity metrics, removal, and process startup. The roughly two-second TrustMark cold start matters only when validation is the first TrustMark use in the process; a workflow that already identified the source pays the warm cost. The sample is deliberately diagnostic rather than statistically representative.

Real-image cohort v1

scripts/watermark_benchmark_real_cohort.py replaces procedural carriers with the 38 committed OpenAI/Meta images in the engine-selection content matrix. The matrix has 19 prompt-matched content strata, one image per provider in every stratum, and no duplicate file hash. Its images are project-collected model outputs rather than camera originals; "real-image" here means real, decoded content carriers rather than generated gradients or checkerboards.

Every source row carries reuse_basis: project-maintainer-public-test-clearance. This records permanent public test and benchmark permission for those exact committed bytes. It does not claim that provider terms grant a general dataset license for arbitrary generations. The builder refuses a source without that explicit basis, a hash or dimension mismatch, repeated pixels, an incomplete provider pair, or a duplicate provider/content stratum.

uv run python scripts/watermark_benchmark_real_cohort.py \
  --output-dir .local-eval/watermark-benchmark-real-v1 \
  --size 512
uv run python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-real-v1/manifest.jsonl \
  --output .local-eval/watermark-benchmark-real-v1-results.jsonl

Each source is deterministically center-cropped only if necessary, resized to the selected square geometry with LANCZOS, converted to RGB, and written as a PNG reference. At the default size all current sources are already square, so only the downscale and RGB conversion apply. The source manifest hash, original file and payload hashes, provider, model, content stratum, native dimensions, reuse basis, prompt hash, and standardization recipe travel with every case.

The default build has 798 cases: all three production DWT-DCT patterns plus TrustMark Variant P, each with matched negatives, marked positives, the same three fixed attacks as the synthetic cohort, and TrustMark remover outputs. Clean sources are expected negative only for the selected DWT-DCT or TrustMark adapter; the result must not be generalized into a claim that no other provenance signal exists. The aggregate report adds provider and content-stratum tables when this source metadata is present.

The 12 existing SynthID and Content Seal signal carriers are deliberately not mixed into this matched-negative baseline. They remain suitable for a separate cross-watermark interference layer, where their original signal and oracle provenance can be named instead of being mislabeled as globally clean.

Development baseline, 2026-09-06

Three sequential local runs reproduced all 798 case verdicts with no unstable case, adapter error, or unavailable result. Both adapters rejected all 38 matched negatives. The carrier set therefore removed the synthetic cohort's false impression that TrustMark was broadly carrier-fragile, while confirming and sharpening the DWT-DCT carrier-dependence finding.

Adapter and state	Unique artifacts	Detected	Not detected
DWT-DCT marked	114	74	40
DWT-DCT JPEG q90	114	76	38
DWT-DCT resize 75% round-trip	114	77	37
DWT-DCT crop 5% per edge	114	0	114
TrustMark marked	38	38	0
TrustMark, all three attacked arms	114	114	0
TrustMark removed	38	0	38

DWT-DCT marked recall was 26/38 for FLUX, 23/38 for Stable Diffusion 1.x, and 25/38 for SDXL. JPEG and resize retained every one of the 74 base detections, then moved two and three previously missed cases above the decoder threshold respectively. They did not create new ground-truth marks; this is why the report presents pairwise transitions rather than describing 76/114 and 77/114 as ordinary attack retention. The crop destroyed every recognized DWT-DCT signal.

The content strata expose a strong dependence that three procedural carriers could not measure. Interior low light, person in context, and single portrait were 0/6 on the un-attacked DWT-DCT embeds, while nine strata were 6/6. Meta carriers were detected in 42/57 marked cases and their prompt-matched OpenAI partners in 32/57. Those are descriptive counts over one image per provider and stratum, not a provider-population estimate.

TrustMark was detected before and after every fixed attack in every stratum. All 38 remover outputs produced no recognized TrustMark signal, with 41.26 to 49.80 dB PSNR and 45.27 dB median PSNR against the standardized carriers. This remains a decoder-scoped post-removal observation, not proof of erasure against another implementation.

The run was based on commit f61a7ec with tracked benchmark changes still uncommitted. The exact inputs and implementations are therefore bound by hash:

• source manifest: sha256:aa37aa97ed3b32a59242fd506fa3cee545df615f98690d373735929a9d7d0089;
• content-manifest validator: sha256:72ddb2e489d5a7f92cd948b880a0fd308d4083ed86d68fc49897d646aceb7ec5;
• real-cohort recipe: sha256:f3851dae5d4eee69510c3dc5b5b15b5116720d944cfeed631894e635c2a5e9c9;
• shared embed/attack helper: sha256:c78342f0d125d6636259b641e67e71b72b577455057e568052d2a0a4df00c9dd;
• benchmark kernel: sha256:a96874a41fa0a38cd29d957c6a69683de887e942b480ce7668090b0f1a9e3329;
• generated manifest: sha256:77dc6094fcfbbaa6fdb9a15d96dee6fb3d4349829b2b1d7cf5f006aa7fb0ae8a;
• case-level results: sha256:bb483be015c012fd28d0979510244c834475a23bbc215226c998fada73485ed3, sha256:35d735de9654a36d6d2a6e096f7a2c3f8a70b1b22ead4f6f414d6c1494e53b2a, and sha256:52a9a2bb78dcc9f40609b5ca13812470d21477e5e553892c2bbef4b803fee433;
• aggregate report: sha256:4496edf231a20cc7977f003c30313af5acbe8c99075e22d2cb2eeec5e104f3eb.

The local timing rows were host-contended and are not used for a new latency claim. The controlled resource profile below remains the performance evidence.

Video rows

media_type: video rows decode every artifact and reference to float RGB frames in [0, 1] as (T, H, W, 3) through the system ffmpeg, staying outside the detector interval like the image and audio decodes. Video fidelity compares decoded frames against the explicit reference clip: mean PSNR over frames (mean_psnr_db, reported separately from image psnr_db), a changed-frame fraction, unbounded_identical for bit-identical pixel sequences, and an explicit shape_mismatch state when frame counts or geometry differ - which is why geometry-changing attack arms carry no reference instead of a fabricated comparison.

The videoseal adapter loads Meta's standalone TorchScript release of VideoSeal (MIT license) from scripts/videoseal_oracle.py - one pinned file, no videoseal package, whose wheel pulls a research dependency chain without macOS arm64 wheels and whose loader resolves cards relative to the working directory. The upstream video evaluation ignores the model's detection head (it prepends a constant) and scores the aggregated 256-bit message decode, so the adapter's detected is its own explicit rule: bit accuracy of the averaged decode against the oracle's fixed message at or above 0.9. The label is the decoded message as 64 hex characters. Because the rule is message accuracy, a different-message row reads not_detected - the matched verifier rejects the foreign payload - unlike the audioseal presence rule above. Every videoseal detection record carries a temporal block - frame count, min/mean/max and the full per-frame bit-accuracy series - and the aggregation is computed in the oracle from raw per-frame logits, because the TorchScript build ignores the aggregation argument of detect_video_and_aggregate. The aggregation matrix itself (avg versus norm-weighted) is evaluated in VideoSeal temporal evaluation, not in the kernel, which keeps plain avg as its canonical rule.

Video cohort v1

scripts/watermark_benchmark_video_cohort.py builds the video analogue of the other cohorts: two deterministic synthetic 64-frame 256x256 clips (moving gradient, moving texture), the pinned 256-bit message, matched clean negatives, one never-embedded hard negative, and three attacks - a realistic H.264 re-encode at crf 23, a 75 percent downscale, and a frame-rate halving that drops half the keyframes. Since v2 the video cohort also carries the four-state arms: removed rows reusing the crf 23 re-encode that measurably takes both synthetic carriers below the rule, and forged rows embedding a foreign message with expected: not_detected, because the videoseal rule is message accuracy and the matched verifier rejects a foreign payload. 15 cases in total.

uv run --extra dev python scripts/watermark_benchmark_video_cohort.py \
  --output-dir .local-eval/watermark-benchmark-video-cohort-v1
uv run --extra dev python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-video-cohort-v1/manifest.jsonl \
  --output .local-eval/video-cohort-v1-results.jsonl

Development baseline, 2026-09-07

The first local run used kernel sha256:7a3906cebc3c7cacbac7e204c014d1d6281615e7acbbc5b3a8fcecc939ca68c7, oracle sha256:6c889bf399e8e3bada439eee4a1e9c0b04177be78dfee756d5daa79e73db71b3, recipe sha256:dd06ae1c40ef7d7f6fd2b258b20444a3250254a5ef258542a4eaf43cc501342b, manifest sha256:bb6a7dda6b3897794f2fbfe47291329bb7b9ff7ee57c0634ebb386e58acd5e7a, and case rows sha256:5793b8ba33c55e135e5bef43f33fd2699bb57bae6ca8ead38b1dab0123d1dd12. The checkpoint is the TorchScript build y_256b_img.jit sha256:5c7a4581c36fc6090aafdcfb3999123bae5172a4847f22e2da4e7fd1a39d1e1b pinned to upstream commit 870ca7f.

State and transform	Cases	Detected	Not detected
clean matched negatives + hard negative	3	0	3
marked, videoseal embed (crf 8 artifact)	2	2	0
attacked, H.264 crf 23	2	0	2
attacked, 75 percent downscale + crf 18	2	0	2
attacked, frame rate halved	2	2	0

Direct oracle readings fill in what the threshold hides: the marked clips decode at 0.992-0.996 bit accuracy (48.5 and 41.4 dB mean PSNR against their clean carriers), the crf 23 re-encode collapses to 0.56-0.57, the downscale to 0.69, and the halved frame rate keeps 0.996. Two cautions travel with these numbers: the watermark survives H.264 only at generous quality - a realistic crf 23 re-encode erases it on these smooth synthetic carriers, while a high-bitrate uniform-noise carrier survived crf 18 in the isolated probe, so retention is carrier- and quality-dependent - and dropping half the frames does not touch the message because the secret repeats across keyframes. These are first-baseline observations on synthetic carriers, not robustness curves; a sweep across crf values and real-content carriers is the natural follow-up.

Four-state baseline, 2026-09-07 (v2 cohorts)

The v2 runs used kernel sha256:22f2c1dd5c3b7aeb96090c4c578c6d1841ee62d90958dd1c5c7ad7db71835eec with audio recipe sha256:31c021144212a43565fb3746fadc835bfc37ea8ac98998fad9bba9e72410e82c (manifest sha256:43c7f77f1d4537c00d4b0dd5128e9becc6c860cdd196a7ba2f7541f4276784ec, case rows sha256:c73d7dddfdb90262e96be4d67ec1567d9046812661928cc816cb9f97f4a01a7e) and video recipe sha256:8f25929304ef4f94721667968ee4bb89ebf584a2108de7e02db36c855dc2ddc8 (manifest sha256:b5659848c0d05e205f80ab36480e70fdf7e5f9df4d8ada6ffadec42c97b88998, case rows sha256:c9ad56848bb7efe200aaf1189a8eb03f7571567e6702edd195a62f812fa05815). Every v1 verdict was unchanged; the new arms read: audio removed 0/1 detected, audio forged 3/3 detected under the presence rule, video removed 0/2, video forged 0/2 under the message rule - zero expected mismatches. The mechanism underneath these rows, including double embedding, is measured in Watermark forgery study.

Aggregate repeated runs

scripts/watermark_benchmark_report.py turns one or more result JSONL files into a standalone Markdown report. Treat each input file as one independent process run: the first measured call for each adapter in that file is cold, and the remaining measured calls are warm.

uv run python scripts/watermark_benchmark_report.py \
  .local-eval/results/run-01.jsonl \
  .local-eval/results/run-02.jsonl \
  --output .local-eval/results/report.md

The report retains raw observation counts but deduplicates detector summaries by artifact SHA-256 within each adapter and state. It reports state and transform breakdowns, post-removal observations, nearest-rank timing, fidelity, expected-result mismatches, unstable repeated verdicts, errors, and unavailable adapters as separate evidence. A repeated case_id whose artifact, transform, or other case identity changes is rejected rather than merged. Input file digests in the final report bind every aggregate to its exact case-level source. The runner decodes an artifact used by multiple cases only once per process and passes the decoded pixels to both local adapters. Synthetic recipes record the seed actually passed to generation; a manifest seed is not a substitute for that generator input. Video observations must decode the saved artifact whose digest the row records, including controls and double embeds. Adapter timing does not include file decoding.

Profile process cost

scripts/watermark_benchmark_resources.py runs each manifest in fresh Python processes. It records parent-observed wall time and the child's absolute peak RSS, while preserving the ordinary case-level JSONL outputs for the detector report above.

uv run python scripts/watermark_benchmark_resources.py \
  .local-eval/cohort-256/manifest.jsonl \
  .local-eval/cohort-512/manifest.jsonl \
  .local-eval/cohort-1024/manifest.jsonl \
  --repeat 3 \
  --output-dir .local-eval/resource-profile

Wall time includes interpreter startup, imports, image decoding, detector calls, fidelity metrics, and result writing. Peak RSS is the process's absolute high-water mark, not incremental memory caused only by validation. Run DWT-DCT and TrustMark manifests separately to avoid attributing one adapter's model load to the other. The profiler uses only local adapters and never calls an oracle or provider API.

Development resource profile, 2026-09-06

Ten sequential fresh-process runs per size were measured on macOS arm64 with Python 3.12.13. DWT-DCT manifests contained 46 cases; TrustMark manifests contained 19. Every status count was stable across all ten repetitions.

Adapter	Max geometry	Cases	Process wall p50	p90	Absolute peak RSS p50	p90
DWT-DCT	256x256	46	263.07 ms	322.61 ms	74.66 MiB	75.31 MiB
DWT-DCT	512x512	46	482.67 ms	498.07 ms	90.56 MiB	91.02 MiB
DWT-DCT	1024x1024	46	1,360.10 ms	1,389.40 ms	159.58 MiB	160.00 MiB
TrustMark	256x256	19	3,191.18 ms	6,074.68 ms	741.11 MiB	743.66 MiB
TrustMark	512x512	19	9,172.26 ms	15,430.20 ms	752.92 MiB	758.17 MiB
TrustMark	1024x1024	19	7,301.26 ms	9,797.40 ms	848.91 MiB	852.92 MiB

The TrustMark wall-time variance, including one 18.34-second 512 px maximum, did not coincide with comparable memory movement. The non-monotonic 512/1024 times prove that this host was contended, so the TrustMark wall rows are a diagnostic range and cannot establish size scaling or throughput. Repeat them on an idle host before making a performance claim. Peak RSS stayed much more stable: its nearly flat 256/512 values establish the fixed model cost, while 1024 px adds about 100 MiB of high-water memory.

These whole-process rows are deliberately not divided by the case count. A fresh-process average would hide the fixed model load and is not the cost of one post-removal check. The current three-run real-image cohort measured warm p50 adapter calls of 0.57 ms for DWT-DCT and 29.11 ms for TrustMark at 512 px, excluding file decoding. A workflow that identified the source before removal already holds TrustMark's singleton decoder, so validation reuses that fixed model cost. Starting a new process or loading TrustMark only after removal is the expensive design and should be avoided.

The exact profiler source was sha256:caf9674dffceddebb8cbf26067581250334fbc11e0fbcb567e8dd45555357b54. The local DWT-DCT and TrustMark resource JSONL files were respectively sha256:b7ceb012db5c2f549720c37d6234526396c7120fb68e040705322be35dfd6743 and sha256:f3639f46a9257a3161ed0197016c089ca5c084b2de2e07cb4b65fc774e20544d. They remain untracked because they contain host-specific paths and timings; these hashes bind the summarized numbers to the exact local evidence.