remove-ai-watermarks/docs/benchmark-kernel.md

Watermark benchmark kernel

scripts/watermark_benchmark.py is a development-only runner for reproducible image-watermark experiments. It calls the existing local DWT-DCT and TrustMark detectors through thin adapters. It does not add a runtime command, download a corpus, or contact a provenance oracle or provider API. The optional TrustMark package can fetch its official Adobe model weights when its local package cache is incomplete; prepare that dependency before an offline run.

The kernel answers three different questions and never merges their answers:

• detection records whether one named adapter recognized its own signal.
• removal records that a removed-state artifact no longer produced a signal, or that the signal remained. It always sets certifies_erasure to false.
• fidelity compares decoded pixels with an explicit reference. It says nothing about whether an invisible watermark remains.

In particular, not_detected means only that the selected detector did not recognize its signal. It is not a general clean verdict.

Manifest

Input is strict JSONL, one case per line. Relative paths resolve from the manifest directory. The runner verifies every named file against its lowercase SHA-256 digest before invoking any detector, rejects duplicate case_id values, and rejects missing or extra fields.

{"schema_version":1,"case_id":"sample-removed","pair_id":"sample","media_type":"image","adapter":"dwt-dct","arm":"positive","state":"removed","path":"removed.png","sha256":"0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef","reference_path":"clean.png","reference_sha256":"abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789","source_revision":"local-corpus@2026-09-05","transform":{"name":"remove","revision":"remove-ai-watermarks@abc123","parameters":{"method":"auto"}},"seed":7,"expected":"not_detected"}

The fixed fields are:

Field	Contract
schema_version	Integer 1.
case_id	Unique case identity.
pair_id	Groups related clean, marked, attacked, and removed cases.
media_type	image in schema v1.
adapter	dwt-dct or trustmark.
arm	positive, matched_negative, wrong_key, or hard_negative.
state	clean, marked, attacked, or removed.
path, sha256	Artifact path and pinned content digest.
reference_path, reference_sha256	Both strings or both null; the fidelity reference.
source_revision	Corpus, generator, or acquisition revision.
transform	Exact name, revision, and JSON parameters.
seed	Integer or null; never an implicit random seed.
expected	detected, not_detected, or unresolved.

The arms prevent a positive-only benchmark from being mistaken for a detector evaluation. A useful detector run needs known positives plus matched negatives, wrong-key cases where the method supports keys, and difficult watermark-free images. Transformations such as resize, crop, recompression, or removal should be separate rows with exact parameters, not edits made outside the manifest.

Run and output

Keep the report outside the repository, and choose a new output path for every run:

uv run python scripts/watermark_benchmark.py \
  /path/to/manifest.jsonl \
  --output /path/to/results/run-2026-09-05.jsonl

The runner refuses to overwrite an existing report. Every output row repeats the case identity and records the repository commit, whether tracked files were dirty, and SHA-256 digests of the benchmark kernel and adapter source. Detector dependency absence is unavailable; an undecodable artifact or adapter exception is error. Neither is counted as not_detected.

detection.adapter_elapsed_ms measures the wall time of the named detector adapter call only. Image decoding and fidelity calculation are outside that interval. The value is null when the adapter was unavailable or the artifact could not be decoded, because no detector call occurred. Preserve the first call separately when aggregating: it can include lazy imports and model loading, while subsequent rows describe warm validation cost.

Fidelity is calculated only when a reference is present, both decoded images are 8-bit, and their dimensions match. Other bit depths remain explicitly incomparable until their sample-range contract is defined. The result records 8-bit MAE, the fraction of changed pixel positions, and PSNR. Identical decoded pixels produce psnr_db: null with psnr_status: unbounded_identical, never a misleading numeric zero. Missing or incomparable references remain explicit states rather than fabricated metrics.

Schema v1 deliberately has no aggregate score, confidence interval, video adapter, automatic attack generator, removal runner, or provider oracle. Add those only after the case-level evidence is large enough to define and test the corresponding statistical or media contract.

Synthetic cohort v1

scripts/watermark_benchmark_cohort.py builds a reproducible local smoke cohort without a provider API or provenance oracle. Its default 65 cases cover three deterministic 512-pixel carriers, SDXL, FLUX, and Stable Diffusion 1.x DWT-DCT patterns, TrustMark Variant P with schema 1, and JPEG quality 90, 75% resize round-trip, and 5% crop-per-edge round-trip attacks. Each adapter also receives a high-frequency hard negative. TrustMark cases include the package remover's output as a separate removed state.

uv run python scripts/watermark_benchmark_cohort.py \
  --output-dir .local-eval/watermark-benchmark-cohort-v1 \
  --size 512
uv run python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-cohort-v1/manifest.jsonl \
  --output .local-eval/watermark-benchmark-cohort-v1-results.jsonl

The builder refuses an existing destination, hashes every generated file, pins its own recipe digest and dependency versions, and validates the finished manifest through the benchmark's strict loader. It reads the DWT-DCT patterns from the production detector so the generated positive cannot silently drift from the matcher. TrustMark's optional package and model weights must already be available.

--size accepts square carrier sides of at least 256 pixels, divisible by 16. The size is part of every case identity and transform record, so resource sweeps at multiple boundaries can be aggregated without merging unlike artifacts.

Attacked and removed rows intentionally use expected: unresolved: they are measurements, not assertions smuggled into fixtures. The cohort is too small and too synthetic for prevalence, confidence, or release-quality claims. It is a deterministic first baseline that proves the harness and exposes gross detector behavior before the real-image cohort below is run.

Development baseline, 2026-09-05

The first local v1 run used cohort recipe sha256:d98fd6402bcc149833005ec03aab77bd0074ee105696f9e1ac5da01ef9c666f7 and kernel sha256:37e552ba53c3d2aff2c5f9e160a33ad47b0d32c877b0a8fbba39c90ca0ab0179. Dependencies were invisible-watermark 0.2.0, TrustMark 0.9.1, NumPy 1.26.4, and Pillow 12.3.0. The tree was based on commit ad53790 with the cohort and timing changes still uncommitted, so the content hashes, not that commit alone, identify the executed code. A second build in a different directory produced a byte-identical manifest and artifact-hash inventory.

Adapter and state	Unique artifacts	Detected	Not detected
DWT-DCT clean	4	0	4
DWT-DCT marked	9	6	3
DWT-DCT attacked	27	13	14
TrustMark clean	4	0	4
TrustMark marked	3	2	1
TrustMark attacked	9	4	5
TrustMark removed	3	0	3

All three plain-gradient DWT-DCT positives were missed, while all texture and low-detail positives were recognized. DWT-DCT retained 7/9 positives through JPEG, 6/9 through resize, and 0/9 through crop. TrustMark retained 2/3 through JPEG and 1/3 through each resize and crop. Its three remover outputs produced no recognized signal and measured 46.17-49.24 dB PSNR against their clean carriers. Those three negatives are a post-removal observation, not proof that the signal is absent beyond this decoder.

Ten independent sequential benchmark processes reproduced every case status. Nearest-rank adapter timings were:

Adapter call	n	p50	p90	max
DWT-DCT cold, first call per process	10	14.96 ms	19.86 ms	27.88 ms
DWT-DCT warm	450	0.65 ms	1.64 ms	13.53 ms
TrustMark cold, first call per process	10	1,989.47 ms	2,085.62 ms	2,086.02 ms
TrustMark warm	180	17.96 ms	34.20 ms	39.95 ms
TrustMark warm, removed state only	30	16.80 ms	17.69 ms	18.56 ms

These intervals include only the detector adapter call. They exclude image decoding, fidelity metrics, removal, and process startup. The roughly two-second TrustMark cold start matters only when validation is the first TrustMark use in the process; a workflow that already identified the source pays the warm cost. The sample is deliberately diagnostic rather than statistically representative.

Real-image cohort v1

scripts/watermark_benchmark_real_cohort.py replaces procedural carriers with the 38 committed OpenAI/Meta images in the engine-selection content matrix. The matrix has 19 prompt-matched content strata, one image per provider in every stratum, and no duplicate file hash. Its images are project-collected model outputs rather than camera originals; "real-image" here means real, decoded content carriers rather than generated gradients or checkerboards.

Every source row carries reuse_basis: project-maintainer-public-test-clearance. This records permanent public test and benchmark permission for those exact committed bytes. It does not claim that provider terms grant a general dataset license for arbitrary generations. The builder refuses a source without that explicit basis, a hash or dimension mismatch, repeated pixels, an incomplete provider pair, or a duplicate provider/content stratum.

uv run python scripts/watermark_benchmark_real_cohort.py \
  --output-dir .local-eval/watermark-benchmark-real-v1 \
  --size 512
uv run python scripts/watermark_benchmark.py \
  .local-eval/watermark-benchmark-real-v1/manifest.jsonl \
  --output .local-eval/watermark-benchmark-real-v1-results.jsonl

Each source is deterministically center-cropped only if necessary, resized to the selected square geometry with LANCZOS, converted to RGB, and written as a PNG reference. At the default size all current sources are already square, so only the downscale and RGB conversion apply. The source manifest hash, original file and payload hashes, provider, model, content stratum, native dimensions, reuse basis, prompt hash, and standardization recipe travel with every case.

The default build has 798 cases: all three production DWT-DCT patterns plus TrustMark Variant P, each with matched negatives, marked positives, the same three fixed attacks as the synthetic cohort, and TrustMark remover outputs. Clean sources are expected negative only for the selected DWT-DCT or TrustMark adapter; the result must not be generalized into a claim that no other provenance signal exists. The aggregate report adds provider and content-stratum tables when this source metadata is present.

The 12 existing SynthID and Content Seal signal carriers are deliberately not mixed into this matched-negative baseline. They remain suitable for a separate cross-watermark interference layer, where their original signal and oracle provenance can be named instead of being mislabeled as globally clean.

Development baseline, 2026-09-06

Three sequential local runs reproduced all 798 case verdicts with no unstable case, adapter error, or unavailable result. Both adapters rejected all 38 matched negatives. The carrier set therefore removed the synthetic cohort's false impression that TrustMark was broadly carrier-fragile, while confirming and sharpening the DWT-DCT carrier-dependence finding.

Adapter and state	Unique artifacts	Detected	Not detected
DWT-DCT marked	114	74	40
DWT-DCT JPEG q90	114	76	38
DWT-DCT resize 75% round-trip	114	77	37
DWT-DCT crop 5% per edge	114	0	114
TrustMark marked	38	38	0
TrustMark, all three attacked arms	114	114	0
TrustMark removed	38	0	38

DWT-DCT marked recall was 26/38 for FLUX, 23/38 for Stable Diffusion 1.x, and 25/38 for SDXL. JPEG and resize retained every one of the 74 base detections, then moved two and three previously missed cases above the decoder threshold respectively. They did not create new ground-truth marks; this is why the report presents pairwise transitions rather than describing 76/114 and 77/114 as ordinary attack retention. The crop destroyed every recognized DWT-DCT signal.

The content strata expose a strong dependence that three procedural carriers could not measure. Interior low light, person in context, and single portrait were 0/6 on the un-attacked DWT-DCT embeds, while nine strata were 6/6. Meta carriers were detected in 42/57 marked cases and their prompt-matched OpenAI partners in 32/57. Those are descriptive counts over one image per provider and stratum, not a provider-population estimate.

TrustMark was detected before and after every fixed attack in every stratum. All 38 remover outputs produced no recognized TrustMark signal, with 41.26 to 49.80 dB PSNR and 45.27 dB median PSNR against the standardized carriers. This remains a decoder-scoped post-removal observation, not proof of erasure against another implementation.

The run was based on commit f61a7ec with tracked benchmark changes still uncommitted. The exact inputs and implementations are therefore bound by hash:

• source manifest: sha256:aa37aa97ed3b32a59242fd506fa3cee545df615f98690d373735929a9d7d0089;
• content-manifest validator: sha256:72ddb2e489d5a7f92cd948b880a0fd308d4083ed86d68fc49897d646aceb7ec5;
• real-cohort recipe: sha256:f3851dae5d4eee69510c3dc5b5b15b5116720d944cfeed631894e635c2a5e9c9;
• shared embed/attack helper: sha256:c78342f0d125d6636259b641e67e71b72b577455057e568052d2a0a4df00c9dd;
• benchmark kernel: sha256:a96874a41fa0a38cd29d957c6a69683de887e942b480ce7668090b0f1a9e3329;
• generated manifest: sha256:77dc6094fcfbbaa6fdb9a15d96dee6fb3d4349829b2b1d7cf5f006aa7fb0ae8a;
• case-level results: sha256:bb483be015c012fd28d0979510244c834475a23bbc215226c998fada73485ed3, sha256:35d735de9654a36d6d2a6e096f7a2c3f8a70b1b22ead4f6f414d6c1494e53b2a, and sha256:52a9a2bb78dcc9f40609b5ca13812470d21477e5e553892c2bbef4b803fee433;
• aggregate report: sha256:4496edf231a20cc7977f003c30313af5acbe8c99075e22d2cb2eeec5e104f3eb.

The local timing rows were host-contended and are not used for a new latency claim. The controlled resource profile below remains the performance evidence.

Aggregate repeated runs

scripts/watermark_benchmark_report.py turns one or more result JSONL files into a standalone Markdown report. Treat each input file as one independent process run: the first measured call for each adapter in that file is cold, and the remaining measured calls are warm.

uv run python scripts/watermark_benchmark_report.py \
  .local-eval/results/run-01.jsonl \
  .local-eval/results/run-02.jsonl \
  --output .local-eval/results/report.md

The report retains raw observation counts but deduplicates detector summaries by artifact SHA-256 within each adapter and state. It reports state and transform breakdowns, post-removal observations, nearest-rank timing, fidelity, expected-result mismatches, unstable repeated verdicts, errors, and unavailable adapters as separate evidence. A repeated case_id whose artifact, transform, or other case identity changes is rejected rather than merged. Input file digests in the final report bind every aggregate to its exact case-level source. The runner decodes an artifact used by multiple cases only once per process and passes the decoded pixels to both local adapters; the adapter timing therefore does not include file decoding.

Profile process cost

scripts/watermark_benchmark_resources.py runs each manifest in fresh Python processes. It records parent-observed wall time and the child's absolute peak RSS, while preserving the ordinary case-level JSONL outputs for the detector report above.

uv run python scripts/watermark_benchmark_resources.py \
  .local-eval/cohort-256/manifest.jsonl \
  .local-eval/cohort-512/manifest.jsonl \
  .local-eval/cohort-1024/manifest.jsonl \
  --repeat 3 \
  --output-dir .local-eval/resource-profile

Wall time includes interpreter startup, imports, image decoding, detector calls, fidelity metrics, and result writing. Peak RSS is the process's absolute high-water mark, not incremental memory caused only by validation. Run DWT-DCT and TrustMark manifests separately to avoid attributing one adapter's model load to the other. The profiler uses only local adapters and never calls an oracle or provider API.

Development resource profile, 2026-09-06

Ten sequential fresh-process runs per size were measured on macOS arm64 with Python 3.12.13. DWT-DCT manifests contained 46 cases; TrustMark manifests contained 19. Every status count was stable across all ten repetitions.

Adapter	Max geometry	Cases	Process wall p50	p90	Absolute peak RSS p50	p90
DWT-DCT	256x256	46	263.07 ms	322.61 ms	74.66 MiB	75.31 MiB
DWT-DCT	512x512	46	482.67 ms	498.07 ms	90.56 MiB	91.02 MiB
DWT-DCT	1024x1024	46	1,360.10 ms	1,389.40 ms	159.58 MiB	160.00 MiB
TrustMark	256x256	19	3,191.18 ms	6,074.68 ms	741.11 MiB	743.66 MiB
TrustMark	512x512	19	9,172.26 ms	15,430.20 ms	752.92 MiB	758.17 MiB
TrustMark	1024x1024	19	7,301.26 ms	9,797.40 ms	848.91 MiB	852.92 MiB

The TrustMark wall-time variance, including one 18.34-second 512 px maximum, did not coincide with comparable memory movement. The non-monotonic 512/1024 times prove that this host was contended, so the TrustMark wall rows are a diagnostic range and cannot establish size scaling or throughput. Repeat them on an idle host before making a performance claim. Peak RSS stayed much more stable: its nearly flat 256/512 values establish the fixed model cost, while 1024 px adds about 100 MiB of high-water memory.

These whole-process rows are deliberately not divided by the case count. A fresh-process average would hide the fixed model load and is not the cost of one post-removal check. The current three-run real-image cohort measured warm p50 adapter calls of 0.57 ms for DWT-DCT and 29.11 ms for TrustMark at 512 px, excluding file decoding. A workflow that identified the source before removal already holds TrustMark's singleton decoder, so validation reuses that fixed model cost. Starting a new process or loading TrustMark only after removal is the expensive design and should be avoided.

The exact profiler source was sha256:caf9674dffceddebb8cbf26067581250334fbc11e0fbcb567e8dd45555357b54. The local DWT-DCT and TrustMark resource JSONL files were respectively sha256:b7ceb012db5c2f549720c37d6234526396c7120fb68e040705322be35dfd6743 and sha256:f3639f46a9257a3161ed0197016c089ca5c084b2de2e07cb4b65fc774e20544d. They remain untracked because they contain host-specific paths and timings; these hashes bind the summarized numbers to the exact local evidence.