remove-ai-watermarks/docs/verification-plan.md

Full verification plan

Verification log and historical plan. Dated results describe the code and datasets at the time of each run. They are evidence for maintainers, not a current CLI contract. Recheck open items against the current code and issue tracker before treating them as work that remains.

How we convince ourselves the library actually works, across its whole surface, on real data.

This is the pre-release and periodic-audit plan. It is deliberately organized by oracle strength rather than by module, because the hard part is never "call the function" -- it is "know what the right answer was". A sweep with no oracle proves only that nothing threw.

Performance depends on format, enabled extras, and hardware. Measure it locally with representative inputs; do not publish private dataset sizes or run statistics.

Public test sources

source	role
data/synthid/	labeled SynthID oracle fixtures and evaluation manifest
data/fixtures/provenance/	deterministic format and provenance fixtures
data/calibration/<vendor>/	minimal controlled inputs used to derive detection silhouettes
synthesized inputs	constructed ground truth for pure and format-level checks

Tier A -- self-evident oracles (full corpus, unattended)

Properties that are true or false without anyone labelling anything. These are the backbone: they scale to large local datasets and catch regressions with zero human cost.

A1. Recorded-verdict regression

A dated local cache of identify verdicts can be replayed against the same inputs to detect drift in verdict, platform, confidence, or signal set.

Caveat that makes this honest: a diff is not automatically a bug -- the sidecars were written by older versions, so intended improvements also show up. The output is therefore a classified diff (new detections / lost detections / changed platform / changed confidence), reviewed once, then re-baselined. Lost detections are the alarm.

Implemented as scripts/sidecar_regression.py (resumable, ~1.5 h at 8 workers).

Local run protocol

Re-run identify against locally recorded sidecars, classify losses separately from intended new detections, and keep generated reports under .local-eval/. Do not commit dataset-derived counts or identifiers.

A2. Parity: whatever we detect, we must be able to remove

For every image where a signal fires: remove, re-scan with the same oracle, assert quiet.

• metadata: scripts/metadata_removal_audit.py;
• visible: scripts/visible_removal_audit.py, once per backend.

Detection does not depend on the fill backend. Run scripts/visible_positives.py once and pass its output through the audit's --paths-file option.

Local parity protocol

Run detection, removal, and re-detection over a representative local set. Confirm that decoded pixels remain unchanged for metadata-only operations. Keep reports untracked.

A3. Byte-level invariants

• no-op remove_visible returns the ORIGINAL bytes (not a re-encode)
• pixels outside the fill mask are bit-identical to the input
• JPEG metadata strip is pixel-lossless on the DEFAULT path (--remove-all re-encodes by design -- see metadata.py; assert the split, not losslessness everywhere)
• lossless source formats survive a misnamed extension

A4. Idempotence and order-independence

• remove_visible(remove_visible(x)) == remove_visible(x)
• strip(remove(x)) == remove(strip(x)) in signal terms
• a second identify on a cleaned output reports no metadata signals

A5. Contract sweep across every parameter choice

scripts/smoke_matrix.py (exists, 68 rows, 0 skipped with --diffusion) covers every choice-valued flag on fixtures. Extend from fixtures to a stratified corpus slice (~500 images spanning format x provenance x aspect ratio), asserting exit-code semantics rather than just absence of crash.

Known trap to encode: exit 2 is triply overloaded (no-visible-mark, no-invisible-signal, Click usage error). A wrapper cannot distinguish them without parsing stderr. Either the sweep asserts on stderr, or the codes get split -- the latter is the better fix.

Coverage method

Compare the flags and values exercised by the matrix against the options declared by the CLI. Include optional backends, batch modes, tiling, region-targeted composition, the ESRGAN upscaler, and the ffmpeg audio/video strip. The gap to find is not only "logic untested" but "never executed on real data", which is precisely what this campaign is for.

Bug found by the extension: --steps below ~7 crashes inside torch

Effective timesteps are int(steps * strength). At the vendor-adaptive default strength (0.15, or 0.10 for OpenAI) any --steps under 7 rounds to zero, and the pipeline dies with a raw traceback:

$ remove-ai-watermarks invisible img.png --steps 5
RuntimeError: cannot reshape tensor of 0 elements into shape [0, -1, 1, 512]

Fully valid CLI arguments, no special flags, no --force. The value is accepted, the crash is a torch internal, and nothing tells the user that steps and strength interact. Fix is either a clamp to >=1 effective step or an up-front validation naming both values.

Method note: the first run of the knob rows failed 12 times with this identical error, which read like twelve broken features. It was one bad harness parameter (--steps 4) sitting on top of one real bug. An error that is IDENTICAL across unrelated rows is evidence of a common cause, not of many faults -- check the shared input first.

Tier B -- constructed ground truth (automatable, no labelling)

Where reality gives no answer key, build one. This is the tier that closes the two biggest holes: fill quality and detector response at the edge of the operating range.

B1. Fill quality with a true reference

Real marks have no clean counterpart, so quality has only ever been eyeballed. Construct it instead: take a clean corpus image, stamp a known mark at a known position (the captured alpha maps make this exact), remove it, and compare against the true original.

Yields PSNR / SSIM per --backend (cv2 / migan / lama), sliced by background class, which is exactly the axis where the docs say quality varies but no number exists.

Implemented as scripts/fill_quality.py. Two reporting rules are load-bearing: score INSIDE the footprint (whole-frame PSNR sits near 60 dB whatever the backend does), and use the MEDIAN (a fill that reproduces a flat background exactly scores PSNR=inf, and one inf makes a mean inf -- the first run reported "+inf" for every flat bucket).

Local fill-quality protocol

Construct marked images from clean local references, compare each backend against the known original inside the affected footprint, and keep the report under .local-eval/.

B2. Detector response curves

Use scripts/detector_response.py to sweep mark size, opacity, background, and aspect. Report detection and maskability separately. Generated reports stay under .local-eval/.

B3. Invisible round-trip, positive-control gated

The open DWT-DCT detector is positive-only and carrier-fragile: "not found" on a fragile carrier proves nothing (measured: chatgpt-1.png recovers 114/128, below the 118 gate). Every invisible assertion must first embed on the SAME carrier and confirm recovery, and degrade to a skip rather than a pass when the control fails. Already implemented in smoke_matrix.py; apply the same discipline anywhere else this detector is used.

B4. Resource ceilings

Peak RSS and wall time per backend x input size, up to 25 MP. The memory-constrained CPU tier is a real constraint (MI-GAN must stay ~0.6-0.9 GB by cropping around the mask); a regression here is invisible today and would only surface under load.

Tier C -- human-labelled accuracy (bounded by labelling effort)

The machinery exists: visible_recall_sample.py -> visible_sheets.py -> visible_groundtruth.py -> visible_eval.py.

• Recall needs representative sampling per mark and aspect ratio.
• Precision must be benchmarked before and after every detector change with --vs <snapshot>.
• Coverage is separate from tuning. A missing vendor detector cannot be repaired by changing another detector's threshold.

Three harness rules are load-bearing and must not be relaxed: score a mark only within its crop's adjudication scope; take provenance from metadata, never from labels; and never report recall from the detector-sampled set.

Tier D -- external oracles (manual, not automatable here)

SynthID removal cannot be verified locally by design -- no public decoder exists. Each vendor has its own oracle and it covers only that vendor's content: openai.com/verify for OpenAI (more accessible, the automation candidate), the Gemini app for Google (manual, rate-limited). A quiet metadata proxy is not proof the pixel watermark is gone.

Scope honestly: this tier certifies strength floors on a handful of images per vendor, and that is all it can do. See docs/synthid.md.

D1. Sampling frame

Select a representative local sample without committing images, identifiers, dataset sizes, or oracle results. Stratify by signal family and preserve the sampling method outside the public repository.

D2. The oracle is the bottleneck, not the GPU

Measured on MPS (2026-07-19), single invocation of invisible:

--max-resolution	256	384	512	768	1024
wall time	37.9 s	58.9 s	59.4 s	65.2 s	118.3 s

384/512/768 are indistinguishable, so below ~768 the cost is dominated by fixed model load, not diffusion. Confirmed by batching: 4 images in one process took 105.5 s (26.4 s/image) against 59.4 s/image one at a time -- roughly 40 s fixed overhead per invocation and ~15 s marginal per image at 512 (rough: run-to-run variance is large).

Two consequences for the harness:

• Amortize the fixed cost: one long-lived process over many images, never one invocation per image. That is a 2-4x win. Shrinking below 512 is not.
• The binding constraint is the external oracle's throughput, which is manual and rate limited. So do not run a uniform grid; spend each oracle check where the answer is uncertain -- bisect strength per content class to certify a floor in ~10 checks instead of ~100.

D3. Mandatory control before trusting any reduced-size run

--max-resolution is downscale -> diffuse -> Lanczos upscale. If the resize round trip alone damages SynthID, the oracle goes quiet for a reason unrelated to removal and the result does not transfer to production at native resolution.

Before any reduced-size sweep, run the resize round trip with no diffusion and put the result through the vendor oracle. If the watermark survives, the reduced size is a valid test bed; if it does not, reduced-size results are measuring the resizer. This is the same failure shape as the imwatermark carrier-fragility trap in B3: an oracle that falls silent for the wrong reason reads exactly like success.

docs/synthid.md cites ~99.98% TPR across 30 transforms including resize, which predicts the control passes -- but that is Google's claim about their own decoder, not our measurement, so it is a hypothesis to test, not a reason to skip the control.

Tier E -- robustness and adversarial inputs

Malformed and hostile inputs, including truncated files: at many offsets, corrupt headers, 16-bit and CMYK, absurd dimensions, decompression bombs, zero-byte files, unicode and RTL filenames, symlinks, read-only output dirs, concurrent runs on one file. The bar is never "handles it" but never raises and never silently degrades.

Build order

1. A1 sidecar regression -- highest value per hour, unattended, needs no new labels.
2. A2/A3/A4 parity and invariants -- representative local set, reusing existing audit scripts.
3. B1 fill quality -- closes the oldest unmeasured claim in the project.
4. B2 detector curves -- cheap, and directly guards the geometry class of bug.
5. A5 contract sweep over a representative local set.
6. B4 resource ceilings, E robustness.
7. C recall expansion -- gated by labelling appetite.
8. D oracles -- manual, per release.

Every tier writes a versioned snapshot so runs are comparable over time; a run that cannot be diffed against the last one is a one-off, not a regression suite.

What the measurements imply for detection work

Recorded here because each item is grounded in a number from this campaign, not because it is a prioritized plan (that lives elsewhere -- see the note at the end of this section).

Metadata absence does not disable the detectors -- it disables the RELAXATION

The detectors are pixel-based and need no metadata. What metadata does is relax the false-positive gate (auto vs strict). So "work better without metadata" means strengthening the strict-path detectors themselves; it is not a gating problem.

Per mark, what actually goes away when metadata is stripped:

• The pill loses its metadata-confirmed arm. Without metadata it depends on the bottom-right Jimeng wordmark.
• The sparkle already runs on pixels. Its threshold remains a deliberate recall-versus-precision trade.
• Uncovered vendors are metadata-independent gaps. They require a detector of their own.

Where the evidence points

1. Prefer per-vendor CJK templates when fitted geometry separates similar marks.
2. Recalibrate every detector when changing its front-end. Thresholds do not transfer between binary, tophat, and gray.
3. Treat the Jimeng wordmark as a load-bearing confirmation path for the pill.

Measure before improving

Use Tier B2 detector-response curves before tuning a detector with sparse labelled examples. Sweep size, contrast, aspect, and background texture so geometry regressions are visible without exposing private evaluation statistics.

This section records what the measurements imply technically. Prioritization is tracked separately, outside this repo.

Local end-to-end verification

Run scripts/real_examples_e2e.py against representative local inputs before releases that affect image handling. The script must read from .local-eval/, write only untracked temporary output, and report behavior without exposing dataset provenance or aggregate private measurements.

Standing gap

None of this is in maintain.sh, and it should not all be -- the sweeps take hours. But that means no detector-accuracy or CLI-contract regression is caught automatically today. The endpoint of this plan is a cheap subset (fixtures-only smoke + a sidecar diff on a fixed 500-image slice) that CI can run, with the full sweeps staying pre-release.