mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-14 21:40:47 +02:00
Victor Kuznetsov
5c6588b3ef
Repair image and metadata publication contracts, detection routing, research artifact validation, release verification, and the published agent skill. Refresh the dependency lock and reconcile documentation with verified behavior. Add regression coverage and record the completed audit and corrected VideoSeal measurements. Preserve the existing Accelerate and Lightning versions. Validation: Ruff and Pyright pass; 2358 tests pass with 3 skipped. The maintenance gate exits 2 on the documented Accelerate and Lightning security findings.
72 lines
3.9 KiB
Markdown
72 lines
3.9 KiB
Markdown
# External watermark benchmarks: ETI and W-Bench licensing verdict
|
|
|
|
Investigation completed 2026-09-07 against the primary sources, closing the
|
|
last open item of the original research map: whether an honest external
|
|
benchmark profile can be built on ETI or W-Bench. The verdict is split:
|
|
W-Bench is legally usable, ETI is license-blocked.
|
|
|
|
## ETI (NeurIPS 2024 "Erasing the Invisible")
|
|
|
|
Sources verified: the Hugging Face dataset
|
|
`furonghuang-lab/ETI_Competition_Data` at revision `d80024580b5a` and the
|
|
organizers' evaluation repository `erasinginvisible/eval-program`.
|
|
|
|
- **License: absent everywhere.** The dataset carries no license tag and no
|
|
license section in its card; the evaluation repository has no LICENSE file.
|
|
With no grant of rights, the corpus is unusable in this public repository,
|
|
and downloading it for local evaluation builds results that cannot be
|
|
published. The only unblock is written permission from the organizers -
|
|
an external action that needs an explicit decision, not a default.
|
|
- **Composition (verified from the card):** 300 Beige-track images
|
|
(Gaussian Shading on Stable Diffusion 2.1, StegaStamp on Flux.1) and 300
|
|
Black-track images (JigMark, PRC, StableSignature, Trufo, plus
|
|
Gaussian-Shading+JigMark and StableSignature+StegaStamp doubles), plus
|
|
~1,072 Beige and ~1,650 Black valid submissions with per-watermark
|
|
detection results, image-quality metrics, and final scores.
|
|
- One earlier blocker is now stale: the attacked submission images ARE part
|
|
of the release per the card note, not missing. The license blocker alone
|
|
remains decisive.
|
|
|
|
## W-Bench / VINE (ICLR 2025)
|
|
|
|
Sources verified: the Hugging Face dataset `Shilin-LU/W-Bench` at revision
|
|
`c81f3924f0fb` and the VINE repository `Shilin-LU/VINE`.
|
|
|
|
- **Dataset license: MIT**, declared on the dataset card. The benchmark is
|
|
legally usable.
|
|
- **Composition (verified from the card):** 10,000 instances sourced from
|
|
COCO, Flickr, ShareGPT4V and similar datasets, eleven watermarking
|
|
methods, evaluated across four editing profiles - regeneration, global
|
|
editing, local editing, image-to-video - plus image distortions. An
|
|
earlier internal note said "15k images"; the card says 10,000, and the
|
|
card wins.
|
|
- **Code license: NTUITIVE Non-Commercial** (NTU technology transfer;
|
|
commercial use requires contacting the authors). The VINE code cannot be
|
|
vendored into this Apache-2.0-licensed project, and pipeline outputs derived
|
|
from running it would inherit that restriction. What remains lawful and
|
|
useful: consuming the MIT dataset directly with our own pinned oracles
|
|
and benchmark kernel, adopting the attack taxonomy as protocol
|
|
definitions reimplemented in our stack.
|
|
|
|
## Verdict for the map item
|
|
|
|
An external W-Bench profile is buildable: MIT dataset, our kernel, our
|
|
oracles, no VINE code. Whether to build it is a scoping decision, not a
|
|
legal one - the image cohorts already cover detection/removal/fidelity
|
|
questions with pinned local evidence, and W-Bench would add its
|
|
editing-profile attack axis (regeneration, image-to-video). The ETI profile
|
|
stays closed until the organizers grant a license; nothing in this
|
|
repository should download or derive from it meanwhile.
|
|
|
|
One protocol detail is worth keeping as a candidate, not a commitment: the
|
|
ETI scoring program runs submissions through median blur and JPEG
|
|
compression before decoding, so an attack only wins if the watermark stays
|
|
dead under ordinary handling - that filter separates fragile
|
|
adversarial-perturbation removals from structural ones. Our removal paths
|
|
are structural (fill, regeneration), so no driving question exists today;
|
|
if removal durability under platform re-encoding ever becomes one, the
|
|
follow-up is an hour-sized handling arm over removed-state artifacts in the
|
|
existing cohorts. Participant attack code was never published (the release
|
|
carries attacked images and metrics only), so there is no implementation to
|
|
learn from even where reading would be lawful.
|