mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-10-14 13:30:55 +02:00
Record the ETI and W-Bench licensing verdict
Verified against primary sources: the ETI Competition Data (NeurIPS 2024) carries no license on the dataset card or in the evaluation repository and stays off limits without organizer permission; the W-Bench dataset (ICLR 2025) is MIT while the VINE code is NTUITIVE non-commercial, so a profile is buildable on our kernel without the VINE code. Also corrects two stale internal notes: ETI attacked outputs are in the release, and W-Bench holds 10,000 instances, not 15k.
This commit is contained in:
3 files changed
+68
No files matched your search
@@ -0,0 +1,59 @@
|
||||
# External watermark benchmarks: ETI and W-Bench licensing verdict
|
||||
|
||||
Investigation completed 2026-09-07 against the primary sources, closing the
|
||||
last open item of the original research map: whether an honest external
|
||||
benchmark profile can be built on ETI or W-Bench. The verdict is split:
|
||||
W-Bench is legally usable, ETI is license-blocked.
|
||||
|
||||
## ETI (NeurIPS 2024 "Erasing the Invisible")
|
||||
|
||||
Sources verified: the Hugging Face dataset
|
||||
`furonghuang-lab/ETI_Competition_Data` at revision `d80024580b5a` and the
|
||||
organizers' evaluation repository `erasinginvisible/eval-program`.
|
||||
|
||||
- **License: absent everywhere.** The dataset carries no license tag and no
|
||||
license section in its card; the evaluation repository has no LICENSE file.
|
||||
With no grant of rights, the corpus is unusable in this public repository,
|
||||
and downloading it for local evaluation builds results that cannot be
|
||||
published. The only unblock is written permission from the organizers -
|
||||
an external action that needs an explicit decision, not a default.
|
||||
- **Composition (verified from the card):** 300 Beige-track images
|
||||
(Gaussian Shading on Stable Diffusion 2.1, StegaStamp on Flux.1) and 300
|
||||
Black-track images (JigMark, PRC, StableSignature, Trufo, plus
|
||||
Gaussian-Shading+JigMark and StableSignature+StegaStamp doubles), plus
|
||||
~1,072 Beige and ~1,650 Black valid submissions with per-watermark
|
||||
detection results, image-quality metrics, and final scores.
|
||||
- One earlier blocker is now stale: the attacked submission images ARE part
|
||||
of the release per the card note, not missing. The license blocker alone
|
||||
remains decisive.
|
||||
|
||||
## W-Bench / VINE (ICLR 2025)
|
||||
|
||||
Sources verified: the Hugging Face dataset `Shilin-LU/W-Bench` at revision
|
||||
`c81f3924f0fb` and the VINE repository `Shilin-LU/VINE`.
|
||||
|
||||
- **Dataset license: MIT**, declared on the dataset card. The benchmark is
|
||||
legally usable.
|
||||
- **Composition (verified from the card):** 10,000 instances sourced from
|
||||
COCO, Flickr, ShareGPT4V and similar datasets, eleven watermarking
|
||||
methods, evaluated across four editing profiles - regeneration, global
|
||||
editing, local editing, image-to-video - plus image distortions. An
|
||||
earlier internal note said "15k images"; the card says 10,000, and the
|
||||
card wins.
|
||||
- **Code license: NTUITIVE Non-Commercial** (NTU technology transfer;
|
||||
commercial use requires contacting the authors). The VINE code cannot be
|
||||
vendored into this MIT-licensed project, and pipeline outputs derived
|
||||
from running it would inherit that restriction. What remains lawful and
|
||||
useful: consuming the MIT dataset directly with our own pinned oracles
|
||||
and benchmark kernel, adopting the attack taxonomy as protocol
|
||||
definitions reimplemented in our stack.
|
||||
|
||||
## Verdict for the map item
|
||||
|
||||
An external W-Bench profile is buildable: MIT dataset, our kernel, our
|
||||
oracles, no VINE code. Whether to build it is a scoping decision, not a
|
||||
legal one - the image cohorts already cover detection/removal/fidelity
|
||||
questions with pinned local evidence, and W-Bench would add its
|
||||
editing-profile attack axis (regeneration, image-to-video). The ETI profile
|
||||
stays closed until the organizers grant a license; nothing in this
|
||||
repository should download or derive from it meanwhile.
|
||||
@@ -54,4 +54,5 @@ The current behavior is defined by the code, tests, README, and user guides.
|
||||
- [VideoSeal temporal evaluation](videoseal-temporal-evaluation.md) (per-frame scores, aggregation matrix, and crf decay on synthetic and real clips)
|
||||
- [Watermark forgery study](watermark-forgery-study.md) (four-state model and double-embedding: the last writer wins under both oracles)
|
||||
- [C2PA resolution research](c2pa-resolution-research.md) (measured resolver contracts, the privacy gradient, and the decision to never resolve over the network)
|
||||
- [External benchmark licensing](external-benchmark-licensing.md) (ETI license-blocked, W-Bench dataset MIT with non-commercial code, verified 2026-09-07)
|
||||
- [Chroma1 engine research](chroma1-engine-research.md)
|
||||
@@ -458,6 +458,14 @@ the vendor someone is inspecting the specific content, which is a leak in
|
||||
the opposite direction of a removal tool's purpose. A registry guard test
|
||||
pins the published-API set so ecosystem changes surface consciously.
|
||||
|
||||
The external-benchmark item is closed as a licensing verdict:
|
||||
[the investigation](external-benchmark-licensing.md) verified ETI and
|
||||
W-Bench against their primary sources - ETI carries no license anywhere and
|
||||
stays off limits without organizer permission, while the W-Bench dataset is
|
||||
MIT with the VINE code non-commercial. A W-Bench profile is therefore
|
||||
buildable on our kernel; whether it is worth building is a scoping decision
|
||||
that now has no legal dependency.
|
||||
|
||||
The same full-clip gate runs the public metadata-only path against its real MP4
|
||||
and verifies unchanged file size, decoded frames, stream properties, and AAC
|
||||
packets. A separate synthetic large-`mdat` test rejects full-source
|
||||
|
||||
Reference in new issue
Block a user