Record the ETI and W-Bench licensing verdict

Verified against primary sources: the ETI Competition Data (NeurIPS 2024)
carries no license on the dataset card or in the evaluation repository and
stays off limits without organizer permission; the W-Bench dataset (ICLR
2025) is MIT while the VINE code is NTUITIVE non-commercial, so a profile
is buildable on our kernel without the VINE code. Also corrects two stale
internal notes: ETI attacked outputs are in the release, and W-Bench holds
10,000 instances, not 15k.
This commit is contained in:
Victor Kuznetsov committed 2026-09-07 18:23:18 -07:00
1 parent 464fd142ea
commit 98d3df9f5a
3 files changed
+68

No files matched your search

+59
View File
@@ -0,0 +1,59 @@
# External watermark benchmarks: ETI and W-Bench licensing verdict
Investigation completed 2026-09-07 against the primary sources, closing the
last open item of the original research map: whether an honest external
benchmark profile can be built on ETI or W-Bench. The verdict is split:
W-Bench is legally usable, ETI is license-blocked.
## ETI (NeurIPS 2024 "Erasing the Invisible")
Sources verified: the Hugging Face dataset
`furonghuang-lab/ETI_Competition_Data` at revision `d80024580b5a` and the
organizers' evaluation repository `erasinginvisible/eval-program`.
- **License: absent everywhere.** The dataset carries no license tag and no
license section in its card; the evaluation repository has no LICENSE file.
With no grant of rights, the corpus is unusable in this public repository,
and downloading it for local evaluation builds results that cannot be
published. The only unblock is written permission from the organizers -
an external action that needs an explicit decision, not a default.
- **Composition (verified from the card):** 300 Beige-track images
(Gaussian Shading on Stable Diffusion 2.1, StegaStamp on Flux.1) and 300
Black-track images (JigMark, PRC, StableSignature, Trufo, plus
Gaussian-Shading+JigMark and StableSignature+StegaStamp doubles), plus
~1,072 Beige and ~1,650 Black valid submissions with per-watermark
detection results, image-quality metrics, and final scores.
- One earlier blocker is now stale: the attacked submission images ARE part
of the release per the card note, not missing. The license blocker alone
remains decisive.
## W-Bench / VINE (ICLR 2025)
Sources verified: the Hugging Face dataset `Shilin-LU/W-Bench` at revision
`c81f3924f0fb` and the VINE repository `Shilin-LU/VINE`.
- **Dataset license: MIT**, declared on the dataset card. The benchmark is
legally usable.
- **Composition (verified from the card):** 10,000 instances sourced from
COCO, Flickr, ShareGPT4V and similar datasets, eleven watermarking
methods, evaluated across four editing profiles - regeneration, global
editing, local editing, image-to-video - plus image distortions. An
earlier internal note said "15k images"; the card says 10,000, and the
card wins.
- **Code license: NTUITIVE Non-Commercial** (NTU technology transfer;
commercial use requires contacting the authors). The VINE code cannot be
vendored into this MIT-licensed project, and pipeline outputs derived
from running it would inherit that restriction. What remains lawful and
useful: consuming the MIT dataset directly with our own pinned oracles
and benchmark kernel, adopting the attack taxonomy as protocol
definitions reimplemented in our stack.
## Verdict for the map item
An external W-Bench profile is buildable: MIT dataset, our kernel, our
oracles, no VINE code. Whether to build it is a scoping decision, not a
legal one - the image cohorts already cover detection/removal/fidelity
questions with pinned local evidence, and W-Bench would add its
editing-profile attack axis (regeneration, image-to-video). The ETI profile
stays closed until the organizers grant a license; nothing in this
repository should download or derive from it meanwhile.
+1
View File
@@ -54,4 +54,5 @@ The current behavior is defined by the code, tests, README, and user guides.
- [VideoSeal temporal evaluation](videoseal-temporal-evaluation.md) (per-frame scores, aggregation matrix, and crf decay on synthetic and real clips)
- [Watermark forgery study](watermark-forgery-study.md) (four-state model and double-embedding: the last writer wins under both oracles)
- [C2PA resolution research](c2pa-resolution-research.md) (measured resolver contracts, the privacy gradient, and the decision to never resolve over the network)
- [External benchmark licensing](external-benchmark-licensing.md) (ETI license-blocked, W-Bench dataset MIT with non-commercial code, verified 2026-09-07)
- [Chroma1 engine research](chroma1-engine-research.md)
+8
View File
@@ -458,6 +458,14 @@ the vendor someone is inspecting the specific content, which is a leak in
the opposite direction of a removal tool's purpose. A registry guard test
pins the published-API set so ecosystem changes surface consciously.
The external-benchmark item is closed as a licensing verdict:
[the investigation](external-benchmark-licensing.md) verified ETI and
W-Bench against their primary sources - ETI carries no license anywhere and
stays off limits without organizer permission, while the W-Bench dataset is
MIT with the VINE code non-commercial. A W-Bench profile is therefore
buildable on our kernel; whether it is worth building is a scoping decision
that now has no legal dependency.
The same full-clip gate runs the public metadata-only path against its real MP4
and verifies unchanged file size, decoded frames, stream properties, and AAC
packets. A separate synthetic large-`mdat` test rejects full-source