From 98d3df9f5aeb8f804a522908b473e044d94a112f Mon Sep 17 00:00:00 2001 From: Victor Kuznetsov Date: Mon, 7 Sep 2026 16:55:13 -0700 Subject: [PATCH] Record the ETI and W-Bench licensing verdict Verified against primary sources: the ETI Competition Data (NeurIPS 2024) carries no license on the dataset card or in the evaluation repository and stays off limits without organizer permission; the W-Bench dataset (ICLR 2025) is MIT while the VINE code is NTUITIVE non-commercial, so a profile is buildable on our kernel without the VINE code. Also corrects two stale internal notes: ETI attacked outputs are in the release, and W-Bench holds 10,000 instances, not 15k. --- docs/external-benchmark-licensing.md | 59 ++++++++++++++++++++++++++++ docs/index.md | 1 + docs/verification-plan.md | 8 ++++ 3 files changed, 68 insertions(+) create mode 100644 docs/external-benchmark-licensing.md diff --git a/docs/external-benchmark-licensing.md b/docs/external-benchmark-licensing.md new file mode 100644 index 0000000..213ef40 --- /dev/null +++ b/docs/external-benchmark-licensing.md @@ -0,0 +1,59 @@ +# External watermark benchmarks: ETI and W-Bench licensing verdict + +Investigation completed 2026-09-07 against the primary sources, closing the +last open item of the original research map: whether an honest external +benchmark profile can be built on ETI or W-Bench. The verdict is split: +W-Bench is legally usable, ETI is license-blocked. + +## ETI (NeurIPS 2024 "Erasing the Invisible") + +Sources verified: the Hugging Face dataset +`furonghuang-lab/ETI_Competition_Data` at revision `d80024580b5a` and the +organizers' evaluation repository `erasinginvisible/eval-program`. + +- **License: absent everywhere.** The dataset carries no license tag and no + license section in its card; the evaluation repository has no LICENSE file. + With no grant of rights, the corpus is unusable in this public repository, + and downloading it for local evaluation builds results that cannot be + published. The only unblock is written permission from the organizers - + an external action that needs an explicit decision, not a default. +- **Composition (verified from the card):** 300 Beige-track images + (Gaussian Shading on Stable Diffusion 2.1, StegaStamp on Flux.1) and 300 + Black-track images (JigMark, PRC, StableSignature, Trufo, plus + Gaussian-Shading+JigMark and StableSignature+StegaStamp doubles), plus + ~1,072 Beige and ~1,650 Black valid submissions with per-watermark + detection results, image-quality metrics, and final scores. +- One earlier blocker is now stale: the attacked submission images ARE part + of the release per the card note, not missing. The license blocker alone + remains decisive. + +## W-Bench / VINE (ICLR 2025) + +Sources verified: the Hugging Face dataset `Shilin-LU/W-Bench` at revision +`c81f3924f0fb` and the VINE repository `Shilin-LU/VINE`. + +- **Dataset license: MIT**, declared on the dataset card. The benchmark is + legally usable. +- **Composition (verified from the card):** 10,000 instances sourced from + COCO, Flickr, ShareGPT4V and similar datasets, eleven watermarking + methods, evaluated across four editing profiles - regeneration, global + editing, local editing, image-to-video - plus image distortions. An + earlier internal note said "15k images"; the card says 10,000, and the + card wins. +- **Code license: NTUITIVE Non-Commercial** (NTU technology transfer; + commercial use requires contacting the authors). The VINE code cannot be + vendored into this MIT-licensed project, and pipeline outputs derived + from running it would inherit that restriction. What remains lawful and + useful: consuming the MIT dataset directly with our own pinned oracles + and benchmark kernel, adopting the attack taxonomy as protocol + definitions reimplemented in our stack. + +## Verdict for the map item + +An external W-Bench profile is buildable: MIT dataset, our kernel, our +oracles, no VINE code. Whether to build it is a scoping decision, not a +legal one - the image cohorts already cover detection/removal/fidelity +questions with pinned local evidence, and W-Bench would add its +editing-profile attack axis (regeneration, image-to-video). The ETI profile +stays closed until the organizers grant a license; nothing in this +repository should download or derive from it meanwhile. diff --git a/docs/index.md b/docs/index.md index cbdcc38..ba67e6a 100644 --- a/docs/index.md +++ b/docs/index.md @@ -54,4 +54,5 @@ The current behavior is defined by the code, tests, README, and user guides. - [VideoSeal temporal evaluation](videoseal-temporal-evaluation.md) (per-frame scores, aggregation matrix, and crf decay on synthetic and real clips) - [Watermark forgery study](watermark-forgery-study.md) (four-state model and double-embedding: the last writer wins under both oracles) - [C2PA resolution research](c2pa-resolution-research.md) (measured resolver contracts, the privacy gradient, and the decision to never resolve over the network) +- [External benchmark licensing](external-benchmark-licensing.md) (ETI license-blocked, W-Bench dataset MIT with non-commercial code, verified 2026-09-07) - [Chroma1 engine research](chroma1-engine-research.md) diff --git a/docs/verification-plan.md b/docs/verification-plan.md index ac9f540..107e80c 100644 --- a/docs/verification-plan.md +++ b/docs/verification-plan.md @@ -458,6 +458,14 @@ the vendor someone is inspecting the specific content, which is a leak in the opposite direction of a removal tool's purpose. A registry guard test pins the published-API set so ecosystem changes surface consciously. +The external-benchmark item is closed as a licensing verdict: +[the investigation](external-benchmark-licensing.md) verified ETI and +W-Bench against their primary sources - ETI carries no license anywhere and +stays off limits without organizer permission, while the W-Bench dataset is +MIT with the VINE code non-commercial. A W-Bench profile is therefore +buildable on our kernel; whether it is worth building is a scoping decision +that now has no legal dependency. + The same full-clip gate runs the public metadata-only path against its real MP4 and verifies unchanged file size, decoded frames, stream properties, and AAC packets. A separate synthetic large-`mdat` test rejects full-source