mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-07 22:48:37 +02:00
Restructure documentation, validate metadata removal, consolidate assets
This commit is contained in:
Victor Kuznetsov
@@ -0,0 +1,164 @@
|
||||
# SynthID oracle fixtures
|
||||
|
||||
A compact set of externally verified originals for two uses:
|
||||
|
||||
1. **Signal and detector-limit research** with exact native resolutions and
|
||||
recorded verification sources.
|
||||
2. **Removal regression set**: verify that a pipeline turns a SynthID-positive
|
||||
image into a negative one.
|
||||
|
||||
There is no reliable local detector of the SynthID pixel watermark (Google's
|
||||
decoder is proprietary). The ground-truth label therefore comes from an
|
||||
external oracle, recorded per image in `verified_via` (see below).
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
data/synthid/
|
||||
README.md # collection and verification protocol
|
||||
manifest.csv # one row per committed original
|
||||
full-pipeline-quality.csv # reusable evaluation selection
|
||||
originals/ # one canonical copy of each verified source
|
||||
```
|
||||
|
||||
The originals are committed so the evaluation set is reproducible and its
|
||||
hashes can be checked in CI. `manifest.csv` is kept in sync with
|
||||
`originals/`, one row per file. Generated outputs do not belong here. Store
|
||||
them outside the repository and record only a reproducible verdict in the
|
||||
evaluation manifest or documentation.
|
||||
|
||||
## Reusable removal-quality set
|
||||
|
||||
`full-pipeline-quality.csv` is the canonical reusable input set for
|
||||
full-pipeline visual-quality and watermark-removal tests. Its fixtures are
|
||||
cleared for permanent public test reuse. Tests, benchmarks, and manual
|
||||
evaluation bundles may read these images repeatedly without requesting new
|
||||
permission.
|
||||
|
||||
The CSV preserves each platform's original filename and maps it to the one
|
||||
canonical corpus copy. Use `corpus_path` to read the input, but preserve
|
||||
`source_filename` in generated result names and keep OpenAI and Gemini outputs
|
||||
in separate provider groups. This avoids duplicating large PNGs while retaining
|
||||
the names needed for the corresponding provider oracle.
|
||||
|
||||
Rows with a dated negative `final_clean_oracle` were checked after the complete
|
||||
`visible -> qwen-zimage -> metadata` route with the matching provider oracle.
|
||||
`not-checked-in-final-candidate` means the input is available as an additional
|
||||
quality stress fixture, but a newly generated output still needs the matching
|
||||
oracle before claiming removal. The checked result applies to those exact
|
||||
output bytes; re-run the oracle after changing the pipeline, seed, model
|
||||
versions, or runtime settings.
|
||||
|
||||
## Verification levels (`verified_via`)
|
||||
|
||||
Ground-truth quality, strongest first:
|
||||
|
||||
- `gemini-app` — checked via the Gemini app "Verify with SynthID" feature. Gold standard for the pixel watermark (Google models).
|
||||
- `openai-verify` — checked via openai.com/verify (gold standard for OpenAI ChatGPT/Codex/API images).
|
||||
- `synthid-portal` — checked via Google's SynthID Detector portal.
|
||||
- `c2pa-metadata` — issuer-only proxy (Google/OpenAI C2PA manifest present). Weaker: the C2PA can be stripped while the pixel watermark remains.
|
||||
- `third-party` — label asserted by an external dataset, not independently verified.
|
||||
- `none` — unverified.
|
||||
|
||||
Prefer `gemini-app` for any Google image that will gate a test.
|
||||
|
||||
## What to collect
|
||||
|
||||
For the **regression set**:
|
||||
|
||||
- A small number of positive originals with a recorded provider oracle.
|
||||
- A quality-set row for every original used by repeatable evaluation.
|
||||
- Re-run the provider oracle on each newly generated output. Do not commit the
|
||||
generated output as another corpus copy.
|
||||
|
||||
The corpus is committed to a public repo: review every image before adding it
|
||||
and keep out anything private or identifiable you would not publish.
|
||||
|
||||
## Ingesting
|
||||
|
||||
Use `scripts/synthid_corpus.py` to copy a file into `originals/`, record its
|
||||
SHA-256, resolution, format, and C2PA issuer through our detector, and append a
|
||||
row to `manifest.csv`:
|
||||
|
||||
```bash
|
||||
uv run python scripts/synthid_corpus.py ingest path/to/*.png \
|
||||
--label pos --source "Gemini app" --model gemini-3-pro \
|
||||
--verified-via gemini-app --notes "1024x1024 batch"
|
||||
|
||||
uv run python scripts/synthid_corpus.py status # counts by label / resolution / verification
|
||||
```
|
||||
|
||||
## Autonomous collection via Chrome MCP
|
||||
|
||||
> Historical collection notes. Browser interfaces and download behavior can
|
||||
> change independently of this repository. These steps do not define package
|
||||
> behavior.
|
||||
|
||||
Generation can be driven through the browser (the account must be logged in):
|
||||
|
||||
- **Gemini** (`gemini.google.com`): type `Create an image: <prompt>`, wait, hover the
|
||||
result, click the download icon (top-right). Single, reliable click. Outputs
|
||||
carry Google C2PA + SynthID. Occasionally the composer stalls in a
|
||||
"generating" state -> start a New chat to reset.
|
||||
- **ChatGPT** (`chatgpt.com`): the UI download is flaky (the fullscreen viewer
|
||||
races and can grab the previous image; the share-modal path works but is
|
||||
multi-step). Reliable path is an in-page fetch of the rendered image, which
|
||||
preserves the original bytes (C2PA intact, unlike a canvas re-encode):
|
||||
|
||||
```js
|
||||
// run in the ChatGPT tab via the browser MCP javascript tool
|
||||
(async () => {
|
||||
const imgs = [...document.querySelectorAll('img')].filter(i => i.naturalWidth >= 400);
|
||||
const img = imgs[imgs.length - 1]; // newest large image
|
||||
const b = await (await fetch(img.currentSrc || img.src)).blob();
|
||||
const a = document.createElement('a');
|
||||
a.href = URL.createObjectURL(b); a.download = 'dl.png';
|
||||
document.body.appendChild(a); a.click(); a.remove();
|
||||
return 'size=' + b.size; // do NOT return the src (privacy guard blocks query strings)
|
||||
})()
|
||||
```
|
||||
|
||||
Gotcha: confirm the returned `size` differs from the previous image before
|
||||
ingesting -- if the new image has not finished rendering, the script grabs the
|
||||
prior one (the corpus dedups by sha256, but the notes would mislabel it).
|
||||
ChatGPT also shows an A/B "which is better?" picker; click Skip first.
|
||||
|
||||
**Originals, not previews.** Some platforms render a low-res preview in the chat
|
||||
(Grok serves a ~20KB 1024px JPEG/PNG; the in-page `<img>` fetch grabs *that*, not
|
||||
the original). Previews are re-encoded and **strip metadata**, so a "clean"
|
||||
preview is not proof the original is clean. Always pull the original via the
|
||||
platform's native Download / lightbox button and sanity-check the file size (a
|
||||
20KB "1024x1024" is a preview). ChatGPT's in-chat `<img>` *is* the full-res
|
||||
oaiusercontent original, so fetch+blob is fine there; Grok needs the lightbox
|
||||
Download; Leonardo serves the original JPEG in-chat (download button matches).
|
||||
|
||||
## Per-platform watermark map (observed, May 2026)
|
||||
|
||||
This is a dated observation table, not a live provider compatibility promise.
|
||||
The detector's coverage is complementary: metadata catches C2PA /
|
||||
IPTC; `exif_generator` catches EXIF `Make`/`Software` + XMP `CreatorTool`;
|
||||
`invisible_watermark.py` (imwatermark) catches the open SD/SDXL/FLUX DWT-DCT
|
||||
watermark on pristine files; the visible detector catches the Gemini-family
|
||||
sparkle; the SynthID *pixel* itself has no local detector (oracle only).
|
||||
|
||||
| Platform | C2PA issuer | SynthID pixel | IPTC "Made with AI" | Visible sparkle | imwatermark | Corpus label |
|
||||
|---|---|---|---|---|---|---|
|
||||
| Gemini app | Google | yes | - | yes | - | pos |
|
||||
| ChatGPT / gpt-image | OpenAI | yes | - | - | - | pos |
|
||||
| Microsoft Designer | OpenAI + Microsoft | yes (via OpenAI) | - | - | - | pos |
|
||||
| Bing Image Creator | Microsoft (MAI-Image) | no | - | - | - | pos (C2PA "Microsoft", not OpenAI) |
|
||||
| Google AI Studio (Nano Banana) | **none** | yes (oracle-confirmed) | - | yes | - | pos (metadata blind spot) |
|
||||
| Stability AI (Brand Studio) | Stability AI Ltd | no | - | - | no | pos (C2PA only) |
|
||||
| Ideogram | none | no | - | - | no | pos (EXIF `Make="Ideogram AI"` only) |
|
||||
| Meta AI | none | no | **yes** | - | - | neg (for SynthID) |
|
||||
| Leonardo.ai | none | no | no | - | no | neg |
|
||||
| Recraft | none (export strips) | no | no | - | no | neg (re-encoded export, no signal) |
|
||||
| Krea (FLUX 2 host) | none | no | no | - | no | neg (host omits the imwatermark encoder) |
|
||||
| Grok (xAI) | none (non-adopter) | no | no | - | no | neg (captured: clean low-res preview) |
|
||||
|
||||
Key takeaways:
|
||||
- The same model differs by *surface*: Gemini app wraps C2PA, AI Studio (API/playground) emits none -- only the pixel + sparkle survive.
|
||||
- Microsoft Designer's DALL-E backend inherits OpenAI's C2PA+SynthID (issuer "OpenAI, Microsoft"); Bing now runs Microsoft's own **MAI-Image** and signs C2PA as "Microsoft" (not OpenAI/DALL-E).
|
||||
- Meta uses the IPTC `digitalSourceType` marker, not C2PA or SynthID.
|
||||
- The open imwatermark fires only on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL) -- not from re-hosts (Krea, Stability hosted SDXL) or re-encoded design exports (Recraft, Canva). Ideogram's only signal is the EXIF `Make` tag.
|
||||
- Bing and Grok web UIs are uncooperative for autonomous capture (no document_idle for screenshots; blob downloads intermittently no-op; low-res in-chat previews). Use their native download button manually if a full-res sample is needed.
|
||||
Reference in New Issue
Block a user