Restructure documentation, validate metadata removal, consolidate assets

2026-08-07 06:28:36 +02:00 · 2026-07-25 21:08:04 -07:00
parent 214c9bb3e7
commit 03cd00f132
172 changed files with 2356 additions and 2863 deletions
@@ -0,0 +1,164 @@
+# SynthID oracle fixtures
+
+A compact set of externally verified originals for two uses:
+
+1. **Signal and detector-limit research** with exact native resolutions and
+   recorded verification sources.
+2. **Removal regression set**: verify that a pipeline turns a SynthID-positive
+   image into a negative one.
+
+There is no reliable local detector of the SynthID pixel watermark (Google's
+decoder is proprietary). The ground-truth label therefore comes from an
+external oracle, recorded per image in `verified_via` (see below).
+
+## Layout
+
+```
+data/synthid/
+  README.md                  # collection and verification protocol
+  manifest.csv               # one row per committed original
+  full-pipeline-quality.csv  # reusable evaluation selection
+  originals/                 # one canonical copy of each verified source
+```
+
+The originals are committed so the evaluation set is reproducible and its
+hashes can be checked in CI. `manifest.csv` is kept in sync with
+`originals/`, one row per file. Generated outputs do not belong here. Store
+them outside the repository and record only a reproducible verdict in the
+evaluation manifest or documentation.
+
+## Reusable removal-quality set
+
+`full-pipeline-quality.csv` is the canonical reusable input set for
+full-pipeline visual-quality and watermark-removal tests. Its fixtures are
+cleared for permanent public test reuse. Tests, benchmarks, and manual
+evaluation bundles may read these images repeatedly without requesting new
+permission.
+
+The CSV preserves each platform's original filename and maps it to the one
+canonical corpus copy. Use `corpus_path` to read the input, but preserve
+`source_filename` in generated result names and keep OpenAI and Gemini outputs
+in separate provider groups. This avoids duplicating large PNGs while retaining
+the names needed for the corresponding provider oracle.
+
+Rows with a dated negative `final_clean_oracle` were checked after the complete
+`visible -> qwen-zimage -> metadata` route with the matching provider oracle.
+`not-checked-in-final-candidate` means the input is available as an additional
+quality stress fixture, but a newly generated output still needs the matching
+oracle before claiming removal. The checked result applies to those exact
+output bytes; re-run the oracle after changing the pipeline, seed, model
+versions, or runtime settings.
+
+## Verification levels (`verified_via`)
+
+Ground-truth quality, strongest first:
+
+- `gemini-app` — checked via the Gemini app "Verify with SynthID" feature. Gold standard for the pixel watermark (Google models).
+- `openai-verify` — checked via openai.com/verify (gold standard for OpenAI ChatGPT/Codex/API images).
+- `synthid-portal` — checked via Google's SynthID Detector portal.
+- `c2pa-metadata` — issuer-only proxy (Google/OpenAI C2PA manifest present). Weaker: the C2PA can be stripped while the pixel watermark remains.
+- `third-party` — label asserted by an external dataset, not independently verified.
+- `none` — unverified.
+
+Prefer `gemini-app` for any Google image that will gate a test.
+
+## What to collect
+
+For the **regression set**:
+
+- A small number of positive originals with a recorded provider oracle.
+- A quality-set row for every original used by repeatable evaluation.
+- Re-run the provider oracle on each newly generated output. Do not commit the
+  generated output as another corpus copy.
+
+The corpus is committed to a public repo: review every image before adding it
+and keep out anything private or identifiable you would not publish.
+
+## Ingesting
+
+Use `scripts/synthid_corpus.py` to copy a file into `originals/`, record its
+SHA-256, resolution, format, and C2PA issuer through our detector, and append a
+row to `manifest.csv`:
+
+```bash
+uv run python scripts/synthid_corpus.py ingest path/to/*.png \
+    --label pos --source "Gemini app" --model gemini-3-pro \
+    --verified-via gemini-app --notes "1024x1024 batch"
+
+uv run python scripts/synthid_corpus.py status   # counts by label / resolution / verification
+```
+
+## Autonomous collection via Chrome MCP
+
+> Historical collection notes. Browser interfaces and download behavior can
+> change independently of this repository. These steps do not define package
+> behavior.
+
+Generation can be driven through the browser (the account must be logged in):
+
+- **Gemini** (`gemini.google.com`): type `Create an image: <prompt>`, wait, hover the
+  result, click the download icon (top-right). Single, reliable click. Outputs
+  carry Google C2PA + SynthID. Occasionally the composer stalls in a
+  "generating" state -> start a New chat to reset.
+- **ChatGPT** (`chatgpt.com`): the UI download is flaky (the fullscreen viewer
+  races and can grab the previous image; the share-modal path works but is
+  multi-step). Reliable path is an in-page fetch of the rendered image, which
+  preserves the original bytes (C2PA intact, unlike a canvas re-encode):
+
+  ```js
+  // run in the ChatGPT tab via the browser MCP javascript tool
+  (async () => {
+    const imgs = [...document.querySelectorAll('img')].filter(i => i.naturalWidth >= 400);
+    const img = imgs[imgs.length - 1];                 // newest large image
+    const b = await (await fetch(img.currentSrc || img.src)).blob();
+    const a = document.createElement('a');
+    a.href = URL.createObjectURL(b); a.download = 'dl.png';
+    document.body.appendChild(a); a.click(); a.remove();
+    return 'size=' + b.size;                            // do NOT return the src (privacy guard blocks query strings)
+  })()
+  ```
+
+  Gotcha: confirm the returned `size` differs from the previous image before
+  ingesting -- if the new image has not finished rendering, the script grabs the
+  prior one (the corpus dedups by sha256, but the notes would mislabel it).
+  ChatGPT also shows an A/B "which is better?" picker; click Skip first.
+
+**Originals, not previews.** Some platforms render a low-res preview in the chat
+(Grok serves a ~20KB 1024px JPEG/PNG; the in-page `<img>` fetch grabs *that*, not
+the original). Previews are re-encoded and **strip metadata**, so a "clean"
+preview is not proof the original is clean. Always pull the original via the
+platform's native Download / lightbox button and sanity-check the file size (a
+20KB "1024x1024" is a preview). ChatGPT's in-chat `<img>` *is* the full-res
+oaiusercontent original, so fetch+blob is fine there; Grok needs the lightbox
+Download; Leonardo serves the original JPEG in-chat (download button matches).
+
+## Per-platform watermark map (observed, May 2026)
+
+This is a dated observation table, not a live provider compatibility promise.
+The detector's coverage is complementary: metadata catches C2PA /
+IPTC; `exif_generator` catches EXIF `Make`/`Software` + XMP `CreatorTool`;
+`invisible_watermark.py` (imwatermark) catches the open SD/SDXL/FLUX DWT-DCT
+watermark on pristine files; the visible detector catches the Gemini-family
+sparkle; the SynthID *pixel* itself has no local detector (oracle only).
+
+| Platform | C2PA issuer | SynthID pixel | IPTC "Made with AI" | Visible sparkle | imwatermark | Corpus label |
+|---|---|---|---|---|---|---|
+| Gemini app | Google | yes | - | yes | - | pos |
+| ChatGPT / gpt-image | OpenAI | yes | - | - | - | pos |
+| Microsoft Designer | OpenAI + Microsoft | yes (via OpenAI) | - | - | - | pos |
+| Bing Image Creator | Microsoft (MAI-Image) | no | - | - | - | pos (C2PA "Microsoft", not OpenAI) |
+| Google AI Studio (Nano Banana) | **none** | yes (oracle-confirmed) | - | yes | - | pos (metadata blind spot) |
+| Stability AI (Brand Studio) | Stability AI Ltd | no | - | - | no | pos (C2PA only) |
+| Ideogram | none | no | - | - | no | pos (EXIF `Make="Ideogram AI"` only) |
+| Meta AI | none | no | **yes** | - | - | neg (for SynthID) |
+| Leonardo.ai | none | no | no | - | no | neg |
+| Recraft | none (export strips) | no | no | - | no | neg (re-encoded export, no signal) |
+| Krea (FLUX 2 host) | none | no | no | - | no | neg (host omits the imwatermark encoder) |
+| Grok (xAI) | none (non-adopter) | no | no | - | no | neg (captured: clean low-res preview) |
+
+Key takeaways:
+- The same model differs by *surface*: Gemini app wraps C2PA, AI Studio (API/playground) emits none -- only the pixel + sparkle survive.
+- Microsoft Designer's DALL-E backend inherits OpenAI's C2PA+SynthID (issuer "OpenAI, Microsoft"); Bing now runs Microsoft's own **MAI-Image** and signs C2PA as "Microsoft" (not OpenAI/DALL-E).
+- Meta uses the IPTC `digitalSourceType` marker, not C2PA or SynthID.
+- The open imwatermark fires only on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL) -- not from re-hosts (Krea, Stability hosted SDXL) or re-encoded design exports (Recraft, Canva). Ideogram's only signal is the EXIF `Make` tag.
+- Bing and Grok web UIs are uncooperative for autonomous capture (no document_idle for screenshots; blob downloads intermittently no-op; low-res in-chat previews). Use their native download button manually if a full-res sample is needed.
@@ -0,0 +1,8 @@
+provider,source_filename,corpus_path,sha256,original_signal_verified_via,final_clean_oracle
+openai,"ChatGPT Image May 30, 2026, 10_31_08 AM.png","originals/ChatGPT Image May 30, 2026, 10_31_08 AM.png",28ff8732b037f98a4ef5bc277bbcdaa32e5eb9ccbd00b6c8c616e46ef68ae8a0,openai-verify,openai-verify-negative-2026-07-25
+openai,"ChatGPT Image May 31, 2026, 02_02_23 PM.png","originals/ChatGPT Image May 31, 2026, 02_02_23 PM.png",228e54413a3ef62b313176dcdba774a56a58c36b54227d1440bc26cbe0aefd82,c2pa-metadata,not-checked-in-final-candidate
+openai,"ChatGPT Image May 31, 2026, 02_03_55 PM.png","originals/ChatGPT Image May 31, 2026, 02_03_55 PM.png",05b836ecfe40fd689177fda74384ae4fdcc446505bbc4281cd3cbb6523eb669e,openai-verify,openai-verify-negative-2026-07-25
+gemini,"Gemini_Generated_Image_3mc4t93mc4t93mc4.png","originals/Gemini_Generated_Image_3mc4t93mc4t93mc4.png",2c33e75a2db614ce74c83cc0a6ac6c3ac735aca83ab88c9c9345843b124f7856,gemini-app,gemini-app-negative-2026-07-25
+gemini,"Gemini_Generated_Image_633uuy633uuy633u.png","originals/Gemini_Generated_Image_633uuy633uuy633u.png",4affd7f27767a445db6abf741355743ba8d95108ad922c9fff045feed8492236,gemini-app,gemini-app-negative-2026-07-25
+gemini,"Gemini_Generated_Image_akdbeiakdbeiakdb.png","originals/Gemini_Generated_Image_akdbeiakdbeiakdb.png",8c1a6fb03ef3d45a1f958fb3401e4264e409ff88c2a793061db7f29023454d0e,gemini-app,gemini-app-negative-2026-07-25
+gemini,"Gemini_Generated_Image_y48j3cy48j3cy48j.png","originals/Gemini_Generated_Image_y48j3cy48j3cy48j.png",45d79a683134fcba1b147b2aedb669783d474e1fb8a4df329729a0904fd1b46b,gemini-app,gemini-app-negative-2026-07-25
@@ -0,0 +1,8 @@
+sha256,filename,label,source,model,width,height,format,c2pa_issuer,synthid_metadata,verified_via,added,notes
+05b836ecfe40fd689177fda74384ae4fdcc446505bbc4281cd3cbb6523eb669e,"ChatGPT Image May 31, 2026, 02_03_55 PM.png",pos,ChatGPT,gpt-image,1122,1402,png,OpenAI,yes,openai-verify,2026-06-04T00:07:53Z,June 2026 strength-study subject; openai.com/verify: SynthID detected (docs/synthid.md 2.2)
+28ff8732b037f98a4ef5bc277bbcdaa32e5eb9ccbd00b6c8c616e46ef68ae8a0,"ChatGPT Image May 30, 2026, 10_31_08 AM.png",pos,ChatGPT,gpt-image,1448,1086,png,OpenAI,yes,openai-verify,2026-06-04T00:07:53Z,June 2026 strength-study subject; openai.com/verify: SynthID detected (docs/synthid.md 2.2)
+4affd7f27767a445db6abf741355743ba8d95108ad922c9fff045feed8492236,Gemini_Generated_Image_633uuy633uuy633u.png,pos,Gemini app,gemini,2816,1536,png,Google LLC,yes,gemini-app,2026-06-04T00:08:05Z,June 2026 strength-study subject; Gemini-app Verify with SynthID: detected (docs/synthid.md 2.2)
+8c1a6fb03ef3d45a1f958fb3401e4264e409ff88c2a793061db7f29023454d0e,Gemini_Generated_Image_akdbeiakdbeiakdb.png,pos,Gemini app,gemini,2816,1536,png,Google LLC,yes,gemini-app,2026-06-04T00:08:05Z,June 2026 strength-study subject; Gemini-app Verify with SynthID: detected (docs/synthid.md 2.2)
+45d79a683134fcba1b147b2aedb669783d474e1fb8a4df329729a0904fd1b46b,Gemini_Generated_Image_y48j3cy48j3cy48j.png,pos,Gemini app,gemini,2816,1536,png,Google LLC,yes,gemini-app,2026-06-04T00:08:05Z,June 2026 strength-study subject; Gemini-app Verify with SynthID: detected (docs/synthid.md 2.2)
+2c33e75a2db614ce74c83cc0a6ac6c3ac735aca83ab88c9c9345843b124f7856,Gemini_Generated_Image_3mc4t93mc4t93mc4.png,pos,Gemini app,gemini,2816,1536,png,Google LLC,yes,gemini-app,2026-06-04T00:08:05Z,June 2026 strength-study subject; Gemini-app Verify with SynthID: detected (docs/synthid.md 2.2)
+228e54413a3ef62b313176dcdba774a56a58c36b54227d1440bc26cbe0aefd82,"ChatGPT Image May 31, 2026, 02_02_23 PM.png",pos,ChatGPT,gpt-image,1122,1402,png,OpenAI,yes,c2pa-metadata,2026-07-25T20:21:33Z,Reusable full-pipeline quality fixture; original filename preserved in quality-set manifest; cleared for public test reuse