diff --git a/.gitignore b/.gitignore index ecf0436..f66ea2b 100644 --- a/.gitignore +++ b/.gitignore @@ -21,6 +21,9 @@ Thumbs.db # Test results data/results/ +# SynthID corpus images (manifest.csv + README.md stay tracked) +data/synthid_corpus/images/ + # Reference materials _refs/ @@ -30,3 +33,4 @@ yolov8n.pt # Claude Code local settings .claude/settings.local.json +data/synthid_corpus/refs/ diff --git a/CLAUDE.md b/CLAUDE.md index 59aa8b2..da020ec 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -11,6 +11,10 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r ## Test and lint - `bash maintain.sh` — uv-outdated, uv-secure, ruff check/fix, ruff format, pyright, pytest -n auto +- `maintain.sh` does not currently finish green (pre-existing, not per-change): `uv-secure` aborts on a fixable transitive `idna` vuln, and strict pyright carries debt in `remove_ai_metadata` / `cli.py` (untyped piexif/PIL/click/rich). To gate a change, run `uv run ruff check`, `uv run pyright `, `uv run pytest` directly. +- Run `uv run` from the repo root — from another cwd it falls back to a bare env without numpy/cv2/torch. +- Metadata/C2PA tests assert against real committed fixtures in `data/samples/` (`chatgpt-*.png` = OpenAI C2PA, `firefly-1.png` = Adobe, `not-ai-*` = clean); synthetic byte blobs cover the JPEG/ISOBMFF format paths. +- SynthID reference corpus: `scripts/synthid_corpus.py` ingests labeled images into `data/synthid_corpus/` (`manifest.csv` tracked, `images/` gitignored); see its README for the collection protocol and verification oracles. ## Configuration @@ -21,14 +25,14 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r - `noai/c2pa.py` — PNG chunk parser; use `extract_c2pa_chunk(path)` to get raw caBX payload, `has_c2pa_metadata(path)` to detect. Do not reimplement chunk parsing. `extract_c2pa_info(path)` sets `synthid_watermark`/`synthid_vendors` when the manifest is signed by a SynthID-using vendor. - `noai/constants.py` — PNG_SIGNATURE, C2PA_CHUNK_TYPE, C2PA_SIGNATURES, C2PA_ISSUERS, and `SYNTHID_C2PA_ISSUERS` (issuers that pair SynthID with C2PA: Google, OpenAI). Add a new issuer here, not inline. -- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None; `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. +- `metadata.py` — `synthid_source(path)` returns the vendor name(s) if the C2PA manifest implies a SynthID pixel watermark, else None. Format-agnostic: PNG via the caBX parser, JPEG/WebP/AVIF/HEIF/JXL via a binary scan (C2PA marker + SynthID issuer + AI-source marker). `get_ai_metadata` surfaces the verdict, and `metadata --check` prints it as a callout. - `face_protector.py` — YOLO detect + soft-blend pattern; mirror this for any "protect region during diffusion" features ## Known limitations - `invisible` pipeline downscales to model-native resolution (1024 px for SDXL) before diffusion. Degrades fine text in infographics. Tracked; fix is tile-based diffusion. -- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs) +- Pyright first run is slow (2-3 min) due to ML deps (torch/diffusers/transformers stubs); full-project `uv run pyright` can stall for many minutes — scope it to changed files. - `ultralytics` monkey-patches `PIL.Image.open` and tries to autoload `pi_heif`. When `pi_heif` is missing, opening files raises `ModuleNotFoundError`, not `UnidentifiedImageError`. Code that opens user-supplied or unknown-format files should `except Exception`, not just `OSError`/`UnidentifiedImageError`. - Metadata detection for AVIF/HEIF/JPEG-XL relies on a binary scan for `C2PA_UUID` + `IPTC_AI_MARKERS`. C2PA removal in those containers is implemented via `noai/isobmff.py` (top-level ``uuid`` / ``jumb`` box stripper, no re-encoding). EXIF/XMP boxes inside those containers are not yet scrubbed. -- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. -- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". The same configuration is used in raiw-app production (`fal-ai/fast-sdxl` at native ~1024 px, strength 0.05, steps 50). SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain. +- **SynthID detection is metadata-only.** There is no reliable *local* detector of the SynthID *pixel* watermark — Google's decoder is proprietary, no public spec or API (only a waitlisted portal). We detect SynthID by its C2PA companion (`synthid_source` / `SYNTHID_C2PA_ISSUERS`), which is reliable while the manifest is intact but says nothing once C2PA is stripped. Google→SynthID is long-standing; OpenAI→SynthID is confirmed by OpenAI's Help Center (ChatGPT/Codex/API "include both C2PA metadata and SynthID watermarks", updated 2026-05-21) but time-gated (pre-rollout OpenAI images carry C2PA without SynthID), so the OpenAI verdict is hedged "likely". Oracles: Gemini app "Verify with SynthID" (Google), openai.com/verify (OpenAI). The spectral phase-coherence approach from `github.com/aloshdenny/reverse-SynthID` was evaluated (May 2026) and **does not work for real-content detection**: on its own shipped codebook + validation set, watermarked and cleaned images were indistinguishable (conf within noise, cleaned often higher); it only fires on pure-black 1024x1024 reference images at exact resolution (the controlled case it was calibrated on). The README's "90% / conf=0.91" reproduces only in that lab condition. Do not build a production detector on it; if revisited, it is experimental/diagnostic only and needs a per-resolution, per-model reference corpus. A from-scratch gpt-image pilot (2026-05-24) confirmed this independently: 5 independent solid-black gpt-image outputs share a near-identical fixed signature (pairwise residual correlation **0.92**, avg-template retains 97% energy), so the watermark/carrier IS strongly present and consistent on flat content — but the carrier frequencies extracted from it do NOT discriminate real content (carrier-to-random ratio: cleaned 1.86 > watermarked 1.53; a non-gpt-image image scored highest at 3.67). The signature drowns in content texture. Net: a perfectly consistent solid-color signature still yields no real-content pixel detector with magnitude/carrier methods. +- **SynthID v2 vs default pipeline:** the SDXL-based default profile (since May 2026) defeats SynthID v2. **Verified end-to-end (May 2026):** local SDXL run on a Gemini 3 Pro output, checked via the Gemini app's "Verify with SynthID" feature, returned "no SynthID watermark detected". Also confirmed against **OpenAI's** SynthID (2026-05-23): a fresh ChatGPT/gpt-image output read "SynthID detected" on openai.com/verify before the local SDXL run and "SynthID not detected" after (corpus regression chain: pos `4ef377bd` -> cleaned `47188e88`). The same configuration is used in raiw-app production (`fal-ai/fast-sdxl` at native ~1024 px, strength 0.05, steps 50). SD-1.5 dreamshaper at 768 px was previously the default and does NOT defeat v2 — verified empirically against the same feature (strength 0.04, 0.10, and elastic warp α∈{5,8} all flagged positive). That SD-1.5 path was removed; only `default` (SDXL) and `ctrlregen` profiles remain. diff --git a/README.md b/README.md index 4888fca..647998b 100644 --- a/README.md +++ b/README.md @@ -187,6 +187,10 @@ remove-ai-watermarks visible image.png -o clean.png remove-ai-watermarks invisible image.png -o clean.png --humanize 4.0 # Check / strip AI metadata (C2PA, EXIF, "Made with AI" labels) +# --check also flags SynthID-bearing sources: a C2PA manifest signed by +# Google or OpenAI implies an invisible SynthID watermark in the pixels +# (both vendors pair the two). Adobe Firefly / Microsoft sign C2PA without +# SynthID, so they are reported as C2PA only. remove-ai-watermarks metadata image.png --check remove-ai-watermarks metadata image.png --remove @@ -253,7 +257,7 @@ pip install certifi Tracked but not yet implemented: -- **SynthID-Image v2 automated regression test**. The default SDXL profile defeats v2 per manual checks against the [Gemini app](https://support.google.com/gemini/answer/16722517)'s "Verify with SynthID" feature on a Gemini 3 Pro output (May 2026). An automated end-to-end test would need either programmatic access to the [SynthID Detector portal](https://blog.google/innovation-and-ai/products/google-synthid-ai-content-detector/) (waitlist for media professionals and researchers) or an offline surrogate detector. Open. +- **SynthID-Image v2 automated regression test**. The default SDXL profile defeats v2 per manual checks against the [Gemini app](https://support.google.com/gemini/answer/16722517)'s "Verify with SynthID" feature on a Gemini 3 Pro output (May 2026). An automated end-to-end test would need either programmatic access to the [SynthID Detector portal](https://blog.google/innovation-and-ai/products/google-synthid-ai-content-detector/) (waitlist for media professionals and researchers) or an offline surrogate detector. The spectral phase-coherence surrogate from [reverse-SynthID](https://github.com/aloshdenny/reverse-SynthID) was evaluated and does not separate watermarked from cleaned real-content images (it only fires on controlled solid-color references at exact resolution), so it is not a usable oracle. Open. - **AVIF / HEIF / JPEG-XL detection limits**. Removal strips top-level C2PA `uuid` and JUMBF `jumb` boxes. EXIF/XMP boxes inside these containers are not yet scrubbed (PNG and JPEG are fully covered). - **Video pipeline (`noai-video`)**: per-frame inpainting and tracking for Sora 2 dynamic logo, Veo 3.1 badge, Kling, Runway. Separate package, not folded into this repo. diff --git a/data/synthid_corpus/README.md b/data/synthid_corpus/README.md new file mode 100644 index 0000000..95124ab --- /dev/null +++ b/data/synthid_corpus/README.md @@ -0,0 +1,75 @@ +# SynthID reference corpus + +A locally-collected, labeled image corpus for SynthID work. Two downstream uses: + +1. **Per-resolution spectral codebook** for an experimental SynthID detector + (carrier frequencies are resolution-dependent, so labels must record the + exact native resolution). +2. **Removal regression set** — verify that our pipeline turns a SynthID-positive + image into a negative one. + +There is no reliable local detector of the SynthID pixel watermark (Google's +decoder is proprietary). The ground-truth label therefore comes from an +external oracle, recorded per image in `verified_via` (see below). + +## Layout + +``` +data/synthid_corpus/ + README.md # this protocol (committed) + manifest.csv # labels + provenance (committed, reviewable) + images/ # the actual files (gitignored, local-only) + pos/ # SynthID present + neg/ # SynthID absent + cleaned/ # our pipeline output from a pos image +``` + +Images are gitignored on purpose: the corpus is large, may contain personal or +licensed content, and SynthID-positive outputs are best kept local. The +`manifest.csv` (sha256 + labels + extracted metadata) is the durable artifact. + +## Verification levels (`verified_via`) + +Ground-truth quality, strongest first: + +- `gemini-app` — checked via the Gemini app "Verify with SynthID" feature. Gold standard for the pixel watermark (Google models). +- `openai-verify` — checked via openai.com/verify (gold standard for OpenAI ChatGPT/Codex/API images). +- `synthid-portal` — checked via Google's SynthID Detector portal. +- `c2pa-metadata` — issuer-only proxy (Google/OpenAI C2PA manifest present). Weaker: the C2PA can be stripped while the pixel watermark remains. +- `third-party` — label asserted by an external dataset, not independently verified. +- `none` — unverified. + +Prefer `gemini-app` for any image that will train the codebook or gate a test. + +## What to collect + +For the **codebook** (per target resolution, e.g. 1024x1024, 1024x1536, 1536x2816): + +- 30-50+ SynthID-positive outputs per resolution (more is better; ~150-200 per + resolution materially improves carrier discovery). +- At each target resolution, also a batch of **pure-black (#000000)** and + **pure-white (#FFFFFF)** fills generated by the SynthID model — these isolate + the content-independent carrier (the watermark is most of the signal there). + +For the **regression set**: + +- A handful of `pos` images, their `cleaned` counterparts (run through our + pipeline), and the cleaned re-verified via `gemini-app` (should read negative). +- `neg` controls: non-AI photos and outputs from non-SynthID models (SD, + Midjourney, Firefly) verified negative. + +Avoid personal or identifiable content; the corpus stays local. + +## Ingesting + +Use `scripts/synthid_corpus.py` — it copies a file in, records its sha256, +resolution, format, and C2PA issuer (via our own detector), and appends a row +to `manifest.csv`: + +```bash +uv run python scripts/synthid_corpus.py ingest path/to/*.png \ + --label pos --source "Gemini app" --model gemini-3-pro \ + --verified-via gemini-app --notes "1024x1024 batch" + +uv run python scripts/synthid_corpus.py status # counts by label / resolution / verification +``` diff --git a/data/synthid_corpus/manifest.csv b/data/synthid_corpus/manifest.csv new file mode 100644 index 0000000..576df5d --- /dev/null +++ b/data/synthid_corpus/manifest.csv @@ -0,0 +1,4 @@ +sha256,filename,label,source,model,width,height,format,c2pa_issuer,synthid_metadata,verified_via,added,notes +4ef377bde1a1d4eff141972841938643b173f5052992a018b9a21b31ac31731e,"4ef377bd-ChatGPT Image May 23, 2026, 02_43_02 PM.png",pos,ChatGPT,gpt-image,1254,1254,png,OpenAI,yes,openai-verify,2026-05-23T21:48:12Z,fresh post-rollout 2026-05-23; openai.com/verify: SynthID+C2PA detected +d09f84c0e4c6d8b336bf4a9a7277314e940dcb5052ae7051e785cbb3bb42d656,d09f84c0-Gemini_Generated_Image_vq7wkwvq7wkwvq7w.png,pos,Gemini app,gemini,2816,1536,png,Google LLC,yes,c2pa-metadata,2026-05-23T21:52:40Z,"user: latest Gemini, SynthID v2" +47188e88f956291bd38ab6906e5f21eb273d4a697ddc8b4479deac9f48915e1a,47188e88-disco_synthid_removed.png,cleaned,our pipeline (invisible/SDXL),stabilityai/stable-diffusion-xl-base-1.0,1254,1254,png,,,openai-verify,2026-05-23T22:06:54Z,cleaned from 4ef377bd disco; openai.com/verify: SynthID NOT detected (defeated) diff --git a/scripts/synthid_corpus.py b/scripts/synthid_corpus.py new file mode 100644 index 0000000..6a81f0a --- /dev/null +++ b/scripts/synthid_corpus.py @@ -0,0 +1,211 @@ +"""Ingest and inspect the local SynthID reference corpus. + +Copies images into ``data/synthid_corpus/images/