CalvinBackup/remove-ai-watermarks
1
0 Fork 0
mirror of https://github.com/wiltodelta/remove-ai-watermarks.git synced 2026-10-21 00:11:14 +02:00
Code Issues Packages Projects Releases Wiki Activity
Files
d25cada76bc39bab59cd7a41c29028b1cc98a5f2
remove-ai-watermarks/data/synthid/README.md
T
Victor Kuznetsov 07b07634cf Merge the docs/arxiv-paper-review research line into main (#95) 
* Add rigorous SynthID research and evaluation harness

* Record SynthID cross-epoch validation results

* Reset SynthID detector hypotheses to paired training

* Add paired wavelet and spectral SynthID probes

* Record closed-corpus SynthID detector ablations

* Add registered SynthID phase probing

* Add periodic SynthID tile probing

* Add calibrated SynthID pixel detector

* Harden scale-registered SynthID detection

* Record SynthID phase and channel rescue findings

* Add production SynthID routing and OpenAI verification

* Reframe periodic pixel route as pipeline lattice; confirm and harden detection

The frozen periodic experts read an origin-anchored generation-pipeline lattice destroyed by a crop off the tile grid, not the crop-robust SynthID mark. Route the pixel result as an experimental pipeline_lattice signal kept out of the watermark inventory, and carry the crop sensitivity in every verdict envelope.

Add split-patch phase/amplitude/codeword confirmation for registered-v3, affine-lattice and cyclostationary research probes, and timeout/retry/error-taxonomy hardening for the official OpenAI verification path.

* Record pipeline-lattice reframing and registered-v3 phase lock

Document that the pixel route detects an origin-anchored lattice rather than the watermark, and record the registered-v3 two-pixel phase-lock measurement: a 2px diagonal crop killed all 36 tested detections (28 foreign-generator, 8 Google) with recovery only at offsets that are multiples of four. Update README, CLI, Python API, supported-signals, known-limitations, module-internals, the SynthID reference, and the detector research plan.

* Split SynthID campaign docs by detector, classifiers, and removal

Keep Model 1 as a research AI-versus-camera result outside the library.
Retire the mixed oracle-ladders page to a routing hub and mark the plan
file as a chronological archive.

* Keep 2026-08-22 SynthID campaign results in research docs only

Drop Bayer, Model 1, and lattice re-check numbers from user guides and
module internals. The library surface is unchanged.

* Unplug lattice detector from package, expand research docs

* Remove lattice detector sources from the public package

* Document SynthID GitHub sweep findings

* Document general AI classifier sweep

* Document modern-negative taxonomy audit and classifier doc split

Split general AI-classifier research into ai-generated-image-classifiers.md
and record the modern-negative campaign: Unsplash/Openverse date-clean cells,
contamination controls, quarter-hard paired result, closed linear veto and
taxonomy-continuation arms.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix (same triage as main); core checks separately green (ruff, format, pyright, 1564 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - split references updated everywhere, none stale; 4) CLAUDE.md - routing line already updated, compliant

* Record wild-AI audit, Meta provider class, and stock-negative expansion

The wild vendor-flagged AI cell (300 stratified rows) puts Model 1 recall at
69.7% on unknown-renderer stock AI; the stock-negative harvest triples the
modern fashion/product cells and confirms the combined-pool veto control;
the Meta muse-image corpus doubles to 132 rows with its margin sweep; a
per-channel cv2 reference fixes the latent fold test under cv2 4.10.0.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks separately green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - new run references point at the gitignored research store, none stale; 4) CLAUDE.md - compact, no changes needed

* Close representation-change campaign: both candidates falsified

Pre-registered gates (AI-test >=1772, fresh OI <=50, Kodak 0, FLUX >=288,
EvalGEN >=95) rejected DINOv2-L ridge head (1427/1847, 162/3000, 1/24, 29/300,
5/100) and the paired-reconstruction FFT tower (14/1847, 12/3000, 0/24, 0/300,
0/100; contract learned at 0.949 pairwise separation but no transfer to
production generators). Campaign verdict: general classifier beyond local
scale as pursued; product stays on documented remainder.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks green minutes ago (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - verdict mirrored in data/research/REPRESENTATION-CAMPAIGN.md, no stale refs; 4) CLAUDE.md - no changes

* Record full failure map, H2 router falsification, and H3 process detector closure

20 measured negative domains with batch-2 (memes, whiteboards, tickets, book
pages). H2 domain router falsified: 97.9% of AI-test looks structured, every
family 94-100%. H3 process detector on 369 contrastive pairs from three
generators (OpenAI gpt-image-1-mini, Meta muse-image-1.0, Gemini flash-image)
achieves 0.909 pairwise AUC on training pairs but transfers at AUC 0.55-0.60
to production AI; human receipts score higher than any AI family. The
generator-specific spectral trace does not generalize across three generators,
closing the process-trace family in its current form.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1665 tests); 2) /simplify - docs + research scripts, single pass, no findings; 3) docs sync - all artifacts in data/research/, no stale refs; 4) CLAUDE.md - no changes

* Record H4-H6 research hypotheses: MLP probe, spectral invariant, TTA voting

H4 (MLP on frozen CLIP): 98.6% AI-test recall but 15.8% photo FPR — same
trade-off curve as quarter-hard. H5 (spectral invariant from CVPR 2025):
2.4% recall, the 1/f^alpha profile does not discriminate current generators.
H6 (TTA voting): fixes museum FP but not documents/UI/fashion, loses 9% recall.
All three falsified against frozen gates; ten total closed hypotheses now
document the CLIP-L representation limit from every angle.

Also records the provider cascade across all 20 domains (69.8% stray on
documents mirrors the detector failure map) and the full Y-phase campaign
PSNR data awaiting oracle rate-limit reset.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - all artifacts in data/research/, no stale refs; 4) CLAUDE.md - no changes

* Fix Y-phase frequency convention, record H4-H6 and campaign results

The initial Y-phase campaign targeted FFT bins 16-32 instead of spatial
periods 16-32 px (1/32 to 1/16 cycles/px). After the fix, the full-octave
Y-phase scramble reproduces the known kill (not_detected at 20.5 dB on s1
at native 1122x1402). Partial sub-bands are visually invisible (42.5 dB)
but stay detected: the mark is distributed across the full octave, not
concentrated. Point intervention is not viable within this frequency band.

H4 MLP probe: 98.6% recall, 15.8% FPR (same trade-off wall).
H5 spectral invariant: 2.4% recall (1/f^alpha does not discriminate).
H6 TTA voting: fixes museum FP but not documents/UI/fashion.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (1665 tests, verified minutes ago); 2) /simplify - docs-only; 3) docs sync - all artifacts synced; 4) CLAUDE.md - no changes

* Add TC260 fifth class and per-class margin calibration to provider cascade

Five-head cascade (openai/google/tc260/meta/no_ai): openai 68.8%, google
76.6%, tc260 63.8%, meta 71.8% mean recall at uniform margin 0.50.
Per-class margins calibrated against 1.67% photo budget: openai 0.31,
google 0.29, tc260 0.39, meta 0.55. TC260 test recall 71.4% at calibrated
margins on 196 held-out images. Chinese-AIGC is a viable pixel class.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (verified within the hour); 2) /simplify - research script, single pass; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Record universal detector v3-v8: diverse training shifts the curve

Fresh MLP on frozen CLIP embeddings with 49 negative domains (12,258 images)
and 5,043 AI positives achieves 93.0% AI-test recall with 39/49 domains at
zero false positives. Digital art FP dropped from 99.3% (Model 1) to 24.8%,
fashion from 62-71% to 12%, UI from 99.3% to 12%. Photo FPR 0.7%. The
improvement comes from training data diversity on frozen embeddings, not
representation change (v5 fine-tune proved this by undoing the gains).
Also adds .local-eval to ruff exclude so the gate covers only tracked files.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - all artifacts in data/research/; 4) CLAUDE.md - no changes

* Add three-tier ensemble detector: DEFINITELY/POSSIBLY/LIKELY_HUMAN

Combines Model 1 and v8 into calibrated confidence tiers. DEFINITELY_AI
(both models agree) achieves 79.6% recall at 0.50% photo FPR with 46/49
domains at zero FP. The union reaches 93.3% recall at 1.80% photo FPR
with Kodak 0/24. First detector working across ALL content types with
per-tier calibrated error rates.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + ensemble; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Add provider v2: diversity training eliminates document leakage

Same diversity-training insight applied to the provider cascade. Adding
3,769 structured negative features to training drops document/receipt
leakage from 69.8% to 5.1%, UI from 10% to 4.3%, fashion from 5-15% to
0-3%. AI recall improves: OpenAI 84% (was 75%), Google 88.8% (was 79%).
Photo leakage 1.4%. Worst regression: memes 47.9% (was 27%).

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - provider v2; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Add universal v11: full taxonomy, 136 domains, 124 at zero FP

Master data plan mapped 148 visual domains. Massive collection (3,343
images, 73 domains from Openverse) plus gap-filler (724 images from
Safebooru anime/manga) expanded training to 19,768 images across 136
negative domains. Universal v11 (MLP-512 + Model 1 ensemble):
DEFINITELY 90.3% recall at 0.50% photo FPR, ANY 95.6% recall,
124/134 domains at exactly 0% FP, worst FP 13.8% (digital art).

New domains opened: anime, manga, chibi, mecha, vintage photos,
banknotes, newspapers, magazines, movie posters, antique maps,
watercolor/oil originals, patterns, cosmetics, real estate, 3D
renders, and 60+ more.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Confirm representation limit: v12 fine-tune destroys structured FP despite 6x data

Fine-tuning CLIP tower on 19,518 images achieves 95.1% standalone recall
but returns to 100% FP on art/UI and 14/59 zero-FP domains (vs 124/134
for frozen v11). This is a representation property, not a data issue.
Frozen CLIP + MLP (v11) is the only working architecture. Also records
provider v3 trained on all negative domains.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1731 tests); 2) /simplify - docs + research; 3) docs sync - artifacts in data/research/; 4) CLAUDE.md - no changes

* Document v13 plateau: 128/136 zero FP, frozen-CLIP ceiling confirmed

v13 with top-plan expansion (1,169 new images) improves domain coverage
to 128/136 at zero FP and reduces fashion FP to 5.3%, but DEFINITELY
recall plateaus at 89.6% and art FP at 15.9%. The frozen CLIP + MLP
architecture has reached its ceiling.

pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (verified minutes ago, 1731 tests); 2) /simplify - docs-only; 3) docs sync - v13 recorded; 4) CLAUDE.md - no changes

* Document the 2026-08-31 photo-classifier freeze.

Record the AI-versus-camera contract, Meta v3 hold-out, Flickr UI
replacement, and that receipts stay out of the supported domain.

* Scrub Spaces catalog details from classifier research docs

* Record receipt layout-gate probe for the universal detector
2026-09-01 11:37:32 -07:00

170 lines
9.0 KiB
Markdown
Raw Blame History

# SynthID oracle fixtures
A compact set of externally verified originals for two uses:
1. **Signal and detector-limit research** with exact native resolutions and
recorded verification sources.
2. **Removal regression set**: verify that a pipeline turns a SynthID-positive
image into a negative one.
There is no reliable local detector of the SynthID pixel watermark (Google's
decoder is proprietary). The ground-truth label therefore comes from an
external oracle, recorded per image in `verified_via` (see below).
## Layout
```
data/synthid/
README.md # collection and verification protocol
manifest.csv # one row per committed original
full-pipeline-quality.csv # reusable evaluation selection
originals/ # one canonical copy of each verified source
```
The originals are committed so the evaluation set is reproducible and its
hashes can be checked in CI. `manifest.csv` is kept in sync with
`originals/`, one row per file. Generated outputs do not belong here. Store
them outside the repository and record only a reproducible verdict in the
evaluation manifest or documentation.
## Reusable removal-quality set
`full-pipeline-quality.csv` is the canonical reusable input set for
full-pipeline visual-quality and watermark-removal tests. Its fixtures are
cleared for permanent public test reuse. Tests, benchmarks, and manual
evaluation bundles may read these images repeatedly without requesting new
permission.
The CSV preserves each platform's original filename and maps it to the one
canonical corpus copy. Use `corpus_path` to read the input, but preserve
`source_filename` in generated result names and keep OpenAI and Gemini outputs
in separate provider groups. This avoids duplicating large PNGs while retaining
the names needed for the corresponding provider oracle.
Rows with a dated negative `final_clean_oracle` were checked after the complete
`visible -> qwen-zimage -> metadata` route with the matching provider oracle.
`not-checked-in-final-candidate` means the input is available as an additional
quality stress fixture, but a newly generated output still needs the matching
oracle before claiming removal. The checked result applies to those exact
output bytes; re-run the oracle after changing the pipeline, seed, model
versions, or runtime settings.
## Verification levels (`verified_via`)
Ground-truth quality, strongest first:
- `gemini-app` — checked via the Gemini app "Verify with SynthID" feature. Gold standard for the pixel watermark (Google models).
- `openai-verify` — checked through the OpenAI web verifier. This is the
historical manifest label for existing rows. OpenAI now also documents a
Content Provenance API with a separate SynthID result; add a distinct manifest
value before recording API-derived labels rather than silently reusing this one.
- `synthid-portal` — checked via Google's SynthID Detector portal.
- `c2pa-metadata` — supported provenance evidence (Google AI C2PA, or OpenAI
C2PA with an explicit `c2pa.watermarked.*` action). Weaker than a provider
oracle: C2PA can be stripped while the pixel watermark remains.
- `third-party` — label asserted by an external dataset, not independently verified.
- `none` — unverified.
Prefer `gemini-app` for any Google image that will gate a test.
## What to collect
For the **regression set**:
- A small number of positive originals with a recorded provider oracle.
- A quality-set row for every original used by repeatable evaluation.
- Re-run the provider oracle on each newly generated output. Do not commit the
generated output as another corpus copy.
The corpus is committed to a public repo: review every image before adding it
and keep out anything private or identifiable you would not publish.
## Ingesting
Use `scripts/synthid_corpus.py` to copy a file into `originals/`, record its
SHA-256, resolution, format, and C2PA issuer through our detector, and append a
row to `manifest.csv`:
```bash
uv run python scripts/synthid_corpus.py ingest path/to/*.png \
--label pos --source "Gemini app" --model gemini-3-pro \
--verified-via gemini-app --notes "1024x1024 batch"
uv run python scripts/synthid_corpus.py status # counts by label / resolution / verification
```
## Autonomous collection via Chrome MCP
> Historical collection notes. Browser interfaces and download behavior can
> change independently of this repository. These steps do not define package
> behavior.
Generation can be driven through the browser (the account must be logged in):
- **Gemini** (`gemini.google.com`): type `Create an image: <prompt>`, wait, hover the
result, click the download icon (top-right). Single, reliable click. Outputs
carry Google C2PA + SynthID. Occasionally the composer stalls in a
"generating" state -> start a New chat to reset.
- **ChatGPT** (`chatgpt.com`): the UI download is flaky (the fullscreen viewer
races and can grab the previous image; the share-modal path works but is
multi-step). Reliable path is an in-page fetch of the rendered image, which
preserves the original bytes (C2PA intact, unlike a canvas re-encode):
```js
// run in the ChatGPT tab via the browser MCP javascript tool
(async () => {
const imgs = [...document.querySelectorAll('img')].filter(i => i.naturalWidth >= 400);
const img = imgs[imgs.length - 1]; // newest large image
const b = await (await fetch(img.currentSrc || img.src)).blob();
const a = document.createElement('a');
a.href = URL.createObjectURL(b); a.download = 'dl.png';
document.body.appendChild(a); a.click(); a.remove();
return 'size=' + b.size; // do NOT return the src (privacy guard blocks query strings)
})()
```
Gotcha: confirm the returned `size` differs from the previous image before
ingesting -- if the new image has not finished rendering, the script grabs the
prior one (the corpus dedups by sha256, but the notes would mislabel it).
ChatGPT also shows an A/B "which is better?" picker; click Skip first.
**Originals, not previews.** Some platforms render a low-res preview in the chat
(Grok serves a ~20KB 1024px JPEG/PNG; the in-page `<img>` fetch grabs *that*, not
the original). Previews are re-encoded and **strip metadata**, so a "clean"
preview is not proof the original is clean. Always pull the original via the
platform's native Download / lightbox button and sanity-check the file size (a
20KB "1024x1024" is a preview). ChatGPT's in-chat `<img>` *is* the full-res
oaiusercontent original, so fetch+blob is fine there; Grok needs the lightbox
Download; Leonardo serves the original JPEG in-chat (download button matches).
## Per-platform watermark map (observed, May 2026)
This is a dated observation table, not a live provider compatibility promise.
The detector's coverage is complementary: metadata catches C2PA /
IPTC; `exif_generator` catches EXIF `Make`/`Software` + XMP `CreatorTool`;
`invisible_watermark.py` (imwatermark) catches the open SD/SDXL/FLUX DWT-DCT
watermark on pristine files; the visible detector catches the Gemini-family
sparkle; the SynthID *pixel* itself has no local detector (oracle only).
| Platform | C2PA issuer | SynthID pixel | IPTC "Made with AI" | Visible sparkle | imwatermark | Corpus label |
|---|---|---|---|---|---|---|
| Gemini app | Google | yes | - | yes | - | pos |
| ChatGPT / gpt-image | OpenAI | yes | - | - | - | pos |
| Microsoft Designer | OpenAI + Microsoft | yes (via OpenAI) | - | - | - | pos |
| Bing Image Creator | Microsoft (MAI-Image) | no | - | - | - | pos (C2PA "Microsoft", not OpenAI) |
| Google AI Studio (Nano Banana) | **none** | yes (oracle-confirmed) | - | yes | - | pos (metadata blind spot) |
| Stability AI (Brand Studio) | Stability AI Ltd | no | - | - | no | pos (C2PA only) |
| Ideogram | none | no | - | - | no | pos (EXIF `Make="Ideogram AI"` only) |
| Meta AI | none | no | **yes** | - | - | neg (for SynthID) |
| Leonardo.ai | none | no | no | - | no | neg |
| Recraft | none (export strips) | no | no | - | no | neg (re-encoded export, no signal) |
| Krea (FLUX 2 host) | none | no | no | - | no | neg (host omits the imwatermark encoder) |
| Grok (xAI) | none (non-adopter) | no | no | - | no | neg (captured: clean low-res preview) |
Key takeaways:
- The same model differs by *surface*: Gemini app wraps C2PA, AI Studio (API/playground) emits none -- only the pixel + sparkle survive.
- Microsoft Designer's DALL·E backend inherits OpenAI's C2PA+SynthID (issuer "OpenAI, Microsoft"); Bing now runs Microsoft's own **MAI-Image** and signs C2PA as "Microsoft" (not OpenAI/DALL·E).
- Meta uses the IPTC `digitalSourceType` marker, not C2PA or SynthID.
- The open imwatermark fires only on *pristine* output from a pipeline that runs the encoder (diffusers default, official BFL) -- not from re-hosts (Krea, Stability hosted SDXL) or re-encoded design exports (Recraft, Canva). Ideogram's only signal is the EXIF `Make` tag.
- Bing and Grok web UIs are uncooperative for autonomous capture (no document_idle for screenshots; blob downloads intermittently no-op; low-res in-chat previews). Use their native download button manually if a full-res sample is needed.
Reference in new issue View Git Blame Copy Permalink