diff --git a/CLAUDE.md b/CLAUDE.md index 9044cff..ba70da8 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -92,4 +92,4 @@ Who embeds what, and whether it is locally detectable (so we know which gaps are - **External AI-vs-real classifier models are out of scope (decided 2026-05-24).** Generic HuggingFace detectors (`Organika/sdxl-detector` Swin Transformer, `umm-maybe/AI-image-detector`, and fine-tunes) exist and report ~0.98 on their *own* SDXL-vs-real validation sets, but they are per-generator and the model cards themselves note degraded accuracy off-distribution; they are untested on gpt-image / Gemini Nano Banana (the metadata-stripped surfaces we care about), and our own light SDXL pass would likely defeat them the same way it defeats SynthID. Detection here stays local + signal-based (metadata + visible sparkle); do not add a bundled classifier dependency. - **DEFAULT STRENGTH IS NOW VENDOR-ADAPTIVE (2026-06-01, SUPERSEDES every fixed-default claim in this bullet and the next).** `resolve_strength(strength, profile, vendor)` + `vendor_for_strength(path)` (`watermark_profiles.py`) read the C2PA issuer (`metadata.synthid_source`) on the ORIGINAL input and pick `OPENAI_STRENGTH` **0.10** / `GEMINI_STRENGTH` **0.15** / `UNKNOWN_STRENGTH` **0.15** when `--strength` is unset; explicit `--strength` always wins. The CLI detects the vendor from the pristine source (before the visible pass / metadata-strip removes C2PA from the temp file) and passes it to the engine, so display and execution agree; `cmd_invisible`/`cmd_all`/`batch` + the module-level `remove_watermark` all thread `vendor`. **This replaces the single 0.30 default AND the prior "do NOT build a vendor-adaptive default" policy** -- both came from the now-debunked region-rescrub-contaminated study (the per-region re-scrub that contaminated those numbers was removed in the controlnet refactor). Basis: the oracle-verified June 2026 controlled study (clean v0.8.6, protect OFF): OpenAI clears at 0.05 across 1024-1600 (n=4, resolution-independent); Google needs 0.15 on the capped-1536 path (n=4). `docs/synthid.md` §2.2 (data) + §5.2 (the adaptive default) are authoritative. **CAVEAT (oracle pass 2026-06-04): the OpenAI 0.10 default is content-dependent, NOT universal -- a flat-graphic OpenAI logo/poster still read SynthID-detected after `default` at 0.10, and photoreal images after controlnet at 0.10/0.15 (low-change regions under-perturbed). Removal at 0.10/0.15 is content×pipeline dependent (see the controlnet Known-limitations bullet); the lever is a higher strength, oracle-revalidated per content type. Do NOT assume the vendor-adaptive default clears every image.** CAVEAT: Google's 0.15 was validated only on `--max-resolution 1536`; native large Gemini (2816) was not locally measurable (OOM on M-series) and is pending GPU validation on raiw.cc -- if it survives 0.15 native, raise `--strength`. **Everything below in this bullet about a fixed 0.10/0.30 default is HISTORICAL; trust the vendor-adaptive constants + docs/synthid.md.** - **SynthID removal: strength + oracle scope.** Default strength is vendor-adaptive (see the bullet above); `docs/synthid.md` §2.2 is authoritative for the numbers. **Oracle scope (load-bearing):** the Gemini app "Verify with SynthID" is the ONLY valid SynthID oracle (detects Google's mark on any image); `openai.com/verify` is scoped to OpenAI provenance (its own C2PA), NOT a SynthID oracle -- a negative there is meaningless for SynthID. There is no local SynthID detector, so the tool cannot self-check; if the oracle still reads SynthID, raise `--strength` to the lowest value that verifies clean. Only the `default` (plain SDXL img2img) and `controlnet` (SDXL + canny ControlNet) profiles exist; the local `invisible` default is weight-for-weight identical to raiw.cc prod (`fal-ai/fast-sdxl` = `stabilityai/stable-diffusion-xl-base-1.0`, runtime-downloaded, not bundled). **Forensic-stealth caveat** (arXiv:2605.09203): defeating the SynthID verifier is NOT forensic invisibility -- independent detectors flag *removal-processed* images vs genuinely-clean ones at >98% TPR@1%FPR, so do not over-claim "indistinguishable from a real photo". -- **`controlnet` pipeline (text/face STRUCTURE preservation, EXPERIMENTAL, opt-in `--pipeline controlnet`).** SDXL + the canny ControlNet `xinsir/controlnet-canny-sdxl-1.0` via `StableDiffusionXLControlNetImg2ImgPipeline` (`watermark_remover._run_controlnet` / `_load_controlnet_pipeline`). **Removal still comes from the img2img regeneration (`strength`); the ControlNet only PRESERVES text and face STRUCTURE by conditioning on the canny edge map** (`cv2.Canny(gray, 100, 200)`, 3-channel). Canny preserves edges, NOT face identity (a regenerated face drifts in likeness); face identity is preserved by the optional `--restore-faces` GFPGAN post-pass (EXPERIMENTAL, opt-in, OFF by default -- see `face_restore.py`, the `restore` extra) -- **but WARNING: that pass can RE-INTRODUCE SynthID into the face regions (oracle-confirmed 2026-06-04, since GFPGAN runs on the ORIGINAL watermarked face and blends ~half its pixels at weight 0.5), so it is a footgun for removal; see the `face_restore.py` bullet.** The CodeFormer alternative stays NON-COMMERCIAL and is not shipped. The earlier `--face-id` IP-Adapter FaceID layer was REMOVED (footgun: it needs high strength and corrupts faces at the low removal strength). No original pixels are copied or frozen, **BUT removal at the low vendor-adaptive strength is CONTENT × PIPELINE dependent and NEITHER pipeline clears all content -- oracle-validated against the OpenAI verifier 2026-06-04 (8 images, strength 0.10/0.15, `--max-resolution 1536`).** The survivors FLIP by content type: **photoreal** (a 9-face grid, a bracelet product photo) SURVIVES controlnet but CLEARS `default` (controlnet's dense edge map keeps the regen too close to the original, so the SynthID-destroying perturbation never happens; plain img2img perturbs photoreal texture enough); **flat graphic** (a logo/poster with large flat color fills) SURVIVES `default` but CLEARS controlnet (at low strength img2img barely changes flat fills so SynthID persists there, while controlnet repaints them more freely); a flat **text** card cleared under both. **Root cause is insufficient STRENGTH, not the pipeline: at 0.10 the low-change regions -- dense-edge photoreal under controlnet, large flat fills under `default` -- are not perturbed enough to destroy SynthID. The vendor-adaptive 0.10 from the June study is NOT universally sufficient (that study's content happened to clear at 0.10).** The robust fix is a HIGHER strength, oracle-revalidated per content type (controlnet can be cranked harder without losing structure; a lower `controlnet_conditioning_scale` also frees the regen on photoreal). So at today's default strength **both pipelines AND `--auto` can LEAVE SynthID on some content** -- a removal-priority caller (raiw.cc) MUST oracle-validate strength across content types before adopting, not pick a pipeline and assume removal. **Follow-up same day: re-running the two photoreal survivors through controlnet at an explicit `--strength 0.15` cleared BOTH on the oracle -- BUT one of them (the bracelet) had SURVIVED the SAME 0.15 controlnet config in the first pass (only the random, unset seed differed). So removal near the threshold is SEED-NON-DETERMINISTIC: the same image+pipeline+strength+resolution can pass or fail run-to-run (img2img uses `seed=None`/random unless `--seed` is passed, and there is no local SynthID detector to self-verify). 0.15 is the borderline, NOT a robust floor -- pick a strength with MARGIN (controlnet ~>= 0.20) rather than exactly on it; the content×pipeline table's 0.15 data point is near-threshold noise. A confirming run at `--strength 0.20` controlnet cleared BOTH photoreal survivors on the oracle (ladder: 0.10 grid detected → 0.15 borderline/non-deterministic → 0.20 both clean), so **0.20 is the recommended robust controlnet floor for OpenAI photoreal** (one margin run, not an N-run repeatability proof -- a service should add margin or verify repeatability since there is no local SynthID detector to self-check). **Engineering follow-up for raiw.cc: the controlnet pipeline should use a HIGHER vendor strength than `default` -- it currently shares `resolve_strength` (0.10/0.15, tuned for plain img2img), but controlnet's edge map preserves structure so it needs ~0.20+; calibrate per vendor/content on the GPU worker, do NOT just reuse the `default` ladder.** See `docs/synthid.md` §5.5.** **Lesson: visual-quality + face-recovery validation does NOT prove watermark removal -- only the SynthID oracle does, across MULTIPLE content types; never infer removal from sharpness/identity, and never conclude from a partial result (the photoreal-only data first read as "controlnet shields, default removes" -- the flat-graphic result reversed it).** `controlnet_conditioning_scale` (CLI `--controlnet-scale`, default 1.0) is the structure-preservation knob (higher = closer to the original structure); fp32 on cpu/mps, fp16-fixed VAE on cuda/xpu. The `controlnet` profile is threaded explicitly (`WatermarkRemover(pipeline=...)` / `InvisibleEngine(pipeline=...)`), NOT inferred from `model_id`. This productionizes the `scripts/controlnet_sweep.py` prototype; see `docs/controlnet-removal-pipeline-research.md`. **Forensic-stealth caveat still applies** (arXiv:2605.09203): defeating the SynthID verifier is not forensic invisibility -- a "this image went through a removal pipeline" classifier can still flag the output. +- **`controlnet` pipeline (text/face STRUCTURE preservation, EXPERIMENTAL, opt-in `--pipeline controlnet`).** SDXL + the canny ControlNet `xinsir/controlnet-canny-sdxl-1.0` via `StableDiffusionXLControlNetImg2ImgPipeline` (`watermark_remover._run_controlnet` / `_load_controlnet_pipeline`). **Removal still comes from the img2img regeneration (`strength`); the ControlNet only PRESERVES text and face STRUCTURE by conditioning on the canny edge map** (`cv2.Canny(gray, 100, 200)`, 3-channel). Canny preserves edges, NOT face identity (a regenerated face drifts in likeness); face identity is preserved by the optional `--restore-faces` GFPGAN post-pass (EXPERIMENTAL, opt-in, OFF by default -- see `face_restore.py`, the `restore` extra) -- **but WARNING: that pass can RE-INTRODUCE SynthID into the face regions (oracle-confirmed 2026-06-04, since GFPGAN runs on the ORIGINAL watermarked face and blends ~half its pixels at weight 0.5), so it is a footgun for removal; see the `face_restore.py` bullet.** The CodeFormer alternative stays NON-COMMERCIAL and is not shipped. The earlier `--face-id` IP-Adapter FaceID layer was REMOVED (footgun: it needs high strength and corrupts faces at the low removal strength). No original pixels are copied or frozen, **BUT removal at the low vendor-adaptive strength is CONTENT × PIPELINE dependent and NEITHER pipeline clears all content -- oracle-validated against the OpenAI verifier 2026-06-04 (8 images, strength 0.10/0.15, `--max-resolution 1536`).** The survivors FLIP by content type: **photoreal** (a 9-face grid, a bracelet product photo) SURVIVES controlnet but CLEARS `default` (controlnet's dense edge map keeps the regen too close to the original, so the SynthID-destroying perturbation never happens; plain img2img perturbs photoreal texture enough); **flat graphic** (a logo/poster with large flat color fills) SURVIVES `default` but CLEARS controlnet (at low strength img2img barely changes flat fills so SynthID persists there, while controlnet repaints them more freely); a flat **text** card cleared under both. **Root cause is insufficient STRENGTH, not the pipeline: at 0.10 the low-change regions -- dense-edge photoreal under controlnet, large flat fills under `default` -- are not perturbed enough to destroy SynthID. The vendor-adaptive 0.10 from the June study is NOT universally sufficient (that study's content happened to clear at 0.10).** The robust fix is a HIGHER strength, oracle-revalidated per content type (controlnet can be cranked harder without losing structure; a lower `controlnet_conditioning_scale` also frees the regen on photoreal). So at today's default strength **both pipelines AND `--auto` can LEAVE SynthID on some content** -- a removal-priority caller (raiw.cc) MUST oracle-validate strength across content types before adopting, not pick a pipeline and assume removal. **Follow-up same day: re-running the two photoreal survivors through controlnet at an explicit `--strength 0.15` cleared BOTH on the oracle -- BUT one of them (the bracelet) had SURVIVED the SAME 0.15 controlnet config in the first pass (only the random, unset seed differed). So removal near the threshold is SEED-NON-DETERMINISTIC: the same image+pipeline+strength+resolution can pass or fail run-to-run (img2img uses `seed=None`/random unless `--seed` is passed, and there is no local SynthID detector to self-verify). 0.15 is the borderline, NOT a robust floor -- pick a strength with MARGIN (controlnet ~>= 0.20) rather than exactly on it; the content×pipeline table's 0.15 data point is near-threshold noise. A confirming run at `--strength 0.20` controlnet cleared BOTH photoreal survivors on the oracle (ladder: 0.10 grid detected → 0.15 borderline/non-deterministic → 0.20 both clean), so **0.20 is the recommended robust controlnet floor for OpenAI photoreal** (one margin run, not an N-run repeatability proof -- a service should add margin or verify repeatability since there is no local SynthID detector to self-check). **Engineering follow-up for raiw.cc: the controlnet pipeline should use a HIGHER vendor strength than `default` -- it currently shares `resolve_strength` (0.10/0.15, tuned for plain img2img), but controlnet's edge map preserves structure so it needs ~0.20+; calibrate per vendor/content on the GPU worker, do NOT just reuse the `default` ladder.** **CERTIFIED 2026-06-04 via the isolated `raiw-controlnet-cert` Modal app (`raiw-app/modal_cert.py`), restore OFF, ≤1536, each vendor on its own oracle: controlnet floors are OpenAI 0.20 (2 photoreal × 3 seeds = 6/6 clean; the 0.15-flipper is seed-robust at 0.20) and Gemini 0.30 (0.20 detected → 0.30 clean on 2/2 seeds). OpenAI 0.20 transfers to prod (resolution-independent); Gemini 0.30 holds only ≤1536 — Gemini is resolution-sensitive and raiw.cc runs NATIVE (`max_resolution=0`), so cap Gemini ≤1536 + use 0.30, or native-calibrate (~0.35+). Prod recipe: controlnet + per-vendor floor in `resolve_strength` (not the default ladder) + FIXED seed (kills the non-determinism) + restore reworked/off.** See `docs/synthid.md` §5.5 + `docs/controlnet-removal-pipeline-research.md` (certified floors table).** **Lesson: visual-quality + face-recovery validation does NOT prove watermark removal -- only the SynthID oracle does, across MULTIPLE content types; never infer removal from sharpness/identity, and never conclude from a partial result (the photoreal-only data first read as "controlnet shields, default removes" -- the flat-graphic result reversed it).** `controlnet_conditioning_scale` (CLI `--controlnet-scale`, default 1.0) is the structure-preservation knob (higher = closer to the original structure); fp32 on cpu/mps, fp16-fixed VAE on cuda/xpu. The `controlnet` profile is threaded explicitly (`WatermarkRemover(pipeline=...)` / `InvisibleEngine(pipeline=...)`), NOT inferred from `model_id`. This productionizes the `scripts/controlnet_sweep.py` prototype; see `docs/controlnet-removal-pipeline-research.md`. **Forensic-stealth caveat still applies** (arXiv:2605.09203): defeating the SynthID verifier is not forensic invisibility -- a "this image went through a removal pipeline" classifier can still flag the output. diff --git a/docs/controlnet-removal-pipeline-research.md b/docs/controlnet-removal-pipeline-research.md index 8dd82bc..18626cc 100644 --- a/docs/controlnet-removal-pipeline-research.md +++ b/docs/controlnet-removal-pipeline-research.md @@ -94,14 +94,36 @@ raising strength cannot fix it — the face is re-pasted from the original after This also corrects the prior "GFPGAN scrubs SynthID / oracle-confirmed clean" claim (it was checked on one lucky image). +### Certified controlnet strength floors (Modal GPU sweep + oracle, 2026-06-04) + +Run via the isolated `raiw-controlnet-cert` Modal app (`raiw-app/modal_cert.py`): +controlnet, `restore_faces` OFF (it re-introduces SynthID), `--max-resolution 1536`, +each image checked on ITS OWN vendor oracle (OpenAI -> openai.com/verify, Gemini -> the +Gemini app; the two payloads are vendor-specific and never cross-checked): + +| vendor | **floor** | evidence (oracle, restore OFF, <= 1536) | +|---|---|---| +| **OpenAI** | **0.20** | 2 photoreal images (9-face grid + bracelet) x seed {1,2,3} = **6/6 clean**; the bracelet that FLIPPED at 0.15 is seed-robust at 0.20 | +| **Gemini** | **0.30** | hardest face (gemini_3): 0.20 detected -> 0.30 clean on **2/2 seeds**; Gemini is the harder vendor (default 0.15 vs OpenAI 0.10) AND resolution-sensitive | + +- **OpenAI 0.20 transfers to prod as-is** (OpenAI removal is resolution-independent: + the study clears it at 0.05 across 1024-1600). +- **Gemini 0.30 is the floor at <= 1536 only.** Gemini is resolution-sensitive (study: + native 2816 likely needs >= 0.30 even on `default`), and **raiw.cc runs NATIVE** + (`max_resolution=0` in `modal_app.py`). So either CAP Gemini to <= 1536 in raiw.cc and + use 0.30, or run a native-resolution Gemini cert and expect a higher floor (~0.35+). + ### Recommendations for a removal pipeline (raiw.cc) - **Treat controlnet as PRESERVATION, not removal.** Choose it for text/structure content, `default` for photoreal; removal efficacy comes from STRENGTH in both. - **Give controlnet a higher, per-vendor strength than `default`** (today both share - `resolve_strength` 0.10/0.15, tuned for plain img2img). Oracle floors so far: OpenAI - ~0.20 with margin, Gemini ~0.20–0.25. Calibrate on the GPU worker with a seed-repeat - sweep (cheap there), not single local runs. + `resolve_strength` 0.10/0.15, tuned for plain img2img). **Certified controlnet floors: + OpenAI 0.20, Gemini 0.30** (see table above) — add a controlnet-specific per-vendor + schedule to `resolve_strength` rather than reusing the `default` ladder. +- **Fix the seed in prod.** The non-determinism is purely `seed=None` (random); a fixed + `--seed` makes every run reproduce the certified-clean result, so you ship a + deterministic, re-certifiable config (and the seed sweep collapses to one config). - **Rework `--restore-faces` before any removal use:** run GFPGAN on the diffusion-CLEANED image (not the original), or drop the weight well below 0.5, or leave it off — then re-validate on the oracle. diff --git a/docs/synthid.md b/docs/synthid.md index 908cbd0..0e77cd1 100644 --- a/docs/synthid.md +++ b/docs/synthid.md @@ -556,11 +556,22 @@ openai.com/verify): robust to downscaling by design, and the study's resolution trend says LOWER processing res needs LESS strength, so 1024 was never the wall.) +**Certified controlnet floors (Modal GPU sweep `raiw-app/modal_cert.py` + oracle, +restore OFF, <= 1536, each vendor on its own oracle):** OpenAI **0.20** (2 photoreal x +seed {1,2,3} = 6/6 clean; the 0.15-flipper is seed-robust at 0.20) and Gemini **0.30** +(0.20 detected -> 0.30 clean on 2/2 seeds). OpenAI 0.20 transfers to prod +(resolution-independent); Gemini 0.30 holds only <= 1536 -- Gemini is +resolution-sensitive and raiw.cc runs NATIVE, so cap Gemini <= 1536 + use 0.30 or +native-calibrate (~0.35+). See `docs/controlnet-removal-pipeline-research.md` for the +table. + **Net for raiw.cc:** (1) controlnet needs a higher, per-vendor strength than -`default` (OpenAI ~0.20, Gemini >= ~0.20-0.25), oracle-calibrated; (2) the +`default` -- CERTIFIED OpenAI 0.20 / Gemini 0.30 (above); add a controlnet-specific +schedule to `resolve_strength`, do not reuse the default ladder; (2) the `--restore-faces` pass can re-add SynthID and must be reworked (restore on the cleaned image / lower weight / off) before it is safe in a removal pipeline; (3) -removal near threshold is seed-non-deterministic, so use margin. +removal near threshold is seed-non-deterministic -> FIX the prod seed (kills the +coin-flip; ship a deterministic certified config). ---