remove-ai-watermarks/docs/controlnet-removal-pipeline-research.md

# ControlNet-as-removal-pipeline research: can structure-conditioned regeneration scrub SynthID and keep text?

Date: 2026-06-02. Source: a manual primary-source pass (WebSearch + WebFetch over the
watermark-removal-attack and SDXL-ControlNet literature). Prompted by issue #35
(@newideas99 / Jacob): "as we use SDXL even at low strength that kills small text ... Do you
think ControlNet could be added to preserve and still remove the watermark?" Clarified scope:
Jacob means **replacing the removal pipeline itself** with a ControlNet-conditioned
regeneration (structure held by the control signal), NOT a separate text-protection add-on.

A deep-research workflow run was attempted first (`wf_3244411d-ffd`) and failed at the harness
level (97 agents completed without emitting StructuredOutput; ~4.3 M tokens, no report). This
note is the hand-run replacement.

## The question, precisely

Can a single full-image ControlNet-conditioned diffusion pass **replace** plain SDXL base 1.0
img2img as the watermark remover, so that one structure-guided regeneration removes the
invisible robust pixel watermark (Google SynthID) **everywhere** while keeping fine detail and
small/CJK **text** legible across the whole image? The hard constraint is unchanged from
`text-protection-research.md`: the watermark must be scrubbed everywhere including inside text,
so any path that freezes or composites original text pixels is disqualified.

## Executive summary

The idea is **already academically validated as a watermark remover and is literally what we
already ship** — CtrlRegen (ICLR 2025) is a canny-ControlNet + DINOv2-semantic pipeline that
regenerates from clean noise. But the make-or-break gap is exact: **none of the
watermark-removal papers validate TEXT or fine-detail preservation at all** — CtrlRegen reports
only FID/PSNR/quality-model scores and explicitly contains no text, fine-detail, or
hallucination analysis. Our shipped ctrlregen's empirical failure ("destroys real content,
hallucinates micro-text in smooth regions") is precisely this unstudied failure mode of the
published method, most likely driven by our **512 px tiling** (text occupies too few pixels per
tile to regenerate legibly; edge-free smooth regions get DINOv2-semantic hallucination). The
constructive path is NOT to keep fixing the SD1.5 CtrlRegen, but to port the structure-control
idea onto an **SDXL-native** ControlNet (xinsir tile-sdxl / ControlNet-Union-SDXL) as a control
add to our existing SDXL base 1.0 img2img, run it at **1024+, not 512 tiles**, and empirically
sweep the (denoise strength x conditioning scale x resolution) cube against the SynthID oracle
AND a text-legibility check. The central tension may be fundamental and must be measured: the
conditioning strong enough to keep text legible may suppress regeneration enough to let SynthID
survive; the regeneration strong enough to scrub may deform text regardless of edges.

## Findings (with confidence and sources)

### Finding 1 — confidence: high

**Claim.** "ControlNet as the removal pipeline" is exactly CtrlRegen (ICLR 2025), and our
shipped `ctrlregen` profile is a faithful implementation of it. Its **spatial control is canny
edges** extracted from the watermarked image; its **semantic control is DINOv2-giant** via a
trainable projection + decoupled cross-attention. Clean-noise (full-strength) regeneration
scrubs the watermark from both pixel and latent space while the two control nets hold structure.

**Evidence.** CtrlRegen: spatial control "conditioned on Canny edge images extracted from the
watermarked image," integrated into the U-Net decoder blocks via a ControlNet structure;
semantic control on "DINOv2-giant" embeddings. Removal is strong: TPR@1%FPR driven from 1.00 ->
0.01 (StegaStamp) and 0.99 -> 0.12 (TreeRing). This matches our `ctrlregen/engine.py` exactly
(canny detector + `facebook/dinov2-giant` + spatial ControlNet from `yepengliu/ctrlregen`).

**Sources.** https://arxiv.org/html/2410.05470v1 · https://github.com/yepengliu/CtrlRegen ·
https://openreview.net/forum?id=mDKxlfraAn

### Finding 2 — confidence: high

**Claim.** Regeneration provably removes any bounded-perturbation pixel watermark **given enough
noise** — the operative constraint is the amount of regeneration, which is the same knob that
trades against fidelity.

**Evidence.** Zhao et al., "Invisible Image Watermarks Are Provably Removable Using Generative
AI" (NeurIPS 2024): a noise-then-reconstruct regeneration attack "guarantees the removal of any
invisible watermark" that perturbs the image within a bounded L2 distance. The guarantee is a
function of injected noise magnitude — low noise preserves detail but leaves the watermark; high
noise scrubs but discards original signal. This is the knob ControlNet conditioning is meant to
make survivable (push regeneration high while the control signal holds composition).

**Sources.** https://arxiv.org/abs/2306.01953 · https://github.com/XuandongZhao/WatermarkAttacker

### Finding 3 — confidence: high

**Claim.** The make-or-break gap: **no watermark-removal paper validates text or fine-detail
preservation.** CtrlRegen's "high perceptual quality" is FID/PSNR/quality-model only and
explicitly omits text, fine-detail, and hallucination analysis. So the literature does NOT
support the specific claim Jacob needs (text survives), it is simply unmeasured.

**Evidence.** CtrlRegen reports CLIP-FID, PSNR, Q-Align, LIQE; the fetched analysis confirms
"the paper contains no discussion of text preservation, fine-detail retention, or hallucination
artifacts," and "explicitly avoids discussing failure modes." Pixel metrics like PSNR are
acknowledged not to reflect perception, and text legibility is a different axis than FID.

**Sources.** https://arxiv.org/html/2410.05470v1

### Finding 4 — confidence: medium-high

**Claim.** Resolution is the prime suspect for our shipped ctrlregen's content destruction. We
tile to **512 px** and run full clean-noise per tile; at 512 px text occupies too few pixels per
tile to regenerate legibly, and smooth edge-free regions (no canny signal) are filled by the
DINOv2 semantic prior, which hallucinates texture/micro-text. The paper omits resolution
entirely, so this is an implementation regime it never characterized.

**Evidence.** Our `ctrlregen/engine.py`: `PROCESS_SIZE = 512`, `TILE_SIZE = 512`, full strength
on each tile. This mirrors the `_run_region_hires` insight (text needs MORE pixels under
regeneration so strokes exceed the VAE's ~8 px latent floor), but ctrlregen runs the regeneration
at LOW res, the opposite. CtrlRegen's paper gives no resolution/tiling spec to contradict this.

**Sources.** internal (`src/remove_ai_watermarks/noai/ctrlregen/engine.py`); resolution-omission
confirmed against https://arxiv.org/html/2410.05470v1

### Finding 5 — confidence: high

**Claim.** SDXL-native ControlNets exist, so the removal-pipeline upgrade need NOT be the SD1.5
re-architecture our current ctrlregen is. xinsir `controlnet-tile-sdxl-1.0` and
`controlnet-union-sdxl-1.0` (ControlNet++) run on SDXL base 1.0. The tile model has a `tile_var`
image-variation mode purpose-built to regenerate detail while preserving structure, at
`controlnet_conditioning_scale = 1.0`, optimal 1024 px. This is a drop-in control add to our
existing SDXL img2img.

**Evidence.** xinsir tile-sdxl model card: use cases = deblur/detail-repaint, **image variation
(preserving structure)**, super-resolution; `controlnet_conditioning_scale = 1.0`, ~30 steps,
optimal 1024x1024, works with `madebyollin/sdxl-vae-fp16-fix` (the same VAE our fp16 path
already swaps in). ControlNet-Union-SDXL / ControlNet++ merges 10+ control types (canny, HED,
tile, depth, lineart) into one SDXL model.

**Sources.** https://huggingface.co/xinsir/controlnet-tile-sdxl-1.0 ·
https://huggingface.co/xinsir/controlnet-union-sdxl-1.0 · https://github.com/xinsir6/ControlNetPlus

### Finding 6 — confidence: high

**Claim.** The community tile-ControlNet upscale workflow runs at **LOW denoise (0.3-0.4)** —
the wrong regime for watermark removal. It preserves detail precisely by regenerating little, so
a naive tile-upscale preserves text AND preserves the watermark. The open empirical question is
whether at `conditioning_scale ~1.0` you can push denoise high enough to scrub SynthID while the
tile conditioning still holds text — the exact cell to test.

**Evidence.** Stable-Diffusion-Art ControlNet-tile upscale: denoise "typically 0.3, max ~0.4 to
avoid artifacts"; some users push 0.6 with ControlNet strength 0.5. Our own data: SynthID
survives below the removal-strength threshold (current Gemini needs notably higher denoise than
the tile-upscale regime). So the detail-preserving regime and the watermark-scrubbing regime are
on opposite ends of the denoise axis; ControlNet conditioning is the bet that they can meet.

**Sources.** https://stable-diffusion-art.com/controlnet-upscale/ ·
internal (`docs/synthid.md` strength data)

### Finding 7 — confidence: high

**Claim.** Forensic-stealth caveat: diffusion-based regeneration is among the MOST detectable
removal families. Even a ControlNet-regeneration that fools the SynthID oracle leaves forensic
traces flagging the output as "removal-processed" at >98% TPR@1%FPR. This bounds the claim (do
not over-promise "indistinguishable from an original") but does not block the use case — the
SynthID oracle still reads negative.

**Evidence.** "Removing the Watermark Is Not Enough: Forensic Stealth in Generative-AI Watermark
Removal" (arXiv:2605.09203, Goonatilake & Ateniese, GMU): across six removal attacks including
diffusion-based regeneration, independent forensic detectors separate removal-processed from
clean content at >98% TPR under a 1% FPR budget.

**Sources.** https://arxiv.org/html/2605.09203v1

### Finding 8 — confidence: low (watch, do not build on yet)

**Claim.** Partial/semantic-guided regeneration is an active sub-direction that explicitly targets
the removal-vs-fidelity tradeoff, but the specific fidelity-on-text claims were not verifiable
from the source in this pass.

**Evidence.** "Removing Watermarks with Partial Regeneration using Semantic Information"
(arXiv:2505.08234) proposes focusing regeneration on watermarked regions with semantic (VLM)
conditioning to preserve untouched areas; the PDF body did not render cleanly enough to confirm
its quantitative text/detail results. Treat as a pointer, not evidence.

**Sources.** https://arxiv.org/pdf/2505.08234

## Recommendation / decision

**ControlNet-as-removal-pipeline is worth prototyping — but not by fixing the SD1.5 ctrlregen.**
Port the structure-control idea onto an SDXL-native ControlNet as a control add to the existing
SDXL base 1.0 img2img, run it at full resolution (1024+, NOT 512 tiles), and treat the
text-vs-scrub tension as an empirical question to measure, not assume.

**Prototype (runs locally on 32 GB MPS — no dedicated GPU required):**

Compute is NOT the bottleneck. On a 32 GB Apple-silicon machine (M5 here) native SDXL already
runs entirely on MPS with no CPU fallback (~155 s at 1122x1402, verified — see `synthid.md` /
CLAUDE.md). The prototype runs at **1024** (fewer pixels than that) with SDXL base + an SDXL
ControlNet + activations in **fp32** (MPS fp16 decodes to all-black NaN — issue #29 — confirmed
on run 1 below; fp32 is the required default on mps/cpu) — fits the 32 GB budget with vae-tiling +
attention-slicing; ~1-2 min/image, so a coarse sweep is a sub-hour background run. A dedicated GPU
is needed ONLY for the separate
native-large-Gemini (2816 px) case, which OOMs even without a ControlNet (that stays a raiw.cc
GPU task). The genuine external dependency is NOT compute but the **manual SynthID oracle**:
there is no local SynthID detector, so removal is verified by hand in the Gemini app
("Verify with SynthID") per image, regardless of where the diffusion runs.

Runner: **`scripts/controlnet_sweep.py`** (built 2026-06-02) implements exactly this sweep —
SDXL base 1.0 + an SDXL-native ControlNet img2img, one output per (control x strength x scale)
cell, plus a `sweep_index.csv` with empty `synthid_oracle` / `text_legible` columns to fill by
hand. It uses the dedicated single-type xinsir models (`controlnet-canny-sdxl-1.0`,
`controlnet-tile-sdxl-1.0`) rather than the Union model to keep the diffusers API path robust.

    uv run python scripts/controlnet_sweep.py watermarked.png -o sweep_out

1. SDXL base 1.0 img2img + `xinsir/controlnet-canny-sdxl-1.0` / `controlnet-tile-sdxl-1.0`
   (sweep both `tile` and `canny` control), full image at 1024, `sdxl-vae-fp16-fix`.
2. Sweep the cube on fresh Gemini + gpt-image inputs that contain small/CJK text:
   - denoise strength {0.15, 0.3, 0.5, 0.7, 1.0}
   - `controlnet_conditioning_scale` {0.5, 0.8, 1.0}
   - control type {tile, canny}
3. Per cell, measure BOTH axes:
   - **removal**: Gemini app "Verify with SynthID" oracle (the only valid SynthID oracle; for
     gpt-image also openai.com/verify for provenance) — must read clean.
   - **text**: OCR round-trip / visual legibility of the small text.
   - secondary: SSIM/FID vs original for global fidelity.
4. Find the Pareto cell where the oracle is clean AND text stays legible.

**The honest fork the prototype resolves:**
- If such a cell exists -> the answer to Jacob is YES, ship an SDXL-native ControlNet removal
  profile (replacing the SD1.5 ctrlregen) tuned to that cell.
- If no cell clears both (the tension is fundamental: scrub-strength always deforms text, or
  text-preserving conditioning always spares the watermark) -> the canny/tile-ControlNet middle
  path is dead for text, and the standing answer reverts to `text-protection-research.md`: a full
  **glyph-conditioned re-render** (EasyText / TextSR on a FLUX-DiT base) is required, which is a
  base-model migration, not a control add.

**Do not:** keep tuning the 512 px SD1.5 ctrlregen for text (wrong resolution, wrong base model);
run tile-ControlNet at the community 0.3-0.4 upscale denoise and expect watermark removal (that
regime preserves the watermark); over-claim forensic invisibility (Finding 7).

## Prototype run 1 — 2026-06-02 (text axis measured; watermark axis pending the oracle)

First sweep on a real, SynthID-positive, text-dense input: the corpus tokyo-street-night
gpt-image (`88e61a38-chatgpt_tokyo.png`, 1023x1537 -> 680x1024, dense small CJK + Latin neon
signage; SynthID + C2PA confirmed, so its valid oracle is openai.com/verify). Grid: control
{canny, tile} x strength {0.3, 0.5, 0.7, 1.0} x `conditioning_scale` 1.0, fp32 on MPS. Outputs +
`sweep_index.csv` (text verdicts filled by visual inspection; `synthid_oracle` left for the
manual run) are under `/tmp/cnsweep/` (not committed — derived regenerations of corpus content).

**Measured — PSNR vs input (proxy for how much was regenerated):**
- canny: 0.3 -> 16.91, 0.5 -> 15.91, 0.7 -> 14.82, 1.0 -> 13.22 (monotonic drop = progressively
  more regeneration as strength rises; canny only pins edges, so flat regions change).
- tile: 0.3 -> 17.89, 0.5 -> 17.84, 0.7 -> 17.83, 1.0 -> 17.74 (**flat and high — near-identity
  even at strength 1.0**; tile@scale1.0 pins the whole image to the input and barely regenerates).

**Measured — text legibility (visual, focused on SMALL text; large high-contrast glyphs survive
everything because canny/tile hold their edges):**
- canny: legible at 0.3, softening at 0.5 (partial), garbling at 0.7, hallucinated pseudo-glyphs
  at 1.0 ("NEC" -> "NWENES"). Same plain-img2img small-text deformation, only big text protected.
- tile: near-identity through 0.7, only tiny alterations at 1.0 — small text preserved throughout.

**Reading (the make-or-break tension, now visible in the data):**
- **tile@scale1.0 does not actually regenerate** (flat PSNR), so it preserves all text but almost
  certainly leaves the watermark intact — it is a near-identity pass, exactly the community
  "tile-upscale preserves detail by not regenerating" regime (Finding 6), confirmed.
- **canny@scale1.0 regenerates progressively** (PSNR drops) and so could scrub — but small text
  breaks at exactly the strength where scrubbing would start to bite. canny saves big edges, not
  sub-stroke small text.
- Net on the text axis: neither cell at scale 1.0 cleanly gives "high regeneration + legible small
  text." This is the literature prior (Findings 3, 6) reproduced empirically. Lowering
  `conditioning_scale` to force small-text regeneration is the same tradeoff knob, not an escape.

**Still pending (the decisive half, cannot be done locally):** run the 8 cells through the SynthID
oracle and fill `synthid_oracle`. The most informative cells: canny 1.0 (text dead — does it at
least scrub? if not, the canny path is dead outright), canny 0.5 (text partial — does it scrub?),
tile 1.0 (text perfect — predicted to still read present). If no cell is `oracle=clean` AND
`text=yes`, the fork resolves to the glyph-re-render path (`text-protection-research.md`).

**Incidental bug caught:** the first run used fp16 on MPS (the script's original default) and
produced **all-black** outputs across every cell (2 KB PNGs, PSNR 9.22 flat) — the issue #29
fp16-VAE-NaN failure, and the fp16-fix VAE did not save it on MPS. Fixed `scripts/controlnet_sweep.py`
to default fp32 on mps/cpu (fp16 only on cuda/xpu), matching the production pipeline.

## Tuning ControlNet for text preservation across image types (research 2026-06-03)

Goal: how to configure the canny-ControlNet path to best preserve text (and faces) on diverse
images. Primary sources: diffusers ControlNet doc, the ControlNet paper (arXiv:2302.05543),
xinsir model cards, practitioner guides. The **critical reframe**: almost all community ControlNet
advice optimizes a txt2img *generation* tradeoff (control vs creative freedom). OUR context is
img2img *watermark removal*, where the objective is the opposite -- maximum faithful preservation
while regenerating just enough to scrub. So several common recommendations INVERT here.

**Removal is `strength`; everything below is preservation and does not change removal efficacy**
(only the watermark-shielding risk -- see the caveat). Set `strength` by the oracle/vendor need;
tune these to keep text/faces intact at that strength.

Knobs, ranked by impact for text:

1. **Canny edge density (the per-image lever, currently hardcoded `_CANNY_LOW=100`/`_CANNY_HIGH=200`).**
   Lower thresholds capture more/finer edges; higher thresholds keep only major outlines (diffusers
   doc + practitioner guides; ControlNet paper uses 100/200 as the default). Small-text strokes and
   fine facial features fall below the default 100/200 and are missed. **For dense small text
   (infographics, signage) lower the thresholds (~50/120, even 30/100 for facial likeness per
   practitioner tests); for high-contrast large text 100/200 already suffices.** Denser canny is
   still a BINARY thresholded edge map, so it does not carry the low-amplitude SynthID pixel pattern
   -- it passes more shape, not the watermark (still oracle-verify). This is the single highest-value
   unexplored lever and should become a CLI knob.

2. **`controlnet_conditioning_scale` -> keep at 1.0 (max structure hold).** Community defaults to 0.5
   for creative balance; we want maximum preservation, so 1.0 (xinsir canny/tile cards also recommend
   1.0). We measured text on a clean high-contrast image surviving across strength 0.1-0.5 at scale
   1.0 (PSNR ~26 flat), so scale 1.0 is the right default; only lower it if a specific image needs
   more regeneration to scrub (raises shielding risk the other way).

3. **`control_guidance_start=0.0`, `control_guidance_end=1.0` (full window) -- KEEP, do not shorten.**
   The common "end=0.5: establish structure early then let the model render detail freely" is a
   creative-generation recipe; for text it is HARMFUL -- the late free steps re-render and deform the
   glyphs. We want the edge control active through ALL denoise steps so text stays pinned. (Our
   pipeline already uses the 0->1 default; the point is to NOT adopt the shorten-the-window advice.)

4. **Control type, per image type:**
   - **Text / graphics / high-contrast -> canny** (the literature's reliable choice for defined edges
     and text; what we ship).
   - **Faces / smooth tonal content -> soft-edge / HED is a candidate worth testing.** Canny's hard
     binary threshold fractures smooth skin gradients; HED/soft-edge gives gradual edges that may hold
     faces better. UNVERIFIED for removal (softer edges may carry slightly more original signal ->
     oracle-check). A face-heavy image is the test (gemini group photos).
   - **tile -> NOT for removal.** It is near-identity (detail-enhancement at low denoise); it shields
     the watermark (measured flat PSNR ~17.8 across strength on the tokyo sweep). Do not use it as the
     removal control.

5. **Resolution** -- higher long-side = strokes span more VAE latent cells = less softening, while
   still fully regenerating. Already a knob (`--max-resolution`); for tiny text prefer native/large.

**Multi-ControlNet (canny + soft-edge), list scales e.g. `[1.0, 0.8]`** (diffusers MultiControlNet):
could hold text edges AND face geometry at once, but doubles ControlNet memory/latency and raises the
shielding risk; defer to a v2 after the single-canny path is dialed in.

**Image-type playbook (proposed, to validate with the oracle):**
- Clean high-contrast text (openai_1-style): canny 100/200, scale 1.0, full window -- already optimal.
- Dense small text / infographics (big_pic3, neon signage): canny **lower thresholds (~50/120)**,
  scale 1.0, full window, larger resolution.
- Faces / portraits: try **soft-edge/HED** control, scale 1.0; or multi-ControlNet canny+softedge.

**Hard caveat:** every change that increases preservation (higher scale, denser canny, fuller window,
softer edges) marginally REDUCES effective regeneration and so raises the chance the watermark
survives -- exactly the shielding failure mode. There is no local SynthID detector, so each tuning
change must be re-confirmed on the oracle. These are img2img-context recommendations derived from
generation-context sources plus our own measurements; treat the playbook as hypotheses to verify, not
settled defaults.

**Sources.** https://huggingface.co/docs/diffusers/en/using-diffusers/controlnet ·
https://arxiv.org/pdf/2302.05543 · https://huggingface.co/xinsir/controlnet-canny-sdxl-1.0 ·
https://huggingface.co/xinsir/controlnet-tile-sdxl-1.0 · https://blog.cephalon.ai/canny-and-softedge/

## FaceID research: identity-preserving face conditioning (research 2026-06-03)

Motivation: canny alone preserves face STRUCTURE/position better than plain SDXL but does NOT hold
IDENTITY -- verified on a real Gemini group photo (gemini_3, s015): faces drift in expression and
likeness (the smile/mouth and eyes change), they are "a similar person," not the same one. Canny
carries edges, not identity, so the regenerated face is identity-drifted. To hold identity WITHOUT
copying original pixels (the hard constraint -- copied pixels carry SynthID), the conditioning must
be an identity EMBEDDING, not pixels. Primary sources: diffusers IP-Adapter doc, InstantID
(arXiv:2401.07519), IP-Adapter (arXiv:2308.06721), practitioner comparisons.

### Findings

**1. IP-Adapter FaceID conditions on an ArcFace identity VECTOR, not pixels (confidence: high).**
FaceID extracts `insightface` ArcFace `normed_embedding` (a ~512-d identity vector) via
`FaceAnalysis`, and passes it as `ip_adapter_image_embeds` -- NOT a CLIP image embedding, NOT the
original pixels. So it is constraint-compatible: the watermark (a pixel-amplitude pattern) is not in
the identity vector, and the img2img still regenerates the pixels (removal via `strength` unchanged).
It loads on any SDXL via `load_ip_adapter` (~100 MB), is fast/low-VRAM, but identity fidelity on SDXL
is ~5-10% lower than the SD1.5 line / dedicated methods.

**2. Multiple distinct faces ARE handled, via regional attention masks (confidence: high -- THE key
unlock).** This is the make-or-break for group photos (our hardest case). diffusers supports a LIST
of IP-Adapter face images each with its own binary region mask: `IPAdapterMaskProcessor` builds the
masks, `set_ip_adapter_scale([[s1, s2, ...]])`, and `cross_attention_kwargs={"ip_adapter_masks":
masks}`. So you detect each face, extract its own ArcFace embedding, assign it a region mask, and one
pass preserves N different identities simultaneously. (InstantID, by contrast, is single-subject --
it averages embeddings for multiple refs, which is wrong for distinct people -- so for group photos
**IP-Adapter FaceID + masks beats InstantID**.)

**3. IP-Adapter + ControlNet + img2img compose (confidence: high).** The doc shows IP-Adapter +
ControlNet (depth) in one pipeline and IP-Adapter + img2img (`strength`). Our target stack is the
union: `StableDiffusionXLControlNetImg2ImgPipeline` (canny = structure) + `load_ip_adapter` (FaceID =
identity) + `strength` (removal). `set_ip_adapter_scale` (1.0 = image-only, 0.5 = balanced) is the
identity-hold knob. API friction to verify in implementation: that `ip_adapter_masks` via
`cross_attention_kwargs` works on the *ControlNet img2img* pipeline (the masking is an attention-
processor feature, so it should be pipeline-agnostic, but confirm).

**4. InstantID / PuLID positioning (confidence: medium).** InstantID does not train the UNet so it
composes with canny/depth ControlNets, and gives better single-face fidelity than FaceID -- but it is
single-subject (needs its own landmark ControlNet + dedicated weights). PuLID has the best identity
fidelity but is heaviest and Flux-leaning. For our multi-face, constraint-bound, SDXL-canny case,
IP-Adapter FaceID + masks is the right first build; InstantID/PuLID are single-portrait upgrades.

### Architecture (proposed)

```
detect faces (insightface) -> per face: ArcFace embed + region mask
one img2img pass:
  image=init, control_image=canny(init),                      # structure (existing)
  ip_adapter_image_embeds=[face_embeds],                       # identity per face
  cross_attention_kwargs={"ip_adapter_masks": face_masks},     # each face -> its region
  controlnet_conditioning_scale=1.0, set_ip_adapter_scale(~0.6),
  strength=vendor-adaptive                                     # removal (unchanged)
```
Pixels are regenerated (SynthID removed by `strength`), structure held by canny, each face's identity
held by its masked ArcFace vector -- no original pixel copied.

### Risks / honest costs

- **Shielding risk (same wall):** FaceID conditioning, like canny, reduces effective regeneration ->
  higher `set_ip_adapter_scale` raises the chance SynthID survives in the face region (echo of why the
  old region-hires failed). MUST oracle-verify removal at the chosen FaceID scale; keep `strength` at
  the vendor threshold.
- **New heavy dependency:** `insightface` + `onnxruntime` + the `buffalo_l` model (~300 MB, downloaded
  on first use). Detection + embedding is CPU/ONNX, separate from the diffusion.
- **Detection floor:** insightface needs faces large enough (det_size ~640); tiny faces in a dense
  group may not be detected -> not preserved (falls back to canny-only for those).
- **Identity ceiling:** SDXL FaceID is ~5-10% off true identity -- a meaningful boost over canny-only
  drift, NOT a perfect face swap. Set expectations; PuLID/InstantID are the higher-fidelity (heavier)
  paths if needed.
- **Value scales with strength:** at low strength (OpenAI 0.10) faces barely drift, so FaceID is
  marginal; at the higher strength a hard vendor (Google 0.30) needs, FaceID earns its keep.

### Build plan (staged)

- v1: optional `--face-id` flag on `--pipeline controlnet`. Detect faces; if any, run the masked
  FaceID pass (works for 1 or N faces -- masks generalize). If none detected, fall through to plain
  canny. Oracle-verify SynthID removal is preserved at the default FaceID scale on a face image.
- v2 (if identity still short): InstantID for single-portrait, or PuLID, as a higher-fidelity opt-in.

**Sources.** https://huggingface.co/docs/diffusers/main/en/using-diffusers/ip_adapter ·
https://huggingface.co/h94/IP-Adapter-FaceID · https://arxiv.org/pdf/2401.07519 (InstantID) ·
https://instantid.github.io/ · https://arxiv.org/abs/2308.06721 (IP-Adapter)

### FaceID prototype run 1 -- 2026-06-03 (NEGATIVE on dense small-face groups)

Built and shipped the masked multi-face FaceID layer (`--face-id`, `face_id.py`, `faceid` extra).
First real run on the gemini_3 group photo (Google, s015, scale 0.6, native 2816 via cap 1536):
insightface detected **17 faces**, the masked multi-face pass composed and ran end-to-end (non-black
output), so the API is correct. **At s015 the result is a clear FAILURE: every face corrupted --
melted/discolored/psychedelic, materially WORSE than canny-only.**

**ROOT CAUSE FOUND (confirmed by ablation, not speculation) -- it is STRENGTH, not scale/masks/faces.**
Investigated the real data: masks are fine (max overlap depth 2, 33% coverage, only 0.2% of pixels
double-covered -- NOT an overlap problem), embeddings are fine (`normed_embedding` norm 1.000), the
FaceID LoRA is not required for SDXL (h94 model card), and faces span 34-181 px (7 medium + 10 tiny).
None of those is the cause. The decisive test: the SAME image + FaceID at **strength 0.5** produces
**clean, coherent faces across the whole group** (no psychedelic artifacts). So FaceID needs
substantial regeneration: the h94 usage is full generation (txt2img, 30 steps); at our removal
strength (0.10-0.15 = ~7 effective steps) the strong identity cross-attention cannot reconcile with
a latent that is ~85% the untouched original, so it smears identity-colored noise onto the faces.

**This is a FUNDAMENTAL tension, not a tuning bug:** watermark removal wants LOW strength (minimal
degradation, just enough to scrub), FaceID wants HIGH strength (regenerate the face to impose
identity). They are opposed. At strength 0.5 FaceID works AND removes the watermark, but the whole
image regenerates much more (canny still holds text/edge structure, but texture/detail drifts well
beyond the 0.15 "minimal degradation" target). So `--face-id` is a HIGH-STRENGTH option: it trades
whole-image fidelity for face identity, and is a footgun at the low default strength (guaranteed
garbage). Required follow-up code guard: when `--face-id` is set, floor `strength` at ~0.5 (or refuse
+ warn) -- never run FaceID at the vendor-adaptive removal strength. Open question: whether
high-strength FaceID's whole-image drift is acceptable for face-centric images, or whether identity
preservation at LOW strength needs a different mechanism entirely (FaceID structurally cannot do it). (Infra lesson: the `faceid` extra must
stay numpy<2.0 -- pin `onnx<1.18` + `scipy<1.18`; pinning numpy UP, as the first build did, leaves a
numpy-1.26 env with a numpy-2-only scipy that crashes the diffusers import via `np.long`.)

## Face preservation, done properly (research 2026-06-03, after the FaceID failure)

The FaceID run failed and I wrongly concluded "faces can't be preserved." Re-research corrected the
understanding. The hard constraint is unchanged: to remove the watermark FROM a face the face MUST
be regenerated (freezing it leaves SynthID), so the goal is identity-preserving REGENERATION of the
face, at minimal overall image degradation. Three things I got wrong and the corrected picture:

**What I got wrong:** (1) I applied FaceID at GLOBAL high strength -- the literature is clear the
architecture must be REGION-ADAPTIVE (face region handled separately, background stays low-strength);
(2) I used IP-Adapter FaceID, the WEAKEST identity tool -- InstantID uses an ArcFace encoder and hits
82-86% face-recognition similarity vs FaceID's weak CLIP-ish signal; (3) I missed the entire
face-restoration class (CodeFormer / GFPGAN), which is purpose-built for "regenerate a face, keep
identity."

**The most promising mechanism -- CodeFormer face-restoration post-pass (confidence: high on the
mechanism, unverified on our watermark).** CodeFormer is a VQ-VAE: a frozen discrete CODEBOOK of HQ
facial priors + a Transformer that predicts code *tokens* from the input, and a frozen decoder that
regenerates the face FROM THE CODEBOOK ENTRIES -- "does not depend on feature fusion with low-quality
cues." So the output face pixels come from a finite learned codebook, NOT from the input pixels:
**the SynthID pixel-amplitude pattern physically cannot survive a codebook re-synthesis** -- a
stronger scrub than low-strength img2img (which keeps ~85% of the latent). Fidelity knob `w` in
[0,1]: higher w preserves identity but fuses MORE low-quality (input) cues (more watermark risk),
lower w leans on the codebook (cleaner scrub, identity drift) -- the same scrub-vs-fidelity tension,
settled per-image by the oracle; there is likely a `w` that holds identity AND clears the oracle.

**Constraint-compatible architecture:** run the normal canny low-strength controlnet removal globally
(minimal degradation everywhere), then detect+align each face, run CodeFormer on the **ORIGINAL** face
crop (to capture true identity AND re-synthesize from the codebook = scrub), and composite the
CodeFormer output (codebook-generated, not original pixels -> no copy, no watermark) into the cleaned
image. Decouples whole-image minimal-degradation from face identity -- no high GLOBAL strength needed.

**Honest costs/caveats:** (a) **License -- CodeFormer is NTU S-Lab 1.0 (non-commercial/research)**, so
it cannot be bundled in this MIT tool for general use; the license-clean alternative is **GFPGAN
(Apache-2.0)**, slightly lower quality. (b) Deps (basicsr/facexlib) are heavy and numpy-version-finicky
(same class of conflict as insightface). (c) CodeFormer is a *restoration* model -- it can subtly
alter expression/asymmetry; identity is held but not pixel-identical. (d) **The watermark-scrub is
mechanistically strong but UNVERIFIED -- must oracle-check.** InstantID + region-adaptive strength is
the alternative if the restoration route disappoints, but it is more complex (differential strength).
Prototype plan: validate CodeFormer on a real face in a THROWAWAY env (identity held? oracle clean?)
before any project-env integration or the license/GFPGAN decision.

### CodeFormer prototype -- VALIDATED end-to-end 2026-06-03 (oracle-confirmed)

Prototyped the CodeFormer face-restoration post-pass (codeformer-pip in a throwaway venv, forced CPU
-- the pip wrapper has an MPS device-mismatch bug) on the gemini_3 group photo (18 faces). Pipeline:
`all --pipeline controlnet --strength 0.15` (sparkle + SynthID removed from the whole image, minimal
degradation) -> CodeFormer on the ORIGINAL faces -> feather-composite the CodeFormer faces into the
all-cleaned image. Oracle results (Gemini app "Verify with SynthID"), isolating each part:
- pure controlnet-0.15 background (no faces): **clean** -> the background scrub works at 0.15 (no
  ControlNet-shielding problem for Google on this image).
- composite with CodeFormer faces at **w0.7**: **SynthID DETECTED** -> high fidelity fuses too much of
  the original face signal (the watermark) through.
- composite at **w0.5**: **clean**. composite at **w0.3**: **clean**.
So the scrub-vs-fidelity threshold is between 0.5 and 0.7; **w=0.5 is the sweet spot** (highest
fidelity / best identity that still clears the oracle). Identity at w0.3-0.7 all looks like the same
person (the face is large enough), so the lower w costs little.

**This VALIDATES the corrected face-preservation approach** (and refutes my earlier "faces can't be
preserved" / FaceID conclusion): controlnet low-strength background scrub + CodeFormer-codebook face
re-synthesis at w~0.5 + feather composite = oracle-clean SynthID removal everywhere (background AND
faces), identity preserved, minimal overall degradation, zero original-pixel copying (CodeFormer faces
are codebook-generated). CodeFormer's discrete-codebook re-synthesis DOES scrub the pixel watermark,
but only when w is low enough that the decoder leans on the codebook rather than fusing the input
(watermark-carrying) features -- exactly the predicted fidelity-vs-scrub tension, with an empirical
clean threshold at w<=0.5.

**Production TODO (not built -- still a throwaway prototype):** (1) license -- CodeFormer is NTU S-Lab
(non-commercial); decide CodeFormer-as-user-installed-extra vs GFPGAN (Apache-2.0, re-verify it scrubs
at its fidelity setting); (2) wire a `--restore-faces` post-pass (detect -> restore w~0.5 -> feather
composite) onto the controlnet pipeline; (3) handle the MPS device bug (force CPU for the face model
or fix); (4) re-verify the w threshold on more images / vendors (w=0.5 confirmed on one Gemini group
photo only).

**Sources.** https://arxiv.org/abs/2206.11253 (CodeFormer) · https://github.com/sczhou/CodeFormer ·
https://arxiv.org/pdf/2401.07519 (InstantID) ·
https://openaccess.thecvf.com/content/WACV2024/papers/Suin_Diffuse_and_Restore... (region-adaptive) ·
https://arxiv.org/pdf/2504.12809 (saliency-aware watermark removal)

## Provenance

Hand-run primary-source pass, 2026-06-02. Sources fetched and quoted above; the central
make-or-break claim (structure-conditioned high-strength regeneration scrubs the watermark while
keeping text) is **unverified and explicitly flagged as the thing the local prototype must
measure** (against the manual Gemini SynthID oracle) — the literature supports removal (Findings 1, 2) and supports structure-preserving
regeneration (Finding 5) but never jointly validated text (Finding 3). No code change implied
until the prototype validates a Pareto cell on the SynthID oracle.