remove-ai-watermarks/docs/synthid-robust-identity-research.md

# SynthID-robust face identity for an SDXL removal pipeline (research)

> **Status (2026-06-08): retired.** Every approach described below was empirically
> tested and rejected -- see `docs/synthid-robust-identity-research-2026-06-08.md`
> "Empirical follow-up" for the final conclusion. The library no longer ships any
> face-restore extra: the cleaned image from the main controlnet 0.20 pass is the
> least-AI face state we can reach without re-introducing SynthID. This document
> is kept as historical record of the exploration.

**Question.** Which face identity-preservation mechanism for an SDXL img2img +
canny-ControlNet watermark-removal pipeline (denoise 0.20-0.30) is BOTH (a)
commercial-safe end-to-end and (b) does not re-introduce the SynthID pixel
watermark the removal pass just destroyed?

**Constraint.** raiw.cc is a paid service, so every component (adapter weights AND
the face embedder it conditions on AND any base model) must be Apache-2.0 / MIT /
BSD or otherwise clearly commercial-permitted. Non-commercial is disqualifying.

**One-line verdict.** Today there is **ONE** SDXL identity-conditioning stack that
is commercial-safe end-to-end: **PhotoMaker-V1** (Apache-2.0, identity encoded as a
fine-tuned OpenCLIP-ViT-H/14 image embedding -- NO InsightFace). Every other
candidate -- **including PhotoMaker-V2**, IP-Adapter FaceID, InstantID, PuLID,
Arc2Face -- inherits InsightFace's non-commercial model-pack license through an
ArcFace-class embedder and is therefore blocked for paid services, regardless of
the adapter's own license header. Below is the evidence per component and the
integration plan.

**Correction notice (2026-06-04).** An earlier version of this doc claimed
PhotoMaker-V2 was commercial-safe end-to-end. That was WRONG -- the V2 model card
phrase *"id_encoder includes finetuned OpenCLIP-ViT-H-14 and a few fuse layers"*
described one of TWO ID branches; the V2 source (model_v2.py) defines
`PhotoMakerIDEncoder_CLIPInsightfaceExtendtoken` whose forward takes an
ArcFace `id_embeds` from `insightface.app.FaceAnalysis`, and the upstream package
`__init__.py` imports InsightFace at module load. A Modal cert sweep caught this
empirically (`No module named 'insightface'` from `restore_faces_photomaker`). V1
is the correct commercial-safe target: its `PhotoMakerIDEncoder` (model.py)
forward takes only `(id_pixel_values, prompt_embeds, class_tokens_mask)` -- no
ArcFace branch -- so identity is CLIP-only.

**Status notice (2026-06-04, end of session).** Two commercial-safe paths were
tried and abandoned:

1. **PhotoMaker-V1** (commercial-safe by license but blocked by upstream compat).
   The cert sweep hit a cascade of upstream compatibility issues with the diffusers
   version we ship (0.38): missing `einops` declaration, missing `peft` declaration,
   default `pm_version='v2'` that mis-loads V1 weights into the V2 encoder, custom
   `id_encoder` left on CPU after `pipe.to(device)`, and a CFG-batch tensor-shape
   mismatch in the denoising loop (`Expected size 2 but got size 1`). 7 cascading
   fixes did not get the pipeline running end-to-end. The PhotoMaker `pipeline.py`
   header notes it was forked from diffusers v0.29.1; SDXL prompt-encoder handling
   changed significantly between 0.29 and 0.38.
2. **GFPGAN on the diffusion-CLEANED image** (commercial-safe, but no identity
   recovery). A one-line change made it SynthID-safe (input pixels are already
   clean, so the partial blend cannot transport the watermark), but visual
   inspection of the cert output showed it only polished the already-drifted face
   without actually restoring identity. Trade-off was real and the value too low
   to keep.

**The shipped path is PhotoMaker-V2** (`photomaker_restore.py`, the `photomaker`
extra). V2 uses a DUAL ID encoder (CLIP image features + ArcFace embedding),
which delivers true identity-from-embedding face regeneration. The cost is that
the ArcFace embedding comes from InsightFace's `antelopev2`/`buffalo_l` model
packs, which are released under a non-commercial / research-only license. **So
the shipped restore path is NON-COMMERCIAL.** raiw.cc and any other monetized
deployment must NOT install the `photomaker` extra. The CLI flag and module
docstring both call this out at every entry point.

A future commercial-safe path would need either (a) the PhotoMaker upstream to
land its diffusers 0.38 compat fix so V1 can run, or (b) an equally good
ArcFace-class face-recognition model released under a permissive license that
PhotoMaker-V2 can be retargeted to. Neither is on a near-term horizon as of
this writing.

## 1. Why identity-by-embedding (not by pixel) is the only SynthID-robust path

The pipeline regenerates pixels to destroy SynthID. Any identity-restoration that
is "faithful to the input pixels" (GFPGAN, CodeFormer, face-swap-by-blending, our
previous restore-on-original pass) reproduces the watermark, because SynthID is
engineered to be robust to fidelity-preserving transforms (resize, JPEG, partial
blend). Oracle-confirmed on a real Gemini face: controlnet @ 0.20/0.25 WITH the
GFPGAN restore pass left SynthID detected; the SAME controlnet @ 0.20 with
`--no-restore-faces` cleared it (clean A/B, see `docs/synthid.md` 5.5 and
`docs/controlnet-removal-pipeline-research.md`).

The only mechanism that can preserve identity AND not re-introduce SynthID is to
carry identity in a SEMANTIC EMBEDDING (a vector that encodes "who is in this
picture") and use it to CONDITION a fresh generation -- the pixels are new, so
the watermark is not transported. Two embedding families exist in practice:

- **ArcFace-class face-recognition embeddings** (the InsightFace family). Used by
  IP-Adapter FaceID, InstantID, PuLID, Arc2Face. Highest identity fidelity, but
  the embedder weights are non-commercial.
- **CLIP image embeddings of a face crop**. Used by PhotoMaker (and the original
  IP-Adapter image variant). Lower identity fidelity at small scale than ArcFace,
  but the encoder (OpenCLIP-ViT-H/14, MIT) is commercial-safe.

## 2. License table (verified against primary sources, 2026-06-04)

| stack | adapter weights | identity encoder | end-to-end commercial-safe? |
|---|---|---|---|
| **PhotoMaker-V1** | **Apache-2.0** ([HF][pmhf]) | **OpenCLIP-ViT-H/14 (MIT)** finetuned, identity from `PhotoMakerIDEncoder` (`model.py`); forward takes only ``(id_pixel_values, prompt_embeds, class_tokens_mask)`` -- no ArcFace branch | **YES** |
| PhotoMaker-V2 | Apache-2.0 (adapter) ([HF][pm2hf]) | DUAL encoder: OpenCLIP-ViT-H/14 AND InsightFace antelopev2/buffalo_l -- `PhotoMakerIDEncoder_CLIPInsightfaceExtendtoken` (`model_v2.py`) forward takes `id_embeds` from `insightface.app.FaceAnalysis`, and `photomaker/__init__.py` imports InsightFace at module load | NO -- InsightFace pack is non-commercial |
| IP-Adapter FaceID | non-commercial per model card: *"AS InsightFace pretrained models are available for non-commercial research purposes, IP-Adapter-FaceID models are released exclusively for research purposes and is not intended for commercial use"* ([HF][ipafhf]) | InsightFace antelopev2 (non-commercial for the model pack) | NO -- both layers block |
| InstantID | Apache-2.0 (adapter only) ([HF][insthf]) | requires InsightFace antelopev2 face-analysis at runtime (`FaceAnalysis(name='antelopev2', ...)` per the README usage snippet, [HF][insthf]) | NO -- embedder pack is non-commercial |
| PuLID | apache-2.0 (HF model metadata, [HF][pulidhf]) | depends on InsightFace face-analysis for ArcFace embedding (per the upstream README; PuLID's own card is sparse and the GitHub README documents the InsightFace install step) | NO -- same embedder issue as IP-Adapter FaceID |
| Arc2Face | MIT (HF model metadata, [HF][arc2hf]) | uses `insightface.app.FaceAnalysis` to extract the ArcFace embedding ([HF][arc2hf]); also based on SD-v1-5 (NOT SDXL) | NO -- non-commercial embedder + not SDXL |

**The crux is InsightFace.** InsightFace explicitly splits its license: *"Code is
MIT licensed; models require separate commercial licensing"* and frames the
pretrained packs as *"Commercial licensing for InsightFace's open-source model
packages"* requiring users to *"obtain commercial usage rights for model
packages"* ([insightface.ai][iflic]). antelopev2 and buffalo_l fall under the
model-pack license, not MIT. So any stack that calls
`insightface.app.FaceAnalysis(name='antelopev2', ...)` to compute its ArcFace
embedding is blocked by default, REGARDLESS of the adapter's own Apache header
above it. This is the same reason IP-Adapter FaceID's card flags itself
non-commercial.

(Note on PuLID's HF metadata: the model card declares apache-2.0 for the adapter
weights but the upstream repo's quickstart requires the InsightFace package to
extract the ID embedding. So PuLID's adapter license is permissive; the BLOCKER
is the embedder it expects at runtime. This is the same trap as InstantID.)

[pmhf]: https://huggingface.co/TencentARC/PhotoMaker
[pm2hf]: https://huggingface.co/TencentARC/PhotoMaker-V2
[ipafhf]: https://huggingface.co/h94/IP-Adapter-FaceID
[insthf]: https://huggingface.co/InstantX/InstantID
[pulidhf]: https://huggingface.co/guozinan/PuLID
[arc2hf]: https://huggingface.co/FoivosPar/Arc2Face
[iflic]: https://www.insightface.ai/solutions/face-recognition-licensing

## 3. Is there a commercial-safe ArcFace replacement?

Short answer: **no clean drop-in**. The widely deployed pretrained ArcFace packs
(antelopev2, buffalo_l, glint360k) come from InsightFace and are non-commercial.
ArcFace as an ARCHITECTURE is published in a paper, so retraining is legally fine,
but you would need:

- a commercial-licensed training dataset (the big public ones -- MS-Celeb-1M,
  Glint360K, WebFace -- carry research-only or licensing-uncertain restrictions);
- compute + time to train an ArcFace-class model on the legal dataset;
- the result would be a one-off effort, not a maintained dependency.

For a removal service this is a multi-month side project that delivers what
PhotoMaker already gives us with one pip install. So the practical answer is to
take the CLIP-embedding path (PhotoMaker-V1; V2 adds InsightFace and is non-commercial), accept the identity-fidelity
trade-off, and revisit ArcFace later if quality is insufficient.

## 4. Does an identity embedding leak SynthID?

This is the load-bearing assumption of the whole approach. The argument:

- SynthID is a low-amplitude, perceptually-invisible pixel watermark engineered
  to be robust to "fidelity-preserving" transforms (it survives JPEG, resize,
  crop, color, noise at >=99% TPR -- see arXiv:2510.09263 referenced in
  `docs/synthid.md`).
- A face-recognition / CLIP-image embedding is by design INVARIANT to such low-
  amplitude pixel changes (compression, brightness, small noise should not change
  "who is in the photo"). That is the whole training objective.
- Therefore the embedding extracted from a watermarked face vs. the same face
  cleaned should be ~identical -- the embedding cannot CARRY the watermark
  pattern, only the identity, because the watermark sits in exactly the
  dimensions the embedding learned to discard.

**MEASURED 2026-06-04 — hypothesis confirmed.** Ran a low-amplitude
perturbation sweep on 31 face crops (3 photoreal originals: gemini_3, gemini_4,
openai_3 grid), comparing `cos(embedding(orig), embedding(perturbed))` for OpenCLIP-
ViT-H/14 (laion2B-s32B-b79K, the same OpenCLIP-ViT-H/14 encoder PhotoMaker V1 and V2 both finetune for CLIP-side identity):

| perturbation | mean cos | min | max |
|---|---|---|---|
| **synthid_proxy** (±2 LSB low-freq noise, σ=4 px Gaussian carrier — same regime SynthID hides in) | **0.9977** | 0.9937 | 0.9996 |
| noise3 (Gaussian σ=3, full-spectrum) | 0.9541 | 0.9055 | 0.9825 |
| jpeg90 (SynthID survives this) | 0.9280 | 0.8806 | 0.9566 |
| blur1 (Gaussian σ=1) | 0.9139 | 0.8103 | 0.9875 |
| jpeg70 | 0.8945 | 0.8125 | 0.9603 |
| (self check: identical crop) | 1.0000 | 1.0000 | 1.0000 |

The SynthID-magnitude perturbation moves the embedding by **0.002** (cosine 0.9977),
an order of magnitude less than JPEG90 — which SynthID survives at >=99% TPR by
design. So the embedding cannot carry the watermark pattern: its discriminative
signal is in dimensions the SynthID payload does not occupy. PhotoMaker-V1
conditioned on a watermarked face will see ~the same identity vector as if
conditioned on a clean face of the same person, so the freshly generated face
inherits the identity, not the watermark.

A first, naive smoke run measured `cos(orig, SDXL-cleaned)` instead — that test is
about diffusion drift, not watermark invariance (diffusion at strength 0.20-0.30 is a
much larger perturbation than SynthID), so its 0.56-0.93 spread is the identity
drift the PhotoMaker pipeline is meant to fix in the first place. The
synthid_proxy result above is the one that actually answers the load-bearing
question. Script: `/tmp/identity_smoke/test2_proxy.py` (not committed; reproducible
from the test set + this doc).

## 5. PhotoMaker-V1 properties for our pipeline

- **SDXL-native.** PhotoMaker v1 and v2 target Stable Diffusion XL; the pipeline
  is a stacked-ID embedding fused into SDXL's cross-attention via the fuse layers
  bundled in the released weights.
- **Identity from a SINGLE reference image works** but the method was designed
  for "stacked" multi-reference; with one image identity fidelity is lower than
  with 3-4, and a service has only one (the upload). This is the failure mode to
  guard.
- **Compatibility with img2img + canny ControlNet.** PhotoMaker is typically
  exposed in txt2img workflows in the upstream demo. SDXL img2img + ControlNet
  is the same denoising backbone, so the cross-attention injection works the same
  way; community examples on Diffusers and ComfyUI confirm PhotoMaker stacks with
  ControlNet. Validate this on a representative image before adopting.
- **Failure modes to expect:**
  - identity drift on small / multi-face groups (the 9-face grid case);
  - "plastic" / over-smoothed faces if PhotoMaker's identity weighting is high
    while the img2img strength is low;
  - canny ControlNet conditioning can fight the ID embedding (edges of the
    ORIGINAL face vs identity of the SAME person regenerated) -- expect to tune
    `controlnet_conditioning_scale` down a notch on photoreal faces;
  - PhotoMaker was trained on a celebrity-skew distribution; real-user faces
    (especially non-white, non-Western, elderly, children) may have lower
    fidelity. Measure on the real upload distribution.

## 6. Integration cost (rough)

- New deps: `diffusers` already in the gpu extra; PhotoMaker ships as a `.bin`
  loaded via `pipeline.load_photomaker_adapter(...)`. The OpenCLIP encoder is the
  same one diffusers already pulls. No new heavy pip dep.
- Weight download: PhotoMaker-V1 weights are ~3 GB. Add to the Modal HF volume
  alongside SDXL.
- VRAM: SDXL + canny ControlNet + PhotoMaker-V1 fits comfortably in A100-40GB.
- Latency: a few extra seconds on cold start (load PhotoMaker), negligible per
  request after warm-up.
- No InsightFace install: huge win for `restore` extra's basicsr/numpy hell --
  this path simply does not touch that ecosystem.

## 7. Recommended path

1. **Embedding-invariance smoke test FIRST** (one afternoon, no codegen):
   - For ~10 OpenAI / Gemini watermarked faces, compute OpenCLIP-ViT-H/14
     embeddings; for the same images after our SDXL `default` pass at the
     certified strength, compute the embeddings again; assert mean cosine
     similarity > ~0.95.
   - If yes -> the embedding does not carry SynthID, proceed.
   - If no -> the assumption is wrong; PhotoMaker would re-introduce the
     watermark. Stop and reconsider.
2. **PhotoMaker-V1 prototype** in the existing `controlnet` pipeline:
   - Mirror the `_load_controlnet_pipeline` path: add a PhotoMaker variant that
     loads SDXL + canny ControlNet + PhotoMaker adapter on the same engine.
   - Extract the OpenCLIP face embedding from the watermarked face crops (use
     OpenCV YuNet, already bundled for `auto`, to find the face boxes).
   - Pass the embedding as PhotoMaker's `id_embeds` to the SDXL pipeline; run
     img2img at the certified strength (0.20 OpenAI, 0.30 Gemini-capped-1536)
     with the canny edge map.
3. **Oracle validation** on the cert sweep: run the new PhotoMaker variant
   through `raiw-app/modal_cert.py` over the same 6 image set, certify on the
   per-vendor oracles. Expected: SynthID cleared (the regeneration is the same)
   AND identity recovered (the embedding adds it back).
4. **Honest exit criteria.** Ship only if BOTH oracle reads clean AND a small
   user-perception test on real uploads says "looks like me". If identity is
   still too soft on small faces -> add stacked-reference (multiple crops of the
   same upload at different scales) before reaching for a non-commercial
   embedder.

## 8. What we are NOT doing, and why

- **No InsightFace.** Non-commercial for model packs (see License table).
- **No CodeFormer.** Non-commercial.
- **No GFPGAN on the original image.** It re-introduces SynthID
  (oracle-confirmed).
- **No GFPGAN on the cleaned image.** It cannot RECOVER identity that the
  diffusion pass already drifted -- it can only smooth/sharpen whatever face is
  already there. Useful as cosmetic polish, not as identity restoration.
- **No retraining of an in-house ArcFace.** Out of scope for a removal service.

---

## Process note

The deep-research harness was run but its verifier subagents failed to call
`StructuredOutput` (same harness bug as the prior 2026-05-XX run), so its synthesis
was unusable. The license claims above were verified by directly fetching the HF
model cards and the InsightFace licensing page and quoting them; the
embedding-invariance argument is mechanistic and explicitly flagged as not yet
measured (it is the first integration step). Do not treat the deep-research
output as ground truth for this file.