remove-ai-watermarks/docs/chroma1-engine-research.md

Chroma1 engine research (2026-08-29/30)

Research archive. This page records experiments and decisions from the dates above. Current package behavior is defined by the supported signals, known limitations, and module internals. Nothing here is shipped.

Cited experiment behind issue #88 ("Please integrate Flux 2"): FLUX.2 has no native strength img2img anywhere (BFL reference code or diffusers; its image inputs are reference conditioning and outputs start from pure noise), so the requestable scrub lane in the FLUX family is Chroma1 -- the FLUX.1 architecture re-trained on open data, Apache-2.0 on weights, with a native ChromaImg2ImgPipeline strength parameter. This page records what was measured before any integration decision.

Setup

All Chroma1 runs: lodestones/Chroma1-HD bf16 on Modal H100, seed 0, the neutral scrub prompt (high quality, sharp, detailed, faithful to the original / blurry, lowres, distorted text, garbled text, artifacts), guidance 5.0, and step compensation: Diffusers truncates the step COUNT (int(steps * strength)), so the requested count is scaled to always spend 4 effective denoising steps, the same semantics as sdxl_zimage_pipeline.requested_steps. Without it, strength 0.05 runs a single step and the floor reads artificially high (the first, uncompensated sweep put the OpenAI boundary at (0.05, 0.10]; compensated it is (0.05, 0.06]).

Two Flux-family traps found while wiring the prototype:

• without explicit width/height, the img2img pipeline silently generates 1024x1024 regardless of input aspect;
• dimensions floor to the /16 latent-patch grid, so the result must be resized back to the input size for pixel-exact comparison.

Scripts: scripts/chroma_scrub_prototype.py (first sweep), scripts/engine_quality_price_probe.py (head-to-head with the production qwen-zimage path), scripts/engine_cohort_calibration.py (the cohort ladders below), scripts/chroma_meta_high_rungs.py (the Meta rungs above 0.08). Generated outputs stay outside the repository (out/, gitignored); the durable verdicts are the tables on this page.

Measured floors

Oracle per cohort: OpenAI via openai.com/verify (SynthID read separately from C2PA); Meta via the anonymous meta.ai/identification oracle. Each fixture's rungs were checked ascending and stopped at the first clean verdict.

cohort	fixture	detected at	clean from
OpenAI	typography card 1122x1402	0.05	0.06
OpenAI	9-face grid 1448x1086	0.06	0.075
OpenAI	full-pipeline quality fixture 1122x1402	0.05	0.06
Meta	gen_lighthouse_watercolor 1600x1600	0.08	0.10
Meta	gen_fox_forest 1600x1600	0.06	0.08
Meta	gen_night_city 1920x1280	0.045	0.06
Meta	gen_studio_mug 1600x1600	0.03	0.045
Meta	gen_text_poster 1600x1600	--	<=0.03

Derived operating points by the shipped rule (worst first-clean boundary plus one observed cross-source spread):

• OpenAI: 0.09 (0.075 + (0.075 - 0.06)). Against qwen-zimage 0.07675 and sdxl-zimage 0.15. The typography boundary (0.05, 0.06] is seed-stable across seeds 0, 1, 2.

• Meta: 0.17 (0.10 + (0.10 - 0.03)), oracle-verified clean on the worst fixture at 0.17. Against qwen-zimage 0.1. Chroma needs ~1.7x qwen's strength on Content Seal.

• Google: 0.40, measured 2026-08-30 across all four fixtures. A metadata-stripped original verified as pixel-DETECTED through the @synthid Verify AI agent (the C2PA-carrying original answers about C2PA instead, which is not a pixel verdict).

  fixture	detected at	clean from
  Gemini_Generated_Image_633uuy	0.20	0.25
  Gemini_Generated_Image_akdbei	0.20	0.25
  Gemini_Generated_Image_y48j3c	0.08	0.12
  Gemini_Generated_Image_3mc4t9	0.08	0.12

  Derived by the shipped rule: worst first-clean 0.25 + spread (0.25 - 0.12 = 0.13) = 0.38, rounded up to the measured rung 0.40, then oracle-verified clean on the worst fixture. Against qwen-zimage 0.27: Chroma needs ~1.5x the strength on Google SynthID. At 0.15-0.20 the tool still reads positive while its wording degrades from "most or all" to "part of", so the boundary band is real signal decay, not noise.

  Oracle mechanics learned while bracketing: the quota is per ACCOUNT and additionally dedupes SIMILAR images ("quota for checking images similar to this one"), roughly three checks per account window; switching accounts inside one browser works. A fresh Playwright context with the same Google session cookies (exported browser-wide via CDP Storage.getCookies, never through the transcript) carries working sign-in and its own check budget. The agent's own instructions leak into the answer on some renders and include an abstention rule for text-dominated images - always read the TOOL outcome, never the prose.

• Microsoft: 0.125, measured 2026-08-30 on the SAME three valid Paint carriers behind the qwen floor (staged under out/cohort-calibration/microsoft/ with pixel-identical metadata-stripped controls; they are real raiw-corpus user uploads and are never committed). Controls re-verified Microsoft AI detected on the day of the run. Against qwen-zimage 0.15: Chroma needs LESS strength on InvisMark.

  source (Paint)	detected at	clean from	qwen first-clean
  paint-1 (47d87e99, 1536x1024)	0.06	0.08	0.055
  paint-2 (a579fb70, 1536x1024)	--	<=0.04	0.04125
  paint-3 (883191b5, worst qwen source)	0.06	0.08	0.095

  Derived by the shipped rule: 0.08 + (0.08 - 0.04) = 0.12, rounded to the measured rung 0.125, which was then oracle-verified clean on both worst sources. Oracle: the public ai.azure.com/nextgen/validate page; its Inconclusive verdict is the site's watermark-negative, weaker than an API-level Watermark: false, the same caveat the qwen cohort carries.

Fidelity at the floors (scripts/fidelity_metrics.py)

qwen-zimage rows ran through the package's own QwenZImagePipeline on the same H100 (production path, both stacks resident, prompt cache warm).

OpenAI typography (chroma 0.06 vs qwen 0.07675):

	CER	img LPIPS	SSIM	PSNR
qwen-zimage	0.241	0.083	0.834	27.7
chroma1	0.138	0.049	0.881	31.2

OpenAI full-pipeline fixture (chroma 0.09 vs qwen 0.07675): chroma wins every metric (CER 0.155 vs 0.259, LPIPS 0.056 vs 0.094, PSNR 29.6 vs 26.1).

OpenAI 9-face grid (chroma 0.075 global-only vs qwen 0.07675 with its Z-Image face stage):

	ID cos	face LPIPS	img LPIPS	SSIM	PSNR
qwen-zimage	0.857	0.051	0.065	0.865	31.1
chroma1 (no face stage)	0.788	0.067	0.060	0.890	32.4

The face stage regenerates from the ORIGINAL crops and is engine-independent, so an integrated chroma-zimage would composite the same faces over the chroma global; whether that closes the identity gap is an integration-time measurement, not a given.

Meta at the respective floors (chroma 0.17 vs qwen 0.1): qwen wins perceptual fidelity on textured content (LPIPS: fox 0.163 vs 0.211, lighthouse 0.173 vs 0.248, night city 0.187 vs 0.277), chroma wins the text poster decisively (LPIPS 0.008 vs 0.012, CER 0.000 vs 0.021) and the studio mug is a split. The extra 0.07 of strength costs real fidelity on photoreal content.

Google at the respective floors (chroma 0.40 vs qwen 0.27, qwen row includes its Z-Image face stage):

fixture	qwen LPIPS/SSIM/ID cos	chroma LPIPS/SSIM/ID cos
633uuy	0.405 / 0.577 / -	0.499 / 0.576 / - (CER 1.000 vs 0.333)
akdbei	0.440 / 0.552 / -	0.540 / 0.557 / - (CER 0.796 vs 0.367)
y48j3c (18 faces)	0.436 / 0.608 / 0.801	0.560 / 0.548 / 0.279
3mc4t9 (5 faces)	0.422 / 0.529 / 0.921	0.562 / 0.461 / 0.143

qwen wins every axis that matters on every fixture; at 0.40 the Chroma global regeneration itself collapses face identity (0.14-0.28 cosine) and destroys dense text (CER up to 1.0). The extra 0.13 of strength is simply too much regeneration. This is the Meta pattern amplified.

Microsoft at the respective floors (chroma 0.125 vs qwen 0.15): chroma wins EVERY metric on EVERY source while scrubbing at lower strength --

source	CER q/ch	img LPIPS q/ch	SSIM q/ch	PSNR q/ch
paint-1	0.741 / 0.241	0.164 / 0.106	0.742 / 0.847	24.8 / 29.3
paint-2	0.826 / 0.391	0.253 / 0.108	0.536 / 0.633	22.5 / 25.5
paint-3	0.000 / 0.000	0.038 / 0.020	0.762 / 0.953	36.3 / 42.2

The Microsoft cohort therefore behaves like the OpenAI cohort, not like Meta: for InvisMark carriers Chroma1 is the better engine at a lower operating point.

Price (warm H100 container)

	qwen-zimage	chroma1
seconds per image	2.44	3.62
VRAM peak	31.9 GiB	29.2 GiB
USD per 1000 images ($3.95/h)	$2.68	$3.97
weights in cache	~70 GB	~28 GB

The chroma number includes a T5 encode per call; the shipped qwen path caches prompt embeddings, and the same treatment would put chroma near 1.5-2 s/image. Both stacks fit an 80 GB card; chroma's smaller footprint likely fits smaller cards untested.

Matched-strength comparison: the floors hide the engine (2026-08-30 addendum)

The floor-based tables above compare chroma at its DERIVED floor (worst case

• spread) against qwen at its shipped floor - different perturbation levels. At the strength where each fixture ACTUALLY turns clean (chroma first-clean rungs, all oracle-verified above), the picture reverses:

Google (chroma@first-clean vs qwen@0.27):

fixture	chroma rung	LPIPS q/ch	SSIM q/ch	PSNR q/ch	CER q/ch	ID q/ch
633uuy	0.25	0.405/0.375	0.577/0.673	18.8/22.3	0.333/0.667	-
akdbei	0.25	0.440/0.409	0.552/0.650	18.6/22.5	0.367/0.163	-
y48j3c	0.12	0.436/0.329	0.608/0.700	21.0/25.8	1.0/1.0	0.801/0.611
3mc4t9	0.12	0.422/0.318	0.529/0.644	17.7/21.3	0.443/0.229	0.921/0.410

Meta (chroma@first-clean vs qwen@0.1):

fixture	chroma rung	LPIPS q/ch	SSIM q/ch	PSNR q/ch	CER q/ch
lighthouse	0.10 (equal)	0.173/0.097	0.783/0.886	25.6/30.8	-
fox	0.08	0.163/0.083	0.778/0.907	25.6/31.4	-
night_city	0.06	0.187/0.092	0.748/0.893	23.9/29.6	0.850/0.250
studio_mug	0.045	0.024/0.009	0.984/0.992	39.0/37.8	-
text_poster	0.03	0.012/0.002	0.984/0.994	34.0/41.2	0.021/0.007

At effective-clean strength Chroma1 wins whole-image fidelity on essentially every fixture (often by 4-6 dB) and text on five of six; qwen's remaining edge is face identity, which its Z-Image face stage buys (0.80-0.92 vs 0.41-0.61 cosine) and a chroma-zimage profile would inherit on top of its own global pass.

Why the floors still matter: a shipped strength must clear the worst unseen content, and Chroma1's per-fixture boundaries scatter much wider (Google 0.12-0.25, Meta 0.03-0.10) than the spread the flat floors were built to absorb - so its derived floors (0.40 / 0.17) pay a large margin penalty and at those floors the extra strength genuinely destroys quality. The engines differ less in "quality per strength" than in how predictable their boundary is. The integration lever this exposes: a content-adaptive strength policy for chroma-zimage (the project already ships resolution-adaptive denoise for qwen) could capture chroma's low per-content boundaries; the two hard Google fixtures are both dense-text cards and the easy ones are photoreal, so a content-class signal may predict the rung.

Pre-ship validation (2026-08-30)

With the chroma-zimage profile implemented (_internal/chroma_zimage_pipeline.py on TwoStageZImagePipeline, floors in watermark_profiles), two ship-gate checks were run:

1. Seed stability at the worst Meta boundary. lighthouse @ 0.10 with seeds 1 and 2: both CLEAN on the meta.ai oracle. The Meta boundary is not a seed-0 fluke. The Google seed probes (633uuy @ 0.25, seeds 1-2) were generated but the oracle's per-image-class quota blocked them on every available account; they remain queued, and the shipped 0.40 Google floor carries 0.15 of margin over that boundary regardless.
2. Full-path face-stage interaction. The complete profile run (Chroma1 global at the shipped Google floor 0.40, then the inherited YuNet/SAM/Z-Image face stage) on the 5-face fixture 3mc4t9: CLEAN on the @synthid oracle. The composited face regions do not re-introduce a detectable signal on top of the floor.

Generation script: scripts/chroma_preship_validation.py; outputs under out/preship-chroma/ (gitignored).

Verdict

Engine quality is cohort-dependent, and after measuring all four cohorts the split is even and sharp:

• On OpenAI-provenance content Chroma1 is better at every measured fidelity axis at its floor (0.09 vs 0.07675), for comparable money.
• On Microsoft InvisMark content Chroma1 is better at every measured fidelity axis at a LOWER floor than qwen (0.125 vs 0.15).
• On Meta Content Seal Chroma1 needs 1.7x the strength (0.17 vs 0.1) and loses LPIPS on textured content at its floor.
• On Google SynthID Chroma1 needs ~1.5x the strength (0.40 vs 0.27) and loses every fidelity axis on every fixture, with face identity collapsing at 0.40 (0.14-0.28 cosine vs qwen's 0.80-0.92).

A single-engine chroma-zimage default at a flat per-vendor floor is NOT supported: Google and Meta floors land too high and the quality at those floors is worse than qwen's. But the matched-strength addendum changes what IS supported: at the strength each image actually needs, Chroma1 regenerates better than qwen almost everywhere except face identity (which the shared face stage would supply). The two honest integration options:

1. Per-cohort engine router at flat floors: chroma for OpenAI/Microsoft, qwen for Google/Meta - conservative, ships on the measurements as they stand.
2. chroma-zimage with a content-adaptive strength policy that predicts the per-image rung - potentially better everywhere, but the predictor is new research (the measured signal so far: dense-text content needs the high rungs, photoreal the low ones) and needs its own oracle validation before any ship.

Both need the seed-sensitivity and face-stage composition checks called out above before shipping. Both checks have now been run (Pre-ship validation above): the Meta boundary is seed-stable across seeds 0-2 and the full profile path stays clean through the face stage on the shipped Google floor. The profile ships with option 1's floors; option 2's adaptive policy remains the documented follow-up.

Oracle session notes

• openai.com/verify throttles anonymous sessions through Cloudflare Turnstile after ~15 rapid checks; a fresh browser context clears it. The programmatic POST /v1/content_provenance_checks API is the better oracle (separate SynthID/C2PA outcomes) but the checked OPENAI_API_KEY returned 401 org-wide, including /v1/models.
• meta.ai/identification enforces a sliding per-IP window: a burst of ~15 checks exhausted it twice; it reopens minutes later, and the page sometimes overstates this as a daily limit. Pacing of roughly 2-3 checks per window worked.
• The Gemini app oracle requires the user's logged-in session; driving the user's live Chrome while it is in active use was refused (wrong-tab risk), so those 20 checks remain queued.