Commit Graph
113 Commits
Author SHA1 Message Date
Victor KuznetsovandClaude Opus 5 84583dd479 Drop the unproven claim that Gemini removal is resolution-sensitive
The document asserted it in two places and recommended capping Gemini at 1536
with 0.30, or native-calibrating to ~0.35+. Nothing measured that, and the one
relevant measurement points the other way: the 2026-06-14 deployed-worker re-test
cleared Gemini at 0.15 on two NATIVE 2816x1536 images, the same rung as capped
1536. The document already recorded that as contradicting the "native >= 0.30"
guess, then kept the guess anyway in the historical-certification paragraph and
restated it as fact in the strength floors.

Replaced with what was measured, plus an explicit statement that the direction is
unproven and the low-resolution end has never been through the Gemini oracle on
any pipeline.

Also removes the same appeal from the qwen-zimage denoise-boundary note added
earlier in this branch, which had used the unproven trend as reassurance that the
bottom of the adaptive curve is safe. It is not reassurance; it is an open
question, and it is now written as one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 19:49:44 -07:00
Victor KuznetsovandClaude Opus 5 56d7aff3a1 Record the qwen-zimage global denoise boundary against the Gemini oracle
A ladder on one native 2816x1536 Gemini original at seed 0, verified through the
Gemini app, puts the boundary between 0.08 and 0.10: 0.154, 0.12 and 0.10 read
clean, 0.08 reads SynthID FOUND. Fidelity rises monotonically all the way down,
so 0.10 buys +1.54 dB whole-image and +0.98 dB inside the face boxes over the
0.154 the profile ships for that size.

Recorded with the two constraints that stop it being acted on directly. It
brackets rather than calibrates - one image, one seed, and shipping the lowest
clean rung means shipping at the measured cliff edge. And the untested end is the
bottom, not the top: every Gemini oracle fixture is 2816x1536, so the Google-side
certification only ever covered 0.154, while the curve sends sub-1 MP images to
0.084-0.094. The resolution trend already recorded in this document says lower
processing resolution needs less strength, which is the shape the curve has, but
that is an inference and no small Gemini original has ever been through the
oracle. Downscaling is valid test material since SynthID survives it by design.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 19:46:12 -07:00
Victor KuznetsovandClaude Opus 5 39523aba09 Serve the static prompt embeddings from a persistent cache
Both qwen-zimage stages prompt with module constants, and at CFG 1.0 DiffSynth's
PipelineUnitRunner reuses the positive embedding for the negative side rather
than encoding it, so exactly one embedding per stage is ever computed. Persist it
and neither text encoder has to be loaded at all.

Measured on an H100 volume: this drops 15.45 GiB (Qwen2.5-VL) and 7.49 GiB
(Z-Image) of an 87.6 GiB per-request read, worth a median 11.76 s and 4.10 s of
load time paired within five containers. A nine-face fixture returned
sha256 c8567e11077de32a both with and without the cache, so the output is
byte-identical and the provider-oracle clearance is untouched.

The cache key carries the cache version, model id, pipeline output params and the
exact prompt, so a model bump or a prompt edit recomputes instead of reading a
stale embedding. The write is atomic because a torn file must never read back as
a hit, and a miss after the text encoder was already dropped raises rather than
calling a model that is not loaded.

_model_cache_dir now prefers HF_HOME: on a scale-to-zero runner that is the only
persistently mounted path, so anything below it is re-derived every request. The
YuNet download follows the same root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-02 15:36:55 -07:00
Victor KuznetsovandClaude Opus 5 36e3743289 Keep the qwen-zimage global stack resident on a card that can hold it
Port of the same change made on the v0.20.1 line, reapplied here because the
package layout moved under _internal/ in the meantime.

The mandatory Qwen stack was configured to offload to disk unconditionally.
DiffSynth implements that by dropping the weights to the meta device and
re-reading every parameter through its DiskMap on the next onload, and the
pipeline moves between text encoder, transformer and VAE on every pass, so
each generation paid a full model reload. That is the right trade on a
consumer card, where it is what makes a 20B model runnable at all, and pure
waste on a card that can simply hold the stack.

Residency is now resolved from total VRAM, mirroring how the optional
Z-Image face stack is already gated. Above the floor the config passes no
"disk" value anywhere, which is what actually disables the behavior:
DiffSynth latches disk_offload once from offload_dtype, so pointing every
device at CUDA while leaving the sentinel would keep both the meta-drop and
the re-read.

Measured on an H100: a warm global pass went from 37.3s at 0.8 GiB resident
to 2.2s at 28.7 GiB, with both stacks resident peaking at 48.0 GiB of 79.2.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-01 21:22:52 -07:00
Victor Kuznetsov a6c0c1c6f0 Rewrite internal watermark pipeline and preserve behavior 2026-07-31 16:53:41 -07:00
Victor Kuznetsov 7c922e5133 Preserve one encoded frame per input frame 2026-07-31 15:31:00 -07:00
Victor Kuznetsov 7e63856a90 Split video encoding from audio muxing 2026-07-31 15:22:41 -07:00
Victor Kuznetsov 9b9f9c6ced Open source input before frame pipe 2026-07-31 15:07:15 -07:00
Victor Kuznetsov 7818e60824 Prevent ffmpeg stderr backpressure 2026-07-31 14:59:57 -07:00
Victor Kuznetsov 0fac8cd000 Bound ffmpeg filter threads 2026-07-31 14:43:01 -07:00
Victor Kuznetsov 166dab9d03 Bound ffmpeg video encoder threads 2026-07-31 14:33:50 -07:00
Victor Kuznetsov 1fd859b58d Prevent ffmpeg pipe probe deadlock 2026-07-31 12:05:42 -07:00
Victor Kuznetsov de3ed75de5 Merge main into video watermark pipeline 2026-07-31 11:13:59 -07:00
Victor Kuznetsov 5767c1d7b3 Finalize video package boundary and product wording 2026-07-31 10:52:44 -07:00
Victor Kuznetsov 6315b58628 feat: complete product video watermark pipeline 2026-07-31 10:42:39 -07:00
Victor Kuznetsov 08dc078d91 Release 0.22.0 with composable feature extras 2026-07-31 10:39:13 -07:00
Victor Kuznetsov 8975916b11 Harden automatic video watermark removal for product use 2026-07-30 18:36:04 -07:00
Victor Kuznetsov 9c9e81c756 Compact CLAUDE.md and route development guidance 2026-07-30 18:32:25 -07:00
Victor Kuznetsov 07b678fa4c Add Hailuo, Kling, AVI, and FLV video coverage 2026-07-29 23:30:33 -07:00
Victor Kuznetsov 63cde7f32a Add streaming video SynthID regeneration 2026-07-29 22:03:23 -07:00
Victor Kuznetsov c9b8585105 Add video SynthID experiment harness 2026-07-29 21:03:46 -07:00
Victor Kuznetsov 41b6fd0682 Add Seedance and Dola video removal 2026-07-29 17:56:52 -07:00
Victor Kuznetsov f330dd9c94 Add video metadata and visible watermark removal 2026-07-29 17:28:11 -07:00
Victor Kuznetsov 7c9bec0eba Ignore collector diagnostics in metadata evidence 2026-07-29 16:46:42 -07:00
Victor Kuznetsov 4ef180541f Add external metadata evidence ingestion 2026-07-29 14:32:25 -07:00
Victor Kuznetsov 408c4c9d5d Separate provenance extraction and detection, release 0.21.0 2026-07-29 13:13:39 -07:00
Victor Kuznetsov 59823b2eaf Recover malformed metadata containers, release 0.20.2 2026-07-28 12:12:26 -07:00
Victor Kuznetsov 1d333fe855 Use a dedicated ComfyUI release token 2026-07-26 12:11:35 -07:00
Victor Kuznetsov 5ea160f4ce Include ComfyUI in library releases 2026-07-26 11:56:09 -07:00
Victor Kuznetsov e562b440db Improve Qwen face fidelity, release 0.20.1 2026-07-26 11:39:29 -07:00
Victor Kuznetsov ab525c1b90 Add global-only preload for qwen-zimage 2026-07-26 10:52:26 -07:00
Victor Kuznetsov 685f0679bd Automate conda recipe synchronization 2026-07-25 21:57:22 -07:00
Victor Kuznetsov d21dc62bbb Sync conda recipe with v0.20.0 2026-07-25 21:32:52 -07:00
Victor Kuznetsov 03cd00f132 Restructure documentation, validate metadata removal, consolidate assets 2026-07-25 21:08:04 -07:00
Victor Kuznetsov 214c9bb3e7 Add Tencent Yuanbao visible watermark removal 2026-07-25 20:52:15 -07:00
Victor Kuznetsov b8b69d93be Finish CPU offload support on current main 2026-07-25 18:44:44 -07:00
Victor Kuznetsov b6204a24ce Add high-fidelity Qwen Z-Image removal pipeline 2026-07-25 16:53:22 -07:00
Victor Kuznetsov f1a5eecf98 Register RunningHub, Baidu, and LibLibAI visible marks; park Qingyan and MiniMax (measured)
New engines, each calibrated on its TC260 USCC cohort and validated by a
full-corpus sweep (42009 files):
- runninghub: top-left corner (new corner="tl"), faint mid-gray text via
  the new raw-grayscale "gray" detection front-end, anchor-position gate
- baidu: text-run-only template (pill is a bright-blob magnet), load-bearing
  Doubao+Qwen rival margins, corner-extended footprint for the white tag
- liblib: bottom-center (new corner="bc"), Arial silhouette (font is the
  discriminative lever against latin UI text), logo-extended footprint

Qingyan parked (no clean-arm separation at any render/box), MiniMax/Hailuo
parked (1 visible frame, the xinghui rule); silhouettes kept as starting
points.
2026-07-22 13:03:03 -07:00
Victor Kuznetsov 5d63b9161f Register the Kling 可灵AI 3.0 visible text mark; park Yuanbao and cat-logo (measured)
Kling (USCC cohort 91110108335469089C, n=30): kling_engine.py, gate 0.35
(clean p99 0.304 / max 0.320), strict-only, unimodal 0.12/short on the
shared ladder, fitted locate box, no rival margin (crossfire 1/400 doubao
below gate, 0 jimeng, 0 clean), parity 9/9 detect->fill->re-detect.
Suppresses the jimeng pill like doubao/qwen. identify gains visible_kling.

Yuanbao: measured negative -- the two-line italic block does not separate
from clean corners on either front-end at any render/box/font setting;
the fitted recipe stays in render_vendor_silhouettes.py MARK_OPTS.

cat-logo: cohort has only 2 unique carriers, parked on evidence; the
draw_catlogo silhouette already separates (0.50 vs clean max 0.333), so
registration is a gate pick once more uniques arrive.

vendor_mark_calibrate: --fit-geometry takes locate-box overrides (two-line
marks were clipped by the inherited box) and the aspect sweep reaches 0.62.
2026-07-22 08:49:06 -07:00
Victor Kuznetsov 3a659ededc chore(release): v0.16.0 2026-07-21 18:35:04 -07:00
Victor Kuznetsov 008319c6a5 Register the Qwen 千问AI生成 visible text mark
Calibrated on the 117-frame TC260-producer cohort (vendor_cohort_harvest +
vendor_mark_calibrate, both committed here): per-mark 2-rung ladder
(0.78, 1.27) for the two measured size modes, fitted locate box (the mark
sits ~0.025 of the short side off the edge; doubao's box clipped the first
glyph), measured template aspect 0.26, gate 0.45 (clean p99 0.301).
Strict-only (the sub-gate band is non-Qwen banners), no rival margin
(0 cross-fires on 400 doubao / 298 jimeng / 286 clean frames).
83/83 real marks detector-clean after cv2 fill.

TextMarkConfig gains a per-mark ladder field; the shipped 3-rung default
is unchanged for every other mark.
2026-07-21 16:41:16 -07:00
Victor KuznetsovandClaude Opus 4.8 1d93704a4d Restructure the open items into a continuation list for the next session
The section had accumulated incremental edits: defects numbered out of order
(1, 2, 3a, 3, 5, 4), closed items mixed into the open table, and "what to do
next" spread across four subsections that partly repeated each other.

Now it opens with START HERE -- the prioritized next actions and the reason each
sits where it does -- followed by open defects only, renumbered 1-5. Closed items
move to their own subsection, keeping the faint-mask post-mortem because how it
escaped both parity and its own regression test is the instructive part.

Adds a table of the completed full runs with their artifacts and row counts, each
verified against the file on disk. That exists because the artifacts are the
answer to "did we actually cover X" -- relaunching a sweep costs hours and returns
nothing new, and this session nearly did exactly that before checking. It also
names the two fast confirmations (real_examples_e2e ~2 min, robustness_suite
~3 min) that a later change should run instead.

Adds invisible_engine.py:346 as an open defect: it still discards imwrite's
success flag, the same shape as the crash fixed in the previous commit.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 08:31:06 -07:00
Victor KuznetsovandClaude Opus 4.8 633fc3aa52 Report failed writes instead of crashing, and reject directories at parse time
The Tier E adversarial sweep (new, scripts/robustness_suite.py) drove the real CLI
over truncated, corrupt, zero-byte, absurdly-shaped and bomb inputs, unicode and
RTL paths, hostile output directories and concurrent runs. It found two crashes;
the /simplify review then reproduced a third and worse one.

1. A FAILED WRITE CRASHED ON THE SIZE REPORT. image_io.imwrite is contractually
   non-raising and returns False, but write_bgr_with_alpha discarded that bool and
   returned None, so no caller could tell a failed write from a successful one.
   Every write site then ran output.stat() to print the size, so a read-only
   destination died with a bare FileNotFoundError pointing at the stat rather than
   the write. The fix is deliberately NOT uniform: single-image commands exit via
   the new cli._write_output_or_exit; api._write_visible_result RAISES so a library
   caller gets an accurate error instead of a confusing FileNotFoundError from the
   downstream metadata strip; and the batch sites raise but never SystemExit,
   because the batch loop counts per-image exceptions and aborting would kill the
   whole run.

2. BATCH LOST DATA SILENTLY. Into a read-only output directory it wrote ZERO files
   for 2 inputs and exited 0 -- no traceback, no error, an empty output directory a
   wrapping service would read as a completed run. The robustness harness could not
   see this class at all, since it scored exit codes and traceback markers and this
   failure has neither; it now asserts on the artifacts written.

3. A DIRECTORY PASSED AS THE IMAGE crashed the metadata scanner with
   IsADirectoryError, because click.Path(exists=True) accepts directories. Fixed
   with dir_okay=False on all six source arguments, so argument parsing refuses it.

Also adds Tier B4 (scripts/resource_ceilings.py): peak RSS per fill backend from
1 MP to 25 MP, one fresh process per cell. migan 603->775 MB and lama 4679->4779 MB
are flat in input size, confirming the crop-around-the-mask design and both
documented figures; cv2 is the only backend that grows (74->440 MB, 5.9x). The
harness's own no-op check originally allocated a full-frame temp before reading
peak RSS and inflated the numbers with input size -- it now compares only the mask
box, and the conclusion survived re-measurement.

And scripts/real_examples_e2e.py, which drives every command over real corpus
examples and checks the outcome rather than the exit code: 6/6 provenance classes
identified, 10/10 metadata strips re-scan clean, all three fill backends write,
diffusion on MPS writes genuinely changed images. It records samsung as a real
partial (the faintest mark, 0.431 -> 0.404 against a 0.40 gate on the weakest of
its 3 corpus positives) and treats the gated pill's refusal to act as correct.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 22:45:02 -07:00
Victor KuznetsovandClaude Opus 4.8 52eb40c2ca Mask faint text marks with the detector's match box, not a response threshold
The faint-mask fallback added for the tophat front-end thresholded the
max-normalized uint8 response at 0.5 -- which selects every non-zero pixel,
not "half the peak" as its comment claimed -- and filled ~120% of the corner
box on textured frames. Measured on 14 real faint-path frames (cv2 fill,
detector re-run after): the detector's own best-match box fills a 58.7%-median
corner box vs 120.9% for the threshold, both 100% detector-clean. Detection and
the mask now read one method, _tophat_best, whose score gates detection and
whose argmax box bounds the fill, so the two cannot drift by construction --
which is how the mismatch arose. The 0.5 constant is deleted.

Parity could not catch this (a mask that fills everything is trivially
detector-clean) and the regression test could not either: its flat fixture
gives every threshold the same box, so mutating the constant to 99.0 stayed
green. The fixture now carries texture and asserts the mask area is bounded,
not merely non-empty; it reproduces the corpus number (127% pre-fix).

Also lands the Tier B2 verification harnesses that found and bounded this:
- detector_response.py: response curves (detected AND maskable per cell); found
  the size response is a comb, contrast is near-irrelevant, no unmaskable cells.
- ladder_headroom.py: measured that a denser scale ladder recovers 7.6% of
  misses for a 2.52%->3.05% false-fire rise, and the one landscape rung that
  helps is a geometry shift that helps and hurts equally (1.7:1) -- do not add.
- cjk_tail_probe.py: a generic shared-tail (AI生成) template does not separate
  uncovered vendors from clean corners (0.407 vs clean p99 0.298).

Records the visible-parity re-run confirming the earlier front-end fix (doubao
91.8% -> 99.3%), and dedups the thrice-written stamp forward model into one
fill_quality.composite.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 15:41:18 -07:00
Victor KuznetsovandClaude Opus 4.8 7d00debdca Record the full visible-parity run and the per-mark residual split
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 11:11:27 -07:00
Victor KuznetsovandClaude Opus 4.8 ac20cf3a3d Record open defects, unrun verification tiers and the stale torch triage
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 11:10:12 -07:00
Victor KuznetsovandClaude Opus 4.8 c150180acf Add full-surface verification harnesses and corpus sweep plan
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 08:14:50 -07:00
Victor KuznetsovandClaude Opus 4.8 cfefd9d819 Remove assume_ai, add tophat front-end and rival margin, fix two CLI defects
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-20 08:14:50 -07:00
Victor Kuznetsov a8f3536d3e Refactor watermark detection and provenance handling 2026-07-16 17:40:46 -07:00
Victor KuznetsovandClaude Opus 4.8 9618ac93c8 docs: record where the HF Space source lives and how it deploys
The Space's demo code (app.py etc.) lives in a SEPARATE private repo,
wiltodelta/raiw-hf-space, which nothing documented -- so finding it cost a long
detour through the Space's commit authorship and a hunt for a write token that
never existed locally. Record it, plus the deploy flow that replaced the old
web-UI editing: push to that repo's main -> sync-to-hf.yml mirrors the files via
HfApi.upload_folder (adds a commit on top of the Space history, never a force
push). Also call out the two automations that both touch the Space so they are
not confused: sync-to-hf.yml ships demo-code changes, while this repo's
distribute.yml factory-rebuilds the Space on a library release so its
`remove-ai-watermarks>=` pin re-resolves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-16 10:24:46 -07:00