From 05f415b5673c33fc7931d356a6201e27c20b1bf9 Mon Sep 17 00:00:00 2001 From: Victor Kuznetsov Date: Sun, 9 Aug 2026 22:27:39 -0700 Subject: [PATCH] Reset SynthID detector hypotheses to paired training --- docs/synthid-detector-removal-plan.md | 101 +++++++++++++++++++++++--- docs/synthid.md | 15 ++++ 2 files changed, 107 insertions(+), 9 deletions(-) diff --git a/docs/synthid-detector-removal-plan.md b/docs/synthid-detector-removal-plan.md index 7204e9d..7ab3f1e 100644 --- a/docs/synthid-detector-removal-plan.md +++ b/docs/synthid-detector-removal-plan.md @@ -219,23 +219,66 @@ Gate: advance the classical detector only if the locked test shows a stable watermark-specific advantage over confound baselines. Do not choose a threshold from the locked test. +### Mechanism hypotheses for D3-D5 + +The next detector epoch tests these hypotheses as one preregistered program: + +1. **Counterfactual labels.** Train and calibrate on source-matched + clean/watermarked examples, ideally the same underlying image before and + after encoding. A result that vanishes when source, date, codec, and + dimensions are balanced is a provider fingerprint, not watermark evidence. +2. **Canonical full-field evidence.** Preserve a 512x512 image-level field, + absolute residual amplitude, and chroma alongside normalized residuals. + Compare it against the frozen patch baseline on source-disjoint pairs. + Failure to improve paired low-FPR detection rejects the added global context. +3. **Registration.** Score several bicubic canonical views spanning small scale, + crop, and offset changes, then aggregate before calibration. This tests + whether synchronization is distributed across the image rather than fixed to + one global phase or local grid. +4. **Independent detection head.** Learn one presence logit directly. Treat + payload-like or phase-consistency heads as auxiliary evidence, not as the + presence decision. Their inclusion must improve a held-out paired test, not + only attack-surface scores. +5. **Symmetric transformation channel.** Apply the identical sampled codec, + resize, crop, color, noise, or overlay operation to both members of a pair. A + transform is admitted only when the reference decoder, where available, + confirms that the transformed positive remains valid. +6. **Encoder versions.** Compare one universal model with version- or + epoch-specific experts on a cross-version transfer matrix. A version split is + useful only if it improves held-out likelihood without source metadata at + inference. +7. **Content-dependent watermarkability.** Measure flat, low-contrast, + monochrome, logo, and pixel-art strata separately. Allow a + content-conditioned expert or abstention instead of forcing one global + operating point. +8. **Two-sided calibration.** Calibrate empirical evidence for both `not + watermarked` and `watermarked`, returning positive, negative, or abstain. + External generator corpora enter afterward as an untouched false-positive + challenge, not as a substitute for counterfactual negatives. + ### Experiment D4: learned residual detector -Goal: learn content-dependent evidence that a fixed template misses. +Goal: learn watermark presence from counterfactual image-level evidence that a +fixed template misses. -Start with a small two-view model: +The primary model uses a full canonical field with raw RGB or luminance/chroma, +absolute-amplitude fine and coarse residuals, and an optional frequency branch. +Locally normalized patch evidence remains a frozen ablation, not the primary +input. Multi-view registration is aggregated into one image-level presence +logit. Payload-like, phase, localization, and content-watermarkability heads are +auxiliary and must prove an incremental held-out benefit. -1. a spatial view of locally normalized high-pass RGB residuals; -2. a frequency view containing log magnitude and phase-derived channels. - -Fuse only late features so each view can be ablated. Use aggressive content and -export balancing, group-aware sampling, and augmentations drawn from the paper's -transformation families. Never expose metadata, path, dimensions without -normalization, or encoder-specific byte patterns to the network. +Use group-aware pair sampling and apply every sampled transformation +symmetrically to the clean and watermarked members. Never expose metadata, +paths, native dimensions without normalization, or encoder-specific byte +patterns to the network. If causal pairs remain unavailable, do not advance a +learned model from provider classification to this experiment. Train provider-specific models first. A shared backbone with provider-specific heads is a later ablation, not the default architecture. Keep a second detector family completely outside remover training so it can reveal surrogate overfit. +Compare a universal head with version-specific experts, and calibrate the +selected model with two-sided empirical evidence so ambiguous inputs abstain. Gate for a detector release candidate: @@ -854,6 +897,46 @@ the candidates above 40 dB are architecture-specific and cannot support a removal claim. No additional public OpenAI verifier requests were made during these experiments. +### 2026-08-09: mechanism reset and paired open-method control + +The first detector epochs optimized the wrong statistical problem. The +[technical reference](synthid.md#11-post-hoc-model-independent-design) now +records the paper's paired training target, symmetric transformation channel, +independent detection logit, and two-sided calibration. Applied to the initial +pilots, those details reject three implicit assumptions: that a carrier phase +can stand in for the detection score, that mean patch classification is an +adequate image-level decoder, and that a sigmoid threshold calibrated on +unrelated negatives is a deployable decision rule. + +A bounded open-method control tested the architecture concern without claiming +that another watermark reproduces SynthID. TrustMark P was used only because +its public encoder creates causal clean/watermarked pairs and its public decoder +can validate transformed labels. Ninety public COCO images were split by source +image into 60 training, 15 calibration, and 15 test pairs. Each pair received +the same identity, JPEG-90, 0.8 resize round trip, or 5% crop transformation. +The official decoder detected 14/15, 15/15, 14/15, and 15/15 transformed marked +test images respectively, with zero detections on the corresponding clean +images. + +The existing normalized residual patch recipe reached only 0.707 identity AUC +and 0.668 aggregate AUC on the held-out pairs. A small full-field model that +retained RGB amplitude plus fine and coarse residuals reached 0.947 identity AUC +and 0.841 aggregate AUC. At a threshold above every calibration negative, their +aggregate TPRs were 5% and 20% respectively. Both models failed to reproduce +the official decoder's JPEG robustness. This small proxy is not a TrustMark +benchmark and says nothing directly about SynthID accuracy. It does falsify the +claim that the current patch-normalized architecture and training recipe are an +adequate generic neural-watermark detector. + +The resulting eight falsifiable hypotheses now define the canonical +[D3-D5 mechanism program](#mechanism-hypotheses-for-d3-d5). In particular, +external generator corpora remain a final false-positive challenge and do not +substitute for counterfactual negatives. + +Until the counterfactual-label and full-field gates pass, the current residual +CNN remains a useful vendor-triage stage for a future cascade, but it is not a +SynthID detector and should not be optimized as the final decision surface. + ## Decision record The program has four possible honest outcomes per provider: diff --git a/docs/synthid.md b/docs/synthid.md index eef328d..a84c23f 100644 --- a/docs/synthid.md +++ b/docs/synthid.md @@ -42,6 +42,21 @@ after generation, by a separate neural network. This means: - Replacing the generative model does not remove the watermarking capability. - The encoder/decoder pair can be updated independently of the generative model. +The detector's training target is narrower than provider or AI-image +classification. Equation 1 trains on watermarked `f(x)` as positive and the +corresponding unwatermarked distribution `x` as negative, with both drawn from +the same target image distribution. Equation 2 applies the same sampled +semantics-preserving transformation to both sides. Unrelated generators and +camera images are useful later for false-positive evaluation, but they do not +replace source-matched clean examples during signal-identifiability training. + +The decoder produces a dedicated detection logit whose threshold is calibrated +after training for a target false-positive rate. Payload recovery is a separate +problem, not the definition of the presence score. The paper additionally +describes two-sided conformal calibration for the `not watermarked` and +`watermarked` hypotheses, which permits a detector to abstain instead of forcing +an unsupported binary verdict. + The paper does not disclose the internal architecture of the encoder/decoder networks (layer types, capacity). The external variant SynthID-O is available to partners; the production internal variant is not published.