From 132107ad9ed217239cc7ea1ae49bb535eece5362 Mon Sep 17 00:00:00 2001 From: Victor Kuznetsov Date: Sun, 9 Aug 2026 19:08:09 -0700 Subject: [PATCH] Record SynthID cross-epoch validation results --- docs/synthid-detector-removal-plan.md | 65 +++++++++++++++++++++++++-- docs/synthid.md | 14 ++++++ 2 files changed, 75 insertions(+), 4 deletions(-) diff --git a/docs/synthid-detector-removal-plan.md b/docs/synthid-detector-removal-plan.md index 12aa426..7204e9d 100644 --- a/docs/synthid-detector-removal-plan.md +++ b/docs/synthid-detector-removal-plan.md @@ -419,6 +419,62 @@ same-provider hard negatives, and a two-class temporal holdout. ## Empirical log +### 2026-08-09: D1 confound pilot and codec challenge + +The manifest-driven D1 harness was run on the frozen Google pilot containing +five oracle-positive images and 330 deduplicated exact-geometry external +photographs. The manifest passed byte, decoded-pixel, lineage, and split +verification. It is not evidence-ready because the locked test contains no +same-provider hard negative. + +The container-only baseline separated the labels perfectly because the pilot +still exposes format and export-geometry differences. That result is a measured +confound, not watermark evidence. The canonical 8x8 decoded-content baseline +was much weaker: locked-test AUC was 0.620 and temporal AUC was 0.533, and its +validation-frozen threshold detected neither held-out positive. The existing +positive-only RGB plus HSV S/V ensemble still detected all five positives and +emitted no positive verdict on the 330 frozen external images. + +A codec challenge re-encoded every positive without changing geometry. The +ensemble remained positive on five of five JPEG-95 outputs and five of five +WebP-95 outputs; JPEG-90 retained three of five. This rejects a bare PNG versus +JPEG container explanation, but does not exclude a generator or source-pipeline +correlate. With only five positives, the one-sided 95% lower bound on TPR is +54.9%. With zero positives among 330 external images, the one-sided 95% upper +bound on FPR is 0.904%, still nine times the 0.1% detector target. Four positives +also participated in model fitting, leaving only one independent temporal +positive. The next valid detector claim still requires new oracle-positive +images and ordinary same-provider oracle-negative controls. + +A post-freeze source-provenance challenge then added three exact-geometry +Google originals that had not influenced fitting or threshold selection. Each +carried the same signed Google LLC C2PA issuer, trained-algorithmic-media source +type, and explicit SynthID-present assertion as the detected temporal control. +The ensemble abstained on all three: two lacked active carrier support in both +branches, and the third passed RGB evidence but missed the HSV S/V evidence and +support gates. These are provider-signed embedding assertions rather than +matching-oracle pixel labels, but zero positives in three new same-geometry +images falsifies the current ensemble as a general Google SynthID detector. The +measured phase family is retained only as an epoch- or surface-specific +correlate pending a broader oracle-labeled corpus. + +A follow-up cross-epoch check found that the correlate is not entirely confined +to the original five images. RGB and HSV models fitted only on the three later +images ranked each of the five earlier images above all 329 external negatives +in both color spaces (AUC 1.0), but thresholds derived from the later fitting +scores transferred poorly: the strict RGB-plus-HSV decision retained only one +of five earlier positives at zero false positives on the 279-image holdout. An +eight-fold leave-one-positive-out refit across both source groups then detected +six of eight excluded positives with the same RGB-plus-HSV conjunction and each +branch frozen just above its 50-image calibration maximum. RGB alone put all +eight excluded positives above that maximum, while HSV missed the geometric and +low-texture images; the per-fold RGB operating points also produced between one +and eight false positives on 279 held-out negatives. +This is evidence for a transferable but content- and epoch-sensitive Google +pixel correlate, not a shippable detector. The experiment still lacks ordinary +same-provider oracle negatives, and its external-negative set is too small for +the 0.1% false-positive target. + ### 2026-08-09: fixed spectral-template baseline rejected An exploratory Google template was reconstructed from four public, purported @@ -750,10 +806,11 @@ has a matching-provider negative oracle verdict. A three-file Google batch is hash-frozen outside the repository for the next healthy quota window: the positive source, the stronger phase-only ablation, and the combined periodic EOT candidate. Results will be accepted only in that fixed order without -between-query adaptation. The first later attempt detected the positive source, -then returned a usage-limit response for the phase-only ablation; that response -is `indeterminate`, and the combined candidate was not submitted out of order. -The OpenAI candidate is not scheduled for the public verifier under the current +between-query adaptation. Two later attempts detected the positive source, then +returned a usage-limit response for the phase-only ablation. In the latest +attempt the verifier requested a retry after 17 hours. Both responses are +`indeterminate`, and the combined candidate was not submitted out of order. The +OpenAI candidate is not scheduled for the public verifier under the current usage restriction. ### 2026-08-09: OpenAI fidelity and architecture-transfer frontier diff --git a/docs/synthid.md b/docs/synthid.md index 8fd023e..eef328d 100644 --- a/docs/synthid.md +++ b/docs/synthid.md @@ -199,6 +199,20 @@ leave-one-positive-out checks, detected all five available positive controls, and emitted no positive verdict on 330 newly collected exact-size images. Almost all external images lacked sufficient measured carrier support and therefore remained abstentions rather than proven negatives. +A later post-freeze challenge used three additional exact-size Google originals +with signed Google LLC C2PA and explicit SynthID-present assertions. The +ensemble abstained on all three. They do not replace matching-oracle pixel +labels, but this zero-of-three source-provenance result rejects the frozen +ensemble as a general Google detector and narrows it to an epoch- or +surface-specific correlate. + +A multi-epoch leave-one-positive-out refit recovered six of the eight total +Google positives under a strict RGB-plus-HSV decision; RGB alone ranked every +excluded positive above the 50-image calibration maximum but produced up to +eight false positives on 279 held-out external negatives. The signal therefore +transfers across the two source groups, but remains content-sensitive and +cannot be separated from provider or generation-pipeline attribution without +same-provider oracle-negative controls. Directly projecting out the ensemble phases cleared the local detector above 51 dB PSNR, but three frozen candidates remained detected by Gemini in a