mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-31 09:40:38 +02:00
Record full failure map, H2 router falsification, and H3 process detector closure
20 measured negative domains with batch-2 (memes, whiteboards, tickets, book pages). H2 domain router falsified: 97.9% of AI-test looks structured, every family 94-100%. H3 process detector on 369 contrastive pairs from three generators (OpenAI gpt-image-1-mini, Meta muse-image-1.0, Gemini flash-image) achieves 0.909 pairwise AUC on training pairs but transfers at AUC 0.55-0.60 to production AI; human receipts score higher than any AI family. The generator-specific spectral trace does not generalize across three generators, closing the process-trace family in its current form. pre-commit: 1) maintain.sh - exit 1, known lightning advisory; core green (ruff, format, pyright, 1665 tests); 2) /simplify - docs + research scripts, single pass, no findings; 3) docs sync - all artifacts in data/research/, no stale refs; 4) CLAUDE.md - no changes
This commit is contained in:
@@ -754,6 +754,99 @@ with pass one (corpus now 127 + 5 checkout originals), moving the Meta class
|
||||
toward margin calibrability. Artifacts:
|
||||
`stock-negative-2026-08-27/`, `meta-muse-corpus-pass2-2026-08-26/`.
|
||||
|
||||
### Documents and receipts, 2026-08-27
|
||||
|
||||
Two public pre-diffusion research datasets filled the unmeasured document
|
||||
domain: CORD-v2 thermal-printed photographed receipts (99 rows) and FUNSD
|
||||
scanned forms (150 rows), both provenance-metadata-clean on a spot scan. The
|
||||
split is the sharpest in the whole failure map: Model 1 accepted **89/99
|
||||
receipts as AI-generated (89.9%, score median 0.469 against the 0.306
|
||||
threshold)** while form scans stayed near the contract at 5/150 (3.3%,
|
||||
median -0.001). Photographed thermal paper (small, noisy, low-texture,
|
||||
layout-symmetric) is the single worst human negative domain measured,
|
||||
surpassing retouched fashion (71.1%); clean high-fidelity scans pass. The
|
||||
document domain therefore decomposes by capture quality rather than content
|
||||
type, and any receipt-adjacent workload inherits a 90% false AI rate from
|
||||
Model 1. Artifacts: `documents-negative-2026-08-27/`.
|
||||
|
||||
Diagnostics then fixed what can be done about it. Receipt embeddings sit
|
||||
**99.0% closer to AI positives than to photos** in Model 1 space, so every
|
||||
veto, head, or continuation over this embedding is structurally dead for
|
||||
receipts, the fashion trap in its worst form; form scans sit at 26.7% and are
|
||||
fine. A trivial white-background rule is also dead (CORD receipts are dark
|
||||
noisy photographs of crumpled thermal paper, white fraction 0.021, at photo
|
||||
levels). A receipt-document **gate** (ridge receipt-vs-rest over 499 receipt
|
||||
embeddings with photo, museum, AI, and FLUX negatives) reaches **100%
|
||||
receipt coverage**, but at the operating point chosen by inner CV it also
|
||||
gates 14.4% of AI-test, 12.0% of FLUX, and 2% of EvalGEN, with family losses
|
||||
up to 25.3% (fal) and 25.2% (xAI) — infographic-like generators trip the
|
||||
document gate. As a product abstain this converts a 90% wrong-verdict domain
|
||||
into unknowns at the price of a double-digit recall loss across AI families,
|
||||
which violates the frozen recall contract; a high-precision variant with a
|
||||
stricter threshold could gate fewer AI rows but the inner-CV F1 was only
|
||||
0.444, so precision headroom is thin. The honest options are therefore:
|
||||
(1) scope receipts out of the supported contract explicitly, or (2) a
|
||||
calibrated abstain band presented as unknown rather than no_ai, accepting a
|
||||
bounded recall loss decided as a product tradeoff, not a model fix.
|
||||
Artifacts: `receipt-gate-2026-08-27/report.json`.
|
||||
|
||||
Batch-1 then filled four more unmeasured domains under the same sieves
|
||||
(428→421 rows after phantom repair: UI screenshots from own products, 150;
|
||||
Danbooru community digital art, 145; software-rendered matplotlib charts,
|
||||
82; OpenStreetMap tiles, 44). Frozen Model 1 false-positive rates: **UI
|
||||
screenshots 149/150 (99.3%)**, **digital art 144/145 (99.3%)**, charts 55/82
|
||||
(67.1%), map tiles 21/44 (47.7%). The 1-NN diagnostic puts every failing
|
||||
domain inside the AI cluster (UI 100%, art 100%, maps 100%, charts 95.1%
|
||||
closer to AI than to photos), confirming the structural pattern: any
|
||||
human-made content that is layout-structured, graphics-like, or
|
||||
software-rendered lives inside the AI region of the CLIP-L space and cannot
|
||||
be repaired there. Combined with receipts (89.9%), fashion (62-71%), logos
|
||||
(38%), and museum art (33.8%), the boundary of the supported domain is now
|
||||
sharp: clean photographs and high-fidelity scans pass; everything structured,
|
||||
stylized, or degraded fails at 33-99%. The product contract and any future
|
||||
routing gate must draw the line at photographic content, not at "images".
|
||||
Artifacts: `batch1-negative-2026-08-27/`.
|
||||
|
||||
### Batch-2 and the full failure map, 2026-08-27
|
||||
|
||||
Batch-2 added four more domains under the same sieves (memes from imgflip
|
||||
top-100, whiteboard photos, ticket/stamp scans, book pages; 396 rows):
|
||||
memes 58/97 (59.8%), book pages 27/100 (27.0%), whiteboards 3/100 (3.0%),
|
||||
tickets 3/99 (3.0%). Whiteboards and tickets join the passing side; memes
|
||||
and book pages join the failing side. The complete failure map now covers
|
||||
**20 measured negative domains**.
|
||||
|
||||
### H2 domain router falsified, 2026-08-27
|
||||
|
||||
A binary photographic-vs-structured router on Model 1 embeddings, trained on
|
||||
all 16 failing domains vs photo cells, reached 98.2% structured coverage and
|
||||
4.4% photo false-structured rate, but routed **97.9% of AI-test and 98.0% of
|
||||
FLUX as structured**: every AI family (94.3-100%) looks non-photographic to
|
||||
the router. This is the third independent confirmation (after 1-NN overlap
|
||||
and linear-veto exchange) that provenance is not encoded in the CLIP-L
|
||||
appearance space: routing cannot separate AI content from structured human
|
||||
content because they occupy the same region.
|
||||
|
||||
### H3 process detector: generator-specific traces do not transfer, 2026-08-27
|
||||
|
||||
Contrastive pairs (same content, different process) were generated from 125
|
||||
source images across five domains (charts, logos, art, documents, UI), each
|
||||
recreated by three generators: OpenAI gpt-image-1-mini (edits/img2img), Meta
|
||||
muse-image-1.0 (prompt), and Gemini 2.5-flash-image (prompt), totaling 369
|
||||
pairs. A patch-level FFT-magnitude CNN trained on these pairs separates
|
||||
human originals from AI recreations at pairwise AUC 0.909 (0.885 with two
|
||||
generators, 0.949 with one VAE), confirming the process signal exists within
|
||||
the training distribution. Transfer to production AI collapses: AI-test
|
||||
score distributions overlap photo distributions (mean +0.013 to +0.026 above
|
||||
photos), giving AUC 0.55-0.60 — near chance — while human receipts score
|
||||
+0.095 above photos, higher than any AI family. Each VAE/decoder leaves a
|
||||
unique spectral signature; three generators are insufficient to cover the
|
||||
space, and the learned features do not generalize to unseen production
|
||||
generators. This closes the process-trace family in its current form;
|
||||
reopening would require 10+ diverse generators in training or a
|
||||
fundamentally different feature space that captures what AI generation
|
||||
processes share beyond their decoder-specific artifacts.
|
||||
|
||||
The combined-pool control then separated the data question from the geometry
|
||||
question: refitting the same pooled/domain/multiclass ridge vetoes on a
|
||||
687-row modern-negative pool (Openverse-clean plus all three stock cells,
|
||||
|
||||
Reference in New Issue
Block a user