mirror of
https://github.com/wiltodelta/remove-ai-watermarks.git
synced 2026-08-31 09:40:38 +02:00
Record wild-AI audit, Meta provider class, and stock-negative expansion
The wild vendor-flagged AI cell (300 stratified rows) puts Model 1 recall at 69.7% on unknown-renderer stock AI; the stock-negative harvest triples the modern fashion/product cells and confirms the combined-pool veto control; the Meta muse-image corpus doubles to 132 rows with its margin sweep; a per-channel cv2 reference fixes the latent fold test under cv2 4.10.0. pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks separately green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - new run references point at the gitignored research store, none stale; 4) CLAUDE.md - compact, no changes needed
This commit is contained in:
@@ -22,6 +22,12 @@ data/
|
||||
|
||||
## Storage rules
|
||||
|
||||
Large local-only research data (the 211 GB Spaces corpus and the frozen
|
||||
checkpoints, research corpora, and campaign reports) lives under gitignored
|
||||
paths documented in `data/research/README.md` (ignored via `.git/info/exclude`,
|
||||
never committed). Worktree `.local-eval/` directories are scratch copies; the
|
||||
durable store is `data/research/` in the main checkout.
|
||||
|
||||
1. Store each binary image once. Evaluation manifests and documentation point
|
||||
to its canonical location.
|
||||
2. Put executable test fixtures in `fixtures/`.
|
||||
|
||||
@@ -694,6 +694,89 @@ family over the frozen Model 1 CLIP-L embedding for the modern domains as well
|
||||
and confirms the earlier aligned-mining finding: the add representation itself
|
||||
must change, not the example selection or the veto geometry.
|
||||
|
||||
### Wild vendor-flagged AI, 2026-08-27
|
||||
|
||||
A free wild-AI positive cell came from the stock providers behind the private
|
||||
veedma-blog image pipeline. Pixabay exposes a vendor-declared
|
||||
`isAiGenerated` flag and the query `ai generated` surfaces that pool (3,287
|
||||
results); 284 content-hash-unique rows were harvested (300 minus 16 lost to a
|
||||
duplicate-download bug, since fixed, manifest repaired to on-disk truth). The
|
||||
vendor flag is ground truth for AI-versus-human, not for the renderer, so the
|
||||
cell is eval-only. A metadata scan found 0/284 provenance signals (the stock
|
||||
CDN strips everything), making it a pure pixel task. Scored with frozen
|
||||
classifiers that never saw stock-site content, this produced the first
|
||||
source-disjoint wild-AI recall measurement: Model 1 accepted 183/284 (64.4%)
|
||||
at its frozen 1%-FPR threshold, far below its 93.0% curated AI-test recall
|
||||
(median score 0.43 above the 0.306 threshold, but a quarter of the cell scores
|
||||
below 0.13). Wild AI images of unknown renderers, re-encoded by a stock
|
||||
pipeline, are therefore a genuinely harder positive distribution, and curated
|
||||
test recall does not bound deployment recall. The frozen provider cascade
|
||||
under its photo-first margin called 255/284 `no_ai` with only 1 `openai`
|
||||
and 0 `meta` calls, so unknown-renderer AI does not fabricate provider
|
||||
attributions; under argmax it leans `google` (199) without evidence. The cell
|
||||
joins the dataset as the first wild-AI eval row source. Artifacts:
|
||||
`pixabay-ai-positive-2026-08-27/` (manifest, images,
|
||||
classification-report.json).
|
||||
|
||||
A bias audit then rejected that cell's composition, though not its verdict:
|
||||
147/284 rows (52%) came from a single contributor and every row from one
|
||||
query, so the 64.4% recall was one author's dump plus a tail. Model 1 recall
|
||||
split 63.3% on that contributor versus 65.7% on the rest, so difficulty was
|
||||
homogeneous, but the sample was not. A stratified reharvest capped at 15 rows
|
||||
per contributor over eight AI-marker queries produced 300 unique images from
|
||||
180 contributors (top-10 share 33%) with the same sieves. On the balanced
|
||||
cell the frozen numbers are: Model 1 recall 209/300 (69.7%, median score 0.482,
|
||||
lowest quartile below 0.144), direct quarter-hard 189/300 repairing 22 Model 1
|
||||
misses and losing 2 (the one place the quarter representation still helps),
|
||||
conditional routing 210/300 adding one repair and nothing else, provider
|
||||
cascade margin 252/300 `no_ai` with 1 `openai` call, and 0/300 metadata
|
||||
signals. The wild-AI recall ceiling under Model 1 therefore settles near
|
||||
65-70% rather than 93%, robust to contributor stratification; unknown-renderer
|
||||
AI re-encoded by stock pipelines is the largest measured positive-side gap.
|
||||
Artifacts: `pixabay-ai-stratified-2026-08-27/` (the eval cell the dataset
|
||||
keeps; the first harvest is retained only as the audit trail of the bias
|
||||
finding).
|
||||
|
||||
The stock-negative harvest from the same providers then tripled the modern
|
||||
negative cells under identical sieves: 150 Pixabay product cutouts
|
||||
(vendor `isAiGenerated=false`), 150 Pixabay fashion rows, and 152 Unsplash
|
||||
API fashion rows date-bounded before 2022-08-01 (452 total, 0/452 metadata
|
||||
signals, all content-hash unique). Frozen Model 1 false-positive rates on
|
||||
them: product cutouts 40/150 (26.7%), Pixabay fashion 93/150 (62.0%),
|
||||
Unsplash fashion 108/152 (71.1%). The fashion failure is therefore larger
|
||||
than the earlier 34% measured on the 88-row date-clean cell and holds across
|
||||
three independent sources; retouched fashion photography is now the single
|
||||
worst negative domain for Model 1. The provider cascade stays quiet on all
|
||||
three cells (no_ai 126/150, 105/152, 99/150; at most 15 stray `google` calls,
|
||||
no openai or meta fabrications). A second stochastic pass over the Meta
|
||||
muse-image-1.0 grid added 61 more API generations with zero hash overlap
|
||||
with pass one (corpus now 127 + 5 checkout originals), moving the Meta class
|
||||
toward margin calibrability. Artifacts:
|
||||
`stock-negative-2026-08-27/`, `meta-muse-corpus-pass2-2026-08-26/`.
|
||||
|
||||
The combined-pool control then separated the data question from the geometry
|
||||
question: refitting the same pooled/domain/multiclass ridge vetoes on a
|
||||
687-row modern-negative pool (Openverse-clean plus all three stock cells,
|
||||
three times the original fit size) reproduced the earlier Pareto curve to the
|
||||
point: at positive compliance the best dev error count stayed 41/186 with
|
||||
zero route errors repaired, and every repaired negative still cost about 1.4
|
||||
AI positives. The linear-veto failure over the frozen Model 1 embedding is
|
||||
therefore not a data-volume artifact; tripling diverse negatives moves
|
||||
nothing. Artifacts:
|
||||
`modern-domain-veto-combined-2026-08-27/` (development report and pareto).
|
||||
|
||||
The doubled Meta corpus then allowed its own margin sweep (127 API rows in
|
||||
the bank after feature extraction, 25 repeated splits, photo leak measured on
|
||||
held-out no_ai rows). The curve is steep exactly where the frozen 0.50 margin
|
||||
sat: margin 0.00 recalls 92.2% meta at 5.3% photo AI-rate, 0.10 gives 90.1%
|
||||
at 3.6%, 0.20 gives 85.2% at 2.2%, then the cliff: 0.30 falls to 70.2%, 0.40
|
||||
to 39.4%, and the frozen 0.50 lands at 12.4%. A workable photo-safe
|
||||
operating point exists at margin 0.20 (85% meta recall at 2.2% photo leak),
|
||||
but 2.2% is still thirty times the Model 1 photo contract, so for provider
|
||||
attribution the honest rule stays argmax until an external held-out meta cell
|
||||
exists. The margin, not the head, was always the limiter on the small class.
|
||||
Artifacts: `meta-margin-2026-08-27/report.json`.
|
||||
|
||||
A taxonomy continuation then changed the training mix itself instead of the
|
||||
veto: two arms re-ran the expanded quarter-hard recipe (ordinary AI replay,
|
||||
EvalGEN pair positives, ordinary photos) with one or two of the eight negative
|
||||
|
||||
@@ -37,6 +37,71 @@ Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57;
|
||||
Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the
|
||||
124-d lattice bank it is 0.989. They are two pipelines, not one class.
|
||||
|
||||
### Renderer, not front-end (cross-carrier validation, 2026-08-26)
|
||||
|
||||
The Spaces catalog's C2PA issuer field allows a provider-label audit the
|
||||
family labels cannot express: Bing Image Creator rows signed `Microsoft,
|
||||
OpenAI` (renderer DALL-E, n=152), Microsoft-native rows (MAI-Image / Designer,
|
||||
n=85), Designer rows signed `Microsoft, Google LLC` (renderer Imagen, n=36),
|
||||
and the unattributed Instagram `made_with_ai` label set (n=275). Scored with
|
||||
the frozen 124-feature openai/google/no_ai cascade, no training, metadata
|
||||
used only to build cells: argmax called 124/152 (81.6%) of the Bing+DALL-E
|
||||
cell `openai` and 34/36 (94.4%) of the Microsoft+Imagen cell `google`. Pixel
|
||||
attribution therefore tracks the renderer behind the front-end, which is the
|
||||
correct semantic for a stripped file: a Bing export is an OpenAI-rendered
|
||||
image. The Microsoft-native cell split 48 openai / 35 google, consistent with
|
||||
Designer historically routing between DALL-E and Imagen rather than one
|
||||
pipeline. The Instagram `made_with_ai` set is not a Meta pixel class: argmax
|
||||
leaned `google` at 196/275 with 68 `openai`, and the photo-first 0.50 margin
|
||||
sent 182/275 to `no_ai`, so the label marks detected AI content of mixed
|
||||
origin, not Meta-rendered pixels. A real Meta class needs fresh
|
||||
`Imagined with AI` generations with known provenance; the catalog label is
|
||||
not one. Artifacts:
|
||||
`provider-renderer-cells-2026-08-26/report.json`.
|
||||
|
||||
### A Meta class from muse-image-1.0, 2026-08-26
|
||||
|
||||
The Meta Model API (`api.meta.ai/v1`, OpenAI-compatible images endpoint) made
|
||||
a known-provenance Meta corpus possible without any account browser session:
|
||||
61 images generated from a 61-prompt grid spanning portraits, product shots,
|
||||
scenes, food, animals, architecture, illustration styles, text posters, and
|
||||
abstract work across all three aspect ratios, plus the five oracle-verified
|
||||
samples already in `data/contentseal/originals/`. All 66 are content-hash
|
||||
unique, delivered at 1600x1600 / 1920x1280 / 1280x1920, and every API row
|
||||
carries IPTC `trainedAlgorithmicMedia` plus a Content Seal generation id
|
||||
recorded in the manifest. A four-head control with a `microsoft_native` class
|
||||
(83 issuer-verified rows) failed first: repeated-split argmax recall 0.38 mean
|
||||
with 0.16-0.56 spread, because Designer routes between renderers and the class
|
||||
is mixed. The same recipe with `meta_muse_image` in its place holds: argmax
|
||||
recall 0.86 mean (0.75-1.00) on just 66 images, openai 0.952 and google 0.956
|
||||
unharmed, and the frozen cross-carrier cells keep their renderer semantics
|
||||
(Bing+DALL-E 119/146 openai, Microsoft+Imagen 32/36 google) with only small
|
||||
meta leakage (14 and 3 argmax rows). The class label names what the corpus is:
|
||||
muse-image-1.0 API output, not the unverified assumption that the consumer
|
||||
Imagine feature renders identically. The full-data model calls 59/66 meta rows
|
||||
correctly, with seven leaking to google and none to openai. The photo-first
|
||||
0.50 margin does not transfer to a 66-image class (0.398 mean recall); argmax
|
||||
is the honest operating rule until the corpus grows. Muse Image is therefore
|
||||
a separable fourth provider class on pixels, unlike the Microsoft front-end.
|
||||
Artifacts: `meta-muse-corpus-2026-08-26/`, `four-head-provider-2026-08-26/`,
|
||||
`meta-provider-cells-2026-08-26/`.
|
||||
|
||||
A paired chat-vs-API check then closed the label question. Two prompts from
|
||||
the API grid (weathered fisherman portrait, white-sneaker product shot) were
|
||||
submitted to the consumer Meta AI chat imagine flow in the user's own logged-in
|
||||
browser session; the chat delivered 1280x1920 and 1920x1280 WebP files from the
|
||||
`t39.105495-1` CDN family, each carrying IPTC `trainedAlgorithmicMedia` in the
|
||||
downloaded bytes. Scored by the frozen API-trained four-head model, which had
|
||||
never seen a chat image: both chat rows landed on `meta_muse_image` under
|
||||
argmax (0.251 and 0.233) with score profiles matching their API twins almost
|
||||
exactly, and the margin rule agreed pairwise (fisherman meta at margin, sneaker
|
||||
no_ai at margin 0.50, same as its API counterpart). The chat imagine pipeline
|
||||
and `muse-image-1.0` are therefore pixel-indistinguishable to the provider
|
||||
classifier on this pair, so `meta_muse_image` honestly names both
|
||||
distributions. Caveat: n=2 paired prompts from one session; this is a
|
||||
consistency result, not a deployment-scale equivalence claim. Artifact:
|
||||
`meta-chat-check-2026-08-26/`.
|
||||
|
||||
Collapsing OpenAI and Gemini into one pixel class versus other generators
|
||||
does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%,
|
||||
Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at
|
||||
|
||||
@@ -689,12 +689,22 @@ def test_non_divisible_fold_matches_modulo_cell_means() -> None:
|
||||
rng = np.random.default_rng(44041)
|
||||
pixels = rng.integers(0, 256, size=(53, 71, 3), dtype=np.uint8)
|
||||
source = pixels.astype(np.float32)
|
||||
residual = source - cv2.GaussianBlur(
|
||||
source,
|
||||
(0, 0),
|
||||
sigmaX=1.25,
|
||||
sigmaY=1.25,
|
||||
borderType=cv2.BORDER_REFLECT_101,
|
||||
# Mirror the module's documented per-channel blur: OpenCV's multi-channel
|
||||
# GaussianBlur is not bit-identical to per-channel calls (observed 3e-5 on
|
||||
# cv2 4.10.0), so a three-channel reference cannot satisfy atol=0.
|
||||
residual = np.stack(
|
||||
[
|
||||
source[:, :, channel]
|
||||
- cv2.GaussianBlur(
|
||||
source[:, :, channel].copy(),
|
||||
(0, 0),
|
||||
sigmaX=1.25,
|
||||
sigmaY=1.25,
|
||||
borderType=cv2.BORDER_REFLECT_101,
|
||||
)
|
||||
for channel in range(3)
|
||||
],
|
||||
axis=2,
|
||||
)
|
||||
expected = np.empty((16, 16, 3), dtype=np.float64)
|
||||
for tile_y in range(16):
|
||||
|
||||
Reference in New Issue
Block a user