Record wild-AI audit, Meta provider class, and stock-negative expansion

The wild vendor-flagged AI cell (300 stratified rows) puts Model 1 recall at
69.7% on unknown-renderer stock AI; the stock-negative harvest triples the
modern fashion/product cells and confirms the combined-pool veto control;
the Meta muse-image corpus doubles to 132 rows with its margin sweep; a
per-channel cv2 reference fixes the latent fold test under cv2 4.10.0.

pre-commit: 1) maintain.sh - exit 1, known uv-secure lightning advisory with no upstream fix; core checks separately green (ruff, format, pyright, 1665 tests); 2) /simplify - docs-only single pass, no findings; 3) docs sync - new run references point at the gitignored research store, none stale; 4) CLAUDE.md - compact, no changes needed
This commit is contained in:
Victor Kuznetsov
2026-08-27 09:49:30 -07:00
parent aa74f0eff7
commit 85b18af804
4 changed files with 170 additions and 6 deletions
+6
View File
@@ -22,6 +22,12 @@ data/
## Storage rules
Large local-only research data (the 211 GB Spaces corpus and the frozen
checkpoints, research corpora, and campaign reports) lives under gitignored
paths documented in `data/research/README.md` (ignored via `.git/info/exclude`,
never committed). Worktree `.local-eval/` directories are scratch copies; the
durable store is `data/research/` in the main checkout.
1. Store each binary image once. Evaluation manifests and documentation point
to its canonical location.
2. Put executable test fixtures in `fixtures/`.
+83
View File
@@ -694,6 +694,89 @@ family over the frozen Model 1 CLIP-L embedding for the modern domains as well
and confirms the earlier aligned-mining finding: the add representation itself
must change, not the example selection or the veto geometry.
### Wild vendor-flagged AI, 2026-08-27
A free wild-AI positive cell came from the stock providers behind the private
veedma-blog image pipeline. Pixabay exposes a vendor-declared
`isAiGenerated` flag and the query `ai generated` surfaces that pool (3,287
results); 284 content-hash-unique rows were harvested (300 minus 16 lost to a
duplicate-download bug, since fixed, manifest repaired to on-disk truth). The
vendor flag is ground truth for AI-versus-human, not for the renderer, so the
cell is eval-only. A metadata scan found 0/284 provenance signals (the stock
CDN strips everything), making it a pure pixel task. Scored with frozen
classifiers that never saw stock-site content, this produced the first
source-disjoint wild-AI recall measurement: Model 1 accepted 183/284 (64.4%)
at its frozen 1%-FPR threshold, far below its 93.0% curated AI-test recall
(median score 0.43 above the 0.306 threshold, but a quarter of the cell scores
below 0.13). Wild AI images of unknown renderers, re-encoded by a stock
pipeline, are therefore a genuinely harder positive distribution, and curated
test recall does not bound deployment recall. The frozen provider cascade
under its photo-first margin called 255/284 `no_ai` with only 1 `openai`
and 0 `meta` calls, so unknown-renderer AI does not fabricate provider
attributions; under argmax it leans `google` (199) without evidence. The cell
joins the dataset as the first wild-AI eval row source. Artifacts:
`pixabay-ai-positive-2026-08-27/` (manifest, images,
classification-report.json).
A bias audit then rejected that cell's composition, though not its verdict:
147/284 rows (52%) came from a single contributor and every row from one
query, so the 64.4% recall was one author's dump plus a tail. Model 1 recall
split 63.3% on that contributor versus 65.7% on the rest, so difficulty was
homogeneous, but the sample was not. A stratified reharvest capped at 15 rows
per contributor over eight AI-marker queries produced 300 unique images from
180 contributors (top-10 share 33%) with the same sieves. On the balanced
cell the frozen numbers are: Model 1 recall 209/300 (69.7%, median score 0.482,
lowest quartile below 0.144), direct quarter-hard 189/300 repairing 22 Model 1
misses and losing 2 (the one place the quarter representation still helps),
conditional routing 210/300 adding one repair and nothing else, provider
cascade margin 252/300 `no_ai` with 1 `openai` call, and 0/300 metadata
signals. The wild-AI recall ceiling under Model 1 therefore settles near
65-70% rather than 93%, robust to contributor stratification; unknown-renderer
AI re-encoded by stock pipelines is the largest measured positive-side gap.
Artifacts: `pixabay-ai-stratified-2026-08-27/` (the eval cell the dataset
keeps; the first harvest is retained only as the audit trail of the bias
finding).
The stock-negative harvest from the same providers then tripled the modern
negative cells under identical sieves: 150 Pixabay product cutouts
(vendor `isAiGenerated=false`), 150 Pixabay fashion rows, and 152 Unsplash
API fashion rows date-bounded before 2022-08-01 (452 total, 0/452 metadata
signals, all content-hash unique). Frozen Model 1 false-positive rates on
them: product cutouts 40/150 (26.7%), Pixabay fashion 93/150 (62.0%),
Unsplash fashion 108/152 (71.1%). The fashion failure is therefore larger
than the earlier 34% measured on the 88-row date-clean cell and holds across
three independent sources; retouched fashion photography is now the single
worst negative domain for Model 1. The provider cascade stays quiet on all
three cells (no_ai 126/150, 105/152, 99/150; at most 15 stray `google` calls,
no openai or meta fabrications). A second stochastic pass over the Meta
muse-image-1.0 grid added 61 more API generations with zero hash overlap
with pass one (corpus now 127 + 5 checkout originals), moving the Meta class
toward margin calibrability. Artifacts:
`stock-negative-2026-08-27/`, `meta-muse-corpus-pass2-2026-08-26/`.
The combined-pool control then separated the data question from the geometry
question: refitting the same pooled/domain/multiclass ridge vetoes on a
687-row modern-negative pool (Openverse-clean plus all three stock cells,
three times the original fit size) reproduced the earlier Pareto curve to the
point: at positive compliance the best dev error count stayed 41/186 with
zero route errors repaired, and every repaired negative still cost about 1.4
AI positives. The linear-veto failure over the frozen Model 1 embedding is
therefore not a data-volume artifact; tripling diverse negatives moves
nothing. Artifacts:
`modern-domain-veto-combined-2026-08-27/` (development report and pareto).
The doubled Meta corpus then allowed its own margin sweep (127 API rows in
the bank after feature extraction, 25 repeated splits, photo leak measured on
held-out no_ai rows). The curve is steep exactly where the frozen 0.50 margin
sat: margin 0.00 recalls 92.2% meta at 5.3% photo AI-rate, 0.10 gives 90.1%
at 3.6%, 0.20 gives 85.2% at 2.2%, then the cliff: 0.30 falls to 70.2%, 0.40
to 39.4%, and the frozen 0.50 lands at 12.4%. A workable photo-safe
operating point exists at margin 0.20 (85% meta recall at 2.2% photo leak),
but 2.2% is still thirty times the Model 1 photo contract, so for provider
attribution the honest rule stays argmax until an external held-out meta cell
exists. The margin, not the head, was always the limiter on the small class.
Artifacts: `meta-margin-2026-08-27/report.json`.
A taxonomy continuation then changed the training mix itself instead of the
veto: two arms re-ran the expanded quarter-hard recipe (ordinary AI replay,
EvalGEN pair positives, ordinary photos) with one or two of the eight negative
+65
View File
@@ -37,6 +37,71 @@ Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57;
Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the
124-d lattice bank it is 0.989. They are two pipelines, not one class.
### Renderer, not front-end (cross-carrier validation, 2026-08-26)
The Spaces catalog's C2PA issuer field allows a provider-label audit the
family labels cannot express: Bing Image Creator rows signed `Microsoft,
OpenAI` (renderer DALL-E, n=152), Microsoft-native rows (MAI-Image / Designer,
n=85), Designer rows signed `Microsoft, Google LLC` (renderer Imagen, n=36),
and the unattributed Instagram `made_with_ai` label set (n=275). Scored with
the frozen 124-feature openai/google/no_ai cascade, no training, metadata
used only to build cells: argmax called 124/152 (81.6%) of the Bing+DALL-E
cell `openai` and 34/36 (94.4%) of the Microsoft+Imagen cell `google`. Pixel
attribution therefore tracks the renderer behind the front-end, which is the
correct semantic for a stripped file: a Bing export is an OpenAI-rendered
image. The Microsoft-native cell split 48 openai / 35 google, consistent with
Designer historically routing between DALL-E and Imagen rather than one
pipeline. The Instagram `made_with_ai` set is not a Meta pixel class: argmax
leaned `google` at 196/275 with 68 `openai`, and the photo-first 0.50 margin
sent 182/275 to `no_ai`, so the label marks detected AI content of mixed
origin, not Meta-rendered pixels. A real Meta class needs fresh
`Imagined with AI` generations with known provenance; the catalog label is
not one. Artifacts:
`provider-renderer-cells-2026-08-26/report.json`.
### A Meta class from muse-image-1.0, 2026-08-26
The Meta Model API (`api.meta.ai/v1`, OpenAI-compatible images endpoint) made
a known-provenance Meta corpus possible without any account browser session:
61 images generated from a 61-prompt grid spanning portraits, product shots,
scenes, food, animals, architecture, illustration styles, text posters, and
abstract work across all three aspect ratios, plus the five oracle-verified
samples already in `data/contentseal/originals/`. All 66 are content-hash
unique, delivered at 1600x1600 / 1920x1280 / 1280x1920, and every API row
carries IPTC `trainedAlgorithmicMedia` plus a Content Seal generation id
recorded in the manifest. A four-head control with a `microsoft_native` class
(83 issuer-verified rows) failed first: repeated-split argmax recall 0.38 mean
with 0.16-0.56 spread, because Designer routes between renderers and the class
is mixed. The same recipe with `meta_muse_image` in its place holds: argmax
recall 0.86 mean (0.75-1.00) on just 66 images, openai 0.952 and google 0.956
unharmed, and the frozen cross-carrier cells keep their renderer semantics
(Bing+DALL-E 119/146 openai, Microsoft+Imagen 32/36 google) with only small
meta leakage (14 and 3 argmax rows). The class label names what the corpus is:
muse-image-1.0 API output, not the unverified assumption that the consumer
Imagine feature renders identically. The full-data model calls 59/66 meta rows
correctly, with seven leaking to google and none to openai. The photo-first
0.50 margin does not transfer to a 66-image class (0.398 mean recall); argmax
is the honest operating rule until the corpus grows. Muse Image is therefore
a separable fourth provider class on pixels, unlike the Microsoft front-end.
Artifacts: `meta-muse-corpus-2026-08-26/`, `four-head-provider-2026-08-26/`,
`meta-provider-cells-2026-08-26/`.
A paired chat-vs-API check then closed the label question. Two prompts from
the API grid (weathered fisherman portrait, white-sneaker product shot) were
submitted to the consumer Meta AI chat imagine flow in the user's own logged-in
browser session; the chat delivered 1280x1920 and 1920x1280 WebP files from the
`t39.105495-1` CDN family, each carrying IPTC `trainedAlgorithmicMedia` in the
downloaded bytes. Scored by the frozen API-trained four-head model, which had
never seen a chat image: both chat rows landed on `meta_muse_image` under
argmax (0.251 and 0.233) with score profiles matching their API twins almost
exactly, and the margin rule agreed pairwise (fisherman meta at margin, sneaker
no_ai at margin 0.50, same as its API counterpart). The chat imagine pipeline
and `muse-image-1.0` are therefore pixel-indistinguishable to the provider
classifier on this pair, so `meta_muse_image` honestly names both
distributions. Caveat: n=2 paired prompts from one session; this is a
consistency result, not a deployment-scale equivalence claim. Artifact:
`meta-chat-check-2026-08-26/`.
Collapsing OpenAI and Gemini into one pixel class versus other generators
does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%,
Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at
+16 -6
View File
@@ -689,12 +689,22 @@ def test_non_divisible_fold_matches_modulo_cell_means() -> None:
rng = np.random.default_rng(44041)
pixels = rng.integers(0, 256, size=(53, 71, 3), dtype=np.uint8)
source = pixels.astype(np.float32)
residual = source - cv2.GaussianBlur(
source,
(0, 0),
sigmaX=1.25,
sigmaY=1.25,
borderType=cv2.BORDER_REFLECT_101,
# Mirror the module's documented per-channel blur: OpenCV's multi-channel
# GaussianBlur is not bit-identical to per-channel calls (observed 3e-5 on
# cv2 4.10.0), so a three-channel reference cannot satisfy atol=0.
residual = np.stack(
[
source[:, :, channel]
- cv2.GaussianBlur(
source[:, :, channel].copy(),
(0, 0),
sigmaX=1.25,
sigmaY=1.25,
borderType=cv2.BORDER_REFLECT_101,
)
for channel in range(3)
],
axis=2,
)
expected = np.empty((16, 16, 3), dtype=np.float64)
for tile_y in range(16):