Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
5052dbe63c | ||
|
|
2fe4d3c3a0 | ||
|
|
adbc6a3046 | ||
|
|
326840bab4 | ||
|
|
29de21256b | ||
|
|
07b07634cf | ||
|
|
54a19059d5 | ||
|
|
5e85f3ac6f | ||
|
|
36c36212f0 | ||
|
|
84ea8ede2e | ||
|
|
8f30f8bdb3 | ||
|
|
925401c905 | ||
|
|
55e7c53699 | ||
|
|
2ff06d4c5a | ||
|
|
1410a8e064 | ||
|
|
11ca7d6054 |
No files matched your search
@@ -54,6 +54,9 @@ Do not classify an entire module as untestable because its main path downloads a
|
||||
stage's; `sdxl-zimage` shipped doing exactly that and crashed on every image with a
|
||||
face. When one profile inherits another's stage, guard the invariants that stage
|
||||
relies on, not just the code path.
|
||||
- the photo-classify gate in `test_classify.py`: DEFINITELY is ridge AND MLP,
|
||||
POSSIBLY does not emit a provider, and `identify` does not import
|
||||
`remove_ai_watermarks.classify`.
|
||||
- the `InvisibleOptions` defaults, in `test_api.py`. When one signature promises to
|
||||
mirror another, compare them field by field rather than pinning the values you happen
|
||||
to know about, so the next field added on one side and not the other fails at the
|
||||
@@ -89,6 +92,23 @@ status set; `TestIdentifyRealSamples::test_no_committed_fixture_reports_a_truste
|
||||
is that guard. Read a capability the stack does not have as a missing input, never as a
|
||||
negative finding.
|
||||
|
||||
## The docs are a seam, and they drift like one
|
||||
|
||||
`tests/test_docs_cover_the_public_surface.py` reads the live Click tree and the
|
||||
live `InvisibleOptions` dataclass and requires every knob to be NAMED in the
|
||||
user-facing pages. The code-to-code seams were already guarded and the
|
||||
code-to-docs seam was not, so it drifted exactly the way an unguarded seam does:
|
||||
`docs/cli.md` named none of `--adaptive-polish`, `--controlnet-scale`,
|
||||
`--detect/--no-detect`, `--humanize`, `--strength`, `--tile-size`,
|
||||
`--tile-overlap` or `--unsharp`, and `docs/python-api.md` named 9 of the 16
|
||||
option fields.
|
||||
|
||||
Group option spellings by the Click PARAMETER, never by string surgery on the
|
||||
names. A boolean pair can be spelled anything -- `--strip-metadata/--keep-metadata`,
|
||||
`--keep-standard/--remove-all` -- so deriving the partner from a `--no-` prefix
|
||||
silently splits those pairs and demands both halves. Either spelling documents
|
||||
the pair.
|
||||
|
||||
## One measurement, one gate seam
|
||||
|
||||
A detector is split into a trust-level-blind scan and a verdict that applies the
|
||||
@@ -103,6 +123,11 @@ rules follow, and both were broken in practice before they were written down:
|
||||
- Detection and the removal mask must read ONE sweep. The winning box travels on
|
||||
`TextMarkDetection.match_box` and the registry threads the detection into the mask
|
||||
builder; a mask path that re-runs its own sweep is how the two drift apart.
|
||||
- When the research lattice runtime in `scripts/synthid_runtime/` routes by image
|
||||
geometry, preserve the returned `SynthIDDetection.detector` in score manifests
|
||||
and downstream routers. Record an inactive expert as explicitly unsupported;
|
||||
never attribute a routed large-image score to the fixed expert. This expert is
|
||||
not part of the public package.
|
||||
|
||||
The C2PA manifest-store JSON is NOT stable across reads: the reader regenerates manifest
|
||||
URNs and instance ids. Compare the derived `c2pa_info`, never the raw store.
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
# HuggingFace token (optional; only needed for gated/private models)
|
||||
# Get yours at: https://huggingface.co/settings/tokens
|
||||
# HF_TOKEN=
|
||||
|
||||
# OpenAI API key (optional; only for explicit verify-openai-synthid uploads)
|
||||
# Create and manage keys at: https://platform.openai.com/api-keys
|
||||
# OPENAI_API_KEY=
|
||||
@@ -0,0 +1,72 @@
|
||||
name: Publish photo classifier to Hugging Face
|
||||
|
||||
# Manual only. Not part of a library release. The Space demo
|
||||
# (wiltodelta/remove-ai-watermarks) stays on distribute.yml / raiw-hf-space.
|
||||
#
|
||||
# Hub model: wiltodelta/raiw-photo-classify
|
||||
# Secret: HF_TOKEN, write-role Hugging Face token (same secret Space sync uses).
|
||||
#
|
||||
# Weights are not in git. Full mode downloads them from a GitHub Release
|
||||
# of this repository, then uploads with the card from docs/photo-classify-hf/.
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
confirm:
|
||||
description: "Type publish"
|
||||
required: true
|
||||
mode:
|
||||
description: "full = card plus weights from the release tag; card = README and operating-point only"
|
||||
type: choice
|
||||
options:
|
||||
- full
|
||||
- card
|
||||
default: full
|
||||
weights_tag:
|
||||
description: "GitHub Release tag that holds clip-l-ft.pt, probe, detector.pt, provider.pt"
|
||||
required: false
|
||||
default: photo-classify-freeze-2026-08-31
|
||||
|
||||
jobs:
|
||||
publish:
|
||||
if: ${{ github.event.inputs.confirm == 'publish' }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 180
|
||||
permissions:
|
||||
contents: read
|
||||
steps:
|
||||
- uses: actions/checkout@v7
|
||||
|
||||
- uses: astral-sh/setup-uv@v7
|
||||
with:
|
||||
python-version: "3.12"
|
||||
|
||||
- name: Download freeze weights from the GitHub Release
|
||||
if: ${{ github.event.inputs.mode == 'full' }}
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
TAG: ${{ github.event.inputs.weights_tag }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
mkdir -p weights
|
||||
gh release download "$TAG" --repo "${{ github.repository }}" --dir weights
|
||||
ls -l weights
|
||||
|
||||
- name: Publish to Hugging Face
|
||||
env:
|
||||
HF_TOKEN: ${{ secrets.HF_TOKEN }}
|
||||
MODE: ${{ github.event.inputs.mode }}
|
||||
SHA: ${{ github.sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ -z "${HF_TOKEN:-}" ]; then
|
||||
echo "HF_TOKEN secret is not set" >&2
|
||||
exit 1
|
||||
fi
|
||||
extra=(--message "Photo-classify freeze (GitHub ${SHA:0:7})")
|
||||
if [ "$MODE" = "card" ]; then
|
||||
extra+=(--card-only)
|
||||
else
|
||||
extra+=(--src weights)
|
||||
fi
|
||||
uv run --no-project --with huggingface-hub python scripts/publish_photo_classify_hf.py "${extra[@]}"
|
||||
@@ -5,6 +5,9 @@ on:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
group: tests-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
@@ -17,13 +17,24 @@ on:
|
||||
workflows: ["Distribute on release"]
|
||||
types: [completed]
|
||||
|
||||
# No step reads the GitHub token any more: the ComfyUI check moved to the
|
||||
# registry's public API. Keep the job read-only so it cannot write on a repo
|
||||
# whose default token permission is read/write.
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
verify:
|
||||
# Only run on a successful distribute (not on manual dispatch, where the
|
||||
# version input or PyPI-latest provides the target).
|
||||
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
timeout-minutes: 25
|
||||
env:
|
||||
# The Space id appears in the Hub API, the raw-file URL and the app host,
|
||||
# spelled three different ways. Name it once so a move cannot half-land.
|
||||
SPACE_REPO: wiltodelta/remove-ai-watermarks
|
||||
SPACE_URL: https://wiltodelta-remove-ai-watermarks.hf.space
|
||||
|
||||
steps:
|
||||
- name: Resolve target version
|
||||
@@ -89,63 +100,124 @@ jobs:
|
||||
echo " Homebrew: formula points to $VERSION"
|
||||
|
||||
# -- ComfyUI ------------------------------------------------------------------
|
||||
- name: Verify ComfyUI sync succeeded
|
||||
- name: Verify ComfyUI registry node requires this release
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
VERSION: ${{ steps.v.outputs.version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Checking ComfyUI node sync..."
|
||||
# The registry itself has no public JSON API; the sync workflow in
|
||||
# the node repo already tests compatibility and publishes only on
|
||||
# success, so verifying its latest run succeeded is the proxy.
|
||||
latest=$(gh run list \
|
||||
--repo wiltodelta/ComfyUI-remove-ai-watermarks \
|
||||
--workflow sync-library.yml \
|
||||
--limit 1 \
|
||||
--json conclusion \
|
||||
| python3 -c "
|
||||
import json, sys
|
||||
runs = json.load(sys.stdin)
|
||||
r = runs[0] if runs else {}
|
||||
print(r.get('conclusion', 'unknown'))
|
||||
")
|
||||
echo " ComfyUI sync: $latest"
|
||||
if [ "$latest" != "success" ]; then
|
||||
echo "::warning::ComfyUI sync latest run is '$latest', not success."
|
||||
echo " Check: https://github.com/wiltodelta/ComfyUI-remove-ai-watermarks/actions"
|
||||
else
|
||||
echo " ComfyUI: sync succeeded, node published"
|
||||
fi
|
||||
echo "Checking the ComfyUI Registry node..."
|
||||
# The registry publishes the node as JSON, including the dependency the
|
||||
# published version declares. Read that directly instead of proxying
|
||||
# through the sync workflow's run status: the node repo runs
|
||||
# sync-library.yml on a daily schedule too, so a green nightly run says
|
||||
# nothing about whether THIS release was ever synced.
|
||||
node=$(curl -fsSL "https://api.comfy.org/nodes/remove-ai-watermarks")
|
||||
echo "$node" | VERSION="$VERSION" python3 -c "
|
||||
import json, os, sys
|
||||
version = os.environ['VERSION']
|
||||
latest = json.load(sys.stdin).get('latest_version') or {}
|
||||
deps = latest.get('dependencies') or []
|
||||
floor = f'>={version}'
|
||||
if latest.get('status') != 'NodeVersionStatusActive':
|
||||
sys.exit(f\"::error::ComfyUI node {latest.get('version')} is {latest.get('status')}, not active\")
|
||||
if not any(floor in d for d in deps):
|
||||
sys.exit(
|
||||
f'::error::ComfyUI node {latest.get(\"version\")} does not require '
|
||||
f'{version}; it declares {deps}'
|
||||
)
|
||||
print(f' ComfyUI: node {latest[\"version\"]} requires {version}')
|
||||
"
|
||||
echo "- ComfyUI: registry node requires $VERSION" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
# -- HF Space -------------------------------------------------------------------
|
||||
- name: Verify HF Space is alive
|
||||
- name: Verify the HF Space serves this release
|
||||
env:
|
||||
VERSION: ${{ steps.v.outputs.version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
echo "Checking HF Space..."
|
||||
space_url="https://wiltodelta-remove-ai-watermarks.hf.space"
|
||||
# The Space may be sleeping after a factory rebuild; wake it.
|
||||
for attempt in $(seq 1 12); do
|
||||
status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 30 "$space_url/" || echo "000")
|
||||
if [ "$status" = "200" ]; then
|
||||
echo " HF Space: alive (HTTP $status)"
|
||||
exit 0
|
||||
fi
|
||||
echo " HF Space: HTTP $status (attempt $attempt), waiting for rebuild..."
|
||||
echo "Checking the HF Space pin..."
|
||||
# The Space pin bump is a MANUAL step (see docs/release-and-distribution.md):
|
||||
# distribute.yml only factory-rebuilds whatever is already pinned. Failing
|
||||
# here is the point -- a demo left on an older release is an incomplete
|
||||
# release, and only this check can say so.
|
||||
reqs=$(curl -fsSL "https://huggingface.co/spaces/$SPACE_REPO/raw/main/requirements.txt")
|
||||
pinned=$(printf '%s\n' "$reqs" | sed -n 's/^remove-ai-watermarks==\([0-9][0-9a-zA-Z.]*\).*/\1/p' | head -1)
|
||||
if [ -z "$pinned" ]; then
|
||||
echo "::error::Space requirements.txt has no remove-ai-watermarks== pin"
|
||||
exit 1
|
||||
fi
|
||||
if [ "$pinned" != "$VERSION" ]; then
|
||||
echo "::error::HF Space is pinned to $pinned, not $VERSION."
|
||||
echo " Bump remove-ai-watermarks in wiltodelta/raiw-hf-space (pyproject.toml,"
|
||||
echo " uv lock, re-exported requirements.txt) and push; sync-to-hf.yml mirrors it."
|
||||
exit 1
|
||||
fi
|
||||
echo " HF Space: requirements pin is $pinned"
|
||||
|
||||
- name: Verify the HF Space is running and answering
|
||||
env:
|
||||
VERSION: ${{ steps.v.outputs.version }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# A factory rebuild leaves the Space BUILDING for a while; wait for the
|
||||
# runtime stage the API reports rather than for a bare HTTP 200, which a
|
||||
# build page also returns.
|
||||
for attempt in $(seq 1 20); do
|
||||
stage=$(curl -fsSL "https://huggingface.co/api/spaces/$SPACE_REPO" \
|
||||
| python3 -c "import json,sys;print((json.load(sys.stdin).get('runtime') or {}).get('stage',''))")
|
||||
echo " attempt $attempt: runtime stage=$stage"
|
||||
[ "$stage" = "RUNNING" ] && break
|
||||
case "$stage" in
|
||||
BUILD_ERROR|RUNTIME_ERROR|CONFIG_ERROR)
|
||||
echo "::error::HF Space runtime stage is $stage"; exit 1 ;;
|
||||
esac
|
||||
sleep 15
|
||||
done
|
||||
echo "::warning::HF Space did not respond with HTTP 200 after 3 minutes."
|
||||
echo " The Space may still be rebuilding after a factory restart."
|
||||
echo " Verify manually at $space_url"
|
||||
if [ "$stage" != "RUNNING" ]; then
|
||||
echo "::error::HF Space never reached RUNNING (last stage: $stage)"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Alive is not the same as working. Drive the two CPU surfaces the demo
|
||||
# exists for, against the Space's own committed example, and require a
|
||||
# real verdict rather than any HTTP 200.
|
||||
export PATH="$HOME/.local/bin:$PATH"
|
||||
curl -fsSL -o /tmp/example.png \
|
||||
"https://huggingface.co/spaces/$SPACE_REPO/resolve/main/examples/doubao-aigc.png"
|
||||
uv run --no-project --with 'gradio_client>=1,<2' python - <<'PYEOF'
|
||||
import os
|
||||
import sys
|
||||
|
||||
from gradio_client import Client, handle_file
|
||||
|
||||
# The committed example is a 5 MB PNG, and httpx's default 5 s write
|
||||
# timeout aborts that upload on a slow link before the Space ever sees
|
||||
# it -- which would read as a broken demo rather than a slow one.
|
||||
client = Client(os.environ["SPACE_URL"], verbose=False, httpx_kwargs={"timeout": 300})
|
||||
|
||||
report = client.predict(path=handle_file("/tmp/example.png"), api_name="/run_identify")
|
||||
if "Verdict:" not in report:
|
||||
sys.exit(f"::error::/run_identify returned no verdict: {report[:200]!r}")
|
||||
print(f" HF Space identify: {report.splitlines()[0]}")
|
||||
|
||||
_image, _file, info = client.predict(
|
||||
path=handle_file("/tmp/example.png"),
|
||||
sensitivity_label="auto (recommended)",
|
||||
api_name="/run_visible",
|
||||
)
|
||||
if "Removed:" not in info:
|
||||
sys.exit(f"::error::/run_visible removed nothing on the Doubao example: {info!r}")
|
||||
print(f" HF Space visible: {info.splitlines()[0]}")
|
||||
PYEOF
|
||||
echo "- HF Space: pinned to $VERSION, RUNNING, identify + visible answered" >> "$GITHUB_STEP_SUMMARY"
|
||||
|
||||
- name: Summary
|
||||
env:
|
||||
VERSION: ${{ steps.v.outputs.version }}
|
||||
run: |
|
||||
cat <<EOF
|
||||
Release $VERSION verified:
|
||||
PyPI: wheel + sdist present
|
||||
Clean install: --version returns $VERSION
|
||||
Homebrew: formula points to $VERSION
|
||||
ComfyUI: sync succeeded
|
||||
HF Space: alive and responding
|
||||
EOF
|
||||
# Every check above appends its own line to the step summary as it passes,
|
||||
# and every one of them fails the job instead of warning. So this heading
|
||||
# can only be reached when all of them actually held -- an unconditional
|
||||
# "verified" block printed here once claimed surfaces nobody had checked.
|
||||
sed -i "1i ## Release $VERSION verified" "$GITHUB_STEP_SUMMARY"
|
||||
cat "$GITHUB_STEP_SUMMARY"
|
||||
@@ -31,6 +31,8 @@ Optional features and installation groups are documented in [`docs/installation.
|
||||
|
||||
`maintain.sh` runs dependency freshness and security checks, Ruff, Pyright scoped to `src/`, and the parallel test suite. Full-project Pyright is not the project gate because the ML dependency graph can exhaust Node memory.
|
||||
|
||||
This gate does NOT pass `--ignore-unfixed` to uv-secure, so a transitive CVE with no released fix stops it before Ruff, Pyright and the tests ever run. That is deliberate and it is not a blocker: run those three separately and report them, per the triage in [`docs/development.md`](docs/development.md). A red gate here is not the same signal as a red gate in a sibling repository that does ignore unfixed findings.
|
||||
|
||||
Command, gate, typing, and model-test invariants auto-load from [`.claude/rules/development.md`](.claude/rules/development.md). Environment recovery, CI behavior, and fixture policy live in [`docs/development.md`](docs/development.md).
|
||||
|
||||
Before a release, read [`docs/release-and-distribution.md`](docs/release-and-distribution.md). Treat the release as complete only after PyPI, Homebrew, the Hugging Face Space, and the ComfyUI Registry are verified; conda is not published. Keep the source-distribution public allowlist.
|
||||
@@ -39,7 +41,7 @@ Before a release, read [`docs/release-and-distribution.md`](docs/release-and-dis
|
||||
|
||||
[`docs/module-internals.md`](docs/module-internals.md) is the canonical per-module map, including design decisions, thresholds, calibration history, incident records, and regression guards. Read the relevant section before changing a subsystem.
|
||||
|
||||
Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md).
|
||||
Research and current constraints are routed through [`docs/index.md`](docs/index.md), especially [`docs/known-limitations.md`](docs/known-limitations.md), [`docs/supported-signals.md`](docs/supported-signals.md), [`docs/synthid.md`](docs/synthid.md), and [`docs/watermarking-landscape.md`](docs/watermarking-landscape.md). Pixel photo classification is [`docs/photo-classify.md`](docs/photo-classify.md); it is not `identify`. Classifier research is split between general [`AI-generated image classifiers`](docs/ai-generated-image-classifiers.md) and [`SynthID source classifiers`](docs/synthid-classifiers.md). Other SynthID campaign logs are [`docs/synthid-detector-research.md`](docs/synthid-detector-research.md) and [`docs/synthid-removal-research.md`](docs/synthid-removal-research.md). The pre-split chronological archive is [`docs/synthid-detector-removal-plan.md`](docs/synthid-detector-removal-plan.md).
|
||||
|
||||
## Data safety
|
||||
|
||||
@@ -51,4 +53,4 @@ Topic-specific rules live in `.claude/rules/*.md` and are auto-loaded when match
|
||||
|
||||
| File | Covers |
|
||||
|---|---|
|
||||
| `development.md` | Command contracts, project gate, typing boundaries, model-adjacent tests, and the detection-path measurement rule |
|
||||
| `development.md` | Command contracts, project gate, typing boundaries, model-adjacent tests, the docs-coverage seam, and the detection-path measurement rule |
|
||||
@@ -7,8 +7,8 @@ Remove AI provenance marks from images and video you generated yourself:
|
||||
- C2PA, EXIF, XMP, IPTC, and related AI metadata.
|
||||
|
||||
Video support covers provenance identification, complete visible-plus-metadata
|
||||
cleaning, directory batches, visible Sora, Veo, Seedance, Dola, Hailuo AI, and
|
||||
Kling AI mark removal, and oracle-certified VAE regeneration for video SynthID
|
||||
cleaning, directory batches, visible Sora, Veo, Seedance, Doubao, Dola,
|
||||
Hailuo AI, and Kling AI mark removal, and oracle-certified VAE regeneration for video SynthID
|
||||
removal.
|
||||
|
||||
> [raiw.cc](https://raiw.cc) runs this library as a hosted service, with the GPU
|
||||
@@ -32,6 +32,8 @@ removal.
|
||||
| Goal | Command | GPU |
|
||||
| --- | --- | --- |
|
||||
| Find provenance signals and watermarks | `identify` | No |
|
||||
| Classify a photograph from pixels (opt-in, not provenance) | `classify` | No |
|
||||
| Verify supported OpenAI SynthID from pixels with the official remote API | `verify-openai-synthid` | No |
|
||||
| Remove known visible AI marks | `visible` | No |
|
||||
| Erase a region you select | `erase` | No |
|
||||
| Strip AI metadata | `metadata` | No |
|
||||
@@ -54,6 +56,8 @@ its linked C2PA manifest; metadata stripping alone removes only the manifest.
|
||||
| Need | Install |
|
||||
| --- | --- |
|
||||
| Metadata inspection and stripping | `remove-ai-watermarks` |
|
||||
| Photograph AI-versus-camera classification | `remove-ai-watermarks[classify]` |
|
||||
| Official remote OpenAI SynthID verification | `remove-ai-watermarks[verify]` |
|
||||
| Visible detection and removal | `remove-ai-watermarks[visible]` |
|
||||
| Visible video processing | `remove-ai-watermarks[video]` |
|
||||
| Video SynthID removal | `remove-ai-watermarks[video,diffusion]` |
|
||||
@@ -80,6 +84,34 @@ Inspect an image:
|
||||
remove-ai-watermarks identify image.png
|
||||
```
|
||||
|
||||
To classify a photograph from pixels (AI versus camera, optional provider),
|
||||
install the extra and call `classify`. `identify` never starts it:
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[classify]"
|
||||
remove-ai-watermarks classify image.png
|
||||
```
|
||||
|
||||
Guide: [photo pixel classification](docs/photo-classify.md).
|
||||
|
||||
Signed provenance is the supported route for SynthID and `identify` reads it.
|
||||
There is no local SynthID pixel detector in the package. Research on a
|
||||
periodic lattice expert is in `scripts/synthid_runtime/` and
|
||||
[synthid-detector-research.md](docs/synthid-detector-research.md).
|
||||
|
||||
For supported OpenAI images, the optional official verifier provides a pixel
|
||||
watermark verdict:
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[verify]"
|
||||
remove-ai-watermarks verify-openai-synthid image.png --acknowledge-upload
|
||||
```
|
||||
|
||||
The command removes AI provenance metadata from a temporary copy, verifies that
|
||||
the decoded pixels are unchanged, uploads only that copy to OpenAI, and consumes
|
||||
only the independent SynthID response. It never runs implicitly from `identify`.
|
||||
An API key, endpoint access, and explicit upload acknowledgement are required.
|
||||
|
||||
For visible watermark removal, install the pixel dependencies:
|
||||
|
||||
```bash
|
||||
@@ -369,6 +401,9 @@ import remove_ai_watermarks as raiw
|
||||
result, removed = raiw.remove_visible("watermarked.png", "clean.png")
|
||||
print(removed)
|
||||
|
||||
openai_synthid = raiw.verify_openai_synthid("image.png", acknowledge_upload=True)
|
||||
print(openai_synthid.status)
|
||||
|
||||
provenance = raiw.identify_video("input.mp4")
|
||||
report = raiw.inspect_video_metadata("input.mp4")
|
||||
complete = raiw.remove_video_all("input.mp4", "clean.mp4")
|
||||
@@ -411,7 +446,7 @@ invisible removal.
|
||||
detail.
|
||||
- Visible video removal recognizes the moving Sora 2 wordmark, the current Veo
|
||||
diamond plus legacy `Veo` text, the Seedance boxed `AI` label, and the fixed
|
||||
Dola, Hailuo AI, and Kling AI labels. It does not recognize the older Sora Turbo
|
||||
Doubao, Dola, Hailuo AI, and Kling AI labels. It does not recognize the older Sora Turbo
|
||||
corner swirl or unregistered layouts from those providers.
|
||||
The classical OpenCV backend can smear structured backgrounds; use MI-GAN or
|
||||
LaMa when recovery quality matters.
|
||||
@@ -451,8 +486,11 @@ reinterpret the pixel result returned `UNAVAILABLE`; that follow-up was not a
|
||||
verifier rerun and does not invalidate the built-in verdicts. A 2026-07-31
|
||||
full-clip check on a public eight-second Veo sample found `0.10` still detected
|
||||
and `0.15` not detected, so `0.15` is now the certified default. The
|
||||
reproducible hashes and verdicts live in
|
||||
`data/evaluations/video-synthid-oracle.csv`.
|
||||
reproducible hashes and verdicts for that full-clip check -- one carrier, two
|
||||
rows -- live in `data/evaluations/video-synthid-oracle.csv`. The earlier
|
||||
two-clip calibration is narrative only: its verdicts were not recorded in a
|
||||
tracked manifest, so treat the certified default as resting on the 2026-07-31
|
||||
rows.
|
||||
|
||||
## Documentation
|
||||
|
||||
|
||||
@@ -17,6 +17,8 @@ data/
|
||||
full-pipeline-quality.csv
|
||||
Reusable full-pipeline evaluation selection
|
||||
evaluations/
|
||||
engine-selection/
|
||||
Prompt-matched AI content matrix for deterministic auto-engine research
|
||||
fidelity/ Evaluation instructions and hand-verified ground truth
|
||||
video-synthid-oracle.csv
|
||||
Reproducible full-clip Gemini SynthID verdicts
|
||||
@@ -24,6 +26,12 @@ data/
|
||||
|
||||
## Storage rules
|
||||
|
||||
Large local-only research data (the 211 GB Spaces corpus and the frozen
|
||||
checkpoints, research corpora, and campaign reports) lives under gitignored
|
||||
paths documented in `data/research/README.md` (ignored via `.git/info/exclude`,
|
||||
never committed). Worktree `.local-eval/` directories are scratch copies; the
|
||||
durable store is `data/research/` in the main checkout.
|
||||
|
||||
1. Store each binary image once. Evaluation manifests and documentation point
|
||||
to its canonical location.
|
||||
2. Put executable test fixtures in `fixtures/`.
|
||||
@@ -34,6 +42,9 @@ data/
|
||||
commands, hashes, and oracle verdicts instead of committing another corpus
|
||||
copy. A small curated before-and-after example may live in `docs/images/`
|
||||
when it is part of the public documentation.
|
||||
`evaluations/engine-selection/` is the bounded exception for ORIGINAL study
|
||||
inputs: its manifest tracks a prompt-matched OpenAI/Meta content matrix, not
|
||||
generated model outputs or watermark-oracle evidence.
|
||||
6. Keep third-party fixture license notices in `licenses/`, outside directories
|
||||
that tests enumerate as media inputs.
|
||||
7. Runtime detector assets belong in `src/remove_ai_watermarks/assets/`.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Engine-selection fixture set
|
||||
|
||||
This directory is the tracked, content-balanced input set for researching a
|
||||
deterministic `auto` engine choice. It is not a training corpus and it is not a
|
||||
watermark-oracle corpus.
|
||||
|
||||
## Content matrix
|
||||
|
||||
`content-manifest.csv` contains 19 prompt-matched pairs. Each pair uses the
|
||||
same prompt and generation index across OpenAI `gpt-image-1-mini` and Meta
|
||||
`muse-image-1.0`, covering the content classes that materially stress image
|
||||
regeneration:
|
||||
|
||||
- natural and urban scenes;
|
||||
- single-person portraits and people in context;
|
||||
- animals, food, products, interiors, architecture, macro detail, and action;
|
||||
- flat, isometric, watercolor, and surreal illustration styles.
|
||||
|
||||
The files are exact copies of generation outputs collected for the
|
||||
`docs/arxiv-paper-review` research branch. The manifest records the SHA-256 of
|
||||
the committed bytes. The old OpenAI collection manifest stored the API payload
|
||||
hash under `sha256`; `source_payload_sha256` preserves that value while
|
||||
`sha256` always means the actual file bytes here. Meta's two hashes are the
|
||||
same.
|
||||
|
||||
One generation per prompt and provider is enough for the first paired
|
||||
discovery pass. Any policy inferred from it must be challenged against unused
|
||||
generation indices from the local research archive before shipping. Generated
|
||||
outputs belong outside this directory.
|
||||
|
||||
`carrier-manifest.csv` adds the canonical, already tracked signal carriers to
|
||||
the study without copying their bytes. Together the two manifests cover
|
||||
ordinary content, text, single faces, face grids, and mixed face/text scenes.
|
||||
|
||||
## Signal carriers
|
||||
|
||||
The content matrix was re-encoded by its collection pipeline and has no local
|
||||
provenance signal. It must not be used to set or certify watermark-removal
|
||||
strengths. The tracked oracle/carrier inputs remain in their canonical homes:
|
||||
|
||||
- OpenAI and Google SynthID: `data/synthid/originals/` with
|
||||
`data/synthid/manifest.csv`;
|
||||
- Meta Content Seal: `data/contentseal/originals/` with
|
||||
`data/contentseal/manifest.csv`.
|
||||
|
||||
There is no tracked, publication-cleared Microsoft InvisMark carrier yet. The
|
||||
three Paint sources used by the earlier calibration are user uploads and remain
|
||||
outside git. A Microsoft row may be added only from a newly generated or
|
||||
otherwise publication-cleared source, with a pixel-identical metadata-stripped
|
||||
control that remains positive in Microsoft's detector.
|
||||
|
||||
## Integrity check
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run python scripts/verify_engine_selection_fixtures.py
|
||||
```
|
||||
|
||||
The check validates paths, byte hashes, dimensions, prompt pairing, and the
|
||||
declared provider/content matrix without loading any ML model.
|
||||
@@ -0,0 +1,13 @@
|
||||
provider,file,content_stratum,signal,verified_via,sha256,verified_text_geometry,verified_text_ground_truth
|
||||
openai,"../../synthid/originals/ChatGPT Image May 30, 2026, 10_31_08 AM.png",face_grid,OpenAI SynthID,openai-verify,28ff8732b037f98a4ef5bc277bbcdaa32e5eb9ccbd00b6c8c616e46ef68ae8a0,no,no
|
||||
openai,"../../synthid/originals/ChatGPT Image May 31, 2026, 02_02_23 PM.png",multilingual_typography,OpenAI SynthID,c2pa-metadata,228e54413a3ef62b313176dcdba774a56a58c36b54227d1440bc26cbe0aefd82,yes,yes
|
||||
openai,"../../synthid/originals/ChatGPT Image May 31, 2026, 02_03_55 PM.png",multilingual_typography,OpenAI SynthID,openai-verify,05b836ecfe40fd689177fda74384ae4fdcc446505bbc4281cd3cbb6523eb669e,yes,yes
|
||||
google,../../synthid/originals/Gemini_Generated_Image_633uuy633uuy633u.png,cjk_typography_scene,Google SynthID,gemini-app,4affd7f27767a445db6abf741355743ba8d95108ad922c9fff045feed8492236,yes,yes
|
||||
google,../../synthid/originals/Gemini_Generated_Image_akdbeiakdbeiakdb.png,cyrillic_typography_scene,Google SynthID,gemini-app,8c1a6fb03ef3d45a1f958fb3401e4264e409ff88c2a793061db7f29023454d0e,no,no
|
||||
google,../../synthid/originals/Gemini_Generated_Image_y48j3cy48j3cy48j.png,crowd_faces,Google SynthID,gemini-app,45d79a683134fcba1b147b2aedb669783d474e1fb8a4df329729a0904fd1b46b,no,no
|
||||
google,../../synthid/originals/Gemini_Generated_Image_3mc4t93mc4t93mc4.png,faces_and_typography,Google SynthID,gemini-app,2c33e75a2db614ce74c83cc0a6ac6c3ac735aca83ab88c9c9345843b124f7856,no,no
|
||||
meta,../../contentseal/originals/gen_fox_forest.webp,photoreal_texture,Meta Content Seal,meta-oracle,6d7cecf322a5ac4b2547e2f32c767b3cb3e8a0e49f274cfd00aa31be6697eed4,no,no
|
||||
meta,../../contentseal/originals/gen_text_poster.webp,typography,Meta Content Seal,meta-oracle,873927662d8198e9c38b4ae71a2919ba5d417dfcd714649db1b0ae231e8cbf9b,no,no
|
||||
meta,../../contentseal/originals/gen_lighthouse_watercolor.webp,watercolor,Meta Content Seal,meta-oracle,69bee7c376608568361085602c5c7406900885849bd94c5279c3924cf793b27c,no,no
|
||||
meta,../../contentseal/originals/gen_night_city.webp,urban_night,Meta Content Seal,meta-oracle,345adf6b9f777b15ef7ce7bbdfb3b355a1d9e882740809e966e9f6caafd94d48,no,no
|
||||
meta,../../contentseal/originals/gen_studio_mug.webp,product_still_life,Meta Content Seal,meta-oracle,2b7bc7fd56fa2250c66ef74f8b3e49d9666fe147d894d5baf9461b5e36077a07,no,no
|
||||
|
@@ -0,0 +1,39 @@
|
||||
pair_id,provider,model,file,width,height,content_stratum,sha256,source_payload_sha256,prompt
|
||||
00,openai,gpt-image-1-mini,originals/openai/00.png,1024,1024,natural_landscape,698896455e0f1de57a227339227bf068143e690873b9c8c9672b31578de93ba0,1345f59fc7b51a1edf837745245bb90811feefd9ba7f9ba6f12dc89846aeba18,"a stunning photograph of a mountain lake at sunrise, highly detailed, professional photography"
|
||||
00,meta,muse-image-1.0,originals/meta/00.png,1600,1600,natural_landscape,83362c189b79cd0e700296b975e9ef8e7f2b2fe89813e407bc795299400ca5e0,83362c189b79cd0e700296b975e9ef8e7f2b2fe89813e407bc795299400ca5e0,"a stunning photograph of a mountain lake at sunrise, highly detailed, professional photography"
|
||||
01,openai,gpt-image-1-mini,originals/openai/01.png,1024,1024,street_scene,b885d8cb1490e4fbbe580b87750008780671390097d28e71ff18b50cf320ec85,772518e4686940d2c68df114298fb007a97a5c42548245092e989b6ca9bd5b90,"a vibrant street market in Bangkok with colorful fruits and spices, candid photo"
|
||||
01,meta,muse-image-1.0,originals/meta/01.png,1600,1600,street_scene,5ba08cf9ab07bf00f64f39fe93f1766a64f737817b00086783476566c5f3317f,5ba08cf9ab07bf00f64f39fe93f1766a64f737817b00086783476566c5f3317f,"a vibrant street market in Bangkok with colorful fruits and spices, candid photo"
|
||||
02,openai,gpt-image-1-mini,originals/openai/02.png,1024,1024,single_portrait,97618d1a5c39ddeba708df3a9448b27d1d26583ee68b490ba79f7f3e1eb2248d,6bace9d2adae4d630e1830449f0945a30d7ddb3635e3005bc7aad80206a57176,"a close-up portrait of a woman with freckles and green eyes, studio lighting, 85mm lens"
|
||||
02,meta,muse-image-1.0,originals/meta/02.png,1600,1600,single_portrait,4daf07fc8bdfe007b96a70b00ec0cfde87f180eb8c2d5501723e218ed29e7589,4daf07fc8bdfe007b96a70b00ec0cfde87f180eb8c2d5501723e218ed29e7589,"a close-up portrait of a woman with freckles and green eyes, studio lighting, 85mm lens"
|
||||
03,openai,gpt-image-1-mini,originals/openai/03.png,1024,1024,animal_action,eb6920aab03bbe6935a01756c8f2d524d383e307369cdfaad3fe44926fbd9e48,2012abb15be639ebb3c596d2210f6d4c50cce0e11dd6cfc88aeb29de1c82de18,"a golden retriever running through a field of sunflowers, action photography"
|
||||
03,meta,muse-image-1.0,originals/meta/03.png,1600,1600,animal_action,93dce3a5049f773adb882ef22243a84e4822b9c43ad30774741c40f3844ef405,93dce3a5049f773adb882ef22243a84e4822b9c43ad30774741c40f3844ef405,"a golden retriever running through a field of sunflowers, action photography"
|
||||
04,openai,gpt-image-1-mini,originals/openai/04.png,1024,1024,reflective_architecture,275d68de881041de989261ced7570dd460028e0406346642d880d0d1664bb9ae,1808d95c69080aafced4234e4c3ad7714121bad00f3f5dc84a744288d9757245,"a modern architecture building with glass facade reflecting sunset clouds"
|
||||
04,meta,muse-image-1.0,originals/meta/04.png,1600,1600,reflective_architecture,209c71f0b737a9ef60b4d620c66f518b34dbc42af4532b2b8e0b3086183c90ec,209c71f0b737a9ef60b4d620c66f518b34dbc42af4532b2b8e0b3086183c90ec,"a modern architecture building with glass facade reflecting sunset clouds"
|
||||
05,openai,gpt-image-1-mini,originals/openai/05.png,1024,1024,food_closeup,316a743ccf29f025904d1bcd047a6996f987715e4f93e8a0d9cf85b624349b07,a8bac088c350ef849bbf78a85a1e44aa1665d1054a0cdfae29461c1cdca56e4a,"a bowl of ramen with steam rising, moody food photography, shallow depth of field"
|
||||
05,meta,muse-image-1.0,originals/meta/05.png,1600,1600,food_closeup,4a50ca3e33ddacd2897b926cdc54615a389416fcd2edf4d8803ad882b85b33af,4a50ca3e33ddacd2897b926cdc54615a389416fcd2edf4d8803ad882b85b33af,"a bowl of ramen with steam rising, moody food photography, shallow depth of field"
|
||||
06,openai,gpt-image-1-mini,originals/openai/06.png,1024,1024,sports_action,390d87d1f59eb4fa38590c0c751b2848383c77ffcf549571a0bf00db0f783763,6cc24eeb0fce95a86741b9ac0c055002793a586ba37f0cf4d64201a5c5e73513,"a surfer riding a barrel wave at golden hour, backlit spray"
|
||||
06,meta,muse-image-1.0,originals/meta/06.png,1600,1600,sports_action,1d9fcbf0f471f733ae96ca94dff892afd9f5852767000e379d3486dc94e30b78,1d9fcbf0f471f733ae96ca94dff892afd9f5852767000e379d3486dc94e30b78,"a surfer riding a barrel wave at golden hour, backlit spray"
|
||||
07,openai,gpt-image-1-mini,originals/openai/07.png,1024,1024,interior_low_light,4c77c2814216bdcde3417d2d228c5b6f275e33777f13cbea7911241bc9bc376b,9e858b95a4f842c917cbf47afeb4b7e0e24e04c4df0b19da4afe00b7316a13a9,"a cozy reading nook with warm lamp light and rain on the window"
|
||||
07,meta,muse-image-1.0,originals/meta/07.png,1600,1600,interior_low_light,ecec76c086a7b33beec1b3cb787f62cad68c09cb362a8888a8af734e9d4f3425,ecec76c086a7b33beec1b3cb787f62cad68c09cb362a8888a8af734e9d4f3425,"a cozy reading nook with warm lamp light and rain on the window"
|
||||
08,openai,gpt-image-1-mini,originals/openai/08.png,1024,1024,isometric_illustration,d688f6da74d4a7b2bd29ef59ba64864d0915134433febd8c90eb5ce1dc80781f,37a801f66b7c1bf63e613a7a5717dbc00397120a7a32ec45414661e03137a180,"an isometric illustration of a cozy coffee shop interior, warm colors, detailed"
|
||||
08,meta,muse-image-1.0,originals/meta/08.png,1600,1600,isometric_illustration,d4683d16a55c5f358f5b3b876b89fbc810ae1e3332e74399395ae20c9f646ef5,d4683d16a55c5f358f5b3b876b89fbc810ae1e3332e74399395ae20c9f646ef5,"an isometric illustration of a cozy coffee shop interior, warm colors, detailed"
|
||||
09,openai,gpt-image-1-mini,originals/openai/09.png,1024,1024,watercolor,d0beefb80ee00a0937a851932a2ed6dab2c670f10104ffe94c5ff22869a9595d,b52401235efc16641d3cb8f9c7c6d20a75c36751ada9409b65ceefa0e57901da,"a watercolor painting of Venice canals at sunset, loose brushstrokes"
|
||||
09,meta,muse-image-1.0,originals/meta/09.png,1600,1600,watercolor,c76ffbca59fbcb2dba9b66c94598f0c0062b0ac4679e348cdf1e18a06d3eb64d,c76ffbca59fbcb2dba9b66c94598f0c0062b0ac4679e348cdf1e18a06d3eb64d,"a watercolor painting of Venice canals at sunset, loose brushstrokes"
|
||||
10,openai,gpt-image-1-mini,originals/openai/10.png,1024,1024,macro_detail,59d9c6dce5ce4c2b77427144ccbafc8a4b7ab87564673677119859c31598b0d4,3740f9de304f050fc7306447c6668dcaa8156a888709f759f05715e7987a321c,"a macro photograph of a dewdrop on a spider web at dawn, bokeh background"
|
||||
10,meta,muse-image-1.0,originals/meta/10.png,1600,1600,macro_detail,c86fd9c3a59aa0bec709c5bce7c2ef1be4ecb1d780f3a3f35460b73cd97d1a12,c86fd9c3a59aa0bec709c5bce7c2ef1be4ecb1d780f3a3f35460b73cd97d1a12,"a macro photograph of a dewdrop on a spider web at dawn, bokeh background"
|
||||
11,openai,gpt-image-1-mini,originals/openai/11.png,1024,1024,urban_night,e4f6f806ed8559973374eae1e4d379807e45046749b7d491edbbc8ad9647728e,06bfc68289ece1b85b169df4cedd5e1cf12d0efa30631bdecd0e3edce4d8c82e,"a cinematic wide shot of Tokyo at night with neon reflections on wet streets"
|
||||
11,meta,muse-image-1.0,originals/meta/11.png,1600,1600,urban_night,7877cf0c13b07fbc6651178d838db4ab253369896292af1cf3f85a957a559db8,7877cf0c13b07fbc6651178d838db4ab253369896292af1cf3f85a957a559db8,"a cinematic wide shot of Tokyo at night with neon reflections on wet streets"
|
||||
12,openai,gpt-image-1-mini,originals/openai/12.png,1024,1024,flat_illustration,6bf230aa0b6f40d7ff0d23b22b8aac2e9b66288eb85c02eebdf43b71a1597f5b,65567d5cecb34241ac755ac4db5eb5a7b271d9a00fa27d826c961d2a2ef6ac3d,"a flat-design poster of Santorini with bold shapes, screen print style"
|
||||
12,meta,muse-image-1.0,originals/meta/12.png,1600,1600,flat_illustration,1b71bd94aa18d737c8a1c319a33dd0f0e9e15c2e93210c9c3aa98242b9a67955,1b71bd94aa18d737c8a1c319a33dd0f0e9e15c2e93210c9c3aa98242b9a67955,"a flat-design poster of Santorini with bold shapes, screen print style"
|
||||
13,openai,gpt-image-1-mini,originals/openai/13.png,1024,1024,storm_landscape,d07b05eef633fcb5b6b54268cc2cdd787dc19f8050198872e462f69e383a054d,a692a8a0daebad375cd57d7c671be6e16ad3ef700f0b9214daacbe80b6148c90,"a dramatic storm cloud over a lonely lighthouse, moody, high contrast photography"
|
||||
13,meta,muse-image-1.0,originals/meta/13.png,1600,1600,storm_landscape,e669128fe2d3b59dc83ae0b216c5378fe5930e77fa3a692219da238dbf55d5f6,e669128fe2d3b59dc83ae0b216c5378fe5930e77fa3a692219da238dbf55d5f6,"a dramatic storm cloud over a lonely lighthouse, moody, high contrast photography"
|
||||
14,openai,gpt-image-1-mini,originals/openai/14.png,1024,1024,monochrome_portrait,9fb1153f4d0041932b35bcd1408d774b642c68f81d7bb403d22f50f8c891ede7,826acf13df78378499e57c7fe1a8309947ab26810f46482be3a64d3328de1ff6,"a black and white portrait of an elderly craftsman in his workshop"
|
||||
14,meta,muse-image-1.0,originals/meta/14.png,1600,1600,monochrome_portrait,8339475f0585491ea08992d29890f3c759f24db2184212c6bf4f7efc0932e177,8339475f0585491ea08992d29890f3c759f24db2184212c6bf4f7efc0932e177,"a black and white portrait of an elderly craftsman in his workshop"
|
||||
15,openai,gpt-image-1-mini,originals/openai/15.png,1024,1024,surreal_composite,8bf02f9028fd049ebfafd3410d2eb9d618819be1c14b4e05d53a490e22ba109c,e71de5a9a2f6579857d2e24b8e7a1516031f19ee824906cece158101d13fe001,"a surreal double exposure of a forest and a human profile, conceptual art"
|
||||
15,meta,muse-image-1.0,originals/meta/15.png,1600,1600,surreal_composite,20628541cc924e623984604337807c7afd453aa8a7652f087add742a13828f6a,20628541cc924e623984604337807c7afd453aa8a7652f087add742a13828f6a,"a surreal double exposure of a forest and a human profile, conceptual art"
|
||||
16,openai,gpt-image-1-mini,originals/openai/16.png,1024,1024,brutalist_architecture,e91ca5128ce8644510c5c4def6bba085e45e1716105b3b6a2761cc18d42018b3,0aa8a1819f160ce26dfc9e59b996f969698fdf4eba3402fe999af598fcad80d4,"a low-angle shot of a brutalist building against dramatic clouds"
|
||||
16,meta,muse-image-1.0,originals/meta/16.png,1600,1600,brutalist_architecture,d22702d26343b6575e89751c95d865696c6085bed053f6008f989434a96bac02,d22702d26343b6575e89751c95d865696c6085bed053f6008f989434a96bac02,"a low-angle shot of a brutalist building against dramatic clouds"
|
||||
17,openai,gpt-image-1-mini,originals/openai/17.png,1024,1024,product_still_life,4f41f4a46b80c0ded61a9a3ce1273412b1728476a0734519d1af34a1797013b0,1f60c25c59138c14bacb0dd485475037e7d6222bb45daa66ed85f9dd91c83e4e,"a minimalist product photo of a ceramic mug on a marble surface, clean commercial look"
|
||||
17,meta,muse-image-1.0,originals/meta/17.png,1600,1600,product_still_life,67bde86edfd3db08b88c0dbc5572958408875e188dac6fa8ed64f821b7e7e994,67bde86edfd3db08b88c0dbc5572958408875e188dac6fa8ed64f821b7e7e994,"a minimalist product photo of a ceramic mug on a marble surface, clean commercial look"
|
||||
18,openai,gpt-image-1-mini,originals/openai/18.png,1024,1024,person_in_context,9b049d3112a654a87022d591b93a01087693c0664be5af560d3c0ec490291963,ba873bc7989720245821f5298f12e36bd7b6051909694b72569645c29cdd6bd0,"a lifestyle photo of a person working on a laptop in a bright cafe"
|
||||
18,meta,muse-image-1.0,originals/meta/18.png,1600,1600,person_in_context,cc4b2bcfc4e80d7dfe6646beb12529168137aef04d2b6765c6382b830ce19ed7,cc4b2bcfc4e80d7dfe6646beb12529168137aef04d2b6765c6382b830ce19ed7,"a lifestyle photo of a person working on a laptop in a bright cafe"
|
||||
|
|
After Width: | Height: | Size: 3.5 MiB |
|
After Width: | Height: | Size: 4.0 MiB |
|
After Width: | Height: | Size: 2.5 MiB |
|
After Width: | Height: | Size: 3.2 MiB |
|
After Width: | Height: | Size: 2.9 MiB |
|
After Width: | Height: | Size: 1.7 MiB |
|
After Width: | Height: | Size: 4.0 MiB |
|
After Width: | Height: | Size: 3.0 MiB |
|
After Width: | Height: | Size: 3.2 MiB |
|
After Width: | Height: | Size: 3.6 MiB |
|
After Width: | Height: | Size: 1.7 MiB |
|
After Width: | Height: | Size: 3.5 MiB |
|
After Width: | Height: | Size: 2.0 MiB |
|
After Width: | Height: | Size: 2.1 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 3.4 MiB |
|
After Width: | Height: | Size: 3.0 MiB |
|
After Width: | Height: | Size: 854 KiB |
|
After Width: | Height: | Size: 2.8 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.9 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.5 MiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 1.7 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.9 MiB |
|
After Width: | Height: | Size: 2.0 MiB |
|
After Width: | Height: | Size: 1.3 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 1.7 MiB |
|
After Width: | Height: | Size: 1.2 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 1.6 MiB |
|
After Width: | Height: | Size: 1.8 MiB |
|
After Width: | Height: | Size: 1.1 MiB |
|
After Width: | Height: | Size: 1.4 MiB |
@@ -54,7 +54,10 @@ versions, or runtime settings.
|
||||
Ground-truth quality, strongest first:
|
||||
|
||||
- `gemini-app` — checked via the Gemini app "Verify with SynthID" feature. Gold standard for the pixel watermark (Google models).
|
||||
- `openai-verify` — checked via openai.com/verify (gold standard for OpenAI ChatGPT/Codex/API images).
|
||||
- `openai-verify` — checked through the OpenAI web verifier. This is the
|
||||
historical manifest label for existing rows. OpenAI now also documents a
|
||||
Content Provenance API with a separate SynthID result; add a distinct manifest
|
||||
value before recording API-derived labels rather than silently reusing this one.
|
||||
- `synthid-portal` — checked via Google's SynthID Detector portal.
|
||||
- `c2pa-metadata` — supported provenance evidence (Google AI C2PA, or OpenAI
|
||||
C2PA with an explicit `c2pa.watermarked.*` action). Weaker than a provider
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
# Private SynthID research manifest
|
||||
|
||||
The research manifest is a local CSV for detector and pixel-only removal
|
||||
experiments. It is intentionally separate from `manifest.csv`, which describes
|
||||
the small public regression corpus.
|
||||
|
||||
Store the CSV and its media below `.local-eval/synthid/`. Keep one manifest per
|
||||
detector target, for example `google-manifest.csv` and `openai-manifest.csv`.
|
||||
Separate manifests may reference the same external negative bytes without
|
||||
duplicating an artifact inside either provider's split graph. Do not commit
|
||||
private media, corpus sizes, oracle sessions, or provider-access details.
|
||||
|
||||
Before assigning labels, build a label-free inventory. It records byte and
|
||||
decoded-pixel hashes, exact duplicates, geometry, and format, but deliberately
|
||||
has no provider, oracle, outcome, or split columns:
|
||||
|
||||
```bash
|
||||
uv run python scripts/synthid_research_inventory.py \
|
||||
--root .local-eval/synthid negatives google openai \
|
||||
--inventory-out .local-eval/synthid/inventory.csv
|
||||
```
|
||||
|
||||
An inventory row becomes a manifest row only after its source, detector target,
|
||||
label evidence, group lineage, and split are established independently. A
|
||||
directory name is not evidence and must not be promoted mechanically.
|
||||
|
||||
Audit each local manifest with:
|
||||
|
||||
```bash
|
||||
uv run python scripts/synthid_research_manifest.py \
|
||||
.local-eval/synthid/google-manifest.csv --verify-files
|
||||
```
|
||||
|
||||
Once a provider manifest has ordinary positives and negatives in train,
|
||||
validation, and locked-test splits, run the D1 confound challenge:
|
||||
|
||||
```bash
|
||||
uv run --extra pixels python scripts/synthid_confound_probe.py \
|
||||
.local-eval/synthid/google-manifest.csv \
|
||||
--target-provider google \
|
||||
--report-out .local-eval/synthid/google-d1-confounds.json
|
||||
```
|
||||
|
||||
D1 excludes candidate, sham, and source-control rows. Its `container`,
|
||||
`thumbnail`, and `canonical` baselines intentionally measure how well export
|
||||
fields, geometry, and coarse generator/content style can imitate detection. A
|
||||
report is evidence-ready only when the locked test includes a same-provider hard
|
||||
negative and the temporal split contains both labels. Evidence readiness does
|
||||
not mean the D1 gate passed; a candidate signal still has to beat the frozen
|
||||
canonical baseline on those controls.
|
||||
|
||||
## Columns
|
||||
|
||||
| Column | Meaning |
|
||||
| --- | --- |
|
||||
| `artifact_sha256` | SHA-256 of the exact file submitted or measured. |
|
||||
| `pixel_sha256` | SHA-256 of decoded RGB bytes, used to catch lossless duplicates. |
|
||||
| `artifact_path` | Safe path relative to the manifest. |
|
||||
| `parent_sha256` | Exact parent artifact for a derivative, empty for an original. |
|
||||
| `group_id` | Leakage boundary shared by an original and all semantic or transformed siblings. |
|
||||
| `target_provider` | Detector target, `openai` or `google`. |
|
||||
| `source_provider` | Actual source family: `openai`, `google`, `camera`, `other_ai`, `synthetic`, or `editor`. |
|
||||
| `surface` | Product or export surface. |
|
||||
| `model_epoch` | Model family plus a dated epoch when the exact version is unavailable. |
|
||||
| `generation_session` | Groups outputs created in one provider session. |
|
||||
| `content_stratum` | Predeclared content class. |
|
||||
| `width`, `height`, `format` | Decoded geometry and file format. |
|
||||
| `transform` | `original` or the reproducible transform applied to `parent_sha256`. |
|
||||
| `split` | `discovery`, `train`, `validation`, `test`, or `temporal`. |
|
||||
| `c2pa_outcome` | C2PA result, recorded independently from the pixel signal. |
|
||||
| `synthid_outcome` | `detected`, `not_detected`, `indeterminate`, `refused`, or `not_checked`. |
|
||||
| `verified_via` | Matching provider oracle, external source evidence, or `none`. |
|
||||
| `evidence_reference` | Stable URL or local evidence-record reference required for `source-evidence`; never infer it from a directory name. |
|
||||
| `oracle_session` | Groups allowed checks made in one session. |
|
||||
| `oracle_role` | `ordinary`, `source_control`, `candidate`, or `sham`. |
|
||||
| `captured_at`, `oracle_checked_at` | Timezone-aware ISO-8601 timestamps. |
|
||||
| `notes` | Verbatim context that does not fit a structured column. |
|
||||
|
||||
`source-evidence` may establish an ordinary external negative, but never a
|
||||
positive or a same-provider negative. Provider positives and same-provider hard
|
||||
negatives require the matching provider verifier. An `indeterminate`, `refused`,
|
||||
or unchecked row may remain in `discovery`; it cannot enter a train, validation,
|
||||
test, or temporal split. Every `source-evidence` row must retain the source URL
|
||||
or evidence-record reference that establishes the claim.
|
||||
|
||||
The auditor also rejects duplicate artifact hashes, identical decoded pixels in
|
||||
different groups, derivatives without parents, cross-provider parentage,
|
||||
lineage cycles, and any group crossing split boundaries. A `not_detected`
|
||||
candidate or sham is valid only when the same provider, group, and oracle
|
||||
session contains a detected `source_control`. A session that misses its control
|
||||
cannot establish removal, even when its candidate response is negative.
|
||||
@@ -475,6 +475,51 @@ That is not a YuNet split: `face_count > 0` would over-strengthen the
|
||||
spread (0.075 - 0.06) is already absorbed by the shipped 0.09 floor.
|
||||
Images API generations remain non-carriers (C2PA only).
|
||||
|
||||
## Content-balanced engine selection check (2026-08-31)
|
||||
|
||||
The earlier tables were intentionally carrier-focused and therefore small. A
|
||||
separate discovery pass tested whether ordinary image content contradicts the
|
||||
cohort choice at the shipped floors. The tracked matrix contains 19 prompts,
|
||||
matched across OpenAI `gpt-image-1-mini` and Meta `muse-image-1.0`, spanning
|
||||
photoreal scenes, faces, action, text-like flat art, products, architecture,
|
||||
watercolor, and surreal illustration. These files were re-encoded by collection
|
||||
and are not signal carriers, so this pass measures fidelity only; removal remains
|
||||
established by the provider-oracle calibration above.
|
||||
|
||||
Qwen and Chroma ran in separate sequential H100 invocations, once per source,
|
||||
with the production provider floor and fixed seed. The comparison isolated the
|
||||
global stage because both profiles inherit the same source-based Z-Image face
|
||||
repair. Per-image differences, rather than differences between aggregate means,
|
||||
were evaluated with exact two-sided sign tests:
|
||||
|
||||
| provider | metric | Chroma wins | Qwen wins | two-sided p |
|
||||
|---|---|---:|---:|---:|
|
||||
| OpenAI | SSIM | 19 | 0 | 0.0000038 |
|
||||
| OpenAI | PSNR | 19 | 0 | 0.0000038 |
|
||||
| OpenAI | MAE | 18 | 1 | 0.000076 |
|
||||
| OpenAI | edge F1 | 15 | 4 | 0.019 |
|
||||
| OpenAI | LPIPS | 12 | 7 | 0.359 |
|
||||
| Meta | LPIPS | 1 | 18 | 0.000076 |
|
||||
| Meta | edge F1 | 1 | 18 | 0.000076 |
|
||||
| Meta | SSIM | 5 | 14 | 0.064 |
|
||||
|
||||
Meta's remaining pixel metrics disagree: Chroma wins PSNR on 16/19 and MAE on
|
||||
14/19, while losing both perceptual distance and edge preservation on 18/19 at
|
||||
its higher removal floor. This is not evidence for a genre classifier. The
|
||||
direction instead follows the already measured provenance cohort: Chroma for
|
||||
OpenAI, Qwen for Meta. On OpenAI, the seven LPIPS exceptions include unrelated
|
||||
classes (landscape, sport, low-light interior, watercolor, monochrome portrait,
|
||||
surreal art, and brutalism), while Chroma still wins SSIM and PSNR on every one;
|
||||
there is no coherent input feature that justifies overriding the cohort choice.
|
||||
|
||||
Inputs and integrity manifests are under
|
||||
`data/evaluations/engine-selection/`. Raw generated outputs remain gitignored
|
||||
under `out/engine-selection-study/`; `scripts/engine_selection_study.py`
|
||||
reproduces the run and `scripts/analyze_engine_selection_study.py` reproduces
|
||||
the paired statistics. One generation per prompt is a discovery set, not a
|
||||
license to fit thresholds. Any future content override must first predict a
|
||||
rule here and then survive unused generation indices without changing it.
|
||||
|
||||
## Oracle session notes
|
||||
|
||||
- openai.com/verify throttles anonymous sessions through Cloudflare Turnstile
|
||||
|
||||
@@ -14,6 +14,8 @@ defaults. This page focuses on choosing the right command.
|
||||
| Command or signal | Required installation |
|
||||
| --- | --- |
|
||||
| `metadata` and metadata-only `identify` | Default package |
|
||||
| `classify` | `remove-ai-watermarks[classify]` |
|
||||
| `verify-openai-synthid` | `remove-ai-watermarks[verify]`, API access, and `OPENAI_API_KEY` |
|
||||
| Visible signals in `identify` | `remove-ai-watermarks[visible]` (`pixels` is the minimal runtime) |
|
||||
| Open DWT-DCT signals in `identify` | `remove-ai-watermarks[detect]` |
|
||||
| Adobe TrustMark signals in `identify` on Python 3.11-3.12 | `remove-ai-watermarks[trustmark]` |
|
||||
@@ -62,8 +64,76 @@ Metadata only inspection:
|
||||
remove-ai-watermarks identify image.png --no-visible
|
||||
```
|
||||
|
||||
Despite the historical option name, `--no-visible` skips both visible and open
|
||||
invisible pixel detectors. Metadata inspection still runs.
|
||||
Despite the historical option name, `--no-visible` skips all pixel detectors,
|
||||
including visible marks, open DWT-DCT, and TrustMark. Metadata inspection still
|
||||
runs.
|
||||
|
||||
## Verify OpenAI SynthID from pixels
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[verify]"
|
||||
remove-ai-watermarks verify-openai-synthid image.png --acknowledge-upload
|
||||
remove-ai-watermarks verify-openai-synthid image.png --acknowledge-upload --json
|
||||
```
|
||||
|
||||
This is an explicit remote check against OpenAI's official Content Provenance
|
||||
API, not the incomplete local OpenAI carrier research model. Before upload, the
|
||||
command writes a temporary copy with AI provenance metadata removed and aborts
|
||||
unless the decoded RGBA pixels are identical to the source. It then reads only
|
||||
the API's independent `synthid` entry; a C2PA-only response cannot become a
|
||||
SynthID detection. The source is never modified.
|
||||
|
||||
The API supports PNG, JPEG, and WebP files up to 50 MiB. The command requires
|
||||
`OPENAI_API_KEY` and an organization with endpoint access. Because the sanitized
|
||||
raster is uploaded to OpenAI and the endpoint is not eligible for Zero Data
|
||||
Retention, `--acknowledge-upload` is mandatory. This command is never called by
|
||||
`identify`. `not_detected` means only that OpenAI's verifier did not recognize a
|
||||
supported watermark in this file; it is not proof of human authorship.
|
||||
|
||||
The built-in client bounds the request at 120 seconds and disables automatic
|
||||
SDK retries, so one acknowledgement cannot silently upload the media multiple
|
||||
times. A timeout, disconnect, malformed response, access failure, or rate limit
|
||||
is an error, never a negative watermark verdict. API failures expose status,
|
||||
error code, request id, `Retry-After`, and whether an explicit caller-controlled
|
||||
retry is appropriate through `OpenAIProvenanceError`; the verifier itself never
|
||||
retries an upload.
|
||||
The JSON result uses the same provider-scope, backend, pixel-preservation, and
|
||||
metadata-use audit fields as `verify_openai_synthid`.
|
||||
|
||||
The Python API enforces the same boundary with the required explicit intent
|
||||
flag `verify_openai_synthid(path, acknowledge_upload=True)`.
|
||||
|
||||
## Classify a photograph from pixels
|
||||
|
||||
This is not provenance. `identify` never starts it, including after a
|
||||
no-signal metadata scan. Install the extra first:
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[classify]"
|
||||
```
|
||||
|
||||
Then:
|
||||
|
||||
```bash
|
||||
remove-ai-watermarks classify image.png
|
||||
remove-ai-watermarks classify image.png --json
|
||||
```
|
||||
|
||||
The command runs the frozen photo detector (CLIP-L-ft ridge AND freeze MLP).
|
||||
Only a DEFINITELY result is reported as `ai`. On that result only, the 124-d
|
||||
provider head may return `openai`, `google`, `muse-image`, or `tc260`. POSSIBLY is
|
||||
`unknown`. Camera-like photographs are `human`. The call does not run cleanup
|
||||
and does not set `is_ai_generated`.
|
||||
|
||||
The contract is AI versus camera. Receipts, UI, and digital art are out of
|
||||
scope. Microsoft is not a pixel class: a DALL-E Bing image scores as `openai`,
|
||||
an Imagen Designer image as `google`. `tc260` is the China AIGC label
|
||||
standard, not one producer. Weights download on first use from
|
||||
[`wiltodelta/raiw-photo-classify`](https://huggingface.co/wiltodelta/raiw-photo-classify),
|
||||
or from `RAIW_CLASSIFY_WEIGHTS` if that directory already holds the freeze
|
||||
files.
|
||||
Guide: [photo pixel classification](photo-classify.md). Hub card:
|
||||
[photo-classify-hf/README.md](photo-classify-hf/README.md).
|
||||
|
||||
## Remove known visible marks
|
||||
|
||||
@@ -105,6 +175,18 @@ Use the strict visual gate without metadata or sibling corroboration:
|
||||
remove-ai-watermarks visible image.png --sensitivity strict -o clean.png
|
||||
```
|
||||
|
||||
Act on a mark you can see but the detector will not confirm, by naming it and
|
||||
skipping the gate:
|
||||
|
||||
```bash
|
||||
remove-ai-watermarks visible image.png --mark gemini --no-detect -o clean.png
|
||||
```
|
||||
|
||||
`--detect` is the default. `--no-detect` removes the named mark's footprint
|
||||
unconditionally, so it only makes sense together with an explicit `--mark`; on
|
||||
an image that never carried that mark it inpaints a region for nothing. Prefer
|
||||
`erase --region` when you know the geometry.
|
||||
|
||||
When no known mark is detected, the command does not write a new output. Use
|
||||
`erase` if you can identify the affected region yourself.
|
||||
|
||||
@@ -436,17 +518,18 @@ user can act on rather than after a model load.
|
||||
### Restore operator-verified text
|
||||
|
||||
`--text-manifest` enables the experimental `vae-glyphs` post-pass. It reconstructs
|
||||
the source with the Qwen VAE, erases the annotated candidate glyphs with LaMa, and
|
||||
composites only the reconstructed glyph cores through source-derived silhouettes. It
|
||||
does not run OCR or choose which strings are correct. The optional `--fidelity-anchor`
|
||||
described below additionally blends 15% of the reconstruction across the full frame.
|
||||
the source with the selected profile's VAE, erases the annotated candidate glyphs
|
||||
with LaMa, and composites only the reconstructed glyph cores through source-derived
|
||||
silhouettes. It does not run OCR or choose which strings are correct. The optional
|
||||
Qwen-only `--fidelity-anchor` described below additionally blends 15% of the
|
||||
reconstruction across the full frame.
|
||||
|
||||
Install the combined extra and run only with an operator-verified manifest:
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[text-restoration]"
|
||||
remove-ai-watermarks invisible image.png -o clean.png \
|
||||
--pipeline qwen-zimage --text-manifest verified-lines.json --force
|
||||
--pipeline auto --text-manifest verified-lines.json --force
|
||||
```
|
||||
|
||||
``verified: true`` may also be set by an automated operator that attests
|
||||
@@ -454,7 +537,8 @@ machine-verified geometry: stability-gated detector boxes inside sane caps. Such
|
||||
operators should use the geometry-only schema 2, which carries no transcription
|
||||
or script metadata.
|
||||
|
||||
Since 0.27.1 the global 15% Qwen-VAE fidelity-anchor blend is off by default: it
|
||||
Since 0.27.1 the global 15% Qwen-VAE fidelity-anchor blend is off by default and
|
||||
remains available only with `qwen-zimage`: it
|
||||
was measured to return detector-visible OpenAI SynthID on poster-scale manifests
|
||||
(official Content Provenance API, 2026-08-19). `--fidelity-anchor` restores the
|
||||
0.27.0 research behavior; text-box fidelity lost by the default is well under one
|
||||
@@ -471,8 +555,9 @@ container changes remain valid while a resized or edited source fails closed. Th
|
||||
experimental helper
|
||||
`remove_ai_watermarks._internal.text_restoration.source_pixel_sha256` computes it.
|
||||
|
||||
This mode is supported only by `qwen-zimage` at native geometry with
|
||||
`humanize=0`, `unsharp=0`, and adaptive polish disabled. `all` also accepts the flag,
|
||||
This mode is supported by `qwen-zimage`, `chroma-zimage`, and `auto` at native
|
||||
geometry with `humanize=0`, `unsharp=0`, and adaptive polish disabled. The legacy
|
||||
`sdxl-zimage` profile does not expose a verified-text VAE donor. `all` also accepts the flag,
|
||||
but its manifest must match the pixels entering the invisible stage; if visible-mark
|
||||
removal changes those pixels, the hash check rejects the run. One oracle verdict does
|
||||
not certify another manifest, seed, model/runtime version, or output hash.
|
||||
@@ -502,7 +587,33 @@ remove-ai-watermarks invisible image.png -o clean.png \
|
||||
```
|
||||
|
||||
Tiling avoids the explicit downscale but each tile is regenerated separately.
|
||||
It is a memory strategy, not a guarantee of better quality.
|
||||
It is a memory strategy, not a guarantee of better quality. `--tile-size` and
|
||||
`--tile-overlap` set the tile geometry (defaults 1024 and 128 px); a larger
|
||||
overlap costs time and hides seams.
|
||||
|
||||
### Every tuning option
|
||||
|
||||
These apply to `invisible`, `all`, and `batch`. Each has a measured default, so
|
||||
reach for one only when you know why the default is wrong for your file.
|
||||
|
||||
| Option | Effect |
|
||||
| --- | --- |
|
||||
| `--strength` | Denoising strength, 0.0-1.0. Default is profile- and vendor-adaptive; see [choose a strength cohort](#choose-a-strength-cohort). Raising it removes more and drifts further from the original. |
|
||||
| `--vendor` | Strength cohort, and an assertion that the watermark is present. |
|
||||
| `--pipeline` | Profile: `qwen-zimage` (default), `sdxl-zimage`, `chroma-zimage`, `auto`. |
|
||||
| `--seed` | Fixed at 0 by default: every profile is certified at that seed, and removal near the floor is seed-dependent. |
|
||||
| `--controlnet-scale` | Canny ControlNet conditioning on the global stage. Higher keeps the original structure and text closer. |
|
||||
| `--humanize` | Analog film grain, 0 = off, typical 2.0-6.0. |
|
||||
| `--unsharp` | Unsharp-mask sharpening, 0 = off. |
|
||||
| `--adaptive-polish` / `--no-adaptive-polish` | Overrides the profile's own choice. Unset lets the profile decide: off for `qwen-zimage`, whose output already matches the input's detail level, on for `sdxl-zimage`. |
|
||||
| `--max-resolution` | Cap the long side before diffusion; 0 = native. |
|
||||
| `--tile`, `--tile-size`, `--tile-overlap` | Tiled regeneration for large inputs, as above. |
|
||||
| `--cpu-offload` | Stream model components between CPU and GPU to lower CUDA memory pressure, at the cost of speed. |
|
||||
| `--hf-token` | Hugging Face token for the model download; also read from the environment and a local `.env`. |
|
||||
|
||||
There is no `--model`, `--steps`, `--guidance-scale` or `--device` option; the
|
||||
profile pins all four. See [exit behavior](#exit-behavior) for what each command
|
||||
returns.
|
||||
|
||||
## Run the full pipeline
|
||||
|
||||
|
||||
@@ -79,8 +79,9 @@ enough yet barely touches flat fills. So the survivors FLIP by content type —
|
||||
choice alone does not guarantee removal.
|
||||
|
||||
**2. Seed non-determinism near threshold.** img2img uses a random seed unless `--seed`
|
||||
is passed, and there is no local SynthID detector to self-verify. The bracelet survived
|
||||
controlnet @0.15 in one run and CLEARED @0.15 in another (same pipeline+strength+res).
|
||||
is passed, and these geometries are outside the current local detector's scope.
|
||||
The bracelet survived controlnet @0.15 in one run and CLEARED @0.15 in another
|
||||
(same pipeline+strength+res).
|
||||
So a single clean run does NOT establish a strength as safe — characterizing a reliable
|
||||
floor needs a seed-repeatability sweep (N runs, varied seed), not one pass.
|
||||
|
||||
@@ -144,8 +145,8 @@ Gemini app; the two payloads are vendor-specific and never cross-checked):
|
||||
but never recovered original identity precisely — every setting traded one problem
|
||||
for another. See `docs/synthid-robust-identity-research-2026-06-08.md`
|
||||
"Empirical follow-up" for the full sweep.
|
||||
- **No local SynthID detector exists** → the service can't self-verify; bake in strength
|
||||
margin and periodic oracle spot-checks.
|
||||
- **No applicable local detector exists for these geometries** → the service
|
||||
can't self-verify; bake in strength margin and periodic oracle spot-checks.
|
||||
- **Lesson:** visual-quality / face-identity recovery does NOT prove removal — only the
|
||||
oracle does, across MULTIPLE content types; never conclude from a partial result (the
|
||||
photoreal-only data first read as "controlnet shields, default removes"; the flat-graphic
|
||||
@@ -298,8 +299,9 @@ attention-slicing; ~1-2 min/image, so a coarse sweep is a sub-hour background ru
|
||||
is needed ONLY for the separate
|
||||
native-large-Gemini (2816 px) case, which OOMs even without a ControlNet (that requires a
|
||||
GPU task). The genuine external dependency is NOT compute but the **manual SynthID oracle**:
|
||||
there is no local SynthID detector, so removal is verified by hand in the Gemini app
|
||||
("Verify with SynthID") per image, regardless of where the diffusion runs.
|
||||
these geometries are outside the current local detector's scope, so removal is
|
||||
verified by hand in the Gemini app ("Verify with SynthID") per image, regardless
|
||||
of where the diffusion runs.
|
||||
|
||||
Runner: **`scripts/controlnet_sweep.py`** (built 2026-06-02) implements exactly this sweep —
|
||||
SDXL base 1.0 + an SDXL-native ControlNet img2img, one output per (control x strength x scale)
|
||||
@@ -441,8 +443,9 @@ shielding risk; defer to a v2 after the single-canny path is dialed in.
|
||||
|
||||
**Hard caveat:** every change that increases preservation (higher scale, denser canny, fuller window,
|
||||
softer edges) marginally REDUCES effective regeneration and so raises the chance the watermark
|
||||
survives -- exactly the shielding failure mode. There is no local SynthID detector, so each tuning
|
||||
change must be re-confirmed on the oracle. These are img2img-context recommendations derived from
|
||||
survives -- exactly the shielding failure mode. These geometries are outside the
|
||||
current local detector's scope, so each tuning change must be re-confirmed on
|
||||
the oracle. These are img2img-context recommendations derived from
|
||||
generation-context sources plus our own measurements; treat the playbook as hypotheses to verify, not
|
||||
settled defaults.
|
||||
|
||||
|
||||
@@ -12,6 +12,8 @@ to run the tool. Use the maintainer references only when changing the code.
|
||||
| [Python API](python-api.md) | You want to call the package from Python. |
|
||||
| [Supported signals](supported-signals.md) | You need to know which visible marks, metadata formats, and invisible signals are covered. |
|
||||
| [Known limitations](known-limitations.md) | You need the quality, device, format, or verification boundaries. |
|
||||
| [Photo pixel classification](photo-classify.md) | You want `classify` / `classify_pixels`: AI versus camera, optional provider. |
|
||||
| [Photo classifier training](photo-classify-training.md) | You need the retrain pack, public sources, and what is not published. |
|
||||
| [Scope, safety, and legal notes](legal-and-safety.md) | You need the intended use and legal context. |
|
||||
|
||||
## Maintainer references
|
||||
@@ -22,9 +24,9 @@ to run the tool. Use the maintainer references only when changing the code.
|
||||
| [Development](development.md) | Environment setup, dependency recovery, CI behavior, and fixture policy. |
|
||||
| [Code provenance](code-provenance.md) | Required notices for licensed derivative work. |
|
||||
| [Verification plan](verification-plan.md) | Verification methods, completed measurements, and remaining validation gaps. |
|
||||
| [Release and distribution](release-and-distribution.md) | PyPI, Homebrew, Hugging Face Space, ComfyUI Registry, and the release workflow. |
|
||||
| [Release and distribution](release-and-distribution.md) | PyPI, Homebrew, Hugging Face Space, photo-classify Hub model, ComfyUI Registry, and the release workflow. |
|
||||
| [Watermarking landscape](watermarking-landscape.md) | Vendor signals and detection approaches. |
|
||||
| [SynthID technical reference](synthid.md) | Mechanism, detector access, robustness, and implications for this project. |
|
||||
| [SynthID technical reference](synthid.md) | Mechanism, provenance, robustness, regeneration. |
|
||||
|
||||
## Research archive
|
||||
|
||||
@@ -35,8 +37,15 @@ The current behavior is defined by the code, tests, README, and user guides.
|
||||
- [ControlNet removal research](controlnet-removal-pipeline-research.md)
|
||||
- [Qwen improvement research](qwen-improvement-research.md)
|
||||
- [Doubao reverse-alpha research](research-doubao-distillation.md)
|
||||
- [AI-generated image classifiers](ai-generated-image-classifiers.md) (photo AI-versus-camera freeze consumed by `classify_pixels`, not `identify`)
|
||||
- [Photo classifier Hugging Face card](photo-classify-hf/README.md) (Hub publication text of the 2026-08-31 freeze)
|
||||
- [SynthID source classifiers](synthid-classifiers.md) (metadata-free OpenAI/Gemini source finding and provider-lineage experiments)
|
||||
- [SynthID local detector research](synthid-detector-research.md)
|
||||
- [SynthID mark removal research](synthid-removal-research.md)
|
||||
- [SynthID identity research](synthid-robust-identity-research.md)
|
||||
- [SynthID identity follow-up](synthid-robust-identity-research-2026-06-08.md)
|
||||
- [Video SynthID quality research](video-synthid-quality-research.md)
|
||||
- [OpenAI SynthID oracle ladders](synthid-oracle-ladders.md) (routing hub)
|
||||
- [SynthID detector and removal plan](synthid-detector-removal-plan.md) (chronological mixed archive)
|
||||
- [Text protection research](text-protection-research.md)
|
||||
- [Chroma1 engine research](chroma1-engine-research.md)
|
||||
@@ -106,17 +106,19 @@ application actually uses:
|
||||
|
||||
| Extra | Capability | Automatically includes | Torch or model download |
|
||||
| --- | --- | --- | --- |
|
||||
| `pixels` | Shared BGR array and image-processing runtime | NumPy, headless OpenCV | No |
|
||||
| `pixels` | Shared BGR runtime and calibrated-size SynthID carrier detection | NumPy, headless OpenCV | No |
|
||||
| `heif` | HEIC, HEIF, and AVIF pixel decoding | pillow-heif | No |
|
||||
| `visible` | Visible mark detection, OpenCV inpainting, and manual erasing | `pixels` | No |
|
||||
| `video` | Visible video identification/removal and timestamp preservation | `visible`, PyAV | No |
|
||||
| `detect` | Open DWT-DCT detection for Stable Diffusion, SDXL, and FLUX | `pixels`, PyWavelets | No |
|
||||
| `trustmark` | Adobe TrustMark detection on Python 3.11-3.12 | trustmark | Yes |
|
||||
| `classify` | Metadata-free photo AI-versus-camera classifier plus gated provider | `pixels`, Torch, Transformers | Yes |
|
||||
| `verify` | Official remote OpenAI SynthID verification | OpenAI SDK | No |
|
||||
| `diffusion` | Torch and Diffusers runtime; video SynthID regeneration | `pixels`, Torch, Diffusers | Yes |
|
||||
| `migan` | MI-GAN ONNX fill backend | `visible`, ONNX Runtime | Model download, no Torch |
|
||||
| `lama` | big-LaMa ONNX fill backend | `visible`, ONNX Runtime | Model download, no Torch |
|
||||
| `qwen-zimage` | Invisible image-watermark removal, all CUDA-only profiles | `diffusion`, DiffSynth | Yes |
|
||||
| `text-restoration` | Opt-in verified Qwen-VAE glyph restoration | `qwen-zimage`, `lama` | Yes |
|
||||
| `text-restoration` | Opt-in verified profile-VAE glyph restoration | `qwen-zimage`, `lama` | Yes |
|
||||
| `text-draft` | Draft OCR proposals for operator verification | PaddleOCR, PaddlePaddle | Model download, no Torch |
|
||||
| `all` | Every production feature available on the active Python | All compatible rows above | Yes |
|
||||
| `dev` | Tests, linting, typing, and upstream parity checks | `video`, `detect`, upstream invisible-watermark | Yes, for parity tests |
|
||||
@@ -137,9 +139,11 @@ flowchart LR
|
||||
draft["text-draft"]
|
||||
heif
|
||||
trustmark
|
||||
classify --> pixels
|
||||
verify
|
||||
```
|
||||
|
||||
`heif`, `trustmark`, and `text-draft` are independent branches. Combine them
|
||||
`heif`, `trustmark`, `verify`, and `text-draft` are independent branches. Combine them
|
||||
explicitly with another feature when required. `text-draft` is excluded from
|
||||
`all` because it proposes unverified OCR annotations and is not a production
|
||||
removal path. TrustMark requires NumPy 1.x, which has no
|
||||
@@ -162,6 +166,9 @@ uv tool install --force "remove-ai-watermarks[video]"
|
||||
# DWT-DCT and TrustMark detection without diffusion removal
|
||||
uv tool install --force "remove-ai-watermarks[detect,trustmark]"
|
||||
|
||||
# Official OpenAI SynthID verification
|
||||
uv tool install --force "remove-ai-watermarks[verify]"
|
||||
|
||||
# Every production capability compatible with this Python
|
||||
uv tool install --force "remove-ai-watermarks[all]"
|
||||
|
||||
@@ -174,6 +181,13 @@ do not install libheif. `detect` uses the in-tree torch-free decoder and does
|
||||
not install the upstream `invisible-watermark` package. Optional models download
|
||||
their weights on first use.
|
||||
|
||||
The `verify` extra makes an explicit remote request. The
|
||||
`verify-openai-synthid` command first removes AI provenance metadata from a
|
||||
temporary copy, checks that its decoded pixels are unchanged, and then uploads
|
||||
that copy to OpenAI. It needs `OPENAI_API_KEY`; the command never runs from
|
||||
`identify` and refuses to upload without `--acknowledge-upload`. The Python API
|
||||
requires the equivalent explicit `acknowledge_upload=True` argument.
|
||||
|
||||
The old `gpu` and `remove` aliases are intentionally not provided. Use
|
||||
`diffusion` and `visible` respectively.
|
||||
|
||||
|
||||
@@ -4,6 +4,18 @@ This page describes current product limits. Historical measurements and
|
||||
superseded experiments live in the research archive listed in
|
||||
[the documentation index](index.md).
|
||||
|
||||
## Pixel classification
|
||||
|
||||
`classify` / `classify_pixels` is AI versus camera, not a universal AI detector.
|
||||
Receipts, UI screenshots, and digital art are out of contract: a thermal receipt
|
||||
can score as AI, a photorealistic AI receipt as human. POSSIBLY is reported as
|
||||
`unknown`. Provider attribution runs only after DEFINITELY and can abstain when
|
||||
124-d extraction refuses the file. Microsoft is not a pixel class.
|
||||
|
||||
The command is never started by `identify`. A no-signal provenance result stays
|
||||
unknown. This is not a clean verdict and it does not run cleanup. Full guide:
|
||||
[photo pixel classification](photo-classify.md).
|
||||
|
||||
## Visible removal
|
||||
|
||||
### Fill quality depends on the background
|
||||
@@ -99,12 +111,14 @@ rates are kept in
|
||||
[`text-protection-research.md`](text-protection-research.md) and the
|
||||
[`fidelity` evaluation record](../data/evaluations/fidelity/README.md).
|
||||
|
||||
### Removal cannot be verified locally for proprietary SynthID
|
||||
### There is no local SynthID pixel detector in the package
|
||||
|
||||
The project has no public local SynthID pixel decoder. It recognizes presence
|
||||
from supported provenance metadata: Google AI C2PA under Google's all-media
|
||||
policy, or current OpenAI C2PA with an explicit watermark action. A missing
|
||||
provenance signal is not a negative pixel verdict.
|
||||
Google does not publish the proprietary SynthID payload decoder, and the
|
||||
package does not ship one. Signed provenance is the supported route:
|
||||
Google AI C2PA or current OpenAI C2PA with an explicit watermark action.
|
||||
`verify-openai-synthid` is the official remote pixel check for OpenAI.
|
||||
Research on a periodic lattice expert is in
|
||||
[synthid-detector-research.md](synthid-detector-research.md).
|
||||
|
||||
For important outputs:
|
||||
|
||||
@@ -260,8 +274,9 @@ per-stage schedule, CFG 1.0 and CUDA.
|
||||
### Small images are processed at their native size
|
||||
|
||||
There is no minimum-resolution floor. It existed to enlarge small inputs toward
|
||||
SDXL's ~1024 training resolution and was removed with the SDXL profiles, which
|
||||
never applied it anyway. Both surviving profiles run at native geometry, so a
|
||||
SDXL's ~1024 training resolution and was removed with the standalone SDXL and
|
||||
ControlNet profiles, which never applied it anyway. The surviving `sdxl-zimage`
|
||||
is a different profile and is unaffected. All three surviving profiles run at native geometry, so a
|
||||
small input is neither enlarged before diffusion nor restored afterward.
|
||||
|
||||
`--max-resolution` still caps very large inputs, and only ever scales down.
|
||||
@@ -445,9 +460,46 @@ with plain `torch.load`. Bump lightning and cut a patch release when a fix
|
||||
ships. The calibration history is in
|
||||
[module internals](module-internals.md#metadata-and-provenance).
|
||||
|
||||
External AI versus real image classifiers are out of scope. The project
|
||||
identifies concrete local provenance signals instead of shipping a generic
|
||||
statistical classifier.
|
||||
A generic metadata-free AI-generated-image classifier is not shipped. It is a
|
||||
separate open research task from provenance detection: the current Model 1
|
||||
result is AI-versus-camera and still confuses some conventional graphics, CGI,
|
||||
product cutouts, and scans with generation. The OpenAI/Gemini source finder is
|
||||
narrower again and cannot substitute for the general classifier. The shipped
|
||||
product identifies concrete local provenance signals instead of presenting
|
||||
either research classifier as a supported verdict.
|
||||
|
||||
Frozen transfers of Community Forensics, SPAI, SAFE, RINE, Nonescape Mini,
|
||||
Dual Data Alignment, PGC, and DGS-Net did not close this gap at the required low
|
||||
false-positive rate. DDA supplies a complementary representation but its
|
||||
independent errors make simple hybrids worse. PGC SD1.4 is strong on OpenAI,
|
||||
but its published output and the useful global/residual ablations misclassify
|
||||
most or all of the Kodak scan set; removing that branch removes the OpenAI
|
||||
gain. DGS-Net is weak at the same operating point and also adds independent
|
||||
photo errors. A source-disjoint public-domain museum audit confirmed a separate
|
||||
human-art failure: Model 1 called 355/1,050 drawings, prints, manuscripts,
|
||||
paintings, textiles, and studio-object or historical-photo rows AI-generated.
|
||||
Its error was 0/150 on the historical-photo control, so ordinary camera accuracy
|
||||
does not bound accuracy on other human-created visual media. A domain veto
|
||||
removed the museum errors in development but failed the existing AI-test and
|
||||
fresh-photo core. A second source-disjoint modern-negative audit measured the
|
||||
commercial cells: across 421 date-clean Unsplash and CC-licensed Flickr images,
|
||||
Model 1 accepted 16.4% as AI, concentrated in logo/graphic design (38.3%),
|
||||
retouched fashion photography (34.1% on the Unsplash cell; 2/40 on the Flickr
|
||||
replication), and product cutouts (16.0%), while composites
|
||||
stayed clean at 0/50 and long exposure near clean at 11/146 after a
|
||||
two-part contamination control (a 0/450 provenance-metadata scan and a
|
||||
pre-2022-08 date bound that dropped 29 rows, correcting one contaminated
|
||||
Flickr fashion cell). The failure pattern therefore spans both historical art
|
||||
and modern stylized graphics, not photographic exposure technique. A
|
||||
source-disjoint linear veto fit on the date-clean modern cells was then
|
||||
rejected one gate earlier: no searched operating point repaired even one
|
||||
modern dev error without dropping below the frozen EvalGEN and FLUX floors,
|
||||
because modern human graphics and AI generations overlap almost one-to-one in
|
||||
the frozen CLIP embedding, at roughly 1.3 to 1.5 AI positives lost per negative
|
||||
repaired. Model 1 also lacks a time/device-disjoint negative contract
|
||||
for modern computational photography. The measured public protocol, GitHub
|
||||
survey, and rejected fusions are recorded in
|
||||
[AI-generated image classifier research](ai-generated-image-classifiers.md#general-ai-classifier-github-sweep-2026-08-25).
|
||||
|
||||
## Output and traceability
|
||||
|
||||
|
||||
@@ -11,7 +11,8 @@ updated in the same change.
|
||||
|
||||
## Architecture
|
||||
|
||||
The package has four main paths:
|
||||
The package has four main paths, plus an opt-in pixel classifier that is not
|
||||
on the provenance or removal graph:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
@@ -19,13 +20,18 @@ flowchart LR
|
||||
Input --> Visible[Visible mark removal]
|
||||
Input --> Invisible[Diffusion regeneration]
|
||||
Input --> Metadata[Metadata stripping]
|
||||
Input --> Classify[classify_pixels]
|
||||
|
||||
Identify --> Report[ProvenanceReport]
|
||||
Visible --> VisibleOutput[Localized and filled image]
|
||||
Invisible --> InvisibleOutput[Regenerated image]
|
||||
Metadata --> MetadataOutput[Container with AI metadata removed]
|
||||
Classify --> PixelClassification
|
||||
```
|
||||
|
||||
`identify` does not call `classify_pixels`. `has_invisible_target` and `all`
|
||||
do not read it.
|
||||
|
||||
The `all` command runs visible removal, optional invisible regeneration, and
|
||||
metadata stripping in that order.
|
||||
|
||||
@@ -533,7 +539,382 @@ metadata extraction from verdict logic:
|
||||
both extractors reach the same answer. It did not, and the record path silently
|
||||
reported no SynthID for images the file path flagged.
|
||||
- `identify` preserves the path-based API and adds the optional registered
|
||||
visible-mark and open invisible-watermark decoders after extraction.
|
||||
visible-mark and open invisible-watermark detectors after extraction.
|
||||
|
||||
### SynthID periodic carrier detector (research only)
|
||||
|
||||
The local lattice expert is not part of the public package. Runtime code lives
|
||||
in [`scripts/synthid_runtime/`](../scripts/synthid_runtime/) and the campaign
|
||||
log is [`synthid-detector-research.md`](synthid-detector-research.md). The
|
||||
notes below are the calibration history of that research expert.
|
||||
|
||||
[`synthid_detector.py`](../scripts/synthid_runtime/synthid_detector.py) is the
|
||||
runtime form of the frozen 2048x2048 periodic-tile experiment. It folds a
|
||||
Gaussian high-pass residual modulo 16x16 within a calibrated pixel-count range
|
||||
and compares the normalized RGB tile with the bundled float64 template
|
||||
`scripts/synthid_runtime/synthid_periodic_tile_2048_v1.npz`. Exact multiples use the original
|
||||
reshape-and-mean path; other sizes use count-correct modulo folding,
|
||||
without resize. Channels are filtered and folded sequentially, and partial edge
|
||||
blocks are accumulated without a full-frame padding buffer so the 18-megapixel
|
||||
ceiling does not require multiple three-channel float workspaces. The model hash
|
||||
is pinned by a test, and the unchanged operating threshold is
|
||||
`0.17357069773071196` through 10 megapixels.
|
||||
|
||||
That fixed threshold is not production-qualified. A later source-fresh
|
||||
Open Images test-split challenge produced 5 crossings among the 211 images in
|
||||
its supported geometry. A precision-first `0.28` replacement is frozen as a
|
||||
research candidate after retaining all 12 available source-diverse native
|
||||
positives and rejecting those five crossings. It then failed the second
|
||||
untouched holdout at 1/213, with score `0.322542963`. The replacement is
|
||||
rejected and fixed-v2 remains an explicit diagnostic only.
|
||||
|
||||
The direct API returns `detected`, `indeterminate`, or `unsupported`. Passing
|
||||
`register_scale=False` selects fixed-v2 from 1,000,000 through 10,000,000
|
||||
decoded pixels as an explicit diagnostic. Its frozen threshold accepted
|
||||
none of 5,000 public COCO views balanced across every observed target geometry,
|
||||
and none of a separate 5,000-view challenge over 256 generated geometries
|
||||
covering every pair of modulo-16 edge remainders. The original 2048x2048
|
||||
verdicts and exact scores remain unchanged. Runtime matches do not attribute a
|
||||
provider. `identify` adds only positive matches as high-confidence
|
||||
evidence and never turns a local negative into a clean verdict.
|
||||
The result envelope also names the signal family, provider scope, backend,
|
||||
whether metadata contributed to the verdict, whether pixels were preserved,
|
||||
and an explicit reason for unsupported or indeterminate results. These fields
|
||||
are shared with the official OpenAI verifier's JSON boundary.
|
||||
|
||||
The production router selects `synthid-periodic-tile-large-v1` above 10 through 18
|
||||
megapixels when both dimensions are at least 2,048 pixels. It evaluates all
|
||||
phase-aligned 2,048-square windows and combines the minimum fixed-template,
|
||||
Red-minus-Green, and Blue-minus-Yellow spatial correlations with the most
|
||||
negative Blue-minus-Yellow mid-band correlation. The 3072x5504 portrait
|
||||
geometry also applies a Green mid-band alias veto. Each component is normalized
|
||||
to its frozen gate and the public threshold is `1.0`.
|
||||
|
||||
All 37 inferred large candidates cross the rule, and all seven metadata-free,
|
||||
pixel-identical candidates checked by the official Gemini verifier were
|
||||
detected. The constants rejected all 17,417 exposed external controls. A
|
||||
post-freeze production-path challenge then rejected all 2,637 decoded-pixel-
|
||||
unique controls drawn from 2,000 COCO images excluded from the earlier large
|
||||
color-phase challenge and 637 deduplicated Picsum controls. Four large
|
||||
geometries and four resampling kernels were balanced; the maximum score was
|
||||
`0.0592777965`. The source collections were not freshly acquired, so this is a
|
||||
feature-unseen holdout rather than a fresh-source estimate.
|
||||
|
||||
A separate post-freeze Open Images download yielded 41 completed,
|
||||
decoded-pixel-unique controls after excluding incomplete `.aria2` files and all
|
||||
prior Open Images hashes. The frozen production path accepted 0/41 and reached
|
||||
a maximum score of `0.4083013324`. This source-fresh audit is too small to
|
||||
replace the main holdout interval but checks the acquisition boundary.
|
||||
|
||||
The same seven official positives were then re-encoded at unchanged dimensions.
|
||||
JPEG-95 and JPEG-90 each reduced detection from 7/7 native files to 0/7. The
|
||||
large operating point is therefore native-pixel and lossless-copy support, not
|
||||
a codec-robust claim.
|
||||
|
||||
Arbitrary geometry is not the same as arbitrary spatial resampling. On a
|
||||
stratified 80-image fixed-positive sample, one-step resizes at seven nonidentity
|
||||
scales from 0.5 through 1.5 reduced the unchanged 16x16 detector from 80 accepted
|
||||
sources to zero at every scale. Restoring the original geometry recovered 58-80
|
||||
sources, showing that the carrier period scaled with the pixels. A discovery
|
||||
bank that scaled the template to integer periods 8, 10, 12, 14, 18, 20, and 24
|
||||
was promising: a threshold frozen above 3,000 resized COCO controls accepted no
|
||||
view in a 2,000-control final partition and accepted 672 of 800 source-disjoint
|
||||
provider positives. It also stayed below threshold on the tracked OpenAI and
|
||||
Adobe controls. The branch remains research-only because noninteger periods at
|
||||
scales 0.8, 0.9, 1.1, 1.2, and 1.333 collapsed, while separate per-period
|
||||
thresholds accepted five final controls. The runtime therefore keeps only the
|
||||
fixed 16-pixel lattice.
|
||||
|
||||
A follow-up fractional-period probe sampled the 30 strongest template harmonics
|
||||
over a continuous 7.5-24.5 period range. The correct period appeared within
|
||||
0.05 pixels among the top three candidates for 58 of 60 transformed positives.
|
||||
Testing nine neighboring reconstructed geometries then recovered 44 of 60 at
|
||||
the native threshold, compared with an upper bound of 48 when the true source
|
||||
geometry was supplied. The complete search still failed its small frozen
|
||||
control split: a threshold above 250 development controls accepted two of 150
|
||||
final controls. Multiplying the canonical score by spectral-period confidence
|
||||
also accepted two. That baseline was rejected rather than shipped at its
|
||||
discovery threshold.
|
||||
|
||||
The research-only
|
||||
[`synthid_affine_lattice_probe.py`](../scripts/synthid_affine_lattice_probe.py)
|
||||
adds split-confirm synchronization. It estimates complex harmonic coherence on
|
||||
one checkerboard of patches, confirms the selected period on the other, and
|
||||
reports amplitude-aware confirmation, a locally content-whitened multichannel
|
||||
code match, and phase-preserving and cyclically registered template scores. A
|
||||
0.1-pixel grid recovered the expected
|
||||
12.8, 14.4, 17.6, and 19.2 periods in 20/20 transformed views from five official
|
||||
positive parents. A provisional conjunction accepted none of 800 corresponding
|
||||
views from 200 oracle-negative parents, none of 469 OpenAI-labeled rows, and
|
||||
three of 276 broad non-Google rows whose TC260 or Samsung provenance prevents
|
||||
treating them as clean oracle negatives. This probe is not runtime routing:
|
||||
positive diversity is still inadequate, period 8 remains rejected, and the
|
||||
amplitude stage currently handles isotropic scale at zero rotation only. Full
|
||||
protocol and caveats are in the detector research plan.
|
||||
|
||||
The whitened match uses neighboring noncarrier bins to estimate complex
|
||||
Green/opponent-color covariance around every selected harmonic. It corrected a
|
||||
period-24 alias on one native official positive, giving the broad native search
|
||||
the correct period 16 on all five ordinary-size positives. It was retained only
|
||||
as a candidate reranker: its positive-to-negative margin was smaller than the
|
||||
existing spatial-template margin in both the native pilot and a locked 0.8
|
||||
resize challenge. Two of the three TC260/Samsung-provenance challenge crossings
|
||||
also matched the full whitened code more strongly than the weakest official
|
||||
large positive, so the pixel result identifies a compatible signal family, not
|
||||
a provider.
|
||||
|
||||
The same probe also exposes a payload-agnostic H5 confirmation. It estimates a
|
||||
complex harmonic vector independently on the two checkerboards and measures
|
||||
their fixed relative-phase inner product; the codeword need not match the known
|
||||
template. Free cyclic-shift selection was rejected because natural phase aliases
|
||||
overlapped the controls. The fixed all-harmonic statistic separated five native
|
||||
official positives from 130 controls with margin `0.1619`, and five 0.8-resized
|
||||
views from 200 fresh-parent controls with margin `0.1541`; all seven large
|
||||
official positives also passed the observed gap. It remains research-only
|
||||
because these tests supplied the period, reuse one negative source family, and
|
||||
contain only 12 independent positive parents.
|
||||
|
||||
The separate research-only
|
||||
[`synthid_cyclostationary_probe.py`](../scripts/synthid_cyclostationary_probe.py)
|
||||
measures full complex cross-channel spectral correlation at carrier shifts
|
||||
against neighboring-shift same-image nulls. It recognizes synthetic amplitude
|
||||
modulation and rejects both independent noise and a purely additive carrier.
|
||||
The real native pilot overlapped completely: five official-positive joint
|
||||
contrasts ranged from `-0.0027` to `0.0337`, while 130 controls reached `0.0593`.
|
||||
The cyclostationary branch is rejected rather than tuned on new frequency bands.
|
||||
|
||||
The affine-lattice research CLI also supports symmetric in-memory JPEG and
|
||||
top-left crop challenges. At fixed native period 16, codec-specific
|
||||
phase-preserving template thresholds retained all five repeated positive views
|
||||
and accepted none of 80 locked controls at JPEG qualities 95, 90, and 80. The
|
||||
locked margins declined from `0.1054` to `0.0257`; whitened and unknown-codeword
|
||||
scores were rejected for lossy JPEG. A separate registered crop conjunction
|
||||
retained all five views and accepted none of 80 controls after both 5% and 10%
|
||||
top-left crops, with registered-template margins above `0.22`. These are
|
||||
research pilots, not runtime experts: the period was supplied, positive parents
|
||||
were reused, and only one codec implementation and one negative family were
|
||||
tested.
|
||||
|
||||
The `register_scale=True` path is the calibrated ordinary-size production
|
||||
expert. It
|
||||
keeps the harmonic search, adds three-level scale-space consistency and quadrant
|
||||
agreement. Its calibrated decision requires the canonical winner to be the
|
||||
strongest spectral-period candidate, its three-way amplitude to cross the
|
||||
threshold for that period bin, and both independent high-frequency template
|
||||
bands to correlate by at least `0.075`. The public registered score is the
|
||||
minimum normalized margin across those gates, so its decision threshold is
|
||||
`1.0`.
|
||||
|
||||
The earlier single-threshold version produced 68 false positives among 11,273
|
||||
symmetrically downscaled Spaces controls. A top-candidate plus period-bin version
|
||||
removed those but still produced 6 false positives among 1,000 source-independent
|
||||
Open Images controls. The high-band gate removed them and then accepted none of
|
||||
499 previously untouched Open Images reserve controls. The resulting rule
|
||||
retained 229 of 355 source-disjoint transformed positives: 0/65 at scale 0.5 and
|
||||
229/290 from scale 0.65 through 1.5. The explicit period-8 rescue is rejected
|
||||
because resize lattices fully overlap its positive distribution.
|
||||
|
||||
A subsequently frozen 1,000-image Open Images reserve accepted zero in
|
||||
registered mode. The fixed expert supported only 81 of those geometries and
|
||||
accepted seven, so fixed and registered results cannot safely be unioned. In
|
||||
overlapping geometry the registered decision remains the validated path;
|
||||
fixed-only evidence is a diagnostic rather than a universal-cascade positive.
|
||||
|
||||
A later source-fresh 3,000-image Open Images test-split challenge superseded the
|
||||
registered-v2 precision claim: it accepted 4 of 2,997 supported controls. The
|
||||
registered-v3 confirmation uses disjoint patch groups for phase, cyclic
|
||||
amplitude, and held-out-codeword evidence. It rejected all four crossings,
|
||||
retained 147 of 148 base-accepted source-diverse positives, and retained all 359
|
||||
base-accepted views in a dense 0.65-1.50 transform matrix over 12 independent
|
||||
parents. Frozen unchanged, it then accepted 0/2,996 controls from a second
|
||||
nonoverlapping Open Images cohort and 0/2,366 supported controls from a
|
||||
3,000-image COCO second-family challenge. Registered-v3 is now the default
|
||||
ordinary-size positive route. The exact gates, acquisition hashes, and rejected
|
||||
weak-signal rescue are recorded in the detector research plan.
|
||||
|
||||
Every one of those control rates is photographic. Against 223 corpus images
|
||||
whose C2PA names a non-Google generator, the unchanged entry point accepted 29
|
||||
(`0.130`, Adobe Firefly `0.241`, highest foreign score `3.01`), all from
|
||||
registered-v3. The branch reads a lattice shared across generation pipelines,
|
||||
which is why it must not be reported as a watermark. The public `identify`
|
||||
path no longer calls this expert.
|
||||
|
||||
The branch is also phase-locked to the image origin, exactly like the large
|
||||
expert. A two-pixel diagonal crop killed all 28 in-geometry foreign detections
|
||||
and all 8 detected Google provenance positives (maximum remaining scores
|
||||
`0.779` and `0.311` against the `1.0` threshold); the signal recovers only at
|
||||
offsets that are multiples of four. Registered-v3 therefore detects the same
|
||||
crop-destroyed generation-pipeline lattice as `large-v1`, on ordinary sizes.
|
||||
|
||||
When registered-v3 abstains, opponent-registered-v1 searches the same frozen
|
||||
template in Red-minus-Green and Blue-minus-Yellow space. It reranks three
|
||||
separated scale candidates with fixed RGB and two spatial opponent-color gates,
|
||||
then accepts only periods 7.9-12.0 on 1-10 megapixel rasters whose sides are at
|
||||
least 768 pixels. Period-8 candidates additionally require Red-Green and
|
||||
Blue-Yellow 8-pixel edge ratios no greater than `1.05`; this vetoes the
|
||||
deterministic JPEG block lattice without using container metadata. The final
|
||||
rule recovered 49/49 lossless 0.5x-0.75x views from seven official-positive
|
||||
parents. It rejected all 1,790 measured period-8 codec crossings, while 350
|
||||
identically resized controls had no base crossing and the earlier period-band
|
||||
rule accepted 0/1,000 post-freeze Picsum controls. Period 12.8 remains excluded,
|
||||
and lossy JPEG/WebP views remain inconclusive.
|
||||
|
||||
The runtime precedence is registered-v3, the bounded opponent fallback, then
|
||||
large-v1 above 10 megapixels. Passing `register_scale=False` selects the legacy
|
||||
fixed diagnostic explicitly. The research bank in
|
||||
`scripts/synthid_routed_expert_bank.py` keeps all three observations for audits.
|
||||
Neither runtime nor research routing returns a clean-image verdict.
|
||||
|
||||
An unchanged registered challenge from 10 to 18 megapixels retained only 1 of
|
||||
37 Google candidates and zero of 89 non-Google controls. Twenty-eight positives
|
||||
cleared amplitude, 21 had matching spatial and spectral periods, but only three
|
||||
cleared high-band agreement. The 10-megapixel ceiling therefore remains.
|
||||
Phase-aligned 2,048-square fixed windows did not provide a fallback: median
|
||||
consensus retained 36 positives and accepted 10 controls, while all-window
|
||||
consensus retained 36 and accepted eight.
|
||||
One frozen full-frame pre-resize to eight megapixels also retained only the same
|
||||
1 of 37 positives and zero controls; just three positives cleared high-band
|
||||
agreement. Large images therefore cannot be routed through a canonical-size
|
||||
registered fallback; the later native opponent-color branch is separate.
|
||||
The remaining phase-aligned window variants closed this branch: a single center
|
||||
2,048-square registered window retained 1 of 37 positives and zero of 89
|
||||
controls, while accepting any phase-aligned 2,048-square window retained 2 of
|
||||
37 and zero controls. The latter control maximum was already 0.968 against the
|
||||
1.0 decision threshold. Neither the coverage nor the exposed specificity
|
||||
margin supports a registered-window expert; these results do not apply to the
|
||||
later native opponent-color branch.
|
||||
|
||||
A separate half-scale patch-consensus branch initially looked promising. Its
|
||||
64-pixel, 90th-percentile patch statistic retained 33 of 49 validation positives
|
||||
and zero of 166 controls, then 27 of 52 locked-test positives and zero of 140
|
||||
controls. The frozen broad challenge rejected it: 61 of 10,906 symmetrically
|
||||
half-scaled Spaces controls and 22 of 1,000 independent Open Images controls
|
||||
crossed the unchanged threshold. A post-challenge Green negative-phase spread
|
||||
gate removed those errors but retained only 2 of 234 later Google rows and zero
|
||||
of 689 later controls. Half-scale support therefore remains explicitly absent.
|
||||
|
||||
An independent reproduction of the public `aloshdenny/reverse-SynthID` V4
|
||||
cross-color codebook did not provide another expert. Its best-of-two-model phase
|
||||
rule retained 141 of 355 positives but accepted 191 of 499 controls and 386 of
|
||||
a fresh 1,000-control reserve; AUC was 0.517. The repository's stricter
|
||||
aspect-ratio routing reduced coverage but preserved the same approximately 38%
|
||||
positive and control acceptance within supported rows. Its older V3 phase score
|
||||
had AUC 0.473, and two prespecified amplitude-aware V4 rerankings reached only
|
||||
0.521 and 0.524 AUC. These external rules are research baselines only. Their useful
|
||||
contribution is the solid-reference phase-consensus construction, not either
|
||||
published single-image threshold. Full methodology and the DALL-E reference-set
|
||||
confound are recorded in the detector research plan.
|
||||
A later exact repeat on the hash-disjoint v7 challenge confirmed the same
|
||||
failure: the published cut accepted 77/200 Google and 76/200 OpenAI, so it did
|
||||
not separate the two target providers.
|
||||
A direct exact-1024 reproduction of the same V4 artifact confirmed rather than
|
||||
rescued that verdict: the union of its two published profiles accepted 177 of
|
||||
443 Google rows and 60 of 162 controls. The artifact's floor left only about two
|
||||
active bins in the inspected profile/channel, explaining why its score acts
|
||||
as a weak generic phase-coherence test rather than a specific codebook match.
|
||||
|
||||
An exact-1086x1448 OpenAI follow-up also rejected the solid-reference transfer.
|
||||
A low-texture 256-bin phase codebook achieved native and JPEG-95 test AUCs of
|
||||
0.512, and independent train halves shared only two exact bins. The paper's
|
||||
content-dependent alternative was tested with 60 OKLab masking-alignment
|
||||
features. Its native/JPEG minimum reached test AUC 0.593 and 15/562 positives
|
||||
versus 1/228 controls at the frozen boundary, but deterministic half-fits shared
|
||||
no positive decision and had coefficient cosine 0.148. Neither branch is a
|
||||
runtime OpenAI expert.
|
||||
|
||||
An exact-1254 assertion-contrast search selected an sRGB period-8 Blue-channel
|
||||
direction at 45 of 259 validation asserted rows and zero of 16 same- or
|
||||
other-provider validation rows. It failed its locked specificity check at 39 of
|
||||
233 asserted rows and one of 18 controls. Across the complete exact-geometry
|
||||
audit it accepted 185 of 992 asserted rows, one of 55 same-provider rows without
|
||||
an assertion, and one of 13 other-provider rows. Raising the threshold above
|
||||
the wider controls reduced recall to a small minority, so this branch remains
|
||||
research-only.
|
||||
|
||||
Three attempted veto or missing-codeword rescues also failed. The official
|
||||
InvisMark decoder checkpoint passed its own embedded-watermark self-test at
|
||||
0.997 confidence and 0.97 bit accuracy, but all OpenAI and provider-control
|
||||
cohorts clustered around 0.18 confidence with no useful separation. DTCWT
|
||||
modulus before periodic folding retained 5 of 158 locked-test OpenAI rows and
|
||||
zero of 135 controls, then zero of 40 fresh OpenAI rows. A 16-codeword whitened
|
||||
period-8 model retained 5 of 158 and zero controls, then 1 of 40 fresh OpenAI
|
||||
rows and 1 of 50 fresh controls. They neither explain the known oracle-positive
|
||||
miss nor justify runtime routing.
|
||||
|
||||
An image-level patch-bag model over nine normalized high-pass patches was the
|
||||
strongest later local OpenAI classifier. It accepted 324/1,364 development-test
|
||||
OpenAI rows with 4/3,373 heterogeneous controls, then transferred to 120/486
|
||||
OpenAI rows, 8/504 general controls, and 5/33 same-provider non-assertion rows
|
||||
in the later temporal challenge. Assertion enrichment over the same-provider
|
||||
stratum was not independently significant (`p = 0.151`, one-sided exact test).
|
||||
JPEG-95 reduced the unchanged conjunction to
|
||||
1/486 positives and zero controls. The native temporal hits were complementary
|
||||
to the signed period-8 hits, but the same-provider rate and codec collapse
|
||||
identify another export noiseprint rather than a runtime SynthID expert.
|
||||
|
||||
The separate OpenAI period-8 DTCWT component is persistent rather than tied to
|
||||
one short rollout: exact-generator asserted hits were 3/16, 46/365, and 26/200
|
||||
from May through July, with 75/581 overall versus 1/52 same-generator rows
|
||||
without an assertion. Its native/JPEG minimum score reached 0.721 AUC between
|
||||
those indeterminate strata. Sorting all 64 cyclic correlation scores removed
|
||||
absolute phase but also removed locked-test discrimination at 0/158 positives
|
||||
and 0/135 controls. This remains research evidence for a weak signed carrier,
|
||||
not a runtime OpenAI detector.
|
||||
|
||||
A four-family open-proxy challenge also failed to justify a generic neural
|
||||
watermark expert. A fixed residual frontend and cross-family residual mixing
|
||||
were trained on three of TrustMark P, VideoSeal, DWT-DCT, and WAM while the
|
||||
fourth encoder and its test sources remained unseen. Held-out AUCs ranged from
|
||||
0.437 to 0.562. Equal-power phase-scrambled hard negatives prevented simple
|
||||
spectral-energy shortcuts, but did not produce architecture transfer. A
|
||||
separate translation-invariant Gemini bicoherence search selected none of 20
|
||||
development positives and finished at 0/50 positives, 0/199 controls, and AUC
|
||||
0.374. Neither branch is part of runtime routing; full split and oracle details
|
||||
are in the detector research plan.
|
||||
|
||||
The separately measured registered geometry range remains 250,000 through
|
||||
10,000,000 decoded pixels with both sides at least 256 pixels. The default path
|
||||
and `identify` use registered-v3, then the narrower opponent-registered-v1
|
||||
fallback in its 1-10 megapixel domain, and large-v1 above 10 megapixels. A
|
||||
20-image real-corpus drift check was byte-identical after the earlier v2 integration.
|
||||
The calibration history and caveats are in the linked detector research plan.
|
||||
|
||||
### Official OpenAI SynthID verifier
|
||||
|
||||
[`openai_provenance.py`](../src/remove_ai_watermarks/openai_provenance.py)
|
||||
provides the explicit remote production backend exposed as
|
||||
`verify-openai-synthid`. It is intentionally separate from `identify`, because
|
||||
one invocation uploads a sanitized raster to OpenAI. The CLI requires
|
||||
`--acknowledge-upload`, and the optional OpenAI SDK lives in the independent
|
||||
`verify` extra.
|
||||
|
||||
The backend accepts only PNG, JPEG, and WebP. It computes a decoded RGBA pixel
|
||||
fingerprint, removes AI provenance metadata into a temporary file through
|
||||
`metadata.strip_and_verify`, recomputes the fingerprint, and aborts before any
|
||||
request if metadata survived, the format changed, the pixels changed, or the
|
||||
sanitized file exceeds the endpoint's 50 MiB limit. It then sends exactly one
|
||||
multipart file to `content_provenance_checks.create` and parses exactly one
|
||||
`type == "synthid"` result. The independent C2PA entry is never returned or
|
||||
used as fallback evidence. Missing, duplicate, or unknown SynthID outcomes are
|
||||
errors rather than negative detections.
|
||||
|
||||
The default SDK client has a 120-second request timeout and zero automatic
|
||||
retries. One upload acknowledgement therefore authorizes at most one media
|
||||
transmission rather than inheriting the SDK's retry default. Request logs keep
|
||||
the endpoint, temporary basename, media type, byte count, timeout, retry policy,
|
||||
duration, HTTP status, error code, and request id when available, but omit the
|
||||
source path, image bytes, credentials, and decoded-pixel fingerprint.
|
||||
|
||||
`OpenAIProvenanceError` preserves the status, API error code, request id,
|
||||
`Retry-After` value, and a transient-only `retryable` flag. The library does not
|
||||
automatically act on that flag: an explicit caller invocation is required for
|
||||
every additional upload. Transport and schema failures remain errors rather
|
||||
than becoming `not_detected` or a local detector result.
|
||||
|
||||
The result remains provider-scoped and positive-evidence-only. `not_detected`
|
||||
does not mean human-created, and the official endpoint's published prohibition
|
||||
on repeated reverse-engineering or evasion queries prevents using this backend
|
||||
as an adaptive training or removal oracle.
|
||||
|
||||
### Portable metadata record
|
||||
|
||||
@@ -576,6 +957,22 @@ provenance normalizer. Integration code publishes the strict
|
||||
`ProvenanceReport.to_dict()` alongside it rather than letting operational fields or
|
||||
derived results influence detection.
|
||||
|
||||
### Photo pixel classifier
|
||||
|
||||
[`classify.py`](../src/remove_ai_watermarks/classify.py) is the 2026-08-31
|
||||
photo freeze: CLIP-L-ft ridge AND freeze MLP, then 124-d focal heads only on
|
||||
DEFINITELY. The public label is `ai` / `human` / `unknown`. POSSIBLY is
|
||||
`unknown`. The named class is `openai` / `google` / `muse-image` / `tc260` /
|
||||
`None`. `tc260` is the China AIGC label standard, not one producer. The freeze
|
||||
file keys the Muse Image head `meta_muse_image`. Tests in
|
||||
`tests/test_classify.py` pin the gate without downloads and pin that
|
||||
`identify` does not import this module.
|
||||
|
||||
Weights stay out of git. The Hub snapshot is `wiltodelta/raiw-photo-classify`.
|
||||
`RAIW_CLASSIFY_WEIGHTS` overrides it. The extra is `classify`. User guide:
|
||||
[photo-classify.md](photo-classify.md). Hub card:
|
||||
[photo-classify-hf/README.md](photo-classify-hf/README.md).
|
||||
|
||||
### Pixel forensics
|
||||
|
||||
[`pixel_evidence.py`](../src/remove_ai_watermarks/pixel_evidence.py) measures six
|
||||
@@ -850,6 +1247,17 @@ box grows), not the whole `footprint_mask`. Baidu extends right to the corner ta
|
||||
and LiblibAI extends left to the triangle logo; both inherit every guard around
|
||||
that arithmetic.
|
||||
|
||||
Doubao is the deliberate exception because its captured alpha supplies more
|
||||
information than a rectangle. Its continuous top-hat response locates the mark,
|
||||
then `DoubaoEngine.footprint_mask` resizes the alpha to that same winning
|
||||
`match_box` and masks only the glyphs. The canonical 2048-pixel fixture has bright
|
||||
branch texture behind the bottom-edge wordmark; bounding the thresholded response,
|
||||
padding it, and dilating it produced a solid 404-by-134 mask clamped to the bottom
|
||||
and right edges. OpenCV then had no context beyond either edge and filled the hole
|
||||
with large triangular wedges. The aligned sparse mask keeps both frame edges
|
||||
untouched and still clears the detector. `force` cannot align a missing detection,
|
||||
so it retains the shared geometry-box fallback.
|
||||
|
||||
Yuanbao uses the polarity-independent `contrast` front end because its standard
|
||||
two-line mark can be light on dark scenes or dark on light scenes. Its detector
|
||||
and footprint both use the same best-match box. The separate one-line overlay
|
||||
@@ -945,9 +1353,10 @@ is the source of truth for:
|
||||
The current profiles are `qwen-zimage` (the default), `sdxl-zimage`,
|
||||
`chroma-zimage`, and `auto`, and all four are CUDA-only. `controlnet`, `sdxl`,
|
||||
`qwen` and `default` were removed rather than kept as a CPU path, and are
|
||||
rejected rather than aliased onward. `auto` is a per-cohort engine router:
|
||||
rejected rather than aliased onward. `auto` is a deterministic per-cohort
|
||||
selection policy:
|
||||
chroma-zimage for OpenAI and Microsoft, qwen-zimage for Google, Meta, and
|
||||
unknown. It is not a content-class split inside one engine.
|
||||
unknown. It does not run a learned router or classify the image's genre.
|
||||
|
||||
`qwen-zimage` normally resolves global denoise from image area for unknown content.
|
||||
Measured provider cohorts bypass that curve with flat operating points. The values,
|
||||
@@ -1105,13 +1514,14 @@ geometry alone in schema 2, plus a SHA-256 over decoded RGB width, height, and p
|
||||
Validation happens before model loading. The library never treats OCR confidence as
|
||||
verification, and geometry-only operators do not need to invent text or script fields.
|
||||
|
||||
When enabled, `QwenZImagePipeline` reconstructs the source once through its already
|
||||
loaded Qwen VAE, runs the ordinary global and face stages, and calls the shared
|
||||
restoration compositor. The optional fidelity anchor first blends 15% of the VAE
|
||||
When enabled, the one profile selected before model loading reconstructs the source
|
||||
once through its already loaded VAE, runs the ordinary global and face stages, and
|
||||
calls the shared restoration compositor. Qwen and Chroma implement this donor hook;
|
||||
SDXL does not. The optional Qwen-only fidelity anchor first blends 15% of the VAE
|
||||
reconstruction into the clean result; it is off by default because the blend returned
|
||||
detector-visible OpenAI SynthID in the measured poster fixtures. The compositor derives
|
||||
binary source and candidate silhouettes, groups nearby lines, uses LaMa for the initial
|
||||
and residual-glyph erase passes, paints fresh silhouette edges, then copies the Qwen-VAE
|
||||
and residual-glyph erase passes, paints fresh silhouette edges, then copies the profile-VAE
|
||||
core with a 0.5-pixel feather. The evaluation script
|
||||
imports these same mask and compositing helpers so the two implementations cannot
|
||||
silently drift. Silhouette crops start 12% of line height beyond each horizontal
|
||||
@@ -1127,6 +1537,21 @@ geometry or final pixels after the verified layer and have no measured oracle re
|
||||
It remains opt-in because annotations are manual and provider verdicts apply only to
|
||||
the exact tested output hashes, not to the mechanism in general.
|
||||
|
||||
The first full-path H100 check on 2026-08-31 ran each generative engine exactly
|
||||
once per source and used that SAME engine's VAE donor; it did not compose Qwen and
|
||||
Chroma in one production result. On the two verified multilingual OpenAI posters,
|
||||
restoration reduced PaddleOCR CER from 0.303 to 0.160 and 0.279 to 0.136 for Qwen,
|
||||
and from 0.213 to 0.130 and 0.204 to 0.108 for Chroma. PSNR improved on all four
|
||||
arms (Qwen 26.1 to 29.9 dB and 27.7 to 31.6 dB; Chroma 29.6 to 33.1 dB and 29.7
|
||||
to 32.9 dB). The verified CJK Google fixture is the counterexample that prevents
|
||||
an unconditional quality claim: Qwen CER stayed 0.074, while Chroma moved from
|
||||
0.037 to 0.074 despite better LPIPS, SSIM, PSNR, and text-box pixel error. The
|
||||
stage is therefore implemented and fidelity-positive on both OpenAI sources, but
|
||||
not assumed to improve every OCR reading. This is a fidelity check, not a provider
|
||||
oracle certification; exact output hashes and per-box metrics are produced by
|
||||
`scripts/chroma_text_restoration_study.py` under the gitignored
|
||||
`out/text-restoration-engine-study/` directory.
|
||||
|
||||
A matched stage-isolation check on the 18-face Gemini portrait grid confirms the
|
||||
division of responsibility. The visible-cleaned, metadata-stripped control and the
|
||||
Z-Image face-only output were both SynthID-positive; Qwen global-only and the full
|
||||
@@ -1165,6 +1590,18 @@ strength-controlled img2img in that family; the full research record,
|
||||
including why FLUX.2 itself is not integrable this way, is
|
||||
[`chroma1-engine-research.md`](chroma1-engine-research.md).
|
||||
|
||||
The profile also implements the shared verified-text donor hook through its already
|
||||
loaded Chroma VAE. It edge-pads native geometry to the /16 latent grid, takes the
|
||||
deterministic mode of the encoder distribution rather than sampling it, decodes with
|
||||
Diffusers' own image processor, and crops back to the exact source dimensions. Thus
|
||||
`auto` still loads and runs only the profile selected by the measured vendor table;
|
||||
the presence of a manifest does not introduce a second generative model or override
|
||||
that decision. A CPU smoke run against the official Chroma1-HD `AutoencoderKL`
|
||||
weights on 2026-08-31 verified the real encode/decode contract and exact restoration
|
||||
of a non-grid-aligned source size; model-free tests pin mode-not-sample and padding.
|
||||
Chroma-specific full-pipeline provider-oracle certification remains an output-hash
|
||||
measurement, not something either check can establish.
|
||||
|
||||
The profile swaps only the global stage (same inheritance invariant as
|
||||
sdxl-zimage, asserted by the same test shape). Three things are bound to the
|
||||
calibration and must not drift: the NEUTRAL prompt (`high quality, sharp,
|
||||
@@ -1196,6 +1633,24 @@ that rung, which is the margin the 0.17 floor already holds. OpenAI stays on the
|
||||
clear at or below 0.06, including a 2-face file, and only the 9-face
|
||||
grid needs 0.075 (seed-stable on 0, 1, 2 via openai.com/research/verify).
|
||||
|
||||
A separate content-balanced check on 2026-08-31 tested the exact production
|
||||
OpenAI and Meta floors over 19 prompt-matched image strata per provider. Each
|
||||
engine ran in its own sequential H100 invocation, and the comparison isolated
|
||||
the global stage because the shared face repair does not change with this choice.
|
||||
On OpenAI content, Chroma won SSIM and PSNR on 19/19 pairs, MAE on 18/19, edge
|
||||
F1 on 15/19, and LPIPS on 12/19. On Meta content, Qwen won LPIPS and edge F1 on
|
||||
18/19 pairs; Chroma's better PSNR on 16/19 did not overcome its systematically
|
||||
worse perceptual and edge preservation at the higher required floor. Exact
|
||||
two-sided sign-test p-values were 3.8e-6 for the OpenAI SSIM and PSNR directions
|
||||
and 7.6e-5 for the Meta LPIPS and edge-F1 directions. The tracked discovery
|
||||
inputs and manifests are in
|
||||
[`data/evaluations/engine-selection/`](../data/evaluations/engine-selection/),
|
||||
the run is reproduced by `scripts/engine_selection_study.py`, and paired analysis
|
||||
by `scripts/analyze_engine_selection_study.py`. These content files were locally
|
||||
re-encoded and are not watermark oracles; they validate fidelity at already
|
||||
oracle-calibrated floors, not removal by themselves. No content-stratum override
|
||||
is supported by this pass, so `auto` keeps the measured provenance-cohort table.
|
||||
|
||||
### SDXL plus Z-Image
|
||||
|
||||
[`_internal/sdxl_zimage_pipeline.py`](../src/remove_ai_watermarks/_internal/sdxl_zimage_pipeline.py)
|
||||
@@ -1380,7 +1835,8 @@ grain, unsharp masking, and adaptive polish helpers.
|
||||
|
||||
`upscaler.py` held an optional Real-ESRGAN path, reachable only when enlarging a
|
||||
small image to the minimum-resolution floor. That floor existed to lift small
|
||||
inputs toward SDXL's ~1024 training size; when the SDXL profiles were removed it
|
||||
inputs toward SDXL's ~1024 training size; when the standalone SDXL and ControlNet
|
||||
profiles were removed (`sdxl-zimage` is a later, surviving profile) it
|
||||
was forced to 0 on every path, so the module, the `--min-resolution` and
|
||||
`--upscaler` options and the `esrgan` extra were all unreachable and went with
|
||||
it. Only the `max_resolution` cap can move geometry now, and it only scales down.
|
||||
|
||||
@@ -0,0 +1,243 @@
|
||||
---
|
||||
language:
|
||||
- en
|
||||
license: apache-2.0
|
||||
library_name: transformers
|
||||
pipeline_tag: image-classification
|
||||
base_model: openai/clip-vit-large-patch14
|
||||
inference: false
|
||||
tags:
|
||||
- clip
|
||||
- image-classification
|
||||
- ai-generated-content
|
||||
- photo
|
||||
- provider-attribution
|
||||
datasets:
|
||||
- detection-datasets/coco
|
||||
---
|
||||
|
||||
# Photo AI-versus-camera classifier
|
||||
|
||||
Two heads, one call. Model 1 decides whether a **photograph** looks generated
|
||||
or camera-like. Model 2 names a provider only after Model 1 is DEFINITELY AI.
|
||||
|
||||
This is not a SynthID decoder. It is not provenance. It is not a universal
|
||||
"AI or not" detector for receipts, UI, or digital art. The Python package
|
||||
[`remove-ai-watermarks`](https://github.com/wiltodelta/remove-ai-watermarks)
|
||||
loads these files through an explicit `classify` extra. `identify` never
|
||||
starts this model, including after a no-signal metadata scan.
|
||||
|
||||
Library guide:
|
||||
[photo pixel classification](https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/photo-classify.md).
|
||||
Research archive:
|
||||
[AI-generated image classifiers](https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/ai-generated-image-classifiers.md).
|
||||
The demo Space
|
||||
[`wiltodelta/remove-ai-watermarks`](https://huggingface.co/spaces/wiltodelta/remove-ai-watermarks)
|
||||
is a watermark-removal UI, not this classifier.
|
||||
|
||||
The training catalog is not in this Hub repository and is not on GitHub.
|
||||
|
||||
## Intended use
|
||||
|
||||
Use it when you have a photographic still and you want a pixel opinion after
|
||||
metadata is gone:
|
||||
|
||||
- `label=ai` plus `provider=openai|google|muse-image|tc260` on a stripped ChatGPT,
|
||||
Gemini, Muse Image, or TC260 photograph
|
||||
- `label=human` on camera photographs at a low false-positive rate
|
||||
- `label=unknown` when the detector is only POSSIBLY AI, or when 124-d
|
||||
features cannot be extracted
|
||||
|
||||
Do not use it to assert that a file is clean. Do not use it as a legal
|
||||
authorship test. Do not run Model 2 on every image.
|
||||
|
||||
## Approach
|
||||
|
||||
Content embeddings and forensic residuals answer different questions.
|
||||
|
||||
CLIP-L, after a light vision-block fine-tune, separates **generated
|
||||
photographs** from **camera photographs**. It does not reliably separate
|
||||
generated photographs from receipts, UI, charts, or illustration. A ridge
|
||||
trained on that embedding at a 1% Open Images false-positive cut is the
|
||||
strongest Model 1 we measured. A small MLP on the same vectors, ANDed with
|
||||
the ridge, is the freeze DEFINITELY gate.
|
||||
|
||||
The 124-d bank is a different feature: patch-local residual ratios (FFT band
|
||||
energy, comb contrast, autocovariance) on 256 px tiles. On that bank, OpenAI
|
||||
versus Gemini is easy (AUC 0.989 in the research sweep) while "AI or not" is
|
||||
the wrong job. Firefly and PixelBin sit near Google on those residuals, which
|
||||
is why a provider finder must keep foreign generators in the test and must
|
||||
abstain.
|
||||
|
||||
The two heads therefore stay specialized:
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
image[Photograph] --> m1[Model 1: CLIP-L-ft ridge AND MLP]
|
||||
m1 -->|likely_human| human[label human, provider none]
|
||||
m1 -->|possibly| unk[label unknown, provider none]
|
||||
m1 -->|definitely| m2[Model 2: 124-d one-vs-rest focal]
|
||||
m2 -->|openai google muse-image tc260| named[label ai plus provider]
|
||||
m2 -->|no_ai or extract fail| aiOnly[label ai, provider none]
|
||||
```
|
||||
|
||||
CLIP content space is the wrong place to train "receipt versus rest" or
|
||||
"UI versus rest" if the product contract is photographic. Those domains
|
||||
remain open. The shipped heads do not include a receipt specialist.
|
||||
|
||||
Provider attribution tracks the **renderer**, not the front-end. Bing Image
|
||||
Creator rows signed Microsoft, OpenAI score as `openai`. Designer rows signed
|
||||
Microsoft, Google LLC score as `google`. Native Designer is mixed. There is
|
||||
no Microsoft pixel class. `openai` and `google` are provider classes.
|
||||
`muse-image` is Muse Image output, not a general Meta class. Instagram
|
||||
`made_with_ai` is not that class. `tc260` is the China AIGC label standard,
|
||||
not one producer: Doubao, Jimeng, Qwen, Kling, and others share that class.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Model 1
|
||||
|
||||
- Backbone: `openai/clip-vit-large-patch14`
|
||||
- Fine-tune: last two vision blocks, visual projection, post-layernorm
|
||||
- Preprocess: 224 letterbox, pad RGB `(123, 117, 104)`, BICUBIC, L2-normalized
|
||||
`get_image_features` (`pooler_output` when present)
|
||||
- Ridge: linear probe on 768-d vectors, threshold `thr_oi_1pct` = 0.3056
|
||||
from the locked `photo_dev_oi` 1% cut
|
||||
- MLP: `768-512-128-1` with dropout 0.3 / 0.1, threshold 5.9586, 1.67%
|
||||
photo-dev quantile
|
||||
- DEFINITELY = ridge AND MLP. POSSIBLY = either. The public `label` is `ai`
|
||||
only on DEFINITELY.
|
||||
|
||||
### Model 2
|
||||
|
||||
- Input: 124-d residual vector, or abstain if the image is smaller than
|
||||
256 px or the extractor refuses
|
||||
- Heads: one-vs-rest focal MLP `124-64-1` per class
|
||||
(`openai`, `google`, `tc260`, `meta_muse_image`, `no_ai`)
|
||||
- Decision: a class wins only if it beats `no_ai` by margin 0.30, then argmax
|
||||
among those that passed
|
||||
- Public name `muse-image` is the Muse Image class. The freeze file still
|
||||
keys that head `meta_muse_image`. `tc260` is the China AIGC label
|
||||
standard (mixed producers), not a company. `no_ai` is not reported as
|
||||
`human`; it becomes `provider=None` on an `ai` label (for example FLUX)
|
||||
|
||||
## Training
|
||||
|
||||
CPU. Two consecutive freeze runs from the same catalog and caches were
|
||||
byte-identical.
|
||||
|
||||
| Item | Value |
|
||||
| --- | --- |
|
||||
| Catalog identity | sha256 of file bytes, 32,690 listed files |
|
||||
| CLIP-L embeddings keyed | 26,573 |
|
||||
| Detector seed | 20260940 |
|
||||
| Provider seed | 20260956 |
|
||||
| Sample seed for provider OpenAI/Google draws | 20260821 |
|
||||
| MLP epochs / width | 40 / 512 |
|
||||
| Provider focal gamma / epochs | 2.0 / 40 |
|
||||
| Private UI in train | 0. Replaced with 150 CC Flickr screenshots via Openverse, host `live.staticflickr.com` only |
|
||||
|
||||
Flickr licenses in that replacement cell: CC BY 2.0 42, CC BY-NC 2.0 42,
|
||||
CC BY-SA 2.0 29, CC BY-NC-SA 2.0 28, CC BY-NC-ND 2.0 6, CC BY-ND 2.0 2,
|
||||
CC0 1.
|
||||
|
||||
The pixel catalog is **not published on the Hub**. Head retrain uses
|
||||
sha256-keyed CLIP-L and 124-d caches plus `catalog.json` roles. The original
|
||||
files remain stored privately so embeddings and 124-d features can be rebuilt.
|
||||
Public named sources in those roles include COCO, Open Images, Kodak, Picsum,
|
||||
Hugging Face FLUX, Openverse CC Flickr, and Meta Muse API hold-out v3.
|
||||
In-house labeled OpenAI, Gemini, and TC260 photographs stay private. See
|
||||
[photo-classify-training.md](https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/photo-classify-training.md).
|
||||
|
||||
## Evaluation (run 1 = run 2)
|
||||
|
||||
DEFINITELY is the shipped operating point.
|
||||
|
||||
| Head | Cell | Result |
|
||||
| --- | --- | ---: |
|
||||
| Detector DEFINITELY | AI-test | 92.4% (n=1,847) |
|
||||
| Detector DEFINITELY | Open Images fresh | 1.3% (n=3,000) |
|
||||
| Detector DEFINITELY | Kodak | 0/24 |
|
||||
| Detector DEFINITELY | FLUX hold | 83.0% (n=300) |
|
||||
| Class | OpenAI test | 90.8% (345/380 of 381) |
|
||||
| Class | Google test | 90.9% (339/373 of 377) |
|
||||
| Class | TC260 test | 78.6% (298/379 of 384) |
|
||||
| Class | Muse Image hold-out v2 | 85.7% (66/77 listed 79) |
|
||||
| Class | Muse Image hold-out v3 | 89.4% (177/198) |
|
||||
| Class | Muse Image hold-out pooled | 88.4% (243/275 listed 277) |
|
||||
| Class | meme templates, ungated | 29.1% leak (86 of 97) |
|
||||
|
||||
The ungated meme leak is why Model 2 must not run on every file. Gated on
|
||||
DEFINITELY, that leak is not a provider attribution.
|
||||
|
||||
Ridge-only Model 1 at the earlier 1% Open Images cut (before the AND with
|
||||
the freeze MLP) is documented in the research page: Kodak 0/24, fresh 1.7%,
|
||||
AI-test 93.0%, FLUX 92.7%, Firefly 94.0%. Nobody in that sweep hit both
|
||||
≤1% fresh FPR and ≥90% TPR. The freeze AND is stricter on photos (1.3%
|
||||
DEF) at a small recall cost.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Photographs only.** Receipts, UI, scans of forms, maps, and community
|
||||
digital art sit in the AI region of CLIP-L. A real thermal receipt can
|
||||
score as AI. A photorealistic AI receipt can score as human.
|
||||
- **Not SynthID.** A `provider=openai` result means the pixels look like an
|
||||
OpenAI-rendered photograph, not that a payload was decoded.
|
||||
- **Not provenance.** C2PA, IPTC, and visible marks stay on `identify`.
|
||||
- **124-d can refuse.** Some sizes and some images yield no vector. The
|
||||
detector can still return DEFINITELY; provider is then `None`.
|
||||
- **Microsoft has no head.** Route by renderer, or use C2PA for the
|
||||
front-end name.
|
||||
- **Open-world generators.** FLUX is a hold-out at 83% DEF, not 100%.
|
||||
Unknown generators may land on `ai` with `provider=None`, or miss
|
||||
DEFINITELY.
|
||||
- **`tc260` is mixed producers.** Doubao, Jimeng, Qwen, Kling, and others
|
||||
share one label-standard head. A later retrain should split by
|
||||
`ContentProducer`. Do not read `tc260` as a named manufacturer.
|
||||
|
||||
## How to use
|
||||
|
||||
Install the extra. First call downloads this repository into the Hugging
|
||||
Face cache, or reads `RAIW_CLASSIFY_WEIGHTS` if you already have the files.
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[classify]"
|
||||
remove-ai-watermarks classify image.png
|
||||
```
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from remove_ai_watermarks.classify import classify_pixels
|
||||
|
||||
result = classify_pixels(Path("image.png"))
|
||||
# result.label, result.detector, result.provider
|
||||
```
|
||||
|
||||
`device` is a library parameter (`None` / `"auto"` / `"cpu"` / `"cuda"`),
|
||||
not a CLI option. Pin a Hub revision in production so a later upload cannot
|
||||
silently change the heads.
|
||||
|
||||
## Files in this repository
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `clip-l-ft.pt` | HeadedCLIP state dict (CLIP-L plus unused linear head) |
|
||||
| `probe-weights-clip-l-ft.npz` | Ridge mean, scale, weights, `thr_oi_1pct` |
|
||||
| `detector.pt` | Freeze MLP |
|
||||
| `provider.pt` | Focal heads keyed `openai`, `google`, `tc260`, `meta_muse_image`, `no_ai` |
|
||||
| `operating-point.json` | Seeds, thresholds, margin |
|
||||
|
||||
The image catalog is not in this repository.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0 for the trained heads and this card. CLIP-L inherits its upstream
|
||||
license from [`openai/clip-vit-large-patch14`](https://huggingface.co/openai/clip-vit-large-patch14).
|
||||
Flickr replacement images are CC-licensed per file; they are not redistributed
|
||||
here.
|
||||
|
||||
## Citation
|
||||
|
||||
Package [`remove-ai-watermarks`](https://github.com/wiltodelta/remove-ai-watermarks).
|
||||
Protocol and rejected variants: [docs/ai-generated-image-classifiers.md](https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/ai-generated-image-classifiers.md).
|
||||
Provider renderer study: [docs/synthid-classifiers.md](https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/synthid-classifiers.md).
|
||||
@@ -0,0 +1,43 @@
|
||||
{
|
||||
"freeze": "2026-08-31",
|
||||
"task": "photo_ai_versus_camera",
|
||||
"not": ["synthid_decoder", "provenance", "universal_ai_detector"],
|
||||
"seeds": {
|
||||
"sample": 20260821,
|
||||
"detector": 20260940,
|
||||
"provider": 20260956
|
||||
},
|
||||
"model1": {
|
||||
"backbone": "openai/clip-vit-large-patch14",
|
||||
"letterbox": 224,
|
||||
"pad_rgb": [123, 117, 104],
|
||||
"ridge_threshold_key": "thr_oi_1pct",
|
||||
"ridge_threshold": 0.3056212276800537,
|
||||
"mlp_threshold": 5.9586493237495395,
|
||||
"mlp_width": 512,
|
||||
"definitely": "ridge_and_mlp",
|
||||
"public_ai_label": "definitely_only"
|
||||
},
|
||||
"model2": {
|
||||
"feature_width": 124,
|
||||
"margin": 0.3,
|
||||
"classes": ["openai", "google", "tc260", "muse-image", "no_ai"],
|
||||
"class_kind": {
|
||||
"openai": "provider",
|
||||
"google": "provider",
|
||||
"tc260": "label-standard",
|
||||
"muse-image": "model",
|
||||
"no_ai": "negative"
|
||||
},
|
||||
"checkpoint_keys": {
|
||||
"muse-image": "meta_muse_image"
|
||||
},
|
||||
"runs_only_after": "definitely"
|
||||
},
|
||||
"files": [
|
||||
"clip-l-ft.pt",
|
||||
"probe-weights-clip-l-ft.npz",
|
||||
"detector.pt",
|
||||
"provider.pt"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
# Photo classifier training and data
|
||||
|
||||
How the 2026-08-31 freeze was fit, what a retrain needs, and where those
|
||||
files come from. Inference is [photo-classify.md](photo-classify.md). The
|
||||
campaign log is [ai-generated-image-classifiers.md](ai-generated-image-classifiers.md).
|
||||
|
||||
The shipped package does not include the catalog or the embeddings. Fitting
|
||||
the heads again uses the cache pack. The original pixels still have to be
|
||||
kept: a missing embedding, a new backbone, or an audit all need the files.
|
||||
|
||||
## Retrain pack (caches)
|
||||
|
||||
CPU retrain of both heads needs sha256-keyed matrices and the catalog of
|
||||
roles. That step does not open JPEG or PNG files. It is not permission to
|
||||
drop the pixels.
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `catalog.json` | 32,690 rows: `sha256`, `roles`, optional `family` / public `source` |
|
||||
| `embeddings-by-sha256.npz` | CLIP-L-ft 768-d vectors for Model 1 |
|
||||
| `extra-clip-l.npz` | CLIP-L extras for the 150 Flickr UI hashes |
|
||||
| `features-124d.npz` | 124-d residual vectors for Model 2 |
|
||||
| `probe-weights-clip-l-ft.npz` | Ridge mean, scale, weights, `thr_oi_1pct` |
|
||||
| `meta-split.json` | Meta train 217, hold-out v2 79, hold-out v3 198 |
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
uv run python scripts/retrain_photo_classify.py --pack DIR --out DIR
|
||||
```
|
||||
|
||||
Seeds are fixed: detector `20260940`, provider `20260956`. Two consecutive
|
||||
runs must be byte-identical.
|
||||
|
||||
Not in the **published** retrain pack: raw photographs, COCO/Open Images/Kodak
|
||||
binaries, CORD, Danbooru. Those files stay in their public corpora or in the
|
||||
private freeze archive. Pixel rebuild of a missing embedding still needs
|
||||
`clip-l-ft.pt` and the original file.
|
||||
|
||||
## Where the rows come from
|
||||
|
||||
Identity is always sha256 of the file bytes.
|
||||
|
||||
### Public, named corpora
|
||||
|
||||
| Role | Source |
|
||||
| --- | --- |
|
||||
| `detector_photo_train`, `detector_photo_coco_hold` | COCO 2017 validation stills |
|
||||
| `detector_open_images_*`, `detector_photo_dev_oi` | Open Images photographs. Locked fresh never enters train |
|
||||
| `detector_photo_kodak` | Kodak lossless true color, 24 files |
|
||||
| `detector_photo_picsum_eval` | Picsum |
|
||||
| `detector_flux_hf_hold` | Hugging Face FLUX stills, hold-out |
|
||||
| Flickr UI cell (150) | Openverse `license_type=all-cc`, host `live.staticflickr.com` only |
|
||||
| Meta hold-out v3 (198) | Meta Muse Image API, eval-only, not in train |
|
||||
|
||||
### In-house labeled photographs, not redistributed
|
||||
|
||||
Provider OpenAI, Google, and TC260 train and test rows, and most detector AI
|
||||
train and test rows, are labeled photographic generations held privately.
|
||||
The published pack carries only their sha256, roles, and precomputed vectors.
|
||||
It does not carry the pixels.
|
||||
|
||||
### Out of the freeze train
|
||||
|
||||
CORD receipts, FUNSD forms, Danbooru, charts, and map tiles were measured as
|
||||
failure domains. They are not Model 1 train. A future universal router treats
|
||||
them as other domains, not as extra CLIP-L negatives.
|
||||
|
||||
## What the freeze script does
|
||||
|
||||
Historically `freeze_both_models.py` built the catalog from local paths, then
|
||||
trained. That campaign code is private and is not in this repository. The
|
||||
public entry is `scripts/retrain_photo_classify.py`, which skips catalog
|
||||
construction and fits the same heads from the pack above.
|
||||
|
||||
1. Load CLIP-L-ft vectors by sha256. Fit the 768-512-128-1 MLP on
|
||||
`detector_ai_train` plus extras versus `detector_photo_train` plus struct.
|
||||
Threshold is the 1.67% quantile on `detector_photo_dev_oi`.
|
||||
2. AND that MLP with the frozen ridge in the probe file. DEFINITELY is both.
|
||||
3. Load 124-d vectors. Fit one-vs-rest focal heads for openai, google, tc260,
|
||||
meta_muse_image, no_ai. Margin 0.30. The public class for the Muse Image
|
||||
head is `muse-image`. `tc260` is the China AIGC label standard, mixed
|
||||
producers, not one company.
|
||||
4. Repeat with the same seeds. Compare bytes.
|
||||
|
||||
CLIP-L fine-tune of the last two vision blocks is earlier, seed 20260822, and
|
||||
is not repeated by the retrain script. The backbone file `clip-l-ft.pt` is an
|
||||
inference and embedding dependency, not a retrain-pack input.
|
||||
|
||||
## Next retrain
|
||||
|
||||
Keep the current `tc260` head until then. The next provider-head fit should
|
||||
split that catalog by `ContentProducer` (Doubao, Jimeng, Qwen, Kling, and
|
||||
the rest) instead of one label-standard class. Do not treat a `tc260` score
|
||||
as a named manufacturer.
|
||||
@@ -0,0 +1,126 @@
|
||||
# Photo pixel classification
|
||||
|
||||
`classify` gives a pixel opinion on a **photograph**: camera-like, generated,
|
||||
or unknown, and optionally which renderer it resembles. It is a separate
|
||||
command from `identify`. Metadata inspection never starts it.
|
||||
|
||||
This page is the library guide. The Hugging Face model card is the same freeze
|
||||
written for Hub publication: [photo-classify-hf/README.md](photo-classify-hf/README.md).
|
||||
Campaign notes and rejected variants live in
|
||||
[AI-generated image classifiers](ai-generated-image-classifiers.md).
|
||||
|
||||
## When to use it
|
||||
|
||||
Use `identify` for C2PA, IPTC, visible marks, and other provenance. Use
|
||||
`classify` when those signals are gone and the file is still a photograph.
|
||||
|
||||
Do not use `classify` to prove a file is clean. Do not use it on receipts, UI,
|
||||
or digital art. Do not treat `provider=openai` as a decoded SynthID payload.
|
||||
|
||||
## Install and run
|
||||
|
||||
```bash
|
||||
uv tool install --force "remove-ai-watermarks[classify]"
|
||||
remove-ai-watermarks classify image.png
|
||||
remove-ai-watermarks classify image.png --json
|
||||
```
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
from remove_ai_watermarks.classify import classify_pixels
|
||||
|
||||
result = classify_pixels(Path("image.png"))
|
||||
print(result.label, result.detector, result.provider)
|
||||
```
|
||||
|
||||
Missing extra raises `RuntimeError` with
|
||||
`pip install 'remove-ai-watermarks[classify]'`. `device` is a library
|
||||
parameter (`None` / `"auto"` / `"cpu"` / `"cuda"`), not a CLI option.
|
||||
|
||||
Weights are not in git. First call downloads
|
||||
[`wiltodelta/raiw-photo-classify`](https://huggingface.co/wiltodelta/raiw-photo-classify)
|
||||
at the freeze revision pinned in `classify.py`, or reads
|
||||
`RAIW_CLASSIFY_WEIGHTS` if that directory already has `clip-l-ft.pt`,
|
||||
`probe-weights-clip-l-ft.npz`, `detector.pt`, and `provider.pt`. The training
|
||||
catalog is not published with the package.
|
||||
|
||||
## What one call returns
|
||||
|
||||
One request runs both heads. Model 2 runs only after Model 1 is DEFINITELY.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
image[Photograph] --> m1[Model 1 CLIP-L-ft ridge AND MLP]
|
||||
m1 -->|likely_human| human["label human, provider none"]
|
||||
m1 -->|possibly| unk["label unknown, provider none"]
|
||||
m1 -->|definitely| m2[Model 2 124-d focal heads]
|
||||
m2 -->|openai google muse-image tc260| named["label ai plus provider"]
|
||||
m2 -->|no_ai or extract fail| aiOnly["label ai, provider none"]
|
||||
```
|
||||
|
||||
| Field | Values | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `label` | `ai`, `human`, `unknown` | Public verdict. `ai` only on DEFINITELY |
|
||||
| `domain` | `photo` | This freeze is photographic only |
|
||||
| `detector` | `definitely`, `possibly`, `likely_human` | Raw Model 1 gate |
|
||||
| `provider` | `openai`, `google`, `muse-image`, `tc260`, or `None` | Model 2, only if `label` is `ai` |
|
||||
|
||||
`human` is camera-like under this contract, not a proof of authorship.
|
||||
`unknown` is POSSIBLY, or a domain this freeze does not support. `provider=None`
|
||||
on an `ai` file is often FLUX or another generator that is not one of the four
|
||||
heads, or a 124-d extract refusal.
|
||||
|
||||
## Approach
|
||||
|
||||
CLIP-L content embeddings separate generated photographs from camera
|
||||
photographs. They do not separate generated photographs from receipts or UI.
|
||||
A ridge on fine-tuned CLIP-L (last two vision blocks, 224 letterbox) is Model
|
||||
1. A small MLP on the same vectors, ANDed with the ridge, is the freeze
|
||||
DEFINITELY gate.
|
||||
|
||||
Provider attribution uses a different feature: 124-d residual ratios on 256 px
|
||||
patches. OpenAI versus Gemini is a strong split there. "AI or not" is the
|
||||
wrong job on that bank, which is why Model 2 is gated.
|
||||
|
||||
Provider names the renderer, not the product UI. Bing Image Creator signed
|
||||
Microsoft, OpenAI scores `openai`. Designer signed Microsoft, Google LLC
|
||||
scores `google`. There is no Microsoft pixel class. `openai` and `google` are
|
||||
provider classes. `muse-image` is Muse Image output, not a general Meta
|
||||
class. `tc260` is the China AIGC label standard, not one producer: Doubao,
|
||||
Jimeng, Qwen, Kling, and others share that residual class.
|
||||
|
||||
## Evaluation
|
||||
|
||||
Two CPU retrains were byte-identical. DEFINITELY is the shipped cut.
|
||||
|
||||
| Head | Cell | Result |
|
||||
| --- | --- | ---: |
|
||||
| Detector DEFINITELY | AI-test | 92.4% (n=1,847) |
|
||||
| Detector DEFINITELY | Open Images fresh | 1.3% (n=3,000) |
|
||||
| Detector DEFINITELY | Kodak | 0/24 |
|
||||
| Detector DEFINITELY | FLUX hold | 83.0% (n=300) |
|
||||
| Class | OpenAI | 90.8% (345/380 of 381) |
|
||||
| Class | Google | 90.9% (339/373 of 377) |
|
||||
| Class | TC260 | 78.6% (298/379 of 384) |
|
||||
| Class | Muse Image hold-out v3 | 89.4% (177/198) |
|
||||
| Class | Muse Image hold-out pooled | 88.4% (243/275 listed 277) |
|
||||
| Class | meme templates, ungated | 29.1% leak |
|
||||
|
||||
The ungated meme row is why Model 2 never runs on `human` or `unknown`.
|
||||
|
||||
## Limits
|
||||
|
||||
- Receipts, UI, scans, maps, and community art are out of contract.
|
||||
- Not SynthID, not C2PA, not `is_ai_generated`.
|
||||
- Images under 256 px cannot yield 124-d features, so provider abstains.
|
||||
- FLUX is a hold-out at 83% DEF, not a named provider class.
|
||||
- `tc260` is mixed producers under one label standard. A later retrain
|
||||
should split it by manufacturer.
|
||||
|
||||
`identify`, `has_invisible_target`, `all`, and `invisible` do not import this
|
||||
module. A no-signal provenance result stays unknown until you call `classify`
|
||||
yourself.
|
||||
|
||||
Training data, the retrain pack (caches only, no images), and
|
||||
`scripts/retrain_photo_classify.py` are in
|
||||
[photo-classify-training.md](photo-classify-training.md).
|
||||
@@ -6,12 +6,29 @@ and pipeline modules are intended for maintainers and specialized workflows.
|
||||
Dependency groups are identical for the CLI and Python API. The default install
|
||||
covers metadata extraction, normalization, verdict logic, and stripping.
|
||||
Array/pixel APIs use `pixels`; visible removal uses `visible`; DWT-DCT detection
|
||||
uses `detect`; invisible image removal uses `qwen-zimage` and an NVIDIA GPU; and
|
||||
uses `detect`; pixel photo classification uses `classify`; invisible image removal uses `qwen-zimage` and an NVIDIA GPU; and
|
||||
visible video processing uses `video`. Video SynthID removal is a separate VAE
|
||||
path that still runs on CPU and combines `video` and `diffusion`. Add `heif`
|
||||
independently when path-based pixel APIs must decode HEIC, HEIF, or AVIF. See
|
||||
the complete [feature-extra matrix](installation.md#feature-extras).
|
||||
|
||||
## Verify OpenAI SynthID
|
||||
|
||||
```python
|
||||
import remove_ai_watermarks as raiw
|
||||
|
||||
result = raiw.verify_openai_synthid("input.png", acknowledge_upload=True)
|
||||
print(result.status)
|
||||
```
|
||||
|
||||
Official OpenAI JSON results expose `signal_family`,
|
||||
`provider_scope`, `backend`, `metadata_used_for_verdict`, and
|
||||
`pixels_preserved`. Remote transport and response failures raise `OpenAIProvenanceError`. Its
|
||||
`status_code`, `error_code`, `request_id`, `retry_after`, and `retryable`
|
||||
attributes let a caller implement bounded backoff or a circuit breaker without
|
||||
turning an API outage into a false `not_detected` result. One function call still
|
||||
performs at most one upload.
|
||||
|
||||
## Remove visible marks
|
||||
|
||||
Install `remove-ai-watermarks[visible]` before using the visible-removal API.
|
||||
@@ -129,6 +146,31 @@ engine's own parameter names and defaults. `force`, which decides whether the
|
||||
engine runs at all, is a parameter of `remove_all` and `remove_batch` alongside
|
||||
`backend` and `sensitivity`.
|
||||
|
||||
The complete field set, with the shipped defaults:
|
||||
|
||||
| Field | Default | Effect |
|
||||
| --- | --- | --- |
|
||||
| `strength` | `None` | Denoising strength. `None` resolves per profile and vendor cohort. |
|
||||
| `pipeline` | `"qwen-zimage"` | Profile: `qwen-zimage`, `sdxl-zimage`, `chroma-zimage`, or `auto`. |
|
||||
| `vendor` | `None` | Strength cohort, and an assertion that the watermark is present. |
|
||||
| `seed` | `None` | `None` uses the certified seed 0. Removal near the floor is seed-dependent. |
|
||||
| `hf_token` | `None` | Hugging Face token for the model download; the environment and a local `.env` are read when unset. |
|
||||
| `humanize` | `0.0` | Analog film grain. 0 is off; 2.0-6.0 is the useful band. |
|
||||
| `unsharp` | `0.0` | Unsharp-mask sharpening. 0 is off. |
|
||||
| `adaptive_polish` | `None` | `None` lets the profile decide (off for `qwen-zimage`, on for `sdxl-zimage`); `True`/`False` override it. |
|
||||
| `max_resolution` | `0` | Cap the long side before diffusion. 0 keeps native geometry and the most detail. |
|
||||
| `controlnet_conditioning_scale` | `1.0` | Canny ControlNet conditioning on the global stage. Higher stays closer to the original structure and text. |
|
||||
| `cpu_offload` | `False` | Stream model components between CPU and GPU. Lower CUDA memory, slower. |
|
||||
| `tile` | `False` | Regenerate large inputs in overlapping tiles instead of downscaling. |
|
||||
| `tile_size` | `1024` | Tile side in pixels when `tile` is set. |
|
||||
| `tile_overlap` | `128` | Tile overlap in pixels. More overlap hides seams and costs time. |
|
||||
| `text_manifest` | `None` | Path to an operator-verified text manifest; see the text-restoration section. |
|
||||
| `fidelity_anchor` | `False` | Opt into the fidelity-anchor pass. |
|
||||
|
||||
`tests/test_docs_cover_the_public_surface.py` fails when a field is added here
|
||||
and not to this table, because this seam had already drifted by eight fields
|
||||
before anything checked it.
|
||||
|
||||
If AI metadata survives the strip, `remove_all` raises `MetadataStripIncomplete`
|
||||
**before** writing anything: an AI-readable output is worse than no output.
|
||||
|
||||
@@ -144,6 +186,34 @@ print(summary.invisible_unavailable) # outputs that still carry the watermark
|
||||
`mode` is `all`, `visible`, `invisible`, or `metadata`. Pass a constructed
|
||||
`InvisibleEngine` as `engine` to load the model once for the whole directory.
|
||||
|
||||
## Classify a photograph from pixels
|
||||
|
||||
This is not provenance. `identify` does not call it. Install
|
||||
`remove-ai-watermarks[classify]` first.
|
||||
|
||||
```python
|
||||
from pathlib import Path
|
||||
|
||||
from remove_ai_watermarks.classify import classify_pixels
|
||||
|
||||
result = classify_pixels(Path("input.png"))
|
||||
print(result.label, result.detector, result.provider)
|
||||
```
|
||||
|
||||
`label` is `ai` only on a DEFINITELY detector result (ridge AND freeze MLP).
|
||||
POSSIBLY is `unknown`. Camera-like photographs are `human`. `provider` is
|
||||
`openai`, `google`, `muse-image`, or `tc260` only when `label` is `ai` and the 124-d
|
||||
head beats `no_ai` by the freeze margin. `tc260` is the China AIGC label
|
||||
standard (mixed producers), not a company. Otherwise it is `None`, including
|
||||
when 124-d extraction refuses the file.
|
||||
|
||||
`device` is a library parameter: `None` / `"auto"` detect, `"cpu"` or `"cuda"`
|
||||
pin. It is not a CLI option.
|
||||
|
||||
Missing extra raises `RuntimeError` with the quoted install command
|
||||
`'remove-ai-watermarks[classify]'`. Guide:
|
||||
[photo pixel classification](photo-classify.md).
|
||||
|
||||
## Inspect provenance
|
||||
|
||||
The default installation evaluates file metadata. Add `visible`, `detect`, or
|
||||
@@ -344,8 +414,8 @@ Timings and spatial artifacts are opt-in. Artifacts include image-identifying da
|
||||
such as a thumbnail and perceptual hash; aggregate feature families do not.
|
||||
|
||||
`identify_from_evidence` does not reopen the source file by default: it evaluates
|
||||
metadata only, and registered visible marks and pixel-backed invisible watermarks
|
||||
remain in the path-based `identify` call.
|
||||
metadata only, and the pixel-backed checks remain in the path-based `identify`
|
||||
call: registered visible marks and open invisible-watermark decoders.
|
||||
|
||||
Pass `image_path` together with `check_visible` or `check_invisible` to add those
|
||||
pixel detectors on top of the SAME evidence. That is how a caller asking one file
|
||||
@@ -645,11 +715,13 @@ engine.remove_watermark(
|
||||
|
||||
Install `remove-ai-watermarks[text-restoration]`. The manifest schema and safety
|
||||
constraints are documented in the CLI guide. The engine verifies its decoded RGB
|
||||
hash before loading the diffusion models and rejects SDXL, downscaling, and
|
||||
postprocessing combinations that were not evaluated. Tiling is also rejected because
|
||||
the combined tiled-restoration path has no provider-oracle calibration. `InvisibleOptions`
|
||||
exposes the same field for `remove_all`; after a visible-stage edit, the manifest must
|
||||
be built against the staged pixels rather than the pristine source.
|
||||
hash before loading the diffusion models. Qwen and Chroma reconstruct the donor with
|
||||
the VAE already loaded for the one profile selected by `auto`; no second generative
|
||||
profile runs. The engine rejects SDXL, downscaling, and postprocessing combinations
|
||||
that were not evaluated. Tiling is also rejected because the combined
|
||||
tiled-restoration path has no provider-oracle calibration. `InvisibleOptions` exposes
|
||||
the same field for `remove_all`; after a visible-stage edit, the manifest must be built
|
||||
against the staged pixels rather than the pristine source.
|
||||
|
||||
Use manifest schema 1 for manually reviewed text plus script metadata. Automated
|
||||
operators that verify only text-region geometry should emit schema 2 lines with a
|
||||
@@ -660,7 +732,8 @@ default** (`fidelity_anchor=False`): that whole-frame blend was measured to
|
||||
return detector-visible OpenAI SynthID on poster-scale manifests (official
|
||||
Content Provenance API, 2026-08-19 - detected x6 with the anchor, clean x6
|
||||
without it, controls and base outputs validated in the same sessions). Pass
|
||||
`fidelity_anchor=True` to reproduce the 0.27.0 research behavior.
|
||||
`fidelity_anchor=True` with `qwen-zimage` to reproduce the 0.27.0 research
|
||||
behavior. Chroma rejects that Qwen-specific reproduction flag.
|
||||
|
||||
### Drafting manifest lines
|
||||
|
||||
|
||||
@@ -73,6 +73,18 @@ Upgrading the Hugging Face Space is a separate, manual step: bump
|
||||
pin. Its callback smoke tests run in that repository's CI on every push; they
|
||||
are the scenario check for the demo surface.
|
||||
|
||||
The Hugging Face **model** `wiltodelta/raiw-photo-classify` is not part of that
|
||||
release fan-out. It holds the photo-classify freeze weights. The library extra
|
||||
`classify` downloads that snapshot on first use, or reads
|
||||
`RAIW_CLASSIFY_WEIGHTS`. Update the Hub repo only with
|
||||
`.github/workflows/publish-photo-classify-hf.yml` (`workflow_dispatch`,
|
||||
type `publish`). Full mode downloads the four weight files from the GitHub
|
||||
Release tag `photo-classify-freeze-2026-08-31` (or the tag you pass) and
|
||||
uploads them with the card in `docs/photo-classify-hf/`. Card mode updates
|
||||
the README and `operating-point.json` only. A library version bump does not
|
||||
upload new heads. Pin a Hub revision in production so a later freeze cannot
|
||||
silently change installed clients.
|
||||
|
||||
The workflow can also be started manually with an optional version input.
|
||||
|
||||
If a distribution job fails because a repository or Hugging Face credential is
|
||||
@@ -123,9 +135,14 @@ requests an older supported one.
|
||||
|
||||
After publication, the `verify-release.yml` workflow runs automatically once
|
||||
`distribute.yml` completes and checks every surface below. It can also be
|
||||
dispatched manually with a specific version. The ComfyUI check verifies the
|
||||
sync workflow's latest run succeeded rather than querying the registry directly
|
||||
(the registry has no public JSON API). The full checklist for completeness:
|
||||
dispatched manually with a specific version. Every check fails the job; none of
|
||||
them warns and continues, because a surface nobody can confirm is not a verified
|
||||
surface. The ComfyUI check reads the published node from the registry's JSON API
|
||||
(`https://api.comfy.org/nodes/remove-ai-watermarks`) and asserts the active
|
||||
version declares this library release. It deliberately does NOT proxy through
|
||||
the sync workflow's run status: the node repository also runs that workflow on a
|
||||
daily schedule, so a green nightly run says nothing about whether this release
|
||||
was synced. The full checklist for completeness:
|
||||
|
||||
- both wheel and source distribution exist on PyPI;
|
||||
- the package version matches the tag;
|
||||
|
||||
@@ -141,12 +141,23 @@ Current pipeline values, all CUDA-only:
|
||||
The `controlnet`, `sdxl`, `qwen` and `default` values were removed. A retired name
|
||||
is rejected at parse time rather than remapped onto a surviving profile.
|
||||
|
||||
SynthID does not have a public local pixel decoder in this project. The tool
|
||||
recognizes presence from supported provenance: Google AI C2PA under Google's
|
||||
all-media watermark policy, and current OpenAI C2PA carrying an explicit
|
||||
`c2pa.watermarked.*` action. Legacy OpenAI C2PA without that action does not
|
||||
assert SynthID. After provenance metadata is removed, a local negative result
|
||||
is still inconclusive.
|
||||
Google does not publish the SynthID payload decoder. This package does not
|
||||
ship a local pixel detector for that watermark. Research on a periodic
|
||||
lattice expert is in [synthid-detector-research.md](synthid-detector-research.md)
|
||||
and `scripts/synthid_runtime/`.
|
||||
|
||||
The tool recognizes presence from supported provenance: Google AI C2PA
|
||||
under Google's all-media watermark policy, and current OpenAI C2PA carrying an
|
||||
explicit `c2pa.watermarked.*` action. Legacy OpenAI C2PA without that action
|
||||
does not assert SynthID.
|
||||
|
||||
The optional `verify-openai-synthid` command is a separate official remote
|
||||
verifier for supported OpenAI watermarks. It strips AI provenance metadata from
|
||||
a temporary PNG, JPEG, or WebP copy, proves that decoded RGBA pixels are
|
||||
unchanged, and uses only the API's SynthID result. It is therefore independent
|
||||
of C2PA for its decision, but it is not local: the sanitized raster is uploaded
|
||||
to OpenAI after explicit acknowledgement. It is intentionally excluded from
|
||||
`identify` and its negative result remains inconclusive.
|
||||
|
||||
Microsoft Paint can name `com.microsoft.invismark.1` in a C2PA soft-binding
|
||||
assertion. Inspection reports both that exact algorithm and its signed `value`,
|
||||
@@ -219,7 +230,7 @@ not a universal clean verdict.
|
||||
|
||||
| Provider or family | Visible | Invisible path | Metadata or provenance |
|
||||
| --- | --- | --- | --- |
|
||||
| Google Gemini | Sparkle | Diffusion regeneration for SynthID | C2PA and related source signals |
|
||||
| Google Gemini | Sparkle | Diffusion regeneration | C2PA and related source signals |
|
||||
| Google Veo video | Veo diamond and legacy text | Oracle-certified VAE removal for SynthID | C2PA and related source signals |
|
||||
| OpenAI image generators | None registered | Diffusion regeneration for supported invisible signals | C2PA and generator provenance |
|
||||
| Meta Muse Image | None on Muse output (legacy `Imagined with AI` unregistered) | Diffusion regeneration for Content Seal, oracle-verified on the default profile | XMP IPTC `trainedAlgorithmicMedia` companion tag; no local Content Seal decoder |
|
||||
|
||||
@@ -0,0 +1,518 @@
|
||||
# SynthID source classifiers (research)
|
||||
|
||||
> Research archive for metadata-free OpenAI/Gemini source finding and
|
||||
> provider-lineage classifiers. These are not SynthID payload decoders and are
|
||||
> not shipped product verdicts. Current behavior:
|
||||
> [supported signals](supported-signals.md) and
|
||||
> [known limitations](known-limitations.md).
|
||||
>
|
||||
> Sister pages: [general AI-generated image classifiers](ai-generated-image-classifiers.md),
|
||||
> [SynthID local detector](synthid-detector-research.md),
|
||||
> [SynthID mark removal](synthid-removal-research.md), and
|
||||
> [mechanism reference](synthid.md).
|
||||
|
||||
A source classifier is reliable only when photographs and non-target generators
|
||||
are explicit negatives, a strict rule can abstain, and any watermark claim uses
|
||||
an independent oracle. CLIP content embeddings and the 124-d origin-locked
|
||||
residual bank are different features for different jobs.
|
||||
|
||||
Krawetz's Gemini-chat TPR critique is a verifier-quality claim, not a
|
||||
feature we can ship. [Lead Stories, 2026-07](https://leadstories.com/analysis/2026/07/google-gemini-synthid-detector-confuses-results-within-same-chat.html)
|
||||
documented Gemini repeating the first file's SynthID verdict inside a
|
||||
chat; Google said that was fixed 2026-07-16. The OpenAI provenance API is
|
||||
a different endpoint.
|
||||
|
||||
## Provider names from pixels
|
||||
|
||||
The task is: given a file with no metadata, is this OpenAI, Gemini, or
|
||||
unknown, with almost no errors on camera photographs or other generators. That
|
||||
is this section. `unknown` does not mean `not AI`; it includes AI images from
|
||||
other providers and target-provider images the strict rule misses. This is not
|
||||
a general AI-generation detector or a SynthID detector. Firefly, PixelBin, and
|
||||
other generators have to sit in the test, because a head that only sees OpenAI
|
||||
versus Gemini versus COCO will call Firefly a provider.
|
||||
|
||||
Three-way `openai` / `google` / `other` on Model 1 embeddings fails the
|
||||
Firefly gate. CLIP-L-ft test accuracy 0.53; Firefly 35/31/18. CLIP-H 0.57;
|
||||
Firefly 36/33/15. OpenAI versus Gemini AUC on CLIP-L-ft is 0.845; on the
|
||||
124-d lattice bank it is 0.989. They are two pipelines, not one class.
|
||||
|
||||
### Renderer, not front-end (cross-carrier validation, 2026-08-26)
|
||||
|
||||
A local C2PA-labeled collection's issuer field allows a provider-label audit the
|
||||
family labels cannot express: Bing Image Creator rows signed `Microsoft,
|
||||
OpenAI` (renderer DALL-E, n=152), Microsoft-native rows (MAI-Image / Designer,
|
||||
n=85), Designer rows signed `Microsoft, Google LLC` (renderer Imagen, n=36),
|
||||
and the unattributed Instagram `made_with_ai` label set (n=275). Scored with
|
||||
the frozen 124-feature openai/google/no_ai cascade, no training, metadata
|
||||
used only to build cells: argmax called 124/152 (81.6%) of the Bing+DALL-E
|
||||
cell `openai` and 34/36 (94.4%) of the Microsoft+Imagen cell `google`. Pixel
|
||||
attribution therefore tracks the renderer behind the front-end, which is the
|
||||
correct semantic for a stripped file: a Bing export is an OpenAI-rendered
|
||||
image. The Microsoft-native cell split 48 openai / 35 google, consistent with
|
||||
Designer historically routing between DALL-E and Imagen rather than one
|
||||
pipeline. The Instagram `made_with_ai` set is not a Meta pixel class: argmax
|
||||
leaned `google` at 196/275 with 68 `openai`, and the photo-first 0.50 margin
|
||||
sent 182/275 to `no_ai`, so the label marks detected AI content of mixed
|
||||
origin, not Meta-rendered pixels. A real Meta class needs fresh
|
||||
`Imagined with AI` generations with known provenance; the catalog label is
|
||||
not one. Artifacts:
|
||||
`provider-renderer-cells-2026-08-26/report.json`.
|
||||
|
||||
### A Meta class from muse-image-1.0, 2026-08-26
|
||||
|
||||
The Meta Model API (`api.meta.ai/v1`, OpenAI-compatible images endpoint) made
|
||||
a known-provenance Meta corpus possible without any account browser session:
|
||||
61 images generated from a 61-prompt grid spanning portraits, product shots,
|
||||
scenes, food, animals, architecture, illustration styles, text posters, and
|
||||
abstract work across all three aspect ratios, plus the five oracle-verified
|
||||
oracle-verified content-seal samples already on disk. All 66 are content-hash
|
||||
unique, delivered at 1600x1600 / 1920x1280 / 1280x1920, and every API row
|
||||
carries IPTC `trainedAlgorithmicMedia` plus a Content Seal generation id
|
||||
recorded in the manifest. A four-head control with a `microsoft_native` class
|
||||
(83 issuer-verified rows) failed first: repeated-split argmax recall 0.38 mean
|
||||
with 0.16-0.56 spread, because Designer routes between renderers and the class
|
||||
is mixed. The same recipe with `meta_muse_image` in its place holds: argmax
|
||||
recall 0.86 mean (0.75-1.00) on just 66 images, openai 0.952 and google 0.956
|
||||
unharmed, and the frozen cross-carrier cells keep their renderer semantics
|
||||
(Bing+DALL-E 119/146 openai, Microsoft+Imagen 32/36 google) with only small
|
||||
meta leakage (14 and 3 argmax rows). The class label names what the corpus is:
|
||||
muse-image-1.0 API output, not the unverified assumption that the consumer
|
||||
Imagine feature renders identically. The full-data model calls 59/66 meta rows
|
||||
correctly, with seven leaking to google and none to openai. The photo-first
|
||||
0.50 margin does not transfer to a 66-image class (0.398 mean recall); argmax
|
||||
is the honest operating rule until the corpus grows. Muse Image is therefore
|
||||
a separable fourth provider class on pixels, unlike the Microsoft front-end.
|
||||
Artifacts: `meta-muse-corpus-2026-08-26/`, `four-head-provider-2026-08-26/`,
|
||||
`meta-provider-cells-2026-08-26/`.
|
||||
|
||||
A paired chat-vs-API check then closed the label question. Two prompts from
|
||||
the API grid (weathered fisherman portrait, white-sneaker product shot) were
|
||||
submitted to the consumer Meta AI chat imagine flow in the user's own logged-in
|
||||
browser session; the chat delivered 1280x1920 and 1920x1280 WebP files from the
|
||||
`t39.105495-1` CDN family, each carrying IPTC `trainedAlgorithmicMedia` in the
|
||||
downloaded bytes. Scored by the frozen API-trained four-head model, which had
|
||||
never seen a chat image: both chat rows landed on `meta_muse_image` under
|
||||
argmax (0.251 and 0.233) with score profiles matching their API twins almost
|
||||
exactly, and the margin rule agreed pairwise (fisherman meta at margin, sneaker
|
||||
no_ai at margin 0.50, same as its API counterpart). The chat imagine pipeline
|
||||
and `muse-image-1.0` are therefore pixel-indistinguishable to the provider
|
||||
classifier on this pair, so `meta_muse_image` honestly names both
|
||||
distributions. Caveat: n=2 paired prompts from one session; this is a
|
||||
consistency result, not a deployment-scale equivalence claim. Artifact:
|
||||
`meta-chat-check-2026-08-26/`.
|
||||
|
||||
Collapsing OpenAI and Gemini into one pixel class versus other generators
|
||||
does not fix that. Binary ridge AUC 0.686, TPR 75% at FPR 45%. Canva 98%,
|
||||
Microsoft 75%, Firefly 68% leak into the union; FLUX HF hold stays out at
|
||||
3%. Training the same union only against photographs recreates Model 1 with
|
||||
a narrower train set (fresh FPR 2.1%, Firefly still 95%).
|
||||
|
||||
`provider-report-clip-l-ft.json`, `provider-union-report.json`.
|
||||
|
||||
### Strict abstention does not recover provider identity
|
||||
|
||||
An `unknown` class makes the claim narrower, but neither existing
|
||||
representation has a useful high-precision operating point. A
|
||||
leave-one-foreign-family-out ridge on the 124-d residual bank set each cut
|
||||
above every calibration negative. Mean OpenAI recall fell to 1.3% and mean
|
||||
Google recall to 4.4%, while a held-out foreign family was still attributed to
|
||||
a provider up to 5.0% of the time and photographs were called OpenAI up to
|
||||
0.52% of the time.
|
||||
|
||||
The CLIP-L-ft head has the same failure at a 0.1% calibration-negative cut.
|
||||
On 400 disjoint queries per provider, OpenAI recall was 1.0% and Google recall
|
||||
was 0.5%. It still called 1/11 held-out FLUX images OpenAI and 1/750 COCO
|
||||
photographs OpenAI. Zero calibration false positives reduced both provider
|
||||
recalls to zero. These are abstaining content classifiers, not precise source
|
||||
finders.
|
||||
|
||||
A nonlinear head and local-neighbor decision do not expose a hidden provider
|
||||
partition in the same embeddings. Across five balanced 768-128-3 MLP seeds,
|
||||
strict test recall ranged from 0.5-7.0% for OpenAI and 0.75-1.0% for Google,
|
||||
with 6-19 false attributions among 6,550 locked negatives. The best
|
||||
calibration-selected `k=3` local-neighbor rule reached 0.5% OpenAI and 3.25%
|
||||
Google recall and still called one TC260 image Google.
|
||||
|
||||
### Pixel finetuning learns the train split, not an exact provider boundary
|
||||
|
||||
The last two CLIP-L vision blocks were then finetuned directly for
|
||||
`openai` / `google` / `unknown`: 9,063 fit images, 3,537 disjoint calibration
|
||||
images, 400 balanced steps, and random JPEG 40-95, 85-100% crop, and mild blur.
|
||||
Each provider cut was placed above every calibration negative. Calibration
|
||||
recall was 4.3% OpenAI and 5.2% Google.
|
||||
|
||||
The time-disjoint locked result was 6/400 OpenAI and 10/400 Google. One Google
|
||||
image and one TC260 image were called OpenAI. All 500 unseen-AI controls and
|
||||
all 4,945 locked photographs stayed `unknown`, including 3,000 fresh Open
|
||||
Images, but that photo specificity does not repair an AI-source error. An
|
||||
oracle cut above both locked OpenAI errors leaves only 1/400 OpenAI; it is an
|
||||
upper bound, not a valid post-test threshold. The model is not shippable.
|
||||
|
||||
The independent high-frequency route is already closed at the tested
|
||||
capacity. A four-layer opponent-residual patch CNN reached AUC 0.44-0.56
|
||||
against foreign generators, reversed to 0.15 on a fresh era, and accepted
|
||||
95-100% of several held-out Firefly, Microsoft, fal.ai, and PixelBin families
|
||||
at its photo-median threshold. It learned AI rendering versus photography,
|
||||
not vendor identity.
|
||||
|
||||
### External surrogate and forensic-descriptor audit
|
||||
|
||||
The public
|
||||
[`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09)
|
||||
does not supply a causal SynthID contrast. Its negatives are lightly
|
||||
regenerated positive images, so the trained ResNet/EfficientNet ensemble can
|
||||
read the regeneration pipeline. On a blind 517-file local pilot, OpenAI versus
|
||||
all AUC was 0.630. At the repository's 0.5 cut it retained 92/100 OpenAI and
|
||||
accepted 307/417 negatives, including 104/120 Open Images, 26/30 COCO, 34/50
|
||||
Google, and 8/10 Firefly. A later exact repeat on the hash-disjoint v7 challenge
|
||||
retained 172/200 OpenAI but accepted 110/200 Google. It therefore fails source
|
||||
specificity before any photograph gate is considered. It is a visual-domain
|
||||
classifier, not an independent confirmation signal.
|
||||
|
||||
The current [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID/tree/b110836)
|
||||
V4 codebook also adds no useful hybrid evidence. A pickle-free exact inference
|
||||
repeat on v7 accepted 77/200 Google and 76/200 OpenAI at its published 0.52
|
||||
threshold. Applied only to v11 `unknown` rows, that threshold would rescue 26
|
||||
Google files while misrouting two OpenAI files. The older V3 published cut
|
||||
would add two v11 Google misses, but it previously accepted 5/499 controls and
|
||||
6/1,000 fresh Open Images. A 1%-recall OR rule with that measured false-positive
|
||||
history is also rejected.
|
||||
|
||||
The public [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847)
|
||||
SPAM model is another forensic descriptor, not a decoder. The audited pickle
|
||||
contained only an sklearn pipeline, scaler, logistic regression, and numeric
|
||||
numpy globals; inference used an exact restricted allowlist. At the published
|
||||
0.5 cut it accepted 139/200 Google and 141/200 OpenAI. On the disjoint public
|
||||
extension, the same published cut also accepted 214/600 ImageNet photographs,
|
||||
14/75 BigGAN, 17/75 Midjourney, 52/75 SDXL, and 10/75 VQDM. It adds no safe v11
|
||||
rescue.
|
||||
|
||||
[`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad)
|
||||
publishes a useful flat-field corpus but an invalid open-world classifier
|
||||
contrast. Its Tier-1 matrix has 500 Gemini-flat positives and 1,500 synthetic,
|
||||
spectrum-matched, or phase-scrambled negatives, with no real negative. The
|
||||
33 features include noise scale and radial power as well as six hand-selected
|
||||
carrier bins. Rebuilding the repository's seed-42 RBF SVM from the safe numeric
|
||||
matrix, without loading joblib, accepted 1/200 current Google and 0/200 OpenAI
|
||||
on v7. The reported AUC above 0.999 measures the synthetic negative recipe and
|
||||
flat renderer epoch; it cannot confirm the current source finder or a SynthID
|
||||
payload.
|
||||
|
||||
[Forensic Self-Descriptions](https://github.com/ductai199x/Forensic-Self-Descriptions-CVPR25/tree/50f2eae)
|
||||
(CVPR 2025) is a genuinely different representation: constrained prediction
|
||||
residuals are summarized as a 960-d forensic descriptor. Its ready attribution
|
||||
head is not usable here. Three current OpenAI files were all called `Real`,
|
||||
while one Microsoft file was called `GPT-Image 1/1.5` at confidence 0.863.
|
||||
|
||||
A custom source head on a native 256-pixel center crop was more informative.
|
||||
PCA-64 plus logistic regression, with each provider cut above every calibration
|
||||
negative, gave OpenAI AUC 0.872 and 3/50 strict OpenAI recall with 0/388 test
|
||||
false attributions. Its Google head failed on Bytedance, Canva, FLUX,
|
||||
Microsoft, and an Open Images photograph. On a later unused-hash challenge the
|
||||
OpenAI FSD head alone accepted 14/300 OpenAI, 3/300 TC260, and 3/90 PixelBin.
|
||||
FSD is therefore an independent source cue, not a sufficient classifier and
|
||||
not a watermark statistic.
|
||||
|
||||
### Rejected narrow two-signal cascades, 2026-08-23
|
||||
|
||||
The visible Gemini sparkle is absent from the target generation and is not a
|
||||
feature or a gate. Two metadata-free intersections did survive a new challenge:
|
||||
|
||||
| Cascade | Development | Unused-hash challenge |
|
||||
| --- | --- | --- |
|
||||
| Google provider probability >= 0.8876249 **and** registered-v3 lattice >= 1.0 | 3/45 Google, 0/30 lattice-positive foreign files | 3/300 Google, 0/690 OpenAI/TC260/PixelBin |
|
||||
| OpenAI provider probability >= 0.8822824 **and** FSD probability >= 0.9912320 | The two cuts were frozen independently | 3/300 OpenAI, 0/690 Google/TC260/PixelBin, 0/500 photographs |
|
||||
|
||||
The Google result is a low-recall intersection of a content head and an
|
||||
origin-locked renderer lattice. A two-pixel crop destroys the lattice, so an
|
||||
indeterminate result says nothing about SynthID. It is evidence for a narrow
|
||||
Gemini/Imagen pipeline, not a public decoder.
|
||||
|
||||
All three OpenAI challenge positives were 1086x1448 RGB PNGs, the measured
|
||||
ChatGPT export geometry. Thirty known non-OpenAI files at exactly 1086x1448
|
||||
(TC260, Apple, Microsoft, Canva, Firefly, Google, and others) produced zero
|
||||
first-gate positives. A scan of 1,194 unique files at that geometry with no
|
||||
provider or platform label produced 29 pixel-gate candidates and four final
|
||||
hybrid candidates. Three had only PNG image-data chunks; the fourth carried
|
||||
generic DPI/orientation fields but no provider or provenance label. Visual
|
||||
inspection found four plausible AI generations. They remain candidates, not
|
||||
ground truth, until a held-out oracle or original source confirms them.
|
||||
|
||||
The full luma 16-32-pixel phase-ring head is not a third confirmation. It
|
||||
rejected all three known and all four unlabeled OpenAI hybrid candidates while
|
||||
accepting 3/30 known non-OpenAI same-geometry controls. Its global test AUC was
|
||||
0.600 for OpenAI and it stays closed.
|
||||
|
||||
These cascades established that independent intersections could suppress the
|
||||
observed false positives, but about 1% recall is not useful for the source-hunt
|
||||
task. They are rejected as operating detectors. The FSD implementation and
|
||||
weights are also research-only under CC BY-NC-SA 4.0.
|
||||
|
||||
Local artifacts: `lattice-pixel-hybrid-challenge.json`,
|
||||
`fsd-openai-challenge-report.json`,
|
||||
`unlabeled-1086x1448-openai-hybrid-candidates.json`. The research entry point
|
||||
is `.local-eval/synthid/ai-photo-2026-08-22/hybrid_source_finder.py`.
|
||||
|
||||
### Canonical forensic source finder, 2026-08-23
|
||||
|
||||
An overlooked pixel representation materially improves the strict result. It
|
||||
normalizes each file to 512x512, then extracts 1,794 metadata-free features
|
||||
from opponent-color residuals: patch power and variation, phase coherence,
|
||||
autocorrelation, low-bit correlations, and residual-error histograms. The
|
||||
OpenAI branch intersects its canonical forensic score with a weak provider
|
||||
pixel vote. The Gemini branch intersects a forensic/provider fusion with the
|
||||
registered-v3 lattice. The visible Gemini sparkle remains excluded.
|
||||
|
||||
The single-scale cuts were frozen before a third hash-disjoint challenge. The
|
||||
result was:
|
||||
|
||||
| Challenge cell | OpenAI candidate | Gemini candidate | Any candidate |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| OpenAI | 51/300 | 1/300 | 52/300 (17.3%) |
|
||||
| Google | 0/300 | 34/300 | 34/300 (11.3%) |
|
||||
| Open Images photographs | 1/1,000 | 0/1,000 | 1/1,000 (0.1%) |
|
||||
| TC260 | 0/7 | 0/7 | 0/7 |
|
||||
|
||||
For the requested union, 86/600 OpenAI-or-Google targets were found, 14.3%
|
||||
recall instead of the rejected cascades' 1%. One OpenAI file crossed through
|
||||
the Gemini branch, and one known photograph crossed through the OpenAI branch.
|
||||
The seven remaining unused TC260 files are too few to establish specificity,
|
||||
and no fresh PixelBin, Firefly, Microsoft, or other generator hashes remained
|
||||
outside the earlier model and challenge manifests.
|
||||
|
||||
This v3 result established a strict source-candidate finder, not an exact
|
||||
identifier or a SynthID decoder. It does not recover a payload, and an
|
||||
`unknown` result says nothing about whether SynthID is present.
|
||||
|
||||
Local artifacts: `source-finder-v3-selection.json`,
|
||||
`source-finder-v3-challenge.json`, `google-canonical-forensic.json`, and
|
||||
`google-forensic-pixel-fusion.json`. The same local research entry point now
|
||||
implements this rule without the FSD dependency.
|
||||
|
||||
### Multiscale strict finder, 2026-08-24
|
||||
|
||||
The 512x512 normalization was discarding usable scale-specific evidence. A
|
||||
frozen union now takes the strongest OpenAI forensic response over 512, 640,
|
||||
768, and native views, gated by the provider pixel head. The Google branch
|
||||
takes the strongest forensic/provider fusion over 384, 512, 640, 768, and
|
||||
native views, then requires the same registered-v3 lattice. Development on
|
||||
the second and third challenges found 539/1,200 targets and 0/1,657
|
||||
photograph/TC260 controls.
|
||||
|
||||
The views and cuts were written before opening a fourth challenge whose hashes
|
||||
were disjoint from every model manifest and the first three challenges:
|
||||
|
||||
| Challenge cell | OpenAI candidate | Gemini candidate | Any candidate |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| OpenAI | 92/300 | 11/300 | 103/300 (34.3%) |
|
||||
| Google | 0/300 | 173/300 | 173/300 (57.7%) |
|
||||
| Open Images photographs | 0/1,000 | 0/1,000 | 0/1,000 |
|
||||
| TC260 | 1/25 | 0/25 | 1/25 |
|
||||
|
||||
For the requested union, the blind result is 276/600, 46.0% recall, with
|
||||
1/1,025 non-target candidates. This is 3.2 times the single-scale v3 recall
|
||||
and 46 times the rejected 1% cascades. The one false candidate is TC260, not a
|
||||
camera photograph. Eleven OpenAI files crossed through the Gemini branch;
|
||||
that is a provider-attribution error but still a correct hit for the declared
|
||||
OpenAI-or-Google union.
|
||||
|
||||
This remains a source-candidate finder, not an exact identifier or a SynthID
|
||||
decoder. Fresh unused paths from the other generator families were not
|
||||
available for v4, so the 0.1% observed non-target rate is not an open-world
|
||||
precision claim. Robustness to crop, resize, re-encoding, and screenshot
|
||||
capture is also not established. Keep the models and paths in `.local-eval`;
|
||||
do not add a runtime or public CLI until a new temporal challenge with fresh
|
||||
foreign-generator families establishes positive precision.
|
||||
|
||||
Local artifacts: `source-finder-v4-selection.json`,
|
||||
`source-finder-v4-rule.json`, `source-finder-v4-challenge.json`, and
|
||||
`multiscale-forensic-development.json`. The local research entry point
|
||||
implements the frozen multiscale rule and still uses no metadata or visible
|
||||
sparkle.
|
||||
|
||||
A post-hoc OR over every per-view zero-development-error OpenAI cut is
|
||||
rejected. It raised v4 OpenAI recall to 153/300 but also accepted 6/1,000
|
||||
photographs and 3/25 TC260 controls. The apparent union of many individually
|
||||
strict cuts was multiple-testing overfit, not additional independent evidence.
|
||||
|
||||
### Original-export hybrids, 2026-08-24
|
||||
|
||||
Three more hash-disjoint challenges tested whether multiscale fusion could be
|
||||
made useful without metadata. The v5 ExtraTrees union improved exact provider
|
||||
recall to 363/600 (60.5%) and provider-union recall to 373/600 (62.2%), but it
|
||||
also accepted 8/1,000 photograph and foreign-generator controls. A revised
|
||||
Google confirmation removed those eight development errors. Adding an
|
||||
AI-versus-camera gate in v6 did not transfer: exact recall fell to 327/600
|
||||
(54.5%) and union recall was 341/600 (56.8%).
|
||||
|
||||
The specificity failures exposed a stronger but narrower signal. Current
|
||||
OpenAI exports in these sets are PNGs produced with adaptive scanline filters,
|
||||
while the earlier TC260 error was a PNG encoded with filter zero on every row.
|
||||
A strict PNG parser now requires a non-interlaced PNG with at least one adaptive
|
||||
filter before the OpenAI branch can emit a result. This reads the image container
|
||||
and pixels, not EXIF, C2PA, a filename, or a visible label. It also changes the
|
||||
claim: a re-encoded OpenAI JPEG must abstain.
|
||||
|
||||
The complete frozen v7 rule reached 215/400 exact provider matches (53.8%) and
|
||||
221/400 provider-union matches (55.3%) on a new challenge. Its cells were
|
||||
114/200 exact OpenAI and 101/200 exact Google. The PNG gate repaired the
|
||||
observed specificity problem, but the old OpenAI forensic head remained the
|
||||
recall bottleneck.
|
||||
|
||||
A subsequent v8 development hybrid trains an ExtraTrees OpenAI head on v4-v5
|
||||
multiscale forensic scores, pixel probabilities, and PNG encoding structure.
|
||||
Model selection used v6. The final 0.47 precision cut was chosen after v7 had
|
||||
been opened, so the following is a post-hoc development measurement, not
|
||||
another blind result:
|
||||
|
||||
| v7 cell under v8 development rule | OpenAI | Gemini | Unknown |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| OpenAI | 190/200 | 3/200 | 7/200 |
|
||||
| Google | 0/200 | 102/200 | 98/200 |
|
||||
|
||||
That is 292/400 exact provider matches (73.0%) and 295/400 provider-union
|
||||
matches (73.8%).
|
||||
|
||||
The remaining Google miss set contained two different export pipelines: PNG
|
||||
and JPEG. A second development branch parses only JPEG codestream parameters,
|
||||
including quantization tables, chroma sampling, and progressive encoding; it
|
||||
explicitly skips APP0-APP15 and COM segments. Training one Google model per
|
||||
encoding class on v4 and selecting zero-validation-error cuts on v5-v6 raised
|
||||
the v11 transfer result to:
|
||||
|
||||
| v7 cell under v11 development rule | OpenAI | Gemini | Unknown |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| OpenAI | 190/200 | 4/200 | 6/200 |
|
||||
| Google | 0/200 | 126/200 | 74/200 |
|
||||
|
||||
This is 316/400 exact provider matches (79.0%) and 320/400 provider-union
|
||||
matches (80.0%).
|
||||
|
||||
This is the best local source finder in the campaign, but it is still not a
|
||||
SynthID detector, payload decoder, or open-world precision proof. The v8 rule
|
||||
is post-hoc, and neither v8 nor v11 has an independent open-world negative
|
||||
proof. The OpenAI branch is intentionally scoped to original-style PNG exports.
|
||||
A new temporal blind challenge with new foreign generators and PNG
|
||||
camera/editor controls is required before a runtime or public CLI is justified.
|
||||
|
||||
Local artifacts: `source-finder-v7-selection.json`,
|
||||
`source-finder-v7-challenge.json`, `source-finder-v8-rule.json`,
|
||||
`source-finder-v8-openai-extra-trees.joblib`, and
|
||||
`source-finder-v11-google-per-codec.joblib`.
|
||||
|
||||
### Published few-shot attribution also fails the open-world gate
|
||||
|
||||
[OmniDFA](https://arxiv.org/abs/2509.25682) is a purpose-built few-shot source
|
||||
attributor rather than a generic content embedding. Its published `part1`
|
||||
checkpoint is the correct unseen-generator fold for DALL-E 2 and DALL-E 3:
|
||||
those generators are in `part1` validation and absent from its training list.
|
||||
The same checkpoint has seen Imagen, so its Google result is not a clean
|
||||
unseen-Imagen benchmark; the OpenAI result is sufficient to reject the shared
|
||||
runtime.
|
||||
|
||||
With 20 support images per provider and provider-specific similarity plus
|
||||
margin cuts calibrated to zero false attributions over 160 negatives, a
|
||||
content-hash-disjoint 745-image evaluation produced:
|
||||
|
||||
| Cell | Result |
|
||||
| --- | ---: |
|
||||
| OpenAI recall | 9/50 (18%) |
|
||||
| Google recall | 4/50 (8%) |
|
||||
| Microsoft called OpenAI | 3/15 (20%) |
|
||||
| Kodak called OpenAI | 3/24 (12.5%) |
|
||||
| Canva called Google | 1/15 (6.7%) |
|
||||
| fal.ai called OpenAI | 1/15 (6.7%) |
|
||||
| xAI called OpenAI | 1/15 (6.7%) |
|
||||
| unseen Higgsfield called OpenAI | 1/11 (9.1%) |
|
||||
| fresh Open Images / COCO false attributions | 0/100 / 0/50 |
|
||||
|
||||
Provider multimodality is not the missing fix. Choosing 1-10 spherical
|
||||
prototypes only by calibration recall selected five: test recall fell to 16%
|
||||
OpenAI and 4% Google, while false attributions remained on Firefly (2/15),
|
||||
Kodak (2/24), Microsoft, ByteDance, TC260, and Made-with-AI samples.
|
||||
|
||||
Native files already fail, so JPEG, resize, crop, and screenshot variants were
|
||||
not run for OmniDFA. Do not add a provider-attribution runtime or CLI from that
|
||||
model. General exact OpenAI/Gemini identification remains unsupported. The
|
||||
strict source finder above emits candidates; it does not read the SynthID
|
||||
payload.
|
||||
|
||||
### 124-d lattice as pipeline ID, not a vendor CLIP head
|
||||
|
||||
Provider-class ridge on 124 native residual features (70/30 once, not a
|
||||
watermark gate). OpenAI L1 n=285, Google corpus n=533, foreign n=218, COCO
|
||||
n=289: OpenAI vs COCO 0.965; Google vs COCO 0.999; OpenAI vs Google 0.989;
|
||||
OpenAI vs foreign 0.725; Google vs foreign 0.922. OpenAI vs Firefly-class
|
||||
is the weak cell.
|
||||
|
||||
One-vs-rest: Google head at TPR 90% has FPR 0% vs COCO, 14% vs foreign, 2%
|
||||
vs OpenAI. A Gemini-like pixel class is close to what `pipeline_lattice`
|
||||
already is. An OpenAI-like pixel class on this bank would label Firefly as
|
||||
OpenAI about half the time and is not shippable.
|
||||
|
||||
Three-class `openai` / `google` / `no_ai` on 2,000 catalog OpenAI, 2,000
|
||||
catalog Google, and 1,936 COCO photos. Photo-first margin 0.50: openai
|
||||
74.7%, google 78.9%, no_ai 99.8%; Kodak 24/24 `no_ai`. Other generators
|
||||
are leakage, not classes:
|
||||
|
||||
| Platform | n | openai | google | no_ai |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Firefly | 106 | 37 | 27 | 42 |
|
||||
| Microsoft | 117 | 39 | 22 | 56 |
|
||||
| PixelBin | 90 | 14 | 46 | 30 |
|
||||
| HuggingFace job | 82 | 3 | 62 | 17 |
|
||||
| ByteDance C2PA | 86 | 5 | 35 | 46 |
|
||||
| SD / Comfy | 120 | 30 | 11 | 79 |
|
||||
| fal.ai | 98 | 11 | 18 | 69 |
|
||||
| Made-with-AI tag | 115 | 4 | 32 | 79 |
|
||||
| TC260 | 118 | 1 | 13 | 104 |
|
||||
| xAI | 114 | 1 | 20 | 93 |
|
||||
| Canva | 76 | 11 | 3 | 62 |
|
||||
| Apple Clean Up | 114 | 1 | 23 | 90 |
|
||||
| Aweme | 37 | 2 | 7 | 28 |
|
||||
| FLUX | 11 | 0 | 0 | 11 |
|
||||
| Reve | 10 | 0 | 0 | 10 |
|
||||
| NovelAI | 9 | 0 | 0 | 9 |
|
||||
| Higgsfield | 11 | 1 | 5 | 5 |
|
||||
|
||||
PixelBin and HuggingFace jobs lean `google` (shared renderer lineage).
|
||||
FLUX, NovelAI, and Reve stay `no_ai`. Local probe:
|
||||
`uv run python .local-eval/synthid/prc-oklab-attack-2026-08-15/classify_openai_gemini.py image.png`.
|
||||
|
||||
## Research lattice expert (google-lineage renderer)
|
||||
|
||||
Not a watermark and not in `identify`. `scripts/synthid_runtime/`
|
||||
`detect_synthid` re-check on 628 frozen holdouts, seed 20260822, threshold
|
||||
1.0.
|
||||
|
||||
| Family | n | detected | rate | max score |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| Google / Gemini | 80 | 45 | 0.56 | 3.03 |
|
||||
| Firefly | 84 | 15 | 0.18 | 3.00 |
|
||||
| PixelBin | 80 | 11 | 0.14 | 2.48 |
|
||||
| Microsoft | 60 | 2 | 0.03 | 2.40 |
|
||||
| OpenAI | 80 | 1 | 0.01 | 1.38 |
|
||||
| xAI | 40 | 1 | 0.03 | 1.26 |
|
||||
| FLUX HF | 40 | 0 | 0 | 0.85 |
|
||||
| TC260 | 40 | 0 | 0 | 0.97 |
|
||||
| Kodak | 24 | 0 | 0 | 0.49 |
|
||||
| Open Images fresh | 60 | 0 | 0 | 0.64 |
|
||||
| COCO hold | 40 | 0 | 0 | 0.74 |
|
||||
|
||||
Firefly 18% and PixelBin 14% match the 2026-08-16 signed-foreign rates
|
||||
(24% and 14%) in order of magnitude. Both Microsoft hits have issuer
|
||||
`Microsoft, Google LLC`. A 2 px crop killed every sampled positive,
|
||||
including Firefly and PixelBin. Google TPR 56% is mixed collection eras, not
|
||||
the oracle-positive 147/148 cell. Honest name:
|
||||
`google_lineage_renderer` = Gemini/Imagen + Firefly + PixelBin.
|
||||
|
||||
Registered-v3 photographic controls remain 0/5,993 Open Images and 0/2,366
|
||||
COCO. Against 223 C2PA-named non-Google generators on 2026-08-16: 29
|
||||
accepted (0.130), Firefly 0.241. `.local-eval/synthid/lattice-check-2026-08-22/`.
|
||||
|
||||
## Local collection note (2026-08-21)
|
||||
|
||||
Unlabeled rows in the local C2PA-labeled collection are not photographs.
|
||||
Microsoft and Firefly rows also carry `synthid_from_provenance=true`, so
|
||||
that flag is not an OpenAI-plus-Gemini class.
|
||||
@@ -0,0 +1,591 @@
|
||||
# SynthID local detector research
|
||||
|
||||
> Research archive for the hunt for a local, keyless SynthID pixel detector.
|
||||
> Not a statement of current product capability. Shipped behavior:
|
||||
> [supported signals](supported-signals.md) and
|
||||
> [known limitations](known-limitations.md).
|
||||
>
|
||||
> Sister pages: [SynthID source classifiers](synthid-classifiers.md),
|
||||
> [SynthID mark removal](synthid-removal-research.md),
|
||||
> [mechanism reference](synthid.md). Dated measurements live in the
|
||||
> [chronological plan](synthid-detector-removal-plan.md).
|
||||
|
||||
## Result
|
||||
|
||||
The hunt for a local, keyless OpenAI SynthID pixel detector is open.
|
||||
Google has no public payload decoder, and nothing in this project
|
||||
reproduces one. Closed routes below failed to read the official oracle
|
||||
contrast on photographs. The remaining work is a new waveform, a new
|
||||
pair mint, or a new statistic, not a rerun of those routes.
|
||||
|
||||
The mark behaves like a keyed spread-spectrum residual: a content-adaptive
|
||||
`x' = x + g(x)` with a paired decoder (DeepMind patent family, optional
|
||||
secret in intermediate layers, encoder/decoder ensembles that do not
|
||||
recognize each other). Without that pair, the residual sits under the
|
||||
scene. Keyless energy, TrustMark transfer, L1 distillation, a flat-field
|
||||
stamp, and a 16-32 band student all failed to read the official oracle
|
||||
contrast on photographs.
|
||||
|
||||
S4, 2026-08-15: the two providers are not doing the same thing. Cross-image
|
||||
correlation of the folded residual is high for Google (tile16 pos-pos
|
||||
`+0.326`, chance `0.036`) and at chance for OpenAI (`+0.032`). Google
|
||||
shares one fixed phase-coherent pattern, also present in its controls at
|
||||
about half the amplitude. OpenAI shares nothing, which is what a
|
||||
content-dependent post-hoc encoder produces. Comb experts that work on
|
||||
Google therefore cannot be reused as an OpenAI watermark detector.
|
||||
|
||||
M2, 2026-08-15, re-verified 2026-08-16: Google unwatermarked pairs cannot
|
||||
be minted. `imagen-*` ids 404 from the model garden. `gemini-*-image`
|
||||
rejects `addWatermark` (`Cannot find field`). Current Gemini API docs
|
||||
state that all generated images include a SynthID watermark. There is no
|
||||
encoder-off Google path.
|
||||
|
||||
What the product uses for the *watermark* is signed provenance and
|
||||
`verify-openai-synthid`. The periodic-lattice expert is research-only under
|
||||
`scripts/synthid_runtime/` and is not called from `identify` or the CLI.
|
||||
Lineage measurements of that expert are in
|
||||
[SynthID source classifiers](synthid-classifiers.md).
|
||||
|
||||
## Public GitHub sweep, 2026-08-24
|
||||
|
||||
GitHub's primary-repository search returned 78 repositories with `synthid` in
|
||||
the name and 81 with it in the description, 133 unique repositories in total.
|
||||
The fork-inclusive searches were much larger and capped or dominated by copies
|
||||
(860 name hits and 1,311 description hits), so this is not a claim that every
|
||||
fork or unindexed private repository was read. The sweep also used targeted
|
||||
code searches for detector, verifier, correlation, codebook, bypass, and
|
||||
removal implementations. SynthID-Text, visible-corner-mark removers, metadata
|
||||
cleaners, wrappers, and literal ports were classified before inspecting the
|
||||
remaining image-pixel candidates at pinned commits.
|
||||
|
||||
The audits are not null results. They separate three claims that must not be
|
||||
collapsed: a repeatable image statistic exists, that statistic is a calibrated
|
||||
single-image SynthID detector, and changing its score transfers to the
|
||||
provider's production decoder. The repositories support parts of the first
|
||||
claim. None supports the second or third on the current challenge data.
|
||||
|
||||
| Repository | Retained contribution | Rejected use | Decisive result |
|
||||
| --- | --- | --- | --- |
|
||||
| [`Rinne414/SynthID-detector`](https://github.com/Rinne414/SynthID-detector/tree/b08abff06b67c37db9dfa7c6ed63760a446d560f) | A reproducible fixed-residual correlator and a useful spoofability negative control | Current-image detector or fixed-template remover | All three templates detected 0/400 current OpenAI and Google images. Adding the expected template spoofed the public detector, while subtracting 0.25-2x from an official positive left SynthID `detected` down to 25.72 dB |
|
||||
| [`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09c62dc26112f9a0e6cfab469f9a10f37e3) | Evidence that a CNN can learn a renderer/regeneration-domain signature, plus a white-box shortcut baseline | Payload decoder, independent hybrid confirmation, or removal oracle | The published cut retained 172/200 OpenAI but accepted 110/200 Google. A gradient drove the surrogate effectively to zero, but official SynthID stayed `detected` through +/-8/255 at 34.30 dB |
|
||||
| [`aloshdenny/reverse-SynthID`](https://github.com/aloshdenny/reverse-SynthID/tree/b11083676fd3ee3ff97ce9d03c0e409e46905902) | Cross-color phase consensus and further evidence for a repeating Google renderer-lineage pattern | The published V3/V4 thresholds as presence detectors, or Round-06 as a verified quiet remover | V4 accepted 77/200 Google and 76/200 OpenAI at the same cut. Round-06 has no published per-image oracle evidence and uses regeneration/distortion with 14/11 dB PSNR floors |
|
||||
| [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847) | A compact 147-d second-order SPAM descriptor and a safely auditable sklearn model | Google or OpenAI detector, or a high-precision hybrid rescue | The published cut accepted 139/200 Google, 141/200 OpenAI, and 214/600 disjoint ImageNet photographs |
|
||||
| [`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad) | A public 2026 Gemini flat-field corpus, safe numeric features, and explicit candidate carrier bins | Its reported near-perfect Tier-1 AUC as open-world SynthID evidence | Training used 500 Gemini flats against 1,500 synthetic or phase-scrambled negatives and zero real photographs. Rebuilding the SVM from the safe matrix accepted 1/200 Google and 0/200 OpenAI on v7 |
|
||||
|
||||
Useful forensic signal is therefore not synonymous with a production
|
||||
watermark decoder. The Google lattice idea is retained only as a
|
||||
research-lineage component; it cannot be labelled SynthID. The fixed template
|
||||
and CNN gradient are retained as negative controls. None of these public
|
||||
implementations is shipped as a detector or remover. Detailed classifier and
|
||||
attack measurements live in [SynthID source classifiers](synthid-classifiers.md) and
|
||||
[mark removal](synthid-removal-research.md).
|
||||
|
||||
### Google verifier surfaces found in the sweep
|
||||
|
||||
Two real but non-local verifier paths surfaced. The historical Vertex SDK
|
||||
exposed `WatermarkVerificationModel.from_pretrained("imageverification@001")`.
|
||||
The class still exists in `google-cloud-aiplatform` 1.165.1, but warns that it
|
||||
was deprecated 2025-06-24 with removal scheduled for 2026-06-24. Calls in two
|
||||
projects with Vertex enabled now reject the publisher model id as invalid. A
|
||||
repository using `response.watermark_detected` is also stale: the actual SDK
|
||||
property was `watermark_verification_result`, backed by the prediction's
|
||||
`decision` field.
|
||||
|
||||
A newer unlisted upload route,
|
||||
`POST https://synthid.googleapis.com/upload/v1:verifyContent`, is live enough
|
||||
to identify itself as the SynthID API and return `SERVICE_DISABLED` for a real
|
||||
public image. It has no public discovery document, did not appear in the
|
||||
project's available-services listing, and no authorized successful response
|
||||
was obtained. Enabling a cloud service was deliberately not attempted. This
|
||||
may be an access-gated research oracle, but it is not a local decoder and does
|
||||
not satisfy the metadata-free offline product requirement. Gemini chat remains
|
||||
the only Google verifier actually evidenced by the public bypass corpora.
|
||||
|
||||
## Closed detector routes
|
||||
|
||||
| Route | Close | Why |
|
||||
| --- | --- | --- |
|
||||
| Wavelets / FFT / cepstrum as a single-image detector | 2026-08-09 | TrustMark proxy: 318-d wavelet/spectral summary AUC 0.653 and 0 TPR at a clean calibration cut. Complex FFT maps AUC 0.516. Spatial RGB still won |
|
||||
| `aloshdenny/reverse-SynthID` V3 phase codebook | 2026-08-09/24 | Pickle-free numeric audit. The original pilot was 5/5 Google positives and 0/194 then-available negatives. The broader audit was 5/355 Google, 5/499 controls, and 6/1,000 fresh Open Images. On v7 it accepted 4/200 Google and 1/200 OpenAI. As a v11 `unknown` rescue it adds 2/200 Google on this test, but the historical control errors reject a 1%-recall OR rule |
|
||||
| `aloshdenny/reverse-SynthID` V4 | 2026-08-13/24 | Commit `b110836`. Better-of-two profiles: 141/355 Google positives and 191/499 controls. Frozen 1,000 Open Images: 386 accepted. Paired AUC 0.517. An exact repeat on the hash-disjoint v7 challenge accepted 77/200 Google and 76/200 OpenAI. Strict GUI aspect routing reduced support but not false acceptance. The threshold sits 0.02 above chance phase similarity |
|
||||
| `cebeuq/Synthid-Bypass` as a local decoder | 2026-08-13 | 12 public before/after diffusion pairs. Fixed carrier 10/12 before, 0/12 after. Gemini Verify still saw SynthID on two `before` files the carrier missed. Reconstruction removal, not a payload read |
|
||||
| S4: one content-independent pattern for both providers | 2026-08-15 | Google has a shared phase-coherent tile. OpenAI is at chance. See Result |
|
||||
| M2: mint Google encoder-off pairs | 2026-08-15 | No current Google path emits an unwatermarked image. See Result |
|
||||
| U1: one feature space for both providers | 2026-08-15 | Transfer 0.535 and 0.469 |
|
||||
| Hand statistics vs a perceptually masked keyed mark (M1b) | 2026-08-15 | At equal RMS, fixed 0.670, keyed 0.669, keyed+masked 0.546. Masking, not keying, defeats hand features |
|
||||
| Decoder-substitution / foreign VAE as a verified negative | 2026-08-17 | OpenAI oracle stayed `detected` at 22.3 dB. Attacking a positive does not mint a labelled negative |
|
||||
| `aoxo/reverse-synthid` as extra Google templates | 2026-08-19 | Hugging Face dump scored against the frozen era bank. Extra family means raised foreign scores instead of isolating a mark |
|
||||
| Origin-locked lattice as SynthID | 2026-08-16 | Two-pixel crop kills it; published SynthID keeps 99.97% TPR under aggressive crop. See classifiers for `google_lineage_renderer` |
|
||||
| Keyless energy in 16-32 px | 2026-08-20 | L1 AUC 0.53; official decoder reads phase structure, not energy |
|
||||
| TrustMark / open-encoder transfer | 2026-08-20 | E3 leave-one-encoder-out at chance on TrustMark (0.505 ridge, 0.510 conv) |
|
||||
| L1 distillation of oracle bits | 2026-08-20 | Geometry-only AUC 0.78-0.83 beats pixel 124-d (0.64). ChatGPT export and `opened` C2PA predict `not_detected`. Inside 1254x1254, permutation p=0.45 |
|
||||
| Flat 16-32 matched filter | 2026-08-21 | Leave-one-out residual correlation 0.645 on gray flats; 0.007-0.025 on COCO photo residuals. L1 AUC 0.70 was a size confound (size-matched 0.59 / 0.44) |
|
||||
| `gpt-image-1` as encoder-off pair | 2026-08-21 | Does not stamp. `gpt-image-2` does |
|
||||
| CNN on raw L1 bits | 2026-08-21 | Do not train. Labels are export geometry and presentation, not the mark |
|
||||
| 128-photo causal wavelet student on the flat template | 2026-08-21 | Photo 16-32 jam flips only at a=24 / 18.6 dB (`a*_photo < 0.53` failed). A student cannot beat a matched filter if the photo-domain mark is that quiet |
|
||||
| Photo 16-32 amplitude without the flat-arm template | 2026-08-23 | 11 COCO `gpt-image-2` edits plus the red-dot pair. Residual is the camera JPEG resized onto the edit raster, top-left 32 px dropped. Median 16-32 residual RMS 12.2, same order as 8-16 (9.0) and 32-64 (16.3). On the five tightest pairs (full residual RMS < 16): 16-32 median 2.69, still ~24x the 0.11 flat stamp and not louder than 8-16 (3.81). The photo residual does not isolate a 16-32 codeword |
|
||||
| Sort-by-luma 1D chroma stego | 2026-08-23 | Flat LOO 0.914 but a luma-order shuffle is already 0.792. COCO plus tiny noise: 0.642 vs shuffle 0.640. The sort invents the correlation |
|
||||
| Retinex / local-gain then 16-32 LOO | 2026-08-23 | 0.742 vs plain 16-32 LOO 0.752 on the same eight gray flats. No new carrier |
|
||||
| Distant 16-32 patch coupling | 2026-08-23 | Near lag 0.13 of a tile: -0.013. Far lag ~500 px: +0.054. No nonlocal lock |
|
||||
| Bispectrum of the 16-32 ring | 2026-08-23 | Degenerate: 1.0 on flats and 1.0 on COCO. Not a statistic |
|
||||
| Length-255 m-sequence vs 16x16 fold | 2026-08-23 | Max abs corr 0.059. Chance for n=255 is `1/sqrt(255) ≈ 0.063` |
|
||||
| Row-as-audio pitch peak | 2026-08-23 | Peak-to-mean 51 on flats, 46 on COCO noise residuals. Shared FFT peakiness, not a pitch |
|
||||
| 16x16 origin search of the flat 16-32 template on photos | 2026-08-23 | Zero-shift NCC 0.006-0.024, best of 256 shifts 0.013-0.026. Origin is not hiding a lock |
|
||||
| Affine search of the 16-32 residual vs the flat template | 2026-08-23 | 11 `gpt-image-2` photos: zero-deg median 0.009, best rotation in ±4° 0.020, lift 0.011. Six COCO noise residuals: lift 0.008. Flats already peak at 0°. No hidden rotation or 3% scale lock |
|
||||
| Fourier-angle scramble of 16-32 (true `(r, θ)` of the annulus) | 2026-08-23 | s1 24.1 dB official `not_detected`, cartesian replicate 24.8 dB `not_detected`. Decoder reads phase in that annulus, including angular organization. Not quieter than scramble |
|
||||
| Baker-map of the 16-32 band | 2026-08-23 | s1 27.8 dB still `detected` |
|
||||
| Poisson / photon-shot embedding | 2026-08-23 | Flat Anscombe LOO 0.7518 vs linear 0.7519. s1 Poisson 30.7 dB still `detected` |
|
||||
| Nested LSB plus SynthID | 2026-08-23 | s1 55.9 dB still `detected`. Extra open stego does not hide or replace the mark |
|
||||
| ICC sRGB rewrite | 2026-08-23 | No-op, PSNR inf. Mark is not in a profile that a pixel-preserving assign changes |
|
||||
| Seam carve 8 columns of 16-32 energy | 2026-08-23 | s1 27.7 dB still `detected` |
|
||||
| Extra OpenAI image ids as a new ensemble | 2026-08-23 | `gpt-image-1.5` and `gpt-image-1-mini` are encoder-off: C2PA `created, converted`, official `not_detected`, 16-32 corr vs the `gpt-image-2` gray stamp 0.003. Dated `gpt-image-2-2026-04-21` is the same family: C2PA `watermarked.unbound`, 16-32 corr 0.189. `chatgpt-image-latest` generate stays 403 until org verification |
|
||||
| DIRE reconstruction error as a payload | 2026-08-23 | SDXL float32, 512 px. VAE RMS: gray 1.24, s1 9.15, photo edit 9.89, camera 11.84. DDIM DIRE RMS: gray 2.40, photo 30.9, s1 31.5, camera 33.0. Rank follows texture. Float16 DDIM NaN'd on MPS. Not a payload |
|
||||
| HEIF / AV1 as a hidden carrier | 2026-08-23 | s1 HEIF q80 46.3 dB, q50 39.3 dB, AV1 CRF 32 37.2 dB, all official `detected`. Codec round-trip does not mint a new statistic |
|
||||
| Print-scan simulation as the kill | 2026-08-23 | s1 24.95 dB still `detected`. Same PSNR class as 16-32 scramble, which kills. Generic 25 dB is not the mark |
|
||||
| Mellin / log-polar \|F\| of the 16-32 residual | 2026-08-23 | Flat LOO 0.996-1.000, photo 0.996, COCO 0.991-0.995. The annulus envelope, not a payload. Scale chirps `r^{iα}`: flat 0.51, photo -0.27, COCO -0.58. No photo lock |
|
||||
| Zernike n≤6 and PCET of the residual | 2026-08-23 | Flat LOO 0.77 / 0.75, same order as cartesian 16-32. Photo -0.007 / 0.044. COCO 0.23 / 0.35, above photos. Moments of the flat stamp, not a photograph decoder |
|
||||
| Fresnel quadratic-phase of 16-32 (z=8,16,32,64) | 2026-08-23 | Flat LOO 0.750-0.753, photo 0.012, COCO -0.006. Unitary copy of the cartesian residual. No new shell |
|
||||
| 8-bin Fourier orientation energy | 2026-08-23 | 0.999 flat, 0.966 photo, 0.995 COCO. Saturates |
|
||||
| Chroma 16-32 as the payload | 2026-08-23 | Gray-flat Y LOO 0.682, Cb 0.064, Cr 0.025. s1 Y scramble 24.6 dB `not_detected`; Cb 45.0 dB and Cr 43.9 dB stay `detected`. Mark is in luma |
|
||||
| One 90° Fourier sector of 16-32 | 2026-08-23 | s1 27.5 dB still `detected`. Decoder needs the whole annulus, not one wedge |
|
||||
| Radial-phase-only scramble of 16-32 | 2026-08-23 | s1 25.0 dB `not_detected`. Phase as a function of radius, over the full ring, is enough to kill. Combined with Fourier-angle scramble, either polar coordinate of the annulus phase is a kill if the whole ring is hit |
|
||||
| [`Rinne414/SynthID-detector`](https://github.com/Rinne414/SynthID-detector/tree/b08abff06b67c37db9dfa7c6ed63760a446d560f) fixed residual template | 2026-08-23/24 | The repository reports 0/33 held-out AI images and identifies its original result as self-correlation. Its pinned pre-May GPT-Image2 template first detected 0/11 current source-matched edits and 0/11 COCO sources. An exact repeat on 200 current OpenAI and 200 current Google images produced zero detections for all three published templates at the 0.08 cut; maximum GPT-Image2 score on OpenAI was 0.0167. Adding the expected template can spoof the public correlator, but that template is not present in held-out content. Independent repeat of the rejected fixed-template route, not a payload decoder |
|
||||
| [`newideas99/gpt-image-synthid-detector`](https://github.com/newideas99/gpt-image-synthid-detector/tree/5495e09) CNN ensemble | 2026-08-24 | Exact frozen inference retained 172/200 OpenAI but accepted 110/200 Google at the published 0.5 cut. The training negatives are regenerated positives, while its confound probe transforms only positives and has no foreign-image control. A full-frame white-box attack drove its probability from 0.99995 to 0.25956 at ±2/255 and effectively zero at ±4/255, but the official OpenAI decoder stayed `detected` through ±8/255. Renderer/confound classifier, not a mark decoder |
|
||||
| [`Ristellise/REGRET`](https://github.com/Ristellise/REGRET/tree/7d449034bf323987e7e608e7886e029ed20fd847) SPAM model | 2026-08-24 | At the published cut, exact model inference accepted 139/200 Google, 141/200 OpenAI, 214/600 disjoint ImageNet photographs, and substantial fractions of foreign generators. The apparent high-precision tail did not transfer |
|
||||
| [`vordme2010/synthid-dataset`](https://github.com/vordme2010/synthid-dataset/tree/133a27088f6f4d695c79db9a1a70fa8e7fa3adad) Tier-1 SVM | 2026-08-24 | Rebuilt without loading joblib, using the repository's 2,000x33 numeric matrix, reference phases, noise template, seed-42 split, scaler, and RBF SVM. It accepted 1/200 current Google and no current OpenAI rows. Its published AUC above 0.999 distinguishes Gemini flats from generated derivatives, not current photographs |
|
||||
| Generic GitHub heuristics and literal ports | 2026-08-24 | `hackerfactor/reverse-SynthID-C`, `BIRSAx2/ripmark`, and `xiaoyao9184/reverse-from-synthid` reproduce or wrap the already-rejected reverse-SynthID codebook. `AI-SCERN` uses uncalibrated annular energy, decoder-grid, and FFT-symmetry heuristics with no weights or held-out corpus. LSB flips, ±1 dither, and 100.5% resize tools publish no matching-oracle evidence. None adds an independent signal |
|
||||
|
||||
## Information budget on photographs (2026-08-21)
|
||||
|
||||
On 11 `gpt-image-2` photo edits, the flat-derived 16-32 template at full
|
||||
amplitude against per-image whitening has median `d' = 0.93` (range
|
||||
0.68-2.46). That is a best-case single-image AUC of about 0.75 even with a
|
||||
perfect, perfectly aligned template. A P5 gate (FPR 0.1%, TPR 90%) needs
|
||||
`d' >= 4.37`. The deficit is 4.7x in amplitude, 13.4 dB, *unless* the
|
||||
photo-domain mark is louder than the flat measurement. Titration said it
|
||||
is not: fish 16-32 additive jam stays `detected` through a=16 / 21.9 dB.
|
||||
|
||||
2026-08-23, without using that flat stamp as `G`: the aligned photo
|
||||
residual in 16-32 is redraw, not a codeword. Median RMS 12.2 across 12
|
||||
pairs; 2.69 on the five tightest. Neighboring octaves are as loud.
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/photo-band-amplitude-2026-08-23.json`.
|
||||
|
||||
Do not report an AUC from those 11 pairs as a detector result. With n=11
|
||||
the standard error on AUC is about 0.12. Notes:
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/agent-detector-claude.md`.
|
||||
|
||||
## External literature (surveyed 2026-08-23)
|
||||
|
||||
Primary sources, not abstracts. Each row is mapped onto a closed or open
|
||||
route in this campaign. Mechanism detail stays in
|
||||
[synthid.md](synthid.md). Removal papers are on
|
||||
[mark removal](synthid-removal-research.md). Classifier papers are on
|
||||
[SynthID source classifiers](synthid-classifiers.md).
|
||||
|
||||
### Official mark, not a public decoder
|
||||
|
||||
| Source | What it is | Map to this campaign |
|
||||
| --- | --- | --- |
|
||||
| Gowal et al., [arXiv:2510.09263](https://arxiv.org/abs/2510.09263) | Post-hoc encoder `f` / decoder `g`. Detection logit is not payload recovery. SynthID-O (partner variant) 136 bits at 512x512. TPR at 0.1% FPR 99.98% aggregated, 99.97% on the hardest spatial crop+resize. Trains against sampled semantics-preserving transforms, including weak VAE regeneration. Production decoder unpublished | Matches the architecture we treat as keyed `x' = x + g(x)`. Explains why a two-pixel crop kills `pipeline_lattice` but not the official oracle, and why a 22.3 dB foreign VAE still reads `detected` |
|
||||
| DeepMind [US12094474B1](https://patents.google.com/patent/US12094474B1/en) and continuation [US20250149048A1](https://patents.google.com/patent/US20250149048A1/en) | Residual U-Net encoder, separate decoder, optional key, encoder/decoder ensembles that need not recognize each other | Constraint, not a recipe. Ensemble non-recognition is why one recovered Google tile cannot be reused as an OpenAI detector (S4) |
|
||||
| Dathathri et al., [Nature 634:818-823 (2024)](https://www.nature.com/articles/s41586-024-08025-4) | SynthID-Text: tournament sampling of LLM tokens, open-source | Different system. Image/audio/video remain proprietary |
|
||||
| OpenAI, [advancing content provenance](https://openai.com/index/advancing-content-provenance/) (2026-05-19, audio 2026-07-31) and [content provenance API](https://developers.openai.com/api/docs/guides/content-provenance) | ChatGPT / API / Codex images carry C2PA plus SynthID. Audio from 2026-07-31. `POST /v1/content_provenance_checks`. `not_detected` does not rule out another vendor | This is the oracle. C2PA and SynthID are independent entries. Do not abuse the endpoint as an adaptive reverse-engineering loop |
|
||||
|
||||
### Keyless presence detectors in the literature
|
||||
|
||||
| Source | Claim | Caveat against our gates |
|
||||
| --- | --- | --- |
|
||||
| Ao et al., [arXiv:2603.06723](https://arxiv.org/abs/2603.06723) (AWPD / FSNet, SAFE@CVPR 2026) | Leave-one-algorithm-out presence detection. SynthID held out: FSNet Acc 0.894 / F1 0.886, ResNet-50 Acc 0.845 / F1 0.812, ConvNeXt V2 Acc 0.866. LSB and Patchwork both fail below 60%. Hypothesis: modern invisible marks share dense high-frequency spectral anomalies | UniFreq Table 3: SynthID is 2,000 images, all AIGC, zero photographs, from `imagen-4.0-fast-generate-001` only, resized to 256x256. No OpenAI. No 0.1% FPR. Closest published analog to E3/L1 distillation. Our TrustMark leave-one-encoder-out was chance (0.505 ridge). Their high-frequency commonality is the same energy that L1 failed to read as an OpenAI payload |
|
||||
| `aloshdenny/reverse-SynthID` V3/V4; Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | Spectral codebook from averaged Gemini flats. Spokesperson Myriam Khan: "It is incorrect to say this tool can systematically remove SynthID watermarks." The author also said the bypass confuses the decoder rather than deleting a payload | Our pickle-free V4 audit: paired AUC 0.517, frozen Open Images 386/1000 accepted. Not 0.1% FPR |
|
||||
| Krawetz, [Hacker Factor "Reversing SynthID"](https://www.hackerfactor.com/blog/index.php?/archives/1092-Reversing-SynthID.html) and ["Meta's Un-Stable Signature"](https://hackerfactor.com/blog/index.php?/archives/1098-Metas-Un-Stable-Signature.html) | Gemini chat TPR closer to 1/20 than the paper's 99.97%. Detector weak on flats. reverse-SynthID field accuracy ~70% vs the author's 90% | Gemini-app chat is not the OpenAI provenance API. Our gray `gpt-image-2` flats are `detected` on the official OpenAI oracle. Do not mix the two verifiers |
|
||||
| vitotitto LAB-a logistic (community, tiny holdout) | Reported 97.7% AUC on 20/20 | Not a P5 gate. Ignore as a detector claim |
|
||||
|
||||
### Different embedding loci (not post-hoc SynthID)
|
||||
|
||||
These papers are often cited as if they were SynthID. They are not.
|
||||
|
||||
| Source | Locus | Why it is not this mark |
|
||||
| --- | --- | --- |
|
||||
| Gunn, Zhao, Song, [arXiv:2410.07369](https://arxiv.org/abs/2410.07369) (PRC, ICLR 2025) | Pseudorandom error-correcting code in the *initial diffusion latent* | Cryptographic undetectability is a latent-code property. SynthID-Image is applied after the VAE has already decoded pixels. Our OKLab "PRC-style" noise was a category error |
|
||||
| Francati et al., [arXiv:2509.10577](https://arxiv.org/abs/2509.10577) (EuroS&P 2026) | Crop-and-resize flipped about half of PRC latent signs and blocked belief-propagation decode | Confirms PRC is origin-locked in latent space. Matches why a 2 px shift kills `pipeline_lattice` and does not kill OpenAI SynthID |
|
||||
| Fernandez et al. Stable Signature; Wen et al. Tree-Ring | Fine-tuned VAE decoder, or ring constraints on initial noise | In-generation. Google's paper is explicit that SynthID-Image does not modify the generator |
|
||||
| TrustMark ([arXiv:2311.18297](https://arxiv.org/abs/2311.18297)), HiDDeN, StegaStamp | Open post-hoc encoder/decoder pairs | Transfer from these is E3: chance on TrustMark. WAVES ([arXiv:2401.08573](https://arxiv.org/abs/2401.08573)) stress-tests this family, not production SynthID |
|
||||
|
||||
Do not train a student on AWPD's Imagen-only 256 crop and call it an
|
||||
OpenAI SynthID detector. That would repeat L1's contamination: a
|
||||
generator/size cue, not a payload.
|
||||
|
||||
### Adjacent literature, not SynthID
|
||||
|
||||
These papers are the published versions of the wild hypotheses and of
|
||||
Model 1. They detect *something else*: a camera pipeline, a generator
|
||||
fingerprint, an open watermark, or steganography. None of them is a
|
||||
keyless OpenAI payload decoder.
|
||||
|
||||
**Classical spread-spectrum and open neural marks**
|
||||
|
||||
| Source | What it is | Map |
|
||||
| --- | --- | --- |
|
||||
| Cox, Kilian, Leighton, Shamoon, IEEE TIP 1997 | i.i.d. Gaussian vector in perceptually significant spectral components. Informed detector. Collusion-resistant by construction | Ancestor of keyed CDMA. Our length-255 m-sequence vs 16x16 fold was at chance because we did not have the key, and because OpenAI is content-adaptive, not a fixed Gold code |
|
||||
| Zhu et al., HiDDeN, ECCV 2018 | Joint encoder / noise layer / decoder | Open pair. AWPD leave-one-out Acc 0.985. Transfer to SynthID is E3 |
|
||||
| Tancik, Mildenhall, Ng, StegaStamp, 2020 | 100 bits at 400x400, trained through print-scan | Open pair. Survives recapture. WAVES: TPR at 1% FPR collapses from 1.00 to 0.01 under regeneration |
|
||||
| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring, NeurIPS 2023) | Pattern in the *initial diffusion noise*, recovered by DDIM inversion | In-generation, origin-locked in latent Fourier space. Crop-invariant by design. Not post-hoc SynthID |
|
||||
| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tune the LDM VAE decoder so every decode carries a bit string | In-generation. Gowal is explicit that SynthID-Image does not modify the generator |
|
||||
| Lin and Juarez, [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) | Public-knowledge attack that removes Tree-Ring | Confirms Tree-Ring is a different object with a different kill |
|
||||
|
||||
**Steganalysis as a presence detector**
|
||||
|
||||
Fridrich and Kodovsky Spatial Rich Models (TIFS 2012) and Boroumand,
|
||||
Chen, Fridrich SRNet (TIFS 2018) detect sub-bit-per-pixel spatial
|
||||
stego by high-pass residuals, with pooling disabled in the front of
|
||||
SRNet so the weak signal is not averaged away. AWPD cites both and
|
||||
says they drift on modern deep / generative marks. That matches our
|
||||
wavelet/FFT single-image detector (AUC 0.653, 0 TPR at a clean cut)
|
||||
and the 16-32 energy miss on photographs: a residual energy detector
|
||||
without the matching key is steganalysis of a mark that was trained
|
||||
not to look like LSB.
|
||||
|
||||
**Generator fingerprints in the Fourier domain**
|
||||
|
||||
Corvi, Cozzolino, Poggi, Nagano, Verdoliva,
|
||||
[arXiv:2304.06408](https://arxiv.org/abs/2304.06408) (CVPRW 2023):
|
||||
GAN, diffusion, and VQ-GAN images show spectral peaks and anomalous
|
||||
autocorrelation; real vs synthetic differ in mid-high radial and
|
||||
angular power. reverse-SynthID averaged Gemini flats and called the
|
||||
peak a watermark codebook. Corvi's result says many generators leave
|
||||
*some* peak. Our V4 Open Images 386/1000 is what a generator-fingerprint
|
||||
detector looks like when you calibrate it as if it were a payload.
|
||||
|
||||
Yao and Juarez, [arXiv:2512.11771](https://arxiv.org/abs/2512.11771)
|
||||
("Smudged Fingerprints"): 14 fingerprinting methods across RGB,
|
||||
frequency, and learned features; removal attacks >80% white-box, >50%
|
||||
black-box. A fingerprint you can see without a key is a fingerprint
|
||||
you can wipe without a key.
|
||||
|
||||
### Image investigation and data hiding (any method)
|
||||
|
||||
These are not SynthID papers. They are the rest of the toolkit: how
|
||||
people hide bits in pictures, and how people tell a picture was
|
||||
touched. Several of our wild hypotheses already had a published form
|
||||
here.
|
||||
|
||||
Hiding is not one problem. Cover modification (change an existing
|
||||
image), coverless / generative (sample an image that already carries
|
||||
the bits), and signed metadata (C2PA) fail under different attacks.
|
||||
|
||||
**Cover modification, classical**
|
||||
|
||||
| Source | Hide how | Detect / limit |
|
||||
| --- | --- | --- |
|
||||
| LSB, Patchwork (Bender et al., IBM SJ 1996) | Flip low bits, or luminance of random pixel pairs | AWPD Acc < 0.60. Sparse or ±1 amplitude. SRNet / FSNet average it away |
|
||||
| Westfeld F5 (2001), Fridrich nsF5 | JPEG DCT coefficients, matrix embedding | Histogram attacks on F5; nsF5 was the shrinkage fix. Domain is the codec, not a mid-band residual |
|
||||
| Pevny, Filler, Bas HUGO (2010); Holub and Fridrich WOW (2012); Holub, Fridrich, Denemark UNIWARD (IH&MMSec 2013 / EURASIP 2014) | Content-adaptive costs, bits placed in texture via syndrome-trellis codes (Filler, IH 2011) | The modern spatial/JPEG floor. Distortion is *designed* to look like the cover. A 16-32 energy detector is the wrong statistic |
|
||||
| Cox et al. 1997 | Spread-spectrum in significant DCT | Keyed. Already mapped above |
|
||||
|
||||
**Cover modification, neural (hide a whole image, not 32 bits)**
|
||||
|
||||
| Source | What it does | Map |
|
||||
| --- | --- | --- |
|
||||
| Baluja, NeurIPS 2017, "Hiding Images in Plain Sight" | Full-resolution secret image distributed across all bits of a same-size cover | Capacity is the point. Not robust, not keyed provenance. HiDDeN later added a noise layer so the secret survives JPEG |
|
||||
| Jing et al., HiNet, ICCV 2021 | Invertible neural net: hide and recover as one bijection | High-capacity lossless-ish hiding. Recovery needs the exact inverse, not an official oracle |
|
||||
| Yang et al., PRIS, [arXiv:2309.13620](https://arxiv.org/abs/2309.13620) | Invertible net plus robustness modules | Same family, trained through distortion |
|
||||
|
||||
A HiNet-style student on OpenAI pairs would learn the *edit*, not a
|
||||
SynthID codeword. That is the L1 contamination again.
|
||||
|
||||
**Coverless / generative hiding**
|
||||
|
||||
The cover is never modified because there is no cover. The sampler
|
||||
emits an image whose latents already encode the message.
|
||||
|
||||
| Source | Locus | Map |
|
||||
| --- | --- | --- |
|
||||
| Yang et al., Gaussian Shading, [arXiv:2404.04956](https://arxiv.org/abs/2404.04956) (CVPR 2024) | Map bits onto Gaussian latents indistinguishable from ordinary noise. Recover by DDIM inversion | Sibling of Tree-Ring / PRC. Training-free, performance-lossless *for the generator*. Not a post-hoc pixel stamp |
|
||||
| Peng et al., StegaDDPM (ACM MM 2023) and later LDStega | Bits in the diffusion sampling distribution | Spatial SRNet is the wrong detector (NS-DSer, [arXiv:2602.10219](https://arxiv.org/abs/2602.10219): move steganalysis into noise space) |
|
||||
| CRoSS, Pulsar, MDDM | Message-to-noise projections | Same locus. A pixel scramble does not invert the sampler |
|
||||
|
||||
**Passive forensics (the picture was touched, no secret assumed)**
|
||||
|
||||
| Source | Cue | Map |
|
||||
| --- | --- | --- |
|
||||
| Krawetz, "A Picture's Worth", 2007 (ELA) | Re-JPEG at lower quality, subtract | Already measured: COCO 3.13, s1 1.97, gray stamp 0.49. Codec history, not a payload. Farid publicly called ELA as likely to mislabel originals as it is to catch edits |
|
||||
| Farid, IEEE TIFS 2009, JPEG ghosts | Difference energy vs a sweep of JPEG qualities; spliced regions ghost at their original Q | Local: s1, a `gpt-image-2` photo, and a camera JPEG all minimize at Q90. Codec, not a payload |
|
||||
| Popescu and Farid, TR2004-515 | Copy-move via duplicated regions | Not generation, not a watermark |
|
||||
| Popescu and Farid, IEEE TSP 2005 | Resampling periodic correlations | Affine search cousin. A rotated SynthID residual is a different question |
|
||||
| Wang et al., DIRE, [arXiv:2303.09295](https://arxiv.org/abs/2303.09295) (ICCV 2023) | Diffusion reconstruction error: generated images reconstruct, cameras do not | Model 1 sibling. SDXL float32 at 512: DIRE RMS camera 33.0, s1 31.5, photo edit 30.9, gray stamp 2.40. Texture rank, not a payload. Inverse of the foreign-VAE remover: there the mark survived 22.3 dB |
|
||||
| Wang, Wang, Zhang, Owens, Efros, [arXiv:1912.11035](https://arxiv.org/abs/1912.11035) (CVPR 2020, CNNDetect) | One ProGAN classifier, heavy JPEG/crop aug, transfers to many CNNs | Ancestor of "train on one generator". Ojha showed the sink-class failure once diffusion arrived. We required Firefly for that reason |
|
||||
|
||||
C2PA is the non-pixel stack: a signed manifest, stripped by
|
||||
`metadata --remove`. Durable Content Credentials (spec 2.4) add a
|
||||
soft binding that can re-link a stripped file to a repository. That
|
||||
is provenance, not hiding.
|
||||
|
||||
Do not train on ELA, JPEG ghosts, DIRE, or a HiNet reconstruction and
|
||||
name the score SynthID.
|
||||
|
||||
### Waveforms that can live in a picture
|
||||
|
||||
A mark is a function on the pixel lattice. The literature does not
|
||||
use one wave. It picks a basis whose symmetries match the attack it
|
||||
fears, then hides a keyed coefficient vector in that basis. Cartesian
|
||||
16-32 is one shell. Polar, scale, and diffraction are different
|
||||
shells.
|
||||
|
||||
**Standing waves on a rectangle.** DFT / DCT / DST. A 2-D sinusoid
|
||||
`cos(2π(ux + vy)/N)`. JPEG lives here. Our octave scramble destroys
|
||||
one annular *radius* of these frequencies, not one orientation. A
|
||||
Gabor packet is the same sinusoid windowed in space.
|
||||
|
||||
**Circular and log-radial waves.** Functions of `(r, θ)`, not
|
||||
`(x, y)`.
|
||||
|
||||
| Basis | Wave | Invariance it buys |
|
||||
| --- | --- | --- |
|
||||
| Fourier-Mellin / log-polar (Ó Ruanaidh and Pun, Signal Processing 1998) | `r^{iα} exp(ikθ)` after a DFT magnitude | Rotation and scale become translations |
|
||||
| Logarithmic radial harmonics (IH 2002) | Same family, added in pixels, detected by complex correlation | RST without going through the unstable log-polar resample |
|
||||
| Polar harmonic transforms: PCET, PCT, PST | `exp(±i2π n r²)` and polar cos/sin | Rotation. Moments, not a dense codebook |
|
||||
| Zernike / pseudo-Zernike | Orthogonal polynomials on the disk, radial part related to Bessel | Rotation. Classical moment watermark |
|
||||
| Circular chirp (SPIE 6072, 2006) | Polar map of a 1-D chirp onto a ring | JPEG (tune chirp rate) plus rotation (the ring) |
|
||||
| Tree-Ring (Wen 2023) | Concentric rings in the *latent* Fourier plane | Crop/flip by construction. Not a pixel wave |
|
||||
|
||||
The 2026-08-23 file named polar-1632 is cartesian annular phase
|
||||
shuffle, a scramble replicate (`not_detected` at 25.6 dB). True
|
||||
Fourier-angle scramble of the same annulus is `not_detected` at 24.1 dB.
|
||||
Radial-phase-only scramble is `not_detected` at 25.0 dB. One 90°
|
||||
sector plus its conjugate stays `detected` at 27.5 dB. Affine/rotation
|
||||
search of the 16-32 residual against the flat template does not lock
|
||||
(lift 0.011, COCO 0.008).
|
||||
|
||||
**Scale chirps.** Hyperbolic / Mellin monomials (arXiv:1208.5842):
|
||||
real 1-D Mellin patterns tiled in 2-D. Run 2026-08-23:
|
||||
`r^{iα}` coefficients flat LOO 0.51, photo -0.27, COCO -0.58.
|
||||
Log-polar `|F|` and a 64-bin radial Mellin profile saturate on COCO
|
||||
(0.99). Not a payload, and not quieter than cartesian 16-32.
|
||||
|
||||
**Directional packets.** Dual-tree complex wavelets (approximate
|
||||
analytic wave), Gabor/Morlet, ridgelets, curvelets, shearlets,
|
||||
contourlets, bandelets. Multiplicative spread-spectrum on curvelet
|
||||
coefficients is a published detector-design paper, not a SynthID
|
||||
decoder. Our wavelet summary AUC 0.653 already said a *generic*
|
||||
packet energy is not the OpenAI payload.
|
||||
|
||||
**Optical diffraction, actual wave physics.**
|
||||
|
||||
| Transform | What the wave is | Map |
|
||||
| --- | --- | --- |
|
||||
| Fresnel | Quadratic phase `exp(iπ r² / λz)`, a radial chirp. Energy concentrates with distance | Holographic watermarks embed the diffracted field, not the picture |
|
||||
| Fractional Fourier | Rotation of the time-frequency plane. Order is an extra key | Extra key space on top of DFT |
|
||||
| Double random phase encoding (Réfrégier and Javidi, Optics Letters 1995) | Random phase at input *and* Fourier plane. Ciphertext is stationary white noise | Closest optical analog of a keyed `x' = x + g(x)` that looks like noise without both screens |
|
||||
| Cascaded Fresnel / CGH | Two holograms at different depths | Hardware key. Not a pixel residual we can match-filter |
|
||||
|
||||
A Fresnel/FrFT student without the distance or the fractional order
|
||||
is another unmatched filter. Fresnel at z=8, 16, 32, 64 on the 16-32
|
||||
residual is a unitary copy of the cartesian band (flat LOO 0.75, photo
|
||||
0.012). FrFT is the same unmatched-filter claim with an extra order
|
||||
key; it does not isolate a photograph lock the four distances missed.
|
||||
|
||||
**Cyclostationary and clock-warped waves.** Periodic clock change
|
||||
(Martin, Chabert, Lacaze, EUSIPCO 2005) turns a stationary host into
|
||||
a cyclostationary one by a periodic time warp. Our H2 multiplicative
|
||||
cyclostationarity test already put official positives *below*
|
||||
controls. Do not rerun the same statistic in a new basis and call it
|
||||
new evidence.
|
||||
|
||||
**How to read a candidate wave.** Name the coordinates `(x,y)` vs
|
||||
`(r,θ)` vs `log r`, the generating function (sinusoid, chirp,
|
||||
Bessel, Zernike), additive vs multiplicative, and the symmetry it
|
||||
claims. Then pick one attack that kills that symmetry and one that
|
||||
should not. Cartesian 16-32 scramble and a 2 px shift are done
|
||||
(scramble kills, shift does not). True Fourier-angle scramble kills s1
|
||||
at 24.1 dB. Radial-phase scramble kills at 25.0 dB. A 90° sector does
|
||||
not. Y scramble kills; Cb/Cr do not. Affine, Mellin, Zernike, PCET,
|
||||
and Fresnel of the 16-32 residual do not lock on photographs.
|
||||
|
||||
## Still open or incomplete
|
||||
|
||||
- Google: no public payload decoder and no usable programmatic oracle.
|
||||
The Gemini app remains quota-limited. Vertex was 403.
|
||||
- `chatgpt-image-latest` is an optional third pair mint, not a blocker.
|
||||
`gpt-image-1`, `gpt-image-1.5`, and `gpt-image-1-mini` are encoder-off.
|
||||
`gpt-image-2` and dated `gpt-image-2-2026-04-21` are encoder-on, same
|
||||
16-32 stamp (corr 0.189 on gray). Latest would only matter if ChatGPT's
|
||||
current consumer id used a different ensemble. `images.generate` is 403
|
||||
until the organization is verified.
|
||||
- Physical print-scan: Brother DCP-L2520DW is idle and accepting. No
|
||||
`scanimage`, no ImageCapture pyobjc. Recapture needs a human at the
|
||||
flatbed. Simulation at 24.95 dB stayed `detected`. Do not print
|
||||
unattended.
|
||||
- Adding unmarked context (frame, pad, hstack, collage) to hunt a
|
||||
residual is a presentation gate, not a mark statistic. Those rungs
|
||||
belong with identity and layout below, and with
|
||||
[mark removal](synthid-removal-research.md).
|
||||
- Camera-to-ChatGPT "unchanged crop" pairs are dirty counterfactuals.
|
||||
The instructed edit redraws; aligned PSNR on the frame pair was
|
||||
16.6 dB. Compare pixels only after registration and a frozen residual
|
||||
bound, as in the `gpt-image-2` red-dot row.
|
||||
|
||||
A Gemini app caption that an image "looks like a photograph" is not a
|
||||
SynthID verdict. Use the provider pixel check or signed provenance.
|
||||
|
||||
## Wild hypotheses
|
||||
|
||||
Battery 2026-08-23, no official oracle. Local numbers:
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/wild-hypotheses-2026-08-23.json`.
|
||||
Prepared attack rasters wait in `wild-attacks-2026-08-23/` for a later
|
||||
verifier window. Jacobian / adaptive queries against
|
||||
`verify-openai-synthid` stay out: the endpoint forbids reverse-engineering.
|
||||
|
||||
### Tested locally, not a mark
|
||||
|
||||
| Hypothesis | Result |
|
||||
| --- | --- |
|
||||
| Chroma ordered by luma is a 1D payload | Correlation is an artifact of the sort |
|
||||
| Mark is `g(x)` of a Retinex envelope | LOO unchanged vs the raw 16-32 residual |
|
||||
| Nonlocal patch agreement in 16-32 | Far patches are not coupled |
|
||||
| Quadratic phase coupling (bispectrum) | Statistic saturates on camera images too |
|
||||
| CDMA Gold / m-sequence in the 16x16 fold | At chance for length 255 |
|
||||
| Rows as a pitched waveform | COCO noise has the same peak-to-mean |
|
||||
| The photo mark is the flat stamp at an unknown 16-phase | Exhaustive shift does not lift NCC |
|
||||
| The photo mark is the flat stamp at a small rotation or 3% scale | Affine search lift 0.011, COCO noise 0.008 |
|
||||
| Self-keyed predictor from a 16 px luma thumb | Already 0.48 vs L1, 0.50 vs COCO (`self-keyed.json`) |
|
||||
| Global multiplicative cyclostationarity (H2) | Official positives below controls, 2026-08-14 |
|
||||
| Mellin radial profile / log-polar \|F\| | Saturates on COCO (0.99). Annulus envelope |
|
||||
| Scale chirps `r^{iα}` | Flat 0.51, photo -0.27, COCO -0.58 |
|
||||
| Zernike n≤6 | Flat 0.77, photo -0.007, COCO 0.23 |
|
||||
| PCET polar harmonics | Flat 0.75, photo 0.044, COCO 0.35 |
|
||||
| Fresnel z=8..64 of 16-32 | Copy of cartesian LOO |
|
||||
| 8-bin orientation energy | Saturates on COCO |
|
||||
| Chroma 16-32 on gray flats | Cb 0.064, Cr 0.025 vs Y 0.682. Stamp is luma |
|
||||
|
||||
### Not a watermark, might still be an AI-vs-camera extra
|
||||
|
||||
Battery 2026-08-23, `remaining-hypotheses-2026-08-23.json`. Kodak corpus was empty here, camera arm is COCO plus the photo-pair sources.
|
||||
|
||||
| Hypothesis | Result |
|
||||
| --- | --- |
|
||||
| Absence of camera PRNU | Gray highpass RMS 0.25 vs COCO 14.6 was texture. Wiener residual on *photographs*: `gpt-image-2` edits 4.61 vs camera 8.05. Direction is camera-like leftover of the donor JPEG, not a missing sensor, and not a payload |
|
||||
| JPEG ELA | Flat 0.49, s1 1.97, COCO 3.13. Compression history |
|
||||
| CFA interpolation error (Bayer as detector) | Ratio photo-edit 0.117 vs camera 0.184 vs gray stamp 0.588. Weak camera vote, overlaps. Inverse of the Bayer remover, not SynthID |
|
||||
| JPEG ghosts (Farid) | s1, a `gpt-image-2` photo, and a camera JPEG all minimize at Q90 with similar spreads. Codec, not a payload |
|
||||
| PNG Paeth / filter-byte residual | `gpt-image-2` PNG is 99.9% Paeth. Camera re-saved PNG 73% Paeth, s1 74%. Export encoder fingerprint |
|
||||
| Perfect-circle / text-edge rate | Hough circles per MP 385 (edits) vs 536 (camera). Edge fraction 0.052 vs 0.072. Too noisy for a payload or a provider |
|
||||
| Face-gated 16-32 | s1: 3 Haar faces, inside RMS 12.6 vs outside 10.9. One photo with faces: outside/in 1.49. No systematic "spend the mark off faces" |
|
||||
| CLIP-space watermark | That is Model 1. Not a payload decoder |
|
||||
|
||||
### Non-local, 2026-08-23
|
||||
|
||||
Battery `.local-eval/synthid/prc-oklab-attack-2026-08-15/nonlocal-2026-08-23/`.
|
||||
Oracle after AI-metadata strip, 25 s gap, then one dated-id retry after a 429.
|
||||
|
||||
| Hypothesis | Result |
|
||||
| --- | --- |
|
||||
| HEIF q80 / q50 | 46.3 / 39.3 dB, official `detected` |
|
||||
| AV1 CRF 32 still | 37.2 dB, `detected` |
|
||||
| Print-scan simulation (0.35 deg, blur σ 0.9, 300/360 dpi, JPEG q42) | 24.95 dB, still `detected`. Same PSNR class as 16-32 scramble, which kills |
|
||||
| Physical print-scan | Brother DCP-L2520DW idle and accepting. No `scanimage`, no ImageCapture pyobjc. Not printed unattended |
|
||||
| `gpt-image-1.5` gray reproduce | C2PA `created, converted`. Oracle `not_detected`. 16-32 corr vs `gpt-image-2` stamp 0.003. Encoder-off. Dirty redraw 24.0 dB |
|
||||
| `gpt-image-1-mini` gray reproduce | Same C2PA. Oracle `not_detected`. Corr 0.003. PSNR 31.1 dB |
|
||||
| `gpt-image-2-2026-04-21` gray reproduce | C2PA `created, converted, watermarked.unbound`. Official `detected`. 16-32 corr vs unversioned `gpt-image-2` stamp 0.189. Same encoder family. PSNR 47.5 dB |
|
||||
| `chatgpt-image-latest` generate | 403, org verification. Listed among image models, not callable |
|
||||
| DIRE SDXL VAE + DDIM float32 | VAE RMS 512: gray 1.24, s1 9.15, photo edit 9.89, camera 11.84. DIRE RMS: gray 2.40, photo 30.9, s1 31.5, camera 33.0. Texture rank, not a payload |
|
||||
|
||||
### Still blocked after that battery
|
||||
|
||||
| Hypothesis | Why it stays blocked |
|
||||
| --- | --- |
|
||||
| Physical print-scan recapture | Human at the flatbed |
|
||||
| Encoder ensemble that does not recognize itself | Needs another actual SynthID encoder/decoder pair. Extra OpenAI ids did not mint one. Open-proxy transfer is already closed: source-matched TrustMark, VideoSeal, DWT-DCT, and WAM pairs produced held-out-family AUC 0.437-0.562, so another generic public pair does not answer this hypothesis |
|
||||
| `chatgpt-image-latest` as a third pair mint | `images.generate` 403 until org verification. Optional: only if ChatGPT latest used a different ensemble than `gpt-image-2` |
|
||||
|
||||
Do not train on official decoder bits beyond the frozen L1 study. Do not
|
||||
mint a 128-photo student on the flat stamp. The information budget and
|
||||
the 2026-08-23 residual measurement both say the photo-domain mark is
|
||||
under the scene.
|
||||
|
||||
## Oracle and seeds
|
||||
|
||||
Official `POST /v1/content_provenance_checks`, metadata stripped, pixels
|
||||
preserved. Raw files stay gitignored under
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/`.
|
||||
|
||||
| Seed | Size |
|
||||
| --- | --- |
|
||||
| s1 | 1024x1536 |
|
||||
| s2 | 1024x1536 |
|
||||
| s3 | 1086x1448 |
|
||||
|
||||
All three `detected` before attacks.
|
||||
|
||||
## Source-matched pairs (2026-08-21)
|
||||
|
||||
`gpt-image-2` (ChatGPT Images 2.0) can mint encoder-on/off pairs.
|
||||
`chatgpt-image-latest` is optional and 403 until org verification.
|
||||
|
||||
| Input | C2PA | Oracle | Aligned PSNR |
|
||||
| --- | --- | --- | ---: |
|
||||
| Camera JPEG 960x1280 | none | not_detected | - |
|
||||
| ChatGPT UI + frame 1086x1448 | watermarked.unbound | detected | 16.6 dB (redraw) |
|
||||
| `gpt-image-1` edit, high fidelity, red dot | created, converted only | not_detected | 30.9 dB |
|
||||
| `gpt-image-1.5` gray reproduce 1024 | created, converted only | not_detected | 24.0 dB (redraw) |
|
||||
| `gpt-image-1-mini` gray reproduce 1024 | created, converted only | not_detected | 31.1 dB |
|
||||
| `gpt-image-2` edit, red dot 1024x1536 | watermarked.unbound | detected | 28.2 dB |
|
||||
| `gpt-image-2-2026-04-21` gray reproduce 1024 | watermarked.unbound | detected | 47.5 dB |
|
||||
|
||||
The `gpt-image-2` photo residual is broadband (std 9.1), not a 16-32 stamp.
|
||||
COCO `gpt-image-2` edits: 11/12 `watermarked.unbound`, one safety block.
|
||||
|
||||
Solid gray/white through `gpt-image-2` "reproduce exactly" is
|
||||
`watermarked.unbound`. Gray 128 at 1024x1536: PSNR 41.9 dB vs the flat,
|
||||
source `not_detected`, output `detected`. Other gray levels 32-220: PSNR
|
||||
38-49 dB. Eight gray 1024x1536 residuals, leave-one-out correlation after
|
||||
highpass: 0.645 (shuffle 0). That shared stamp does not transfer to COCO
|
||||
photo residuals (0.007-0.025).
|
||||
|
||||
## Presentation gate versus residual
|
||||
|
||||
An official `not_detected` is not one event. Collage, photo-pad, two-panel
|
||||
self-concat, and 8x8 scramble can be a *presentation gate* (this upload is
|
||||
not a single photograph). Blur sigma 7 and in-band 16-32 destruction sit on
|
||||
the single-photo manifold and are residual damage; those belong in
|
||||
[mark removal](synthid-removal-research.md).
|
||||
|
||||
Do not train a surrogate on mixed collage misses if the goal is a mark
|
||||
student.
|
||||
|
||||
Identity (s1, 2026-08-19): JPEG q80/q20, crop 2 px, crop 8 px, keep 0.75
|
||||
and 0.50, rot90, rotate 5, flip, grayscale all stay `detected`.
|
||||
|
||||
Token / layout (s1): each quadrant at 512x768 `detected`; `hstack` and
|
||||
`vstack` of marked\|marked `not_detected`; 4x4 tile scramble `detected`;
|
||||
8x8 `not_detected`. Two copies of a detecting image still miss. Tomography
|
||||
3x3: all nine cells `not_detected` (a 1/9 window is below support).
|
||||
|
||||
Preprocess E1 (s1): stretch 2x `detected`; centre-crop of marked\|marked
|
||||
hstack back to native `detected`; 0.20x pixels on a native-size white
|
||||
canvas `not_detected` (the same 0.20x file uploaded alone was `detected`).
|
||||
|
||||
## L1 is not a mark task
|
||||
|
||||
A missing `watermarked.unbound` assertion is not a clean negative. On
|
||||
2026-08-16 the official verifier called 65 of 94 OpenAI rows without that
|
||||
assertion `detected` (69%), with 9 of 9 interleaved health positives
|
||||
detected so the endpoint was answering. Google's own API emits watermarked
|
||||
images with no assertion, per its documentation. Corpus AUCs that treat
|
||||
"no assertion" as unmarked rest on a negative class that is roughly
|
||||
two-thirds positive.
|
||||
|
||||
Control-only 283 rows: 203 detected, 80 not_detected. Forward-temporal
|
||||
nested ridge on 124 pixel features: AUC 0.649 / 0.641. Geometry-only:
|
||||
0.781 / 0.826. All 21 `claim_generator=ChatGPT` controls are
|
||||
`not_detected`. C2PA `opened`: 24/24 `not_detected`. Inside 1254x1254,
|
||||
mean-feature permutation p=0.45.
|
||||
|
||||
OpenAI-supervised ridge on 16-32 band-passed `image_features`:
|
||||
forward-temporal AUC 0.53 vs L1 `not_detected`, 0.95 vs COCO, 0.97 vs COCO
|
||||
after a 2 px crop. The student learns OpenAI-versus-photo and still does
|
||||
not see the oracle mark contrast.
|
||||
|
||||
## Product remainder for the watermark
|
||||
|
||||
Signed provenance (`identify`) and `verify-openai-synthid` (remote, explicit
|
||||
upload). A research lattice miss is not a clean SynthID negative.
|
||||
@@ -0,0 +1,54 @@
|
||||
# OpenAI SynthID oracle ladders
|
||||
|
||||
> This page is a routing hub. The mixed 2026-08 campaign log was split by
|
||||
> purpose on 2026-08-22. Numeric tables now live on the page for that
|
||||
> purpose. Raw images stay gitignored under
|
||||
> `.local-eval/synthid/prc-oklab-attack-2026-08-15/`.
|
||||
>
|
||||
> Seeds s1, s2, s3 are listed in
|
||||
> [SynthID local detector research](synthid-detector-research.md).
|
||||
|
||||
| Page | Use it for |
|
||||
| --- | --- |
|
||||
| [SynthID local detector research](synthid-detector-research.md) | Hunt for a keyless local mark detector. Open. Closed routes on that page. |
|
||||
| [AI-generated image classifiers](ai-generated-image-classifiers.md) | Primary metadata-free AI-generation task, Model 1 partial result, and frozen-checkpoint sweep. |
|
||||
| [SynthID source classifiers](synthid-classifiers.md) | OpenAI/Gemini source finding and `pipeline_lattice` as google-lineage. |
|
||||
| [SynthID mark removal research](synthid-removal-research.md) | Quiet-remover hunt. Closed except ~19-24 dB 16-32 scramble and blur sigma 7. |
|
||||
| [Mechanism reference](synthid.md) | How SynthID works, provenance, robustness, regeneration. |
|
||||
| [Chronological plan archive](synthid-detector-removal-plan.md) | Dated H-gates, corpora, and session notes in original order. |
|
||||
|
||||
## Where former sections went
|
||||
|
||||
| Former heading | Now |
|
||||
| --- | --- |
|
||||
| 2026-08-21 pairs, flats, L1 labels | [detector](synthid-detector-research.md) |
|
||||
| Identity, token/layout, tomography, preprocess E1 | [detector](synthid-detector-research.md) (presentation gate) |
|
||||
| L1 repair, L1 geometry, camera vs edit pair, E3 | [detector](synthid-detector-research.md) |
|
||||
| CLIP-L photo vs AI, provider CLIP, union | [AI classifiers](ai-generated-image-classifiers.md), [source classifiers](synthid-classifiers.md) |
|
||||
| 124-d three-class and binary AI | [source classifiers](synthid-classifiers.md) |
|
||||
| `pipeline_lattice` re-check, Spaces census | [source classifiers](synthid-classifiers.md) |
|
||||
| Attack / kill / flip / add / surround ladders | [removal](synthid-removal-research.md) |
|
||||
| 16-32 titration, E2 scramble, Bayer and geometry | [removal](synthid-removal-research.md) |
|
||||
| S4 provider split, M2 Imagen `addWatermark`, reverse-SynthID, Bypass | [detector](synthid-detector-research.md) |
|
||||
| Photo `d'` budget 13.4 dB, 128-photo student, 16-32 residual without flat `G` | [detector](synthid-detector-research.md) |
|
||||
| OKLab codeword replacement, add-context as presentation | [removal](synthid-removal-research.md) |
|
||||
| Three-class OpenAI / Gemini / photo ask | [source classifiers](synthid-classifiers.md) |
|
||||
| Wild hypotheses 2026-08-23 (sort, CDMA, bispectrum, PRNU, affine 16-32 NCC no lock) | [detector](synthid-detector-research.md) |
|
||||
| Wild oracle 2026-08-23: misnamed polar-1632 is cartesian scramble replicate `not_detected`; band-transplant `not_detected`; palette64 `detected` | [removal](synthid-removal-research.md) |
|
||||
| External literature 2026-08-23 (Gowal, AWPD, PRC, Zhao, UnMarker, CtrlRegen, MarkNull, reverse-SynthID) | [detector](synthid-detector-research.md), [removal](synthid-removal-research.md), [AI classifiers](ai-generated-image-classifiers.md) |
|
||||
| Adjacent literature, not SynthID (Cox, HiDDeN, StegaStamp, Tree-Ring, Ojha CLIP, Corvi Fourier, DCCT CFA, PRNU PCE) | [detector](synthid-detector-research.md), [AI classifiers](ai-generated-image-classifiers.md), [removal](synthid-removal-research.md) |
|
||||
| Image investigation and data hiding (LSB, UNIWARD, Baluja, HiNet, Gaussian Shading, ELA, JPEG ghosts, DIRE, CNNDetect) | [detector](synthid-detector-research.md), [AI classifiers](ai-generated-image-classifiers.md) |
|
||||
| Waveforms in a picture (DFT, Fourier-Mellin, Zernike, chirps, Fresnel, DRPE, cyclostationary) | [detector](synthid-detector-research.md) |
|
||||
| Remaining hypotheses 2026-08-23 (CFA, JPEG ghost, Paeth, face-gate, Baker, Poisson, nested LSB, angular scramble) | [detector](synthid-detector-research.md), [removal](synthid-removal-research.md), [AI classifiers](ai-generated-image-classifiers.md) |
|
||||
| Non-local 2026-08-23 (HEIF/AV1 survive, print-scan sim 24.95 dB still detected, gpt-image-1.5/mini encoder-off, dated gpt-image-2 same stamp, DIRE DDIM texture) | [detector](synthid-detector-research.md), [removal](synthid-removal-research.md), [AI classifiers](ai-generated-image-classifiers.md) |
|
||||
| Waveform shells 2026-08-23 (Mellin/Zernike/PCET/Fresnel no photo lock; Y scramble kills, chroma and 90° sector do not; radial-phase kills) | [detector](synthid-detector-research.md), [removal](synthid-removal-research.md) |
|
||||
| Public decoder sweep 2026-08-23/24 (`Rinne414` fixed templates 0/400 on current OpenAI/Google images; injected template spoofs the public correlator, but 0.25-2x subtraction stays official `detected` at 43.13-25.72 dB) | [detector](synthid-detector-research.md), [removal](synthid-removal-research.md) |
|
||||
| Public CNN sweep 2026-08-24 (`newideas99` retains 172/200 OpenAI but accepts 110/200 Google; a whole-frame gradient makes the ensemble effectively zero but stays official `detected` through ±8/255 / 34.30 dB) | [detector](synthid-detector-research.md), [source classifiers](synthid-classifiers.md), [removal](synthid-removal-research.md) |
|
||||
| `reverse-SynthID` V4 repeat 2026-08-24 (77/200 Google and 76/200 OpenAI at the published cut; Round-06 manual Gemini verdicts are not published as per-image evidence) | [detector](synthid-detector-research.md), [source classifiers](synthid-classifiers.md), [removal](synthid-removal-research.md) |
|
||||
| Broad GitHub sweep 2026-08-24 (133 unique primary repositories; REGRET and the vordme flat-field SVM fail the strict v7 transfer; literal ports and generic heuristics add no signal) | [detector](synthid-detector-research.md), [source classifiers](synthid-classifiers.md) |
|
||||
| Additional public removal sweep 2026-08-24 (DeSynth stays official `detected`; froggeric and other ControlNet/diffusion projects corroborate lossy regeneration; PortOS closes high-fidelity phase/noise jamming) | [removal](synthid-removal-research.md) |
|
||||
| Google verifier surfaces 2026-08-24 (retired Vertex `imageverification@001`; live but unlisted and inaccessible `synthid.googleapis.com/upload/v1:verifyContent`) | [detector](synthid-detector-research.md) |
|
||||
| Metadata-free source hunt 2026-08-24 (1% cascades rejected; frozen multiscale fusion found 276/600 OpenAI-or-Google targets, 0/1,000 photographs and 1/25 TC260 on blind v4) | [source classifiers](synthid-classifiers.md) |
|
||||
| Original-export source hybrids 2026-08-24 (frozen v7 215/400 exact; post-hoc v8 292/400 and per-codec v11 316/400 exact) | [source classifiers](synthid-classifiers.md) |
|
||||
| Frozen general-detector transfer 2026-08-24 (Community Forensics and SPAI do not replace or safely fuse with Model 1 at the 1% calibration FPR; B-Free weights unavailable and license noncommercial) | [AI classifiers](ai-generated-image-classifiers.md#frozen-public-checkpoint-transfer-2026-08-24) |
|
||||
| General-classifier GitHub sweep 2026-08-25 (2,006 unique repositories found; DDA and PGC expose complementary frozen signals, but DDA raises independent FPR and PGC's OpenAI gain confounds 24/24 Kodak scans; DGS-Net, SAFE, RINE, and Nonescape Mini rejected) | [AI classifiers](ai-generated-image-classifiers.md#general-ai-classifier-github-sweep-2026-08-25) |
|
||||
@@ -0,0 +1,243 @@
|
||||
# SynthID mark removal research
|
||||
|
||||
> Research archive for pixel-only SynthID removal, not diffusion
|
||||
> regeneration. Not a statement of current product capability. Shipped
|
||||
> invisible removal is lossy regeneration:
|
||||
> [known limitations](known-limitations.md).
|
||||
>
|
||||
> Sister pages: [SynthID local detector](synthid-detector-research.md),
|
||||
> [SynthID source classifiers](synthid-classifiers.md),
|
||||
> [mechanism reference](synthid.md).
|
||||
|
||||
## Result
|
||||
|
||||
The quality-preserving OpenAI SynthID remover hunt closed 2026-08-20.
|
||||
Bayer, VNG demosaic, upscale-then-Bayer, barrel distortion, scanline
|
||||
jitter, and a 2 px shift closed 2026-08-22 on s1/s2 and 2026-08-23 on s3
|
||||
and fish: they leave the official oracle `detected`.
|
||||
|
||||
Working residual kills on photographs cost about 19-26 dB:
|
||||
|
||||
- 16-32 px cartesian phase scramble (s1 24.6-24.8 dB, fish 23.2 dB, s2 19.0 dB)
|
||||
- Fourier-angle scramble of the same annulus (s1 24.1 dB)
|
||||
- Radial-phase scramble of the same annulus (s1 25.0 dB)
|
||||
- Y-only 16-32 scramble (s1 24.6 dB); Cb/Cr-only do not kill
|
||||
- File named polar-1632, actually cartesian (s1 25.6 dB)
|
||||
- Replace 16-32 with a COCO photo's 16-32 (s1 25.2 dB)
|
||||
- Gaussian blur sigma 7 (23.8 dB), holds 3/3 versus sigma 6
|
||||
- Additive 16-32 jam only at a=24 / 18.6 dB, worse than scramble
|
||||
|
||||
Baker-map, 8-seam carve, Poisson, nested LSB, palette64, and ICC rewrite
|
||||
do not kill at a better PSNR.
|
||||
|
||||
JPEG q5, noise sigma 16, grayscale, rot90, flip, 5°, downscale 0.20x,
|
||||
median 7, posterize 4, VAE round-trip, and white pad to 40% linear stay
|
||||
`detected`. Elastic warp is not a stable kill (s3 still `detected` at
|
||||
22.0 dB).
|
||||
|
||||
The product remainder is diffusion regeneration (`qwen-zimage` /
|
||||
`sdxl-zimage`), which does not decode and delete a payload. Defeating the
|
||||
verifier does not restore forensic deniability; see
|
||||
[synthid.md](synthid.md#23-removal-attacks-and-forensic-detectability).
|
||||
|
||||
Official `not_detected` on collage, photo-pad, and two-panel layouts is a
|
||||
presentation gate, not residual damage. Those rungs are in
|
||||
[detector research](synthid-detector-research.md).
|
||||
|
||||
## Closed quiet removers
|
||||
|
||||
| Attack | Close | Notes |
|
||||
| --- | --- | --- |
|
||||
| Quality-preserving photo remover | 2026-08-20 | Every residual `not_detected` that is not a collage is below usable quality |
|
||||
| Additive in-band jam as a quiet remover | 2026-08-21 | Fish flips only at 18.6 dB; 4-8 px at the same PSNR stays `detected` |
|
||||
| Bayer bilinear / VNG / upscale-Bayer | 2026-08-23 | s1/s2/s3 and fish still `detected`. s3 bilinear 37.1 dB, fish 34.7 dB. VNG on s2 is dirtier than scramble (18.5 vs 19.0 dB) and the mark remains |
|
||||
| Barrel k1=0.06, scanline ±0.8 px, and shift 2 px | 2026-08-23 | s1/s2/s3/fish `detected` even at 14-20 dB barrel. Fish scanline 32.5 dB and shift 22.7 dB still `detected` |
|
||||
| TrustMark-style micro-warp | 2026-08-21 | 0.25 px / 32 dB still 100% detect on TrustMark P; OpenAI elastic ~21 dB unreplicated |
|
||||
| OKLab random-codeword replacement as a quiet wipe | 2026-08-15 | On four public Google-oracle positives, replacing the period-16 tile at 0.95 dropped the *local* lattice score below 0.173 at ~54 dB. Gemini pixel verify never accepted the candidates (`Connecting to Verify AI`). Local-score intervention, not an official wipe |
|
||||
| PRC-style OKLab a/b noise plus spectral peak cleaning | 2026-08-15 | Independent high-pass OKLab noise at 32.6 dB moved the median local score by only `-0.0219`. Comb suppression moved one of four below threshold at 58 dB. Not a quiet official kill |
|
||||
| Rinne414 fixed-template subtraction | 2026-08-24 | The pinned pre-May GPT-Image2 residual was resized onto a current verified OpenAI image and subtracted at 0.25x, 0.5x, 1x, and 2x. Official SynthID stayed `detected` at 43.13, 37.49, 31.60, and 25.72 dB after metadata stripping. Even the dirtiest rung is no better than the existing 24-25 dB annulus-phase scramble. The template can spoof its own public correlator but does not cancel the current mark |
|
||||
| `newideas99` CNN gradient | 2026-08-24 | A 3x3 crop-grid objective spread the white-box perturbation over the full marked frame. The frozen ensemble fell from 0.99995 to 0.25956 at ±2/255 (45.92 dB) and to effectively zero at ±4/255 (40.30 dB). Official OpenAI SynthID stayed `detected` at ±1, ±2, ±4, and ±8/255, through 34.30 dB. The gradient attacks the surrogate's shortcut, not the production mark |
|
||||
| `reverse-SynthID` V4 Round-06 | 2026-08-24 | The advertised `final`/`nuke` path is SD-VAE regeneration plus elastic and affine warps, resize squeeze, color change, residual FFT subtraction, and a JPEG/noise chain, with PSNR floors of only 14/11 dB. The repository claims 20 manual Gemini-app successes but contains no manifest, tally, or per-image verdicts. Its four bundled older cleaned pairs are 45.7-50.2 dB, yet the reproduced V4 score rises on three and is nearly unchanged on the fourth. No callable Google pixel oracle was available for an independent Round-06 verdict. This is an unverified lossy regeneration/distortion stack, not a quiet pattern cancellation |
|
||||
| [`0xROOTPLS/DeSynth`](https://github.com/0xROOTPLS/DeSynth/tree/96db920731c2a3d04bf13163a5077b9a67706d1b) | 2026-08-24 | Qwen Image img2img at strength 0.25, followed by a Gaussian frequency split that restores the source high frequencies above sigma 1.95. The repository's public OpenAI original, default output, and edge-guided output all returned official `detected` in a current repeat after metadata stripping. The published `not found` claim does not reproduce against the current OpenAI oracle |
|
||||
| [`froggeric/gemini-watermark-and-synthid-remover`](https://github.com/froggeric/gemini-watermark-and-synthid-remover/tree/5918384ce403968de0560cefd889e50eba0163bc) | 2026-08-24 | SDXL img2img with a documented manual Google-verifier ladder. The author reports 7/8 clears at strength 0.08 and 9/9, including a double mark, at strength 0.10 with five effective denoise steps and PSNR 29-41 dB. The exact nine before/after verdict artifacts are not tracked, so this is useful external regeneration corroboration, not an independently reproduced oracle result |
|
||||
| [`atomantic/PortOS`](https://github.com/atomantic/PortOS/tree/b11a93e110262925c64a1b145a154ca87b340055) adversarial-jamming experiment | 2026-08-24 | Its own one-image manual OpenAI run found that quality-preserving phase noise, band noise, blur, and 0.70 resize squeeze stayed detected. Only visibly destructive phase perturbation cleared. A 0.85-0.90 resize caused repeated detector timeouts, which the repository correctly keeps separate from `not_detected`. This independently closes high-fidelity additive/phase jamming, but the source artifacts are not published |
|
||||
| Generic regeneration claims | 2026-08-24 | `mertizci/noai-watermark`, `BovineOverlord/Loyal-Bear`, `obaskly/NeuralBleach`, and `tymongumienik/unwatermark` are SD/CtrlRegen/ControlNet redraws. Their algorithms are already covered by the product's regeneration remainder and the CtrlRegen literature. README verdicts without a frozen manifest do not establish a quieter or more reliable operating point |
|
||||
| Add unmarked context (frame, pad, hstack, collage) | 2026-08-19 | Official `not_detected` is a presentation gate. White pad 40% linear stays `detected`; photo fill at the same geometry is `not_detected`. See detector identity/layout |
|
||||
| `cebeuq/Synthid-Bypass` diffusion reconstruction | 2026-08-13 | 12 public pairs. Lattice and Gemini Verify go quiet on the *after* images because the image was redrawn, not because a payload was deleted. Faces and text are a quality claim for regeneration, not a local wipe |
|
||||
|
||||
Oracle: `verify_openai_synthid` after AI-metadata strip. Seeds s1, s2, s3
|
||||
as in [detector research](synthid-detector-research.md). Raw files:
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/`.
|
||||
|
||||
## Band that actually carries the mark
|
||||
|
||||
Phase-randomize one octave at a time, preserve energy (E2, s1, replicated
|
||||
p16_32 on s2 and s3):
|
||||
|
||||
| Octave (period px) | Verdict |
|
||||
| --- | --- |
|
||||
| 2-4 | detected |
|
||||
| 4-8 | detected |
|
||||
| 8-16 | detected |
|
||||
| 16-32 | not_detected |
|
||||
| 32-64 | detected |
|
||||
| 64-128 | detected |
|
||||
|
||||
Destroying only 16-32 px periods silences the official decoder. The
|
||||
lattice band 8-16 px does not. On a marked gray flat the same scramble is
|
||||
51.1 dB `not_detected` because the band is almost empty. On a photograph
|
||||
the band holds the scene (24.3 dB). Replacing native 16-32 with a
|
||||
2x-pyramid prediction is 48.4 dB and still `detected`.
|
||||
|
||||
Fish additive jam, luma RMS `a`:
|
||||
|
||||
| Band | a | PSNR | Verdict |
|
||||
| --- | ---: | ---: | --- |
|
||||
| 16-32 | 0.06-4.0 | 51.2-33.6 | detected |
|
||||
| 16-32 | 8 | 27.7 | detected |
|
||||
| 16-32 | 16 | 21.9 | detected |
|
||||
| 16-32 | 24 | 18.6 | not_detected |
|
||||
| 4-8 | 16 | 21.9 | detected |
|
||||
| 4-8 | 24 | 18.6 | detected |
|
||||
|
||||
Preregistered close `a*_photo < 0.53` failed (`a* > 16`).
|
||||
|
||||
## Bayer and geometry (2026-08-22)
|
||||
|
||||
Frozen one-pass batch. Lattice scores on these OpenAI sources are already
|
||||
`indeterminate` and are not the label.
|
||||
|
||||
| Attack | s1 | s2 | s3 | fish |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| source | detected | detected | detected | detected |
|
||||
| Bayer bilinear | 32.7 detected | 28.4 detected | 37.1 detected | 34.7 detected |
|
||||
| Bayer VNG | 25.0 detected | 18.5 detected | 26.4 detected | 25.4 detected |
|
||||
| upscale 1.15 then Bayer | 33.9 detected | 29.8 detected | 38.4 detected | 36.0 detected |
|
||||
| barrel k1=0.06 | 19.9 detected | 14.2 detected | 20.2 detected | 18.5 detected |
|
||||
| scanline ±0.8 px | 31.5 detected | 26.7 detected | 35.7 detected | 32.5 detected |
|
||||
| shift 2 px | 23.1 detected | 15.9 detected | 24.7 detected | 22.7 detected |
|
||||
| 16-32 phase scramble | 24.6 not_detected | 19.0 not_detected | 25.2 not_detected | 23.2 not_detected |
|
||||
|
||||
Camera pipeline and mild geometry do not hit the decoder basis. A 2 px
|
||||
shift, which kills `pipeline_lattice`, left s1/s2/s3/fish `detected`.
|
||||
Scramble remains the residual kill on 4/4 sources.
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/bayer-geometry-2026-08-22/`.
|
||||
Manifest `status` is `oracle_complete` (2026-08-22 22:07).
|
||||
|
||||
## Attack ladders (OpenAI oracle)
|
||||
|
||||
Pilot 2026-08-17 (n=3): JPEG q40, 0.5x, noise8+jpeg85, 0.35x+jpeg50 all
|
||||
`detected` (s2 0.35x+jpeg50 at 20.1 dB still detected).
|
||||
|
||||
Kill and flip (s1), then replication 3/3:
|
||||
|
||||
| Rung | s1 | s2 | s3 |
|
||||
| --- | --- | --- | --- |
|
||||
| blur sigma 6 | detected | detected | detected |
|
||||
| blur sigma 7 | not_detected | not_detected | not_detected |
|
||||
| keep 0.50 linear | detected | detected | detected |
|
||||
| keep 0.40 linear | not_detected | not_detected | not_detected |
|
||||
| elastic ~24 dB | detected 24.6 | detected 17.9 | detected 26.1 |
|
||||
| elastic ~21 dB | not_detected 20.8 | not_detected 14.8 | detected 22.0 |
|
||||
| pad white 40% linear | detected | detected | detected |
|
||||
| hstack with photo | not_detected | not_detected | not_detected |
|
||||
| blend 25% marked | detected | detected | detected |
|
||||
| inset native 1:1 on 2x photo | not_detected | not_detected | not_detected |
|
||||
| JPEG q5 | detected | detected | detected |
|
||||
|
||||
Holds on 3/3: blur 6 vs 7, crop keep 0.50 vs 0.40, white pad 40%
|
||||
`detected`, hstack `not_detected`, blend 25% `detected`, JPEG q5
|
||||
`detected`. Elastic at ~21 dB does not hold. Crop keep-0.40 and hstack
|
||||
are presentation or support, not a quiet residual wipe.
|
||||
|
||||
Surround: photo fill at the white-pad geometries is `not_detected` on
|
||||
3/3; white pad at those sizes was `detected`. Unmarked photo strip flips
|
||||
between 25% width (`detected`) and 35% (`not_detected`). Photo frame 15%
|
||||
each side is `not_detected` on 3/3; 12% does not hold.
|
||||
|
||||
## Wild removers (oracle 2026-08-23)
|
||||
|
||||
Preregistered on s1, one pass, 25 s gap. Manifest:
|
||||
`.local-eval/synthid/prc-oklab-attack-2026-08-15/wild-attacks-2026-08-23/oracle-manifest.json`.
|
||||
|
||||
| Attack | PSNR | Verdict | Note |
|
||||
| --- | ---: | --- | --- |
|
||||
| Replace 16-32 with a COCO photo's 16-32 | 25.2 | not_detected | Same annulus as cartesian scramble (24.6 dB). Substituting a camera band kills the decoder, 0.6 dB quieter than scramble on s1, not a new quality class |
|
||||
| File named polar-1632 | 25.6 | not_detected | Misnamed. The raster randomizes cartesian FFT phase in the 16-32 annulus, same family as scramble (24.6 dB). It is a scramble replicate, not a `(r, θ)` polar test |
|
||||
| 64-color median-cut palette | 35.6 | detected | Quieter than scramble and still marked. Posterize-4 already stayed `detected`; a smarter quantizer is not enough |
|
||||
| PIL RGB to CMYK to RGB | inf | not submitted | No-op on this PNG |
|
||||
|
||||
The 16-32 kill is luma phase in that annulus. Cartesian, Fourier-angle,
|
||||
and radial-phase all silence the decoder near 25 dB. A 90° sector of the
|
||||
same ring does not (27.5 dB `detected`). Cb-only and Cr-only 16-32
|
||||
scrambles stay `detected` at 44-45 dB. A foreign-scene transplant of the
|
||||
same band also silences the decoder near 25 dB. Palette, Baker-map, seam
|
||||
carve, Poisson, nested LSB, and ICC do not.
|
||||
|
||||
Follow-up 2026-08-23, s1, 25 s gap, `remaining-2026-08-23/oracle-manifest.json`:
|
||||
|
||||
| Attack | PSNR | Verdict |
|
||||
| --- | ---: | --- |
|
||||
| Fourier-angle 16-32 scramble | 24.1 | not_detected |
|
||||
| Cartesian 16-32 scramble (replicate) | 24.8 | not_detected |
|
||||
| Baker-map of the 16-32 band | 27.8 | detected |
|
||||
| Poisson noise | 30.7 | detected |
|
||||
| Nested LSB in blue | 55.9 | detected |
|
||||
| Seam carve 8 | 27.7 | detected |
|
||||
| ICC sRGB rewrite | inf | not submitted, no-op |
|
||||
|
||||
Non-local codecs, 2026-08-23, s1, 25 s gap:
|
||||
|
||||
| Attack | PSNR | Verdict |
|
||||
| --- | ---: | --- |
|
||||
| HEIF q80 | 46.3 | detected |
|
||||
| HEIF q50 | 39.3 | detected |
|
||||
| AV1 CRF 32 | 37.2 | detected |
|
||||
| Print-scan simulation | 24.95 | detected |
|
||||
|
||||
Physical print-scan is still blocked unattended (Brother DCP-L2520DW idle and
|
||||
accepting, no `scanimage`, no ImageCapture pyobjc). Face-gated scramble is
|
||||
unnecessary: Haar on s1 put *more* 16-32 energy on faces. Generic 25 dB is
|
||||
not the kill: this simulation stays `detected` at the PSNR where 16-32
|
||||
phase scramble does not.
|
||||
|
||||
Waveform-shell splits, 2026-08-23, s1, 25 s gap,
|
||||
`waveforms-shells-2026-08-23/oracle-manifest.json`:
|
||||
|
||||
| Attack | PSNR | Verdict |
|
||||
| --- | ---: | --- |
|
||||
| Y-only 16-32 scramble | 24.6 | not_detected |
|
||||
| Cb-only 16-32 scramble | 45.0 | detected |
|
||||
| Cr-only 16-32 scramble | 43.9 | detected |
|
||||
| 90° Fourier sector of 16-32 | 27.5 | detected |
|
||||
| Radial-phase-only 16-32 | 25.0 | not_detected |
|
||||
|
||||
## External literature (surveyed 2026-08-23)
|
||||
|
||||
Primary sources. Detector papers live in
|
||||
[detector research](synthid-detector-research.md). Forensic stealth of
|
||||
regeneration is already in [synthid.md](synthid.md#23-removal-attacks-and-forensic-detectability).
|
||||
|
||||
| Source | Attack | Against SynthID? | Map to this campaign |
|
||||
| --- | --- | --- | --- |
|
||||
| Zhao et al., [arXiv:2306.01953](https://arxiv.org/abs/2306.01953) (NeurIPS 2024) | Add noise, then denoise or regenerate (VAE / diffusion). Pixel-level invisible marks are provably removable. Semantic watermarks proposed as the alternative | Open post-hoc schemes, not production SynthID | This is the family our product uses (`qwen-zimage` / `sdxl-zimage`). Gowal trains SynthID-O against *weak* VAE regeneration. Our foreign-VAE round-trip at 22.3 dB stayed `detected`. Regeneration works when it redraws, not when it is a light codec |
|
||||
| Liu et al., [arXiv:2410.05470](https://arxiv.org/abs/2410.05470) (CtrlRegen, ICLR 2025) | Controllable diffusion from clean noise, with a knob on how many noise steps to add | SOTA open watermarks | Same family. Goonatilake later finds CtrlRegen+ the *most* forensically detectable remover (AUROC 0.9999) |
|
||||
| Kassis and Hengartner, [arXiv:2405.08363](https://arxiv.org/abs/2405.08363) (UnMarker, IEEE S&P 2025) | No decoder feedback. Two adversarial spectral optimizations. Breaks even some semantic watermarks (best remaining detection 43%) | Not production SynthID | Spectral disruption without an oracle is the honest analog of our 16-32 scramble, except UnMarker is optimized and we used a one-octave phase shuffle. Goonatilake: UnMarker TPR 98.28% at 0.1% FPR as a *forensic* leftover |
|
||||
| Tallam et al., [arXiv:2505.08234](https://arxiv.org/abs/2505.08234) (SemanticRegen) | Partial, label-free regeneration of main objects | Tree-Ring, StegaStamp, StableSig, DWT/DCT. Not SynthID | Partial redraw. Our collage / photo-pad `not_detected` is a presentation gate, not this attack |
|
||||
| Cao et al., [arXiv:2608.10166](https://arxiv.org/abs/2608.10166) (MarkNull, USENIX Security 2026) | On-manifold latent decorrelation via a public diffusion proxy. Claims 100% on 20 Imagen-3 Gemini-verify images. PSNR 25.36 dB, SSIM 0.80 | Small Gemini-verify set | Independent evidence that a no-box latent reconstruction can confuse Gemini. Does not meet this project's 40 dB / 0.99 SSIM release gate. Still generation, not a pixel-only wipe |
|
||||
| Goonatilake and Ateniese, [arXiv:2605.09203](https://arxiv.org/abs/2605.09203) | Six removers all leave a forensic residue a ResNet-50 sees at >98% TPR @ 1% FPR | Applies to UnMarker, Zhao's WatermarkAttacker, CtrlRegen+ | Defeating `verify-openai-synthid` is not deniability. This is the product remainder |
|
||||
| An et al., [arXiv:2401.08573](https://arxiv.org/abs/2401.08573) (WAVES, ICML 2024) | 26 attacks on StegaStamp, Stable Signature, Tree-Ring. Regeneration, not JPEG, is the attack that matters. StegaStamp TPR@1%FPR 1.00 to 0.01; Tree-Ring 0.99 to 0.12 | Open watermarks | Protocol. Our blur-sigma-7 and 16-32 scramble are closer to WAVES "distortion" than to regeneration |
|
||||
| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring) | In-generation Fourier pattern in initial noise | Not SynthID | Kill is DDIM inversion plus latent wipe, not a pixel scramble. Lin and Juarez [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) remove it from public knowledge |
|
||||
| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tuned VAE decoder | Not SynthID | Regeneration that replaces the decoder is in-family. Our foreign VAE at 22.3 dB did not kill OpenAI SynthID, which is the expected mismatch |
|
||||
| Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | reverse-SynthID does not systematically remove SynthID | Google spokesperson on the spectral bypass | Agrees with our V4 audit and with Bypass: reconstruction or decoder confusion, not a payload delete |
|
||||
|
||||
Quiet pixel-only removal is not a published result on production
|
||||
OpenAI or Google SynthID. Every paper that "beats" a watermark either
|
||||
redraws the image, attacks an open encoder, or reports PSNR in the same
|
||||
19-25 dB band as our 16-32 scramble.
|
||||
|
||||
## Product remainder for removal
|
||||
|
||||
Invisible removal in this package regenerates through `qwen-zimage` or
|
||||
`sdxl-zimage`. It is lossy. There is no shipped pixel-only OpenAI or
|
||||
Google SynthID wipe. Do not add Bayer or geometry as remover arms.
|
||||
@@ -1,8 +1,12 @@
|
||||
# SynthID: technical reference
|
||||
|
||||
> Technical research reference. Current package behavior is defined by the
|
||||
> Technical research reference for the mark itself and for what this package
|
||||
> ships. Current package behavior is defined by the
|
||||
> [supported signals](supported-signals.md), [known limitations](known-limitations.md),
|
||||
> and [module internals](module-internals.md). Dated measurements below are
|
||||
> and [module internals](module-internals.md). Campaign results are split:
|
||||
> [local detector](synthid-detector-research.md),
|
||||
> [SynthID source classifiers](synthid-classifiers.md),
|
||||
> [mark removal](synthid-removal-research.md). Dated measurements below are
|
||||
> historical evidence and should not be read as current CLI defaults.
|
||||
|
||||
This document covers how Google SynthID for images works mechanically, what it
|
||||
@@ -42,11 +46,58 @@ after generation, by a separate neural network. This means:
|
||||
- Replacing the generative model does not remove the watermarking capability.
|
||||
- The encoder/decoder pair can be updated independently of the generative model.
|
||||
|
||||
The detector's training target is narrower than provider or AI-image
|
||||
classification. Equation 1 trains on watermarked `f(x)` as positive and the
|
||||
corresponding unwatermarked distribution `x` as negative, with both drawn from
|
||||
the same target image distribution. Equation 2 applies the same sampled
|
||||
semantics-preserving transformation to both sides. Unrelated generators and
|
||||
camera images are useful later for false-positive evaluation, but they do not
|
||||
replace source-matched clean examples during signal-identifiability training.
|
||||
|
||||
The decoder produces a dedicated detection logit whose threshold is calibrated
|
||||
after training for a target false-positive rate. Payload recovery is a separate
|
||||
problem, not the definition of the presence score. The paper additionally
|
||||
describes two-sided conformal calibration for the `not watermarked` and
|
||||
`watermarked` hypotheses, which permits a detector to abstain instead of forcing
|
||||
an unsupported binary verdict.
|
||||
|
||||
The paper does not disclose the internal architecture of the encoder/decoder
|
||||
networks (layer types, capacity). The external variant SynthID-O is available
|
||||
to partners; the production internal variant is not published.
|
||||
|
||||
### 1.2 How it differs from classical DWT-DCT watermarks
|
||||
### 1.2 Patent-backed architecture clues
|
||||
|
||||
A related [DeepMind patent family](https://patents.google.com/patent/US12094474B1/en),
|
||||
filed in 2023 and naming several authors of the SynthID-Image paper, describes
|
||||
the likely architectural design space in more detail. Patent alternatives are
|
||||
not proof that every option is deployed in production SynthID, so the following
|
||||
items are constraints and research clues rather than implementation claims.
|
||||
|
||||
The patent's image example forms a content-dependent residual, `x' = x + g(x)`.
|
||||
It describes a U-Net-like watermark generator with convolution, attention, and
|
||||
skip connections; a separately trained convolutional decoder; optional message
|
||||
or secret-key injection into intermediate layers; and paired training on clean
|
||||
and watermarked images under sampled differentiable transformations. It also
|
||||
allows an image to be resized to a trained target size for watermarking and then
|
||||
resized back to its original dimensions.
|
||||
|
||||
Most importantly for blind detection, a
|
||||
[continuation patent](https://patents.google.com/patent/US20250149048A1/en)
|
||||
describes groups or ensembles of paired encoder/decoder networks. A decoder can
|
||||
be trained not to recognize marks from another pair, while deployment can
|
||||
select one pair or combine several outputs. This supplies a concrete mechanism
|
||||
for coexisting watermark versions, provider-specific states, or distinct
|
||||
codewords without requiring one universal pixel template.
|
||||
|
||||
The fixed periodic carriers measured in this project should therefore be
|
||||
treated as linear experts for particular observable states, not as the
|
||||
definition of SynthID. A broader detector needs independently validated experts
|
||||
for additional encoder states and transformations, followed by joint
|
||||
false-positive calibration. The paper's separate detection logit and two-sided
|
||||
conformal decision provide the correct target behavior: present, absent, or
|
||||
abstain when neither hypothesis is supported.
|
||||
|
||||
### 1.3 How it differs from classical DWT-DCT watermarks
|
||||
|
||||
The open watermarks used by Stable Diffusion / SDXL / FLUX (via the
|
||||
`imwatermark` library) use classical **DWT-DCT** frequency-domain embedding: a
|
||||
@@ -70,7 +121,7 @@ embeds in fixed, predictable frequency bins, making it brittle to any
|
||||
operation that hits those bins (e.g., JPEG re-quantization wipes it cleanly at
|
||||
quality <= 90).
|
||||
|
||||
### 1.3 Payload capacity
|
||||
### 1.4 Payload capacity
|
||||
|
||||
SynthID-O (the external/partnership variant) encodes:
|
||||
|
||||
@@ -90,7 +141,7 @@ paper separates watermark **detection** (is this watermarked?) from payload
|
||||
**recovery** (what does the payload say?): the detection path is what oracles
|
||||
like the Gemini app's "Verify with SynthID" exercise.
|
||||
|
||||
### 1.4 Where in the pipeline it lives
|
||||
### 1.5 Where in the pipeline it lives
|
||||
|
||||
```
|
||||
[Diffusion model]
|
||||
@@ -176,6 +227,232 @@ conversion (the body text of Section 10 is absent from the HTML).
|
||||
|
||||
**What is known empirically from our own oracle-verified testing.**
|
||||
|
||||
A 2026-08-09 non-generative pilot found a promising Google phase-correlate,
|
||||
but did not establish a releasable local detector or pixel-only remover.
|
||||
The 2026-08-20/22 OpenAI campaign is recorded in
|
||||
[synthid-detector-research.md](synthid-detector-research.md),
|
||||
[synthid-classifiers.md](synthid-classifiers.md), and
|
||||
[synthid-removal-research.md](synthid-removal-research.md).
|
||||
JPEG q5 and 16-32 px phase structure survive as the official mark; a
|
||||
quality-preserving local remover and a local SynthID detector for
|
||||
photographs were not found. `gpt-image-2` source-matched flat pairs exist;
|
||||
their residual does not transfer to photographs. The
|
||||
best independently fitted spectral model relearned phase and magnitude from four of our
|
||||
positives while using third-party candidate coordinates; its second frozen
|
||||
epoch had zero false positives on 279 new exact-size external images and
|
||||
detected the one new confirmed positive used for validation. An additional
|
||||
3,000 upscaled photographs all fell outside the measured active-carrier support
|
||||
and therefore count as abstentions, not negatives. The corpus is still far too
|
||||
small and lacks same-provider hard negatives needed for a 0.1% FPR claim.
|
||||
Repeating the spectral fit in six color spaces produced zero false positives
|
||||
on the same 279-image comparison set for every branch. HSV had the best
|
||||
observed worst-negative margin, but paired normalized negative scores did not
|
||||
show a general improvement over RGB. Its apparent benefit came from saturation
|
||||
and value; hue failed its channel-level separation check. Luminance-like
|
||||
channels dominated YCbCr, YCoCg, opponent, and Lab. This narrows the next
|
||||
hypothesis to intensity/contrast and HSV S+V projections rather than a
|
||||
hue-specific carrier, but it does not add independent positives or certify an
|
||||
operating point.
|
||||
The resulting positive-only RGB plus S/V ensemble passed four
|
||||
leave-one-positive-out checks, detected all five available positive controls,
|
||||
and emitted no positive verdict on 330 newly collected exact-size images.
|
||||
Almost all external images lacked sufficient measured carrier support and
|
||||
therefore remained abstentions rather than proven negatives.
|
||||
A later post-freeze challenge used three additional exact-size Google originals
|
||||
with signed Google LLC C2PA and explicit SynthID-present assertions. The
|
||||
ensemble abstained on all three. They do not replace matching-oracle pixel
|
||||
labels, but this zero-of-three source-provenance result rejects the frozen
|
||||
ensemble as a general Google detector and narrows it to an epoch- or
|
||||
surface-specific correlate.
|
||||
|
||||
A multi-epoch leave-one-positive-out refit recovered six of the eight total
|
||||
Google positives under a strict RGB-plus-HSV decision; RGB alone ranked every
|
||||
excluded positive above the 50-image calibration maximum but produced up to
|
||||
eight false positives on 279 held-out external negatives. The signal therefore
|
||||
transfers across the two source groups, but remains content-sensitive and
|
||||
cannot be separated from provider or generation-pipeline attribution without
|
||||
same-provider oracle-negative controls.
|
||||
|
||||
Directly projecting out the ensemble phases cleared the local detector above
|
||||
51 dB PSNR, but three frozen candidates remained detected by Gemini in a
|
||||
healthy control session. A subsequent spatial analysis found that 81.25% of
|
||||
the top 256 carrier bins lie on a lattice corresponding to a repeating 16x32
|
||||
pixel cell. Modulo-folding and subtracting the complete high-pass tile gave
|
||||
phase-specific matched-control separation across all five Google positives at
|
||||
at least 55.67 dB PSNR: aligned outputs cleared the local ensemble, while
|
||||
one-pixel-shifted controls remained positive. The mild locally clearing tile
|
||||
candidate nevertheless remained detected in a healthy Google Verify AI
|
||||
session, so the tile correlate is not a sufficient removal loss.
|
||||
Sparse subtraction reduced the local score sharply at more than 50 dB PSNR,
|
||||
yet healthy Gemini sessions still detected SynthID. A one-off negative was
|
||||
discarded because the same session also missed the source-positive control.
|
||||
An OpenAI-specific 8-pixel phase candidate likewise changed its local score at
|
||||
46.70 dB PSNR but remained `SynthID detected` in one frozen OpenAI Verify
|
||||
check. No Google threshold or carrier was used in that experiment, and no
|
||||
further public OpenAI checks were made because the documented verifier guidance
|
||||
rules out repeated watermark-removal queries.
|
||||
|
||||
A later local EOT pilot combined periodic full-frame residuals with JPEG-aware
|
||||
optimization against provider-specific CNN surrogates. It produced candidates
|
||||
that stayed below three local models after actual JPEG round trips, but the
|
||||
models remain provider classifiers without same-provider oracle-negative
|
||||
controls. The selected Google candidate retained 36.80 dB PSNR and 0.9241
|
||||
SSIM; the selected OpenAI candidate retained 34.97 dB and 0.9356. Both miss the
|
||||
research fidelity gate. The OpenAI periodic residual also reduced all three
|
||||
local model scores on 11 of 11 additional images before and after JPEG-90, but
|
||||
neither provider candidate has a negative matching-provider oracle verdict, so
|
||||
neither is an established remover.
|
||||
|
||||
A later paired open-method control tested stationary wavelet and complex
|
||||
spectral representations without treating another watermark as SynthID. On a
|
||||
source-disjoint TrustMark P proxy, the SWT branch moved 59 of 60 transformed
|
||||
clean/watermarked pairs in the correct direction but did not improve the
|
||||
spatial model's aggregate AUC. Complex FFT maps were indistinguishable from
|
||||
chance, and fixed phase scoring collapsed under crop. Wavelets therefore remain
|
||||
a gated late-fusion ablation, while complex spectral analysis remains a
|
||||
residual-discovery tool rather than a presence score.
|
||||
|
||||
Expanding that proxy to four causally paired watermark families did not produce
|
||||
a universal residual detector. TrustMark P, VideoSeal Y-256, DWT-DCT, and WAM
|
||||
each used 90 source-matched images and random payloads; official decoders
|
||||
validated which transformed positives still carried readable marks. A fixed
|
||||
steganalysis residual bank and a neural model trained with cross-family
|
||||
residual mixing, sign and phase augmentation, and equal-power phase-scrambled
|
||||
hard negatives were evaluated leave-one-family-out. The four unseen-family
|
||||
AUCs were 0.562, 0.474, 0.497, and 0.437. A network can therefore learn each
|
||||
available encoder without learning a transferable class called "neural
|
||||
watermark." This open-proxy route is not evidence for Google or OpenAI SynthID.
|
||||
|
||||
The same conclusion held on a larger temporally split closed corpus. A
|
||||
three-seed normalized-patch ensemble reached 0.884 AUC, but its threshold above
|
||||
all validation negatives produced a test false positive and detected only
|
||||
2.45% and 8.97% of the two positive-source classes. Naive full-field pooling,
|
||||
late-fused SWT, robust patch aggregation, and a separate binary detection head
|
||||
did not improve that low-FPR operating point. Vendor classification was highly
|
||||
accurate among accepted positives, but it cannot rescue an unreliable presence
|
||||
decision. Source labels therefore remain suitable for vendor triage and hard
|
||||
negative evaluation, not for establishing a SynthID detector without
|
||||
counterfactual or oracle watermark labels.
|
||||
|
||||
A later low-content matrix also rejected wavelet energy and normalized
|
||||
wavelet-shape classifiers: they separated real target outputs from artificial
|
||||
flat, Gaussian, and phase-random controls, then accepted 61.6-100% of real
|
||||
external negatives. The surviving branch is narrower. A provider-specific
|
||||
1536x2816 phase carrier detected all four temporal-test positives, bounded
|
||||
translation registration recovered all four one-pixel shifts, and the joint
|
||||
phase/support rule produced zero positives on 194 frozen negatives. A
|
||||
scale-and-translation discovery rule also produced zero positives on a
|
||||
preregistered 3,000-image COCO challenge, but every COCO image was outside
|
||||
carrier support and the scale threshold was selected post hoc. The result is
|
||||
therefore a positive-only, geometry- and epoch-specific expert with abstention,
|
||||
not a universal SynthID detector. Exact measurements and remaining calibration
|
||||
gates are in the
|
||||
[`detector and removal research plan`](synthid-detector-removal-plan.md#2026-08-10-low-content-controls-and-registered-phase-carrier).
|
||||
|
||||
The next native-geometry experiment isolated a stronger mechanism. At
|
||||
2048x2048, 108 selected spatial frequencies formed a 128-bin lattice, implying
|
||||
a 16x16 periodic residual tile. Folding and averaging 16,384 tile repetitions
|
||||
produced a spatial detector that accepted 29 of 30 test positives, none of 49
|
||||
calibration negatives, and none of 38 held-out test negatives. It also accepted
|
||||
the same two of 182 earlier external-source images as the independent RGB and
|
||||
HSV phase branches. Those cases count against operational source-label FPR,
|
||||
but may contain the same carrier through an upstream encoder; only an oracle
|
||||
can distinguish the two explanations. A normalized tile challenge accepted
|
||||
none of 3,000 general images, while symmetric attacks showed strong resize but
|
||||
limited JPEG and crop robustness. The pickle-free research implementation is
|
||||
`scripts/synthid_periodic_tile_probe.py`; exact evidence and caveats are in the
|
||||
[`2048 periodic-tile experiment`](synthid-detector-removal-plan.md#2026-08-10-2048-periodic-tile-detector).
|
||||
|
||||
The historical 2048 reports retain aggregate counts and model hashes, but not
|
||||
the 111 fitting paths. A later exact-1024 audit demonstrated that perceptual
|
||||
siblings can cross an image-level split despite different file and decoded-pixel
|
||||
hashes, so the old 2048 train/validation/test split cannot be retrospectively
|
||||
cleared of that leakage mode. Its external control challenges remain valid;
|
||||
its positive rates are conditional on the historical split. The next model
|
||||
calibration must preserve content-group membership across every partition.
|
||||
|
||||
An aligned-subtraction ablation strengthened the mechanism finding without
|
||||
clearing the oracle gate. At a discovery-selected amplitude, it reversed both
|
||||
the fixed-tile and independently fitted phase decisions on all 30 test images
|
||||
at a median 53.74 dB PSNR and 0.99681 SSIM. The aligned edit reduced both scores
|
||||
more than cyclic-shifted and orthogonal random tile controls on every paired
|
||||
image. A shifted tile nevertheless suppressed the phase score enough to leave
|
||||
only one accepted image, so these local reversals remain surrogate evidence,
|
||||
not verified SynthID removal. The exact controls and caveats are recorded in
|
||||
the linked experiment section.
|
||||
|
||||
A later adaptive projection reduced that local cost substantially. It repeats
|
||||
the frozen unit-norm 16x16 carrier over the decoded image and uses a bounded
|
||||
scalar search for the smallest subtraction amplitude that reaches a requested
|
||||
fixed-tile score. There is no generative model, spatial optimizer, or change of
|
||||
image dimensions. Among the 30 historical 2048x2048 ablation sources, 27 were
|
||||
positive under both the fixed-tile and independent phase rules before editing.
|
||||
Searching only the scalar amplitude cleared both rules on all 27 at a median
|
||||
59.83 dB PSNR and 0.99890 SSIM; the minima were 55.19 dB and 0.99705. Applying
|
||||
the same selected amplitudes to a one-pixel cyclic shift of the carrier cleared
|
||||
none of the 27. Of the 16 sources whose carrier survived an otherwise identical
|
||||
JPEG-95 round trip, aligned projection cleared all 16 after JPEG-95 while the
|
||||
shifted control cleared one.
|
||||
|
||||
The same native-period edit works without resizing when dimensions are not
|
||||
multiples of 16: repeat the carrier past both image boundaries and crop it to
|
||||
the decoded geometry. A deterministic development challenge selected three
|
||||
first-carrier-positive Spaces images from each of the 20 most frequent native
|
||||
geometries, for 60 images total. Searching to a zero fixed-tile target cleared
|
||||
60 of 60, compared with 5 of 60 norm-matched one-pixel-shifted controls. Median
|
||||
fidelity was 60.37 dB PSNR and 0.99919 SSIM; the minima were 53.56 dB and
|
||||
0.99581. A more conservative target score of -0.25 retained 60 of 60 native
|
||||
local clearances at median 57.70 dB PSNR and 0.99863 SSIM, with minima of
|
||||
52.38 dB and 0.99443.
|
||||
|
||||
The frozen -0.25 rule was then evaluated once on one deterministic first-carrier
|
||||
positive from every native geometry represented by that Spaces subset: 647
|
||||
images at 647 distinct decoded sizes. It reached the local target on all 647,
|
||||
with no maximum-amplitude failure. Median fidelity was 57.67 dB PSNR and
|
||||
0.99867 SSIM; the worst case was 51.78 dB and 0.99400. Among the 323 images
|
||||
whose source carrier survived JPEG-95, the aligned candidate cleared 323 of
|
||||
323 after the same codec round trip, compared with 58 of 323 one-pixel-shifted
|
||||
controls. At the deeper transform margin, shifted controls also clear more
|
||||
often natively, 153 of 647, so exact carrier phase remains causal but the edit
|
||||
is no longer phase-exclusive on every content. The all-geometry corpus was
|
||||
already inspected during detector development and contributes mechanism
|
||||
coverage, not an independent generalization split.
|
||||
|
||||
The conservative target also survived every tested transform for which the
|
||||
same transformed source remained locally positive: 32 of 32 JPEG-95, 13 of 13
|
||||
JPEG-90, 8 of 8 JPEG-85, 15 of 15 WebP-95, and 54 of 54 0.75x resize round
|
||||
trips. The shifted controls cleared 7, 1, 1, 5, and 17 of those respective
|
||||
source-positive subsets. These results establish fast, geometry-independent
|
||||
control over the recovered carrier and quantify the quality-versus-transport
|
||||
margin. They do not establish SynthID removal. The previous provider check
|
||||
showed that Google can remain positive after a locally favorable analytical
|
||||
edit, and this adaptive recipe has not received a matching-provider negative
|
||||
oracle verdict. It therefore remains a research candidate rather than a public
|
||||
removal path.
|
||||
|
||||
The reproducible local tool is
|
||||
`scripts/synthid_adaptive_carrier_suppress.py`. It accepts one locally positive
|
||||
image, writes a lossless PNG without overwriting existing files, and records
|
||||
the input/output hashes, target, selected amplitude, scores, fidelity, and
|
||||
runtime in JSON. Its default -0.25 target is the transform-margin setting above.
|
||||
The tool intentionally remains outside the public package CLI and refuses
|
||||
locally negative inputs; its output status names carrier suppression rather
|
||||
than provider-verified removal.
|
||||
|
||||
The immutable oracle-batch and result evaluator are implemented. On 2026-08-10,
|
||||
four new 2048x2048 Gemini images were generated after the rule was frozen and
|
||||
registered as a 20-request confirmatory batch. The first source group exhausted
|
||||
the account's verification quota after five requests. The untouched source
|
||||
returned Google C2PA Content Credentials without a separate SynthID verdict;
|
||||
the lossless re-encode, aligned subtraction, and cyclic-shifted control all
|
||||
still returned a Google AI signal. The orthogonal control was refused because
|
||||
the quota had been exceeded, and the remaining 15 requests were not submitted.
|
||||
This incomplete run is already negative evidence for the frozen pixel-only
|
||||
recipe: the aligned candidate cleared the phase detector but remained positive
|
||||
under the tile detector and the provider oracle. The local carrier expert
|
||||
therefore ships only as a positive-only, exact-geometry detector; it is not a
|
||||
universal SynthID detector or a remover.
|
||||
|
||||
A controlled study (June 2026, clean v0.8.6 with text/face protection OFF,
|
||||
native resolution on this repo's default SDXL pipeline) measured the minimum
|
||||
img2img strength that removes the SynthID pixel watermark, verified per image on
|
||||
@@ -186,14 +463,15 @@ Generated cleaned outputs are not committed; the table below is the durable
|
||||
record of the historical oracle verdicts. One third-party image from issue #14
|
||||
was oracle-verified but is not committed.
|
||||
|
||||
**Oracle validation order: start with OpenAI.** When validating removal across
|
||||
vendors, run the OpenAI arm first. `openai.com/verify` is more accessible than the
|
||||
Gemini app -- fewer per-check restrictions, so it gives the fastest signal and is
|
||||
the strongest candidate for automation (Playwright / Chrome MCP driving
|
||||
`openai.com/verify`); the Gemini "Verify with SynthID" flow is more manual. This is
|
||||
an ordering/throughput choice, not a substitution: each oracle only reads its own
|
||||
vendor's SynthID (`openai.com/verify` is OpenAI-scoped), so Google content still
|
||||
needs the Gemini app.
|
||||
**Historical oracle validation order: start with OpenAI.** The June study used
|
||||
the OpenAI web verifier first because it was more accessible than the Gemini app.
|
||||
OpenAI now documents a Content Provenance API, but its usage guidance explicitly
|
||||
rules out repeated queries for reverse engineering or watermark removal. New
|
||||
adaptive experiments require separate authorization and must follow the oracle
|
||||
boundary in the
|
||||
[`detector and removal research plan`](synthid-detector-removal-plan.md). This is
|
||||
not a cross-provider substitution: each oracle reads only its own vendor's
|
||||
SynthID, so Google content still needs the Gemini flow.
|
||||
|
||||
| Vendor | Images | Resolution(s) | Pipeline | Removed at |
|
||||
|--------|--------|---------------|----------|------------|
|
||||
@@ -265,11 +543,29 @@ notable because it is also the most powerful removal attack. The paper notes
|
||||
that diffusion regeneration "leaves a strong reconstruction signature from the
|
||||
diffusion prior."
|
||||
|
||||
A newer [MarkNull study](https://arxiv.org/abs/2608.10166), accepted at USENIX
|
||||
Security 2026, evaluates a no-box latent-space attack on 20 Imagen-3 images and
|
||||
checks the outputs with Gemini's SynthID verification flow. MarkNull, its
|
||||
amortized variant, and most advanced regeneration or adversarial baselines
|
||||
reached 100% attack success in that small evaluation; a simple VAE round trip
|
||||
reached 0%. The per-image method inverts the input with a clean public diffusion
|
||||
proxy, optimizes its latent to decorrelate the reconstructed initial noise, and
|
||||
constrains LPIPS, MSE, and SSIM before decoding it again.
|
||||
|
||||
This is important independent evidence that removal need not know SynthID's
|
||||
carrier or decoder, but it is still a generative latent reconstruction, not the
|
||||
pixel-only analytical path pursued here. The paper's milder SynthID setting
|
||||
reports PSNR 25.36 dB and SSIM 0.80 at 100% attack success. Those values do not
|
||||
meet this project's 40 dB median PSNR and 0.99 median SSIM release gate, despite
|
||||
the paper's favorable composite quality score. MarkNull-A's reported 0.50-second
|
||||
runtime and 6.3 GB VRAM are attractive as a future optional fallback, but they
|
||||
do not establish a fast, visually lossless pixel-only remover.
|
||||
|
||||
---
|
||||
|
||||
## 3. Detectability and verifier access
|
||||
|
||||
### 3.1 No public local detector
|
||||
### 3.1 No public payload decoder
|
||||
|
||||
The SynthID decoder is proprietary and not released:
|
||||
|
||||
@@ -278,18 +574,78 @@ The SynthID decoder is proprietary and not released:
|
||||
> available to trusted testers."
|
||||
> -- Gowal et al., arXiv:2510.09263
|
||||
|
||||
There is no public API, no released decoder weights, and no reproducible
|
||||
algorithm for local detection. The verification service (SynthID Detector) is:
|
||||
There are no released payload-decoder weights or public algorithm. Google
|
||||
provides verification in Gemini and a limited SynthID Detector
|
||||
portal. OpenAI now documents a synchronous Content Provenance API whose image
|
||||
response contains separate C2PA and SynthID outcomes. That API is a remote,
|
||||
OpenAI-scoped verifier, not a local decoder. Its documentation also says not to
|
||||
use repeated queries to reverse-engineer, remove, or evade a watermark, so an
|
||||
adaptive research loop requires separate authorization.
|
||||
|
||||
Google's SynthID Detector service is:
|
||||
|
||||
> "a verification portal" in early testing with "journalists and media
|
||||
> professionals" on a waitlist
|
||||
> -- deepmind.google/models/synthid/
|
||||
|
||||
The external variant SynthID-O is available "through partnerships" only. Our
|
||||
tool cannot locally detect SynthID presence or absence -- this is by design,
|
||||
not a gap we can fill.
|
||||
The external variant SynthID-O is available "through partnerships" only. This
|
||||
package does not ship a local payload decoder. Research on a periodic lattice
|
||||
expert is in [synthid-detector-research.md](synthid-detector-research.md).
|
||||
A 2026-08-23 literature map (Gowal, AWPD/FSNet, PRC, reverse-SynthID) is
|
||||
in that page's external-literature section. The product routes are signed
|
||||
provenance and `verify-openai-synthid`.
|
||||
|
||||
### 3.2 How our tool recognizes SynthID from provenance
|
||||
### 3.2 How our tool detects the supported carrier
|
||||
|
||||
This heading is kept because older README and CLI links pointed here. The
|
||||
package does not ship a local SynthID pixel detector. The periodic-lattice
|
||||
expert is research-only under `scripts/synthid_runtime/` and
|
||||
[synthid-detector-research.md](synthid-detector-research.md). It is not a
|
||||
payload decoder and is not called from `identify` or the CLI.
|
||||
|
||||
The product routes for SynthID are signed provenance, documented in the next
|
||||
sections, and `verify-openai-synthid`.
|
||||
|
||||
### 3.3 Official OpenAI pixel verification
|
||||
|
||||
`remove-ai-watermarks verify-openai-synthid image.png
|
||||
--acknowledge-upload` uses OpenAI's official Content Provenance API for the
|
||||
OpenAI half of the production detector. This is not the weak local period-8
|
||||
research expert and it does not use C2PA as a proxy.
|
||||
|
||||
Before making one request, the command strips AI provenance metadata into a
|
||||
temporary PNG, JPEG, or WebP file and compares hashes of the decoded RGBA raster
|
||||
before and after. It refuses the upload if any AI marker survived, if the format
|
||||
or pixels changed, or if the sanitized file exceeds 50 MiB. It then reads
|
||||
exactly one independent `type == "synthid"` response entry and ignores the C2PA
|
||||
entry. The source is not modified. Tests deliberately cover C2PA-only positive
|
||||
responses, pixel mutation, surviving metadata, malformed response shapes, and
|
||||
documented access and rate-limit failures.
|
||||
|
||||
The default SDK client fixes a 120-second request timeout and disables
|
||||
automatic retries. One explicit acknowledgement therefore cannot silently
|
||||
transmit the sanitized raster more than once. Timeout and connection failures
|
||||
are errors, and request logs omit source paths and decoded-pixel hashes.
|
||||
|
||||
A live 2026-08-14 web-verifier smoke used the same sanitization invariant. Two
|
||||
metadata-stripped, pixel-identical OpenAI images at 1536 by 1024 and 1024 by
|
||||
1536 both returned `SynthID detected` with `Content Credentials not detected`.
|
||||
An oracle-positive Google SynthID image and a COCO photograph, sanitized through
|
||||
the same path, both returned OpenAI `SynthID not detected` and no Content
|
||||
Credentials. This directly validates pixel-only and provider-specific behavior
|
||||
for the production design. Four files are only a functional smoke test, not a
|
||||
false-positive calibration, and the credentialed SDK endpoint itself was not
|
||||
called because no API key was available.
|
||||
|
||||
This backend is deliberately excluded from `identify`: verification uploads
|
||||
the sanitized raster to OpenAI, the endpoint is not eligible for Zero Data
|
||||
Retention, and explicit acknowledgement is required. It is suitable for
|
||||
individual supported OpenAI provenance checks, not adaptive detector fitting or
|
||||
removal search. The API documentation prohibits repeated queries for watermark
|
||||
reverse engineering or evasion. As with every detector in this project,
|
||||
`not_detected` is absence of recognized evidence, not proof of human authorship.
|
||||
|
||||
### 3.4 How our tool recognizes SynthID from provenance
|
||||
|
||||
We recognize SynthID indirectly from supported C2PA evidence; this is not a
|
||||
pixel watermark decode. Google states that all media generated by its tools is
|
||||
@@ -310,28 +666,26 @@ This is why:
|
||||
- A quiet `identify` output is not proof that SynthID was removed -- it only
|
||||
means the metadata signal is gone.
|
||||
|
||||
### 3.3 Oracle scope: each vendor detects only their own
|
||||
### 3.5 Oracle scope: each vendor detects only their own
|
||||
|
||||
From openai.com/research/verify (verbatim, verified 2026-05-31):
|
||||
|
||||
> "OpenAI generation signals will only be detected if the image was generated
|
||||
> with our tools."
|
||||
> "Content could also still be AI-generated by another company's model, which
|
||||
> the tool currently does not detect."
|
||||
OpenAI's current Content Provenance API documentation says it checks supported
|
||||
OpenAI signals and is not a general-purpose AI detector. Google's current Gemini
|
||||
documentation likewise says Gemini recognizes SynthID from Google AI tools,
|
||||
not other companies' payloads.
|
||||
|
||||
SynthID technology is used by multiple vendors, but each verifier is keyed to
|
||||
its own payload:
|
||||
|
||||
| Oracle | Detects | Does NOT detect |
|
||||
|-------------------------------|------------------|-------------------------|
|
||||
| Gemini app "Verify with SynthID" | Google SynthID | OpenAI SynthID |
|
||||
| openai.com/research/verify | OpenAI SynthID | Google SynthID |
|
||||
| Oracle | Detects | Does not detect |
|
||||
| --- | --- | --- |
|
||||
| Gemini app verification | Google SynthID | OpenAI SynthID |
|
||||
| OpenAI Content Provenance API or web verifier | Supported OpenAI SynthID | Google SynthID |
|
||||
|
||||
A Google-SynthID image reads clean on openai.com/verify. An OpenAI image reads
|
||||
clean in the Gemini oracle. They are different payloads within the same
|
||||
framework.
|
||||
|
||||
### 3.4 Video verification and attack harness
|
||||
### 3.6 Video verification and attack harness
|
||||
|
||||
Gemini's built-in verification flow reports whether and where it detects Google
|
||||
SynthID in a video. This remains a proprietary oracle: invoke `@synthid`, use
|
||||
@@ -560,7 +914,8 @@ photoreal, `sdxl` on flat graphics, the §5.1 content-x-pipeline table), BUT on
|
||||
hard case (flat fills) `sdxl` is the WEAKER remover (plain img2img barely perturbs a
|
||||
flat region at low strength), so it needs AT LEAST controlnet's strength -- the
|
||||
certified floor is therefore the right floor for `sdxl` too. This is a MARGIN argument
|
||||
for `sdxl`, not a separate certification (no local SynthID detector to self-verify).
|
||||
for `sdxl`, not a separate certification (the tested geometries are outside the
|
||||
current local detector's scope).
|
||||
The higher strength costs little quality where it matters, because `controlnet` is now
|
||||
the default pipeline, so `sdxl` is reached only via an explicit `--pipeline sdxl` (a
|
||||
deliberate opt-down), where over-regeneration has no faces/text to damage.
|
||||
@@ -620,8 +975,10 @@ Two constraints on reading this:
|
||||
subjects.** JPEG re-encoding removes C2PA metadata but does NOT remove the
|
||||
SynthID pixel watermark (verified June 2026 on issue #14 pic3). Do not
|
||||
dismiss these as "not faithful originals" for SynthID-removal tests.
|
||||
- **The correct oracle for OpenAI images is openai.com/verify**, not the Gemini
|
||||
app. The two oracles detect different payloads.
|
||||
- **The correct oracle for OpenAI images is an authorized OpenAI provenance
|
||||
verifier**, not the Gemini app. OpenAI now documents both a web tool and a
|
||||
Content Provenance API; the API's published use restrictions still apply.
|
||||
The OpenAI and Google oracles detect different payloads.
|
||||
- **A quiet `identify` output after processing is not proof of removal.** It
|
||||
means the provenance evidence is gone. The pixel watermark state is unknown without
|
||||
an oracle check.
|
||||
@@ -669,7 +1026,7 @@ random (unset) seed differed between runs. So **0.15 is the borderline floor for
|
||||
controlnet photoreal, not a robust guarantee**: at the threshold the same
|
||||
image+settings can pass or fail run-to-run. img2img runs with `seed=None` (random)
|
||||
unless `--seed` is passed, so a removal SERVICE gets a coin-flip near threshold and
|
||||
has no local SynthID detector to self-verify.
|
||||
has no applicable local SynthID detector at these geometries to self-verify.
|
||||
|
||||
**Controlnet strength ladder on the two photoreal images (oracle, `--auto`,
|
||||
`--max-resolution 1536`):**
|
||||
@@ -684,7 +1041,7 @@ has no local SynthID detector to self-verify.
|
||||
non-deterministic borderline); both photoreal survivors cleared at 0.20. Honest
|
||||
caveat: 0.20 is one confirming run WITH margin, not an N-run repeatability proof --
|
||||
for a removal service, add a little more margin or validate repeatability, since
|
||||
there is no local SynthID detector to self-check. **Implications:** (1) the
|
||||
these geometries are outside the current local detector's scope. **Implications:** (1) the
|
||||
content×pipeline table above conflates a borderline/non-deterministic 0.15 result
|
||||
with deterministic content behavior -- the photoreal-survives-controlnet effect is
|
||||
solid at 0.10 but at 0.15 it is near-threshold noise; (2) for reliable removal pick
|
||||
@@ -735,7 +1092,7 @@ a resolution trend here; measure it.
|
||||
**Current implication:** the old floor table remains evidence about the dated
|
||||
test set, not the current resolver. The SDXL and ControlNet profiles it measured
|
||||
no longer exist; the shipped defaults are defined in `watermark_profiles.py`, and
|
||||
both surviving profiles run face repair as a built-in second stage rather than as
|
||||
all three surviving profiles run face repair as a built-in second stage rather than as
|
||||
an optional restore. Removal near a threshold remains seed dependent, so
|
||||
reproducible verification requires a fixed seed.
|
||||
|
||||
@@ -756,8 +1113,8 @@ reproducible verification requires a fixed seed.
|
||||
Forensic Stealth in Generative-AI Watermark Removal.** arXiv:2605.09203.
|
||||
https://arxiv.org/abs/2605.09203
|
||||
|
||||
5. OpenAI. **Verify tool for AI-generated images.** openai.com/research/verify.
|
||||
Accessed 2026-05-31.
|
||||
5. OpenAI. **Content provenance.**
|
||||
https://developers.openai.com/api/docs/guides/content-provenance
|
||||
|
||||
6. Google. **Verify AI-generated images, videos, and audio.**
|
||||
https://support.google.com/gemini/answer/16722517
|
||||
@@ -770,6 +1127,99 @@ reproducible verification requires a fixed seed.
|
||||
Watermarking.** arXiv:2505.21620.
|
||||
https://arxiv.org/abs/2505.21620
|
||||
|
||||
9. OpenAI. **ChatGPT Images 2.0 system card.**
|
||||
https://deploymentsafety.openai.com/chatgpt-images-2-0/automated-evaluations-and-adversarial-testing
|
||||
|
||||
10. Cao et al. (2026). **MarkNull: Model-Agnostic Watermark Removal in
|
||||
AI-Generated Images via On-Manifold Latent Manipulation.** USENIX Security
|
||||
2026, arXiv:2608.10166. https://arxiv.org/abs/2608.10166
|
||||
|
||||
11. Ao, Du, Wang, Chen, Lu (2026). **AWPD: Frequency Shield Network for
|
||||
Agnostic Watermark Presence Detection.** arXiv:2603.06723.
|
||||
https://arxiv.org/abs/2603.06723
|
||||
|
||||
12. Gunn, Zhao, Song (2025). **An Undetectable Watermark for Generative
|
||||
Image Models.** ICLR 2025, arXiv:2410.07369.
|
||||
https://arxiv.org/abs/2410.07369
|
||||
|
||||
13. Francati, Goonatilake, Pawar, Venturi, Ateniese (2026). **The Coding
|
||||
Limits of Robust Watermarking for Generative Models.** IEEE EuroS&P 2026,
|
||||
arXiv:2509.10577. https://arxiv.org/abs/2509.10577
|
||||
|
||||
14. Kassis and Hengartner (2025). **UnMarker: A Universal Attack on
|
||||
Defensive Image Watermarking.** IEEE S&P 2025, arXiv:2405.08363.
|
||||
https://arxiv.org/abs/2405.08363
|
||||
|
||||
15. Liu et al. (2025). **Image Watermarks are Removable Using Controllable
|
||||
Regeneration from Clean Noise.** ICLR 2025, arXiv:2410.05470.
|
||||
https://arxiv.org/abs/2410.05470
|
||||
|
||||
16. An et al. (2024). **WAVES: Benchmarking the Robustness of Image
|
||||
Watermarks.** ICML 2024, arXiv:2401.08573.
|
||||
https://arxiv.org/abs/2401.08573
|
||||
|
||||
17. Bui, Agarwal, Collomosse (2023). **TrustMark: Universal Watermarking
|
||||
for Arbitrary Resolution Images.** arXiv:2311.18297.
|
||||
https://arxiv.org/abs/2311.18297
|
||||
|
||||
18. Dathathri et al. (2024). **Scalable watermarking for identifying large
|
||||
language model outputs.** Nature 634, 818-823.
|
||||
https://www.nature.com/articles/s41586-024-08025-4
|
||||
|
||||
19. DeepMind. **US12094474B1** and continuation **US20250149048A1**.
|
||||
https://patents.google.com/patent/US12094474B1/en
|
||||
|
||||
20. Weatherbed (2026-04-14). **Has Google's AI watermarking system been
|
||||
reverse-engineered?** The Verge.
|
||||
https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered
|
||||
|
||||
21. Cox, Kilian, Leighton, Shamoon (1997). **Secure spread spectrum
|
||||
watermarking for multimedia.** IEEE Trans. Image Process. 6(12).
|
||||
|
||||
22. Wen, Kirchenbauer, Geiping, Goldstein (2023). **Tree-Ring Watermarks.**
|
||||
NeurIPS 2023, arXiv:2305.20030. https://arxiv.org/abs/2305.20030
|
||||
|
||||
23. Ojha, Li, Lee (2023). **Towards Universal Fake Image Detectors that
|
||||
Generalize Across Generative Models.** CVPR 2023, arXiv:2302.10174.
|
||||
https://arxiv.org/abs/2302.10174
|
||||
|
||||
24. Corvi, Cozzolino, Poggi, Nagano, Verdoliva (2023). **Intriguing
|
||||
properties of synthetic images.** CVPRW 2023, arXiv:2304.06408.
|
||||
https://arxiv.org/abs/2304.06408
|
||||
|
||||
25. Zhong, Xu, Zou (2026). **Color Matters: Demosaicing-Guided Color
|
||||
Correlation Training.** arXiv:2601.22778.
|
||||
https://arxiv.org/abs/2601.22778
|
||||
|
||||
26. Baluja (2017). **Hiding Images in Plain Sight: Deep Steganography.**
|
||||
NeurIPS 2017.
|
||||
|
||||
27. Holub, Fridrich, Denemark (2014). **Universal distortion function for
|
||||
steganography in an arbitrary domain.** EURASIP J. Information Security.
|
||||
|
||||
28. Jing et al. (2021). **HiNet: Deep Image Hiding by Invertible Network.**
|
||||
ICCV 2021.
|
||||
|
||||
29. Yang et al. (2024). **Gaussian Shading: Provable Performance-Lossless
|
||||
Image Watermarking for Diffusion Models.** CVPR 2024, arXiv:2404.04956.
|
||||
https://arxiv.org/abs/2404.04956
|
||||
|
||||
30. Farid (2009). **Exposing digital forgeries from JPEG ghosts.** IEEE
|
||||
Trans. Information Forensics and Security 4(1).
|
||||
|
||||
31. Wang et al. (2023). **DIRE for Diffusion-Generated Image Detection.**
|
||||
ICCV 2023, arXiv:2303.09295. https://arxiv.org/abs/2303.09295
|
||||
|
||||
32. Wang, Wang, Zhang, Owens, Efros (2020). **CNN-generated images are
|
||||
surprisingly easy to spot... for now.** CVPR 2020, arXiv:1912.11035.
|
||||
https://arxiv.org/abs/1912.11035
|
||||
|
||||
33. Ó Ruanaidh and Pun (1998). **Rotation, scale and translation invariant
|
||||
spread spectrum digital image watermarking.** Signal Processing 66(3).
|
||||
|
||||
34. Réfrégier and Javidi (1995). **Optical image encryption based on input
|
||||
plane and Fourier plane random encoding.** Optics Letters 20(7).
|
||||
|
||||
**Google floor on qwen-zimage (0.27.2, remeasured 2026-08-25).** The bracket above did not
|
||||
survive the full production path: on the 4.33 MP CJK-sign fixture (visible-stage
|
||||
sparkle removal -> qwen-zimage seed 0 at the curve's 0.154 top -> resize-back ->
|
||||
@@ -779,5 +1229,6 @@ initially took a flat `0.30` floor instead of the area curve. The later provider
|
||||
remeasurement established a common pass at `0.25` across three valid sources and
|
||||
separately repeated `QWEN_ZIMAGE_GOOGLE_STRENGTH = 0.27` across those sources and
|
||||
three accounts. OpenAI now has its own measured flat operating point; unknown
|
||||
content keeps the resolution curve. An explicit strength still wins. Certification artifacts: raiw-app
|
||||
`data/certification/text-restoration-2026-08-18/` (the failing re-baseline).
|
||||
content keeps the resolution curve. An explicit strength still wins.
|
||||
Certification artifacts: private archive `text-restoration-2026-08-18` (the
|
||||
failing re-baseline).
|
||||
@@ -367,5 +367,5 @@ against base MAEs of 24.7/16.8/10.2. Since 0.27.1 the anchor is off by default
|
||||
(`fidelity_anchor` parameter on `remove_watermark` / `InvisibleOptions` /
|
||||
`--fidelity-anchor`); Google rows were not re-queried because the default
|
||||
strength leaves that fixture's base detected (the vendor-strength lane tracks
|
||||
that separately). Certification artifacts: raiw-app
|
||||
`data/certification/text-restoration-2026-08-19-A1/` (sha256-stamped rows).
|
||||
that separately). Certification artifacts: private archive
|
||||
`text-restoration-2026-08-19-A1` (sha256-stamped rows).
|
||||
@@ -184,7 +184,7 @@ Three harness rules are load-bearing and must not be relaxed: score a mark only
|
||||
crop's adjudication scope; take provenance from metadata, never from labels; and never
|
||||
report recall from the detector-sampled set.
|
||||
|
||||
## Tier D -- external oracles (manual, not automatable here)
|
||||
## Tier D -- external oracles
|
||||
|
||||
Proprietary watermark removal cannot be verified locally by design -- no public decoder
|
||||
exists. Each vendor has its own oracle and it covers only that vendor's content:
|
||||
@@ -195,6 +195,13 @@ needs a pixel-identical metadata-stripped control: the control must lose `C2PA`
|
||||
remaining `Watermark`-positive before a candidate's negative result can be attributed to
|
||||
pixel regeneration. A quiet metadata proxy is **not** proof the pixel watermark is gone.
|
||||
|
||||
OpenAI's API documentation says not to use repeated queries to reverse-engineer, remove,
|
||||
or evade a watermark. Using it as an adaptive research oracle therefore requires explicit
|
||||
authorization. Without that authorization it must not become a training loss, search loop,
|
||||
or automated removal gate. The provider-specific detector and pixel-only removal research
|
||||
protocol is in [`synthid-detector-removal-plan.md`](synthid-detector-removal-plan.md).
|
||||
|
||||
|
||||
Scope honestly: this tier certifies strength floors on a handful of images per vendor, and
|
||||
that is all it can do. See `docs/synthid.md`.
|
||||
|
||||
|
||||
@@ -4,7 +4,7 @@
|
||||
# on it, including the ComfyUI node package. The console script below carries
|
||||
# the same weight, since users have it on PATH.
|
||||
name = "remove-ai-watermarks"
|
||||
version = "0.36.0"
|
||||
version = "0.37.0"
|
||||
description = "AI watermark remover for visible, invisible, and provenance marks in images and video"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.11,<3.15"
|
||||
@@ -142,6 +142,20 @@ trustmark = [
|
||||
# wheels. Keep this feature installable only on its real runtime range.
|
||||
"trustmark>=0.8.0; python_version < '3.13'",
|
||||
]
|
||||
# Metadata-free photo classifier (Model 1 then gated Model 2). CPU torch is
|
||||
# enough. identify never imports this extra. Weights download on first use.
|
||||
classify = [
|
||||
"remove-ai-watermarks[pixels]",
|
||||
"torch>=2.0.0",
|
||||
"transformers>=5,<6",
|
||||
"tokenizers>=0.22,<0.23",
|
||||
"huggingface-hub>=0.20.0",
|
||||
]
|
||||
# Official remote OpenAI SynthID verification. The command strips AI provenance
|
||||
# metadata and proves pixel identity before upload; it is never called implicitly.
|
||||
verify = [
|
||||
"openai>=2.52.0",
|
||||
]
|
||||
# Universal region eraser backend -- big-LaMa via onnxruntime (Carve/LaMa-ONNX,
|
||||
# Apache-2.0). CPU, no torch. Model (~200 MB) is downloaded on first use and
|
||||
# cached by huggingface_hub; it is never bundled in this repo. The default cv2
|
||||
@@ -174,7 +188,7 @@ dev = [
|
||||
]
|
||||
# ``qwen-zimage`` already pulls ``diffusion``; naming both would suggest diffusion is
|
||||
# independently sufficient for a removal, which it is not.
|
||||
all = ["remove-ai-watermarks[video,heif,detect,trustmark,text-restoration,migan]"]
|
||||
all = ["remove-ai-watermarks[video,heif,detect,trustmark,text-restoration,migan,classify,verify]"]
|
||||
|
||||
[project.scripts]
|
||||
remove-ai-watermarks = "remove_ai_watermarks.cli:main"
|
||||
@@ -207,13 +221,13 @@ exclude = [
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
pythonpath = ["src"]
|
||||
pythonpath = ["src", "scripts"]
|
||||
addopts = "-v --tb=short"
|
||||
|
||||
[tool.ruff]
|
||||
target-version = "py311"
|
||||
line-length = 120
|
||||
exclude = ["_refs"]
|
||||
exclude = ["_refs", ".local-eval"]
|
||||
extend-exclude = ["*.md"]
|
||||
|
||||
[tool.ruff.lint]
|
||||
@@ -224,7 +238,7 @@ ignore = [
|
||||
]
|
||||
|
||||
[tool.ruff.lint.per-file-ignores]
|
||||
"scripts/*.py" = ["G004", "S108", "S310", "T20"]
|
||||
"scripts/**/*.py" = ["G004", "S108", "S310", "T20"]
|
||||
"tests/*.py" = ["ANN", "S101", "S105", "S106", "S108"]
|
||||
"src/remove_ai_watermarks/_internal/watermark_remover.py" = ["S603", "S606", "S607"] # nvidia-smi capability probe
|
||||
"src/remove_ai_watermarks/_internal/c2pa.py" = ["S110"] # try-except-pass for corrupt file handling
|
||||
@@ -236,7 +250,7 @@ indent-style = "space"
|
||||
[tool.pyright]
|
||||
pythonVersion = "3.11"
|
||||
typeCheckingMode = "strict"
|
||||
exclude = ["_refs"]
|
||||
exclude = ["_refs", ".local-eval"]
|
||||
|
||||
[[tool.pyright.executionEnvironments]]
|
||||
root = "tests"
|
||||
|
||||
@@ -43,13 +43,19 @@ explicitly names a tracked canonical result.
|
||||
| `visible_positives.py` | List corpus images carrying a registered visible mark. |
|
||||
| `visible_recall_sample.py` | Build an unbiased blinded sample for recall measurement. |
|
||||
| `visible_sheets.py` | Build blinded contact sheets for relaxation candidates. |
|
||||
| `retrain_photo_classify.py` | CPU-retrain the 2026-08-31 photo heads from a sha256-keyed cache pack, no images. |
|
||||
| `publish_photo_classify_hf.py` | Upload the photo-classify card and freeze weights to Hugging Face `wiltodelta/raiw-photo-classify`. Manual; the Action `publish-photo-classify-hf.yml` is the write-token path. |
|
||||
| `verify_engine_selection_fixtures.py` | Verify hashes, dimensions, and prompt pairing in the tracked auto-engine content matrix. |
|
||||
|
||||
## Research and diagnostic prototypes
|
||||
|
||||
| Script | Purpose |
|
||||
| --- | --- |
|
||||
| `chroma_text_restoration_study.py` | Compare verified-text restoration for Qwen and Chroma on H100 without running both engines in production. |
|
||||
| `cjk_tail_probe.py` | Test a generic template for otherwise uncovered CJK labels. |
|
||||
| `controlnet_sweep.py` | Sweep the historical ControlNet removal prototype. |
|
||||
| `engine_selection_study.py` | Run one global engine at a time over the tracked prompt-matched content matrix. |
|
||||
| `analyze_engine_selection_study.py` | Reduce engine-study output to paired deltas and exact sign tests. |
|
||||
| `infer_text_lines.py` | Draft stable source-text lines without modifying pixels. |
|
||||
| `qwen_scrub_prototype.py` | Probe low-strength Qwen regeneration on a GPU. |
|
||||
| `selective_text_restoration.py` | Evaluate text restoration over a scrubbed image. |
|
||||
|
||||
@@ -0,0 +1,121 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarize paired fidelity results from the engine-selection study.
|
||||
|
||||
Run from the repository root after ``engine_selection_study.py``:
|
||||
|
||||
uv run scripts/analyze_engine_selection_study.py
|
||||
|
||||
The report uses per-image paired differences and exact two-sided sign tests.
|
||||
It does not turn the discovery set into a production routing rule.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import statistics
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Callable
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_METRICS: dict[str, tuple[Callable[[float], float], str]] = {
|
||||
"lpips_512": (lambda value: value, "lower"),
|
||||
"ssim": (lambda value: -value, "higher"),
|
||||
"psnr_db": (lambda value: -value, "higher"),
|
||||
"mae": (lambda value: value, "lower"),
|
||||
"edge_f1": (lambda value: -value, "higher"),
|
||||
"laplacian_ratio": (lambda value: abs(math.log(max(value, 1e-12))), "closer_to_one"),
|
||||
}
|
||||
|
||||
|
||||
def _sign_test_two_sided(wins: int, losses: int) -> float:
|
||||
"""Return the exact two-sided sign-test p-value, excluding ties."""
|
||||
trials = wins + losses
|
||||
if trials == 0:
|
||||
return 1.0
|
||||
tail = sum(math.comb(trials, index) for index in range(min(wins, losses) + 1))
|
||||
return min(1.0, 2.0 * tail / (2**trials))
|
||||
|
||||
|
||||
def _index(rows: list[dict[str, Any]]) -> dict[tuple[str, str], dict[str, Any]]:
|
||||
return {(row["provider"], row["pair_id"]): row for row in rows}
|
||||
|
||||
|
||||
def analyze(report: dict[str, Any]) -> dict[str, Any]:
|
||||
qwen = _index(report["qwen-zimage"]["rows"])
|
||||
chroma = _index(report["chroma-zimage"]["rows"])
|
||||
if qwen.keys() != chroma.keys():
|
||||
raise ValueError("Qwen and Chroma result keys differ")
|
||||
|
||||
by_provider: dict[str, list[tuple[dict[str, Any], dict[str, Any]]]] = defaultdict(list)
|
||||
for key in sorted(qwen):
|
||||
by_provider[key[0]].append((qwen[key], chroma[key]))
|
||||
|
||||
providers: dict[str, Any] = {}
|
||||
for provider, pairs in sorted(by_provider.items()):
|
||||
metrics: dict[str, Any] = {}
|
||||
for metric, (loss_transform, direction) in _METRICS.items():
|
||||
deltas: list[float] = []
|
||||
chroma_wins = 0
|
||||
qwen_wins = 0
|
||||
ties = 0
|
||||
for qwen_row, chroma_row in pairs:
|
||||
qwen_value = float(qwen_row["fidelity"][metric])
|
||||
chroma_value = float(chroma_row["fidelity"][metric])
|
||||
delta = loss_transform(chroma_value) - loss_transform(qwen_value)
|
||||
deltas.append(delta)
|
||||
if math.isclose(delta, 0.0, abs_tol=1e-12):
|
||||
ties += 1
|
||||
elif delta < 0:
|
||||
chroma_wins += 1
|
||||
else:
|
||||
qwen_wins += 1
|
||||
metrics[metric] = {
|
||||
"preferred_direction": direction,
|
||||
"delta_definition": "chroma_loss_minus_qwen_loss; negative favors Chroma",
|
||||
"chroma_wins": chroma_wins,
|
||||
"qwen_wins": qwen_wins,
|
||||
"ties": ties,
|
||||
"mean_delta": statistics.fmean(deltas),
|
||||
"median_delta": statistics.median(deltas),
|
||||
"sign_test_two_sided_p": _sign_test_two_sided(chroma_wins, qwen_wins),
|
||||
}
|
||||
providers[provider] = {"pairs": len(pairs), "metrics": metrics}
|
||||
|
||||
return {
|
||||
"method": "paired per-image differences with exact two-sided sign tests; ties excluded",
|
||||
"warning": "Discovery-set evidence only; challenge any proposed rule on unused generations.",
|
||||
"providers": providers,
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument(
|
||||
"--input",
|
||||
type=Path,
|
||||
default=Path("out/engine-selection-study/summary.json"),
|
||||
)
|
||||
parser.add_argument(
|
||||
"--output",
|
||||
type=Path,
|
||||
default=Path("out/engine-selection-study/analysis.json"),
|
||||
)
|
||||
args = parser.parse_args()
|
||||
report = json.loads(args.input.read_text(encoding="utf-8"))
|
||||
analysis = analyze(report)
|
||||
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||
args.output.write_text(json.dumps(analysis, indent=2) + "\n", encoding="utf-8")
|
||||
log.info("Wrote %s", args.output)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
|
||||
main()
|
||||
@@ -0,0 +1,232 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["modal>=1.5"]
|
||||
# ///
|
||||
"""Paired offline study of verified-text restoration on Qwen and Chroma.
|
||||
|
||||
Each generative profile runs exactly once per fixture. Its already loaded VAE
|
||||
produces the restoration donor, then the shared production compositor creates
|
||||
the restored candidate from that one base output. The experiment therefore
|
||||
measures four outputs (Qwen/Chroma x base/restored) without accidentally timing
|
||||
or evaluating a second generation for either restoration arm.
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
uvx modal run scripts/chroma_text_restoration_study.py
|
||||
|
||||
Outputs stay under ``out/text-restoration-engine-study``. Submit exact output
|
||||
hashes to the matching provider oracle before using fidelity metrics to choose
|
||||
between engines; fidelity only ranks candidates after the scrub verdict passes.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import modal
|
||||
|
||||
_ROOT = Path(__file__).resolve().parent.parent
|
||||
_GPU = "H100"
|
||||
|
||||
app = modal.App("chroma-text-restoration-study")
|
||||
|
||||
image = (
|
||||
modal.Image.debian_slim(python_version="3.12")
|
||||
.pip_install(
|
||||
"torch",
|
||||
"torchvision>=0.20.0",
|
||||
"diffusers>=0.38,<1",
|
||||
"diffsynth>=2.0.17,<3",
|
||||
"transformers>=4.53",
|
||||
"accelerate",
|
||||
"sentencepiece",
|
||||
"protobuf",
|
||||
"pillow",
|
||||
"numpy",
|
||||
"opencv-python-headless<5",
|
||||
"onnxruntime>=1.24.0",
|
||||
"huggingface-hub>=0.20.0",
|
||||
)
|
||||
.env({"HF_HOME": "/cache"})
|
||||
.add_local_dir(_ROOT / "src" / "remove_ai_watermarks", remote_path="/pkg/remove_ai_watermarks")
|
||||
)
|
||||
|
||||
cache = modal.Volume.from_name("chroma-hf-cache", create_if_missing=True)
|
||||
|
||||
|
||||
def _png_bytes(image: Any) -> bytes:
|
||||
import io
|
||||
|
||||
buffer = io.BytesIO()
|
||||
image.save(buffer, format="PNG")
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
def _metric_inputs(source: Any, boxes: list[tuple[int, int, int, int]]) -> tuple[Any, Any]:
|
||||
import numpy as np
|
||||
|
||||
source_rgb = np.asarray(source.convert("RGB"), dtype=np.uint8)
|
||||
mask = np.zeros(source_rgb.shape[:2], dtype=bool)
|
||||
for x1, y1, x2, y2 in boxes:
|
||||
mask[y1:y2, x1:x2] = True
|
||||
return source_rgb, mask
|
||||
|
||||
|
||||
def _metrics(source_rgb: Any, candidate: Any, mask: Any) -> dict[str, float]:
|
||||
import math
|
||||
|
||||
import numpy as np
|
||||
|
||||
candidate_rgb = np.asarray(candidate.convert("RGB"), dtype=np.uint8)
|
||||
signed_delta = source_rgb.astype(np.float32) - candidate_rgb
|
||||
absolute_delta = np.abs(signed_delta)
|
||||
mse = float((signed_delta**2).mean())
|
||||
return {
|
||||
"whole_mae": round(float(absolute_delta.mean()), 6),
|
||||
"text_box_mae": round(float(absolute_delta[mask].mean()), 6),
|
||||
"outside_box_mae": round(float(absolute_delta[~mask].mean()), 6),
|
||||
"psnr_db": round(99.0 if mse <= 1e-12 else 10.0 * math.log10(255.0**2 / mse), 6),
|
||||
}
|
||||
|
||||
|
||||
@app.function(image=image, gpu=_GPU, volumes={"/cache": cache}, timeout=3600)
|
||||
def run(payload: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
|
||||
import gc
|
||||
import io
|
||||
import sys
|
||||
import time
|
||||
|
||||
import torch
|
||||
from PIL import Image
|
||||
|
||||
sys.path.insert(0, "/pkg")
|
||||
|
||||
from remove_ai_watermarks._internal.chroma_zimage_pipeline import ChromaZImagePipeline
|
||||
from remove_ai_watermarks._internal.qwen_zimage_pipeline import QwenZImagePipeline
|
||||
from remove_ai_watermarks._internal.text_restoration import VerifiedTextLine, restore_verified_text
|
||||
from remove_ai_watermarks._internal.two_stage_pipeline import detect_faces
|
||||
from remove_ai_watermarks._internal.watermark_profiles import resolve_strength
|
||||
|
||||
sources: dict[str, tuple[Image.Image, str, tuple[VerifiedTextLine, ...]]] = {}
|
||||
for name, item in payload.items():
|
||||
source = Image.open(io.BytesIO(item["source"])).convert("RGB")
|
||||
lines = tuple(
|
||||
VerifiedTextLine(
|
||||
tuple(line["box"]),
|
||||
line.get("text"),
|
||||
line.get("script"),
|
||||
float(line.get("angle", 0.0)),
|
||||
)
|
||||
for line in item["lines"]
|
||||
)
|
||||
sources[name] = source, str(item["vendor"]), lines
|
||||
|
||||
results: dict[str, dict[str, Any]] = {
|
||||
name: {"vendor": vendor, "engines": {}} for name, (_, vendor, _) in sources.items()
|
||||
}
|
||||
metric_inputs = {
|
||||
name: _metric_inputs(source, [line.box for line in lines]) for name, (source, _vendor, lines) in sources.items()
|
||||
}
|
||||
|
||||
for profile, pipeline_type in (
|
||||
("qwen-zimage", QwenZImagePipeline),
|
||||
("chroma-zimage", ChromaZImagePipeline),
|
||||
):
|
||||
print(f"[{profile}] loading...", flush=True)
|
||||
pipeline = pipeline_type(
|
||||
device="cuda",
|
||||
torch_dtype=torch.bfloat16,
|
||||
keep_global_models_on_device=True,
|
||||
keep_face_models_on_device=True,
|
||||
progress_callback=lambda message, p=profile: print(f"[{p}] {message}", flush=True),
|
||||
)
|
||||
for name, (source, vendor, lines) in sources.items():
|
||||
face_count = len(detect_faces(source)) if profile == "chroma-zimage" else 0
|
||||
strength = resolve_strength(
|
||||
None,
|
||||
vendor,
|
||||
profile,
|
||||
size=source.size,
|
||||
face_count=face_count,
|
||||
)
|
||||
print(f"[{profile}] {name}: strength={strength:.6f}, faces={face_count}", flush=True)
|
||||
|
||||
started = time.perf_counter()
|
||||
donor = pipeline._vae_roundtrip(source)
|
||||
donor_seconds = time.perf_counter() - started
|
||||
|
||||
started = time.perf_counter()
|
||||
base = pipeline.run(source, strength=strength, seed=0)
|
||||
generation_seconds = time.perf_counter() - started
|
||||
|
||||
started = time.perf_counter()
|
||||
restored = restore_verified_text(source, base, donor, lines)
|
||||
restoration_seconds = time.perf_counter() - started
|
||||
|
||||
source_rgb, text_mask = metric_inputs[name]
|
||||
base_png = _png_bytes(base)
|
||||
restored_png = _png_bytes(restored)
|
||||
donor_png = _png_bytes(donor)
|
||||
results[name]["engines"][profile] = {
|
||||
"strength": strength,
|
||||
"face_count": face_count,
|
||||
"donor_seconds": round(donor_seconds, 3),
|
||||
"generation_seconds": round(generation_seconds, 3),
|
||||
"restoration_seconds": round(restoration_seconds, 3),
|
||||
"base_sha256": hashlib.sha256(base_png).hexdigest(),
|
||||
"restored_sha256": hashlib.sha256(restored_png).hexdigest(),
|
||||
"donor_sha256": hashlib.sha256(donor_png).hexdigest(),
|
||||
"base_metrics": _metrics(source_rgb, base, text_mask),
|
||||
"restored_metrics": _metrics(source_rgb, restored, text_mask),
|
||||
"base_png": base_png,
|
||||
"restored_png": restored_png,
|
||||
"donor_png": donor_png,
|
||||
}
|
||||
|
||||
del pipeline
|
||||
gc.collect()
|
||||
torch.cuda.empty_cache()
|
||||
|
||||
cache.commit()
|
||||
return results
|
||||
|
||||
|
||||
@app.local_entrypoint()
|
||||
def main(out: str = "out/text-restoration-engine-study") -> None:
|
||||
annotations = json.loads((_ROOT / "data/evaluations/fidelity/text-lines.json").read_text(encoding="utf-8"))
|
||||
vendors = {
|
||||
"ChatGPT Image May 31, 2026, 02_02_23 PM.png": "openai",
|
||||
"ChatGPT Image May 31, 2026, 02_03_55 PM.png": "openai",
|
||||
"Gemini_Generated_Image_633uuy633uuy633u.png": "google",
|
||||
}
|
||||
source_root = _ROOT / "data/synthid/originals"
|
||||
payload = {
|
||||
name: {
|
||||
"source": (source_root / name).read_bytes(),
|
||||
"vendor": vendor,
|
||||
"lines": annotations[name],
|
||||
}
|
||||
for name, vendor in vendors.items()
|
||||
}
|
||||
|
||||
results = run.remote(payload)
|
||||
out_root = Path(out)
|
||||
summary: dict[str, Any] = {"gpu": _GPU, "fixtures": {}}
|
||||
for name, fixture in results.items():
|
||||
fixture_dir = out_root / Path(name).stem
|
||||
fixture_dir.mkdir(parents=True, exist_ok=True)
|
||||
summary["fixtures"][name] = {"vendor": fixture["vendor"], "engines": {}}
|
||||
for profile, result in fixture["engines"].items():
|
||||
for variant in ("base", "restored", "donor"):
|
||||
(fixture_dir / f"{profile}_{variant}.png").write_bytes(result[f"{variant}_png"])
|
||||
summary["fixtures"][name]["engines"][profile] = {
|
||||
key: value for key, value in result.items() if not key.endswith("_png")
|
||||
}
|
||||
|
||||
out_root.mkdir(parents=True, exist_ok=True)
|
||||
(out_root / "summary.json").write_text(json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
print(json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
print(f"outputs in {out_root}")
|
||||
@@ -0,0 +1,221 @@
|
||||
# /// script
|
||||
# requires-python = ">=3.11"
|
||||
# dependencies = ["modal>=1.5"]
|
||||
# ///
|
||||
"""Compare one engine at a time on the tracked content-selection matrix.
|
||||
|
||||
This is an offline research harness, not a production router. Each Modal call
|
||||
loads exactly one global engine and runs all 38 fixed inputs at that engine's
|
||||
shipped provider floor. The shared Z-Image face repair is deliberately omitted:
|
||||
the model choice changes only the global stage, while the face stage regenerates
|
||||
the same original crops for both profiles.
|
||||
|
||||
Run from the repository root:
|
||||
|
||||
uvx modal run scripts/engine_selection_study.py
|
||||
|
||||
The combined metrics land in the gitignored
|
||||
``out/engine-selection-study/summary.json``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import modal
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_ROOT = Path(__file__).resolve().parent.parent
|
||||
_FIXTURE_ROOT = _ROOT / "data" / "evaluations" / "engine-selection"
|
||||
_GPU = "H100"
|
||||
|
||||
app = modal.App("engine-selection-study")
|
||||
|
||||
image = (
|
||||
modal.Image.debian_slim(python_version="3.12")
|
||||
.pip_install(
|
||||
"torch",
|
||||
"torchvision>=0.20.0",
|
||||
"diffusers>=0.38,<1",
|
||||
"diffsynth>=2.0.17,<3",
|
||||
"transformers>=4.53",
|
||||
"accelerate",
|
||||
"sentencepiece",
|
||||
"protobuf",
|
||||
"pillow",
|
||||
"numpy",
|
||||
"opencv-python-headless",
|
||||
"scikit-image",
|
||||
"lpips",
|
||||
)
|
||||
.env({"HF_HOME": "/cache", "TORCH_HOME": "/cache/torch"})
|
||||
.add_local_dir(
|
||||
_ROOT / "src" / "remove_ai_watermarks",
|
||||
remote_path="/pkg/remove_ai_watermarks",
|
||||
)
|
||||
.add_local_dir(_FIXTURE_ROOT, remote_path="/fixtures")
|
||||
)
|
||||
|
||||
cache = modal.Volume.from_name("chroma-hf-cache", create_if_missing=True)
|
||||
|
||||
_STRENGTHS = {
|
||||
"qwen-zimage": {"openai": 0.07675, "meta": 0.10},
|
||||
"chroma-zimage": {"openai": 0.09, "meta": 0.17},
|
||||
}
|
||||
|
||||
|
||||
@app.function(image=image, gpu=_GPU, volumes={"/cache": cache}, timeout=3600)
|
||||
def run_engine(engine: str) -> dict[str, Any]:
|
||||
import csv
|
||||
import hashlib
|
||||
import sys
|
||||
import time
|
||||
|
||||
import cv2
|
||||
import lpips
|
||||
import numpy as np
|
||||
import torch
|
||||
from PIL import Image
|
||||
from skimage.metrics import peak_signal_noise_ratio, structural_similarity
|
||||
|
||||
sys.path.insert(0, "/pkg")
|
||||
logging.basicConfig(level=logging.INFO, format="%(levelname)s: %(message)s")
|
||||
|
||||
if engine == "qwen-zimage":
|
||||
from remove_ai_watermarks._internal.qwen_zimage_pipeline import QwenZImagePipeline
|
||||
|
||||
pipeline_type: Any = QwenZImagePipeline
|
||||
elif engine == "chroma-zimage":
|
||||
from remove_ai_watermarks._internal.chroma_zimage_pipeline import ChromaZImagePipeline
|
||||
|
||||
pipeline_type = ChromaZImagePipeline
|
||||
else:
|
||||
raise ValueError(f"unknown engine: {engine}")
|
||||
pipeline = pipeline_type(
|
||||
device="cuda",
|
||||
torch_dtype=torch.bfloat16,
|
||||
keep_global_models_on_device=True,
|
||||
keep_face_models_on_device=False,
|
||||
)
|
||||
|
||||
perceptual = lpips.LPIPS(net="alex", verbose=False).to("cuda").eval()
|
||||
|
||||
def rgb_array(pil_image: Image.Image) -> np.ndarray:
|
||||
return np.asarray(pil_image.convert("RGB"), dtype=np.uint8)
|
||||
|
||||
def gray(rgb: np.ndarray) -> np.ndarray:
|
||||
return cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)
|
||||
|
||||
def lpips_distance(source: np.ndarray, candidate: np.ndarray) -> float:
|
||||
height, width = source.shape[:2]
|
||||
scale = min(1.0, 512.0 / max(height, width))
|
||||
size = (max(1, round(width * scale)), max(1, round(height * scale)))
|
||||
|
||||
def tensor(array: np.ndarray) -> torch.Tensor:
|
||||
resized = cv2.resize(array, size, interpolation=cv2.INTER_AREA)
|
||||
normalized = resized.astype(np.float32) / 127.5 - 1.0
|
||||
return torch.from_numpy(normalized).permute(2, 0, 1).unsqueeze(0).to("cuda")
|
||||
|
||||
with torch.inference_mode():
|
||||
return float(perceptual(tensor(source), tensor(candidate)).item())
|
||||
|
||||
def edge_f1(source_edges: np.ndarray, candidate_gray: np.ndarray) -> float:
|
||||
candidate_edges = cv2.Canny(candidate_gray, 100, 200) > 0
|
||||
kernel = np.ones((3, 3), np.uint8)
|
||||
source_near = cv2.dilate(source_edges.astype(np.uint8), kernel) > 0
|
||||
candidate_near = cv2.dilate(candidate_edges.astype(np.uint8), kernel) > 0
|
||||
precision = float((candidate_edges & source_near).sum()) / max(1, int(candidate_edges.sum()))
|
||||
recall = float((source_edges & candidate_near).sum()) / max(1, int(source_edges.sum()))
|
||||
return 2.0 * precision * recall / max(1e-9, precision + recall)
|
||||
|
||||
def source_features(
|
||||
source: np.ndarray,
|
||||
source_gray: np.ndarray,
|
||||
source_edges: np.ndarray,
|
||||
source_lapvar: float,
|
||||
) -> dict[str, float]:
|
||||
histogram = np.bincount(source_gray.reshape(-1), minlength=256).astype(np.float64)
|
||||
probabilities = histogram[histogram > 0] / histogram.sum()
|
||||
hsv = cv2.cvtColor(source, cv2.COLOR_RGB2HSV)
|
||||
return {
|
||||
"edge_density": round(float(source_edges.mean()), 6),
|
||||
"laplacian_variance": round(source_lapvar, 6),
|
||||
"entropy_bits": round(float(-(probabilities * np.log2(probabilities)).sum()), 6),
|
||||
"mean_saturation": round(float(hsv[..., 1].mean() / 255.0), 6),
|
||||
}
|
||||
|
||||
def fidelity(
|
||||
source: np.ndarray,
|
||||
candidate: np.ndarray,
|
||||
source_gray: np.ndarray,
|
||||
source_edges: np.ndarray,
|
||||
source_lapvar: float,
|
||||
) -> dict[str, float]:
|
||||
candidate_gray = gray(candidate)
|
||||
candidate_lapvar = float(cv2.Laplacian(candidate_gray, cv2.CV_64F).var())
|
||||
return {
|
||||
"lpips_512": round(lpips_distance(source, candidate), 6),
|
||||
"ssim": round(float(structural_similarity(source_gray, candidate_gray)), 6),
|
||||
"psnr_db": round(float(peak_signal_noise_ratio(source, candidate)), 6),
|
||||
"mae": round(float(np.abs(source.astype(np.float32) - candidate).mean()), 6),
|
||||
"edge_f1": round(edge_f1(source_edges, candidate_gray), 6),
|
||||
"laplacian_ratio": round(candidate_lapvar / max(source_lapvar, 1e-9), 6),
|
||||
}
|
||||
|
||||
with Path("/fixtures/content-manifest.csv").open(newline="", encoding="utf-8") as stream:
|
||||
rows = list(csv.DictReader(stream))
|
||||
|
||||
results: list[dict[str, Any]] = []
|
||||
for index, row in enumerate(rows, start=1):
|
||||
source_path = Path("/fixtures") / row["file"]
|
||||
source_image = Image.open(source_path).convert("RGB")
|
||||
strength = _STRENGTHS[engine][row["provider"]]
|
||||
started = time.perf_counter()
|
||||
output = pipeline._run_global(source_image, strength, 0)
|
||||
elapsed = time.perf_counter() - started
|
||||
source = rgb_array(source_image)
|
||||
candidate = rgb_array(output)
|
||||
source_gray = gray(source)
|
||||
source_edges = cv2.Canny(source_gray, 100, 200) > 0
|
||||
source_lapvar = float(cv2.Laplacian(source_gray, cv2.CV_64F).var())
|
||||
result = {
|
||||
"pair_id": row["pair_id"],
|
||||
"provider": row["provider"],
|
||||
"content_stratum": row["content_stratum"],
|
||||
"strength": strength,
|
||||
"seconds": round(elapsed, 3),
|
||||
"output_rgb_sha256": hashlib.sha256(output.tobytes()).hexdigest(),
|
||||
"source_features": source_features(source, source_gray, source_edges, source_lapvar),
|
||||
"fidelity": fidelity(source, candidate, source_gray, source_edges, source_lapvar),
|
||||
}
|
||||
results.append(result)
|
||||
log.info(
|
||||
"%s %s/%s (%s/%s)",
|
||||
engine,
|
||||
index,
|
||||
len(rows),
|
||||
row["provider"],
|
||||
row["content_stratum"],
|
||||
)
|
||||
|
||||
cache.commit()
|
||||
return {"engine": engine, "gpu": _GPU, "rows": results}
|
||||
|
||||
|
||||
@app.local_entrypoint()
|
||||
def main(out: str = "out/engine-selection-study") -> None:
|
||||
qwen = run_engine.remote("qwen-zimage")
|
||||
chroma = run_engine.remote("chroma-zimage")
|
||||
report = {
|
||||
"protocol": "one engine per sequential H100 invocation; global-stage factorial ablation",
|
||||
"qwen-zimage": qwen,
|
||||
"chroma-zimage": chroma,
|
||||
}
|
||||
destination = Path(out) / "summary.json"
|
||||
destination.parent.mkdir(parents=True, exist_ok=True)
|
||||
destination.write_text(json.dumps(report, indent=2), encoding="utf-8")
|
||||
log.info("Wrote %s", destination)
|
||||
@@ -0,0 +1,119 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Publish the photo-classify freeze to Hugging Face wiltodelta/raiw-photo-classify.
|
||||
|
||||
Manual. Not a library release step. Weights are not in git: pass --src, a
|
||||
directory that holds clip-l-ft.pt and the probe, with detector.pt and
|
||||
provider.pt either at the root or in run1/.
|
||||
|
||||
uv run python scripts/publish_photo_classify_hf.py --src DIR
|
||||
uv run python scripts/publish_photo_classify_hf.py --card-only
|
||||
|
||||
Needs HF_TOKEN with write role. The GitHub Action
|
||||
publish-photo-classify-hf.yml injects secrets.HF_TOKEN.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import logging
|
||||
import os
|
||||
import shutil
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
HUB_REPO = "wiltodelta/raiw-photo-classify"
|
||||
CLIP_FILE = "clip-l-ft.pt"
|
||||
PROBE_FILE = "probe-weights-clip-l-ft.npz"
|
||||
DETECTOR_FILE = "detector.pt"
|
||||
PROVIDER_FILE = "provider.pt"
|
||||
WEIGHT_FILES = (CLIP_FILE, PROBE_FILE, DETECTOR_FILE, PROVIDER_FILE)
|
||||
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
CARD_DIR = REPO_ROOT / "docs" / "photo-classify-hf"
|
||||
|
||||
|
||||
def weight_paths(src: Path) -> dict[str, Path]:
|
||||
"""Map Hub filenames to files under ``src`` or ``src/run1``."""
|
||||
src = src.expanduser().resolve()
|
||||
if not src.is_dir():
|
||||
raise SystemExit(f"missing weights directory: {src}")
|
||||
found: dict[str, Path] = {}
|
||||
for name in WEIGHT_FILES:
|
||||
direct = src / name
|
||||
nested = src / "run1" / name
|
||||
if direct.is_file():
|
||||
found[name] = direct
|
||||
elif nested.is_file():
|
||||
found[name] = nested
|
||||
else:
|
||||
raise SystemExit(f"missing {name} in {src} or {src / 'run1'}")
|
||||
return found
|
||||
|
||||
|
||||
def stage_card(dest: Path) -> None:
|
||||
readme = CARD_DIR / "README.md"
|
||||
operating = CARD_DIR / "operating-point.json"
|
||||
if not readme.is_file() or not operating.is_file():
|
||||
raise SystemExit(f"Hub card missing under {CARD_DIR}")
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(readme, dest / "README.md")
|
||||
shutil.copy2(operating, dest / "operating-point.json")
|
||||
|
||||
|
||||
def _place(src: Path, dest: Path) -> None:
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
try:
|
||||
os.link(src, dest)
|
||||
except OSError:
|
||||
shutil.copy2(src, dest)
|
||||
|
||||
|
||||
def stage_weights(dest: Path, src: Path) -> None:
|
||||
for name, path in weight_paths(src).items():
|
||||
_place(path, dest / name)
|
||||
log.info("stage %s (%s bytes)", name, path.stat().st_size)
|
||||
|
||||
|
||||
def publish(stage: Path, *, token: str, message: str) -> str:
|
||||
from huggingface_hub import HfApi, create_repo
|
||||
|
||||
create_repo(HUB_REPO, repo_type="model", exist_ok=True, private=False, token=token)
|
||||
api = HfApi(token=token)
|
||||
commit = api.upload_folder(
|
||||
folder_path=str(stage),
|
||||
repo_id=HUB_REPO,
|
||||
repo_type="model",
|
||||
commit_message=message,
|
||||
allow_patterns=["README.md", "operating-point.json", *WEIGHT_FILES],
|
||||
)
|
||||
return getattr(commit, "oid", "") or ""
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--src", type=Path, help="Freeze or weights directory")
|
||||
parser.add_argument("--card-only", action="store_true")
|
||||
parser.add_argument("--message", default="")
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
token = os.environ.get("HF_TOKEN", "").strip()
|
||||
if not token:
|
||||
raise SystemExit("HF_TOKEN is not set (write role required)")
|
||||
if args.card_only == bool(args.src):
|
||||
raise SystemExit("pass exactly one of --src or --card-only")
|
||||
message = args.message.strip() or (
|
||||
"Update photo-classify card" if args.card_only else "Publish photo-classify freeze"
|
||||
)
|
||||
with tempfile.TemporaryDirectory(prefix="raiw-hf-") as tmp:
|
||||
stage = Path(tmp)
|
||||
stage_card(stage)
|
||||
if args.src is not None:
|
||||
stage_weights(stage, args.src)
|
||||
oid = publish(stage, token=token, message=message)
|
||||
log.info("published %s revision=%s", HUB_REPO, oid or "(unknown)")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,198 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Retrain the 2026-08-31 photo heads from a sha256-keyed cache pack.
|
||||
|
||||
No image I/O. Requires catalog.json, embeddings-by-sha256.npz or extra-clip-l.npz
|
||||
plus embeddings keyed by sha256, features-124d.npz, and probe-weights-clip-l-ft.npz.
|
||||
|
||||
uv run python scripts/retrain_photo_classify.py --pack DIR --out DIR
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
from torch import nn
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
DETECTOR_SEED = 20260940
|
||||
PROVIDER_SEED = 20260956
|
||||
EPOCHS = 40
|
||||
MLP_WIDTH = 512
|
||||
LR = 5e-4
|
||||
BATCH = 64
|
||||
TARGET_PHOTO_FPR = 0.0167
|
||||
FOCAL_GAMMA = 2.0
|
||||
FOCAL_EPOCHS = 40
|
||||
FOCAL_LR = 1e-3
|
||||
FOCAL_BATCH = 64
|
||||
LABELS = ("openai", "google", "tc260", "meta_muse_image", "no_ai")
|
||||
|
||||
|
||||
def _load_sha_vectors(path: Path) -> dict[str, np.ndarray]:
|
||||
payload = np.load(path, allow_pickle=True)
|
||||
if "sha256" in payload.files:
|
||||
keys = payload["sha256"]
|
||||
vectors = payload["vectors"]
|
||||
return {str(key): np.asarray(vector) for key, vector in zip(keys, vectors, strict=True)}
|
||||
raise SystemExit(f"{path} must contain sha256 and vectors")
|
||||
|
||||
|
||||
def _mlp(dim: int) -> nn.Module:
|
||||
return nn.Sequential(
|
||||
nn.Linear(dim, MLP_WIDTH),
|
||||
nn.ReLU(),
|
||||
nn.Dropout(0.3),
|
||||
nn.Linear(MLP_WIDTH, MLP_WIDTH // 4),
|
||||
nn.ReLU(),
|
||||
nn.Dropout(0.1),
|
||||
nn.Linear(MLP_WIDTH // 4, 1),
|
||||
)
|
||||
|
||||
|
||||
def _provider_head() -> nn.Module:
|
||||
return nn.Sequential(
|
||||
nn.Linear(124, 64),
|
||||
nn.ReLU(),
|
||||
nn.Dropout(0.2),
|
||||
nn.Linear(64, 1),
|
||||
)
|
||||
|
||||
|
||||
def _focal_loss(logits: torch.Tensor, targets: torch.Tensor) -> torch.Tensor:
|
||||
bce = nn.functional.binary_cross_entropy_with_logits(logits, targets, reduction="none")
|
||||
prob = torch.sigmoid(logits)
|
||||
pt = torch.where(targets > 0.5, prob, 1.0 - prob)
|
||||
return ((1.0 - pt) ** FOCAL_GAMMA * bce).mean()
|
||||
|
||||
|
||||
def _train_detector(emb: dict[str, np.ndarray], catalog: dict) -> tuple[nn.Module, float]:
|
||||
torch.manual_seed(DETECTOR_SEED)
|
||||
roles = {row["sha256"]: set(row["roles"]) for row in catalog["rows"]}
|
||||
pos, neg = [], []
|
||||
for sha, vector in emb.items():
|
||||
role = roles.get(sha, set())
|
||||
if "detector_ai_train" in role or "detector_ai_extra" in role:
|
||||
pos.append(vector)
|
||||
if "detector_photo_train" in role or "detector_struct" in role:
|
||||
neg.append(vector)
|
||||
x_pos = torch.from_numpy(np.stack(pos)).float()
|
||||
x_neg = torch.from_numpy(np.stack(neg)).float()
|
||||
log.info("detector train pos=%d neg=%d", len(x_pos), len(x_neg))
|
||||
model = _mlp(x_pos.shape[1])
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=LR, weight_decay=1e-3)
|
||||
loss_fn = nn.BCEWithLogitsLoss()
|
||||
rng = np.random.default_rng(DETECTOR_SEED)
|
||||
model.train()
|
||||
for _epoch in range(EPOCHS):
|
||||
pp = rng.permutation(len(x_pos))
|
||||
pn = rng.permutation(len(x_neg))
|
||||
for start in range(0, min(len(pp), len(pn)), BATCH):
|
||||
features = torch.cat([x_pos[pp[start : start + BATCH]], x_neg[pn[start : start + BATCH]]])
|
||||
labels = torch.cat([torch.ones(min(BATCH, len(pp) - start)), torch.zeros(min(BATCH, len(pn) - start))])
|
||||
optimizer.zero_grad()
|
||||
loss_fn(model(features).squeeze(-1), labels).backward()
|
||||
optimizer.step()
|
||||
model.eval()
|
||||
dev = np.stack(
|
||||
[
|
||||
emb[row["sha256"]]
|
||||
for row in catalog["rows"]
|
||||
if "detector_photo_dev_oi" in row["roles"] and row["sha256"] in emb
|
||||
]
|
||||
)
|
||||
with torch.inference_mode():
|
||||
scores = model(torch.from_numpy(dev).float()).squeeze(-1).numpy()
|
||||
threshold = float(np.quantile(scores, 1.0 - TARGET_PHOTO_FPR))
|
||||
return model, threshold
|
||||
|
||||
|
||||
def _train_provider(feat: dict[str, np.ndarray], catalog: dict) -> dict[str, nn.Module]:
|
||||
def stack(role: str) -> np.ndarray:
|
||||
vecs = [feat[row["sha256"]] for row in catalog["rows"] if role in row["roles"] and row["sha256"] in feat]
|
||||
return np.stack(vecs) if vecs else np.empty((0, 124), dtype=np.float32)
|
||||
|
||||
extra = stack("provider_openai_extra")
|
||||
openai = stack("provider_openai")
|
||||
bank = {
|
||||
"openai": np.concatenate([openai, extra]) if len(extra) else openai,
|
||||
"google": stack("provider_google"),
|
||||
"tc260": stack("provider_tc260"),
|
||||
"meta_muse_image": stack("provider_meta_train"),
|
||||
"no_ai": stack("provider_no_ai"),
|
||||
}
|
||||
struct = stack("provider_struct")
|
||||
all_neg = np.concatenate([bank["no_ai"], struct]) if len(struct) else bank["no_ai"]
|
||||
heads = {}
|
||||
for i, name in enumerate(LABELS):
|
||||
if name == "no_ai":
|
||||
positive = all_neg
|
||||
negative = np.concatenate([bank[n] for n in LABELS if n != "no_ai"])
|
||||
else:
|
||||
negative = np.concatenate([bank[n] for n in LABELS if n not in (name, "no_ai")] + [all_neg])
|
||||
positive = bank[name]
|
||||
torch.manual_seed(PROVIDER_SEED + i)
|
||||
rng = np.random.default_rng(PROVIDER_SEED + i)
|
||||
max_neg = min(len(negative), len(positive) * 2)
|
||||
neg_idx = rng.permutation(len(negative))[:max_neg]
|
||||
x_pos = torch.from_numpy(positive.astype(np.float32))
|
||||
x_neg = torch.from_numpy(negative[neg_idx].astype(np.float32))
|
||||
model = _provider_head()
|
||||
optimizer = torch.optim.AdamW(model.parameters(), lr=FOCAL_LR, weight_decay=1e-3)
|
||||
model.train()
|
||||
for _epoch in range(FOCAL_EPOCHS):
|
||||
pos_idx = rng.permutation(len(x_pos))
|
||||
neg_order = rng.permutation(len(x_neg))
|
||||
for start in range(0, min(len(pos_idx), len(neg_order)), FOCAL_BATCH):
|
||||
features = torch.cat(
|
||||
[x_pos[pos_idx[start : start + FOCAL_BATCH]], x_neg[neg_order[start : start + FOCAL_BATCH]]]
|
||||
)
|
||||
labels = torch.cat(
|
||||
[
|
||||
torch.ones(min(FOCAL_BATCH, len(pos_idx) - start)),
|
||||
torch.zeros(min(FOCAL_BATCH, len(neg_order) - start)),
|
||||
]
|
||||
)
|
||||
optimizer.zero_grad()
|
||||
_focal_loss(model(features).squeeze(-1), labels).backward()
|
||||
optimizer.step()
|
||||
heads[name] = model.eval()
|
||||
log.info("provider %s pos=%d", name, len(positive))
|
||||
return heads
|
||||
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--pack", type=Path, required=True)
|
||||
parser.add_argument("--out", type=Path, required=True)
|
||||
args = parser.parse_args()
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
pack = args.pack.expanduser().resolve()
|
||||
catalog = json.loads((pack / "catalog.json").read_text())
|
||||
emb: dict[str, np.ndarray] = {}
|
||||
for name in ("embeddings-by-sha256.npz", "extra-clip-l.npz"):
|
||||
path = pack / name
|
||||
if path.is_file():
|
||||
emb.update(_load_sha_vectors(path))
|
||||
feat = _load_sha_vectors(pack / "features-124d.npz")
|
||||
probe = np.load(pack / "probe-weights-clip-l-ft.npz")
|
||||
log.info("catalog=%d embeddings=%d features=%d", catalog["counts"]["files"], len(emb), len(feat))
|
||||
args.out.mkdir(parents=True, exist_ok=True)
|
||||
detector, threshold = _train_detector(emb, catalog)
|
||||
torch.save(detector.state_dict(), args.out / "detector.pt")
|
||||
heads = _train_provider(feat, catalog)
|
||||
torch.save({name: head.state_dict() for name, head in heads.items()}, args.out / "provider.pt")
|
||||
(args.out / "retrain.json").write_text(
|
||||
json.dumps({"detector_threshold": threshold, "probe_thr": float(probe["thr_oi_1pct"])}, indent=2) + "\n"
|
||||
)
|
||||
log.info("wrote %s threshold=%.6f", args.out, threshold)
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,197 @@
|
||||
"""Suppress the recovered periodic carrier without image regeneration.
|
||||
|
||||
This research tool controls the project's local fixed-template score. A local
|
||||
score reversal is not evidence that a provider SynthID verifier will stop
|
||||
detecting the image.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
import click
|
||||
from PIL import Image
|
||||
from synthid_pixel_attack import load_rgb, measure # pyright: ignore[reportUnknownVariableType]
|
||||
from synthid_research_manifest import artifact_sha256
|
||||
from synthid_runtime.synthid_detector import (
|
||||
TILE_THRESHOLD,
|
||||
_geometry_supported, # pyright: ignore[reportPrivateUsage]
|
||||
_load_template, # pyright: ignore[reportPrivateUsage]
|
||||
folded_template_score,
|
||||
)
|
||||
from synthid_tile_attack import subtract_tiled_template
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from numpy.typing import NDArray
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def apply_template(pixels: NDArray[Any], template: NDArray[Any], *, amplitude: float) -> NDArray[Any]:
|
||||
"""Subtract AMPLITUDE times periodic TEMPLATE from arbitrary RGB PIXELS."""
|
||||
return subtract_tiled_template(pixels, template, strength=amplitude)
|
||||
|
||||
|
||||
def carrier_score(pixels: NDArray[Any], template: NDArray[Any], sigma: float) -> float:
|
||||
"""Return the local fixed-template carrier score for PIXELS."""
|
||||
score, _folded = folded_template_score(pixels, template, sigma)
|
||||
return score
|
||||
|
||||
|
||||
def find_minimum_amplitude(
|
||||
pixels: NDArray[Any],
|
||||
template: NDArray[Any],
|
||||
sigma: float,
|
||||
*,
|
||||
target_score: float,
|
||||
maximum_amplitude: float,
|
||||
iterations: int,
|
||||
) -> tuple[float, NDArray[Any], float]:
|
||||
"""Return the smallest searched amplitude whose score reaches TARGET_SCORE."""
|
||||
if not math.isfinite(target_score):
|
||||
raise ValueError("target score must be finite")
|
||||
if not math.isfinite(maximum_amplitude) or maximum_amplitude <= 0.0:
|
||||
raise ValueError("maximum amplitude must be finite and positive")
|
||||
if iterations < 1:
|
||||
raise ValueError("iterations must be positive")
|
||||
maximum_pixels = apply_template(pixels, template, amplitude=maximum_amplitude)
|
||||
maximum_score = carrier_score(maximum_pixels, template, sigma)
|
||||
if maximum_score > target_score:
|
||||
raise ValueError(
|
||||
f"maximum amplitude {maximum_amplitude:g} reached score {maximum_score:.6f}, "
|
||||
f"above target {target_score:.6f}"
|
||||
)
|
||||
|
||||
low = 0.0
|
||||
high = maximum_amplitude
|
||||
best_pixels = maximum_pixels
|
||||
best_score = maximum_score
|
||||
for _iteration in range(iterations):
|
||||
middle = (low + high) / 2.0
|
||||
candidate = apply_template(pixels, template, amplitude=middle)
|
||||
candidate_score = carrier_score(candidate, template, sigma)
|
||||
if candidate_score <= target_score:
|
||||
high = middle
|
||||
best_pixels = candidate
|
||||
best_score = candidate_score
|
||||
else:
|
||||
low = middle
|
||||
return high, best_pixels, best_score
|
||||
|
||||
|
||||
def suppress_carrier(
|
||||
pixels: NDArray[Any],
|
||||
*,
|
||||
target_score: float = -0.25,
|
||||
maximum_amplitude: float = 40.0,
|
||||
iterations: int = 8,
|
||||
) -> tuple[NDArray[Any], dict[str, float | int | str]]:
|
||||
"""Suppress a locally detected carrier and return pixels plus measurements."""
|
||||
height, width = pixels.shape[:2]
|
||||
if not _geometry_supported(width, height):
|
||||
raise ValueError(f"unsupported decoded geometry: {width}x{height}")
|
||||
if target_score >= TILE_THRESHOLD:
|
||||
raise ValueError(f"target score must be below the detector threshold {TILE_THRESHOLD:.6f}")
|
||||
template, sigma, _model_height, _model_width, tile_height, tile_width = _load_template()
|
||||
original_score = carrier_score(pixels, template, sigma)
|
||||
if original_score < TILE_THRESHOLD:
|
||||
raise ValueError(
|
||||
f"local carrier is not detected: score {original_score:.6f} is below threshold {TILE_THRESHOLD:.6f}"
|
||||
)
|
||||
|
||||
started = time.perf_counter()
|
||||
amplitude, candidate, candidate_score = find_minimum_amplitude(
|
||||
pixels,
|
||||
template,
|
||||
sigma,
|
||||
target_score=target_score,
|
||||
maximum_amplitude=maximum_amplitude,
|
||||
iterations=iterations,
|
||||
)
|
||||
quality = measure(pixels, candidate, name="adaptive-carrier", path=Path("<memory>"))
|
||||
return candidate, {
|
||||
"status": "local_carrier_suppressed",
|
||||
"detector_scope": "local fixed-template carrier, not provider-verified SynthID removal",
|
||||
"width": width,
|
||||
"height": height,
|
||||
"tile_height": tile_height,
|
||||
"tile_width": tile_width,
|
||||
"threshold": TILE_THRESHOLD,
|
||||
"target_score": target_score,
|
||||
"original_score": original_score,
|
||||
"candidate_score": candidate_score,
|
||||
"amplitude": amplitude,
|
||||
"maximum_amplitude": maximum_amplitude,
|
||||
"iterations": iterations,
|
||||
"residual_rms": quality.residual_rms,
|
||||
"psnr_db": quality.psnr_db,
|
||||
"ssim": quality.ssim,
|
||||
"changed_pixel_fraction": quality.changed_pixel_fraction,
|
||||
"elapsed_seconds": time.perf_counter() - started,
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("output", type=click.Path(dir_okay=False, path_type=Path))
|
||||
@click.option("--target-score", type=float, default=-0.25, show_default=True)
|
||||
@click.option("--maximum-amplitude", type=click.FloatRange(min=0.0, min_open=True), default=40.0, show_default=True)
|
||||
@click.option("--iterations", type=click.IntRange(min=1), default=8, show_default=True)
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path))
|
||||
def main(
|
||||
source: Path,
|
||||
output: Path,
|
||||
target_score: float,
|
||||
maximum_amplitude: float,
|
||||
iterations: int,
|
||||
report_out: Path | None,
|
||||
) -> None:
|
||||
"""Write a lossless PNG with the recovered local carrier suppressed."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
if output.suffix.lower() != ".png":
|
||||
raise click.BadParameter("output must use the .png extension", param_hint="output")
|
||||
report_path = report_out or output.with_suffix(".json")
|
||||
for path in (output, report_path):
|
||||
if path.exists():
|
||||
raise click.ClickException(f"refusing to overwrite existing file: {path}")
|
||||
try:
|
||||
candidate, report = suppress_carrier(
|
||||
load_rgb(source), # pyright: ignore[reportUnknownArgumentType]
|
||||
target_score=target_score,
|
||||
maximum_amplitude=maximum_amplitude,
|
||||
iterations=iterations,
|
||||
)
|
||||
except ValueError as error:
|
||||
raise click.ClickException(str(error)) from error
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
Image.fromarray(candidate, mode="RGB").save(output, format="PNG", compress_level=9)
|
||||
report.update(
|
||||
{
|
||||
"source": str(source.resolve()),
|
||||
"source_sha256": artifact_sha256(source),
|
||||
"output": str(output.resolve()),
|
||||
"output_sha256": artifact_sha256(output),
|
||||
}
|
||||
)
|
||||
report_path.write_text(json.dumps(report, indent=2, allow_nan=False) + "\n", encoding="utf-8")
|
||||
log.info(
|
||||
"Suppressed local carrier %.6f -> %.6f at %.2f dB PSNR; wrote %s",
|
||||
report["original_score"],
|
||||
report["candidate_score"],
|
||||
report["psnr_db"],
|
||||
output,
|
||||
)
|
||||
log.info("Research caveat: this is not provider-verified SynthID removal")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,353 @@
|
||||
"""Compare an exact-geometry phase carrier across color spaces.
|
||||
|
||||
Every model uses the same spatial-frequency candidates. Phases, expected
|
||||
magnitudes, selected channels, and weights are learned independently from the
|
||||
supplied positive images. The resulting scores are experimental evidence, not
|
||||
a proprietary SynthID decoder or a certified production detector.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from synthid_phase_carrier import _leave_one_out_coherence
|
||||
from synthid_phase_registration import extract_frequency_values
|
||||
from synthid_v3_codebook_probe import load_v3_model
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
COLOR_SPACES = ("rgb", "ycbcr", "ycocg", "opponent", "lab", "hsv")
|
||||
CHANNEL_NAMES = {
|
||||
"rgb": ("R", "G", "B"),
|
||||
"ycbcr": ("Y", "Cb", "Cr"),
|
||||
"ycocg": ("Y", "Co", "Cg"),
|
||||
"opponent": ("L", "R-G", "R+G-2B"),
|
||||
"lab": ("L*", "a*", "b*"),
|
||||
"hsv": ("H", "S", "V"),
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ColorPhaseModel:
|
||||
"""Sparse phase carrier learned in one color space."""
|
||||
|
||||
color_space: str
|
||||
height: int
|
||||
width: int
|
||||
rows: np.ndarray
|
||||
columns: np.ndarray
|
||||
channels: np.ndarray
|
||||
phases: np.ndarray
|
||||
weights: np.ndarray
|
||||
expected_magnitudes: np.ndarray
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ColorPhaseScore:
|
||||
"""Phase-carrier evidence and channel contributions for one image."""
|
||||
|
||||
path: str
|
||||
color_space: str
|
||||
phase_score: float
|
||||
active_weight_fraction: float
|
||||
evidence_score: float
|
||||
channel_evidence: tuple[float, float, float]
|
||||
selected_peak_counts: tuple[int, int, int]
|
||||
peak_count: int
|
||||
|
||||
|
||||
def transform_color_space(rgb: np.ndarray, color_space: str) -> np.ndarray:
|
||||
"""Transform float RGB values in [0, 255] into COLOR_SPACE."""
|
||||
if rgb.ndim != 3 or rgb.shape[2] != 3:
|
||||
raise ValueError("rgb must have shape (height, width, 3)")
|
||||
pixels = np.asarray(rgb, dtype=np.float64)
|
||||
red, green, blue = np.moveaxis(pixels, 2, 0)
|
||||
if color_space == "rgb":
|
||||
transformed = pixels
|
||||
elif color_space == "ycbcr":
|
||||
transformed = np.stack(
|
||||
(
|
||||
0.299 * red + 0.587 * green + 0.114 * blue,
|
||||
128.0 - 0.168736 * red - 0.331264 * green + 0.5 * blue,
|
||||
128.0 + 0.5 * red - 0.418688 * green - 0.081312 * blue,
|
||||
),
|
||||
axis=2,
|
||||
)
|
||||
elif color_space == "ycocg":
|
||||
transformed = np.stack(
|
||||
(
|
||||
0.25 * red + 0.5 * green + 0.25 * blue,
|
||||
0.5 * red - 0.5 * blue,
|
||||
-0.25 * red + 0.5 * green - 0.25 * blue,
|
||||
),
|
||||
axis=2,
|
||||
)
|
||||
elif color_space == "opponent":
|
||||
transformed = np.stack(
|
||||
(
|
||||
(red + green + blue) / np.sqrt(3.0),
|
||||
(red - green) / np.sqrt(2.0),
|
||||
(red + green - 2.0 * blue) / np.sqrt(6.0),
|
||||
),
|
||||
axis=2,
|
||||
)
|
||||
elif color_space == "lab":
|
||||
transformed = cv2.cvtColor((pixels / 255.0).astype(np.float32), cv2.COLOR_RGB2LAB).astype(np.float64)
|
||||
elif color_space == "hsv":
|
||||
transformed = cv2.cvtColor((pixels / 255.0).astype(np.float32), cv2.COLOR_RGB2HSV).astype(np.float64)
|
||||
else:
|
||||
raise ValueError(f"unsupported color space: {color_space}")
|
||||
if not np.all(np.isfinite(transformed)):
|
||||
raise ValueError(f"{color_space} transform produced non-finite values")
|
||||
return transformed
|
||||
|
||||
|
||||
def candidate_bins_from_codebook(
|
||||
codebook_path: Path,
|
||||
*,
|
||||
height: int,
|
||||
width: int,
|
||||
source_peak_count: int = 256,
|
||||
) -> np.ndarray:
|
||||
"""Expand the codebook's unique spatial coordinates over three channels."""
|
||||
prior = load_v3_model(
|
||||
codebook_path,
|
||||
height=height,
|
||||
width=width,
|
||||
peak_count=source_peak_count,
|
||||
)
|
||||
spatial = np.unique(np.column_stack((prior.rows, prior.columns)), axis=0)
|
||||
return np.asarray(
|
||||
[(int(row), int(column), channel) for row, column in spatial for channel in range(3)],
|
||||
dtype=np.int32,
|
||||
)
|
||||
|
||||
|
||||
def _load_rgb(path: Path, *, height: int, width: int) -> np.ndarray:
|
||||
"""Load an exact-geometry image as float64 RGB."""
|
||||
with Image.open(path) as image:
|
||||
rgb = image.convert("RGB")
|
||||
if rgb.size != (width, height):
|
||||
raise ValueError(f"{path}: geometry {rgb.width}x{rgb.height} does not match {width}x{height}")
|
||||
return np.asarray(rgb, dtype=np.float64)
|
||||
|
||||
|
||||
def discover_model(
|
||||
paths: list[Path],
|
||||
*,
|
||||
color_space: str,
|
||||
candidate_bins: np.ndarray,
|
||||
peak_count: int = 256,
|
||||
) -> ColorPhaseModel:
|
||||
"""Learn one color-space phase carrier from exact-geometry PATHS."""
|
||||
if len(paths) < 3:
|
||||
raise ValueError("at least three positive images are required")
|
||||
if color_space not in COLOR_SPACES:
|
||||
raise ValueError(f"unsupported color space: {color_space}")
|
||||
with Image.open(paths[0]) as first:
|
||||
width, height = first.size
|
||||
bins = np.asarray(candidate_bins, dtype=np.int32)
|
||||
if bins.ndim != 2 or bins.shape[1] != 3:
|
||||
raise ValueError("candidate_bins must have shape (count, 3)")
|
||||
if len(bins) < peak_count:
|
||||
raise ValueError(f"only {len(bins)} candidate bins for {peak_count} peaks")
|
||||
if (
|
||||
np.any(bins[:, 0] < 0)
|
||||
or np.any(bins[:, 0] >= height)
|
||||
or np.any(bins[:, 1] <= 0)
|
||||
or np.any(bins[:, 1] > width // 2)
|
||||
or np.any(bins[:, 2] < 0)
|
||||
or np.any(bins[:, 2] > 2)
|
||||
):
|
||||
raise ValueError("candidate_bins contain out-of-range coordinates")
|
||||
|
||||
image_values = np.empty((len(paths), len(bins)), dtype=np.complex128)
|
||||
for index, path in enumerate(paths):
|
||||
rgb = _load_rgb(path, height=height, width=width)
|
||||
image_values[index] = extract_frequency_values(
|
||||
transform_color_space(rgb, color_space),
|
||||
bins[:, 0],
|
||||
bins[:, 1],
|
||||
bins[:, 2],
|
||||
)
|
||||
magnitudes = np.abs(image_values)
|
||||
units = np.divide(image_values, magnitudes, out=np.zeros_like(image_values), where=magnitudes != 0.0)
|
||||
unit_sum = np.sum(units, axis=0)
|
||||
count = float(len(paths))
|
||||
minimum_loo = np.ones(len(bins), dtype=np.float64)
|
||||
for unit in units:
|
||||
minimum_loo = np.minimum(minimum_loo, _leave_one_out_coherence(unit_sum, unit, count))
|
||||
expected_magnitudes = np.mean(magnitudes, axis=0)
|
||||
selection = np.power(minimum_loo, 4.0) * np.log1p(expected_magnitudes)
|
||||
chosen = np.argpartition(selection, -peak_count)[-peak_count:]
|
||||
chosen = chosen[np.argsort(selection[chosen])[::-1]]
|
||||
raw_weights = selection[chosen]
|
||||
if np.sum(raw_weights) <= 0.0:
|
||||
raise ValueError("candidate bins have no usable phase consensus")
|
||||
selected = bins[chosen]
|
||||
return ColorPhaseModel(
|
||||
color_space=color_space,
|
||||
height=height,
|
||||
width=width,
|
||||
rows=selected[:, 0].astype(np.int32),
|
||||
columns=selected[:, 1].astype(np.int32),
|
||||
channels=selected[:, 2].astype(np.int8),
|
||||
phases=np.angle(unit_sum[chosen] / count).astype(np.float64),
|
||||
weights=(raw_weights / np.sum(raw_weights)).astype(np.float64),
|
||||
expected_magnitudes=expected_magnitudes[chosen].astype(np.float64),
|
||||
)
|
||||
|
||||
|
||||
def score_image(path: Path, model: ColorPhaseModel) -> ColorPhaseScore:
|
||||
"""Score PATH against MODEL and expose additive channel evidence."""
|
||||
rgb = _load_rgb(path, height=model.height, width=model.width)
|
||||
bins = np.column_stack((model.rows, model.columns, model.channels))
|
||||
values = extract_frequency_values(
|
||||
transform_color_space(rgb, model.color_space),
|
||||
bins[:, 0],
|
||||
bins[:, 1],
|
||||
bins[:, 2],
|
||||
)
|
||||
magnitude_gate = np.minimum(np.abs(values) / (model.expected_magnitudes + 1e-12), 1.0)
|
||||
contributions = model.weights * magnitude_gate * np.cos(np.angle(values) - model.phases)
|
||||
active_weights = model.weights * magnitude_gate
|
||||
active_weight = float(np.sum(active_weights))
|
||||
evidence = float(np.sum(contributions))
|
||||
phase_score = 0.0 if active_weight == 0.0 else evidence / active_weight
|
||||
channel_evidence = tuple(float(np.sum(contributions[model.channels == channel])) for channel in range(3))
|
||||
selected_peak_counts = tuple(int(np.sum(model.channels == channel)) for channel in range(3))
|
||||
return ColorPhaseScore(
|
||||
path=str(path),
|
||||
color_space=model.color_space,
|
||||
phase_score=phase_score,
|
||||
active_weight_fraction=active_weight,
|
||||
evidence_score=evidence,
|
||||
channel_evidence=channel_evidence,
|
||||
selected_peak_counts=selected_peak_counts,
|
||||
peak_count=len(model.rows),
|
||||
)
|
||||
|
||||
|
||||
def save_model(path: Path, model: ColorPhaseModel) -> None:
|
||||
"""Save MODEL as a pickle-free numeric NPZ."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
np.savez_compressed(
|
||||
path,
|
||||
format_version=np.asarray(1, dtype=np.int32),
|
||||
color_space=np.asarray(model.color_space),
|
||||
height=np.asarray(model.height, dtype=np.int32),
|
||||
width=np.asarray(model.width, dtype=np.int32),
|
||||
rows=model.rows.astype(np.int32),
|
||||
columns=model.columns.astype(np.int32),
|
||||
channels=model.channels.astype(np.int8),
|
||||
phases=model.phases.astype(np.float32),
|
||||
weights=model.weights.astype(np.float32),
|
||||
expected_magnitudes=model.expected_magnitudes.astype(np.float64),
|
||||
)
|
||||
|
||||
|
||||
def load_model(path: Path) -> ColorPhaseModel:
|
||||
"""Load and validate one color-space phase-carrier artifact."""
|
||||
with np.load(path, allow_pickle=False) as artifact:
|
||||
if int(artifact["format_version"]) != 1:
|
||||
raise ValueError("unsupported color-phase model format version")
|
||||
model = ColorPhaseModel(
|
||||
color_space=str(artifact["color_space"]),
|
||||
height=int(artifact["height"]),
|
||||
width=int(artifact["width"]),
|
||||
rows=np.asarray(artifact["rows"], dtype=np.int32),
|
||||
columns=np.asarray(artifact["columns"], dtype=np.int32),
|
||||
channels=np.asarray(artifact["channels"], dtype=np.int8),
|
||||
phases=np.asarray(artifact["phases"], dtype=np.float64),
|
||||
weights=np.asarray(artifact["weights"], dtype=np.float64),
|
||||
expected_magnitudes=np.asarray(artifact["expected_magnitudes"], dtype=np.float64),
|
||||
)
|
||||
count = len(model.rows)
|
||||
arrays = (model.columns, model.channels, model.phases, model.weights, model.expected_magnitudes)
|
||||
if model.color_space not in COLOR_SPACES:
|
||||
raise ValueError("invalid model color space")
|
||||
if model.height < 64 or model.width < 64 or count == 0 or any(array.shape != (count,) for array in arrays):
|
||||
raise ValueError("invalid color-phase model shapes")
|
||||
if np.any(model.rows < 0) or np.any(model.rows >= model.height):
|
||||
raise ValueError("invalid color-phase row indices")
|
||||
if np.any(model.columns <= 0) or np.any(model.columns > model.width // 2):
|
||||
raise ValueError("invalid color-phase column indices")
|
||||
if np.any(model.channels < 0) or np.any(model.channels > 2):
|
||||
raise ValueError("invalid color-phase channel indices")
|
||||
if not np.isclose(np.sum(model.weights), 1.0, atol=1e-5) or np.any(model.weights < 0.0):
|
||||
raise ValueError("invalid color-phase weights")
|
||||
if np.any(model.expected_magnitudes < 0.0):
|
||||
raise ValueError("invalid expected magnitudes")
|
||||
return model
|
||||
|
||||
|
||||
@click.group()
|
||||
def main() -> None:
|
||||
"""Discover and score phase carriers in multiple color spaces."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("codebook", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("positives", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--color-space", type=click.Choice(COLOR_SPACES), required=True)
|
||||
@click.option("--source-peak-count", type=click.IntRange(min=1), default=256, show_default=True)
|
||||
@click.option("--peak-count", type=click.IntRange(min=1), default=256, show_default=True)
|
||||
@click.option("--model-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def discover(
|
||||
codebook: Path,
|
||||
positives: tuple[Path, ...],
|
||||
color_space: str,
|
||||
source_peak_count: int,
|
||||
peak_count: int,
|
||||
model_out: Path,
|
||||
) -> None:
|
||||
"""Learn a color-space carrier from exact-geometry POSITIVES."""
|
||||
with Image.open(positives[0]) as first:
|
||||
width, height = first.size
|
||||
candidates = candidate_bins_from_codebook(
|
||||
codebook,
|
||||
height=height,
|
||||
width=width,
|
||||
source_peak_count=source_peak_count,
|
||||
)
|
||||
model = discover_model(
|
||||
list(positives),
|
||||
color_space=color_space,
|
||||
candidate_bins=candidates,
|
||||
peak_count=peak_count,
|
||||
)
|
||||
save_model(model_out, model)
|
||||
log.info("Wrote %s color-phase model with %s candidates: %s", color_space, len(candidates), model_out)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def score(model_path: Path, images: tuple[Path, ...], report_out: Path) -> None:
|
||||
"""Score exact-geometry IMAGES with MODEL_PATH."""
|
||||
model = load_model(model_path)
|
||||
payload = {
|
||||
"model": str(model_path),
|
||||
"color_space": model.color_space,
|
||||
"channel_names": CHANNEL_NAMES[model.color_space],
|
||||
"height": model.height,
|
||||
"width": model.width,
|
||||
"peak_count": len(model.rows),
|
||||
"scores": [asdict(score_image(image, model)) for image in images],
|
||||
}
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||
log.info("Wrote %s color-phase score report: %s", model.color_space, report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,352 @@
|
||||
"""Calibrate a versioned SynthID expert bank without forcing binary verdicts.
|
||||
|
||||
This research utility combines already-computed pixel-only expert scores. It
|
||||
does not inspect provenance, metadata, filenames, or provider labels at
|
||||
inference. Expert support must be determined from predeclared geometry or model
|
||||
scope, never from the observed score.
|
||||
|
||||
The clean null is a union test: any supported expert may provide positive
|
||||
evidence, so its smallest empirical upper-tail p-value receives a Bonferroni
|
||||
correction. The watermarked hypothesis is itself a union over possible encoder
|
||||
states and can be rejected only when every configured expert has complete
|
||||
coverage and gives a small empirical lower-tail p-value.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import bisect
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import Literal, cast
|
||||
|
||||
import click
|
||||
from synthid_research_manifest import artifact_sha256
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
CascadeVerdict = Literal["detected", "not_detected", "abstain"]
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpertCalibration:
|
||||
"""Frozen positive and negative score distributions for one expert."""
|
||||
|
||||
name: str
|
||||
positive_scores: tuple[float, ...]
|
||||
negative_scores: tuple[float, ...]
|
||||
higher_is_positive: bool = True
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
raise ValueError("expert name must not be empty")
|
||||
if not self.positive_scores or not self.negative_scores:
|
||||
raise ValueError(f"expert {self.name!r} needs positive and negative calibration scores")
|
||||
if not all(math.isfinite(score) for score in (*self.positive_scores, *self.negative_scores)):
|
||||
raise ValueError(f"expert {self.name!r} contains a non-finite calibration score")
|
||||
direction = 1.0 if self.higher_is_positive else -1.0
|
||||
object.__setattr__(self, "positive_scores", tuple(sorted(direction * score for score in self.positive_scores)))
|
||||
object.__setattr__(self, "negative_scores", tuple(sorted(direction * score for score in self.negative_scores)))
|
||||
|
||||
def orient(self, score: float) -> float:
|
||||
"""Return SCORE in the common higher-means-more-positive direction."""
|
||||
return score if self.higher_is_positive else -score
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CascadeConfig:
|
||||
"""Calibration distributions and two-sided decision levels."""
|
||||
|
||||
experts: tuple[ExpertCalibration, ...]
|
||||
positive_alpha: float
|
||||
negative_alpha: float
|
||||
coverage_complete: bool
|
||||
scope: str
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.experts:
|
||||
raise ValueError("at least one expert is required")
|
||||
names = [expert.name for expert in self.experts]
|
||||
if len(set(names)) != len(names):
|
||||
raise ValueError("expert names must be unique")
|
||||
for label, value in (("positive_alpha", self.positive_alpha), ("negative_alpha", self.negative_alpha)):
|
||||
if not 0.0 < value <= 1.0:
|
||||
raise ValueError(f"{label} must be in (0, 1]")
|
||||
if not self.scope:
|
||||
raise ValueError("detector scope must not be empty")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpertObservation:
|
||||
"""One expert score, or an explicit unsupported result."""
|
||||
|
||||
name: str
|
||||
supported: bool
|
||||
score: float | None
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
if not self.name:
|
||||
raise ValueError("observation expert name must not be empty")
|
||||
if self.supported:
|
||||
if self.score is None or not math.isfinite(self.score):
|
||||
raise ValueError(f"supported expert {self.name!r} needs a finite score")
|
||||
elif self.score is not None:
|
||||
raise ValueError(f"unsupported expert {self.name!r} must not provide a score")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ExpertEvidence:
|
||||
"""Two empirical p-values for one supported expert."""
|
||||
|
||||
name: str
|
||||
score: float
|
||||
clean_null_p_value: float
|
||||
watermarked_p_value: float
|
||||
positive_calibration_count: int
|
||||
negative_calibration_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CascadeResult:
|
||||
"""Auditable tri-state verdict for one observation record."""
|
||||
|
||||
verdict: CascadeVerdict
|
||||
reason: str
|
||||
clean_null_p_value: float | None
|
||||
watermarked_p_value: float | None
|
||||
supported_expert_count: int
|
||||
configured_expert_count: int
|
||||
coverage_complete: bool
|
||||
evidence: tuple[ExpertEvidence, ...]
|
||||
|
||||
|
||||
def _upper_tail_p_value(sorted_scores: tuple[float, ...], score: float) -> float:
|
||||
"""Smoothed empirical probability of a calibration score at least SCORE."""
|
||||
tail_count = len(sorted_scores) - bisect.bisect_left(sorted_scores, score)
|
||||
return (tail_count + 1.0) / (len(sorted_scores) + 1.0)
|
||||
|
||||
|
||||
def _lower_tail_p_value(sorted_scores: tuple[float, ...], score: float) -> float:
|
||||
"""Smoothed empirical probability of a calibration score at most SCORE."""
|
||||
tail_count = bisect.bisect_right(sorted_scores, score)
|
||||
return (tail_count + 1.0) / (len(sorted_scores) + 1.0)
|
||||
|
||||
|
||||
def classify_observations(config: CascadeConfig, observations: tuple[ExpertObservation, ...]) -> CascadeResult:
|
||||
"""Combine one explicit observation from every configured expert."""
|
||||
calibration_by_name = {expert.name: expert for expert in config.experts}
|
||||
observation_by_name = {observation.name: observation for observation in observations}
|
||||
if len(observation_by_name) != len(observations):
|
||||
raise ValueError("observation expert names must be unique")
|
||||
if observation_by_name.keys() != calibration_by_name.keys():
|
||||
missing = sorted(calibration_by_name.keys() - observation_by_name.keys())
|
||||
unknown = sorted(observation_by_name.keys() - calibration_by_name.keys())
|
||||
raise ValueError(f"observations must cover the configured bank; missing={missing}, unknown={unknown}")
|
||||
|
||||
evidence: list[ExpertEvidence] = []
|
||||
for calibration in config.experts:
|
||||
observation = observation_by_name[calibration.name]
|
||||
if not observation.supported:
|
||||
continue
|
||||
if observation.score is None:
|
||||
raise RuntimeError("validated supported observation lost its score")
|
||||
oriented_score = calibration.orient(observation.score)
|
||||
evidence.append(
|
||||
ExpertEvidence(
|
||||
name=calibration.name,
|
||||
score=observation.score,
|
||||
clean_null_p_value=_upper_tail_p_value(calibration.negative_scores, oriented_score),
|
||||
watermarked_p_value=_lower_tail_p_value(calibration.positive_scores, oriented_score),
|
||||
positive_calibration_count=len(calibration.positive_scores),
|
||||
negative_calibration_count=len(calibration.negative_scores),
|
||||
)
|
||||
)
|
||||
|
||||
if not evidence:
|
||||
return CascadeResult(
|
||||
verdict="abstain",
|
||||
reason="unsupported",
|
||||
clean_null_p_value=None,
|
||||
watermarked_p_value=None,
|
||||
supported_expert_count=0,
|
||||
configured_expert_count=len(config.experts),
|
||||
coverage_complete=config.coverage_complete,
|
||||
evidence=(),
|
||||
)
|
||||
|
||||
supported_count = len(evidence)
|
||||
clean_null_p_value = min(1.0, supported_count * min(item.clean_null_p_value for item in evidence))
|
||||
watermarked_p_value = max(item.watermarked_p_value for item in evidence)
|
||||
rejects_clean_null = clean_null_p_value <= config.positive_alpha
|
||||
full_support = supported_count == len(config.experts)
|
||||
rejects_watermarked = config.coverage_complete and full_support and watermarked_p_value <= config.negative_alpha
|
||||
|
||||
if rejects_clean_null and rejects_watermarked:
|
||||
verdict: CascadeVerdict = "abstain"
|
||||
reason = "conflicting_evidence"
|
||||
elif rejects_clean_null:
|
||||
verdict = "detected"
|
||||
reason = "watermarked_hypothesis_supported"
|
||||
elif rejects_watermarked:
|
||||
verdict = "not_detected"
|
||||
reason = "unwatermarked_hypothesis_supported"
|
||||
elif config.coverage_complete and not full_support:
|
||||
verdict = "abstain"
|
||||
reason = "incomplete_support"
|
||||
elif not config.coverage_complete and watermarked_p_value <= config.negative_alpha:
|
||||
verdict = "abstain"
|
||||
reason = "incomplete_coverage"
|
||||
else:
|
||||
verdict = "abstain"
|
||||
reason = "insufficient_evidence"
|
||||
|
||||
return CascadeResult(
|
||||
verdict=verdict,
|
||||
reason=reason,
|
||||
clean_null_p_value=clean_null_p_value,
|
||||
watermarked_p_value=watermarked_p_value,
|
||||
supported_expert_count=supported_count,
|
||||
configured_expert_count=len(config.experts),
|
||||
coverage_complete=config.coverage_complete,
|
||||
evidence=tuple(evidence),
|
||||
)
|
||||
|
||||
|
||||
def _mapping(value: object, label: str) -> dict[str, object]:
|
||||
if not isinstance(value, dict):
|
||||
raise ValueError(f"{label} must be an object")
|
||||
return cast("dict[str, object]", value)
|
||||
|
||||
|
||||
def _sequence(value: object, label: str) -> list[object]:
|
||||
if not isinstance(value, list):
|
||||
raise ValueError(f"{label} must be an array")
|
||||
return cast("list[object]", value)
|
||||
|
||||
|
||||
def _scores(value: object, label: str) -> tuple[float, ...]:
|
||||
scores: list[float] = []
|
||||
for index, score in enumerate(_sequence(value, label)):
|
||||
if isinstance(score, bool) or not isinstance(score, (int, float)):
|
||||
raise ValueError(f"{label}[{index}] must be a number")
|
||||
scores.append(float(score))
|
||||
return tuple(scores)
|
||||
|
||||
|
||||
def _number(value: object, label: str) -> float:
|
||||
if isinstance(value, bool) or not isinstance(value, (int, float)):
|
||||
raise ValueError(f"{label} must be a number")
|
||||
return float(value)
|
||||
|
||||
|
||||
def _boolean(value: object, label: str) -> bool:
|
||||
if not isinstance(value, bool):
|
||||
raise ValueError(f"{label} must be a boolean")
|
||||
return value
|
||||
|
||||
|
||||
def _string(value: object, label: str) -> str:
|
||||
if not isinstance(value, str) or not value:
|
||||
raise ValueError(f"{label} must be a non-empty string")
|
||||
return value
|
||||
|
||||
|
||||
def load_config(path: Path) -> CascadeConfig:
|
||||
"""Load a schema-versioned calibration manifest."""
|
||||
payload = _mapping(json.loads(path.read_text(encoding="utf-8")), "calibration manifest")
|
||||
if payload.get("schema_version") != 1:
|
||||
raise ValueError("unsupported calibration manifest schema")
|
||||
experts: list[ExpertCalibration] = []
|
||||
for index, raw_expert in enumerate(_sequence(payload.get("experts"), "experts")):
|
||||
expert = _mapping(raw_expert, f"experts[{index}]")
|
||||
experts.append(
|
||||
ExpertCalibration(
|
||||
name=_string(expert.get("name"), f"experts[{index}].name"),
|
||||
positive_scores=_scores(expert.get("positive_scores"), f"experts[{index}].positive_scores"),
|
||||
negative_scores=_scores(expert.get("negative_scores"), f"experts[{index}].negative_scores"),
|
||||
higher_is_positive=_boolean(
|
||||
expert.get("higher_is_positive", True),
|
||||
f"experts[{index}].higher_is_positive",
|
||||
),
|
||||
)
|
||||
)
|
||||
return CascadeConfig(
|
||||
experts=tuple(experts),
|
||||
positive_alpha=_number(payload.get("positive_alpha"), "positive_alpha"),
|
||||
negative_alpha=_number(payload.get("negative_alpha"), "negative_alpha"),
|
||||
coverage_complete=_boolean(payload.get("coverage_complete", False), "coverage_complete"),
|
||||
scope=_string(payload.get("scope"), "scope"),
|
||||
)
|
||||
|
||||
|
||||
def load_observation_records(path: Path) -> list[tuple[str, tuple[ExpertObservation, ...]]]:
|
||||
"""Load named score records with explicit support for every expert."""
|
||||
payload = _mapping(json.loads(path.read_text(encoding="utf-8")), "observation manifest")
|
||||
if payload.get("schema_version") != 1:
|
||||
raise ValueError("unsupported observation manifest schema")
|
||||
records: list[tuple[str, tuple[ExpertObservation, ...]]] = []
|
||||
for record_index, raw_record in enumerate(_sequence(payload.get("records"), "records")):
|
||||
record = _mapping(raw_record, f"records[{record_index}]")
|
||||
record_id = _string(record.get("id"), f"records[{record_index}].id")
|
||||
observations: list[ExpertObservation] = []
|
||||
for observation_index, raw_observation in enumerate(
|
||||
_sequence(record.get("observations"), f"records[{record_index}].observations")
|
||||
):
|
||||
observation = _mapping(raw_observation, f"records[{record_index}].observations[{observation_index}]")
|
||||
raw_score = observation.get("score")
|
||||
observations.append(
|
||||
ExpertObservation(
|
||||
name=_string(
|
||||
observation.get("name"),
|
||||
f"records[{record_index}].observations[{observation_index}].name",
|
||||
),
|
||||
supported=_boolean(
|
||||
observation.get("supported", False),
|
||||
f"records[{record_index}].observations[{observation_index}].supported",
|
||||
),
|
||||
score=None
|
||||
if raw_score is None
|
||||
else _number(raw_score, f"records[{record_index}].observations[{observation_index}].score"),
|
||||
)
|
||||
)
|
||||
records.append((record_id, tuple(observations)))
|
||||
return records
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("calibration_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("observation_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def main(calibration_path: Path, observation_path: Path, report_out: Path) -> None:
|
||||
"""Classify precomputed expert scores using CALIBRATION_PATH."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
config = load_config(calibration_path)
|
||||
rows: list[dict[str, object]] = []
|
||||
verdict_counts: dict[CascadeVerdict, int] = {"detected": 0, "not_detected": 0, "abstain": 0}
|
||||
for record_id, observations in load_observation_records(observation_path):
|
||||
result = classify_observations(config, observations)
|
||||
verdict_counts[result.verdict] += 1
|
||||
rows.append({"id": record_id, "result": asdict(result)})
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"scope": config.scope,
|
||||
"calibration_sha256": artifact_sha256(calibration_path),
|
||||
"observation_sha256": artifact_sha256(observation_path),
|
||||
"counts": verdict_counts,
|
||||
"records": rows,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d conformal cascade verdicts: %s", len(rows), report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,478 @@
|
||||
"""Measure how well non-watermark confounds predict provider SynthID labels.
|
||||
|
||||
The experiment deliberately trains three provider-specific baselines:
|
||||
|
||||
``container``
|
||||
File size, decoded geometry, aspect ratio, and format.
|
||||
``thumbnail``
|
||||
Container features plus a small RGB thumbnail that can learn generator and
|
||||
content style.
|
||||
``canonical``
|
||||
Decoded, orientation-normalized RGB at a fixed geometry, with no container,
|
||||
original-resolution, metadata, filename, or path features.
|
||||
|
||||
These are challenge baselines, not SynthID detectors. A candidate detector must
|
||||
beat the canonical baseline on same-provider hard negatives and a temporal
|
||||
holdout before its result can be attributed to a watermark-specific signal.
|
||||
|
||||
Usage:
|
||||
uv run --extra pixels python scripts/synthid_confound_probe.py \
|
||||
.local-eval/synthid/manifest.csv --target-provider google \
|
||||
--report-out .local-eval/synthid/google-d1-confounds.json
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
from collections import Counter
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from PIL import Image, ImageOps
|
||||
from synthid_research_manifest import artifact_sha256, audit_manifest, resolve_artifact_path
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
FEATURE_FAMILIES = ("container", "thumbnail", "canonical")
|
||||
FINAL_SPLITS = ("train", "validation", "test", "temporal")
|
||||
THUMBNAIL_SIZE = 8
|
||||
FORMAT_NAMES = ("png", "jpeg", "webp")
|
||||
LOGISTIC_ITERATIONS = 2_000
|
||||
LOGISTIC_LEARNING_RATE = 0.2
|
||||
LOGISTIC_L2 = 0.01
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Example:
|
||||
"""One ordinary, provider-targeted manifest row eligible for D1."""
|
||||
|
||||
artifact_path: Path
|
||||
artifact_sha256: str
|
||||
group_id: str
|
||||
split: str
|
||||
label: int
|
||||
negative_cohort: str | None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LogisticModel:
|
||||
"""Standardization and regularized logistic-regression parameters."""
|
||||
|
||||
mean: np.ndarray
|
||||
scale: np.ndarray
|
||||
weights: np.ndarray
|
||||
bias: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Metrics:
|
||||
"""Binary metrics at one validation-frozen threshold."""
|
||||
|
||||
count: int
|
||||
positives: int
|
||||
negatives: int
|
||||
true_positives: int
|
||||
false_positives: int
|
||||
true_negatives: int
|
||||
false_negatives: int
|
||||
tpr: float | None
|
||||
fpr: float | None
|
||||
auc: float | None
|
||||
|
||||
|
||||
def _safe_artifact_path(root: Path, value: str) -> Path:
|
||||
"""Resolve a manifest-relative path without accepting traversal."""
|
||||
candidate = resolve_artifact_path(root, value)
|
||||
if candidate is None:
|
||||
raise ValueError(f"unsafe artifact_path {value!r}")
|
||||
return candidate
|
||||
|
||||
|
||||
def _read_manifest(path: Path) -> list[dict[str, str]]:
|
||||
"""Read manifest rows after the caller has run the canonical auditor."""
|
||||
with path.open(newline="", encoding="utf-8") as stream:
|
||||
return list(csv.DictReader(stream))
|
||||
|
||||
|
||||
def load_examples(manifest: Path, target_provider: str) -> list[Example]:
|
||||
"""Load ordinary final-label examples for one provider target.
|
||||
|
||||
Candidate, sham, and source-control rows are deliberately excluded. A
|
||||
remover-generated negative must not certify the detector that created it,
|
||||
and repeated controls must not receive extra sample weight.
|
||||
"""
|
||||
root = manifest.parent
|
||||
examples: list[Example] = []
|
||||
for row in _read_manifest(manifest):
|
||||
if row.get("target_provider") != target_provider:
|
||||
continue
|
||||
if row.get("split") not in FINAL_SPLITS or row.get("oracle_role") != "ordinary":
|
||||
continue
|
||||
outcome = row.get("synthid_outcome")
|
||||
if outcome not in {"detected", "not_detected"}:
|
||||
continue
|
||||
source_provider = row.get("source_provider", "")
|
||||
if outcome == "detected" and source_provider != target_provider:
|
||||
raise ValueError(
|
||||
f"positive group {row.get('group_id')!r} targets {target_provider!r} "
|
||||
f"but declares source_provider {source_provider!r}"
|
||||
)
|
||||
negative_cohort: str | None = None
|
||||
if outcome == "not_detected":
|
||||
if source_provider == target_provider:
|
||||
negative_cohort = "same_provider"
|
||||
elif source_provider in {"openai", "google", "other_ai"}:
|
||||
negative_cohort = "other_ai"
|
||||
else:
|
||||
negative_cohort = "external"
|
||||
examples.append(
|
||||
Example(
|
||||
artifact_path=_safe_artifact_path(root, row.get("artifact_path", "")),
|
||||
artifact_sha256=row.get("artifact_sha256", ""),
|
||||
group_id=row.get("group_id", ""),
|
||||
split=row.get("split", ""),
|
||||
label=1 if outcome == "detected" else 0,
|
||||
negative_cohort=negative_cohort,
|
||||
)
|
||||
)
|
||||
if not examples:
|
||||
raise ValueError(f"manifest has no eligible ordinary rows for target_provider={target_provider!r}")
|
||||
return examples
|
||||
|
||||
|
||||
def _decoded_rgb(path: Path) -> Image.Image:
|
||||
"""Return rendered RGB pixels with EXIF orientation applied once."""
|
||||
with Image.open(path) as image:
|
||||
return ImageOps.exif_transpose(image).convert("RGB")
|
||||
|
||||
|
||||
def _thumbnail_features(image: Image.Image) -> np.ndarray:
|
||||
"""Return a fixed-size RGB content fingerprint with no source geometry."""
|
||||
thumbnail = image.resize((THUMBNAIL_SIZE, THUMBNAIL_SIZE), Image.Resampling.LANCZOS)
|
||||
return np.asarray(thumbnail, dtype=np.float64).reshape(-1) / 255.0
|
||||
|
||||
|
||||
def extract_feature_families(example: Example) -> dict[str, np.ndarray]:
|
||||
"""Extract every confounded feature family with one image decode."""
|
||||
with _decoded_rgb(example.artifact_path) as image:
|
||||
width, height = image.size
|
||||
thumbnail = _thumbnail_features(image)
|
||||
file_size = example.artifact_path.stat().st_size
|
||||
pixels = width * height
|
||||
image_format = example.artifact_path.suffix.lower().lstrip(".")
|
||||
if image_format == "jpg":
|
||||
image_format = "jpeg"
|
||||
container = np.asarray(
|
||||
[
|
||||
np.log1p(width),
|
||||
np.log1p(height),
|
||||
np.log1p(pixels),
|
||||
np.log1p(file_size),
|
||||
width / height,
|
||||
file_size / pixels,
|
||||
*(1.0 if image_format == name else 0.0 for name in FORMAT_NAMES),
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
return {
|
||||
"container": container,
|
||||
"thumbnail": np.concatenate((container, thumbnail)),
|
||||
"canonical": thumbnail,
|
||||
}
|
||||
|
||||
|
||||
def extract_features(example: Example, family: str) -> np.ndarray:
|
||||
"""Extract one deliberately confounded feature family."""
|
||||
if family not in FEATURE_FAMILIES:
|
||||
raise ValueError(f"unsupported feature family {family!r}")
|
||||
return extract_feature_families(example)[family]
|
||||
|
||||
|
||||
def feature_matrix(examples: list[Example], family: str) -> np.ndarray:
|
||||
"""Extract a dense matrix in manifest order."""
|
||||
return np.stack([extract_features(example, family) for example in examples])
|
||||
|
||||
|
||||
def feature_matrices(examples: list[Example]) -> dict[str, np.ndarray]:
|
||||
"""Extract all dense feature matrices while decoding each artifact once."""
|
||||
rows = [extract_feature_families(example) for example in examples]
|
||||
return {family: np.stack([row[family] for row in rows]) for family in FEATURE_FAMILIES}
|
||||
|
||||
|
||||
def _balanced_sample_weights(labels: np.ndarray) -> np.ndarray:
|
||||
"""Give each class equal total weight regardless of corpus imbalance."""
|
||||
positives = int(np.sum(labels == 1))
|
||||
negatives = int(np.sum(labels == 0))
|
||||
if positives == 0 or negatives == 0:
|
||||
raise ValueError("training requires at least one positive and one negative")
|
||||
return np.where(labels == 1, 0.5 / positives, 0.5 / negatives)
|
||||
|
||||
|
||||
def _sigmoid(values: np.ndarray) -> np.ndarray:
|
||||
"""Evaluate a numerically stable logistic sigmoid."""
|
||||
result = np.empty_like(values, dtype=np.float64)
|
||||
positive = values >= 0
|
||||
result[positive] = 1.0 / (1.0 + np.exp(-values[positive]))
|
||||
exponent = np.exp(values[~positive])
|
||||
result[~positive] = exponent / (1.0 + exponent)
|
||||
return result
|
||||
|
||||
|
||||
def fit_logistic(
|
||||
features: np.ndarray,
|
||||
labels: np.ndarray,
|
||||
*,
|
||||
iterations: int = LOGISTIC_ITERATIONS,
|
||||
learning_rate: float = LOGISTIC_LEARNING_RATE,
|
||||
l2: float = LOGISTIC_L2,
|
||||
) -> LogisticModel:
|
||||
"""Fit deterministic class-balanced L2 logistic regression."""
|
||||
if features.ndim != 2 or labels.shape != (features.shape[0],):
|
||||
raise ValueError("feature and label shapes are inconsistent")
|
||||
if iterations < 1 or learning_rate <= 0.0 or l2 < 0.0:
|
||||
raise ValueError("iterations and learning_rate must be positive; l2 must be nonnegative")
|
||||
mean = np.mean(features, axis=0)
|
||||
scale = np.std(features, axis=0)
|
||||
scale = np.where(scale > 1e-12, scale, 1.0)
|
||||
standardized = (features - mean) / scale
|
||||
sample_weights = _balanced_sample_weights(labels)
|
||||
weights = np.zeros(features.shape[1], dtype=np.float64)
|
||||
bias = 0.0
|
||||
for _ in range(iterations):
|
||||
probabilities = _sigmoid(standardized @ weights + bias)
|
||||
error = (probabilities - labels) * sample_weights
|
||||
gradient = standardized.T @ error + l2 * weights
|
||||
weights -= learning_rate * gradient
|
||||
bias -= learning_rate * float(np.sum(error))
|
||||
return LogisticModel(mean=mean, scale=scale, weights=weights, bias=bias)
|
||||
|
||||
|
||||
def predict_scores(model: LogisticModel, features: np.ndarray) -> np.ndarray:
|
||||
"""Return positive-class probabilities for FEATURES."""
|
||||
standardized = (features - model.mean) / model.scale
|
||||
return _sigmoid(standardized @ model.weights + model.bias)
|
||||
|
||||
|
||||
def select_threshold(labels: np.ndarray, scores: np.ndarray, *, max_fpr: float) -> float:
|
||||
"""Choose the validation threshold with maximum TPR under MAX_FPR."""
|
||||
if labels.shape != scores.shape or labels.ndim != 1:
|
||||
raise ValueError("validation labels and scores must be one-dimensional and aligned")
|
||||
if not 0.0 <= max_fpr <= 1.0:
|
||||
raise ValueError("max_fpr must be between zero and one")
|
||||
if not np.any(labels == 1) or not np.any(labels == 0):
|
||||
raise ValueError("threshold selection requires positive and negative validation examples")
|
||||
candidates = [float(np.nextafter(np.max(scores), np.inf)), *sorted(set(map(float, scores)), reverse=True)]
|
||||
best: tuple[float, float, float] | None = None
|
||||
for threshold in candidates:
|
||||
predicted = scores >= threshold
|
||||
tpr = float(np.mean(predicted[labels == 1]))
|
||||
fpr = float(np.mean(predicted[labels == 0]))
|
||||
if fpr > max_fpr:
|
||||
continue
|
||||
candidate = (tpr, -fpr, threshold)
|
||||
if best is None or candidate > best:
|
||||
best = candidate
|
||||
if best is None:
|
||||
raise RuntimeError("threshold search found no feasible operating point")
|
||||
return best[2]
|
||||
|
||||
|
||||
def _auc(labels: np.ndarray, scores: np.ndarray) -> float | None:
|
||||
"""Return tie-aware ROC AUC, or None when one class is absent."""
|
||||
positive_count = int(np.sum(labels == 1))
|
||||
negative_count = int(np.sum(labels == 0))
|
||||
if positive_count == 0 or negative_count == 0:
|
||||
return None
|
||||
order = np.argsort(scores, kind="stable")
|
||||
sorted_scores = scores[order]
|
||||
ranks = np.empty(len(scores), dtype=np.float64)
|
||||
start = 0
|
||||
while start < len(scores):
|
||||
end = start + 1
|
||||
while end < len(scores) and sorted_scores[end] == sorted_scores[start]:
|
||||
end += 1
|
||||
ranks[order[start:end]] = (start + 1 + end) / 2.0
|
||||
start = end
|
||||
positive_rank_sum = float(np.sum(ranks[labels == 1]))
|
||||
return (positive_rank_sum - positive_count * (positive_count + 1) / 2.0) / (positive_count * negative_count)
|
||||
|
||||
|
||||
def calculate_metrics(labels: np.ndarray, scores: np.ndarray, threshold: float) -> Metrics:
|
||||
"""Calculate confusion counts, rates, and AUC at THRESHOLD."""
|
||||
predicted = scores >= threshold
|
||||
positives = labels == 1
|
||||
negatives = ~positives
|
||||
true_positives = int(np.sum(predicted & positives))
|
||||
false_positives = int(np.sum(predicted & negatives))
|
||||
true_negatives = int(np.sum(~predicted & negatives))
|
||||
false_negatives = int(np.sum(~predicted & positives))
|
||||
positive_count = int(np.sum(positives))
|
||||
negative_count = int(np.sum(negatives))
|
||||
return Metrics(
|
||||
count=len(labels),
|
||||
positives=positive_count,
|
||||
negatives=negative_count,
|
||||
true_positives=true_positives,
|
||||
false_positives=false_positives,
|
||||
true_negatives=true_negatives,
|
||||
false_negatives=false_negatives,
|
||||
tpr=true_positives / positive_count if positive_count else None,
|
||||
fpr=false_positives / negative_count if negative_count else None,
|
||||
auc=_auc(labels, scores),
|
||||
)
|
||||
|
||||
|
||||
def _split_indices(examples: list[Example], split: str) -> np.ndarray:
|
||||
"""Return integer indices for SPLIT."""
|
||||
return np.asarray([index for index, example in enumerate(examples) if example.split == split], dtype=np.int64)
|
||||
|
||||
|
||||
def _cohort_metrics(
|
||||
examples: list[Example], scores: np.ndarray, threshold: float, split: str
|
||||
) -> dict[str, dict[str, int | float | None]]:
|
||||
"""Report negative-only false-positive rates by provenance cohort."""
|
||||
result: dict[str, dict[str, int | float | None]] = {}
|
||||
for cohort in ("same_provider", "other_ai", "external"):
|
||||
indices = np.asarray(
|
||||
[
|
||||
index
|
||||
for index, example in enumerate(examples)
|
||||
if example.split == split and example.negative_cohort == cohort
|
||||
],
|
||||
dtype=np.int64,
|
||||
)
|
||||
if not len(indices):
|
||||
result[cohort] = {"count": 0, "false_positives": 0, "fpr": None}
|
||||
continue
|
||||
cohort_scores = scores[indices]
|
||||
false_positives = int(np.sum(cohort_scores >= threshold))
|
||||
result[cohort] = {
|
||||
"count": len(indices),
|
||||
"false_positives": false_positives,
|
||||
"fpr": false_positives / len(indices),
|
||||
}
|
||||
return result
|
||||
|
||||
|
||||
def _require_split_classes(examples: list[Example], split: str) -> None:
|
||||
"""Require both labels in the train, validation, and locked test splits."""
|
||||
labels = {example.label for example in examples if example.split == split}
|
||||
if labels != {0, 1}:
|
||||
raise ValueError(f"split {split!r} must contain at least one ordinary positive and negative")
|
||||
|
||||
|
||||
def run_experiment(
|
||||
manifest: Path,
|
||||
target_provider: str,
|
||||
*,
|
||||
max_fpr: float = 0.001,
|
||||
verify_files: bool = True,
|
||||
) -> dict[str, object]:
|
||||
"""Audit MANIFEST, train all confound baselines, and return a JSON-safe report."""
|
||||
errors = audit_manifest(manifest, verify_files=verify_files)
|
||||
if errors:
|
||||
preview = "; ".join(errors[:5])
|
||||
raise ValueError(f"manifest audit failed with {len(errors)} error(s): {preview}")
|
||||
examples = load_examples(manifest, target_provider)
|
||||
for split in ("train", "validation", "test"):
|
||||
_require_split_classes(examples, split)
|
||||
|
||||
labels = np.asarray([example.label for example in examples], dtype=np.int64)
|
||||
split_counts = Counter(example.split for example in examples)
|
||||
negative_counts = Counter(example.negative_cohort for example in examples if example.negative_cohort is not None)
|
||||
train_indices = _split_indices(examples, "train")
|
||||
validation_indices = _split_indices(examples, "validation")
|
||||
report_families: dict[str, object] = {}
|
||||
matrices = feature_matrices(examples)
|
||||
|
||||
for family in FEATURE_FAMILIES:
|
||||
features = matrices[family]
|
||||
model = fit_logistic(features[train_indices], labels[train_indices])
|
||||
scores = predict_scores(model, features)
|
||||
threshold = select_threshold(labels[validation_indices], scores[validation_indices], max_fpr=max_fpr)
|
||||
split_metrics: dict[str, object] = {}
|
||||
cohort_metrics: dict[str, object] = {}
|
||||
for split in FINAL_SPLITS:
|
||||
indices = _split_indices(examples, split)
|
||||
if not len(indices):
|
||||
split_metrics[split] = None
|
||||
cohort_metrics[split] = {
|
||||
cohort: {"count": 0, "false_positives": 0, "fpr": None}
|
||||
for cohort in ("same_provider", "other_ai", "external")
|
||||
}
|
||||
continue
|
||||
split_metrics[split] = asdict(calculate_metrics(labels[indices], scores[indices], threshold))
|
||||
cohort_metrics[split] = _cohort_metrics(examples, scores, threshold, split)
|
||||
report_families[family] = {
|
||||
"feature_count": features.shape[1],
|
||||
"threshold": threshold,
|
||||
"metrics": split_metrics,
|
||||
"negative_cohorts": cohort_metrics,
|
||||
}
|
||||
|
||||
manifest_digest = artifact_sha256(manifest)
|
||||
has_same_provider_test = any(
|
||||
example.split == "test" and example.negative_cohort == "same_provider" for example in examples
|
||||
)
|
||||
has_temporal_both_classes = {example.label for example in examples if example.split == "temporal"} == {0, 1}
|
||||
return {
|
||||
"schema_version": 1,
|
||||
"experiment": "synthid-d1-confounds",
|
||||
"target_provider": target_provider,
|
||||
"manifest_sha256": manifest_digest,
|
||||
"max_validation_fpr": max_fpr,
|
||||
"eligible_examples": len(examples),
|
||||
"split_counts": dict(sorted(split_counts.items())),
|
||||
"negative_cohort_counts": dict(sorted(negative_counts.items())),
|
||||
"evidence_ready": has_same_provider_test and has_temporal_both_classes,
|
||||
"evidence_missing": [
|
||||
reason
|
||||
for missing, reason in (
|
||||
(not has_same_provider_test, "locked test has no same-provider hard negative"),
|
||||
(not has_temporal_both_classes, "temporal split does not contain both labels"),
|
||||
)
|
||||
if missing
|
||||
],
|
||||
"feature_schema": {
|
||||
"families": list(FEATURE_FAMILIES),
|
||||
"thumbnail_size": THUMBNAIL_SIZE,
|
||||
"format_names": list(FORMAT_NAMES),
|
||||
},
|
||||
"logistic_regression": {
|
||||
"iterations": LOGISTIC_ITERATIONS,
|
||||
"learning_rate": LOGISTIC_LEARNING_RATE,
|
||||
"l2": LOGISTIC_L2,
|
||||
"class_balancing": "equal total weight per class",
|
||||
},
|
||||
"families": report_families,
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("manifest", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--target-provider", required=True, type=click.Choice(["openai", "google"]))
|
||||
@click.option("--report-out", required=True, type=click.Path(dir_okay=False, path_type=Path))
|
||||
@click.option("--max-fpr", type=click.FloatRange(0.0, 1.0), default=0.001, show_default=True)
|
||||
@click.option("--verify-files/--no-verify-files", default=True, show_default=True)
|
||||
def main(manifest: Path, target_provider: str, report_out: Path, max_fpr: float, verify_files: bool) -> None:
|
||||
"""Run D1 confound baselines from a provider-specific research MANIFEST."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
try:
|
||||
report = run_experiment(manifest, target_provider, max_fpr=max_fpr, verify_files=verify_files)
|
||||
except (OSError, ValueError) as exc:
|
||||
raise click.ClickException(str(exc)) from exc
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(json.dumps(report, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
log.info("Wrote D1 confound report: %s", report_out)
|
||||
if report["evidence_ready"]:
|
||||
log.info("D1 report contains same-provider locked negatives and a two-class temporal holdout")
|
||||
else:
|
||||
log.warning("D1 report lacks required evidence: %s", "; ".join(report["evidence_missing"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,259 @@
|
||||
"""Discover a polarity-invariant spectral carrier from low-texture groups.
|
||||
|
||||
This research harness is intentionally separate from the shipping detector. It
|
||||
uses repeated, independently generated low-texture images to find Fourier bins
|
||||
whose phase is stable within a content group and whose phase axis is stable
|
||||
across groups. Treat its output as a carrier hypothesis until it passes the
|
||||
provider-oracle and hard-negative gates in the SynthID research plan.
|
||||
|
||||
Usage:
|
||||
uv run --extra pixels python scripts/synthid_consensus_probe.py discover \
|
||||
refs/black refs/white refs/red --limit 5 \
|
||||
--model-out .local-eval/synthid/consensus.npz
|
||||
|
||||
uv run --extra pixels python scripts/synthid_consensus_probe.py score \
|
||||
.local-eval/synthid/consensus.npz images/*.png
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from PIL import Image, ImageFilter
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
IMAGE_SUFFIXES = {".jpeg", ".jpg", ".png", ".webp"}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConsensusScore:
|
||||
"""One image's alignment with a frozen carrier hypothesis."""
|
||||
|
||||
path: str
|
||||
score: float
|
||||
active_weight_fraction: float
|
||||
peak_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ConsensusModel:
|
||||
"""A compact, pickle-free carrier hypothesis."""
|
||||
|
||||
size: int
|
||||
peaks: np.ndarray
|
||||
axial_phase: np.ndarray
|
||||
weights: np.ndarray
|
||||
expected_magnitude: np.ndarray
|
||||
|
||||
|
||||
def _image_paths(directory: Path, limit: int | None) -> list[Path]:
|
||||
"""Return a deterministic list of supported images in DIRECTORY."""
|
||||
paths = sorted(path for path in directory.iterdir() if path.suffix.lower() in IMAGE_SUFFIXES)
|
||||
if limit is not None:
|
||||
paths = paths[:limit]
|
||||
if not paths:
|
||||
raise ValueError(f"no supported images in {directory}")
|
||||
return paths
|
||||
|
||||
|
||||
def _high_pass_rgb(path: Path, size: int, blur_radius: float) -> np.ndarray:
|
||||
"""Decode PATH, canonicalize its geometry, and remove local image content."""
|
||||
with Image.open(path) as source:
|
||||
image = source.convert("RGB").resize((size, size), Image.Resampling.LANCZOS)
|
||||
pixels = np.asarray(image, dtype=np.float64)
|
||||
blurred = np.asarray(image.filter(ImageFilter.GaussianBlur(radius=blur_radius)), dtype=np.float64)
|
||||
return pixels - blurred
|
||||
|
||||
|
||||
def _spectrum(path: Path, size: int, blur_radius: float) -> np.ndarray:
|
||||
"""Return a centered channel-wise spectrum for PATH."""
|
||||
residual = _high_pass_rgb(path, size, blur_radius)
|
||||
return np.fft.fftshift(np.fft.fft2(residual, axes=(0, 1)), axes=(0, 1))
|
||||
|
||||
|
||||
def _group_statistics(paths: list[Path], size: int, blur_radius: float) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Return phase coherence and mean magnitude for one reference group."""
|
||||
spectra = [_spectrum(path, size, blur_radius) for path in paths]
|
||||
units = [spectrum / (np.abs(spectrum) + 1e-12) for spectrum in spectra]
|
||||
mean_unit = np.mean(units, axis=0)
|
||||
coherence = np.abs(mean_unit)
|
||||
mean_magnitude = np.mean([np.abs(spectrum) for spectrum in spectra], axis=0)
|
||||
phase = np.angle(mean_unit)
|
||||
return coherence * np.exp(1j * phase), mean_magnitude
|
||||
|
||||
|
||||
def _valid_half_plane(size: int, min_radius: float, max_radius_fraction: float) -> np.ndarray:
|
||||
"""Return the nonredundant Fourier region allowed for carrier selection."""
|
||||
center = size // 2
|
||||
yy, xx = np.ogrid[:size, :size]
|
||||
dy = yy - center
|
||||
dx = xx - center
|
||||
radius = np.sqrt(np.square(dy) + np.square(dx))
|
||||
half_plane = (dy > 0) | ((dy == 0) & (dx > 0))
|
||||
off_axis = (dy != 0) & (dx != 0)
|
||||
return half_plane & off_axis & (radius >= min_radius) & (radius <= size * max_radius_fraction)
|
||||
|
||||
|
||||
def discover_model(
|
||||
groups: list[list[Path]],
|
||||
*,
|
||||
size: int = 512,
|
||||
blur_radius: float = 2.0,
|
||||
peak_count: int = 256,
|
||||
min_radius: float = 8.0,
|
||||
max_radius_fraction: float = 0.4,
|
||||
) -> ConsensusModel:
|
||||
"""Discover a polarity-invariant carrier from independent image GROUPS."""
|
||||
if len(groups) < 2:
|
||||
raise ValueError("at least two reference groups are required")
|
||||
if any(len(group) < 2 for group in groups):
|
||||
raise ValueError("each reference group requires at least two images")
|
||||
|
||||
group_units: list[np.ndarray] = []
|
||||
group_magnitudes: list[np.ndarray] = []
|
||||
for paths in groups:
|
||||
unit, magnitude = _group_statistics(paths, size, blur_radius)
|
||||
group_units.append(unit)
|
||||
group_magnitudes.append(magnitude)
|
||||
|
||||
stacked = np.stack(group_units, axis=0)
|
||||
within_coherence = np.abs(stacked)
|
||||
group_phase = np.angle(stacked)
|
||||
axial_mean = np.mean(np.exp(2j * group_phase) * within_coherence, axis=0)
|
||||
axial_coherence = np.abs(axial_mean) / (np.mean(within_coherence, axis=0) + 1e-12)
|
||||
mean_within = np.mean(within_coherence, axis=0)
|
||||
expected_magnitude = np.mean(group_magnitudes, axis=0)
|
||||
|
||||
magnitude_scale = np.median(expected_magnitude, axis=(0, 1), keepdims=True) + 1e-12
|
||||
magnitude_score = np.log1p(expected_magnitude / magnitude_scale)
|
||||
selection_score = np.square(mean_within) * np.square(axial_coherence) * magnitude_score
|
||||
selection_score *= _valid_half_plane(size, min_radius, max_radius_fraction)[:, :, None]
|
||||
|
||||
candidate_count = int(np.count_nonzero(selection_score))
|
||||
if candidate_count < peak_count:
|
||||
raise ValueError(f"only {candidate_count} valid carrier candidates for {peak_count} peaks")
|
||||
flat = selection_score.ravel()
|
||||
indices = np.argpartition(flat, -peak_count)[-peak_count:]
|
||||
indices = indices[np.argsort(flat[indices])[::-1]]
|
||||
rows, columns, channels = np.unravel_index(indices, selection_score.shape)
|
||||
peaks = np.column_stack((rows - size // 2, columns - size // 2, channels)).astype(np.int32)
|
||||
|
||||
axial_phase = 0.5 * np.angle(axial_mean[rows, columns, channels])
|
||||
weights = selection_score[rows, columns, channels]
|
||||
weights /= np.sum(weights)
|
||||
magnitudes = expected_magnitude[rows, columns, channels]
|
||||
return ConsensusModel(
|
||||
size=size,
|
||||
peaks=peaks,
|
||||
axial_phase=axial_phase.astype(np.float64),
|
||||
weights=weights.astype(np.float64),
|
||||
expected_magnitude=magnitudes.astype(np.float64),
|
||||
)
|
||||
|
||||
|
||||
def score_image(path: Path, model: ConsensusModel, *, blur_radius: float = 2.0) -> ConsensusScore:
|
||||
"""Score PATH against a frozen polarity-invariant carrier model."""
|
||||
spectrum = _spectrum(path, model.size, blur_radius)
|
||||
center = model.size // 2
|
||||
rows = center + model.peaks[:, 0]
|
||||
columns = center + model.peaks[:, 1]
|
||||
channels = model.peaks[:, 2]
|
||||
values = spectrum[rows, columns, channels]
|
||||
phase_alignment = np.cos(2.0 * (np.angle(values) - model.axial_phase))
|
||||
magnitude_gate = np.minimum(np.abs(values) / (model.expected_magnitude + 1e-12), 1.0)
|
||||
active_weights = model.weights * magnitude_gate
|
||||
active_weight = float(np.sum(active_weights))
|
||||
score = 0.0 if active_weight == 0.0 else float(np.sum(active_weights * phase_alignment) / active_weight)
|
||||
return ConsensusScore(
|
||||
path=str(path),
|
||||
score=score,
|
||||
active_weight_fraction=active_weight,
|
||||
peak_count=len(model.peaks),
|
||||
)
|
||||
|
||||
|
||||
def save_model(path: Path, model: ConsensusModel) -> None:
|
||||
"""Save MODEL as a pickle-free NPZ artifact."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
np.savez_compressed(
|
||||
path,
|
||||
size=np.asarray(model.size, dtype=np.int32),
|
||||
peaks=model.peaks.astype(np.int32),
|
||||
axial_phase=model.axial_phase.astype(np.float32),
|
||||
weights=model.weights.astype(np.float32),
|
||||
expected_magnitude=model.expected_magnitude.astype(np.float32),
|
||||
)
|
||||
|
||||
|
||||
def load_model(path: Path) -> ConsensusModel:
|
||||
"""Load and validate a pickle-free carrier model."""
|
||||
with np.load(path, allow_pickle=False) as artifact:
|
||||
model = ConsensusModel(
|
||||
size=int(artifact["size"]),
|
||||
peaks=np.asarray(artifact["peaks"], dtype=np.int32),
|
||||
axial_phase=np.asarray(artifact["axial_phase"], dtype=np.float64),
|
||||
weights=np.asarray(artifact["weights"], dtype=np.float64),
|
||||
expected_magnitude=np.asarray(artifact["expected_magnitude"], dtype=np.float64),
|
||||
)
|
||||
count = len(model.peaks)
|
||||
if model.peaks.ndim != 2 or model.peaks.shape[1] != 3:
|
||||
raise ValueError("invalid peak shape")
|
||||
if any(array.shape != (count,) for array in (model.axial_phase, model.weights, model.expected_magnitude)):
|
||||
raise ValueError("model arrays do not match peak count")
|
||||
if model.size < 64 or np.any(np.abs(model.peaks[:, :2]) >= model.size // 2):
|
||||
raise ValueError("invalid canonical size or peak coordinates")
|
||||
if np.any(model.peaks[:, 2] < 0) or np.any(model.peaks[:, 2] > 2):
|
||||
raise ValueError("invalid channel index")
|
||||
if not np.isclose(np.sum(model.weights), 1.0, atol=1e-5):
|
||||
raise ValueError("model weights must sum to one")
|
||||
return model
|
||||
|
||||
|
||||
@click.group()
|
||||
def main() -> None:
|
||||
"""Run low-texture carrier discovery and scoring experiments."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("group_dirs", nargs=-1, required=True, type=click.Path(exists=True, file_okay=False, path_type=Path))
|
||||
@click.option("--limit", type=click.IntRange(min=2))
|
||||
@click.option("--size", type=click.IntRange(min=64), default=512, show_default=True)
|
||||
@click.option("--peak-count", type=click.IntRange(min=1), default=256, show_default=True)
|
||||
@click.option("--model-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def discover(group_dirs: tuple[Path, ...], limit: int | None, size: int, peak_count: int, model_out: Path) -> None:
|
||||
"""Discover a carrier from the images in each GROUP_DIRS directory."""
|
||||
groups = [_image_paths(directory, limit) for directory in group_dirs]
|
||||
model = discover_model(groups, size=size, peak_count=peak_count)
|
||||
save_model(model_out, model)
|
||||
log.info("Wrote consensus model: %s", model_out)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path))
|
||||
def score(model_path: Path, images: tuple[Path, ...], report_out: Path | None) -> None:
|
||||
"""Score IMAGES against MODEL_PATH."""
|
||||
model = load_model(model_path)
|
||||
payload = {
|
||||
"model": str(model_path),
|
||||
"scores": [asdict(score_image(image, model)) for image in images],
|
||||
}
|
||||
rendered = json.dumps(payload, indent=2) + "\n"
|
||||
if report_out is None:
|
||||
log.info("%s", rendered.rstrip())
|
||||
return
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(rendered, encoding="utf-8")
|
||||
log.info("Wrote score report: %s", report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,228 @@
|
||||
"""Probe complex cross-spectral coupling at preregistered carrier shifts."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING, Any
|
||||
|
||||
import click
|
||||
import cv2
|
||||
import numpy as np
|
||||
from synthid_affine_lattice_probe import (
|
||||
_canonical_pixels,
|
||||
_opponent_channels,
|
||||
_patch_origins,
|
||||
template_harmonics,
|
||||
)
|
||||
from synthid_pixel_attack import load_rgb
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from numpy.typing import NDArray
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
_OFF_CARRIER_OFFSETS = ((1, 0), (-1, 0), (0, 1), (0, -1), (1, 1), (-1, -1))
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class CyclostationaryScore:
|
||||
"""Complex carrier-versus-neighbor contrast on disjoint patch groups."""
|
||||
|
||||
selection_carrier: float
|
||||
selection_off_carrier_median: float
|
||||
selection_contrast: float
|
||||
confirmation_carrier: float
|
||||
confirmation_off_carrier_median: float
|
||||
confirmation_contrast: float
|
||||
joint_contrast: float
|
||||
harmonic_count: int
|
||||
selection_patches: int
|
||||
confirmation_patches: int
|
||||
|
||||
|
||||
def _patch_cyclic_matrices(
|
||||
pixels: NDArray[Any],
|
||||
harmonics: NDArray[Any],
|
||||
*,
|
||||
tile_size: int,
|
||||
denoise_sigma: float,
|
||||
band_min: float,
|
||||
band_max: float,
|
||||
) -> NDArray[Any]:
|
||||
"""Return normalized complex cross-channel matrices for one patch."""
|
||||
patch_size = pixels.shape[0]
|
||||
if pixels.shape[:2] != (patch_size, patch_size) or patch_size % tile_size:
|
||||
raise ValueError("cyclostationary patches must be square multiples of the tile size")
|
||||
channels = _opponent_channels(np.asarray(pixels, dtype=np.float64))
|
||||
for channel in range(channels.shape[2]):
|
||||
residual = channels[:, :, channel]
|
||||
channels[:, :, channel] = residual - cv2.GaussianBlur(
|
||||
residual,
|
||||
(0, 0),
|
||||
sigmaX=denoise_sigma,
|
||||
sigmaY=denoise_sigma,
|
||||
borderType=cv2.BORDER_REFLECT_101,
|
||||
)
|
||||
window_1d = np.hanning(patch_size)
|
||||
window = window_1d[:, None] * window_1d[None, :]
|
||||
spectrum = np.fft.fft2(channels * window[:, :, None], axes=(0, 1))
|
||||
frequency_y = np.fft.fftfreq(patch_size)[:, None]
|
||||
frequency_x = np.fft.fftfreq(patch_size)[None, :]
|
||||
radius = np.sqrt(frequency_y * frequency_y + frequency_x * frequency_x)
|
||||
base_mask = (radius >= band_min) & (radius <= band_max)
|
||||
|
||||
offset_values = ((0, 0), *_OFF_CARRIER_OFFSETS)
|
||||
matrices = np.empty((len(harmonics), len(offset_values), 3, 3), dtype=np.complex128)
|
||||
for harmonic_index, (signed_row_value, signed_column_value) in enumerate(harmonics):
|
||||
alpha_y = round(float(signed_row_value) * patch_size / tile_size)
|
||||
alpha_x = round(float(signed_column_value) * patch_size / tile_size)
|
||||
for offset_index, (offset_y, offset_x) in enumerate(offset_values):
|
||||
shift_y = alpha_y + offset_y
|
||||
shift_x = alpha_x + offset_x
|
||||
shifted = np.roll(spectrum, shift=(-shift_y, -shift_x), axis=(0, 1))
|
||||
mask = base_mask & np.roll(base_mask, shift=(-shift_y, -shift_x), axis=(0, 1))
|
||||
base_values = spectrum[mask]
|
||||
shifted_values = shifted[mask]
|
||||
normalizer = math.sqrt(float(np.sum(np.abs(base_values) ** 2)) * float(np.sum(np.abs(shifted_values) ** 2)))
|
||||
if normalizer <= 1e-12:
|
||||
matrices[harmonic_index, offset_index] = 0.0
|
||||
else:
|
||||
matrices[harmonic_index, offset_index] = shifted_values.T @ np.conj(base_values) / normalizer
|
||||
return matrices
|
||||
|
||||
|
||||
def _group_score(values: list[NDArray[Any]], harmonic_weights: NDArray[Any]) -> tuple[float, float, float]:
|
||||
if not values:
|
||||
raise ValueError("cyclostationary score needs at least one patch")
|
||||
mean_matrices = np.mean(np.stack(values), axis=0)
|
||||
coherence = np.linalg.norm(mean_matrices, axis=(2, 3))
|
||||
carrier = float(np.sum(coherence[:, 0] * harmonic_weights))
|
||||
off_scores = [
|
||||
float(np.sum(coherence[:, offset_index] * harmonic_weights)) for offset_index in range(1, coherence.shape[1])
|
||||
]
|
||||
off_median = float(np.median(off_scores))
|
||||
return carrier, off_median, carrier - off_median
|
||||
|
||||
|
||||
def score_cyclostationary(
|
||||
pixels: NDArray[Any],
|
||||
template: NDArray[Any],
|
||||
*,
|
||||
period: float,
|
||||
patch_size: int = 256,
|
||||
grid_size: int = 4,
|
||||
harmonic_count: int = 8,
|
||||
denoise_sigma: float = 1.0,
|
||||
band_min: float = 0.05,
|
||||
band_max: float = 0.35,
|
||||
) -> CyclostationaryScore:
|
||||
"""Measure split-confirmed complex spectral coupling at one period."""
|
||||
if pixels.ndim != 3 or pixels.shape[2] != 3:
|
||||
raise ValueError("pixels must have shape (height, width, 3)")
|
||||
if not math.isfinite(period) or period <= 0.0:
|
||||
raise ValueError("period must be finite and positive")
|
||||
if not (0.0 < band_min < band_max < 0.5):
|
||||
raise ValueError("frequency band must satisfy 0 < min < max < 0.5")
|
||||
if not math.isfinite(denoise_sigma) or denoise_sigma <= 0.0:
|
||||
raise ValueError("denoise sigma must be finite and positive")
|
||||
|
||||
canonical = _canonical_pixels(pixels, template, period)
|
||||
tile_size = template.shape[0]
|
||||
harmonics, channel_weights, _coefficient_units = template_harmonics(template, harmonic_count)
|
||||
harmonic_weights = np.linalg.norm(channel_weights, axis=1)
|
||||
harmonic_weights /= np.sum(harmonic_weights)
|
||||
grouped_values: dict[int, list[NDArray[Any]]] = {0: [], 1: []}
|
||||
for origin_y, origin_x, group in _patch_origins(*canonical.shape[:2], patch_size, grid_size):
|
||||
aligned_y = (origin_y // tile_size) * tile_size
|
||||
aligned_x = (origin_x // tile_size) * tile_size
|
||||
patch = canonical[aligned_y : aligned_y + patch_size, aligned_x : aligned_x + patch_size]
|
||||
grouped_values[group].append(
|
||||
_patch_cyclic_matrices(
|
||||
patch,
|
||||
harmonics,
|
||||
tile_size=tile_size,
|
||||
denoise_sigma=denoise_sigma,
|
||||
band_min=band_min,
|
||||
band_max=band_max,
|
||||
)
|
||||
)
|
||||
selection_carrier, selection_null, selection_contrast = _group_score(grouped_values[0], harmonic_weights)
|
||||
confirmation_carrier, confirmation_null, confirmation_contrast = _group_score(grouped_values[1], harmonic_weights)
|
||||
return CyclostationaryScore(
|
||||
selection_carrier=selection_carrier,
|
||||
selection_off_carrier_median=selection_null,
|
||||
selection_contrast=selection_contrast,
|
||||
confirmation_carrier=confirmation_carrier,
|
||||
confirmation_off_carrier_median=confirmation_null,
|
||||
confirmation_contrast=confirmation_contrast,
|
||||
joint_contrast=min(selection_contrast, confirmation_contrast),
|
||||
harmonic_count=len(harmonics),
|
||||
selection_patches=len(grouped_values[0]),
|
||||
confirmation_patches=len(grouped_values[1]),
|
||||
)
|
||||
|
||||
|
||||
def _load_template(path: Path) -> NDArray[Any]:
|
||||
with np.load(path, allow_pickle=False) as artifact:
|
||||
template = np.asarray(artifact["template"], dtype=np.float64)
|
||||
if template.shape != (16, 16, 3) or not np.all(np.isfinite(template)):
|
||||
raise ValueError("template artifact does not contain a finite 16x16 RGB template")
|
||||
return template
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("template_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--period", type=click.FloatRange(min=1.0), required=True)
|
||||
@click.option("--input-scale", type=click.FloatRange(min=0.01), default=1.0, show_default=True)
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def main(
|
||||
template_path: Path,
|
||||
images: tuple[Path, ...],
|
||||
period: float,
|
||||
input_scale: float,
|
||||
report_out: Path,
|
||||
) -> None:
|
||||
"""Score IMAGES for complex cross-spectral carrier coupling."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
template = _load_template(template_path)
|
||||
rows = []
|
||||
for path in images:
|
||||
try:
|
||||
pixels = load_rgb(path)
|
||||
if input_scale != 1.0:
|
||||
width = max(1, round(pixels.shape[1] * input_scale))
|
||||
height = max(1, round(pixels.shape[0] * input_scale))
|
||||
interpolation = cv2.INTER_AREA if input_scale < 1.0 else cv2.INTER_CUBIC
|
||||
pixels = cv2.resize(pixels, (width, height), interpolation=interpolation)
|
||||
score = score_cyclostationary(pixels, template, period=period)
|
||||
except ValueError as error:
|
||||
rows.append({"path": str(path), "status": "unsupported", "reason": str(error)})
|
||||
log.warning("%s: unsupported: %s", path, error)
|
||||
continue
|
||||
rows.append({"path": str(path), "status": "scored", "score": asdict(score)})
|
||||
log.info("%s: joint contrast=%.6f", path, score.joint_contrast)
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"schema_version": 1,
|
||||
"template": str(template_path),
|
||||
"period": period,
|
||||
"input_scale": input_scale,
|
||||
"records": rows,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d cyclostationary records: %s", len(rows), report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,223 @@
|
||||
"""Build pixel-only alternating-projection candidates for the ensemble detector.
|
||||
|
||||
The attack removes only the positive complex-spectrum projection onto the
|
||||
learned RGB phases and HSV saturation/value phases. It preserves geometry and
|
||||
does not invoke a generative model. Clearing the local research detector is not
|
||||
proof that a provider oracle will clear SynthID.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import click
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from synthid_ensemble_detector import EnsembleConfig, detect_image, load_config, load_models
|
||||
from synthid_pixel_attack import load_rgb, measure, norm_matched_noise
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from synthid_color_space_probe import ColorPhaseModel
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _remove_positive_projection(
|
||||
channel: np.ndarray,
|
||||
*,
|
||||
rows: np.ndarray,
|
||||
columns: np.ndarray,
|
||||
phases: np.ndarray,
|
||||
strength: float,
|
||||
) -> np.ndarray:
|
||||
"""Remove STRENGTH of each positive phase projection from CHANNEL."""
|
||||
if strength < 0.0 or strength > 1.0:
|
||||
raise ValueError("strength must be between zero and one")
|
||||
height, width = channel.shape
|
||||
spectrum = np.fft.fft2(channel.astype(np.float64))
|
||||
for row, column, phase in zip(rows, columns, phases, strict=True):
|
||||
row_index = int(row)
|
||||
column_index = int(column)
|
||||
direction = np.exp(1j * float(phase))
|
||||
value = spectrum[row_index, column_index]
|
||||
projection = max(0.0, float(np.real(value * np.conj(direction))))
|
||||
delta = strength * projection * direction
|
||||
conjugate_row = (-row_index) % height
|
||||
conjugate_column = (-column_index) % width
|
||||
spectrum[row_index, column_index] -= delta
|
||||
if (conjugate_row, conjugate_column) == (row_index, column_index):
|
||||
spectrum[row_index, column_index] = complex(spectrum[row_index, column_index].real, 0.0)
|
||||
else:
|
||||
spectrum[conjugate_row, conjugate_column] -= np.conj(delta)
|
||||
return np.fft.ifft2(spectrum).real
|
||||
|
||||
|
||||
def _project_model_channels(
|
||||
pixels: np.ndarray,
|
||||
model: ColorPhaseModel,
|
||||
*,
|
||||
included_channels: frozenset[int],
|
||||
strength: float,
|
||||
) -> np.ndarray:
|
||||
"""Apply positive-projection removal to selected MODEL channels."""
|
||||
result = pixels.astype(np.float64, copy=True)
|
||||
for channel in included_channels:
|
||||
positions = np.flatnonzero(model.channels == channel)
|
||||
if len(positions) == 0:
|
||||
continue
|
||||
result[:, :, channel] = _remove_positive_projection(
|
||||
result[:, :, channel],
|
||||
rows=model.rows[positions],
|
||||
columns=model.columns[positions],
|
||||
phases=model.phases[positions],
|
||||
strength=strength,
|
||||
)
|
||||
return result
|
||||
|
||||
|
||||
def alternating_projection(
|
||||
pixels: np.ndarray,
|
||||
rgb_model: ColorPhaseModel,
|
||||
hsv_model: ColorPhaseModel,
|
||||
*,
|
||||
strength: float,
|
||||
iterations: int,
|
||||
) -> np.ndarray:
|
||||
"""Alternate RGB and HSV S/V phase projections without regeneration."""
|
||||
if iterations < 1:
|
||||
raise ValueError("iterations must be positive")
|
||||
expected_shape = (rgb_model.height, rgb_model.width, 3)
|
||||
if pixels.shape != expected_shape or pixels.shape != (hsv_model.height, hsv_model.width, 3):
|
||||
raise ValueError("pixel and model geometries do not match")
|
||||
result = pixels.astype(np.float64)
|
||||
for _ in range(iterations):
|
||||
result = _project_model_channels(
|
||||
result,
|
||||
rgb_model,
|
||||
included_channels=frozenset({0, 1, 2}),
|
||||
strength=strength,
|
||||
)
|
||||
rgb_unit = np.clip(result / 255.0, 0.0, 1.0).astype(np.float32)
|
||||
hsv = cv2.cvtColor(rgb_unit, cv2.COLOR_RGB2HSV).astype(np.float64)
|
||||
hsv = _project_model_channels(
|
||||
hsv,
|
||||
hsv_model,
|
||||
included_channels=frozenset({1, 2}),
|
||||
strength=strength,
|
||||
)
|
||||
hsv[:, :, 0] = np.mod(hsv[:, :, 0], 360.0)
|
||||
hsv[:, :, 1:] = np.clip(hsv[:, :, 1:], 0.0, 1.0)
|
||||
result = cv2.cvtColor(hsv.astype(np.float32), cv2.COLOR_HSV2RGB).astype(np.float64) * 255.0
|
||||
return np.clip(np.rint(result), 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def parse_positive_floats(value: str) -> tuple[float, ...]:
|
||||
"""Parse strictly increasing strengths in the interval (0, 1]."""
|
||||
try:
|
||||
values = tuple(float(item.strip()) for item in value.split(","))
|
||||
except ValueError as error:
|
||||
raise click.BadParameter("strengths must be comma-separated numbers") from error
|
||||
if not values or any(not np.isfinite(item) or item <= 0.0 or item > 1.0 for item in values):
|
||||
raise click.BadParameter("strengths must be finite and in the interval (0, 1]")
|
||||
if tuple(sorted(set(values))) != values:
|
||||
raise click.BadParameter("strengths must be unique and strictly increasing")
|
||||
return values
|
||||
|
||||
|
||||
def parse_positive_integers(value: str) -> tuple[int, ...]:
|
||||
"""Parse strictly increasing positive iteration counts."""
|
||||
try:
|
||||
values = tuple(int(item.strip()) for item in value.split(","))
|
||||
except ValueError as error:
|
||||
raise click.BadParameter("iterations must be comma-separated integers") from error
|
||||
if not values or any(item < 1 for item in values):
|
||||
raise click.BadParameter("iterations must be positive")
|
||||
if tuple(sorted(set(values))) != values:
|
||||
raise click.BadParameter("iterations must be unique and strictly increasing")
|
||||
return values
|
||||
|
||||
|
||||
def _load_source(path: Path, config: EnsembleConfig) -> np.ndarray:
|
||||
"""Load an exact-geometry RGB source."""
|
||||
pixels = load_rgb(path)
|
||||
if pixels.shape != (config.height, config.width, 3):
|
||||
raise ValueError("source geometry does not match detector config")
|
||||
return pixels
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("config_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("output_dir", type=click.Path(file_okay=False, path_type=Path))
|
||||
@click.option("--strengths", default="0.25,0.5,0.75,1", show_default=True)
|
||||
@click.option("--iterations", default="1,2,4", show_default=True)
|
||||
def main(config_path: Path, source: Path, output_dir: Path, strengths: str, iterations: str) -> None:
|
||||
"""Write a frozen pixel-only alternating-projection batch for SOURCE."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
config = load_config(config_path)
|
||||
rgb_model, hsv_model = load_models(config)
|
||||
reference = _load_source(source, config)
|
||||
strength_values = parse_positive_floats(strengths)
|
||||
iteration_values = parse_positive_integers(iterations)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
variants: list[dict[str, object]] = []
|
||||
strongest = reference
|
||||
for iteration_count in iteration_values:
|
||||
for strength in strength_values:
|
||||
pixels = alternating_projection(
|
||||
reference,
|
||||
rgb_model,
|
||||
hsv_model,
|
||||
strength=strength,
|
||||
iterations=iteration_count,
|
||||
)
|
||||
strength_name = f"{strength:g}".replace(".", "p")
|
||||
name = f"project-s{strength_name}-i{iteration_count}"
|
||||
path = output_dir / f"{name}.png"
|
||||
Image.fromarray(pixels, mode="RGB").save(path)
|
||||
variants.append(
|
||||
{
|
||||
**asdict(measure(reference, pixels, name=name, path=path)),
|
||||
**asdict(detect_image(path, config, rgb_model, hsv_model)),
|
||||
"strength": strength,
|
||||
"iterations": iteration_count,
|
||||
}
|
||||
)
|
||||
strongest = pixels
|
||||
|
||||
sham = norm_matched_noise(reference, strongest, seed=20260823)
|
||||
sham_path = output_dir / "sham-strongest-rms.png"
|
||||
Image.fromarray(sham, mode="RGB").save(sham_path)
|
||||
variants.append(
|
||||
{
|
||||
**asdict(measure(reference, sham, name="sham-strongest-rms", path=sham_path)),
|
||||
**asdict(detect_image(sham_path, config, rgb_model, hsv_model)),
|
||||
"strength": None,
|
||||
"iterations": None,
|
||||
}
|
||||
)
|
||||
|
||||
report_path = output_dir / "report.json"
|
||||
report_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"source": str(source),
|
||||
"config": str(config_path),
|
||||
"variants": variants,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d alternating-projection candidates: %s", len(variants), report_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,188 @@
|
||||
"""Run a positive-only exact-geometry SynthID research detector.
|
||||
|
||||
The detector requires independently frozen RGB and HSV phase models. It emits
|
||||
``positive`` only when both branches and both support gates pass; every other
|
||||
case is ``abstain``. It never claims that SynthID is absent.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from PIL import Image
|
||||
from synthid_color_space_probe import ColorPhaseModel, ColorPhaseScore, load_model, score_image
|
||||
from synthid_research_manifest import artifact_sha256
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EnsembleConfig:
|
||||
"""Frozen paths, hashes, geometry, and positive thresholds."""
|
||||
|
||||
width: int
|
||||
height: int
|
||||
rgb_model_path: Path
|
||||
rgb_model_sha256: str
|
||||
rgb_evidence_threshold: float
|
||||
rgb_active_threshold: float
|
||||
hsv_model_path: Path
|
||||
hsv_model_sha256: str
|
||||
hsv_sv_evidence_threshold: float
|
||||
hsv_active_threshold: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class EnsembleVerdict:
|
||||
"""Positive-only verdict with the evidence needed to audit it."""
|
||||
|
||||
path: str
|
||||
verdict: str
|
||||
reason: str
|
||||
rgb_evidence: float | None
|
||||
rgb_active_support: float | None
|
||||
hsv_sv_evidence: float | None
|
||||
hsv_active_support: float | None
|
||||
|
||||
|
||||
def load_config(path: Path) -> EnsembleConfig:
|
||||
"""Load a frozen epoch manifest and verify both model artifacts."""
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
if payload.get("verdict_scope") != "positive-only exact-geometry research detector":
|
||||
raise ValueError("config is not a positive-only exact-geometry detector")
|
||||
rgb = payload["rgb_model"]
|
||||
hsv = payload["hsv_model"]
|
||||
geometry = payload["geometry"]
|
||||
config = EnsembleConfig(
|
||||
width=int(geometry["width"]),
|
||||
height=int(geometry["height"]),
|
||||
rgb_model_path=Path(rgb["path"]),
|
||||
rgb_model_sha256=str(rgb["sha256"]),
|
||||
rgb_evidence_threshold=float(rgb["evidence_threshold"]),
|
||||
rgb_active_threshold=float(rgb["active_support_threshold"]),
|
||||
hsv_model_path=Path(hsv["path"]),
|
||||
hsv_model_sha256=str(hsv["sha256"]),
|
||||
hsv_sv_evidence_threshold=float(hsv["sv_evidence_threshold"]),
|
||||
hsv_active_threshold=float(hsv["active_support_threshold"]),
|
||||
)
|
||||
if config.width < 64 or config.height < 64:
|
||||
raise ValueError("invalid detector geometry")
|
||||
for model_path, expected_hash in (
|
||||
(config.rgb_model_path, config.rgb_model_sha256),
|
||||
(config.hsv_model_path, config.hsv_model_sha256),
|
||||
):
|
||||
if not model_path.is_file():
|
||||
raise ValueError(f"model artifact does not exist: {model_path}")
|
||||
if artifact_sha256(model_path) != expected_hash:
|
||||
raise ValueError(f"model artifact hash mismatch: {model_path}")
|
||||
return config
|
||||
|
||||
|
||||
def load_models(config: EnsembleConfig) -> tuple[ColorPhaseModel, ColorPhaseModel]:
|
||||
"""Load and cross-check the RGB and HSV models in CONFIG."""
|
||||
rgb_model = load_model(config.rgb_model_path)
|
||||
hsv_model = load_model(config.hsv_model_path)
|
||||
if rgb_model.color_space != "rgb" or hsv_model.color_space != "hsv":
|
||||
raise ValueError("detector requires one RGB model and one HSV model")
|
||||
expected_geometry = (config.height, config.width)
|
||||
if (rgb_model.height, rgb_model.width) != expected_geometry:
|
||||
raise ValueError("RGB model geometry does not match config")
|
||||
if (hsv_model.height, hsv_model.width) != expected_geometry:
|
||||
raise ValueError("HSV model geometry does not match config")
|
||||
return rgb_model, hsv_model
|
||||
|
||||
|
||||
def classify_scores(
|
||||
path: Path,
|
||||
rgb_score: ColorPhaseScore,
|
||||
hsv_score: ColorPhaseScore,
|
||||
config: EnsembleConfig,
|
||||
) -> EnsembleVerdict:
|
||||
"""Apply CONFIG's positive-only rule to precomputed branch scores."""
|
||||
hsv_sv_evidence = float(sum(hsv_score.channel_evidence[1:]))
|
||||
rgb_support = rgb_score.active_weight_fraction >= config.rgb_active_threshold
|
||||
hsv_support = hsv_score.active_weight_fraction >= config.hsv_active_threshold
|
||||
rgb_pass = rgb_score.evidence_score >= config.rgb_evidence_threshold
|
||||
hsv_pass = hsv_sv_evidence >= config.hsv_sv_evidence_threshold
|
||||
if rgb_support and hsv_support and rgb_pass and hsv_pass:
|
||||
verdict = "positive"
|
||||
reason = "ensemble_pass"
|
||||
elif not rgb_support or not hsv_support:
|
||||
verdict = "abstain"
|
||||
reason = "insufficient_support"
|
||||
elif rgb_pass != hsv_pass:
|
||||
verdict = "abstain"
|
||||
reason = "branch_disagreement"
|
||||
else:
|
||||
verdict = "abstain"
|
||||
reason = "below_positive_threshold"
|
||||
return EnsembleVerdict(
|
||||
path=str(path),
|
||||
verdict=verdict,
|
||||
reason=reason,
|
||||
rgb_evidence=rgb_score.evidence_score,
|
||||
rgb_active_support=rgb_score.active_weight_fraction,
|
||||
hsv_sv_evidence=hsv_sv_evidence,
|
||||
hsv_active_support=hsv_score.active_weight_fraction,
|
||||
)
|
||||
|
||||
|
||||
def detect_image(
|
||||
path: Path,
|
||||
config: EnsembleConfig,
|
||||
rgb_model: ColorPhaseModel,
|
||||
hsv_model: ColorPhaseModel,
|
||||
) -> EnsembleVerdict:
|
||||
"""Evaluate PATH or abstain when its geometry is unsupported."""
|
||||
with Image.open(path) as image:
|
||||
if image.size != (config.width, config.height):
|
||||
return EnsembleVerdict(
|
||||
path=str(path),
|
||||
verdict="abstain",
|
||||
reason="unsupported_geometry",
|
||||
rgb_evidence=None,
|
||||
rgb_active_support=None,
|
||||
hsv_sv_evidence=None,
|
||||
hsv_active_support=None,
|
||||
)
|
||||
return classify_scores(
|
||||
path,
|
||||
score_image(path, rgb_model),
|
||||
score_image(path, hsv_model),
|
||||
config,
|
||||
)
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("config_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def main(config_path: Path, images: tuple[Path, ...], report_out: Path) -> None:
|
||||
"""Score IMAGES with the frozen positive-only detector CONFIG_PATH."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
config = load_config(config_path)
|
||||
rgb_model, hsv_model = load_models(config)
|
||||
verdicts = [detect_image(image, config, rgb_model, hsv_model) for image in images]
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"config": str(config_path),
|
||||
"positive_count": sum(verdict.verdict == "positive" for verdict in verdicts),
|
||||
"abstain_count": sum(verdict.verdict == "abstain" for verdict in verdicts),
|
||||
"verdicts": [asdict(verdict) for verdict in verdicts],
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d positive-only detector verdicts: %s", len(verdicts), report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,243 @@
|
||||
"""Build non-generative spatial-fragmentation SynthID attack candidates.
|
||||
|
||||
The candidates combine deterministic smooth local warps with mild global
|
||||
resampling, color changes, and codec round-trips. These are pixel transforms,
|
||||
not semantic reconstruction or generative inpainting. Provider-oracle results
|
||||
must be evaluated in a frozen batch with a source-positive control.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import cv2
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from synthid_pixel_attack import (
|
||||
jpeg_round_trip,
|
||||
load_rgb,
|
||||
measure,
|
||||
norm_matched_noise,
|
||||
resize_squeeze,
|
||||
smooth_warp,
|
||||
)
|
||||
from synthid_v3_codebook_probe import load_v3_model, score_image
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def bounded_smooth_warp(
|
||||
pixels: np.ndarray,
|
||||
*,
|
||||
max_displacement: float,
|
||||
sigma: float,
|
||||
seed: int,
|
||||
) -> np.ndarray:
|
||||
"""Apply a smooth warp whose per-axis displacement is absolutely bounded."""
|
||||
if max_displacement < 0.0 or sigma <= 0.0:
|
||||
raise ValueError("max_displacement must be nonnegative and sigma positive")
|
||||
height, width = pixels.shape[:2]
|
||||
rng = np.random.default_rng(seed)
|
||||
fields: list[np.ndarray] = []
|
||||
for _ in range(2):
|
||||
noise = rng.normal(size=(height, width)).astype(np.float32)
|
||||
field = cv2.GaussianBlur(noise, (0, 0), sigmaX=sigma, sigmaY=sigma)
|
||||
maximum = float(np.max(np.abs(field)))
|
||||
fields.append(np.zeros_like(field) if maximum == 0.0 else field * (max_displacement / maximum))
|
||||
yy, xx = np.mgrid[:height, :width].astype(np.float32)
|
||||
return cv2.remap(
|
||||
pixels,
|
||||
xx + fields[0],
|
||||
yy + fields[1],
|
||||
interpolation=cv2.INTER_LANCZOS4,
|
||||
borderMode=cv2.BORDER_REFLECT_101,
|
||||
)
|
||||
|
||||
|
||||
def affine_combo(pixels: np.ndarray, *, rotation_degrees: float, zoom: float) -> np.ndarray:
|
||||
"""Apply one centered rotation-and-zoom resampling operation."""
|
||||
if zoom < 0.0:
|
||||
raise ValueError("zoom must be nonnegative")
|
||||
height, width = pixels.shape[:2]
|
||||
matrix = cv2.getRotationMatrix2D(
|
||||
center=((width - 1) / 2.0, (height - 1) / 2.0),
|
||||
angle=rotation_degrees,
|
||||
scale=1.0 + zoom,
|
||||
)
|
||||
return cv2.warpAffine(
|
||||
pixels,
|
||||
matrix,
|
||||
(width, height),
|
||||
flags=cv2.INTER_LANCZOS4,
|
||||
borderMode=cv2.BORDER_REFLECT_101,
|
||||
)
|
||||
|
||||
|
||||
def color_nudge(
|
||||
pixels: np.ndarray,
|
||||
*,
|
||||
brightness: float,
|
||||
contrast: float,
|
||||
saturation: float,
|
||||
hue_degrees: float,
|
||||
) -> np.ndarray:
|
||||
"""Apply bounded global RGB contrast and HSV saturation/hue changes."""
|
||||
rgb = pixels.astype(np.float32) / 255.0
|
||||
rgb = np.clip((rgb - 0.5) * (1.0 + contrast) + 0.5 + brightness, 0.0, 1.0)
|
||||
hsv = cv2.cvtColor(rgb, cv2.COLOR_RGB2HSV)
|
||||
hsv[:, :, 0] = np.mod(hsv[:, :, 0] + hue_degrees, 360.0)
|
||||
hsv[:, :, 1] = np.clip(hsv[:, :, 1] * (1.0 + saturation), 0.0, 1.0)
|
||||
result = cv2.cvtColor(hsv, cv2.COLOR_HSV2RGB)
|
||||
return np.clip(np.rint(result * 255.0), 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def jpeg_chain(pixels: np.ndarray, qualities: tuple[int, ...]) -> np.ndarray:
|
||||
"""Apply sequential JPEG round-trips at QUALITIES."""
|
||||
result = pixels
|
||||
for quality in qualities:
|
||||
result = jpeg_round_trip(result, quality)
|
||||
return result
|
||||
|
||||
|
||||
def build_candidates(source: np.ndarray) -> dict[str, np.ndarray]:
|
||||
"""Build the frozen spatial-fragmentation ladder for SOURCE."""
|
||||
candidates: dict[str, np.ndarray] = {
|
||||
"control": source.copy(),
|
||||
"elastic-075": smooth_warp(source, amplitude=0.75, sigma=56.0, seed=20260812),
|
||||
"elastic-125": smooth_warp(source, amplitude=1.25, sigma=52.0, seed=20260813),
|
||||
"bounded-100": bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=1.0,
|
||||
sigma=56.0,
|
||||
seed=20260817,
|
||||
),
|
||||
"bounded-180": bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=1.8,
|
||||
sigma=56.0,
|
||||
seed=20260818,
|
||||
),
|
||||
"bounded-280": bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=2.8,
|
||||
sigma=44.0,
|
||||
seed=20260819,
|
||||
),
|
||||
}
|
||||
|
||||
balanced = smooth_warp(source, amplitude=0.75, sigma=56.0, seed=20260814)
|
||||
balanced = affine_combo(balanced, rotation_degrees=0.2, zoom=0.004)
|
||||
balanced = resize_squeeze(balanced, 0.94)
|
||||
balanced = color_nudge(
|
||||
balanced,
|
||||
brightness=0.004,
|
||||
contrast=0.006,
|
||||
saturation=-0.005,
|
||||
hue_degrees=0.15,
|
||||
)
|
||||
balanced = jpeg_chain(balanced, (94, 90))
|
||||
candidates["fragment-balanced"] = balanced
|
||||
|
||||
strong = smooth_warp(source, amplitude=1.5, sigma=48.0, seed=20260815)
|
||||
strong = affine_combo(strong, rotation_degrees=0.4, zoom=0.01)
|
||||
strong = resize_squeeze(strong, 0.88)
|
||||
strong = color_nudge(
|
||||
strong,
|
||||
brightness=0.008,
|
||||
contrast=0.012,
|
||||
saturation=-0.01,
|
||||
hue_degrees=0.3,
|
||||
)
|
||||
strong = jpeg_chain(strong, (92, 88))
|
||||
candidates["fragment-strong"] = strong
|
||||
candidates["sham-strong-rms"] = norm_matched_noise(source, strong, seed=20260816)
|
||||
|
||||
bounded_balanced = bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=1.8,
|
||||
sigma=56.0,
|
||||
seed=20260820,
|
||||
)
|
||||
bounded_balanced = resize_squeeze(bounded_balanced, 0.98)
|
||||
bounded_balanced = color_nudge(
|
||||
bounded_balanced,
|
||||
brightness=0.002,
|
||||
contrast=0.003,
|
||||
saturation=-0.003,
|
||||
hue_degrees=0.1,
|
||||
)
|
||||
bounded_balanced = jpeg_chain(bounded_balanced, (96,))
|
||||
candidates["bounded-fragment-balanced"] = bounded_balanced
|
||||
|
||||
bounded_strong = bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=2.8,
|
||||
sigma=44.0,
|
||||
seed=20260821,
|
||||
)
|
||||
bounded_strong = affine_combo(bounded_strong, rotation_degrees=0.2, zoom=0.004)
|
||||
bounded_strong = resize_squeeze(bounded_strong, 0.94)
|
||||
bounded_strong = color_nudge(
|
||||
bounded_strong,
|
||||
brightness=0.004,
|
||||
contrast=0.006,
|
||||
saturation=-0.005,
|
||||
hue_degrees=0.15,
|
||||
)
|
||||
bounded_strong = jpeg_chain(bounded_strong, (94, 90))
|
||||
candidates["bounded-fragment-strong"] = bounded_strong
|
||||
candidates["sham-bounded-strong-rms"] = norm_matched_noise(source, bounded_strong, seed=20260822)
|
||||
return candidates
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("codebook", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("output_dir", type=click.Path(file_okay=False, path_type=Path))
|
||||
@click.option("--height", type=click.IntRange(min=64), required=True)
|
||||
@click.option("--width", type=click.IntRange(min=64), required=True)
|
||||
@click.option("--peak-count", type=click.IntRange(min=1), default=256, show_default=True)
|
||||
def main(codebook: Path, source: Path, output_dir: Path, height: int, width: int, peak_count: int) -> None:
|
||||
"""Write a frozen spatial-fragmentation batch for SOURCE."""
|
||||
reference = load_rgb(source)
|
||||
if reference.shape != (height, width, 3):
|
||||
raise click.BadParameter("source geometry does not match --height and --width")
|
||||
model = load_v3_model(codebook, height=height, width=width, peak_count=peak_count)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
rows: list[dict[str, object]] = []
|
||||
for name, pixels in build_candidates(reference).items():
|
||||
path = output_dir / f"{name}.png"
|
||||
Image.fromarray(pixels, mode="RGB").save(path)
|
||||
rows.append(
|
||||
{
|
||||
**asdict(measure(reference, pixels, name=name, path=path)),
|
||||
**asdict(score_image(path, model)),
|
||||
}
|
||||
)
|
||||
report_path = output_dir / "report.json"
|
||||
report_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"source": str(source),
|
||||
"codebook": str(codebook),
|
||||
"height": height,
|
||||
"width": width,
|
||||
"peak_count": peak_count,
|
||||
"variants": rows,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d frozen fragmentation candidates: %s", len(rows), report_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
main()
|
||||
@@ -0,0 +1,140 @@
|
||||
"""Build non-generative hybrid phase-projection and fragmentation candidates.
|
||||
|
||||
The matrix combines two independently measured mechanisms: sparse RGB/HSV
|
||||
phase projection and spatially varying subpixel displacement. It is intended
|
||||
for frozen provider-oracle batches with a visible-mark-removed source control.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import click
|
||||
from PIL import Image
|
||||
from synthid_ensemble_attack import alternating_projection
|
||||
from synthid_ensemble_detector import detect_image, load_config, load_models
|
||||
from synthid_fragment_attack import bounded_smooth_warp, color_nudge, jpeg_chain
|
||||
from synthid_pixel_attack import load_rgb, measure, norm_matched_noise, resize_squeeze, smooth_warp
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy as np
|
||||
from synthid_color_space_probe import ColorPhaseModel
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def build_candidates(
|
||||
source: np.ndarray,
|
||||
rgb_model: ColorPhaseModel,
|
||||
hsv_model: ColorPhaseModel,
|
||||
) -> dict[str, np.ndarray]:
|
||||
"""Return a frozen mechanism matrix derived from SOURCE."""
|
||||
projected_075 = alternating_projection(
|
||||
source,
|
||||
rgb_model,
|
||||
hsv_model,
|
||||
strength=0.75,
|
||||
iterations=1,
|
||||
)
|
||||
projected_100 = alternating_projection(
|
||||
source,
|
||||
rgb_model,
|
||||
hsv_model,
|
||||
strength=1.0,
|
||||
iterations=1,
|
||||
)
|
||||
bounded_100 = bounded_smooth_warp(
|
||||
source,
|
||||
max_displacement=1.0,
|
||||
sigma=56.0,
|
||||
seed=20260824,
|
||||
)
|
||||
projected_bounded_100 = bounded_smooth_warp(
|
||||
projected_075,
|
||||
max_displacement=1.0,
|
||||
sigma=56.0,
|
||||
seed=20260824,
|
||||
)
|
||||
|
||||
bounded_polish = resize_squeeze(projected_bounded_100, 0.98)
|
||||
bounded_polish = color_nudge(
|
||||
bounded_polish,
|
||||
brightness=0.002,
|
||||
contrast=0.003,
|
||||
saturation=-0.003,
|
||||
hue_degrees=0.1,
|
||||
)
|
||||
bounded_polish = jpeg_chain(bounded_polish, (96,))
|
||||
|
||||
elastic_combo = smooth_warp(projected_100, amplitude=0.75, sigma=56.0, seed=20260825)
|
||||
elastic_combo = resize_squeeze(elastic_combo, 0.98)
|
||||
elastic_combo = jpeg_chain(elastic_combo, (96,))
|
||||
|
||||
return {
|
||||
"projection-075": projected_075,
|
||||
"bounded-100": bounded_100,
|
||||
"projection-075-bounded-100": projected_bounded_100,
|
||||
"projection-075-bounded-polish": bounded_polish,
|
||||
"projection-100-elastic-075": elastic_combo,
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("config_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("output_dir", type=click.Path(file_okay=False, path_type=Path))
|
||||
def main(config_path: Path, source: Path, output_dir: Path) -> None:
|
||||
"""Write a frozen non-generative hybrid attack matrix for SOURCE."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
config = load_config(config_path)
|
||||
rgb_model, hsv_model = load_models(config)
|
||||
reference = load_rgb(source)
|
||||
if reference.shape != (config.height, config.width, 3):
|
||||
raise click.BadParameter("source geometry does not match detector config")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
variants: list[dict[str, object]] = []
|
||||
candidates = build_candidates(reference, rgb_model, hsv_model)
|
||||
for name, pixels in candidates.items():
|
||||
path = output_dir / f"{name}.png"
|
||||
Image.fromarray(pixels, mode="RGB").save(path)
|
||||
variants.append(
|
||||
{
|
||||
**asdict(measure(reference, pixels, name=name, path=path)),
|
||||
**asdict(detect_image(path, config, rgb_model, hsv_model)),
|
||||
}
|
||||
)
|
||||
|
||||
selected = candidates["projection-075-bounded-100"]
|
||||
sham = norm_matched_noise(reference, selected, seed=20260826)
|
||||
sham_path = output_dir / "sham-projection-bounded-rms.png"
|
||||
Image.fromarray(sham, mode="RGB").save(sham_path)
|
||||
variants.append(
|
||||
{
|
||||
**asdict(measure(reference, sham, name="sham-projection-bounded-rms", path=sham_path)),
|
||||
**asdict(detect_image(sham_path, config, rgb_model, hsv_model)),
|
||||
}
|
||||
)
|
||||
|
||||
report_path = output_dir / "report.json"
|
||||
report_path.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"source": str(source),
|
||||
"config": str(config_path),
|
||||
"variants": variants,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d frozen hybrid candidates: %s", len(variants), report_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,467 @@
|
||||
"""Build and verify an immutable confirmatory batch for a SynthID oracle.
|
||||
|
||||
The batch contains a referenced untouched source plus four lossless PNG views:
|
||||
an exact-pixel re-encode, aligned periodic-tile subtraction, a cyclically
|
||||
shifted tile, and an orthogonal random tile. Building the batch performs no
|
||||
network requests. Oracle results remain empty until a separately authorized
|
||||
submission records them.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import click
|
||||
from PIL import Image
|
||||
from synthid_periodic_tile_ablation import (
|
||||
candidate_quality,
|
||||
control_templates,
|
||||
phase_score_pixels,
|
||||
tile_score_pixels,
|
||||
)
|
||||
from synthid_periodic_tile_probe import PeriodicTileModel
|
||||
from synthid_periodic_tile_probe import load_model as load_tile_model
|
||||
from synthid_phase_carrier import PhaseCarrierModel
|
||||
from synthid_phase_carrier import load_model as load_phase_model
|
||||
from synthid_pixel_attack import load_rgb
|
||||
from synthid_research_manifest import artifact_sha256, pixel_fingerprint
|
||||
from synthid_tile_attack import subtract_tiled_template
|
||||
|
||||
if TYPE_CHECKING:
|
||||
import numpy as np
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
ROLE_ORDER = ("source", "reencode_control", "aligned", "shifted", "orthogonal_random")
|
||||
DERIVATIVE_ROLES = ROLE_ORDER[1:]
|
||||
FORMAT_VERSION = 1
|
||||
SYNTHID_RESULTS = {"detected", "not_detected", "indeterminate", "refused"}
|
||||
C2PA_RESULTS = {"present", "absent", "indeterminate", "unavailable"}
|
||||
|
||||
|
||||
def _inside(path: Path, parent: Path) -> bool:
|
||||
"""Return whether resolved PATH is inside resolved PARENT."""
|
||||
try:
|
||||
path.resolve().relative_to(parent.resolve())
|
||||
except ValueError:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _json_float(value: float) -> float | None:
|
||||
"""Return finite VALUE or None for JSON interoperability."""
|
||||
return value if math.isfinite(value) else None
|
||||
|
||||
|
||||
def _write_png(path: Path, pixels: np.ndarray) -> None:
|
||||
"""Write exact RGB PIXELS as a deterministic lossless PNG."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
if path.exists():
|
||||
raise ValueError(f"refusing to overwrite oracle artifact: {path}")
|
||||
Image.fromarray(pixels, mode="RGB").save(path, format="PNG", compress_level=9)
|
||||
|
||||
|
||||
def _score_row(
|
||||
pixels: np.ndarray,
|
||||
*,
|
||||
source: np.ndarray,
|
||||
tile_model: PeriodicTileModel,
|
||||
phase_model: PhaseCarrierModel,
|
||||
tile_threshold: float,
|
||||
phase_threshold: float,
|
||||
active_threshold: float,
|
||||
) -> dict[str, object]:
|
||||
"""Return local frozen scores and paired fidelity for PIXELS."""
|
||||
tile_score = tile_score_pixels(pixels, tile_model)
|
||||
phase_score, active_support = phase_score_pixels(pixels, phase_model)
|
||||
quality = candidate_quality(source, pixels)
|
||||
return {
|
||||
"tile_score": tile_score,
|
||||
"tile_accepted": tile_score >= tile_threshold,
|
||||
"phase_score": phase_score,
|
||||
"active_support": active_support,
|
||||
"phase_accepted": phase_score >= phase_threshold and active_support >= active_threshold,
|
||||
"residual_rms": quality["residual_rms"],
|
||||
"psnr_db": _json_float(quality["psnr_db"]),
|
||||
"ssim": quality["ssim"],
|
||||
"changed_pixel_fraction": quality["changed_pixel_fraction"],
|
||||
}
|
||||
|
||||
|
||||
def _artifact_row(
|
||||
path: Path,
|
||||
*,
|
||||
role: str,
|
||||
source_id: str,
|
||||
in_batch: bool,
|
||||
batch_root: Path,
|
||||
scores: dict[str, object],
|
||||
) -> dict[str, object]:
|
||||
"""Return one hash-frozen manifest row for PATH."""
|
||||
pixel_sha256, width, height = pixel_fingerprint(path)
|
||||
return {
|
||||
"source_id": source_id,
|
||||
"role": role,
|
||||
"path": str(path.relative_to(batch_root) if in_batch else path.resolve()),
|
||||
"in_batch": in_batch,
|
||||
"artifact_sha256": artifact_sha256(path),
|
||||
"pixel_sha256": pixel_sha256,
|
||||
"width": width,
|
||||
"height": height,
|
||||
"synthid_result": None,
|
||||
"c2pa_result": None,
|
||||
"submitted_at": None,
|
||||
**scores,
|
||||
}
|
||||
|
||||
|
||||
def build_batch(
|
||||
sources: list[Path],
|
||||
*,
|
||||
output_dir: Path,
|
||||
tile_model_path: Path,
|
||||
phase_model_path: Path,
|
||||
tile_threshold: float,
|
||||
phase_threshold: float,
|
||||
active_threshold: float,
|
||||
strength: float,
|
||||
seed: int,
|
||||
provider: str,
|
||||
repository_root: Path,
|
||||
) -> Path:
|
||||
"""Build a preregistered, unsubmitted oracle batch and return its manifest."""
|
||||
if not sources:
|
||||
raise ValueError("at least one source is required")
|
||||
if strength <= 0.0 or not math.isfinite(strength):
|
||||
raise ValueError("strength must be finite and positive")
|
||||
if provider not in {"google", "openai"}:
|
||||
raise ValueError("provider must be google or openai")
|
||||
if _inside(output_dir, repository_root):
|
||||
raise ValueError("oracle batches must be written outside the repository")
|
||||
if output_dir.exists() and any(output_dir.iterdir()):
|
||||
raise ValueError("output directory must not already contain files")
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
tile_model = load_tile_model(tile_model_path)
|
||||
phase_model = load_phase_model(phase_model_path)
|
||||
if (tile_model.height, tile_model.width) != (phase_model.height, phase_model.width):
|
||||
raise ValueError("tile and phase model geometries differ")
|
||||
templates = control_templates(tile_model.template, seed=seed)
|
||||
|
||||
rows: list[dict[str, object]] = []
|
||||
seen_source_hashes: set[str] = set()
|
||||
for source_path in sources:
|
||||
source_hash = artifact_sha256(source_path)
|
||||
if source_hash in seen_source_hashes:
|
||||
raise ValueError("duplicate source artifact")
|
||||
seen_source_hashes.add(source_hash)
|
||||
source = load_rgb(source_path)
|
||||
if source.shape != (tile_model.height, tile_model.width, 3):
|
||||
raise ValueError(f"{source_path}: geometry does not match the frozen models")
|
||||
source_id = source_hash[:16]
|
||||
score_args = {
|
||||
"source": source,
|
||||
"tile_model": tile_model,
|
||||
"phase_model": phase_model,
|
||||
"tile_threshold": tile_threshold,
|
||||
"phase_threshold": phase_threshold,
|
||||
"active_threshold": active_threshold,
|
||||
}
|
||||
source_scores = _score_row(source, **score_args)
|
||||
rows.append(
|
||||
_artifact_row(
|
||||
source_path,
|
||||
role="source",
|
||||
source_id=source_id,
|
||||
in_batch=False,
|
||||
batch_root=output_dir,
|
||||
scores=source_scores,
|
||||
)
|
||||
)
|
||||
for role in DERIVATIVE_ROLES:
|
||||
variant = (
|
||||
source
|
||||
if role == "reencode_control"
|
||||
else subtract_tiled_template(
|
||||
source,
|
||||
templates[role] * tile_model.expected_norm,
|
||||
strength=strength,
|
||||
)
|
||||
)
|
||||
output_path = output_dir / source_id / f"{role}.png"
|
||||
_write_png(output_path, variant)
|
||||
rows.append(
|
||||
_artifact_row(
|
||||
output_path,
|
||||
role=role,
|
||||
source_id=source_id,
|
||||
in_batch=True,
|
||||
batch_root=output_dir,
|
||||
scores=(source_scores if role == "reencode_control" else _score_row(variant, **score_args)),
|
||||
)
|
||||
)
|
||||
|
||||
manifest = {
|
||||
"format_version": FORMAT_VERSION,
|
||||
"status": "preregistered_unsubmitted",
|
||||
"source_count": len(sources),
|
||||
"request_count": len(rows),
|
||||
"request_order": ROLE_ORDER,
|
||||
"provider": provider,
|
||||
"decision_rule": (
|
||||
"Count causal success only when source, reencode_control, shifted, and "
|
||||
"orthogonal_random are detected in the matching provider SynthID oracle, "
|
||||
"aligned is not_detected, and indeterminate is never treated as negative."
|
||||
),
|
||||
"strength": strength,
|
||||
"seed": seed,
|
||||
"tile_threshold": tile_threshold,
|
||||
"phase_threshold": phase_threshold,
|
||||
"active_threshold": active_threshold,
|
||||
"tile_model": str(tile_model_path.resolve()),
|
||||
"tile_model_sha256": artifact_sha256(tile_model_path),
|
||||
"phase_model": str(phase_model_path.resolve()),
|
||||
"phase_model_sha256": artifact_sha256(phase_model_path),
|
||||
"rows": rows,
|
||||
}
|
||||
manifest_path = output_dir / "manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest, indent=2, allow_nan=False) + "\n", encoding="utf-8")
|
||||
manifest_hash = artifact_sha256(manifest_path)
|
||||
(output_dir / "manifest.sha256").write_text(manifest_hash + " manifest.json\n", encoding="utf-8")
|
||||
results_template = {
|
||||
"format_version": FORMAT_VERSION,
|
||||
"manifest_sha256": manifest_hash,
|
||||
"provider": provider,
|
||||
"rows": [
|
||||
{
|
||||
"source_id": row["source_id"],
|
||||
"role": row["role"],
|
||||
"artifact_sha256": row["artifact_sha256"],
|
||||
"synthid_result": None,
|
||||
"c2pa_result": None,
|
||||
"raw_response": None,
|
||||
"submitted_at": None,
|
||||
}
|
||||
for row in rows
|
||||
],
|
||||
}
|
||||
(output_dir / "results-template.json").write_text(
|
||||
json.dumps(results_template, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
verify_batch(manifest_path, repository_root=repository_root)
|
||||
return manifest_path
|
||||
|
||||
|
||||
def verify_batch(manifest_path: Path, *, repository_root: Path) -> dict[str, object]:
|
||||
"""Verify manifest, model, source, and derivative hashes without mutation."""
|
||||
batch_root = manifest_path.parent
|
||||
if _inside(batch_root, repository_root):
|
||||
raise ValueError("oracle batches must remain outside the repository")
|
||||
digest_path = batch_root / "manifest.sha256"
|
||||
expected_manifest_hash = digest_path.read_text(encoding="utf-8").split()[0]
|
||||
if artifact_sha256(manifest_path) != expected_manifest_hash:
|
||||
raise ValueError("manifest hash mismatch")
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
if manifest.get("format_version") != FORMAT_VERSION:
|
||||
raise ValueError("unsupported oracle-batch manifest version")
|
||||
rows = manifest.get("rows")
|
||||
if not isinstance(rows, list) or len(rows) != manifest.get("request_count"):
|
||||
raise ValueError("manifest request count mismatch")
|
||||
if artifact_sha256(Path(manifest["tile_model"])) != manifest["tile_model_sha256"]:
|
||||
raise ValueError("tile model hash mismatch")
|
||||
if artifact_sha256(Path(manifest["phase_model"])) != manifest["phase_model_sha256"]:
|
||||
raise ValueError("phase model hash mismatch")
|
||||
|
||||
groups: dict[str, list[str]] = {}
|
||||
artifact_hashes: set[str] = set()
|
||||
for row in rows:
|
||||
source_id = str(row["source_id"])
|
||||
groups.setdefault(source_id, []).append(str(row["role"]))
|
||||
path = batch_root / str(row["path"]) if row["in_batch"] else Path(str(row["path"]))
|
||||
if artifact_sha256(path) != row["artifact_sha256"]:
|
||||
raise ValueError(f"artifact hash mismatch for {source_id}/{row['role']}")
|
||||
pixel_sha256, width, height = pixel_fingerprint(path)
|
||||
if (pixel_sha256, width, height) != (row["pixel_sha256"], row["width"], row["height"]):
|
||||
raise ValueError(f"pixel fingerprint mismatch for {source_id}/{row['role']}")
|
||||
artifact_hashes.add(str(row["artifact_sha256"]))
|
||||
if any(tuple(roles) != ROLE_ORDER for roles in groups.values()):
|
||||
raise ValueError("each source must have the fixed request-role order")
|
||||
if len(groups) != manifest.get("source_count"):
|
||||
raise ValueError("manifest source count mismatch")
|
||||
if len(artifact_hashes) < len(rows) - len(groups):
|
||||
raise ValueError("unexpected duplicate derivative artifacts")
|
||||
template = json.loads((batch_root / "results-template.json").read_text(encoding="utf-8"))
|
||||
expected_identities = [(row["source_id"], row["role"], row["artifact_sha256"]) for row in rows]
|
||||
template_identities = [(row["source_id"], row["role"], row["artifact_sha256"]) for row in template.get("rows", [])]
|
||||
if template.get("manifest_sha256") != expected_manifest_hash or template.get("provider") != manifest["provider"]:
|
||||
raise ValueError("results template does not identify the preregistered batch")
|
||||
if template_identities != expected_identities:
|
||||
raise ValueError("results template row identities differ from the manifest")
|
||||
return manifest
|
||||
|
||||
|
||||
def _parse_submitted_at(value: object) -> None:
|
||||
"""Reject timestamps that are absent or not timezone-aware ISO-8601."""
|
||||
if not isinstance(value, str):
|
||||
raise ValueError("submitted_at must be a timezone-aware ISO-8601 timestamp")
|
||||
try:
|
||||
parsed = datetime.fromisoformat(value.replace("Z", "+00:00"))
|
||||
except ValueError as error:
|
||||
raise ValueError("submitted_at must be a timezone-aware ISO-8601 timestamp") from error
|
||||
if parsed.tzinfo is None:
|
||||
raise ValueError("submitted_at must be a timezone-aware ISO-8601 timestamp")
|
||||
|
||||
|
||||
def evaluate_results(
|
||||
manifest_path: Path,
|
||||
results_path: Path,
|
||||
*,
|
||||
repository_root: Path,
|
||||
) -> dict[str, object]:
|
||||
"""Validate a complete result file and apply the preregistered decision rule."""
|
||||
manifest = verify_batch(manifest_path, repository_root=repository_root)
|
||||
results = json.loads(results_path.read_text(encoding="utf-8"))
|
||||
manifest_hash = artifact_sha256(manifest_path)
|
||||
if results.get("format_version") != FORMAT_VERSION:
|
||||
raise ValueError("unsupported oracle-results version")
|
||||
if results.get("manifest_sha256") != manifest_hash or results.get("provider") != manifest["provider"]:
|
||||
raise ValueError("results do not identify the preregistered batch")
|
||||
manifest_rows = manifest["rows"]
|
||||
result_rows = results.get("rows")
|
||||
if not isinstance(result_rows, list) or len(result_rows) != len(manifest_rows):
|
||||
raise ValueError("oracle results must cover every preregistered request")
|
||||
|
||||
grouped: dict[str, dict[str, str]] = {}
|
||||
for expected, result in zip(manifest_rows, result_rows, strict=True):
|
||||
identity = (result.get("source_id"), result.get("role"), result.get("artifact_sha256"))
|
||||
expected_identity = (expected["source_id"], expected["role"], expected["artifact_sha256"])
|
||||
if identity != expected_identity:
|
||||
raise ValueError("oracle result order or artifact identity differs from the manifest")
|
||||
synthid_result = result.get("synthid_result")
|
||||
c2pa_result = result.get("c2pa_result")
|
||||
if synthid_result not in SYNTHID_RESULTS:
|
||||
raise ValueError("invalid or missing SynthID result")
|
||||
if c2pa_result not in C2PA_RESULTS:
|
||||
raise ValueError("invalid or missing C2PA result")
|
||||
if not isinstance(result.get("raw_response"), str) or not result["raw_response"].strip():
|
||||
raise ValueError("raw_response must preserve the nonempty verbatim oracle result")
|
||||
_parse_submitted_at(result.get("submitted_at"))
|
||||
grouped.setdefault(str(expected["source_id"]), {})[str(expected["role"])] = str(synthid_result)
|
||||
|
||||
source_results: list[dict[str, str]] = []
|
||||
for source_id, roles in grouped.items():
|
||||
if tuple(roles) != ROLE_ORDER:
|
||||
raise ValueError("oracle results do not preserve the fixed role order")
|
||||
values = set(roles.values())
|
||||
if values & {"indeterminate", "refused"}:
|
||||
verdict = "indeterminate"
|
||||
elif (
|
||||
roles["source"] == "detected"
|
||||
and roles["reencode_control"] == "detected"
|
||||
and roles["shifted"] == "detected"
|
||||
and roles["orthogonal_random"] == "detected"
|
||||
and roles["aligned"] == "not_detected"
|
||||
):
|
||||
verdict = "causal_success"
|
||||
elif any(roles[role] != "detected" for role in ("source", "reencode_control", "shifted", "orthogonal_random")):
|
||||
verdict = "control_failed"
|
||||
else:
|
||||
verdict = "aligned_still_detected"
|
||||
source_results.append({"source_id": source_id, "verdict": verdict})
|
||||
counts = {
|
||||
verdict: sum(row["verdict"] == verdict for row in source_results)
|
||||
for verdict in ("causal_success", "aligned_still_detected", "control_failed", "indeterminate")
|
||||
}
|
||||
return {
|
||||
"format_version": FORMAT_VERSION,
|
||||
"manifest_sha256": manifest_hash,
|
||||
"provider": manifest["provider"],
|
||||
"source_count": manifest["source_count"],
|
||||
"counts": counts,
|
||||
"sources": source_results,
|
||||
}
|
||||
|
||||
|
||||
@click.group()
|
||||
def main() -> None:
|
||||
"""Build and verify an immutable confirmatory oracle batch."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
|
||||
|
||||
@main.command("build")
|
||||
@click.argument("tile_model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("phase_model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("sources", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--output-dir", type=click.Path(file_okay=False, path_type=Path), required=True)
|
||||
@click.option("--tile-threshold", type=float, required=True)
|
||||
@click.option("--phase-threshold", type=float, required=True)
|
||||
@click.option("--active-threshold", type=float, required=True)
|
||||
@click.option("--strength", type=click.FloatRange(min=0.0, min_open=True), default=2.0, show_default=True)
|
||||
@click.option("--seed", type=int, default=20260810, show_default=True)
|
||||
@click.option("--provider", type=click.Choice(["google", "openai"]), required=True)
|
||||
def build_command(
|
||||
tile_model_path: Path,
|
||||
phase_model_path: Path,
|
||||
sources: tuple[Path, ...],
|
||||
output_dir: Path,
|
||||
tile_threshold: float,
|
||||
phase_threshold: float,
|
||||
active_threshold: float,
|
||||
strength: float,
|
||||
seed: int,
|
||||
provider: str,
|
||||
) -> None:
|
||||
"""Build a frozen batch from exact-geometry SOURCES."""
|
||||
manifest = build_batch(
|
||||
list(sources),
|
||||
output_dir=output_dir,
|
||||
tile_model_path=tile_model_path,
|
||||
phase_model_path=phase_model_path,
|
||||
tile_threshold=tile_threshold,
|
||||
phase_threshold=phase_threshold,
|
||||
active_threshold=active_threshold,
|
||||
strength=strength,
|
||||
seed=seed,
|
||||
provider=provider,
|
||||
repository_root=Path(__file__).resolve().parent.parent,
|
||||
)
|
||||
log.info("Wrote preregistered oracle batch: %s", manifest)
|
||||
|
||||
|
||||
@main.command("verify")
|
||||
@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
def verify_command(manifest_path: Path) -> None:
|
||||
"""Verify an existing batch without changing it."""
|
||||
manifest = verify_batch(manifest_path, repository_root=Path(__file__).resolve().parent.parent)
|
||||
log.info("Verified %d immutable oracle requests", manifest["request_count"])
|
||||
|
||||
|
||||
@main.command("evaluate")
|
||||
@click.argument("manifest_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("results_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def evaluate_command(manifest_path: Path, results_path: Path, report_out: Path) -> None:
|
||||
"""Validate RESULTS_PATH and write the preregistered batch verdict."""
|
||||
repository_root = Path(__file__).resolve().parent.parent
|
||||
if _inside(report_out, repository_root):
|
||||
raise click.BadParameter("oracle result reports must be written outside the repository")
|
||||
report = evaluate_results(
|
||||
manifest_path,
|
||||
results_path,
|
||||
repository_root=repository_root,
|
||||
)
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
if report_out.exists():
|
||||
raise click.BadParameter("refusing to overwrite an oracle result report")
|
||||
report_out.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||
log.info("Wrote oracle batch verdict: %s", report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Compatibility imports for shared periodic-residual helpers."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
from synthid_runtime.synthid_detector import fold_residual_template, unit_tile
|
||||
|
||||
__all__ = ["cyclic_tile_correlations", "fold_residual_template", "unit_tile"]
|
||||
|
||||
|
||||
def cyclic_tile_correlations(template: np.ndarray, tile: np.ndarray) -> np.ndarray:
|
||||
"""Return correlations for every cyclic spatial shift of TILE."""
|
||||
if template.shape != tile.shape or template.ndim != 3:
|
||||
raise ValueError("template and tile must have identical three-dimensional shapes")
|
||||
template_spectrum = np.fft.fft2(template, axes=(0, 1))
|
||||
tile_spectrum = np.fft.fft2(tile, axes=(0, 1))
|
||||
return np.fft.ifft2(np.sum(template_spectrum * np.conj(tile_spectrum), axis=2)).real
|
||||
@@ -0,0 +1,288 @@
|
||||
"""Measure whether a frozen periodic tile causally controls local carrier scores.
|
||||
|
||||
This harness compares subtraction of the learned tile with cyclically shifted
|
||||
and orthogonal random tiles of the same norm. It measures local research
|
||||
detectors only. A score reversal is not evidence that a provider oracle would
|
||||
stop detecting SynthID.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from synthid_periodic_tile import unit_tile
|
||||
from synthid_periodic_tile_probe import PeriodicTileModel
|
||||
from synthid_periodic_tile_probe import load_model as load_tile_model
|
||||
from synthid_periodic_tile_probe import score_pixels as score_tile_pixels
|
||||
from synthid_phase_carrier import PhaseCarrierModel, score_pixels
|
||||
from synthid_phase_carrier import load_model as load_phase_model
|
||||
from synthid_pixel_attack import load_rgb, measure
|
||||
from synthid_research_manifest import artifact_sha256
|
||||
from synthid_tile_attack import parse_positive_floats, subtract_tiled_template
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def exact_sign_test(negative: int, positive: int) -> float:
|
||||
"""Return an exact two-sided sign-test p-value after excluding ties."""
|
||||
count = negative + positive
|
||||
if count == 0:
|
||||
return 1.0
|
||||
tail = sum(math.comb(count, index) for index in range(min(negative, positive) + 1)) / 2**count
|
||||
return min(1.0, 2.0 * tail)
|
||||
|
||||
|
||||
def control_templates(template: np.ndarray, *, seed: int) -> dict[str, np.ndarray]:
|
||||
"""Return aligned, shifted, and norm-matched orthogonal control tiles."""
|
||||
rng = np.random.default_rng(seed)
|
||||
random_tile = rng.normal(size=template.shape)
|
||||
random_tile -= np.mean(random_tile, axis=(0, 1), keepdims=True)
|
||||
random_tile -= np.sum(random_tile * template) * template
|
||||
random_tile, norm = unit_tile(random_tile)
|
||||
if norm == 0.0 or abs(float(np.sum(random_tile * template))) > 1e-12:
|
||||
raise ValueError("could not construct an orthogonal random control")
|
||||
return {
|
||||
"aligned": template,
|
||||
"shifted": np.roll(template, shift=(1, 1), axis=(0, 1)),
|
||||
"orthogonal_random": random_tile,
|
||||
}
|
||||
|
||||
|
||||
def phase_score_pixels(pixels: np.ndarray, model: PhaseCarrierModel) -> tuple[float, float]:
|
||||
"""Return the unregistered phase score and active support for PIXELS."""
|
||||
result = score_pixels(pixels, model)
|
||||
return result.score, result.active_weight_fraction
|
||||
|
||||
|
||||
def tile_score_pixels(pixels: np.ndarray, model: PeriodicTileModel) -> float:
|
||||
"""Return the fixed-phase periodic-tile score for PIXELS."""
|
||||
return score_tile_pixels(pixels, model).score
|
||||
|
||||
|
||||
def summarize(values: list[float]) -> dict[str, float]:
|
||||
"""Return bounded descriptive statistics for VALUES."""
|
||||
return {
|
||||
"minimum": float(np.min(values)),
|
||||
"median": float(np.median(values)),
|
||||
"maximum": float(np.max(values)),
|
||||
}
|
||||
|
||||
|
||||
def direction_summary(values: list[float]) -> dict[str, float | int]:
|
||||
"""Return direction counts and a two-sided sign test for VALUES."""
|
||||
negative = sum(value < 0.0 for value in values)
|
||||
positive = sum(value > 0.0 for value in values)
|
||||
return {
|
||||
"negative": negative,
|
||||
"positive": positive,
|
||||
"ties": len(values) - negative - positive,
|
||||
"two_sided_sign_p": exact_sign_test(negative, positive),
|
||||
}
|
||||
|
||||
|
||||
def candidate_quality(reference: np.ndarray, candidate: np.ndarray) -> dict[str, float]:
|
||||
"""Return paired fidelity metrics for one equal-geometry candidate."""
|
||||
measurement = measure(reference, candidate, name="candidate", path=Path("<memory>"))
|
||||
return {
|
||||
"residual_rms": measurement.residual_rms,
|
||||
"psnr_db": measurement.psnr_db,
|
||||
"ssim": measurement.ssim,
|
||||
"changed_pixel_fraction": measurement.changed_pixel_fraction,
|
||||
}
|
||||
|
||||
|
||||
def run_ablation(
|
||||
sources: list[Path],
|
||||
*,
|
||||
tile_model: PeriodicTileModel,
|
||||
phase_model: PhaseCarrierModel,
|
||||
tile_threshold: float,
|
||||
phase_threshold: float,
|
||||
active_threshold: float,
|
||||
strengths: tuple[float, ...],
|
||||
phase_strength: float,
|
||||
seed: int,
|
||||
) -> dict[str, object]:
|
||||
"""Evaluate aligned subtraction and controls on exact-geometry SOURCES."""
|
||||
if not sources:
|
||||
raise ValueError("at least one source is required")
|
||||
if phase_strength not in strengths:
|
||||
raise ValueError("phase strength must be one of the swept strengths")
|
||||
if (tile_model.height, tile_model.width) != (phase_model.height, phase_model.width):
|
||||
raise ValueError("tile and phase model geometries differ")
|
||||
if not all(np.isfinite(value) for value in (tile_threshold, phase_threshold, active_threshold)):
|
||||
raise ValueError("thresholds must be finite")
|
||||
|
||||
templates = control_templates(tile_model.template, seed=seed)
|
||||
rows: list[dict[str, object]] = []
|
||||
for source_path in sources:
|
||||
source = load_rgb(source_path)
|
||||
if source.shape != (tile_model.height, tile_model.width, 3):
|
||||
raise ValueError(f"{source_path}: geometry does not match the models")
|
||||
source_hash = artifact_sha256(source_path)
|
||||
original_tile = tile_score_pixels(source, tile_model)
|
||||
original_phase, original_support = phase_score_pixels(source, phase_model)
|
||||
for strength in strengths:
|
||||
for control, template in templates.items():
|
||||
candidate = subtract_tiled_template(
|
||||
source,
|
||||
template * tile_model.expected_norm,
|
||||
strength=strength,
|
||||
)
|
||||
tile_score = tile_score_pixels(candidate, tile_model)
|
||||
row: dict[str, object] = {
|
||||
"path": str(source_path),
|
||||
"artifact_sha256": source_hash,
|
||||
"control": control,
|
||||
"strength": strength,
|
||||
"original_tile_score": original_tile,
|
||||
"tile_score": tile_score,
|
||||
"tile_delta": tile_score - original_tile,
|
||||
"tile_accepted": tile_score >= tile_threshold,
|
||||
"original_phase_score": original_phase,
|
||||
"original_active_support": original_support,
|
||||
}
|
||||
if strength == phase_strength:
|
||||
phase_score, active_support = phase_score_pixels(candidate, phase_model)
|
||||
row.update(
|
||||
{
|
||||
**candidate_quality(source, candidate),
|
||||
"phase_score": phase_score,
|
||||
"active_support": active_support,
|
||||
"phase_delta": phase_score - original_phase,
|
||||
"phase_accepted": phase_score >= phase_threshold and active_support >= active_threshold,
|
||||
}
|
||||
)
|
||||
rows.append(row)
|
||||
|
||||
grouped: dict[tuple[str, float], list[dict[str, object]]] = defaultdict(list)
|
||||
for row in rows:
|
||||
grouped[(str(row["control"]), float(row["strength"]))].append(row)
|
||||
tile_summaries: list[dict[str, object]] = []
|
||||
for (control, strength), group in sorted(grouped.items()):
|
||||
deltas = [float(row["tile_delta"]) for row in group]
|
||||
tile_summaries.append(
|
||||
{
|
||||
"control": control,
|
||||
"strength": strength,
|
||||
"accepted": sum(bool(row["tile_accepted"]) for row in group),
|
||||
"delta": summarize(deltas),
|
||||
"direction": direction_summary(deltas),
|
||||
}
|
||||
)
|
||||
|
||||
selected = [row for row in rows if float(row["strength"]) == phase_strength]
|
||||
phase_summaries: list[dict[str, object]] = []
|
||||
for control in templates:
|
||||
group = [row for row in selected if row["control"] == control]
|
||||
phase_summaries.append(
|
||||
{
|
||||
"control": control,
|
||||
"accepted": sum(bool(row["phase_accepted"]) for row in group),
|
||||
"delta": summarize([float(row["phase_delta"]) for row in group]),
|
||||
"active_support": summarize([float(row["active_support"]) for row in group]),
|
||||
"psnr_db": summarize([float(row["psnr_db"]) for row in group]),
|
||||
"ssim": summarize([float(row["ssim"]) for row in group]),
|
||||
"changed_pixel_fraction": summarize([float(row["changed_pixel_fraction"]) for row in group]),
|
||||
}
|
||||
)
|
||||
|
||||
paired_comparisons: list[dict[str, object]] = []
|
||||
for control in ("shifted", "orthogonal_random"):
|
||||
for metric in ("tile_delta", "phase_delta"):
|
||||
aligned = {
|
||||
str(row["artifact_sha256"]): float(row[metric]) for row in selected if row["control"] == "aligned"
|
||||
}
|
||||
comparison = {
|
||||
str(row["artifact_sha256"]): float(row[metric]) for row in selected if row["control"] == control
|
||||
}
|
||||
differences = [aligned[key] - comparison[key] for key in sorted(aligned)]
|
||||
paired_comparisons.append(
|
||||
{
|
||||
"aligned_minus": control,
|
||||
"metric": metric,
|
||||
"difference": summarize(differences),
|
||||
"direction": direction_summary(differences),
|
||||
}
|
||||
)
|
||||
|
||||
return {
|
||||
"source_count": len(sources),
|
||||
"tile_threshold": tile_threshold,
|
||||
"phase_threshold": phase_threshold,
|
||||
"active_threshold": active_threshold,
|
||||
"strengths": strengths,
|
||||
"phase_strength": phase_strength,
|
||||
"seed": seed,
|
||||
"original": {
|
||||
"tile_accepted": sum(
|
||||
float(row["original_tile_score"]) >= tile_threshold for row in selected if row["control"] == "aligned"
|
||||
),
|
||||
"phase_accepted": sum(
|
||||
float(row["original_phase_score"]) >= phase_threshold
|
||||
and float(row["original_active_support"]) >= active_threshold
|
||||
for row in selected
|
||||
if row["control"] == "aligned"
|
||||
),
|
||||
},
|
||||
"tile_summaries": tile_summaries,
|
||||
"phase_summaries": phase_summaries,
|
||||
"paired_comparisons": paired_comparisons,
|
||||
"items": rows,
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("tile_model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("phase_model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("sources", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--tile-threshold", type=float, required=True)
|
||||
@click.option("--phase-threshold", type=float, required=True)
|
||||
@click.option("--active-threshold", type=float, required=True)
|
||||
@click.option("--strengths", default="1,1.5,2,3,4", show_default=True)
|
||||
@click.option("--phase-strength", type=click.FloatRange(min=0.0, min_open=True), default=2.0, show_default=True)
|
||||
@click.option("--seed", type=int, default=20260810, show_default=True)
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def main(
|
||||
tile_model_path: Path,
|
||||
phase_model_path: Path,
|
||||
sources: tuple[Path, ...],
|
||||
tile_threshold: float,
|
||||
phase_threshold: float,
|
||||
active_threshold: float,
|
||||
strengths: str,
|
||||
phase_strength: float,
|
||||
seed: int,
|
||||
report_out: Path,
|
||||
) -> None:
|
||||
"""Run a fixed periodic-tile causal ablation on exact-geometry SOURCES."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
strength_values = parse_positive_floats(strengths, option_name="strengths")
|
||||
report = run_ablation(
|
||||
list(sources),
|
||||
tile_model=load_tile_model(tile_model_path),
|
||||
phase_model=load_phase_model(phase_model_path),
|
||||
tile_threshold=tile_threshold,
|
||||
phase_threshold=phase_threshold,
|
||||
active_threshold=active_threshold,
|
||||
strengths=strength_values,
|
||||
phase_strength=phase_strength,
|
||||
seed=seed,
|
||||
)
|
||||
report["tile_model"] = str(tile_model_path)
|
||||
report["tile_model_sha256"] = artifact_sha256(tile_model_path)
|
||||
report["phase_model"] = str(phase_model_path)
|
||||
report["phase_model_sha256"] = artifact_sha256(phase_model_path)
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||
log.info("Wrote periodic-tile causal ablation: %s", report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,273 @@
|
||||
"""Discover and evaluate an exact-geometry periodic residual carrier.
|
||||
|
||||
The model folds a high-pass residual modulo a fixed tile, averaging thousands
|
||||
of spatial repetitions before normalized correlation. It is a positive-only
|
||||
research signal, not a universal or certified SynthID decoder.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from synthid_periodic_tile import cyclic_tile_correlations, fold_residual_template, unit_tile
|
||||
from synthid_pixel_attack import load_rgb
|
||||
from synthid_research_manifest import artifact_sha256
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PeriodicTileModel:
|
||||
"""One exact-geometry normalized periodic-residual template."""
|
||||
|
||||
height: int
|
||||
width: int
|
||||
tile_height: int
|
||||
tile_width: int
|
||||
denoise_sigma: float
|
||||
template: np.ndarray
|
||||
expected_norm: float
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PeriodicTileScore:
|
||||
"""Normalized tile correlation and support for one image."""
|
||||
|
||||
path: str
|
||||
score: float
|
||||
active_support: float
|
||||
row_shift: int
|
||||
column_shift: int
|
||||
repeat_count: int
|
||||
|
||||
|
||||
def _load_rgb(path: Path, *, height: int, width: int) -> np.ndarray:
|
||||
"""Load PATH as exact-geometry uint8 RGB."""
|
||||
pixels = load_rgb(path)
|
||||
if pixels.shape != (height, width, 3):
|
||||
raise ValueError(f"{path}: geometry {pixels.shape[1]}x{pixels.shape[0]} does not match {width}x{height}")
|
||||
return pixels
|
||||
|
||||
|
||||
def discover_model(
|
||||
paths: list[Path],
|
||||
*,
|
||||
tile_height: int,
|
||||
tile_width: int,
|
||||
denoise_sigma: float = 1.0,
|
||||
) -> PeriodicTileModel:
|
||||
"""Learn a normalized periodic template from positive PATHS."""
|
||||
if len(paths) < 3:
|
||||
raise ValueError("at least three positive images are required")
|
||||
first_pixels = load_rgb(paths[0])
|
||||
height, width = first_pixels.shape[:2]
|
||||
unit_sum = np.zeros((tile_height, tile_width, 3), dtype=np.float64)
|
||||
norms: list[float] = []
|
||||
for index, path in enumerate(paths):
|
||||
folded = fold_residual_template(
|
||||
first_pixels if index == 0 else _load_rgb(path, height=height, width=width),
|
||||
tile_height=tile_height,
|
||||
tile_width=tile_width,
|
||||
denoise_sigma=denoise_sigma,
|
||||
)
|
||||
unit, norm = unit_tile(folded)
|
||||
unit_sum += unit
|
||||
norms.append(norm)
|
||||
template, template_norm = unit_tile(unit_sum / len(paths))
|
||||
if template_norm == 0.0:
|
||||
raise ValueError("positive images expose no periodic residual template")
|
||||
return PeriodicTileModel(
|
||||
height=height,
|
||||
width=width,
|
||||
tile_height=tile_height,
|
||||
tile_width=tile_width,
|
||||
denoise_sigma=denoise_sigma,
|
||||
template=template,
|
||||
expected_norm=float(np.median(norms)),
|
||||
)
|
||||
|
||||
|
||||
def score_pixels(
|
||||
pixels: np.ndarray,
|
||||
model: PeriodicTileModel,
|
||||
*,
|
||||
register: bool = False,
|
||||
path: str = "<array>",
|
||||
) -> PeriodicTileScore:
|
||||
"""Score exact-geometry RGB PIXELS, optionally searching cyclic tile shifts."""
|
||||
if pixels.shape != (model.height, model.width, 3):
|
||||
raise ValueError("pixel geometry does not match periodic-tile model")
|
||||
folded = fold_residual_template(
|
||||
pixels,
|
||||
tile_height=model.tile_height,
|
||||
tile_width=model.tile_width,
|
||||
denoise_sigma=model.denoise_sigma,
|
||||
)
|
||||
unit, norm = unit_tile(folded)
|
||||
if register:
|
||||
correlations = cyclic_tile_correlations(model.template, unit)
|
||||
row_shift, column_shift = np.unravel_index(int(np.argmax(correlations)), correlations.shape)
|
||||
score = float(correlations[row_shift, column_shift])
|
||||
else:
|
||||
score = float(np.sum(model.template * unit))
|
||||
row_shift = column_shift = 0
|
||||
return PeriodicTileScore(
|
||||
path=path,
|
||||
score=score,
|
||||
active_support=min(norm / (model.expected_norm + 1e-12), 1.0),
|
||||
row_shift=row_shift,
|
||||
column_shift=column_shift,
|
||||
repeat_count=(model.height // model.tile_height) * (model.width // model.tile_width),
|
||||
)
|
||||
|
||||
|
||||
def score_image(path: Path, model: PeriodicTileModel, *, register: bool = False) -> PeriodicTileScore:
|
||||
"""Score PATH against MODEL, optionally searching cyclic tile shifts."""
|
||||
pixels = _load_rgb(path, height=model.height, width=model.width)
|
||||
return score_pixels(pixels, model, register=register, path=str(path))
|
||||
|
||||
|
||||
def calibrate_threshold(paths: list[Path], model: PeriodicTileModel, *, register: bool = False) -> float:
|
||||
"""Return the first float above every negative score in PATHS."""
|
||||
if not paths:
|
||||
raise ValueError("at least one calibration negative is required")
|
||||
maximum = max(score_image(path, model, register=register).score for path in paths)
|
||||
return float(np.nextafter(maximum, np.inf))
|
||||
|
||||
|
||||
def save_model(path: Path, model: PeriodicTileModel) -> None:
|
||||
"""Save MODEL as a pickle-free numeric artifact without precision loss."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
np.savez_compressed(
|
||||
path,
|
||||
format_version=np.asarray(1, dtype=np.int32),
|
||||
height=np.asarray(model.height, dtype=np.int32),
|
||||
width=np.asarray(model.width, dtype=np.int32),
|
||||
tile_height=np.asarray(model.tile_height, dtype=np.int32),
|
||||
tile_width=np.asarray(model.tile_width, dtype=np.int32),
|
||||
denoise_sigma=np.asarray(model.denoise_sigma, dtype=np.float64),
|
||||
template=model.template.astype(np.float64),
|
||||
expected_norm=np.asarray(model.expected_norm, dtype=np.float64),
|
||||
)
|
||||
|
||||
|
||||
def load_model(path: Path) -> PeriodicTileModel:
|
||||
"""Load and validate one numeric periodic-tile model."""
|
||||
with np.load(path, allow_pickle=False) as artifact:
|
||||
if int(artifact["format_version"]) != 1:
|
||||
raise ValueError("unsupported periodic-tile model format version")
|
||||
model = PeriodicTileModel(
|
||||
height=int(artifact["height"]),
|
||||
width=int(artifact["width"]),
|
||||
tile_height=int(artifact["tile_height"]),
|
||||
tile_width=int(artifact["tile_width"]),
|
||||
denoise_sigma=float(artifact["denoise_sigma"]),
|
||||
template=np.asarray(artifact["template"], dtype=np.float64),
|
||||
expected_norm=float(artifact["expected_norm"]),
|
||||
)
|
||||
if model.height < 1 or model.width < 1 or model.tile_height < 1 or model.tile_width < 1:
|
||||
raise ValueError("invalid periodic-tile geometry")
|
||||
if model.height % model.tile_height or model.width % model.tile_width:
|
||||
raise ValueError("image geometry is not divisible by periodic-tile geometry")
|
||||
if model.template.shape != (model.tile_height, model.tile_width, 3):
|
||||
raise ValueError("invalid periodic-tile template shape")
|
||||
if not np.all(np.isfinite(model.template)) or not np.isclose(np.linalg.norm(model.template), 1.0):
|
||||
raise ValueError("invalid periodic-tile template")
|
||||
if not np.isfinite(model.denoise_sigma) or model.denoise_sigma <= 0.0:
|
||||
raise ValueError("invalid periodic-tile denoise sigma")
|
||||
if not np.isfinite(model.expected_norm) or model.expected_norm <= 0.0:
|
||||
raise ValueError("invalid periodic-tile expected norm")
|
||||
return model
|
||||
|
||||
|
||||
@click.group()
|
||||
def main() -> None:
|
||||
"""Discover and evaluate an exact-geometry periodic carrier."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("positives", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--tile-height", type=click.IntRange(min=1), required=True)
|
||||
@click.option("--tile-width", type=click.IntRange(min=1), required=True)
|
||||
@click.option("--denoise-sigma", type=click.FloatRange(min=0.0, min_open=True), default=1.0, show_default=True)
|
||||
@click.option("--model-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def discover(
|
||||
positives: tuple[Path, ...],
|
||||
tile_height: int,
|
||||
tile_width: int,
|
||||
denoise_sigma: float,
|
||||
model_out: Path,
|
||||
) -> None:
|
||||
"""Learn a periodic tile from exact-geometry POSITIVES."""
|
||||
save_model(
|
||||
model_out,
|
||||
discover_model(
|
||||
list(positives),
|
||||
tile_height=tile_height,
|
||||
tile_width=tile_width,
|
||||
denoise_sigma=denoise_sigma,
|
||||
),
|
||||
)
|
||||
log.info("Wrote periodic-tile model: %s", model_out)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("negatives", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--register", is_flag=True, help="Search every cyclic shift within the learned tile.")
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def calibrate(model_path: Path, negatives: tuple[Path, ...], register: bool, report_out: Path) -> None:
|
||||
"""Calibrate a zero-observed-error threshold on NEGATIVES."""
|
||||
model = load_model(model_path)
|
||||
threshold = calibrate_threshold(list(negatives), model, register=register)
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"model": str(model_path),
|
||||
"model_sha256": artifact_sha256(model_path),
|
||||
"register": register,
|
||||
"negative_count": len(negatives),
|
||||
"threshold": threshold,
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote periodic-tile calibration report: %s", report_out)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--register", is_flag=True, help="Search every cyclic shift within the learned tile.")
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def score(model_path: Path, images: tuple[Path, ...], register: bool, report_out: Path) -> None:
|
||||
"""Score exact-geometry IMAGES with MODEL_PATH."""
|
||||
model = load_model(model_path)
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(
|
||||
json.dumps(
|
||||
{
|
||||
"model": str(model_path),
|
||||
"model_sha256": artifact_sha256(model_path),
|
||||
"register": register,
|
||||
"scores": [asdict(score_image(image, model, register=register)) for image in images],
|
||||
},
|
||||
indent=2,
|
||||
)
|
||||
+ "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote periodic-tile score report: %s", report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,384 @@
|
||||
"""Discover and evaluate an exact-geometry phase-carrier hypothesis.
|
||||
|
||||
The model is learned only from supplied images and stored as numeric arrays in
|
||||
a pickle-free NPZ. It is a research baseline, not a proprietary SynthID
|
||||
decoder. A valid detector claim still requires provider labels, same-provider
|
||||
hard negatives, group-aware splits, and a locked operating point.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
from synthid_phase_registration import MAX_TRANSLATION_SHIFT, extract_frequency_values, register_phase_translations
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PhaseCarrierModel:
|
||||
"""Sparse exact-geometry phase carrier learned from positive images."""
|
||||
|
||||
height: int
|
||||
width: int
|
||||
rows: np.ndarray
|
||||
columns: np.ndarray
|
||||
channels: np.ndarray
|
||||
phases: np.ndarray
|
||||
weights: np.ndarray
|
||||
expected_magnitudes: np.ndarray
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class PhaseCarrierScore:
|
||||
"""Alignment of one exact-geometry image with a phase-carrier model."""
|
||||
|
||||
path: str
|
||||
score: float
|
||||
active_weight_fraction: float
|
||||
peak_count: int
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RegisteredPhaseCarrierScore:
|
||||
"""Best phase-carrier score across a bounded translation search."""
|
||||
|
||||
path: str
|
||||
score: float
|
||||
active_weight_fraction: float
|
||||
peak_count: int
|
||||
row_shift: int
|
||||
column_shift: int
|
||||
|
||||
|
||||
def _load_rgb(path: Path, *, height: int, width: int, canonicalize_geometry: bool = False) -> np.ndarray:
|
||||
"""Load PATH as float64 RGB, optionally resizing to model geometry."""
|
||||
with Image.open(path) as image:
|
||||
rgb = image.convert("RGB")
|
||||
if rgb.size != (width, height):
|
||||
if not canonicalize_geometry:
|
||||
raise ValueError(f"{path}: geometry {rgb.width}x{rgb.height} does not match {width}x{height}")
|
||||
rgb = rgb.resize((width, height), Image.Resampling.LANCZOS)
|
||||
return np.asarray(rgb, dtype=np.float64)
|
||||
|
||||
|
||||
def _valid_frequency_mask(height: int, width: int, min_radius: float) -> np.ndarray:
|
||||
"""Return eligible non-DC bins in an rFFT half-plane."""
|
||||
rows = np.arange(height)
|
||||
signed_rows = np.where(rows > height // 2, rows - height, rows)
|
||||
columns = np.arange(width // 2 + 1)
|
||||
radius = np.sqrt(np.square(signed_rows[:, None]) + np.square(columns[None, :]))
|
||||
return (radius >= min_radius) & (columns[None, :] > 0)
|
||||
|
||||
|
||||
def _leave_one_out_coherence(unit_sum: np.ndarray, held_out_unit: np.ndarray, count: float) -> np.ndarray:
|
||||
"""Return phase coherence after removing HELD_OUT_UNIT from UNIT_SUM."""
|
||||
if count <= 1.0:
|
||||
raise ValueError("leave-one-out coherence requires at least two samples")
|
||||
return np.abs((unit_sum - held_out_unit) / (count - 1.0))
|
||||
|
||||
|
||||
def discover_model(
|
||||
paths: list[Path],
|
||||
*,
|
||||
peak_count: int = 256,
|
||||
min_radius: float = 15.0,
|
||||
candidate_bins: np.ndarray | None = None,
|
||||
) -> PhaseCarrierModel:
|
||||
"""Learn a sparse phase-consensus model from exact-geometry PATHS."""
|
||||
if len(paths) < 3:
|
||||
raise ValueError("at least three positive images are required")
|
||||
with Image.open(paths[0]) as first:
|
||||
width, height = first.size
|
||||
if min(height, width) < 64:
|
||||
raise ValueError("images must be at least 64 pixels per side")
|
||||
|
||||
half_width = width // 2 + 1
|
||||
unit_sum = np.zeros((height, half_width, 3), dtype=np.complex64)
|
||||
magnitude_sum = np.zeros((height, half_width, 3), dtype=np.float64)
|
||||
for path in paths:
|
||||
pixels = _load_rgb(path, height=height, width=width)
|
||||
for channel in range(3):
|
||||
spectrum = np.fft.rfft2(pixels[:, :, channel])
|
||||
magnitude = np.abs(spectrum)
|
||||
unit_sum[:, :, channel] += np.divide(
|
||||
spectrum,
|
||||
magnitude,
|
||||
out=np.zeros_like(spectrum),
|
||||
where=magnitude != 0.0,
|
||||
).astype(np.complex64)
|
||||
magnitude_sum[:, :, channel] += magnitude
|
||||
|
||||
count = float(len(paths))
|
||||
mean_unit = unit_sum / count
|
||||
minimum_loo_coherence = np.ones((height, half_width, 3), dtype=np.float32)
|
||||
for path in paths:
|
||||
pixels = _load_rgb(path, height=height, width=width)
|
||||
for channel in range(3):
|
||||
spectrum = np.fft.rfft2(pixels[:, :, channel])
|
||||
magnitude = np.abs(spectrum)
|
||||
unit = np.divide(
|
||||
spectrum,
|
||||
magnitude,
|
||||
out=np.zeros_like(spectrum),
|
||||
where=magnitude != 0.0,
|
||||
)
|
||||
loo_coherence = _leave_one_out_coherence(unit_sum[:, :, channel], unit, count)
|
||||
np.minimum(minimum_loo_coherence[:, :, channel], loo_coherence, out=minimum_loo_coherence[:, :, channel])
|
||||
expected_magnitude = magnitude_sum / count
|
||||
selection = np.power(minimum_loo_coherence.astype(np.float64), 4.0) * np.log1p(expected_magnitude)
|
||||
selection *= _valid_frequency_mask(height, width, min_radius)[:, :, None]
|
||||
if candidate_bins is None:
|
||||
candidate_indices = np.flatnonzero(selection)
|
||||
else:
|
||||
bins = np.asarray(candidate_bins, dtype=np.int64)
|
||||
if bins.ndim != 2 or bins.shape[1] != 3:
|
||||
raise ValueError("candidate_bins must have shape (count, 3)")
|
||||
if (
|
||||
np.any(bins[:, 0] < 0)
|
||||
or np.any(bins[:, 0] >= height)
|
||||
or np.any(bins[:, 1] <= 0)
|
||||
or np.any(bins[:, 1] > width // 2)
|
||||
or np.any(bins[:, 2] < 0)
|
||||
or np.any(bins[:, 2] > 2)
|
||||
):
|
||||
raise ValueError("candidate_bins contain out-of-range coordinates")
|
||||
candidate_indices = np.unique(np.ravel_multi_index(bins.T, selection.shape))
|
||||
candidate_indices = candidate_indices[selection.ravel()[candidate_indices] > 0.0]
|
||||
candidate_count = len(candidate_indices)
|
||||
if candidate_count < peak_count:
|
||||
raise ValueError(f"only {candidate_count} eligible bins for {peak_count} peaks")
|
||||
|
||||
flat = selection.ravel()
|
||||
candidate_scores = flat[candidate_indices]
|
||||
chosen = np.argpartition(candidate_scores, -peak_count)[-peak_count:]
|
||||
indices = candidate_indices[chosen]
|
||||
indices = indices[np.argsort(flat[indices])[::-1]]
|
||||
rows, columns, channels = np.unravel_index(indices, selection.shape)
|
||||
raw_weights = selection[rows, columns, channels]
|
||||
return PhaseCarrierModel(
|
||||
height=height,
|
||||
width=width,
|
||||
rows=rows.astype(np.int32),
|
||||
columns=columns.astype(np.int32),
|
||||
channels=channels.astype(np.int8),
|
||||
phases=np.angle(mean_unit[rows, columns, channels]).astype(np.float64),
|
||||
weights=(raw_weights / np.sum(raw_weights)).astype(np.float64),
|
||||
expected_magnitudes=expected_magnitude[rows, columns, channels].astype(np.float64),
|
||||
)
|
||||
|
||||
|
||||
def _frequency_values(pixels: np.ndarray, model: PhaseCarrierModel) -> np.ndarray:
|
||||
"""Return MODEL's selected complex coefficients from PIXELS."""
|
||||
return extract_frequency_values(pixels, model.rows, model.columns, model.channels)
|
||||
|
||||
|
||||
def score_pixels(pixels: np.ndarray, model: PhaseCarrierModel, *, path: str = "<array>") -> PhaseCarrierScore:
|
||||
"""Score exact-geometry RGB PIXELS against MODEL."""
|
||||
if pixels.shape != (model.height, model.width, 3):
|
||||
raise ValueError("pixel geometry does not match phase-carrier model")
|
||||
values = _frequency_values(np.asarray(pixels, dtype=np.float64), model)
|
||||
magnitude_gate = np.minimum(np.abs(values) / (model.expected_magnitudes + 1e-12), 1.0)
|
||||
active_weights = model.weights * magnitude_gate
|
||||
active_weight = float(np.sum(active_weights))
|
||||
score = (
|
||||
0.0
|
||||
if active_weight == 0.0
|
||||
else float(np.sum(active_weights * np.cos(np.angle(values) - model.phases)) / active_weight)
|
||||
)
|
||||
return PhaseCarrierScore(
|
||||
path=path,
|
||||
score=score,
|
||||
active_weight_fraction=active_weight,
|
||||
peak_count=len(model.rows),
|
||||
)
|
||||
|
||||
|
||||
def score_image(
|
||||
path: Path,
|
||||
model: PhaseCarrierModel,
|
||||
*,
|
||||
canonicalize_geometry: bool = False,
|
||||
) -> PhaseCarrierScore:
|
||||
"""Score PATH against MODEL, with optional geometry canonicalization."""
|
||||
pixels = _load_rgb(
|
||||
path,
|
||||
height=model.height,
|
||||
width=model.width,
|
||||
canonicalize_geometry=canonicalize_geometry,
|
||||
)
|
||||
return score_pixels(pixels, model, path=str(path))
|
||||
|
||||
|
||||
def score_translations(
|
||||
path: Path,
|
||||
model: PhaseCarrierModel,
|
||||
*,
|
||||
max_shift: int = 4,
|
||||
canonicalize_geometry: bool = False,
|
||||
) -> RegisteredPhaseCarrierScore:
|
||||
"""Return the best score after compensating bounded integer translations."""
|
||||
pixels = _load_rgb(
|
||||
path,
|
||||
height=model.height,
|
||||
width=model.width,
|
||||
canonicalize_geometry=canonicalize_geometry,
|
||||
)
|
||||
registration = register_phase_translations(
|
||||
_frequency_values(pixels, model),
|
||||
phases=model.phases,
|
||||
weights=model.weights,
|
||||
expected_magnitudes=model.expected_magnitudes,
|
||||
rows=model.rows,
|
||||
columns=model.columns,
|
||||
height=model.height,
|
||||
width=model.width,
|
||||
max_shift=max_shift,
|
||||
)
|
||||
return RegisteredPhaseCarrierScore(
|
||||
path=str(path),
|
||||
score=registration.score,
|
||||
active_weight_fraction=registration.active_weight_fraction,
|
||||
peak_count=len(model.rows),
|
||||
row_shift=registration.row_shift,
|
||||
column_shift=registration.column_shift,
|
||||
)
|
||||
|
||||
|
||||
def save_model(path: Path, model: PhaseCarrierModel) -> None:
|
||||
"""Save MODEL as a validated numeric NPZ artifact."""
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
np.savez_compressed(
|
||||
path,
|
||||
format_version=np.asarray(1, dtype=np.int32),
|
||||
height=np.asarray(model.height, dtype=np.int32),
|
||||
width=np.asarray(model.width, dtype=np.int32),
|
||||
rows=model.rows.astype(np.int32),
|
||||
columns=model.columns.astype(np.int32),
|
||||
channels=model.channels.astype(np.int8),
|
||||
phases=model.phases.astype(np.float32),
|
||||
weights=model.weights.astype(np.float32),
|
||||
expected_magnitudes=model.expected_magnitudes.astype(np.float64),
|
||||
)
|
||||
|
||||
|
||||
def load_model(path: Path) -> PhaseCarrierModel:
|
||||
"""Load and validate a numeric phase-carrier artifact."""
|
||||
with np.load(path, allow_pickle=False) as artifact:
|
||||
if int(artifact["format_version"]) != 1:
|
||||
raise ValueError("unsupported phase-carrier format version")
|
||||
model = PhaseCarrierModel(
|
||||
height=int(artifact["height"]),
|
||||
width=int(artifact["width"]),
|
||||
rows=np.asarray(artifact["rows"], dtype=np.int32),
|
||||
columns=np.asarray(artifact["columns"], dtype=np.int32),
|
||||
channels=np.asarray(artifact["channels"], dtype=np.int8),
|
||||
phases=np.asarray(artifact["phases"], dtype=np.float64),
|
||||
weights=np.asarray(artifact["weights"], dtype=np.float64),
|
||||
expected_magnitudes=np.asarray(artifact["expected_magnitudes"], dtype=np.float64),
|
||||
)
|
||||
count = len(model.rows)
|
||||
arrays = (model.columns, model.channels, model.phases, model.weights, model.expected_magnitudes)
|
||||
if model.height < 64 or model.width < 64 or any(array.shape != (count,) for array in arrays):
|
||||
raise ValueError("invalid phase-carrier model shapes")
|
||||
if count == 0 or np.any(model.rows < 0) or np.any(model.rows >= model.height):
|
||||
raise ValueError("invalid phase-carrier row indices")
|
||||
if np.any(model.columns <= 0) or np.any(model.columns > model.width // 2):
|
||||
raise ValueError("invalid phase-carrier column indices")
|
||||
if np.any(model.channels < 0) or np.any(model.channels > 2):
|
||||
raise ValueError("invalid phase-carrier channel indices")
|
||||
if not np.isclose(np.sum(model.weights), 1.0, atol=1e-5) or np.any(model.weights < 0.0):
|
||||
raise ValueError("invalid phase-carrier weights")
|
||||
return model
|
||||
|
||||
|
||||
@click.group()
|
||||
def main() -> None:
|
||||
"""Discover and evaluate an exact-geometry phase carrier."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("positives", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--peak-count", type=click.IntRange(min=1), default=256, show_default=True)
|
||||
@click.option("--candidate-codebook", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--candidate-count", type=click.IntRange(min=1), default=16384, show_default=True)
|
||||
@click.option("--model-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
def discover(
|
||||
positives: tuple[Path, ...],
|
||||
peak_count: int,
|
||||
candidate_codebook: Path | None,
|
||||
candidate_count: int,
|
||||
model_out: Path,
|
||||
) -> None:
|
||||
"""Learn a phase carrier from exact-geometry POSITIVES."""
|
||||
candidate_bins: np.ndarray | None = None
|
||||
if candidate_codebook is not None:
|
||||
from synthid_v3_codebook_probe import load_v3_model
|
||||
|
||||
with Image.open(positives[0]) as first:
|
||||
width, height = first.size
|
||||
prior = load_v3_model(
|
||||
candidate_codebook,
|
||||
height=height,
|
||||
width=width,
|
||||
peak_count=candidate_count,
|
||||
)
|
||||
candidate_bins = np.column_stack((prior.rows, prior.columns, prior.channels))
|
||||
model = discover_model(list(positives), peak_count=peak_count, candidate_bins=candidate_bins)
|
||||
save_model(model_out, model)
|
||||
log.info("Wrote phase-carrier model: %s", model_out)
|
||||
|
||||
|
||||
@main.command()
|
||||
@click.argument("model_path", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("images", nargs=-1, required=True, type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.option("--report-out", type=click.Path(dir_okay=False, path_type=Path), required=True)
|
||||
@click.option("--canonicalize-geometry", is_flag=True, help="Resize inputs to the model geometry before scoring.")
|
||||
@click.option("--max-shift", type=click.IntRange(min=0, max=MAX_TRANSLATION_SHIFT), default=0, show_default=True)
|
||||
def score(
|
||||
model_path: Path,
|
||||
images: tuple[Path, ...],
|
||||
report_out: Path,
|
||||
canonicalize_geometry: bool,
|
||||
max_shift: int,
|
||||
) -> None:
|
||||
"""Score IMAGES with MODEL_PATH."""
|
||||
model = load_model(model_path)
|
||||
scores = (
|
||||
[asdict(score_image(image, model, canonicalize_geometry=canonicalize_geometry)) for image in images]
|
||||
if max_shift == 0
|
||||
else [
|
||||
asdict(
|
||||
score_translations(
|
||||
image,
|
||||
model,
|
||||
max_shift=max_shift,
|
||||
canonicalize_geometry=canonicalize_geometry,
|
||||
)
|
||||
)
|
||||
for image in images
|
||||
]
|
||||
)
|
||||
payload = {
|
||||
"model": str(model_path),
|
||||
"height": model.height,
|
||||
"width": model.width,
|
||||
"peak_count": len(model.rows),
|
||||
"canonicalize_geometry": canonicalize_geometry,
|
||||
"max_shift": max_shift,
|
||||
"scores": scores,
|
||||
}
|
||||
report_out.parent.mkdir(parents=True, exist_ok=True)
|
||||
report_out.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
|
||||
log.info("Wrote phase-carrier score report: %s", report_out)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,92 @@
|
||||
"""Shared bounded translation registration for phase-carrier probes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
MAX_TRANSLATION_SHIFT = 32
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class TranslationRegistration:
|
||||
"""Best phase score and offset from one bounded translation search."""
|
||||
|
||||
score: float
|
||||
active_weight_fraction: float
|
||||
row_shift: int
|
||||
column_shift: int
|
||||
|
||||
|
||||
def phase_adjustment(
|
||||
rows: np.ndarray,
|
||||
columns: np.ndarray,
|
||||
*,
|
||||
height: int,
|
||||
width: int,
|
||||
row_shift: int | np.ndarray,
|
||||
column_shift: int | np.ndarray,
|
||||
) -> np.ndarray:
|
||||
"""Return the Fourier phase adjustment for one integer translation."""
|
||||
signed_rows = np.where(rows > height // 2, rows - height, rows)
|
||||
return 2.0 * np.pi * (signed_rows * row_shift / height + columns * column_shift / width)
|
||||
|
||||
|
||||
def extract_frequency_values(
|
||||
pixels: np.ndarray,
|
||||
rows: np.ndarray,
|
||||
columns: np.ndarray,
|
||||
channels: np.ndarray,
|
||||
) -> np.ndarray:
|
||||
"""Extract sparse three-channel rFFT coefficients from PIXELS."""
|
||||
values = np.empty(len(rows), dtype=np.complex128)
|
||||
for channel in range(3):
|
||||
positions = np.flatnonzero(channels == channel)
|
||||
if len(positions) == 0:
|
||||
continue
|
||||
spectrum = np.fft.rfft2(pixels[:, :, channel])
|
||||
values[positions] = spectrum[rows[positions], columns[positions]]
|
||||
return values
|
||||
|
||||
|
||||
def register_phase_translations(
|
||||
values: np.ndarray,
|
||||
*,
|
||||
phases: np.ndarray,
|
||||
weights: np.ndarray,
|
||||
expected_magnitudes: np.ndarray,
|
||||
rows: np.ndarray,
|
||||
columns: np.ndarray,
|
||||
height: int,
|
||||
width: int,
|
||||
max_shift: int,
|
||||
) -> TranslationRegistration:
|
||||
"""Find the strongest phase alignment over bounded integer translations."""
|
||||
if not 0 <= max_shift <= MAX_TRANSLATION_SHIFT:
|
||||
raise ValueError(f"max_shift must be between 0 and {MAX_TRANSLATION_SHIFT}")
|
||||
magnitude_gate = np.minimum(np.abs(values) / (expected_magnitudes + 1e-12), 1.0)
|
||||
active_weights = weights * magnitude_gate
|
||||
active_weight = float(np.sum(active_weights))
|
||||
if active_weight == 0.0:
|
||||
return TranslationRegistration(0.0, 0.0, 0, 0)
|
||||
|
||||
shifts = np.arange(-max_shift, max_shift + 1)
|
||||
adjustment = phase_adjustment(
|
||||
rows[:, None, None],
|
||||
columns[:, None, None],
|
||||
height=height,
|
||||
width=width,
|
||||
row_shift=shifts[None, :, None],
|
||||
column_shift=shifts[None, None, :],
|
||||
)
|
||||
difference = np.angle(values) - phases
|
||||
scores = np.sum(active_weights[:, None, None] * np.cos(difference[:, None, None] + adjustment), axis=0)
|
||||
scores /= active_weight
|
||||
row_index, column_index = np.unravel_index(int(np.argmax(scores)), scores.shape)
|
||||
return TranslationRegistration(
|
||||
score=float(scores[row_index, column_index]),
|
||||
active_weight_fraction=active_weight,
|
||||
row_shift=int(shifts[row_index]),
|
||||
column_shift=int(shifts[column_index]),
|
||||
)
|
||||
@@ -0,0 +1,199 @@
|
||||
"""Build deterministic, pixel-only SynthID attack candidates and controls.
|
||||
|
||||
The generated variants use quantization, resampling, and a smooth sub-pixel
|
||||
warp. No generative model or image synthesis stage is involved. The command
|
||||
also emits a norm-matched random-noise control so an oracle change cannot be
|
||||
attributed to pixel distance alone.
|
||||
|
||||
This is a research harness. A candidate is successful only when the matching
|
||||
provider oracle changes from detected to not detected while the crop-only
|
||||
control remains detected.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import logging
|
||||
import math
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
import cv2
|
||||
import numpy as np
|
||||
from invisible_quality_audit import _ssim
|
||||
from PIL import Image
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FidelityMeasurement:
|
||||
"""Paired pixel metrics for one attack candidate."""
|
||||
|
||||
name: str
|
||||
path: str
|
||||
width: int
|
||||
height: int
|
||||
psnr_db: float
|
||||
ssim: float
|
||||
changed_pixel_fraction: float
|
||||
residual_rms: float
|
||||
residual_max: float
|
||||
|
||||
|
||||
def load_rgb(path: Path) -> np.ndarray:
|
||||
"""Load PATH as uint8 RGB pixels."""
|
||||
with Image.open(path) as image:
|
||||
return np.asarray(image.convert("RGB"), dtype=np.uint8)
|
||||
|
||||
|
||||
def crop_visible_badge(pixels: np.ndarray, margin: int) -> np.ndarray:
|
||||
"""Remove the bottom and right margins that contain the visible badge."""
|
||||
height, width = pixels.shape[:2]
|
||||
if margin < 0 or margin >= min(height, width):
|
||||
raise ValueError("crop margin must be nonnegative and smaller than the image")
|
||||
if margin == 0:
|
||||
return pixels.copy()
|
||||
return pixels[: height - margin, : width - margin].copy()
|
||||
|
||||
|
||||
def quantize(pixels: np.ndarray, step: int) -> np.ndarray:
|
||||
"""Round RGB samples to the nearest multiple of STEP."""
|
||||
if step < 2 or step > 64:
|
||||
raise ValueError("quantization step must be between 2 and 64")
|
||||
values = np.rint(pixels.astype(np.float64) / step) * step
|
||||
return np.clip(values, 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def smooth_warp(pixels: np.ndarray, *, amplitude: float, sigma: float, seed: int) -> np.ndarray:
|
||||
"""Apply a deterministic smooth sub-pixel displacement field."""
|
||||
if amplitude < 0.0 or sigma <= 0.0:
|
||||
raise ValueError("warp amplitude must be nonnegative and sigma positive")
|
||||
height, width = pixels.shape[:2]
|
||||
rng = np.random.default_rng(seed)
|
||||
fields: list[np.ndarray] = []
|
||||
for _ in range(2):
|
||||
noise = rng.normal(size=(height, width)).astype(np.float32)
|
||||
field = cv2.GaussianBlur(noise, (0, 0), sigmaX=sigma, sigmaY=sigma)
|
||||
field_std = float(np.std(field))
|
||||
fields.append(np.zeros_like(field) if field_std == 0.0 else field * (amplitude / field_std))
|
||||
yy, xx = np.mgrid[:height, :width].astype(np.float32)
|
||||
return cv2.remap(
|
||||
pixels,
|
||||
xx + fields[0],
|
||||
yy + fields[1],
|
||||
interpolation=cv2.INTER_LANCZOS4,
|
||||
borderMode=cv2.BORDER_REFLECT_101,
|
||||
)
|
||||
|
||||
|
||||
def resize_squeeze(pixels: np.ndarray, factor: float) -> np.ndarray:
|
||||
"""Downsample and restore the original geometry without synthesis."""
|
||||
if not 0.5 <= factor < 1.0:
|
||||
raise ValueError("resize factor must be in [0.5, 1.0)")
|
||||
height, width = pixels.shape[:2]
|
||||
reduced = cv2.resize(
|
||||
pixels,
|
||||
(max(1, round(width * factor)), max(1, round(height * factor))),
|
||||
interpolation=cv2.INTER_AREA,
|
||||
)
|
||||
return cv2.resize(reduced, (width, height), interpolation=cv2.INTER_LANCZOS4)
|
||||
|
||||
|
||||
def jpeg_round_trip(pixels: np.ndarray, quality: int) -> np.ndarray:
|
||||
"""Apply one in-memory JPEG encode/decode while returning RGB pixels."""
|
||||
if quality < 1 or quality > 100:
|
||||
raise ValueError("JPEG quality must be between 1 and 100")
|
||||
success, encoded = cv2.imencode(
|
||||
".jpg",
|
||||
cv2.cvtColor(pixels, cv2.COLOR_RGB2BGR),
|
||||
[cv2.IMWRITE_JPEG_QUALITY, quality],
|
||||
)
|
||||
if not success:
|
||||
raise RuntimeError("JPEG encoding failed")
|
||||
decoded = cv2.imdecode(encoded, cv2.IMREAD_COLOR)
|
||||
if decoded is None:
|
||||
raise RuntimeError("JPEG decoding failed")
|
||||
return cv2.cvtColor(decoded, cv2.COLOR_BGR2RGB)
|
||||
|
||||
|
||||
def norm_matched_noise(reference: np.ndarray, target: np.ndarray, *, seed: int) -> np.ndarray:
|
||||
"""Return random RGB noise with approximately TARGET's residual RMS."""
|
||||
target_residual = target.astype(np.float64) - reference.astype(np.float64)
|
||||
target_rms = float(np.sqrt(np.mean(np.square(target_residual))))
|
||||
rng = np.random.default_rng(seed)
|
||||
noise = rng.normal(size=reference.shape)
|
||||
noise *= target_rms / (float(np.sqrt(np.mean(np.square(noise)))) + 1e-12)
|
||||
return np.clip(np.rint(reference.astype(np.float64) + noise), 0, 255).astype(np.uint8)
|
||||
|
||||
|
||||
def measure(reference: np.ndarray, candidate: np.ndarray, *, name: str, path: Path) -> FidelityMeasurement:
|
||||
"""Measure paired fidelity between equal-shaped RGB arrays."""
|
||||
if reference.shape != candidate.shape:
|
||||
raise ValueError("reference and candidate shapes differ")
|
||||
residual = candidate.astype(np.float64) - reference.astype(np.float64)
|
||||
mse = float(np.mean(np.square(residual)))
|
||||
psnr = math.inf if mse == 0.0 else 20.0 * math.log10(255.0 / math.sqrt(mse))
|
||||
reference_gray = cv2.cvtColor(reference, cv2.COLOR_RGB2GRAY)
|
||||
candidate_gray = cv2.cvtColor(candidate, cv2.COLOR_RGB2GRAY)
|
||||
return FidelityMeasurement(
|
||||
name=name,
|
||||
path=str(path),
|
||||
width=int(reference.shape[1]),
|
||||
height=int(reference.shape[0]),
|
||||
psnr_db=psnr,
|
||||
ssim=float(_ssim(reference_gray, candidate_gray)),
|
||||
changed_pixel_fraction=float(np.mean(np.any(residual != 0.0, axis=2))),
|
||||
residual_rms=float(math.sqrt(mse)),
|
||||
residual_max=float(np.max(np.abs(residual))),
|
||||
)
|
||||
|
||||
|
||||
def build_candidates(source: np.ndarray) -> dict[str, np.ndarray]:
|
||||
"""Build the preregistered attack batch from cropped SOURCE pixels."""
|
||||
candidates: dict[str, np.ndarray] = {
|
||||
"control-crop": source.copy(),
|
||||
"quantize-2": quantize(source, 2),
|
||||
"quantize-4": quantize(source, 4),
|
||||
"quantize-8": quantize(source, 8),
|
||||
"warp-035": smooth_warp(source, amplitude=0.35, sigma=48.0, seed=20260809),
|
||||
}
|
||||
combo = smooth_warp(source, amplitude=0.55, sigma=48.0, seed=20260810)
|
||||
combo = resize_squeeze(combo, 0.96)
|
||||
combo = quantize(combo, 4)
|
||||
combo = jpeg_round_trip(combo, 96)
|
||||
candidates["combo-mild"] = combo
|
||||
candidates["sham-combo-rms"] = norm_matched_noise(source, combo, seed=20260811)
|
||||
return candidates
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.argument("source", type=click.Path(exists=True, dir_okay=False, path_type=Path))
|
||||
@click.argument("output_dir", type=click.Path(file_okay=False, path_type=Path))
|
||||
@click.option("--crop-margin", type=click.IntRange(min=0), default=160, show_default=True)
|
||||
def main(source: Path, output_dir: Path, crop_margin: int) -> None:
|
||||
"""Write a frozen pixel-only attack batch for SOURCE into OUTPUT_DIR."""
|
||||
reference = crop_visible_badge(load_rgb(source), crop_margin)
|
||||
output_dir.mkdir(parents=True, exist_ok=True)
|
||||
measurements: list[FidelityMeasurement] = []
|
||||
for name, pixels in build_candidates(reference).items():
|
||||
path = output_dir / f"{name}.png"
|
||||
Image.fromarray(pixels, mode="RGB").save(path)
|
||||
measurements.append(measure(reference, pixels, name=name, path=path))
|
||||
report_path = output_dir / "fidelity.json"
|
||||
payload = {
|
||||
"source": str(source),
|
||||
"crop_margin": crop_margin,
|
||||
"variants": [asdict(row) for row in measurements],
|
||||
}
|
||||
report_path.write_text(
|
||||
json.dumps(payload, indent=2) + "\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
log.info("Wrote %d candidates and fidelity report: %s", len(measurements), report_path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
main()
|
||||
@@ -0,0 +1,170 @@
|
||||
"""Inventory local SynthID research images without assigning evidence labels.
|
||||
|
||||
The inventory is deliberately weaker than the research manifest. It records
|
||||
artifact hashes, decoded RGB hashes, geometry, format, and exact duplicates,
|
||||
but contains no provider, SynthID outcome, oracle, or split fields. Promotion
|
||||
from inventory to manifest therefore remains an explicit evidence decision.
|
||||
|
||||
Usage:
|
||||
uv run python scripts/synthid_research_inventory.py \
|
||||
--root .local-eval/synthid negatives google openai \
|
||||
--inventory-out .local-eval/synthid/inventory.csv
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import logging
|
||||
from collections import Counter
|
||||
from dataclasses import asdict, dataclass
|
||||
from pathlib import Path
|
||||
|
||||
import click
|
||||
from synthid_research_manifest import artifact_sha256, decoded_image_fingerprint
|
||||
|
||||
log = logging.getLogger(__name__)
|
||||
|
||||
IMAGE_SUFFIXES = {".jpeg", ".jpg", ".png", ".webp"}
|
||||
FIELDNAMES = (
|
||||
"artifact_sha256",
|
||||
"pixel_sha256",
|
||||
"artifact_path",
|
||||
"width",
|
||||
"height",
|
||||
"format",
|
||||
"exact_pixel_group",
|
||||
"artifact_duplicate_of",
|
||||
"pixel_duplicate_of",
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InventoryRow:
|
||||
"""One decoded local image with exact-duplicate provenance."""
|
||||
|
||||
artifact_sha256: str
|
||||
pixel_sha256: str
|
||||
artifact_path: str
|
||||
width: int
|
||||
height: int
|
||||
format: str
|
||||
exact_pixel_group: str
|
||||
artifact_duplicate_of: str
|
||||
pixel_duplicate_of: str
|
||||
|
||||
|
||||
def _inside_root(root: Path, path: Path) -> Path:
|
||||
"""Resolve PATH and reject anything outside ROOT."""
|
||||
resolved_root = root.resolve()
|
||||
resolved = path.resolve()
|
||||
try:
|
||||
resolved.relative_to(resolved_root)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"source is outside inventory root: {path}") from exc
|
||||
return resolved
|
||||
|
||||
|
||||
def discover_images(root: Path, sources: tuple[Path, ...]) -> list[Path]:
|
||||
"""Return supported images below explicit in-root sources in stable order."""
|
||||
resolved_root = root.resolve()
|
||||
discovered: set[Path] = set()
|
||||
for source in sources:
|
||||
candidate = source if source.is_absolute() else resolved_root / source
|
||||
candidate = _inside_root(resolved_root, candidate)
|
||||
if not candidate.exists():
|
||||
raise ValueError(f"inventory source does not exist: {source}")
|
||||
if candidate.is_file():
|
||||
if candidate.suffix.lower() in IMAGE_SUFFIXES:
|
||||
discovered.add(candidate)
|
||||
continue
|
||||
for path in candidate.rglob("*"):
|
||||
if path.is_file() and path.suffix.lower() in IMAGE_SUFFIXES:
|
||||
discovered.add(_inside_root(resolved_root, path))
|
||||
return sorted(discovered, key=lambda path: path.relative_to(resolved_root).as_posix())
|
||||
|
||||
|
||||
def build_inventory(root: Path, sources: tuple[Path, ...]) -> list[InventoryRow]:
|
||||
"""Hash and decode all selected images without inferring any labels."""
|
||||
resolved_root = root.resolve()
|
||||
first_artifact: dict[str, str] = {}
|
||||
first_pixels: dict[str, str] = {}
|
||||
rows: list[InventoryRow] = []
|
||||
for path in discover_images(resolved_root, sources):
|
||||
relative = path.relative_to(resolved_root).as_posix()
|
||||
artifact_digest = artifact_sha256(path)
|
||||
pixel_digest, width, height, image_format = decoded_image_fingerprint(path)
|
||||
rows.append(
|
||||
InventoryRow(
|
||||
artifact_sha256=artifact_digest,
|
||||
pixel_sha256=pixel_digest,
|
||||
artifact_path=relative,
|
||||
width=width,
|
||||
height=height,
|
||||
format=image_format,
|
||||
exact_pixel_group=f"pixel-{pixel_digest[:16]}",
|
||||
artifact_duplicate_of=first_artifact.get(artifact_digest, ""),
|
||||
pixel_duplicate_of=first_pixels.get(pixel_digest, ""),
|
||||
)
|
||||
)
|
||||
first_artifact.setdefault(artifact_digest, relative)
|
||||
first_pixels.setdefault(pixel_digest, relative)
|
||||
return rows
|
||||
|
||||
|
||||
def write_inventory(path: Path, rows: list[InventoryRow], *, replace: bool = False) -> None:
|
||||
"""Write a complete inventory atomically enough to avoid partial decode results."""
|
||||
if path.exists() and not replace:
|
||||
raise FileExistsError(f"inventory already exists: {path}; pass --replace to overwrite it")
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_name(f".{path.name}.tmp")
|
||||
try:
|
||||
with temporary.open("w", newline="", encoding="utf-8") as stream:
|
||||
writer = csv.DictWriter(stream, fieldnames=FIELDNAMES)
|
||||
writer.writeheader()
|
||||
writer.writerows(asdict(row) for row in rows)
|
||||
temporary.replace(path)
|
||||
finally:
|
||||
if temporary.exists():
|
||||
temporary.unlink()
|
||||
|
||||
|
||||
def inventory_summary(rows: list[InventoryRow]) -> dict[str, object]:
|
||||
"""Return aggregate coverage without exposing media paths."""
|
||||
formats = Counter(row.format for row in rows)
|
||||
geometries = Counter(f"{row.width}x{row.height}" for row in rows)
|
||||
return {
|
||||
"images": len(rows),
|
||||
"unique_artifacts": len({row.artifact_sha256 for row in rows}),
|
||||
"unique_pixels": len({row.pixel_sha256 for row in rows}),
|
||||
"formats": dict(sorted(formats.items())),
|
||||
"geometries": dict(sorted(geometries.items())),
|
||||
}
|
||||
|
||||
|
||||
@click.command()
|
||||
@click.option("--root", required=True, type=click.Path(exists=True, file_okay=False, path_type=Path))
|
||||
@click.argument("sources", nargs=-1, required=True, type=click.Path(path_type=Path))
|
||||
@click.option("--inventory-out", required=True, type=click.Path(dir_okay=False, path_type=Path))
|
||||
@click.option("--replace", is_flag=True, help="Replace an existing generated inventory.")
|
||||
def main(root: Path, sources: tuple[Path, ...], inventory_out: Path, replace: bool) -> None:
|
||||
"""Inventory image SOURCES below ROOT without assigning SynthID labels."""
|
||||
logging.basicConfig(level=logging.INFO, format="%(message)s")
|
||||
try:
|
||||
rows = build_inventory(root, sources)
|
||||
if not rows:
|
||||
raise ValueError("selected sources contain no supported images")
|
||||
write_inventory(inventory_out, rows, replace=replace)
|
||||
except (OSError, ValueError) as exc:
|
||||
raise click.ClickException(str(exc)) from exc
|
||||
summary = inventory_summary(rows)
|
||||
log.info(
|
||||
"Wrote inventory: %s images=%s unique_artifacts=%s unique_pixels=%s",
|
||||
inventory_out,
|
||||
summary["images"],
|
||||
summary["unique_artifacts"],
|
||||
summary["unique_pixels"],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||