remove-ai-watermarks/docs/provider-oracles.md

Provider oracle tooling

scripts/provider_oracles.py is the development-only entry point for the four remote watermark-oracle families used by this project. It drives Google through the user's real authenticated Chrome and drives the other public Web surfaces through isolated Playwright. It keeps provider scope, execution identity, immutable media, and verbatim results in one format without exposing any oracle through the installed CLI or public Python API.

Surfaces

Surface key	Provider signal	Access	Project evidence
openai-api	OpenAI SynthID	Official Content Provenance API	Images
microsoft-api	Microsoft InvisMark	Azure Content Provenance Detection API	Images
gemini-web	Google SynthID	Signed-in Gemini web account	Images and videos
meta-web	Meta Content Seal	Anonymous Meta identification page	Images
microsoft-web	Microsoft InvisMark	Public Azure AI validation page	Images
openai-web	OpenAI SynthID	Public web verifier	Images

Surface priority

The default selection order is API first, then Web. OpenAI therefore plans openai-api before openai-web, and Microsoft plans microsoft-api before microsoft-web. Google and Meta currently have no usable API adapter in this tooling, so their Web surfaces are primary rather than fallback routes.

Inspect the machine-readable order before preparing a batch:

uv run python scripts/provider_oracles.py plan openai --json
uv run python scripts/provider_oracles.py plan microsoft

API-first is a selection policy, not automatic failover. Use Web only when the provider, requested media, or local configuration has no usable API route, or when the operator explicitly requests a second surface. An API refusal, quota response, transport failure, or indeterminate verdict is recorded as-is and never silently triggers a Web upload.

The old Google SynthID Detector portal does not work. On 2026-09-07, the official labs.google/synthid entry point redirected a signed-in browser to a Google 404. Historical corpus rows may still name synthid-portal; that value describes how an old result was obtained and is not an operational surface. Google's newer Search verification entry point did not provide an independent oracle allowance in the session tested on 2026-09-13. Lens accepted the metadata-stripped, previously Gemini-positive image sha256:c86feaf788df2c423eeb53fcaee52b74f12a8e83829f94414e6dec2966e6e62e, but ordinary Lens results only described and matched the visible content. An explicit Is this made with AI? Check whether it contains SynthID. request in Search AI Mode invoked the technical verifier and returned The verification tool's daily limit has been exceeded. after that session's Gemini image verification allowance was already exhausted. Record this as a quota refusal, not a watermark verdict and not a second quota pool.

A second Search check on 2026-09-15 used a real Qwen 0.15 no-face/no-text variant rather than a control. Lens and Search AI Mode accepted the upload, but both answered from visible-content reasoning and Web matches instead of returning a SynthID result. The exact Google-AI verification question even misattributed the file to an Imagen 4 Fast demonstration and invented a matching prompt. Search is therefore neither a usable independent oracle nor a substitute verdict when its response does not explicitly report the verification tool's SynthID outcome.

Fresh independent-session rechecks on 2026-09-14 split: one returned detected for the same control while another returned unable to process your request due to a tool error or quota limit. A later session returned five usable real-variant verdicts and refused the sixth, while the other tested sessions still refused real variants. Recovery is therefore not a global calendar event. Google documents approximately ten image checks in a rolling 24-hour window, but that approximation is orientation rather than a per-session reset schedule. When a campaign is conserving that allowance, probe each explicitly selected session with its next real pending variant and record a refusal without assigning a watermark verdict. Reserve a positive-control probe for surface diagnosis, not ordinary batch allocation.

A second independent-session sweep used one real pending variant per session and again obtained no watermark verdict: three responses explicitly reported a temporary verification quota, while one reported a verification tool-call error. Four completed immutable result batches preserve the exact uploads and settled responses. Twenty-four newly generated no-face/no-text scene variants form six four-rung ladders; the six lowest rungs were prepared in immutable batches, and none had been submitted at that point. The later completed sweep followed the planned lowest-rung-first escalation. Do not treat batch preparation or visual quality scores as SynthID results.

A session recovered earlier than the estimate derived from the oldest usable call preserved locally. Its next real pending variant, Chroma 0.16 on the face carrier, returned detected; the immutable batch is complete and hash-verified. This demonstrates why estimated rolling-window times are retry guidance rather than reset timestamps, and closes the tested face bracket at 0.16 detected -> 0.22 not_detected without spending a duplicate request on the already settled 0.22 file.

The same recovered session returned not_detected for Chroma 0.40 on the Cyrillic carrier, then explicitly quota-refused SDXL CJK 0.35. The other selected sessions were checked with real pending CJK variants, not controls, and returned quota or tool limitations. One settled response exposed a request-specific instruction to wait 18 hours. Treat that as the retry point for that session and request class, not as a replacement for Google's documented approximate rolling 24-hour allowance or a universal reset time. All four sessions were unavailable again at their latest checks. Because one response did not distinguish service failure from quota, its real Chroma CJK 0.40 candidate was retried. This time the verifier explicitly named a temporary image-analysis quota. The refusal was recorded and the same bytes were prepared in a new immutable retry batch; no watermark verdict was inferred. One session was probed once more with a real Qwen 0.15 no-face/no-text candidate. It returned a verification-tool technical error rather than a SynthID result. That failure was recorded as refused, the batch was hash-verified complete, and a fresh immutable retry batch was prepared for the evidence-based session window.

All four sessions returned usable results again on 2026-09-15, earlier than the previous retry guidance. The remaining real variants were submitted adaptively without controls. SDXL CJK remained detected at 0.35, 0.40, and 0.45 and cleared at 0.50. Chroma CJK remained detected at 0.40 and cleared at 0.50. The two no-face/no-text carriers cleared Qwen at 0.15; Chroma cleared the flat carrier at 0.12 and the photographic carrier at 0.30 after detections at 0.12 and 0.20; SDXL cleared the photographic carrier at 0.15 and the quality-qualified flat carrier at 0.35. One quota refusal and one tool-call failure encountered while routing SDXL CJK 0.40 were kept as separate completed batches; another session then returned the usable detected result for the same upload hash. Every settled batch passed immutable-source, upload, manifest, and result verification. The observed early recovery reinforces that a retry time is guidance for the next real request, not a promised reset.

A same-day production-boundary replication then used three new independent Gemini-generated controls: busy CJK plus many faces, flat CJK without faces, and a portrait plus dense CJK. Only processed variants were submitted. Across Qwen 0.35, Chroma 0.50, and SDXL 0.50, all nine profile-by-carrier cells returned not_detected. The night-market lower rungs Chroma 0.40 and SDXL 0.45 also returned not_detected; they do not supersede the earlier harder-source detections at those rungs. Two sessions returned explicit quota failures during this pass, one candidate retry also hit quota, and a byte-normalized duplicate after the usable SDXL result returned a tool-call error. Those four calls were recorded as unreachable, not spliced into the nine usable verdicts. All result batches were hash-verified. Google did not expose a reset timestamp: the returned quota text supports only waiting and retrying, not a universal 24-hour claim.

The documented Vertex AI Media Studio verification flow was also unavailable in the live console on 2026-09-14. Its old console URL redirected to Agent Media Studio after the Vertex-to-Agent Platform rename. The current Image playground was inspected in eligible configurations; its task menu exposed generation and image editing but no watermark upload or validation action. The deprecated Python SDK can still resolve imageverification@001 to its publisher-model endpoint. A 2026-09-15 request reached predict, but the real Qwen 0.15 variant returned HTTP 403 The caller does not have permission in both tested configurations. The current Python reference still documents WatermarkVerificationModel.verify_image, but the backing model was access-restricted in both tested configurations and the SDK path is past its announced removal date. Recheck the live console and a real pending variant before adding Search or Agent Media Studio to the surface table. Adobe Inspect and local C2PA, DWT-DCT, TrustMark, and research detectors are useful evidence readers, not interchangeable provider pixel-watermark oracles.

The surface capability and this project's evidence coverage are separate. The current Google, Microsoft, and Meta pages accept image, video, and audio inputs; the OpenAI API accepts image and audio. A media type does not become certified merely because a surface accepts it. Run list --json to read both fields.

Execution slots

Keep account, browser-profile, and network labels in the gitignored repository-local .oracle-slots.json. Labels and proxy environment-variable names are provenance, not credentials. Do not commit this file or put cookies, tokens, passwords, proxy URLs, or account secrets in it. When --slot is present, the CLI reads this path by default; --slots PATH explicitly overrides it.

{
  "format_version": 1,
  "slots": [
    {
      "name": "gemini-personal",
      "surface": "gemini-web",
      "account_label": "google-account-a",
      "browser_profile": "Your Chrome",
      "google_account_index": 0,
      "network_label": "home"
    },
    {
      "name": "gemini-research",
      "surface": "gemini-web",
      "account_label": "google-account-b",
      "browser_profile": "Your Chrome",
      "google_account_index": 2,
      "network_label": "office"
    },
    {
      "name": "openai-web-thordata-us",
      "surface": "openai-web",
      "account_label": null,
      "browser_profile": null,
      "network_label": "thordata-us-residential",
      "proxy_url_env": "THORDATA_ROUTE_US"
    },
    {
      "name": "microsoft-web-thordata-de",
      "surface": "microsoft-web",
      "account_label": null,
      "browser_profile": null,
      "network_label": "thordata-de-residential",
      "proxy_url_env": "THORDATA_ROUTE_DE"
    },
    {
      "name": "meta-web-thordata-us",
      "surface": "meta-web",
      "account_label": null,
      "browser_profile": null,
      "network_label": "thordata-us-residential",
      "proxy_url_env": "THORDATA_ROUTE_US"
    },
    {
      "name": "openai-web-thordata-de",
      "surface": "openai-web",
      "account_label": null,
      "browser_profile": null,
      "network_label": "thordata-de-residential",
      "proxy_url_env": "THORDATA_ROUTE_DE"
    },
    {
      "name": "openai-api-1",
      "surface": "openai-api",
      "account_label": "project-1",
      "browser_profile": null,
      "network_label": "home",
      "api_key_env": "OPENAI_API_KEY_1"
    },
    {
      "name": "openai-api-2",
      "surface": "openai-api",
      "account_label": "project-2",
      "browser_profile": null,
      "network_label": "home",
      "api_key_env": "OPENAI_API_KEY_2"
    },
    {
      "name": "openai-api-3",
      "surface": "openai-api",
      "account_label": "project-3",
      "browser_profile": null,
      "network_label": "home",
      "api_key_env": "OPENAI_API_KEY_3"
    },
    {
      "name": "microsoft-api-1",
      "surface": "microsoft-api",
      "account_label": "content-safety-project",
      "browser_profile": null,
      "network_label": "home"
    }
  ]
}

A routed batch selects one slot explicitly. A Gemini slot must name the Google account, its real Chrome profile, and its nonnegative /u/N/ account index; an API slot must name its account or project and cannot name a browser profile. The tool records the network route that the operator has already established. Non-Google Web slots may name proxy_url_env; Playwright reads the secret proxy URL from that environment variable without writing or logging it. The tool does not rotate accounts or networks, retry through another IP after throttling, or merge sessions. Record a throttled response as refused; choosing another authorized slot is a separate operator decision and a separate batch. This preserves per-account and per-IP history without turning provider limits into an automatic evasion mechanism.

Each openai-api slot must name exactly one api_key_env. Put the corresponding values in the gitignored local .env, never in the slot JSON:

OPENAI_API_KEY_1=replace-with-first-key
OPENAI_API_KEY_2=replace-with-second-key
OPENAI_API_KEY_3=replace-with-third-key
THORDATA_ROUTE_US=http://td-customer-USERNAME-country-US:PASSWORD@HOST.pr.thordata.net:9999
THORDATA_ROUTE_DE=http://td-customer-USERNAME-country-DE:PASSWORD@HOST.pr.thordata.net:9999
THORDATA_ROUTE_GB=http://td-customer-USERNAME-country-GB:PASSWORD@HOST.pr.thordata.net:9999
THORDATA_ROUTE_NL=http://td-customer-USERNAME-country-NL:PASSWORD@HOST.pr.thordata.net:9999
THORDATA_ROUTE_JP=http://td-customer-USERNAME-country-JP:PASSWORD@HOST.pr.thordata.net:9999

The tool reads only the variable selected by the requested slot. A nonempty process environment value overrides the matching .env entry. It never prints the value or falls through to another token after authentication, entitlement, quota, or transport failure. The no-slot compatibility path reads OPENAI_API_KEY.

Several Google accounts may share one real Chrome profile: give them distinct slot names and google_account_index values while keeping browser_profile the same.

The ThorData examples use its residential proxy country parameter. The proxy username encodes one explicit two-letter ISO country code. Route selection remains explicit and never changes after a refusal. OpenAI, Microsoft, and Meta Web slots may reuse any route variable; the surface remains part of each distinct slot. OpenAI API slots never accept proxy_url_env and use the current direct network.

Validate and inspect the slots:

uv run python scripts/provider_oracles.py slots .oracle-slots.json
uv run python scripts/provider_oracles.py guide gemini-web --slot gemini-personal

Immutable manual batches

Prepare a batch outside the repository:

uv run python scripts/provider_oracles.py prepare gemini-web input.png \
  --output /tmp/gemini-check-001 \
  --slot gemini-personal

For PNG, JPEG, and WebP, preparation strips AI provenance metadata and proves that the decoded RGBA pixels and format did not change. This prevents a C2PA result from masquerading as a pixel-watermark result. Video and audio are copied byte for byte; the manifest explicitly records that their metadata was not stripped. Use the matched-control protocols in synthid.md for video claims.

The batch contains:

• manifest.json, immutable surface, slot, source, upload, and hash identities;
• manifest.sha256, the manifest digest;
• uploads/, sanitized images or byte-identical non-image media;
• results.json, the only mutable document.

Submit each upload in index order, one file per fresh provider result. Preserve the settled response verbatim, including an unclear or refused response. Record watermark and provenance separately:

uv run python scripts/provider_oracles.py record /tmp/gemini-check-001/manifest.json \
  --artifact-id 000-0123456789abcdef \
  --watermark-result detected \
  --provenance-result absent \
  --raw-response "SynthID watermark detected" \
  --checked-at 2026-09-07T18:30:00Z

uv run python scripts/provider_oracles.py verify \
  /tmp/gemini-check-001/manifest.json --require-complete

The verifier rereads every source and upload hash, binds result rows to exact upload bytes, checks timezone-aware timestamps, and never interprets indeterminate or refused as not_detected.

Web automation

Install Playwright's isolated Chromium once after the development dependencies:

uv sync --extra dev
uv run playwright install chromium

Google is intentionally excluded from the standalone runner because its oracle requires the user's existing authenticated Chrome session. Invoke the project provider-oracles skill to prepare the batch, select the named real Chrome profile and /u/N/ Google account, upload each prepared artifact, capture the settled SynthID tool outcome, and record it.

OpenAI Web, Microsoft Web, and Meta run entirely through a fresh isolated Playwright browser. A direct-route example:

uv run python scripts/provider_oracles.py prepare meta-web input.png \
  --output /tmp/meta-check-001 \
  --slot meta-direct

uv run python scripts/provider_oracles.py run-web \
  /tmp/meta-check-001/manifest.json --acknowledge-uploads

For a ThorData-routed OpenAI, Microsoft, or Meta Web slot, keep the generated proxy URLs in the gitignored .env, then select one route explicitly:

uv run python scripts/provider_oracles.py run-web \
  /tmp/openai-check-001/manifest.json --acknowledge-uploads \
  --env-file .env

The runner reuses one context and one route for the batch, performs a fresh page navigation before each upload, does not retry, and stores the settled page text. Each completed verdict is persisted immediately with its own observation timestamp. A later navigation or upload failure preserves those completed rows and leaves unobserved rows unresolved. ThorData user-and-password endpoints without a sessid rotate the exit on each request. That is unsuitable inside one browser context: page resources can arrive through different exits and trigger the provider's bot checks. For a ThorData hostname, the runner derives a non-secret sticky session id from the immutable manifest hash and appends a 90-minute session window when the configured username has no explicit session. The same batch therefore keeps one route while the provider retains it, a new prepared batch gets a distinct route, and an operator-supplied sessid remains unchanged. This pins a selected route; it does not retry or fail over after a refusal. For a selected proxy route it enables certificate handling in that isolated Chromium launch and context; direct launches keep Playwright's normal TLS verification. A 2026-09-07 preflight sent one read-only request through each configured route and observed five distinct egress addresses with the requested US, DE, GB, NL, and JP country codes. A deliberately invalid country code failed instead of silently using a random route. The addresses are intentionally not recorded because residential egress is ephemeral; this proves route selection, not reachability of any provider page. One observation per surface is not a reliability baseline. It is headed by default: a live 2026-09-07 probe reached OpenAI's upload control that way while the same isolated Chromium received a Cloudflare gate in headless mode. Use --headless only for a surface already verified in that mode. Bot or quota refusal is refused, a transport failure is unreachable, and unknown or changed page wording is indeterminate.

Each provider mounts its file input before the upload handler is necessarily ready. Microsoft and Meta reach a usable handler at network idle. OpenAI keeps background requests alive and never reaches network idle, so its observable readiness seam is the page load state instead. The adapter waits for the surface-specific state before assigning the file. OpenAI's static FAQ contains the lowercase phrase generated with OpenAI tools; only the exact title-cased result label may settle a positive result. Its generic Something went wrong response settles as indeterminate, not clean. Meta's live positive wording changed from AI signatures from Meta were found to AI signatures were found by 2026-09-07; both exact forms remain accepted.

Isolation is a transport property, not a verdict requirement. On 2026-09-15, the isolated OpenAI runner returned one generic error and two Cloudflare refusals for a prepared three-file batch. Uploading those exact prepared hashes through an authenticated first-party browser session reached the official verifier and returned explicit SynthID results. A later Chroma 0.12 upload remained in Verifying for more than 90 seconds and was recorded as indeterminate rather than clean. The official API then returned HTTP 200 and SynthID not_detected for the same processed source hash. Keep each route in its own immutable batch; do not splice an isolated-runner failure and a first-party browser or API verdict into one response.

On 2026-09-16, a rotating ThorData browser batch demonstrated the failure mode: one upload reached OpenAI's generic error, four encountered Cloudflare, and ten became unreachable after the browser context closed. Five fresh connections to the same rotating endpoint had five distinct exits, while five connections with one sticky session had one exit. After session pinning, one direct route and five independently prepared country routes all reached the upload handler with the same prepared test image, but every route returned OpenAI's exact generic error. Record those responses as indeterminate. The result distinguishes the fixed session-continuity defect from the remaining anonymous-Web availability problem; further route changes are not a substitute for an API verdict.

OpenAI API

The one automated path calls the hardened metadata-independent adapter already covered by tests/test_openai_provenance.py:

uv run python scripts/provider_oracles.py check-openai input.png \
  --acknowledge-upload \
  --slot openai-api-1 \
  --env-file .env

The command accepts one image and one explicit acknowledgement, makes at most one request with SDK retries disabled, and emits the selected slot beside the result. Its SDK transport sets trust_env=False, so HTTP_PROXY, HTTPS_PROXY, and related process settings cannot silently proxy an API slot. On 2026-09-07, a no-upload models.list() authentication probe succeeded for all three configured keys over that direct transport. The model counts are not recorded because they are operationally volatile. The official response keeps SynthID separate from C2PA. The current adapter deliberately handles images only even though the current API surface also accepts audio.

OpenAI documents not_detected as absence of a supported signal, not proof that content was not AI-generated. The API must not become an adaptive removal loss or automatic search loop. See synthid.md for the oracle boundary.

The fresh no-face/no-text calibration on 2026-09-15 used three independent official API requests after the public Web surface stalled. All three returned HTTP 200 with C2PA not_present: Chroma 0.12 and SDXL 0.06 were SynthID not_detected, while SDXL 0.04 was detected. Together with settled Web results for the same carrier, the tested 0.02-grid brackets are Qwen 0.04 detected -> 0.06 not_detected, Chroma 0.10 detected -> 0.12 not_detected, and SDXL 0.04 detected -> 0.06 not_detected. The generated source control was not uploaded.

A separate face-plus-multilingual-text carrier then returned Qwen 0.04 detected -> 0.06 not_detected, Chroma 0.10 not_detected, and SDXL 0.04 not_detected. The lower Chroma and SDXL results did not change the global candidate because the no-face/no-text carrier remained the limiting OpenAI stratum.

Microsoft API

The direct Microsoft path uses the Azure Content Provenance Detection preview API. Run az login first, then keep the non-secret resource configuration in the gitignored local .env. The script reads both resource keys into memory through Azure CLI and never writes or logs them:

AZURE_CONTENT_SAFETY_ENDPOINT=https://your-resource.cognitiveservices.azure.com/
AZURE_CONTENT_SAFETY_SUBSCRIPTION_ID=your-subscription-id
AZURE_CONTENT_SAFETY_RESOURCE_GROUP=your-content-safety-resource-group
AZURE_CONTENT_SAFETY_ACCOUNT_NAME=your-content-safety-account
AZURE_ORACLE_STORAGE_RESOURCE_GROUP=your-storage-resource-group

The live preview API accepts only an Azure Blob Storage HTTPS endpoint, despite the how-to page also describing a general public URL. Give the Content Safety resource's system-assigned identity Storage Blob Data Reader on the private input storage account. The command downloads that private blob through Azure CLI and refuses to submit unless its SHA-256 equals the named local file:

uv run python scripts/provider_oracles.py check-microsoft input.png \
  --content-uri https://storage-account.blob.core.windows.net/oracle-inputs/input.png \
  --acknowledge-upload \
  --slot microsoft-api-1 \
  --env-file .env

The adapter sends one detection request with no submit retry, polls only that operation, ignores process proxy settings, and records Watermark separately from C2PA. The current adapter accepts JPEG, PNG, and WebP up to the provider's 100 MiB limit. See Microsoft's Content Provenance workflow.

Live cross-provider smoke

On 2026-09-08, the public data/contentseal/originals/gen_fox_forest.webp fixture, or a metadata-stripped pixel-identical copy of it, was submitted once per configured account or surface. These are transport and parser smokes, not error rate estimates:

Surface	Route and repetitions	Recorded result
Gemini Web	Official browser verifier, three independent authenticated sessions	3/3 not_detected; Gemini reported no reliable Google AI signal.
OpenAI API	Official API, three independent requests	3/3 SynthID not_detected; C2PA not_present; all responses were HTTP 200.
OpenAI Web	ThorData GB, isolated Playwright	indeterminate; after accepting the prepared upload, the provider returned Something went wrong. Please try again.
Microsoft API	Direct API, private hash-matched Azure blob	HTTP 202, operation Succeeded; Watermark not_detected, C2PA absent, provider outcome NoProvenanceDetected.
Microsoft Web	ThorData US, isolated Playwright	not_detected; the page's exact settled label was Inconclusive.
Meta Web	ThorData DE, isolated Playwright	detected; attribution Muse Image 1 - Meta and the existing fixture generation ID were preserved.

The same setup verified distinct US, DE, GB, NL, and JP egress addresses whose reported countries matched their requested routes. A deliberately invalid country code failed rather than falling back to a random location. The Google 404 and the OpenAI Web error remain availability findings; neither is evidence that a watermark is absent.