Compare commits
No files matched your search
@@ -6,7 +6,7 @@
|
|||||||
},
|
},
|
||||||
"metadata": {
|
"metadata": {
|
||||||
"description": "Agent skill and Claude Code plugin for the remove-ai-watermarks CLI.",
|
"description": "Agent skill and Claude Code plugin for the remove-ai-watermarks CLI.",
|
||||||
"version": "1.0.10"
|
"version": "1.0.13"
|
||||||
},
|
},
|
||||||
"plugins": [
|
"plugins": [
|
||||||
{
|
{
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ description: Command contracts, project gate, typing boundaries, model-adjacent
|
|||||||
|
|
||||||
Every single-image command declares `source` with `dir_okay=False`; `batch` declares its directory with `file_okay=False`. Keep `tests/test_cli_robustness.py::TestDirectoryInputIsRejected` as the regression guard.
|
Every single-image command declares `source` with `dir_okay=False`; `batch` declares its directory with `file_okay=False`. Keep `tests/test_cli_robustness.py::TestDirectoryInputIsRejected` as the regression guard.
|
||||||
|
|
||||||
Exit-code and no-signal behavior is a public contract. Read the command-line section of [`../../docs/module-internals.md`](../../docs/module-internals.md) before changing it. Keep [`../../skills/remove-ai-watermarks/`](../../skills/remove-ai-watermarks/) in the same change when command routing, extras, exit codes, mark keys, or the intended-use boundary move.
|
Exit-code and no-signal behavior is a public contract. Read the command-line section of [`../../docs/module-internals.md`](../../docs/module-internals.md) before changing it. The dev `.venv` always carries the pixel stack, so it cannot show what the default install does. To check a no-signal or fail-closed change there, build a throwaway venv outside the repository (`uv venv /tmp/bare && VIRTUAL_ENV=/tmp/bare uv pip install .`) and run `/tmp/bare/bin/remove-ai-watermarks` against it. Keep [`../../skills/remove-ai-watermarks/`](../../skills/remove-ai-watermarks/) in the same change when command routing, extras, exit codes, mark keys, or the intended-use boundary move.
|
||||||
|
|
||||||
`tests/test_agent_skill.py` enforces that by comparing the skill's prose against the live Click tree. Guard the SET, not the instances: per-flag comparisons only check what the skill already documents, so `--vendor`, `--pipeline auto` and then the whole `classify` and `verify-openai-synthid` commands each shipped with a green suite. `test_every_cli_command_is_named_somewhere_in_the_skill` closes that; raise `MIN_CLI_VERSION` in the skill's probe whenever a reference names a command, flag or value the previous release lacked, or the probe tells an agent its CLI is current right before it types something that build rejects.
|
`tests/test_agent_skill.py` enforces that by comparing the skill's prose against the live Click tree. Guard the SET, not the instances: per-flag comparisons only check what the skill already documents, so `--vendor`, `--pipeline auto` and then the whole `classify` and `verify-openai-synthid` commands each shipped with a green suite. `test_every_cli_command_is_named_somewhere_in_the_skill` closes that; raise `MIN_CLI_VERSION` in the skill's probe whenever a reference names a command, flag or value the previous release lacked, or the probe tells an agent its CLI is current right before it types something that build rejects.
|
||||||
|
|
||||||
@@ -29,9 +29,15 @@ The unified maintainer entry point is `scripts/provider_oracles.py`; its workflo
|
|||||||
slot schema live in `docs/provider-oracles.md`. Multi-account and multi-network work
|
slot schema live in `docs/provider-oracles.md`. Multi-account and multi-network work
|
||||||
selects each named slot explicitly from the gitignored repository-local
|
selects each named slot explicitly from the gitignored repository-local
|
||||||
`.oracle-slots.json` by default and snapshots it into a separate immutable batch
|
`.oracle-slots.json` by default and snapshots it into a separate immutable batch
|
||||||
manifest. A bounded provider check may continue through its finite set of authorized
|
manifest. A provider check walks that provider's existing order: the official API
|
||||||
account slots after a refusal; do not add new logins, IP rotation, network-route
|
surface, then its Web oracle, then the remaining configured account slots.
|
||||||
failover, unbounded retries, or cross-session result merging. Never commit the local
|
After a refusal, quota response, transport error, or indeterminate verdict,
|
||||||
|
record the response and continue with the next item as its own command. Do not
|
||||||
|
ask for confirmation before the switch. Anyone doing the task may run the
|
||||||
|
configured checks. Do not add new logins, IP rotation, or unbounded retries.
|
||||||
|
Do not walk a prepared route list or rotate addresses inside one call; the next
|
||||||
|
run is a new command and picks a new session. Do not merge results from
|
||||||
|
different sessions into one response. Never commit the local
|
||||||
slot file.
|
slot file.
|
||||||
Every manual result stays bound to the exact sanitized upload hash and preserves the
|
Every manual result stays bound to the exact sanitized upload hash and preserves the
|
||||||
provider response verbatim.
|
provider response verbatim.
|
||||||
@@ -39,35 +45,49 @@ provider response verbatim.
|
|||||||
Provider surface selection is API-first. OpenAI and Microsoft use their official API
|
Provider surface selection is API-first. OpenAI and Microsoft use their official API
|
||||||
adapters before their Web surfaces when the requested media and local configuration are
|
adapters before their Web surfaces when the requested media and local configuration are
|
||||||
supported. Google and Meta currently use Web because this tooling has no usable API route
|
supported. Google and Meta currently use Web because this tooling has no usable API route
|
||||||
for them. Web remains an explicit second surface, never an automatic retry after an API
|
for them. Google's Web surfaces are `gemini-web` and `synthid-web`
|
||||||
refusal, quota response, transport error, or indeterminate verdict.
|
(`https://synthid.com/`). After the current surface is exhausted, record the
|
||||||
|
response and continue with the next surface as its own command. One call does
|
||||||
|
not upload the same file to the next surface, and the verdicts stay separate.
|
||||||
|
|
||||||
Google Web oracle automation uses only the user's existing authenticated real Chrome
|
A signed-in account is required only for `gemini-web`. That surface uses the user's
|
||||||
session and explicit `/u/N/` account index. OpenAI, Microsoft, and Meta Web automation
|
existing authenticated real Chrome session and an explicit `/u/N/` account index.
|
||||||
uses isolated Playwright contexts; it must never attach to or copy state from the user's
|
Every other Web surface, including `synthid-web`, uses an isolated Playwright
|
||||||
real browser. Proxy credentials are read only through a slot's named environment variable
|
context and may name a proxy. Those contexts must never attach to or copy state
|
||||||
|
from the user's real browser. `synthid-web` has no `/u/N/` index. Its terms forbid
|
||||||
|
bulk verification and quota evasion, so one run submits the prepared rows once
|
||||||
|
and does not start a sign-in. If the page says `Please sign in before detection.`
|
||||||
|
and returns no watermark label, keep that row and continue the same prepared
|
||||||
|
upload in the user's existing real Chrome. That Chrome session stays on the
|
||||||
|
direct route: no proxy, no new login, and no cookie copy into Playwright. If
|
||||||
|
that Chrome also asks for sign-in, leave its row unsubmitted. API calls
|
||||||
|
stay on the direct SDK transport and do not take `proxy_url_env`. Proxy credentials are read only through a slot's named environment variable
|
||||||
and never enter a manifest or log. Dotenv interpolation may derive routed proxy URLs from
|
and never enter a manifest or log. Dotenv interpolation may derive routed proxy URLs from
|
||||||
one secret key without mutating the process environment. A selected proxy context may
|
one secret key without mutating the process environment. A selected proxy context may
|
||||||
accept that proxy's TLS interception certificate; direct contexts keep normal TLS checks.
|
accept that proxy's TLS interception certificate; direct contexts keep normal TLS checks.
|
||||||
ThorData rotating endpoints are pinned to one manifest-derived sticky session for the
|
A ThorData exit is not prepared ahead of time. Each unauthenticated Web run picks
|
||||||
life of a Web batch so one browser context does not change exit identity between page
|
a new session and a country in Europe or the United States, and keeps that one exit
|
||||||
requests. This is route continuity, not account or IP failover; a refusal still ends on
|
for the browser context so page requests do not change identity. A country or
|
||||||
the selected slot and never selects another route automatically.
|
`sessid` already written into the proxy URL is discarded. The next run picks again.
|
||||||
Upload readiness is surface-specific: OpenAI reaches it at page load but keeps background
|
Upload readiness is surface-specific: OpenAI reaches it at page load but keeps background
|
||||||
requests alive, while Microsoft and Meta require network idle. Preserve exact provider
|
requests alive, while Microsoft, Meta, and SynthID Detector require network idle. Preserve exact provider
|
||||||
errors as indeterminate evidence rather than retrying or interpreting them as clean.
|
errors as indeterminate evidence, then continue with the next configured route or surface.
|
||||||
|
|
||||||
OpenAI API slots likewise carry only `api_key_env`. Store multiple key values in the
|
OpenAI API slots likewise carry only `api_key_env`. Store multiple key values in the
|
||||||
gitignored `.env`, select one slot explicitly, and never rotate to another token after a
|
gitignored `.env` and select one slot explicitly. The process environment may override
|
||||||
refusal or rate limit. The process environment may override the matching `.env` value.
|
the matching `.env` value. Their SDK transport sets `trust_env=False`, so ambient proxy
|
||||||
Their SDK transport sets `trust_env=False`; environment proxy variables must not reroute
|
variables do not reroute an API check, and the slot does not take `proxy_url_env`.
|
||||||
an API check.
|
After a refusal or rate limit, continue with the next configured slot in a new call,
|
||||||
|
then with the Web surface.
|
||||||
|
|
||||||
Microsoft API slots use the Azure Content Safety and private Blob Storage resource
|
Microsoft API slots use the Azure Content Safety and private Blob Storage resource
|
||||||
identities from the gitignored `.env`. The adapter reads resource keys only into memory
|
identities from the gitignored `.env`. The adapter reads resource keys only into memory
|
||||||
through authenticated Azure CLI, downloads and hashes the private blob before requiring
|
through authenticated Azure CLI, downloads and hashes the private blob before requiring
|
||||||
equality with the named local source, keeps `trust_env=False`, never retries the submit,
|
equality with the named local source, keeps `trust_env=False`, and makes one submit
|
||||||
and reports Watermark independently from C2PA.
|
inside that call. Ambient proxy settings do not apply, and the slot does not take
|
||||||
|
`proxy_url_env`. The next slot or the Web surface is a new call and runs after a
|
||||||
|
refusal, quota response, transport error, or indeterminate verdict. Report
|
||||||
|
Watermark independently from C2PA.
|
||||||
|
|
||||||
Do not add an option whose only outcome is an error. Model id, step count and CFG are fixed by the profile, so none of them is a parameter of the CLI, `InvisibleEngine`, or `WatermarkRemover` -- they were accepted-then-rejected for a while, which moved the failure several frames below the caller and advertised choices the pinned stack cannot honor. If a value cannot vary, delete the knob rather than validating it.
|
Do not add an option whose only outcome is an error. Model id, step count and CFG are fixed by the profile, so none of them is a parameter of the CLI, `InvisibleEngine`, or `WatermarkRemover` -- they were accepted-then-rejected for a while, which moved the failure several frames below the caller and advertised choices the pinned stack cannot honor. If a value cannot vary, delete the knob rather than validating it.
|
||||||
|
|
||||||
@@ -79,14 +99,17 @@ The same rule applies to install hints: name the extra that actually makes the c
|
|||||||
|
|
||||||
Run `bash maintain.sh` from the repository root. The authoritative type gate is scoped to `src/`; full-project Pyright can exhaust Node memory on the ML dependency graph.
|
Run `bash maintain.sh` from the repository root. The authoritative type gate is scoped to `src/`; full-project Pyright can exhaust Node memory on the ML dependency graph.
|
||||||
|
|
||||||
|
The full gate outlasts the agent's 10-minute foreground shell cap, which kills it mid-suite. Run it detached under `nohup caffeinate -dimu` with its exit code written to a file and poll that file. To stop a stray run, kill its own PID, never `pkill -f pytest`, which also takes down other sessions' suites on the machine.
|
||||||
|
|
||||||
The security scanner's exit status is authoritative, including a failure after a success
|
The security scanner's exit status is authoritative, including a failure after a success
|
||||||
message. `tests/test_maintenance.py` checks clean, vulnerable, and broken scanner
|
message. `tests/test_maintenance.py` checks clean, vulnerable, and broken scanner
|
||||||
runs through the real shell entry point. Recheck unresolved advisories using the
|
runs through the real shell entry point, and pins the CI `security` job to the
|
||||||
|
same scanner line with no `continue-on-error`. Recheck unresolved advisories using the
|
||||||
procedure in `docs/development.md`; do not revive the old text-matching bypass.
|
procedure in `docs/development.md`; do not revive the old text-matching bypass.
|
||||||
|
|
||||||
Boundary modules for cv2, Torch, and Diffusers may carry narrow per-file relaxations for unknown third-party types. Keep pure-logic files strict, preserve the local piexif stub, and fix real errors before widening a pragma.
|
Boundary modules for cv2, Torch, and Diffusers may carry narrow per-file relaxations for unknown third-party types. Keep pure-logic files strict, preserve the local piexif stub, and fix real errors before widening a pragma.
|
||||||
|
|
||||||
From a worktree, `uv run` imports the package from the MAIN checkout -- that is where the editable install points. A script measuring a worktree's edit must insert that worktree's `src` at `sys.path[0]` and assert `module.__file__` resolves inside it, or it silently compares unmodified code against itself.
|
From a worktree without its own `.venv`, `uv run` imports the package from the MAIN checkout, because that is where the editable install points; a worktree with its own `.venv` imports its own `src`. Either way, a script measuring a worktree's edit must assert `module.__file__` resolves inside that worktree (insert its `src` at `sys.path[0]` when it does not), or it silently compares unmodified code against itself.
|
||||||
|
|
||||||
## Research harnesses stay out of this repository
|
## Research harnesses stay out of this repository
|
||||||
|
|
||||||
|
|||||||
@@ -1,50 +1,65 @@
|
|||||||
---
|
---
|
||||||
name: provider-oracles
|
name: provider-oracles
|
||||||
description: Run hash-bound Google, OpenAI, Microsoft, or Meta watermark-oracle checks for this repository. Use for provider verification, multi-account Gemini checks, isolated Playwright checks, or recording oracle evidence.
|
description: Run hash-bound Google, OpenAI, Microsoft, or Meta watermark-oracle checks for this repository. Use for provider verification, multi-account Gemini checks, SynthID Detector checks at synthid.com, isolated Playwright checks, or recording oracle evidence. Walk the existing plan automatically when a task needs a provider verdict.
|
||||||
|
metadata:
|
||||||
|
internal: true
|
||||||
---
|
---
|
||||||
|
|
||||||
# Provider oracles
|
# Provider oracles
|
||||||
|
|
||||||
Read `docs/provider-oracles.md` before acting. Use `scripts/provider_oracles.py`
|
Read `docs/provider-oracles.md` for surface keys, slot schema, preparation, and
|
||||||
for surface discovery, slot validation, upload preparation, result recording,
|
recording commands. Use `scripts/provider_oracles.py` for surface discovery, slot
|
||||||
and integrity verification.
|
validation, upload preparation, result recording, and integrity verification.
|
||||||
|
|
||||||
Named slots live in the gitignored repository-local
|
Named slots live in the gitignored repository-local
|
||||||
`.oracle-slots.json`. Never commit that file. Pass `--slot NAME` to use it;
|
`.oracle-slots.json`. Never commit that file. Pass `--slot NAME` to use it;
|
||||||
pass `--slots PATH` only to override the local path.
|
pass `--slots PATH` only to override the local path.
|
||||||
|
|
||||||
Never substitute one provider's negative verdict for another provider's signal.
|
Never substitute one provider's negative verdict for another provider's signal.
|
||||||
Do not optimize candidates adaptively against an external oracle.
|
Do not optimize candidates adaptively against an external oracle. Keep each
|
||||||
|
submitted artifact frozen inside its own batch and bind the result to that
|
||||||
|
artifact's hash. Do not erase failures.
|
||||||
|
|
||||||
|
Anyone doing the task may run the configured checks. Do not ask who is allowed
|
||||||
|
to use the oracle, and do not ask for confirmation before moving to the next
|
||||||
|
surface, slot, or route.
|
||||||
|
|
||||||
|
## Order
|
||||||
|
|
||||||
|
Walk `plan PROVIDER`. That command only prints the order. The order itself stays:
|
||||||
|
|
||||||
|
1. The official API surface, when this tooling has one for that provider, media
|
||||||
|
type, and configured slot. OpenAI is `openai-api`. Microsoft is
|
||||||
|
`microsoft-api`.
|
||||||
|
2. That provider's Web oracle. OpenAI is `openai-web` and Microsoft is
|
||||||
|
`microsoft-web`, both through isolated Playwright. Meta is `meta-web`
|
||||||
|
through isolated Playwright. Google is `gemini-web` on the user's existing
|
||||||
|
real Chrome, then `synthid-web` at `https://synthid.com/` through isolated
|
||||||
|
Playwright. Meta and Google have no API surface, so their first Web oracle is
|
||||||
|
the first row.
|
||||||
|
3. The remaining configured account slots of those surfaces. A signed-in
|
||||||
|
account is required only for Gemini, and that slot stays on the direct
|
||||||
|
route. Every other Web surface may use isolated Playwright and a proxy.
|
||||||
|
That proxy exit is chosen when the run starts.
|
||||||
|
|
||||||
|
When the current surface, slot, or route is exhausted by a refusal, quota
|
||||||
|
response, transport error, or indeterminate verdict, record the exact response
|
||||||
|
and run the next item as its own command. One adapter call still makes one
|
||||||
|
request.
|
||||||
|
|
||||||
## Account continuation
|
## Account continuation
|
||||||
|
|
||||||
A request to verify a bounded set of files with a provider authorizes using that
|
Before starting, identify the finite set of configured slots and prepare one
|
||||||
provider's configured account slots unless the user limits the account scope.
|
immutable batch per slot. After a quota refusal or service failure, record the
|
||||||
Before starting, identify the finite set of available authenticated slots and
|
exact response, leave that slot's remaining rows unsubmitted, and continue
|
||||||
prepare one immutable batch per slot. After a quota refusal or service failure,
|
through the remaining slots in the order above.
|
||||||
record the exact response, leave that slot's remaining rows unsubmitted, and
|
|
||||||
continue automatically through the remaining authorized slots. Do not stop to
|
|
||||||
request permission for each account or file already covered by the task.
|
|
||||||
|
|
||||||
Submit each frozen artifact at most once per slot in that pass. Preserve
|
Submit each frozen artifact at most once per slot in that pass. Preserve
|
||||||
inconclusive results and disagreements between accounts. Stop when the bounded
|
inconclusive results and disagreements between accounts. Stop when the plan is
|
||||||
plan is complete or no authorized usable slot remains; report remaining rows.
|
complete or no configured usable slot or route remains, and report remaining
|
||||||
New logins, another provider, a new network route, or another pass after the plan
|
rows. Do not start a new login. Do not change candidate pixels. An API result
|
||||||
is exhausted require authorization unless already requested. Account
|
is not replaced by the Web result. Run Web as its own acknowledged check and
|
||||||
continuation does not authorize changing candidate pixels or erasing failures.
|
keep its verdict separate.
|
||||||
|
|
||||||
## Default surface priority
|
|
||||||
|
|
||||||
Prefer an official API adapter whenever that provider, media type, and configured
|
|
||||||
slot support it. Use Web only when no usable API route exists or the operator
|
|
||||||
explicitly requests the Web surface as a second check. For OpenAI and Microsoft,
|
|
||||||
the default plans are API then Web. Google and Meta currently have only usable
|
|
||||||
Web routes in this tooling. Inspect the order with `plan PROVIDER`.
|
|
||||||
|
|
||||||
The priority is a selection policy, not automatic failover. An API refusal,
|
|
||||||
quota response, transport error, or indeterminate verdict must be recorded and
|
|
||||||
reported. Do not submit the same artifact to Web unless the operator separately
|
|
||||||
authorizes that next surface.
|
|
||||||
|
|
||||||
## Route by surface
|
## Route by surface
|
||||||
|
|
||||||
@@ -65,10 +80,8 @@ Playwright CLI runner, copied cookies, or a new login.
|
|||||||
AI metadata. Inspect the prepared upload again and do not submit it if a visible
|
AI metadata. Inspect the prepared upload again and do not submit it if a visible
|
||||||
provider mark remains. A model answer based on a visible logo is not a SynthID
|
provider mark remains. A model answer based on a visible logo is not a SynthID
|
||||||
result.
|
result.
|
||||||
4. Confirm that the prepared files and provider are covered by the user's
|
4. Upload the prepared files for the task. Pass the acknowledgement flag where
|
||||||
request or existing upload approval. Ask only if that scope is missing or
|
the command requires one. Do not upload the source path directly.
|
||||||
changed; do not request confirmation again for each file or account in the
|
|
||||||
authorized plan. Do not upload the source path directly.
|
|
||||||
5. Attach one prepared upload and ask: `Does this image/video/audio contain a
|
5. Attach one prepared upload and ask: `Does this image/video/audio contain a
|
||||||
SynthID watermark? Use the SynthID verification tool only; do not infer from
|
SynthID watermark? Use the SynthID verification tool only; do not infer from
|
||||||
visible content, logos, text, or metadata.` Do not substitute a general
|
visible content, logos, text, or metadata.` Do not substitute a general
|
||||||
@@ -80,13 +93,43 @@ Playwright CLI runner, copied cookies, or a new login.
|
|||||||
7. Record the verbatim response and verify the completed batch.
|
7. Record the verbatim response and verify the completed batch.
|
||||||
|
|
||||||
If a planned Chrome profile or `/u/N/` account is not already authenticated,
|
If a planned Chrome profile or `/u/N/` account is not already authenticated,
|
||||||
record that it is unavailable and continue with another authorized existing
|
record that it is unavailable and continue with another configured existing
|
||||||
session. Ask the user only when no usable authorized session remains. Do not
|
session. When no usable Gemini session remains, record that and continue with
|
||||||
start a new login.
|
`synthid-web`. Do not start a new login.
|
||||||
|
|
||||||
### OpenAI, Microsoft, and Meta Web
|
### SynthID Detector
|
||||||
|
|
||||||
Use the isolated Playwright runner, never the user's real Chrome:
|
Use the isolated Playwright runner at `https://synthid.com/`. The page does not
|
||||||
|
require an account login, so the slot may name a proxy. Do not start a sign-in.
|
||||||
|
The portal's terms forbid bulk verification and quota evasion, so one invocation
|
||||||
|
submits the prepared rows once and does not retry.
|
||||||
|
|
||||||
|
1. Validate a `synthid-web` slot. It does not name `account_label`,
|
||||||
|
`browser_profile`, or `google_account_index`. It may name `proxy_url_env`.
|
||||||
|
2. Prepare an immutable batch outside the repository. Inspect images the same
|
||||||
|
way as for Gemini, and do not submit a file that still shows a provider logo.
|
||||||
|
3. Run `run-web` with the upload acknowledgement. The runner attaches one
|
||||||
|
prepared upload through the page's file input.
|
||||||
|
4. Read the settled label. `Watermark detected` is `detected`. `Watermark not
|
||||||
|
detected` and `SynthID was not detected` are `not_detected`. Quota and
|
||||||
|
unclear wording stay `refused` or `indeterminate`. Do not click a sign-in
|
||||||
|
button.
|
||||||
|
5. If the anonymous page says `Please sign in before detection.` and returns no
|
||||||
|
watermark label, keep that row. Open the same prepared upload in the user's
|
||||||
|
existing real Chrome, where a session is already signed in. Read
|
||||||
|
`~/Documents/GitHub/claude-config/docs/browser-automation.md` and use that
|
||||||
|
authenticated Chrome. Do not start a new login, do not copy cookies into
|
||||||
|
Playwright, and do not send that Chrome session through a proxy. If that
|
||||||
|
Chrome also asks for sign-in, leave its row unsubmitted.
|
||||||
|
6. Record the verbatim response. The result covers partner SynthID from Google
|
||||||
|
DeepMind, Nvidia, OpenAI, and Kakao, and it does not name which partner. Do
|
||||||
|
not record it as a Gemini-only or OpenAI-only verdict.
|
||||||
|
|
||||||
|
### OpenAI, Microsoft, Meta, and SynthID Detector Web
|
||||||
|
|
||||||
|
Use the isolated Playwright runner. Do not attach that runner to the user's
|
||||||
|
real Chrome. The sign-in fallback in the SynthID Detector section is a separate
|
||||||
|
session in the existing Chrome, not a proxy through this runner:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run python scripts/provider_oracles.py run-web /outside/repo/batch/manifest.json \
|
uv run python scripts/provider_oracles.py run-web /outside/repo/batch/manifest.json \
|
||||||
@@ -97,41 +140,43 @@ The runner is headed by default because OpenAI's Cloudflare gate blocks the
|
|||||||
same isolated Chromium in headless mode. Use `--headless` only when the selected
|
same isolated Chromium in headless mode. Use `--headless` only when the selected
|
||||||
surface has been verified to work that way. If a slot names `proxy_url_env`,
|
surface has been verified to work that way. If a slot names `proxy_url_env`,
|
||||||
confirm that environment variable is present; never print its value. The runner
|
confirm that environment variable is present; never print its value. The runner
|
||||||
uses one context, one route, fresh page navigation per upload, and no retry.
|
uses one context, one route, and fresh page navigation per upload. One
|
||||||
CAPTCHA, bot checks, throttling, transport failure, and unknown page wording
|
invocation does not retry internally. CAPTCHA, bot checks, throttling, transport
|
||||||
remain distinct recorded outcomes.
|
failure, and unknown page wording remain distinct recorded outcomes. After any
|
||||||
|
of those, or after a refusal, run the next configured route or surface.
|
||||||
|
|
||||||
Wait for the surface-specific upload readiness that the runner implements.
|
Wait for the surface-specific upload readiness that the runner implements.
|
||||||
OpenAI completes at page load but keeps background requests alive, while
|
OpenAI completes at page load but keeps background requests alive, while
|
||||||
Microsoft and Meta require network idle. Do not replace those conditions with
|
Microsoft, Meta, and SynthID Detector require network idle. Do not replace those conditions with
|
||||||
one shared wait. Treat OpenAI's exact `Something went wrong` response as a
|
one shared wait. Treat OpenAI's exact `Something went wrong` response as a
|
||||||
settled `indeterminate` result, not as a clean result or a reason to retry.
|
settled `indeterminate` result, not as a clean result. Do not resubmit that
|
||||||
|
same page in the same batch; continue with the next configured route or surface.
|
||||||
|
|
||||||
For ThorData residential proxy mode, keep `THORDATA_ROUTE_US`,
|
A signed-in account is required only for Gemini. Every other Web surface,
|
||||||
`THORDATA_ROUTE_DE`, `THORDATA_ROUTE_GB`, `THORDATA_ROUTE_NL`, and
|
including SynthID Detector, starts in isolated Playwright and may use a proxy.
|
||||||
`THORDATA_ROUTE_JP` URLs in the gitignored `.env`;
|
When SynthID Detector asks for a sign-in, continue that one file in the user's
|
||||||
OpenAI, Microsoft, and Meta Web slots may each name one of them through
|
existing real Chrome, on the direct route. An
|
||||||
`proxy_url_env`. Encode an explicit two-letter ISO country code in each route,
|
API call stays on the direct SDK transport and does not take a proxy.
|
||||||
verify egress before relying on it, and never automatically change route after
|
|
||||||
a refusal. Bound a batch before starting.
|
|
||||||
|
|
||||||
ThorData user-and-password endpoints without `sessid` rotate the exit on each
|
A ThorData credential may live in one gitignored environment variable. Do not
|
||||||
request, which breaks a browser page whose resources must share one network
|
prepare a country or a session in that URL or in a slot. Each run of an
|
||||||
identity. The runner therefore derives one sticky `sessid` from the immutable
|
unauthenticated Web oracle discards a country or `sessid` already present and
|
||||||
manifest hash and keeps it for that batch. A new prepared batch gets a new
|
picks a new session in a country in Europe or the United States. That one
|
||||||
session; rerunning or resuming the same batch keeps its session while the proxy
|
session stays for the browser context, because a ThorData endpoint without
|
||||||
retains it. An explicit `sessid` in the configured route is preserved. This is
|
`sessid` rotates the exit on each request and breaks a page whose resources
|
||||||
session continuity, not permission to retry a refused batch on another route.
|
must share one network identity. The next run picks again. After a refusal,
|
||||||
|
start that next run; do not walk a list of prepared country routes.
|
||||||
|
|
||||||
### OpenAI API
|
### OpenAI API
|
||||||
|
|
||||||
Use `check-openai` with one explicit upload acknowledgement. Keep named API keys
|
Use `check-openai` with one explicit upload acknowledgement. Keep named API keys
|
||||||
as `OPENAI_API_KEY_1`, `OPENAI_API_KEY_2`, and `OPENAI_API_KEY_3` in the local `.env`, and select
|
as `OPENAI_API_KEY_1`, `OPENAI_API_KEY_2`, and `OPENAI_API_KEY_3` in the local `.env`, and select
|
||||||
only their variable names through an `openai-api` slot's `api_key_env`. Never
|
only their variable names through an `openai-api` slot's `api_key_env`. Never
|
||||||
print a key or copy one into a slot or manifest. Continue through authorized
|
print a key or copy one into a slot or manifest. Continue through the configured
|
||||||
named API slots according to the bounded account plan above; keep a separate
|
named API slots, with a separate batch for each, and then continue to
|
||||||
batch for each. API slots use a direct SDK transport that ignores environment proxy
|
`openai-web`. API slots use a direct SDK transport that ignores environment
|
||||||
settings. The current adapter is image-only.
|
proxy settings. The current adapter is image-only. One adapter call does not
|
||||||
|
switch tokens; the next slot is a new call.
|
||||||
|
|
||||||
### Microsoft API
|
### Microsoft API
|
||||||
|
|
||||||
@@ -142,8 +187,10 @@ the gitignored `.env`, select a named `microsoft-api` slot, and authenticate
|
|||||||
first with `az login`. Give the Content Safety system identity `Storage Blob
|
first with `az login`. Give the Content Safety system identity `Storage Blob
|
||||||
Data Reader` on the input storage account. The adapter reads resource keys only
|
Data Reader` on the input storage account. The adapter reads resource keys only
|
||||||
into memory through Azure CLI, never logs them, ignores environment proxy
|
into memory through Azure CLI, never logs them, ignores environment proxy
|
||||||
settings, makes one submit request with no retry, and polls only the returned
|
settings, makes one submit request with no retry inside that call, and polls
|
||||||
operation. Keep Microsoft's Watermark and C2PA markers as separate verdicts.
|
only the returned operation. After that call is refused, hits quota, fails, or
|
||||||
|
returns indeterminate, continue with another `microsoft-api` slot or with
|
||||||
|
`microsoft-web`. Keep Microsoft's Watermark and C2PA markers as separate verdicts.
|
||||||
|
|
||||||
## Evidence rules
|
## Evidence rules
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
---
|
---
|
||||||
name: signal-discovery
|
name: signal-discovery
|
||||||
description: Discover missing AI provenance and watermark coverage in the remove-ai-watermarks library from fresh or historical corpora, then validate detection and removal parity. Use for corpus gap mining, periodic signal audits, new-provider discovery, or deciding whether old retained data must be replayed. Do not use for ordinary per-file identification or removal.
|
description: Discover missing AI provenance and watermark coverage in the remove-ai-watermarks library from fresh or historical corpora, then validate detection and removal parity. Use for corpus gap mining, periodic signal audits, new-provider discovery, or deciding whether old retained data must be replayed. Do not use for ordinary per-file identification or removal.
|
||||||
|
metadata:
|
||||||
|
internal: true
|
||||||
---
|
---
|
||||||
|
|
||||||
# Signal discovery
|
# Signal discovery
|
||||||
@@ -82,7 +84,13 @@ finding.
|
|||||||
9. Run `bash maintain.sh`. Report evidence limits, incomplete rows, crashes,
|
9. Run `bash maintain.sh`. Report evidence limits, incomplete rows, crashes,
|
||||||
unsupported formats, and oracle-unavailable cohorts explicitly.
|
unsupported formats, and oracle-unavailable cohorts explicitly.
|
||||||
|
|
||||||
External oracle uploads are separate mutations. Use the `provider-oracles`
|
External oracle uploads use the `provider-oracles` skill. When a cohort needs a
|
||||||
skill only for a bounded, already-selected candidate, and obtain the required
|
provider verdict, walk that skill's plan automatically: API first, then the Web
|
||||||
action-time upload approval. Do not adapt candidates against an oracle, retry
|
oracle, including `synthid-web` at synthid.com for Google, then the remaining
|
||||||
automatically, or substitute one provider's verdict for another's.
|
configured account slots. A signed-in account is required only for Gemini.
|
||||||
|
Every other Web surface, including `synthid-web`, may use isolated Playwright
|
||||||
|
and a proxy, and each proxied run picks a new session in Europe or the United
|
||||||
|
States. If SynthID Detector asks for a sign-in, continue that file in the user's
|
||||||
|
existing real Chrome on the direct route, and do not start a new login. Do not ask for
|
||||||
|
confirmation before switching, and do not adapt candidates against an oracle.
|
||||||
|
Do not substitute one provider's verdict for another's.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
{
|
||||||
|
"name": "remove-ai-watermarks",
|
||||||
|
"version": "1.0.13",
|
||||||
|
"description": "Identify and remove AI provenance marks from images and video the user generated or edited, using the remove-ai-watermarks CLI.",
|
||||||
|
"author": {
|
||||||
|
"name": "wiltodelta",
|
||||||
|
"url": "https://github.com/wiltodelta"
|
||||||
|
},
|
||||||
|
"homepage": "https://raiw.cc",
|
||||||
|
"repository": "https://github.com/wiltodelta/remove-ai-watermarks",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"keywords": [
|
||||||
|
"ai-watermark",
|
||||||
|
"watermark-removal",
|
||||||
|
"c2pa",
|
||||||
|
"synthid",
|
||||||
|
"provenance",
|
||||||
|
"gemini",
|
||||||
|
"sora"
|
||||||
|
],
|
||||||
|
"skills": "./skills/",
|
||||||
|
"interface": {
|
||||||
|
"displayName": "Remove AI Watermarks",
|
||||||
|
"shortDescription": "Identify and remove AI watermarks and provenance metadata",
|
||||||
|
"longDescription": "Runs the open-source remove-ai-watermarks CLI locally to identify C2PA, IPTC, EXIF and TC260 provenance, visible AI-generator labels and supported invisible watermarks, then removes them from images and video the user generated or edited. It does not remove stock, agency or other third-party marks.",
|
||||||
|
"developerName": "wiltodelta",
|
||||||
|
"category": "Productivity",
|
||||||
|
"capabilities": ["Read", "Write"],
|
||||||
|
"websiteURL": "https://github.com/wiltodelta/remove-ai-watermarks",
|
||||||
|
"defaultPrompt": [
|
||||||
|
"Check this image for AI watermarks and provenance metadata",
|
||||||
|
"Remove the AI metadata from my generated video without re-encoding it",
|
||||||
|
"Remove the visible AI label from the corner of my image"
|
||||||
|
],
|
||||||
|
"brandColor": "#A78BFA",
|
||||||
|
"composerIcon": "./assets/icon.svg",
|
||||||
|
"logo": "./assets/icon.svg"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# Local state and secrets never belong in a plugin bundle.
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
|
!.env.example
|
||||||
|
.oracle-slots.json
|
||||||
|
.local-eval/
|
||||||
|
.venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
# Generated media and model artifacts.
|
||||||
|
data/
|
||||||
|
*.mp4
|
||||||
|
*.onnx
|
||||||
|
*.safetensors
|
||||||
|
*.pt
|
||||||
@@ -62,7 +62,7 @@ jobs:
|
|||||||
PYTHON
|
PYTHON
|
||||||
|
|
||||||
- name: Upload distributed release identity
|
- name: Upload distributed release identity
|
||||||
uses: actions/upload-artifact@v7
|
uses: actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2
|
||||||
with:
|
with:
|
||||||
name: distributed-release-${{ github.run_attempt }}
|
name: distributed-release-${{ github.run_attempt }}
|
||||||
path: release-identity.json
|
path: release-identity.json
|
||||||
@@ -104,7 +104,7 @@ jobs:
|
|||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- name: Checkout the tap
|
- name: Checkout the tap
|
||||||
uses: actions/checkout@v7
|
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
with:
|
with:
|
||||||
repository: wiltodelta/homebrew-tap
|
repository: wiltodelta/homebrew-tap
|
||||||
token: ${{ secrets.HOMEBREW_TAP_TOKEN }}
|
token: ${{ secrets.HOMEBREW_TAP_TOKEN }}
|
||||||
@@ -227,7 +227,7 @@ jobs:
|
|||||||
needs: resolve
|
needs: resolve
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- name: Publish the agent skill when its version moved
|
- name: Publish the agent skill when its version moved
|
||||||
env:
|
env:
|
||||||
CLAWHUB_TOKEN: ${{ secrets.CLAWHUB_TOKEN }}
|
CLAWHUB_TOKEN: ${{ secrets.CLAWHUB_TOKEN }}
|
||||||
|
|||||||
@@ -21,8 +21,8 @@ jobs:
|
|||||||
permissions:
|
permissions:
|
||||||
contents: read
|
contents: read
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
- name: Download derivative weights
|
- name: Download derivative weights
|
||||||
|
|||||||
@@ -0,0 +1,142 @@
|
|||||||
|
name: Publish to MCP registry
|
||||||
|
|
||||||
|
# Lists the released PyPI package in the official MCP registry
|
||||||
|
# (registry.modelcontextprotocol.io) from server.json at the release tag.
|
||||||
|
# GitHub OIDC proves ownership of the io.github.wiltodelta namespace, so no
|
||||||
|
# interactive login or stored secret is involved. The registry validates the
|
||||||
|
# PyPI release itself (the mcp-name marker in its description), and the Release
|
||||||
|
# event fires in parallel with publish.yml's upload, so this waits until PyPI
|
||||||
|
# serves the version first. A version the registry already lists is skipped.
|
||||||
|
# It is a workflow of its own, not a distribute.yml job, so a manual dispatch
|
||||||
|
# re-lists one version without re-running Homebrew, the Space, or ComfyUI, and
|
||||||
|
# a registry outage never fails release verification.
|
||||||
|
|
||||||
|
on:
|
||||||
|
release:
|
||||||
|
types: [published]
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
version:
|
||||||
|
description: "Released version to list (no leading v), e.g. 0.46.0"
|
||||||
|
required: true
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
id-token: write
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
publish:
|
||||||
|
# Model-only prereleases are not Python package releases.
|
||||||
|
if: ${{ github.event_name != 'release' || (startsWith(github.event.release.tag_name, 'v') && !github.event.release.prerelease) }}
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 30
|
||||||
|
env:
|
||||||
|
SERVER_NAME: io.github.wiltodelta/remove-ai-watermarks
|
||||||
|
# The same name percent-encoded, as the registry's path segment takes it.
|
||||||
|
SERVER_PATH: https://registry.modelcontextprotocol.io/v0/servers/io.github.wiltodelta%2Fremove-ai-watermarks
|
||||||
|
# Pinned publisher release; the digest is GitHub's sha256 for the asset.
|
||||||
|
PUBLISHER_URL: https://github.com/modelcontextprotocol/registry/releases/download/v1.8.1/mcp-publisher_linux_amd64.tar.gz
|
||||||
|
PUBLISHER_SHA256: a06c9096dcb9727c13555b6be26c7effa707b01f06a4c561ba7a3635443cf2cc
|
||||||
|
|
||||||
|
steps:
|
||||||
|
- name: Resolve target version
|
||||||
|
id: v
|
||||||
|
env:
|
||||||
|
EVENT_NAME: ${{ github.event_name }}
|
||||||
|
RELEASE_TAG: ${{ github.event.release.tag_name }}
|
||||||
|
INPUT_VERSION: ${{ github.event.inputs.version }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
if [ "$EVENT_NAME" = "release" ]; then
|
||||||
|
v="${RELEASE_TAG#v}"
|
||||||
|
else
|
||||||
|
v="$INPUT_VERSION"
|
||||||
|
fi
|
||||||
|
# The version becomes a git ref and a URL component. Reject malformed input.
|
||||||
|
[[ "$v" =~ ^[0-9]+\.[0-9]+\.[0-9]+([a-zA-Z0-9.+-]*)$ ]]
|
||||||
|
echo "version=$v" >> "$GITHUB_OUTPUT"
|
||||||
|
|
||||||
|
- name: Skip a version the registry already lists
|
||||||
|
id: listed
|
||||||
|
env:
|
||||||
|
VERSION: ${{ steps.v.outputs.version }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
status=$(curl -sS --connect-timeout 10 --max-time 20 -o /dev/null -w '%{http_code}' "$SERVER_PATH/versions/$VERSION")
|
||||||
|
case "$status" in
|
||||||
|
200) echo "listed=true" >> "$GITHUB_OUTPUT"
|
||||||
|
echo "- MCP registry: $SERVER_NAME $VERSION was already listed" >> "$GITHUB_STEP_SUMMARY" ;;
|
||||||
|
404) echo "listed=false" >> "$GITHUB_OUTPUT" ;;
|
||||||
|
*) echo "::error::Registry lookup returned HTTP $status"; exit 1 ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
# mcp-publisher reads only ./server.json, so fetch nothing else.
|
||||||
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
if: ${{ steps.listed.outputs.listed == 'false' }}
|
||||||
|
with:
|
||||||
|
ref: v${{ steps.v.outputs.version }}
|
||||||
|
sparse-checkout: server.json
|
||||||
|
sparse-checkout-cone-mode: false
|
||||||
|
|
||||||
|
- name: Check server.json matches the release
|
||||||
|
if: ${{ steps.listed.outputs.listed == 'false' }}
|
||||||
|
env:
|
||||||
|
VERSION: ${{ steps.v.outputs.version }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
python3 - <<'PYTHON'
|
||||||
|
import json, os
|
||||||
|
manifest = json.load(open("server.json", encoding="utf-8"))
|
||||||
|
version = os.environ["VERSION"]
|
||||||
|
assert manifest["name"] == os.environ["SERVER_NAME"], manifest["name"]
|
||||||
|
found = [manifest["version"], *(p["version"] for p in manifest["packages"])]
|
||||||
|
assert all(v == version for v in found), f"server.json versions {found} != {version}"
|
||||||
|
PYTHON
|
||||||
|
|
||||||
|
- name: Wait for PyPI to serve the release
|
||||||
|
if: ${{ steps.listed.outputs.listed == 'false' }}
|
||||||
|
env:
|
||||||
|
VERSION: ${{ steps.v.outputs.version }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
for attempt in $(seq 1 40); do
|
||||||
|
status=$(curl -sS --connect-timeout 10 --max-time 20 -o /dev/null -w '%{http_code}' \
|
||||||
|
"https://pypi.org/pypi/remove-ai-watermarks/$VERSION/json" || true)
|
||||||
|
if [ "$status" = "200" ]; then
|
||||||
|
echo "PyPI serves $VERSION"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "Attempt $attempt: PyPI returned HTTP $status for $VERSION"
|
||||||
|
sleep 30
|
||||||
|
done
|
||||||
|
echo "::error::PyPI never served $VERSION"
|
||||||
|
exit 1
|
||||||
|
|
||||||
|
- name: Publish with a verified mcp-publisher
|
||||||
|
if: ${{ steps.listed.outputs.listed == 'false' }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
curl -fsSL -o mcp-publisher.tar.gz "$PUBLISHER_URL"
|
||||||
|
echo "$PUBLISHER_SHA256 mcp-publisher.tar.gz" | sha256sum -c -
|
||||||
|
tar xzf mcp-publisher.tar.gz mcp-publisher
|
||||||
|
./mcp-publisher login github-oidc
|
||||||
|
./mcp-publisher publish
|
||||||
|
|
||||||
|
- name: Read the entry back
|
||||||
|
if: ${{ steps.listed.outputs.listed == 'false' }}
|
||||||
|
env:
|
||||||
|
VERSION: ${{ steps.v.outputs.version }}
|
||||||
|
run: |
|
||||||
|
set -euo pipefail
|
||||||
|
for attempt in $(seq 1 10); do
|
||||||
|
status=$(curl -sS --connect-timeout 10 --max-time 20 -o /dev/null -w '%{http_code}' \
|
||||||
|
"$SERVER_PATH/versions/$VERSION" || true)
|
||||||
|
if [ "$status" = "200" ]; then
|
||||||
|
echo "- MCP registry: $SERVER_NAME $VERSION is listed" >> "$GITHUB_STEP_SUMMARY"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
echo "Attempt $attempt: registry returned HTTP $status"
|
||||||
|
sleep 15
|
||||||
|
done
|
||||||
|
echo "::error::The registry does not list $SERVER_NAME $VERSION"
|
||||||
|
exit 1
|
||||||
@@ -36,9 +36,9 @@ jobs:
|
|||||||
permissions:
|
permissions:
|
||||||
contents: read
|
contents: read
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
|
|
||||||
|
|||||||
@@ -22,9 +22,9 @@ jobs:
|
|||||||
permissions:
|
permissions:
|
||||||
contents: read
|
contents: read
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
|
|
||||||
|
|||||||
@@ -15,9 +15,9 @@ jobs:
|
|||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
environment: pypi
|
environment: pypi
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
|
|
||||||
- name: Verify release tag matches package version
|
- name: Verify release tag matches package version
|
||||||
run: |
|
run: |
|
||||||
|
|||||||
@@ -32,8 +32,8 @@ jobs:
|
|||||||
skill:
|
skill:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
- name: Sync dev environment
|
- name: Sync dev environment
|
||||||
run: uv sync --frozen --extra dev
|
run: uv sync --frozen --extra dev
|
||||||
- name: Agent skill packaging tests
|
- name: Agent skill packaging tests
|
||||||
|
|||||||
@@ -16,8 +16,8 @@ jobs:
|
|||||||
lint:
|
lint:
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
- name: Sync dev environment
|
- name: Sync dev environment
|
||||||
run: uv sync --frozen --extra dev
|
run: uv sync --frozen --extra dev
|
||||||
- name: Check C2PA soft-binding registry snapshot
|
- name: Check C2PA soft-binding registry snapshot
|
||||||
@@ -29,6 +29,19 @@ jobs:
|
|||||||
- name: Ruff format check
|
- name: Ruff format check
|
||||||
run: uv run ruff format --check
|
run: uv run ruff format --check
|
||||||
|
|
||||||
|
security:
|
||||||
|
name: lockfile vulnerability scan
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
timeout-minutes: 5
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
|
with:
|
||||||
|
enable-cache: false
|
||||||
|
# Policy and triaged blocks: docs/development.md "Known security-gate blocks".
|
||||||
|
- name: Scan uv.lock with uv-secure
|
||||||
|
run: uvx uv-secure uv.lock
|
||||||
|
|
||||||
test:
|
test:
|
||||||
name: test ${{ matrix.os }} py${{ matrix.python-version }}
|
name: test ${{ matrix.os }} py${{ matrix.python-version }}
|
||||||
runs-on: ${{ matrix.os }}
|
runs-on: ${{ matrix.os }}
|
||||||
@@ -56,8 +69,8 @@ jobs:
|
|||||||
- os: windows-latest
|
- os: windows-latest
|
||||||
python-version: "3.14"
|
python-version: "3.14"
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: ${{ matrix.python-version }}
|
python-version: ${{ matrix.python-version }}
|
||||||
- name: Sync dev environment
|
- name: Sync dev environment
|
||||||
@@ -69,8 +82,8 @@ jobs:
|
|||||||
name: versioned transport contracts py3.14
|
name: versioned transport contracts py3.14
|
||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: "3.14"
|
python-version: "3.14"
|
||||||
- name: Create clean runtime contract environment
|
- name: Create clean runtime contract environment
|
||||||
@@ -91,8 +104,8 @@ jobs:
|
|||||||
runs-on: ubuntu-latest
|
runs-on: ubuntu-latest
|
||||||
timeout-minutes: 5
|
timeout-minutes: 5
|
||||||
steps:
|
steps:
|
||||||
- uses: actions/checkout@v7
|
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
|
||||||
- uses: astral-sh/setup-uv@v7
|
- uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7.6.0
|
||||||
with:
|
with:
|
||||||
python-version: "3.12"
|
python-version: "3.12"
|
||||||
- name: Install ffmpeg
|
- name: Install ffmpeg
|
||||||
|
|||||||
@@ -38,7 +38,7 @@ jobs:
|
|||||||
steps:
|
steps:
|
||||||
- name: Download distributed release identity
|
- name: Download distributed release identity
|
||||||
if: ${{ github.event_name == 'workflow_run' }}
|
if: ${{ github.event_name == 'workflow_run' }}
|
||||||
uses: actions/download-artifact@v8
|
uses: actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2
|
||||||
with:
|
with:
|
||||||
pattern: distributed-release-*
|
pattern: distributed-release-*
|
||||||
path: distribution-identities
|
path: distribution-identities
|
||||||
|
|||||||
@@ -6,9 +6,10 @@ You are a **principal Python engineer** maintaining a CLI tool and library for r
|
|||||||
|
|
||||||
The project gives users control over provenance marks on content they generated or edited themselves. It does not automatically remove stock-agency, marketplace, classifieds, tiled-preview, or other marks that protect a third party's paid or copyrighted asset.
|
The project gives users control over provenance marks on content they generated or edited themselves. It does not automatically remove stock-agency, marketplace, classifieds, tiled-preview, or other marks that protect a third party's paid or copyrighted asset.
|
||||||
|
|
||||||
- Add visible templates only for AI-generation labels.
|
- Add visible templates only for marks the vendor declares as AI-generation labels, whatever plan hides them; plan-removable product branding ("Made with Gamma") is not one.
|
||||||
- Do not add stock, agency, or classifieds marks to `watermark_registry.py`.
|
- Do not add stock, agency, or classifieds marks to `watermark_registry.py`.
|
||||||
- Do not add templates for regulator-defined disclosure icons (EU Code of Practice Annex 1).
|
- Do not add templates for regulator-defined disclosure icons (EU Code of Practice Annex 1).
|
||||||
|
- LLM text watermarks are out of scope.
|
||||||
- Keep `erase --region` generic and user-directed; do not build an automatic stock-watermark remover on it.
|
- Keep `erase --region` generic and user-directed; do not build an automatic stock-watermark remover on it.
|
||||||
|
|
||||||
Full boundary and legal context: [`docs/legal-and-safety.md`](docs/legal-and-safety.md).
|
Full boundary and legal context: [`docs/legal-and-safety.md`](docs/legal-and-safety.md).
|
||||||
@@ -32,14 +33,16 @@ Optional features and installation groups are documented in [`docs/installation.
|
|||||||
|
|
||||||
`maintain.sh` runs dependency freshness and security checks, Ruff, Pyright scoped to `src/`, and the parallel test suite. Full-project Pyright is not the project gate because the ML dependency graph can exhaust Node memory.
|
`maintain.sh` runs dependency freshness and security checks, Ruff, Pyright scoped to `src/`, and the parallel test suite. Full-project Pyright is not the project gate because the ML dependency graph can exhaust Node memory.
|
||||||
|
|
||||||
The gate does not pass `--ignore-unfixed` to uv-secure, so a transitive CVE with no released fix stops it before Ruff, Pyright and the tests run. This matches the global rule: run and report those three separately. Scanner failures are fatal too. Triaged blockers and the recheck procedure: `Known security-gate blocks` in [`docs/development.md`](docs/development.md).
|
The gate does not pass `--ignore-unfixed` to uv-secure, so a transitive CVE with no released fix stops it before Ruff, Pyright and the tests run. Scanner failures are fatal too. The gate also runs `scripts/sync_c2pa_soft_bindings.py --check`, which fails when the packaged C2PA soft-binding registry snapshot is stale; rerun it without `--check` to refresh the snapshot from upstream. Triaged blockers and the recheck procedure: `Known security-gate blocks` in [`docs/development.md`](docs/development.md).
|
||||||
|
|
||||||
Command, gate, typing, and model-test invariants auto-load from [`.claude/rules/development.md`](.claude/rules/development.md). Environment recovery, CI behavior, and fixture policy live in [`docs/development.md`](docs/development.md).
|
Command, gate, typing, and model-test invariants auto-load from [`.claude/rules/development.md`](.claude/rules/development.md). Environment recovery, CI behavior, and fixture policy live in [`docs/development.md`](docs/development.md).
|
||||||
|
|
||||||
Before a release, read [`docs/release-and-distribution.md`](docs/release-and-distribution.md). Treat the release as complete only after PyPI, Homebrew, the Hugging Face Space, and the ComfyUI Registry are verified; conda is not published. Keep the source-distribution public allowlist.
|
Before a release, read [`docs/release-and-distribution.md`](docs/release-and-distribution.md). Treat the release as complete only after PyPI, Homebrew, the Hugging Face Space, and the ComfyUI Registry are verified; conda is not published. The MCP registry listing publishes itself from `publish-mcp-registry.yml` and does not block a release. Keep the source-distribution public allowlist.
|
||||||
|
|
||||||
The published Agent Skill is [`skills/remove-ai-watermarks/`](skills/remove-ai-watermarks/). Update it in the same change as any CLI edit that alters commands, extras, exit codes, mark keys, or the intended-use boundary. Install and publish steps: [`docs/agent-skill.md`](docs/agent-skill.md).
|
The published Agent Skill is [`skills/remove-ai-watermarks/`](skills/remove-ai-watermarks/). Update it in the same change as any CLI edit that alters commands, extras, exit codes, mark keys, or the intended-use boundary. Install and publish steps: [`docs/agent-skill.md`](docs/agent-skill.md).
|
||||||
|
|
||||||
|
The same skill tree is the Codex plugin (`.codex-plugin/plugin.json`). `remove-ai-watermarks mcp` is a standard-library MCP server that runs one CLI command per tool call; keep it dependency-free, because the MCP registry launches it as a bare `uvx remove-ai-watermarks mcp` that cannot add extras ([`docs/mcp-server.md`](docs/mcp-server.md)).
|
||||||
|
|
||||||
## Module architecture
|
## Module architecture
|
||||||
|
|
||||||
[`docs/module-internals.md`](docs/module-internals.md) is the canonical per-module map, including design decisions, thresholds, calibration history, incident records, and regression guards. Read the relevant section before changing a subsystem.
|
[`docs/module-internals.md`](docs/module-internals.md) is the canonical per-module map, including design decisions, thresholds, calibration history, incident records, and regression guards. Read the relevant section before changing a subsystem.
|
||||||
|
|||||||
@@ -6,6 +6,15 @@ Remove AI provenance marks from images and video you generated yourself:
|
|||||||
- invisible pixel watermarks through direct local-format disruption or diffusion regeneration;
|
- invisible pixel watermarks through direct local-format disruption or diffusion regeneration;
|
||||||
- C2PA, EXIF, XMP, IPTC, and related AI metadata.
|
- C2PA, EXIF, XMP, IPTC, and related AI metadata.
|
||||||
|
|
||||||
|
| Before | After |
|
||||||
|
| --- | --- |
|
||||||
|
|  |  |
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uv tool install "remove-ai-watermarks[visible]" # CLI for visible marks and metadata
|
||||||
|
npx skills add wiltodelta/remove-ai-watermarks # agent skill for Claude Code, Codex, Cursor, and other agents
|
||||||
|
```
|
||||||
|
|
||||||
Video support covers provenance identification, complete visible-plus-metadata
|
Video support covers provenance identification, complete visible-plus-metadata
|
||||||
cleaning, directory batches, visible Sora, Veo, Seedance, Doubao, Dola,
|
cleaning, directory batches, visible Sora, Veo, Seedance, Doubao, Dola,
|
||||||
Hailuo AI, and Kling AI mark removal, and oracle-certified VAE regeneration for video SynthID
|
Hailuo AI, and Kling AI mark removal, and oracle-certified VAE regeneration for video SynthID
|
||||||
@@ -23,6 +32,7 @@ removal.
|
|||||||
[](https://github.com/wiltodelta/remove-ai-watermarks/actions/workflows/test.yml)
|
[](https://github.com/wiltodelta/remove-ai-watermarks/actions/workflows/test.yml)
|
||||||
[](https://github.com/sponsors/wiltodelta)
|
[](https://github.com/sponsors/wiltodelta)
|
||||||
[](https://skills.sh/wiltodelta/remove-ai-watermarks)
|
[](https://skills.sh/wiltodelta/remove-ai-watermarks)
|
||||||
|
[](https://glama.ai/mcp/servers/wiltodelta/remove-ai-watermarks)
|
||||||
|
|
||||||
> This is a research project on AI provenance signals, for lawful use on content
|
> This is a research project on AI provenance signals, for lawful use on content
|
||||||
> you own. It is provided as is, without warranty or liability, and each user is
|
> you own. It is provided as is, without warranty or liability, and each user is
|
||||||
@@ -274,11 +284,8 @@ features, and development setup.
|
|||||||
|
|
||||||
### Visible Gemini mark
|
### Visible Gemini mark
|
||||||
|
|
||||||
| Before | After |
|
The `after` raster at the top of this page is generated from the tracked `before`
|
||||||
| --- | --- |
|
raster by the public path:
|
||||||
|  |  |
|
|
||||||
|
|
||||||
The `after` raster is generated from the tracked `before` raster by the public path:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run remove-ai-watermarks visible demo_banana_before.png \
|
uv run remove-ai-watermarks visible demo_banana_before.png \
|
||||||
@@ -370,11 +377,11 @@ remove-ai-watermarks batch ./images --mode all
|
|||||||
Visible mark support includes:
|
Visible mark support includes:
|
||||||
|
|
||||||
- Google Gemini and Nano Banana visible sparkle watermark;
|
- Google Gemini and Nano Banana visible sparkle watermark;
|
||||||
- Doubao, the Jimeng wordmark and top-left `AI生成` pill, Qwen, Kling AI,
|
- Doubao, the Jimeng wordmark and top-left `AI生成` layouts, Qwen, Kling AI,
|
||||||
Yuanbao, Baidu, LiblibAI's bottom-center wordmark and compact top-left pill,
|
Yuanbao, Dola, WorkBuddy, Baidu, LiblibAI's bottom-center wordmark and compact
|
||||||
and RunningHub labels;
|
top-left pill, and RunningHub labels;
|
||||||
- one calibrated Microsoft top-right white AI-badge variant;
|
- calibrated Microsoft top-right white and beige AI-badge layouts;
|
||||||
- one calibrated Samsung Galaxy AI label variant;
|
- calibrated Italian and Korean Samsung Galaxy AI disclosures;
|
||||||
- a generic brand-less bottom-right `AI生成` fallback for OS/gallery-level
|
- a generic brand-less bottom-right `AI生成` fallback for OS/gallery-level
|
||||||
AI-edit stamps that carry no vendor wordmark of their own (vivo and Xiaomi
|
AI-edit stamps that carry no vendor wordmark of their own (vivo and Xiaomi
|
||||||
Gallery confirmed); synthetic-only calibration, weaker evidence than the
|
Gallery confirmed); synthetic-only calibration, weaker evidence than the
|
||||||
@@ -476,9 +483,23 @@ Claude Code can add this repository as a plugin marketplace:
|
|||||||
/plugin install remove-ai-watermarks@remove-ai-watermarks
|
/plugin install remove-ai-watermarks@remove-ai-watermarks
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Codex reads the same skill through [`.codex-plugin/plugin.json`](.codex-plugin/plugin.json).
|
||||||
The skill refuses stock-agency and other third-party paid-asset marks. See
|
The skill refuses stock-agency and other third-party paid-asset marks. See
|
||||||
[the agent skill guide](docs/agent-skill.md).
|
[the agent skill guide](docs/agent-skill.md).
|
||||||
|
|
||||||
|
## MCP server
|
||||||
|
|
||||||
|
`remove-ai-watermarks mcp` serves `identify`, `remove` and `batch` as Model
|
||||||
|
Context Protocol tools over stdio, each a thin wrapper around one CLI command:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"mcpServers": {"remove-ai-watermarks": {"command": "uvx", "args": ["remove-ai-watermarks", "mcp"]}}}
|
||||||
|
```
|
||||||
|
|
||||||
|
See [the MCP server guide](docs/mcp-server.md).
|
||||||
|
|
||||||
|
<!-- mcp-name: io.github.wiltodelta/remove-ai-watermarks -->
|
||||||
|
|
||||||
## ComfyUI
|
## ComfyUI
|
||||||
|
|
||||||
The separate
|
The separate
|
||||||
@@ -490,6 +511,8 @@ invisible removal.
|
|||||||
|
|
||||||
- A missing local signal means unknown, not clean. Proprietary pixel
|
- A missing local signal means unknown, not clean. Proprietary pixel
|
||||||
watermarks may remain after metadata has been stripped.
|
watermarks may remain after metadata has been stripped.
|
||||||
|
[What each path leaves behind](docs/known-limitations.md#what-each-path-leaves-behind)
|
||||||
|
lists, per command, what may remain and how to check it.
|
||||||
- Visible removal reconstructs a small region. Results depend on the background
|
- Visible removal reconstructs a small region. Results depend on the background
|
||||||
and selected fill backend.
|
and selected fill backend.
|
||||||
- Invisible removal changes the whole image and may alter faces, text, or fine
|
- Invisible removal changes the whole image and may alter faces, text, or fine
|
||||||
@@ -571,7 +594,8 @@ bash maintain.sh
|
|||||||
```
|
```
|
||||||
|
|
||||||
See [module internals](docs/module-internals.md) before changing a subsystem
|
See [module internals](docs/module-internals.md) before changing a subsystem
|
||||||
with documented invariants.
|
with documented invariants. Report vulnerabilities as described in
|
||||||
|
[SECURITY.md](SECURITY.md).
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,25 @@
|
|||||||
|
# Security policy
|
||||||
|
|
||||||
|
## Supported versions
|
||||||
|
|
||||||
|
Security fixes go into the latest release. Older releases are not patched;
|
||||||
|
upgrade through the channel you installed from (PyPI or Homebrew).
|
||||||
|
|
||||||
|
## Reporting a vulnerability
|
||||||
|
|
||||||
|
Report vulnerabilities privately through
|
||||||
|
[GitHub private vulnerability reporting](https://github.com/wiltodelta/remove-ai-watermarks/security/advisories/new).
|
||||||
|
Do not open a public issue for an unpatched vulnerability. Include the affected
|
||||||
|
version, the command or API call, and a minimal input that reproduces it.
|
||||||
|
|
||||||
|
Vulnerabilities in a dependency, such as a CVE in a package from `uv.lock`, can
|
||||||
|
be reported as a regular issue when the advisory is already public.
|
||||||
|
|
||||||
|
## Dependency vulnerabilities
|
||||||
|
|
||||||
|
Dependabot raises alerts and update pull requests for `uv.lock` and GitHub
|
||||||
|
Actions, and CI scans `uv.lock` with
|
||||||
|
[`uv-secure`](https://github.com/owenlamont/uv-secure) on every push and pull
|
||||||
|
request. A vulnerable dependency is upgraded to a fixed release; advisories
|
||||||
|
without one are tracked under
|
||||||
|
[Known security-gate blocks](docs/development.md#known-security-gate-blocks).
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="512" height="512" fill="none" stroke="url(#lg)" stroke-width="2" stroke-linecap="round" stroke-linejoin="round">
|
||||||
|
<!--
|
||||||
|
The raiw.cc brand mark: the Lucide "sparkle" and "x" glyphs.
|
||||||
|
Lucide, https://lucide.dev, ISC License, Copyright (c) Lucide Contributors.
|
||||||
|
Permission to use, copy, modify, and/or distribute this software for any purpose
|
||||||
|
with or without fee is hereby granted, provided that the above copyright notice
|
||||||
|
and this permission notice appear in all copies.
|
||||||
|
-->
|
||||||
|
<path d="M11.017 2.814a1 1 0 0 1 1.966 0l1.051 5.558a2 2 0 0 0 1.594 1.594l5.558 1.051a1 1 0 0 1 0 1.966l-5.558 1.051a2 2 0 0 0-1.594 1.594l-1.051 5.558a1 1 0 0 1-1.966 0l-1.051-5.558a2 2 0 0 0-1.594-1.594l-5.558-1.051a1 1 0 0 1 0-1.966l5.558-1.051a2 2 0 0 0 1.594-1.594z" fill="url(#lg)" fill-opacity="0.15"/>
|
||||||
|
<g transform="translate(1 17) scale(0.3)" stroke-width="5">
|
||||||
|
<path d="M18 6 6 18"/>
|
||||||
|
<path d="m6 6 12 12"/>
|
||||||
|
</g>
|
||||||
|
<defs>
|
||||||
|
<linearGradient id="lg" x1="0" y1="0" x2="24" y2="24">
|
||||||
|
<stop stop-color="#a78bfa"/>
|
||||||
|
<stop offset="1" stop-color="#06b6d4"/>
|
||||||
|
</linearGradient>
|
||||||
|
</defs>
|
||||||
|
</svg>
|
||||||
|
After Width: | Height: | Size: 1.2 KiB |
|
After Width: | Height: | Size: 2.0 MiB |
|
After Width: | Height: | Size: 2.0 MiB |
@@ -6,9 +6,9 @@ qwen,Alibaba Qwen consumer app,visible text,data/fixtures/visible/qwen/provider-
|
|||||||
kling,Kuaishou Kling AI,visible text,data/fixtures/visible/kling/provider-original-direct.png,available,authenticated Kling IMAGE 3.0 original; Latin KlingAI 3.0 mark detected at confidence 0.43; TC260 producer maps to Kuaishou,subject to Kling AI terms; noncommercial fixture
|
kling,Kuaishou Kling AI,visible text,data/fixtures/visible/kling/provider-original-direct.png,available,authenticated Kling IMAGE 3.0 original; Latin KlingAI 3.0 mark detected at confidence 0.43; TC260 producer maps to Kuaishou,subject to Kling AI terms; noncommercial fixture
|
||||||
yuanbao,Tencent Yuanbao,visible two-line text,data/fixtures/visible/yuanbao/provider-original.png,available,authenticated Yuanbao original; detected by Yuanbao engine at confidence 0.49; TC260 producer metadata maps to Tencent,repository
|
yuanbao,Tencent Yuanbao,visible two-line text,data/fixtures/visible/yuanbao/provider-original.png,available,authenticated Yuanbao original; detected by Yuanbao engine at confidence 0.49; TC260 producer metadata maps to Tencent,repository
|
||||||
samsung,Samsung Galaxy AI,visible Italian text,data/calibration/samsung/samsung_gray_1.png,available,original provider capture and detector calibration,repository
|
samsung,Samsung Galaxy AI,visible Italian text,data/calibration/samsung/samsung_gray_1.png,available,original provider capture and detector calibration,repository
|
||||||
runninghub,RunningHub,visible text,,missing,no redistributable provider original found,
|
runninghub,RunningHub,visible text,data/fixtures/visible/runninghub/example.png,partial,synthetic detector fixture available; no redistributable provider original found,repository
|
||||||
baidu,Baidu,visible text and tag,data/fixtures/visible/baidu/provider-original.jpeg,available,detected by Baidu engine; distributed under the OpenNoMark repository's CC BY-NC-SA 4.0 license,CC-BY-NC-SA-4.0; excluded from Apache-2.0
|
baidu,Baidu,visible text and tag,data/fixtures/visible/baidu/provider-original.jpeg,available,detected by Baidu engine; distributed under the OpenNoMark repository's CC BY-NC-SA 4.0 license,CC-BY-NC-SA-4.0; excluded from Apache-2.0
|
||||||
liblib,LiblibAI,visible wordmark,data/fixtures/visible/liblib/current-provider-output.png,partial,"authenticated current LiblibAI original carries a top-left AI生成 pill rather than the registered historical bottom-center LiblibAI wordmark; TC260 producer metadata maps to LiblibAI; historical positive remains missing",repository
|
liblib,LiblibAI,visible wordmark,data/fixtures/visible/liblib/current-provider-output.png,partial,"authenticated current LiblibAI original carries a top-left AI生成 pill rather than the registered historical bottom-center LiblibAI wordmark; TC260 producer metadata maps to LiblibAI; a synthetic historical-wordmark fixture is available at data/fixtures/visible/liblib/example.png, but a historical provider positive remains missing",repository
|
||||||
microsoft,Microsoft image output,visible top-right AI badge,data/fixtures/visible/microsoft/provider-original.png,available,authenticated Copilot original detected by Microsoft engine at confidence 0.54; also has valid C2PA and InvisMark,repository
|
microsoft,Microsoft image output,visible top-right AI badge,data/fixtures/visible/microsoft/provider-original.png,available,authenticated Copilot original detected by Microsoft engine at confidence 0.54; also has valid C2PA and InvisMark,repository
|
||||||
jimeng_pill,ByteDance Jimeng,visible AI pill,data/fixtures/visible/jimeng_pill/provider-published-example.jpg,available,"real Jimeng output embedded in page 29 of Zhonglun W&D's 2026 Bank Finance Legal Newsletter; direct jimeng_pill detection confidence 0.28; the same image also carries the Jimeng wordmark",fair-use research fixture; excluded from Apache-2.0
|
jimeng_pill,ByteDance Jimeng,visible AI pill,data/fixtures/visible/jimeng_pill/provider-published-example.jpg,available,"real Jimeng output embedded in page 29 of Zhonglun W&D's 2026 Bank Finance Legal Newsletter; direct jimeng_pill detection confidence 0.28; the same image also carries the Jimeng wordmark",fair-use research fixture; excluded from Apache-2.0
|
||||||
google_synthid,Google,image SynthID,data/synthid/originals/Gemini_Generated_Image_633uuy633uuy633u.png,available,provider oracle recorded in data/synthid/manifest.csv,repository
|
google_synthid,Google,image SynthID,data/synthid/originals/Gemini_Generated_Image_633uuy633uuy633u.png,available,provider oracle recorded in data/synthid/manifest.csv,repository
|
||||||
|
|||||||
|
@@ -1,8 +1,8 @@
|
|||||||
key,provider,watermark_type,path,status,verification,license
|
key,provider,watermark_type,path,status,verification,license
|
||||||
sora,OpenAI Sora,visible moving mascot and wordmark,data/fixtures/visible/sora/provider-original.mp4,available,"Apache-2.0 upstream original; detected by Sora engine on 300/300 frames; C2PA claim names Sora and OpenAI/Truepic, with valid integrity and signature but an expired untrusted signer certificate",Apache-2.0
|
sora,OpenAI Sora,visible moving mascot and wordmark,data/fixtures/visible/sora/provider-original.mp4,available,"Apache-2.0 upstream original; detected by Sora engine on 300/300 frames; C2PA claim names Sora and OpenAI/Truepic, with valid integrity and signature but an expired untrusted signer certificate",Apache-2.0
|
||||||
veo,Google Veo,visible fixed text,data/fixtures/visible/veo/provider-original.mp4,available,"authenticated Google Flow Veo 3.1 Fast original; detected by Veo engine on 192/192 frames; valid C2PA and provider-confirmed SynthID",repository
|
veo,Google Veo,visible fixed text,data/fixtures/visible/veo/provider-original.mp4,available,"authenticated Google Flow Veo 3.1 Fast original; detected by Veo engine on 192/192 frames; valid C2PA and provider-confirmed SynthID",repository
|
||||||
seedance,ByteDance Seedance,visible boxed AI label,,missing,"OpenRouter Seedance 2.0 Mini with watermark=true produced BytePlus C2PA but no visible boxed AI label; direct consumer original still needed",
|
seedance,ByteDance Seedance,visible boxed AI label,data/fixtures/visible/seedance/example.mp4,partial,"synthetic detector fixture available; OpenRouter Seedance 2.0 Mini with watermark=true produced BytePlus C2PA but no visible boxed AI label; direct consumer original still needed",repository
|
||||||
doubao,ByteDance Doubao,visible fixed text,,missing,no redistributable provider original found,
|
doubao,ByteDance Doubao,visible fixed text,data/fixtures/visible/doubao/example.mp4,partial,"synthetic detector fixture available; a real public carrier is retained locally but is not publication-cleared",repository
|
||||||
dola,ByteDance Dola,visible fixed text,,missing,no redistributable provider original found,
|
dola,ByteDance Dola,visible fixed text,data/fixtures/visible/dola/example.mp4,partial,synthetic detector fixture available; no redistributable provider original found,repository
|
||||||
hailuo,MiniMax Hailuo AI,visible composite label,data/fixtures/visible/hailuo/provider-original.mp4,available,"authenticated Hailuo AI MiniMax H3 original downloaded with With Watermark selected; detected by Hailuo engine on 124/124 frames; TC260 AIGC label names MiniMax",repository
|
hailuo,MiniMax Hailuo AI,visible composite label,data/fixtures/visible/hailuo/provider-original.mp4,available,"authenticated Hailuo AI MiniMax H3 original downloaded with With Watermark selected; detected by Hailuo engine on 124/124 frames; TC260 AIGC label names MiniMax",repository
|
||||||
kling,Kuaishou Kling AI,visible fixed logo and wordmark,data/fixtures/visible/kling/provider-original.mp4,available,"authenticated Kling AI VIDEO 3.0 original; detected by Kling engine on 121/121 frames; no locally readable provenance metadata",Kling AI terms; noncommercial fixture
|
kling,Kuaishou Kling AI,visible fixed logo and wordmark,data/fixtures/visible/kling/provider-original.mp4,available,"authenticated Kling AI VIDEO 3.0 original; detected by Kling engine on 121/121 frames; no container provenance metadata, but the Kling H.264 SEI tag (kling_sei) is readable",Kling AI terms; noncommercial fixture
|
||||||
|
@@ -26,6 +26,11 @@ SkillsMP indexes public GitHub `SKILL.md` files and does not need a separate
|
|||||||
upload. ClawHub and the Claude community marketplace are explicit publishes;
|
upload. ClawHub and the Claude community marketplace are explicit publishes;
|
||||||
see [Release and distribution](release-and-distribution.md#agent-skill).
|
see [Release and distribution](release-and-distribution.md#agent-skill).
|
||||||
|
|
||||||
|
Every maintainer skill under `.claude/skills/` carries `metadata.internal: true`.
|
||||||
|
The skills CLI discovers every `SKILL.md` in that directory, so without the flag
|
||||||
|
`npx skills add` offers it and skills.sh lists it beside the published skill.
|
||||||
|
`tests/test_agent_skill.py` fails on a project skill that lacks it.
|
||||||
|
|
||||||
## Where the skill can actually be listed
|
## Where the skill can actually be listed
|
||||||
|
|
||||||
Most skill catalogs are install registries, not discovery engines. A listing
|
Most skill catalogs are install registries, not discovery engines. A listing
|
||||||
@@ -47,24 +52,44 @@ Do these in order after the skill is on the default branch.
|
|||||||
people run `npx skills add`. The listing is live and has recorded real
|
people run `npx skills add`. The listing is live and has recorded real
|
||||||
installs; the README install line remains the seed, not a registration.
|
installs; the README install line remains the seed, not a registration.
|
||||||
4. **Awesome-list PRs.** Hand-curated GitHub lists are how people browse skills
|
4. **Awesome-list PRs.** Hand-curated GitHub lists are how people browse skills
|
||||||
today. One accurate PR per list, with the scope sentence intact.
|
today. One accurate PR per list, with the scope sentence intact. This list
|
||||||
- `ComposioHQ/awesome-claude-skills`: submitted 2026-09-05 as PR 1841
|
keeps each list's entry rules and the submission; read live review state
|
||||||
(Creative & Media); no maturity bar in its contribution rules. Rechecked
|
from GitHub (`gh search prs --author @me --state open remove-ai-watermarks`
|
||||||
2026-09-11: still open, CI green, no review comments, waiting on the
|
lists every open one), not from here.
|
||||||
list maintainers.
|
- `ComposioHQ/awesome-claude-skills`: submitted 2026-09-05 as
|
||||||
- `heilcheng/awesome-agent-skills`: submitted 2026-09-09 as PR 483 under
|
[PR 1841](https://github.com/ComposioHQ/awesome-claude-skills/pull/1841)
|
||||||
|
(Creative & Media); no maturity bar in its contribution rules.
|
||||||
|
- `heilcheng/awesome-agent-skills`: submitted 2026-09-09 as
|
||||||
|
[PR 483](https://github.com/heilcheng/awesome-agent-skills/pull/483) under
|
||||||
Community Skills > Media. The skill's three concrete input/output examples
|
Community Skills > Media. The skill's three concrete input/output examples
|
||||||
satisfy the list's working-example requirement. Rechecked 2026-09-11:
|
satisfy the list's working-example requirement. The maintainer put it
|
||||||
still open; the only check failure is their Vercel deploy authorization,
|
`on hold` on 2026-10-15: "Tool strips AI provenance (SynthID/C2PA); policy
|
||||||
|
call for maintainer". Its Vercel check fails on deploy authorization,
|
||||||
which a contributor cannot grant.
|
which a contributor cannot grant.
|
||||||
- `VoltAgent/awesome-agent-skills`: DEFERRED. Its CONTRIBUTING rejects
|
- `VoltAgent/awesome-agent-skills`: submitted 2026-10-17 as
|
||||||
brand-new skills ("give your skill time to mature and gain users before
|
[PR 1225](https://github.com/VoltAgent/awesome-agent-skills/pull/1225), a
|
||||||
submitting"). The skill now has real installs and shipped releases, but
|
one-line entry. Its CONTRIBUTING requires "real community usage" ("Give
|
||||||
adoption remains too small to make the list's maturity claim comfortably.
|
your skill time to mature and gain users before submitting").
|
||||||
- `VoltAgent/awesome-openclaw-skills`: DEFERRED on adoption only. Entries
|
- `VoltAgent/awesome-openclaw-skills`: DEFERRED on adoption only. Entries
|
||||||
are indexed via clawskills.sh, and the ClawHub prerequisite in step 6 is
|
are indexed via clawskills.sh, and the ClawHub prerequisite in step 6 is
|
||||||
now met. The list still requires proven real-world usage. Re-attempt once
|
now met. The list still requires proven real-world usage. Re-attempt once
|
||||||
the ClawHub listing shows organic installs.
|
the ClawHub listing shows organic installs.
|
||||||
|
- `hashgraph-online/awesome-codex-plugins`: merged 2026-10-19 as
|
||||||
|
[PR 541](https://github.com/hashgraph-online/awesome-codex-plugins/pull/541).
|
||||||
|
It lists only Codex plugins, so the entry depends on
|
||||||
|
`.codex-plugin/plugin.json` and the icon in `assets/icon.svg`. Its
|
||||||
|
contribution CI runs `scripts/validate-plugin-pr.py` against the
|
||||||
|
repository's `HEAD.zip`, then the HOL plugin scanner with a minimum score
|
||||||
|
of 80 and no high finding, as `continue-on-error`. The scanner's
|
||||||
|
`\beval\s*\(` rule also matches PyTorch's `.eval()`, which is why the code
|
||||||
|
calls `.train(False)`. Before changing the plugin, rescan with
|
||||||
|
`uvx --from plugin-scanner==3.49.0 plugin-scanner scan . --min-score 80 --fail-on-severity high`
|
||||||
|
(86 with no high finding on 2026-10-17; the loss is a tracked video over
|
||||||
|
the scanner's 16 MiB file budget).
|
||||||
|
- `punkpeye/awesome-mcp-servers`: submitted 2026-10-18 as
|
||||||
|
[PR 16924](https://github.com/punkpeye/awesome-mcp-servers/pull/16924) for
|
||||||
|
the MCP server. The list requires a Glama listing and the Glama score
|
||||||
|
badge; its bot checks both and labels the PR `has-glama`.
|
||||||
5. **Claude community marketplace.** Form at
|
5. **Claude community marketplace.** Form at
|
||||||
[platform.claude.com/plugins/submit](https://platform.claude.com/plugins/submit)
|
[platform.claude.com/plugins/submit](https://platform.claude.com/plugins/submit)
|
||||||
(individuals) or the claude.ai directory form (Team/Enterprise). Validate
|
(individuals) or the claude.ai directory form (Team/Enterprise). Validate
|
||||||
@@ -186,8 +211,8 @@ that names a command, flag or value the previous release lacked. Left behind, it
|
|||||||
CLI is current right before it types an option that build rejects.
|
CLI is current right before it types an option that build rejects.
|
||||||
|
|
||||||
Bump `metadata.version` in `SKILL.md` plus `version` in
|
Bump `metadata.version` in `SKILL.md` plus `version` in
|
||||||
`skills/.claude-plugin/plugin.json` and `.claude-plugin/marketplace.json` when
|
`skills/.claude-plugin/plugin.json`, `.claude-plugin/marketplace.json` and
|
||||||
the skill instructions change. That version is independent of the PyPI package
|
`.codex-plugin/plugin.json` when the skill instructions change. That version is independent of the PyPI package
|
||||||
version.
|
version.
|
||||||
|
|
||||||
## Behavior evals
|
## Behavior evals
|
||||||
|
|||||||
@@ -1597,6 +1597,38 @@ Local reproducibility artifacts are under
|
|||||||
`contamination-scan-2026-08-26`, and `taxonomy-continuation-clean-2026-08-26`
|
`contamination-scan-2026-08-26`, and `taxonomy-continuation-clean-2026-08-26`
|
||||||
run directories.
|
run directories.
|
||||||
|
|
||||||
|
## Modern frozen-backbone campaign closed, 2026-09-27
|
||||||
|
|
||||||
|
A second pre-registered representation campaign tested exact-revision,
|
||||||
|
Apache-2.0 frozen vision backbones with universal ridge heads against the same
|
||||||
|
product contract. The candidates were `google/siglip2-base-patch16-224` at
|
||||||
|
revision `75de2d55ec2d0b4efc50b3e9ad70dba96a7b2fa2` and
|
||||||
|
`facebook/PE-Core-B16-224` at revision
|
||||||
|
`a16450b46fef32363459920c2685a1b4ef13dcd9`. The protocol was frozen before
|
||||||
|
inference and used 28,743 unique source images. Its comparison cells contained
|
||||||
|
1,873 AI-test images, 3,000 fresh Open Images photographs, 24 Kodak images, and
|
||||||
|
300 FLUX images.
|
||||||
|
|
||||||
|
| Model at its frozen operating point | AI test | Fresh photo false positives | Kodak false positives | FLUX |
|
||||||
|
| --- | ---: | ---: | ---: | ---: |
|
||||||
|
| Production Model 1 baseline | 1,676/1,873 (89.5%) | 35/3,000 (1.17%) | 0/24 | 256/300 (85.3%) |
|
||||||
|
| SigLIP2 universal ridge | 1,602/1,873 (85.5%) | 74/3,000 (2.47%) | 0/24 | 125/300 (41.7%) |
|
||||||
|
| PE-Core B universal ridge | 1,715/1,873 (91.6%) | 86/3,000 (2.87%) | 0/24 | 170/300 (56.7%) |
|
||||||
|
|
||||||
|
SigLIP2 failed the AI-test, fresh-photo, and FLUX gates. PE-Core improved the
|
||||||
|
AI-test cell but failed the fresh-photo and FLUX gates. Matching the baseline's
|
||||||
|
exact 35-false-positive photo budget after the fact did not recover the lost
|
||||||
|
coverage: SigLIP2 reached 1,505/1,873 AI and 98/300 FLUX, PE-Core universal
|
||||||
|
reached 1,625/1,873 and 133/300, and PE-Core photo-only reached 1,709/1,873 and
|
||||||
|
97/300. No threshold matched the baseline photo budget, AI coverage, and FLUX
|
||||||
|
coverage simultaneously.
|
||||||
|
|
||||||
|
The bounded newer-backbone hypothesis is therefore closed as a Model 1
|
||||||
|
replacement. Keep `clip-l-ft.pt`, its ONNX graph, hashes, and Hub revision
|
||||||
|
unchanged. Reproducibility artifacts are local under
|
||||||
|
`.local-eval/synthid/ai-photo-2026-08-22/modern-representation-campaign-2026-09-27/`;
|
||||||
|
the campaign did not enter the publication stage.
|
||||||
|
|
||||||
### Wild extras, not SynthID
|
### Wild extras, not SynthID
|
||||||
|
|
||||||
| Hypothesis | 2026-08-23 | Use |
|
| Hypothesis | 2026-08-23 | Use |
|
||||||
|
|||||||
@@ -28,6 +28,7 @@ defaults. This page focuses on choosing the right command.
|
|||||||
| `video invisible` and `video all --invisible` | `remove-ai-watermarks[video,diffusion]` plus ffmpeg on PATH |
|
| `video invisible` and `video all --invisible` | `remove-ai-watermarks[video,diffusion]` plus ffmpeg on PATH |
|
||||||
| HEIC/HEIF/AVIF pixel input | Add `remove-ai-watermarks[heif]` |
|
| HEIC/HEIF/AVIF pixel input | Add `remove-ai-watermarks[heif]` |
|
||||||
| Every production command and Python-compatible backend | `remove-ai-watermarks[all]` |
|
| Every production command and Python-compatible backend | `remove-ai-watermarks[all]` |
|
||||||
|
| `mcp` | Default package; each tool needs the extra of the command it runs |
|
||||||
|
|
||||||
`batch` requires the same extra as its selected mode. Extras can be combined in
|
`batch` requires the same extra as its selected mode. Extras can be combined in
|
||||||
one installation, for example `remove-ai-watermarks[visible,detect,heif]`.
|
one installation, for example `remove-ai-watermarks[visible,detect,heif]`.
|
||||||
@@ -469,6 +470,11 @@ detected. Use `--force` when you know the image should be processed:
|
|||||||
remove-ai-watermarks invisible image.png -o clean.png --force
|
remove-ai-watermarks invisible image.png -o clean.png --force
|
||||||
```
|
```
|
||||||
|
|
||||||
|
For transparent inputs, use PNG or WebP output to retain transparency. The engine
|
||||||
|
preserves the original alpha channel, including partially transparent edges,
|
||||||
|
while regenerating RGB pixels. JPEG output cannot preserve transparency. This
|
||||||
|
also applies to `batch --mode invisible`.
|
||||||
|
|
||||||
### Choose a strength cohort
|
### Choose a strength cohort
|
||||||
|
|
||||||
`--vendor` selects the cohort the default strength resolves from. `auto`
|
`--vendor` selects the cohort the default strength resolves from. `auto`
|
||||||
@@ -685,12 +691,23 @@ For `visible` and `all` modes, the final summary includes counts for
|
|||||||
validation results are printed explicitly, but the once-filled outputs are kept
|
validation results are printed explicitly, but the once-filled outputs are kept
|
||||||
and the batch does not retry them automatically.
|
and the batch does not retry them automatically.
|
||||||
|
|
||||||
|
## Serve the CLI to agent clients
|
||||||
|
|
||||||
|
```bash
|
||||||
|
remove-ai-watermarks mcp
|
||||||
|
```
|
||||||
|
|
||||||
|
Runs a Model Context Protocol server on stdin and stdout whose `identify`,
|
||||||
|
`remove` and `batch` tools each run one command of this CLI. Client setup and
|
||||||
|
the tool contract are in the [MCP server guide](mcp-server.md).
|
||||||
|
|
||||||
## Exit behavior
|
## Exit behavior
|
||||||
|
|
||||||
The CLI uses nonzero exit codes for meaningful incomplete outcomes, including
|
The CLI uses nonzero exit codes for meaningful incomplete outcomes, including
|
||||||
no detected target on commands that would otherwise regenerate or create a
|
no detected target on commands that would otherwise regenerate or create a
|
||||||
misleading unchanged result, processing errors, and a required invisible step
|
misleading unchanged result, a batch directory with nothing to process,
|
||||||
that could not run.
|
processing errors, an input that is not a readable image or video, and a
|
||||||
|
required invisible step that could not run.
|
||||||
|
|
||||||
Scripts should check the process exit code and the output path. The detailed
|
Scripts should check the process exit code and the output path. The detailed
|
||||||
per-command contract is maintained in
|
per-command contract is maintained in
|
||||||
|
|||||||
@@ -14,7 +14,7 @@ Read this reference for environment setup, dependency recovery, CI behavior, and
|
|||||||
- On an unreliable connection, sync `dev` plus only the required feature extras, such as `diffusion`, and run the checks directly instead of downloading every optional learned backend.
|
- On an unreliable connection, sync `dev` plus only the required feature extras, such as `diffusion`, and run the checks directly instead of downloading every optional learned backend.
|
||||||
- Run `uv` from the repository root or it may create a bare environment without the project dependencies.
|
- Run `uv` from the repository root or it may create a bare environment without the project dependencies.
|
||||||
|
|
||||||
`maintain.sh` starts with `uv sync --all-extras`, so a lean environment is replaced whenever the gate runs. Syncing back down can leave `opencv-python-headless` half-removed: `import cv2` then succeeds against an empty namespace while `cv2.INTER_LINEAR` is gone, which surfaces as a suite that stalls or reports hundreds of unrelated failures. Repair it with `uv sync --frozen --extra dev --reinstall-package opencv-python-headless` rather than re-reading the diff.
|
`maintain.sh` starts with `uv sync --all-extras`, so a lean environment is replaced whenever the gate runs. Syncing back down can leave `opencv-python-headless` half-removed: `import cv2` then succeeds against an empty namespace while `cv2.INTER_LINEAR` is gone, which surfaces as a suite that stalls or reports hundreds of unrelated failures. Repair it with `uv sync --frozen --extra dev --reinstall-package opencv-python-headless` rather than re-reading the diff. A stale worktree environment can fail differently: `import cv2` raises `partially initialized module 'cv2' has no attribute 'gapi_wip_gst_GStreamerPipeline'`, because the lock installs `opencv-contrib-python` 4.x beside `opencv-python` 5.x into one `cv2` directory. Reinstalling the three OpenCV packages did not fix that on 2026-10-17; deleting the worktree's `.venv` and running `uv sync --frozen --all-extras` did.
|
||||||
|
|
||||||
Pyright is only meaningful with the extras its imports need. Scoped to `src/`, a lean environment reports `TrustMark`, `transformers` and `huggingface_hub` as unresolved, which reads as a type error and is not one; add `--extra trustmark --extra classify` before believing it.
|
Pyright is only meaningful with the extras its imports need. Scoped to `src/`, a lean environment reports `TrustMark`, `transformers` and `huggingface_hub` as unresolved, which reads as a type error and is not one; add `--extra trustmark --extra classify` before believing it.
|
||||||
|
|
||||||
@@ -29,7 +29,7 @@ Do not add advisory ignores or suppress the scanner's exit status. Run and
|
|||||||
report Ruff, Pyright scoped to `src/`, and tests separately when security blocks
|
report Ruff, Pyright scoped to `src/`, and tests separately when security blocks
|
||||||
the gate.
|
the gate.
|
||||||
|
|
||||||
Rechecked on 2026-09-10 against the locked versions and the advisory sources:
|
Rechecked on 2026-10-02 against the locked versions and the advisory sources:
|
||||||
|
|
||||||
- `lightning` and `pytorch-lightning`, pulled by the optional `trustmark`
|
- `lightning` and `pytorch-lightning`, pulled by the optional `trustmark`
|
||||||
extra, are pinned to 2.6.6. That release clears PYSEC-2026-3624 and
|
extra, are pinned to 2.6.6. That release clears PYSEC-2026-3624 and
|
||||||
@@ -37,18 +37,19 @@ Rechecked on 2026-09-10 against the locked versions and the advisory sources:
|
|||||||
- `accelerate`, required by the diffusion stack, is affected by
|
- `accelerate`, required by the diffusion stack, is affected by
|
||||||
[GHSA-4j2p-28q2-5m79](https://github.com/advisories/GHSA-4j2p-28q2-5m79)
|
[GHSA-4j2p-28q2-5m79](https://github.com/advisories/GHSA-4j2p-28q2-5m79)
|
||||||
and PYSEC-2026-3804. The sharded-checkpoint loader accepts unsafe
|
and PYSEC-2026-3804. The sharded-checkpoint loader accepts unsafe
|
||||||
`weight_map` paths; neither advisory lists a patched release. 1.15.0 is
|
`weight_map` paths; neither advisory lists a patched release. The lock
|
||||||
on PyPI and is outside the advisory's `<= 1.14.0` range, but its
|
carries 1.15.0, which is outside the advisory's `<= 1.14.0` range, so the
|
||||||
|
scanner no longer blocks the gate. Its
|
||||||
`load_checkpoint_in_model` still joins `weight_map` entries with no
|
`load_checkpoint_in_model` still joins `weight_map` entries with no
|
||||||
containment check, and the sanitizer PRs
|
containment check, and the sanitizer PRs
|
||||||
([#4138](https://github.com/huggingface/accelerate/pull/4138),
|
([#4138](https://github.com/huggingface/accelerate/pull/4138),
|
||||||
[#4214](https://github.com/huggingface/accelerate/pull/4214)) did not
|
[#4214](https://github.com/huggingface/accelerate/pull/4214)) were closed
|
||||||
land. Do not bump solely to silence the scanner. The project has no
|
without merging. Do not treat the passing scan as a fix. The project has no
|
||||||
direct call to `load_checkpoint_in_model` or
|
direct call to `load_checkpoint_in_model` or
|
||||||
`load_checkpoint_and_dispatch`, but this does not prove indirect
|
`load_checkpoint_and_dispatch`, but this does not prove indirect
|
||||||
model-loading paths are unaffected.
|
model-loading paths are unaffected.
|
||||||
|
|
||||||
Before carrying the remaining block forward, rerun `uvx uv-secure uv.lock` and check
|
When a new block appears, or before carrying one forward, rerun `uvx uv-secure uv.lock` and check
|
||||||
PyPI for a fixed release. Upgrade a fixed package with
|
PyPI for a fixed release. Upgrade a fixed package with
|
||||||
`uv lock --upgrade-package <package>` and rerun the gate. Removing the current
|
`uv lock --upgrade-package <package>` and rerun the gate. Removing the current
|
||||||
dependency would remove a supported optional feature, so that is not a
|
dependency would remove a supported optional feature, so that is not a
|
||||||
@@ -98,6 +99,11 @@ run the full-clip video test. Diffusion and model-running tests skip in that
|
|||||||
matrix; metadata, identification, visible removal, the DWT-DCT decoder, and the
|
matrix; metadata, identification, visible removal, the DWT-DCT decoder, and the
|
||||||
OpenCV eraser remain covered across operating systems.
|
OpenCV eraser remain covered across operating systems.
|
||||||
|
|
||||||
|
A separate `security` job runs the `maintain.sh` lockfile scan, so an advisory
|
||||||
|
without a released fix turns `main` red; see
|
||||||
|
[Known security-gate blocks](#known-security-gate-blocks). Vulnerability
|
||||||
|
reporting is described in [`SECURITY.md`](../SECURITY.md).
|
||||||
|
|
||||||
Keep `uv.lock` compatible with `uv sync --frozen`. Dependency pull-request checks use GitHub's merge result against current `main`; if `main` moves, merge it locally and rerun the full gate because a newer linter can expose stale directives in later code.
|
Keep `uv.lock` compatible with `uv sync --frozen`. Dependency pull-request checks use GitHub's merge result against current `main`; if `main` moves, merge it locally and rerun the full gate because a newer linter can expose stale directives in later code.
|
||||||
|
|
||||||
`test.yml` runs one concurrency group per ref with `cancel-in-progress: true`, so a newer push cancels the previous run mid-flight. A cancelled run is not a failure verdict: the superseding run's head includes the cancelled commits as ancestors, so watch that run to completion instead of retrying the push.
|
`test.yml` runs one concurrency group per ref with `cancel-in-progress: true`, so a newer push cancels the previous run mid-flight. A cancelled run is not a failure verdict: the superseding run's head includes the cancelled commits as ancestors, so watch that run to completion instead of retrying the push.
|
||||||
|
|||||||
@@ -27,7 +27,8 @@ to run the tool. Use the maintainer references only when changing the code.
|
|||||||
| [Code provenance](code-provenance.md) | Required notices for licensed derivative work. |
|
| [Code provenance](code-provenance.md) | Required notices for licensed derivative work. |
|
||||||
| [Verification plan](verification-plan.md) | Verification methods, completed measurements, and remaining validation gaps. |
|
| [Verification plan](verification-plan.md) | Verification methods, completed measurements, and remaining validation gaps. |
|
||||||
| [Release and distribution](release-and-distribution.md) | PyPI, Homebrew, Hugging Face Space, photo-classify Hub model, ComfyUI Registry, and the release workflow. |
|
| [Release and distribution](release-and-distribution.md) | PyPI, Homebrew, Hugging Face Space, photo-classify Hub model, ComfyUI Registry, and the release workflow. |
|
||||||
| [Agent skill](agent-skill.md) | Install and maintain the published Agent Skill and Claude plugin. |
|
| [Agent skill](agent-skill.md) | Install and maintain the published Agent Skill, the Claude plugin and the Codex plugin. |
|
||||||
|
| [MCP server](mcp-server.md) | Connect an MCP client to `remove-ai-watermarks mcp`, or publish it to the MCP registry. |
|
||||||
| [Watermarking landscape](watermarking-landscape.md) | Vendor signals and detection approaches. |
|
| [Watermarking landscape](watermarking-landscape.md) | Vendor signals and detection approaches. |
|
||||||
| [SynthID technical reference](synthid.md) | Mechanism, provenance, robustness, regeneration. |
|
| [SynthID technical reference](synthid.md) | Mechanism, provenance, robustness, regeneration. |
|
||||||
|
|
||||||
|
|||||||
@@ -4,6 +4,21 @@ This page describes current product limits. Historical measurements and
|
|||||||
superseded experiments live in the research archive listed in
|
superseded experiments live in the research archive listed in
|
||||||
[the documentation index](index.md).
|
[the documentation index](index.md).
|
||||||
|
|
||||||
|
## What each path leaves behind
|
||||||
|
|
||||||
|
No command produces a "clean" verdict. Each one removes a specific layer, and
|
||||||
|
the remaining layers need their own check.
|
||||||
|
|
||||||
|
| Path | Removes | May remain | How to check what remains |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `metadata --remove`, `video metadata --remove` | Supported AI metadata: C2PA manifests, TC260 labels, IPTC AI fields, app-export JSON, generation parameters | Every pixel and audio watermark (SynthID, Meta Content Seal, InvisMark, TrustMark, DWT-DCT). A C2PA soft binding can still lead a resolver to a manifest stored remotely. Strings inside encoded video streams, such as H.264 SEI messages, stay too, except Kling's SEI tag, which `video metadata --remove` blanks | `identify` re-reads the metadata. Only the provider's verifier can confirm a pixel watermark. The pixels are not regenerated, so an AI-image classifier scores the output the same as the input |
|
||||||
|
| `visible`, `erase`, `video visible` | The selected label region, refilled. AI metadata too, unless `--keep-metadata` is given | Any invisible watermark across the rest of the frame. The fill can leave a visible patch on structured backgrounds | Look at the output. Use `identify` for metadata and the provider's verifier for invisible marks |
|
||||||
|
| `invisible`, `all`, `video invisible` | Regenerates pixels to disrupt invisible watermarks. `all` also removes visible marks and metadata | No local decoder confirms that SynthID is gone from a given file, and the outcome depends on content and seed. `video invisible` copies the audio track unchanged, so an audio watermark survives | The provider's verifier, such as Google's SynthID Detector or OpenAI's provenance check. A pass there says nothing about AI-image classifiers |
|
||||||
|
| `identify`, `video identify` | Nothing | Not applicable | `unknown` means no supported local signal was found. It is not a clean verdict |
|
||||||
|
|
||||||
|
AI-image classifiers are a separate question from provenance. They score how
|
||||||
|
the pixels look rather than read a mark, and none of these paths targets them.
|
||||||
|
|
||||||
## Pixel classification
|
## Pixel classification
|
||||||
|
|
||||||
`classify` / `classify_pixels` is AI versus camera, not a universal AI detector.
|
`classify` / `classify_pixels` is AI versus camera, not a universal AI detector.
|
||||||
@@ -179,9 +194,12 @@ fresh-font and silhouette variants visibly changed typography. The higher-fideli
|
|||||||
and line geometry. It builds its donor internally but still requires an independently
|
and line geometry. It builds its donor internally but still requires an independently
|
||||||
clean global anchor. Automatic OCR and line-box proposals are not reliable enough to
|
clean global anchor. Automatic OCR and line-box proposals are not reliable enough to
|
||||||
remove those requirements, and exact oracle results do not establish a general mask,
|
remove those requirements, and exact oracle results do not establish a general mask,
|
||||||
seed, runtime, or provider operating range. Qwen-Image-2.0 is hosted-only and exposes
|
seed, runtime, or provider operating range. Qwen-Image-2.0 is hosted-only. Qwen-Image-2.1
|
||||||
no equivalent low-strength denoise control. Exact experiments, controls, and pass
|
(issue #111) has open weights, but its Diffusers pipeline uses `image=` only as
|
||||||
rates are kept in
|
reference conditioning and starts the target latents from noise, with no `strength`
|
||||||
|
or latent noise-mask input. Its Qwen Research License also limits the weights to
|
||||||
|
non-commercial research and evaluation, so it is not a supported-profile candidate.
|
||||||
|
Exact experiments, controls, and pass rates are kept in
|
||||||
[`text-protection-research.md`](text-protection-research.md) and the
|
[`text-protection-research.md`](text-protection-research.md) and the
|
||||||
[`fidelity` evaluation record](../data/evaluations/fidelity/README.md).
|
[`fidelity` evaluation record](../data/evaluations/fidelity/README.md).
|
||||||
|
|
||||||
@@ -260,7 +278,9 @@ guarantee, and check the audio separately when it matters. The question is no
|
|||||||
longer Google-only: since 2026 OpenAI (ChatGPT and API audio) and ElevenLabs
|
longer Google-only: since 2026 OpenAI (ChatGPT and API audio) and ElevenLabs
|
||||||
also embed SynthID in generated audio, and Microsoft lists provenance for its
|
also embed SynthID in generated audio, and Microsoft lists provenance for its
|
||||||
Azure speech and GPT audio models, so voice-over or music from those services
|
Azure speech and GPT audio models, so voice-over or music from those services
|
||||||
muxed into any video survives this path unchanged.
|
muxed into any video survives this path unchanged. Suno announced audio
|
||||||
|
watermarking and fingerprinting on 2026-08-06 without naming the technology, so
|
||||||
|
Suno music belongs on the same list.
|
||||||
|
|
||||||
The public video results make that boundary explicit without decoding audio:
|
The public video results make that boundary explicit without decoding audio:
|
||||||
`visual_invisible_action` records whether video pixels were regenerated, while
|
`visual_invisible_action` records whether video pixels were regenerated, while
|
||||||
@@ -397,12 +417,20 @@ global Qwen stage is tiled; the face stage runs after tile blending.
|
|||||||
every profile, streaming weights instead of pinning them. It reduces CUDA memory
|
every profile, streaming weights instead of pinning them. It reduces CUDA memory
|
||||||
pressure at the cost of speed.
|
pressure at the cost of speed.
|
||||||
|
|
||||||
It reaches the global stack of `qwen-zimage` only. That stack is each
|
It reaches the global stack of `qwen-zimage` and `sdxl-zimage`. That stack is
|
||||||
profile's own, and `chroma-zimage` and `sdxl-zimage` load theirs with a plain
|
each profile's own, and `chroma-zimage` loads its stack with a plain `.to(device)`.
|
||||||
`.to(device)`. On a card below the face-stage residency floor, where the face
|
On a card below the face-stage residency floor, where the face stack was already
|
||||||
stack was already offloading, the flag changes nothing at all for those two: the
|
offloading, the flag changes nothing at all for that profile: the run behaves as
|
||||||
run behaves as if it were absent. The library says so on the way past, as a
|
if it were absent. The library says so on the way past, as a warning ahead of the
|
||||||
warning ahead of the model load.
|
model load.
|
||||||
|
|
||||||
|
`sdxl-zimage` streams its global stack without being asked on a card below
|
||||||
|
`RESIDENT_SDXL_GLOBAL_MIN_VRAM_GIB` (12 GiB). Its UNet and Canny ControlNet run
|
||||||
|
together on every step, so the stack does not fit an 8 GiB card even with
|
||||||
|
model-level offload. A resident load there does not always fail loudly: under
|
||||||
|
the Windows display driver, including WSL2, CUDA can spill the overflow into
|
||||||
|
shared system memory, and the run goes on at a fraction of the speed instead of
|
||||||
|
raising an out-of-memory error.
|
||||||
|
|
||||||
Residency is otherwise chosen from the card's total VRAM. On a card large enough
|
Residency is otherwise chosen from the card's total VRAM. On a card large enough
|
||||||
to hold a stack, offloading is pure waste: DiffSynth drops weights to the meta
|
to hold a stack, offloading is pure waste: DiffSynth drops weights to the meta
|
||||||
@@ -593,8 +621,9 @@ stream safely.
|
|||||||
raising. User facing callers must use `strip_and_verify` and inspect its
|
raising. User facing callers must use `strip_and_verify` and inspect its
|
||||||
surviving marker mapping before reporting success. `strip_and_verify` recovers
|
surviving marker mapping before reporting success. `strip_and_verify` recovers
|
||||||
when `image_io` can still decode the raster by normalizing the container and
|
when `image_io` can still decode the raster by normalizing the container and
|
||||||
checking again. A truly undecodable file still reports the surviving markers.
|
checking again. A truly undecodable file raises `ValueError` before anything is
|
||||||
The CLI uses this verified path.
|
written (since 2026-10-18; it used to be copied through with its surviving
|
||||||
|
markers). The CLI uses this verified path.
|
||||||
|
|
||||||
### Sixteen bit PNG metadata stripping
|
### Sixteen bit PNG metadata stripping
|
||||||
|
|
||||||
|
|||||||
@@ -46,6 +46,21 @@ It also does not add templates for regulator-defined disclosure icons, such as
|
|||||||
the EU icons in Annex 1 of the Code of Practice on marking and labelling of
|
the EU icons in Annex 1 of the Code of Practice on marking and labelling of
|
||||||
AI-generated content, which deployers use to disclose deepfakes.
|
AI-generated content, which deployers use to disclose deepfakes.
|
||||||
|
|
||||||
|
It does not remove product branding that a paid plan of the same product turns
|
||||||
|
off, such as the "Made with Gamma" badge. A visible mark stays in scope when its
|
||||||
|
vendor describes it as identifying AI-generated content, whichever plan turns it
|
||||||
|
off: the Gemini sparkle, or the
|
||||||
|
[visible watermark](https://support.google.com/gemininotebook/answer/16213268)
|
||||||
|
that Gemini Notebook, formerly NotebookLM, adds to generated images and videos.
|
||||||
|
|
||||||
|
Watermarks in LLM-generated text are out of scope. The Claude and OpenAI text
|
||||||
|
marks change which words the model chooses and add no hidden characters
|
||||||
|
([Anthropic](https://www.anthropic.com/news/claude-text-watermark),
|
||||||
|
[OpenAI](https://help.openai.com/en/articles/8912793-provenance-signals-in-openai-generated-content)).
|
||||||
|
Stripping invisible Unicode does not touch them, only a substantial rewrite
|
||||||
|
removes them, and their detectors are not public, so no removal could be
|
||||||
|
verified.
|
||||||
|
|
||||||
The `erase` command is a generic region tool. Users are responsible for having
|
The `erase` command is a generic region tool. Users are responsible for having
|
||||||
the right to edit the selected content.
|
the right to edit the selected content.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# MCP server
|
||||||
|
|
||||||
|
`remove-ai-watermarks mcp` runs a [Model Context Protocol](https://modelcontextprotocol.io/)
|
||||||
|
server on stdin and stdout, so an agent client can identify and clean the
|
||||||
|
user's own images and video without a shell. It is a thin wrapper: each tool
|
||||||
|
runs one command of this CLI in a child process and returns that command's exit
|
||||||
|
code, output path and text.
|
||||||
|
|
||||||
|
The subcommand ships since 0.46.0. The server uses only the Python standard
|
||||||
|
library, so the default package runs it. The commands behind a tool need the
|
||||||
|
same extras they need on the command line
|
||||||
|
([CLI guide](cli.md#command-dependency-map)). A missing extra comes back as a
|
||||||
|
tool error carrying the CLI's install hint.
|
||||||
|
|
||||||
|
## Client configuration
|
||||||
|
|
||||||
|
Most clients take a command and arguments. With [uv](https://docs.astral.sh/uv/):
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"remove-ai-watermarks": {
|
||||||
|
"command": "uvx",
|
||||||
|
"args": ["remove-ai-watermarks", "mcp"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
To make visible-mark removal and pixel detectors available, name the extras:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mcpServers": {
|
||||||
|
"remove-ai-watermarks": {
|
||||||
|
"command": "uvx",
|
||||||
|
"args": ["--from", "remove-ai-watermarks[visible,video]", "remove-ai-watermarks", "mcp"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
An existing installation can be started directly as `remove-ai-watermarks mcp`.
|
||||||
|
Paths in tool calls are read and written on the machine that runs the server.
|
||||||
|
|
||||||
|
## Tools
|
||||||
|
|
||||||
|
| Tool | Runs | Writes |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `identify` | `identify --json`, or `video identify --json` for a video suffix; `check_visible: false` adds `--no-visible` | Nothing |
|
||||||
|
| `remove` | `metadata --remove`, `visible` or `all` for an image; `video metadata --remove`, `video visible` or `video all` for a video | One output file, `<name>_clean` beside the source unless `output` is given |
|
||||||
|
| `batch` | `batch --mode` for images or `video batch --mode` for video | A separate output directory, `<dir>_clean` unless `output_dir` is given |
|
||||||
|
|
||||||
|
`mode` is `metadata` (the default), `visible` or `all`. For an image, `all`
|
||||||
|
adds the invisible-watermark scrub where the installed extras and a CUDA GPU
|
||||||
|
allow it; `video all` removes visible marks and metadata only. The server never
|
||||||
|
writes over an input: an `output` equal to the source, or an `output_dir` equal
|
||||||
|
to the input directory, is refused before anything runs.
|
||||||
|
|
||||||
|
Exit codes keep their CLI meaning. `0` and `2` are results, not tool errors, and
|
||||||
|
`2` means no supported signal was found, or a `batch` directory held nothing to
|
||||||
|
process, which is unknown rather than clean. An
|
||||||
|
input that is not a readable image or video exits `1`, so `identify` on it is a
|
||||||
|
tool error even though the command still printed a report.
|
||||||
|
Click also exits `2` on a usage error such as a missing file; its `Usage:` line on
|
||||||
|
stderr marks that case as a tool error. Every other nonzero exit code is returned
|
||||||
|
with `isError: true` and the command's stderr.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
The server carries the project's intended-use boundary in its `instructions`:
|
||||||
|
it is for content the user generated or edited themselves, and not for stock,
|
||||||
|
agency, marketplace or other marks that protect a third party's asset. See
|
||||||
|
[legal and safety](legal-and-safety.md).
|
||||||
|
|
||||||
|
## Protocol
|
||||||
|
|
||||||
|
The server answers the initialize handshake for protocol versions 2025-11-25,
|
||||||
|
2025-06-18, 2025-03-26 and 2024-11-05, and lists its tools with input schemas
|
||||||
|
and annotations: `identify` is read-only, and `remove` and `batch` write new
|
||||||
|
files without destroying inputs. It does not implement `server/discover`. A
|
||||||
|
client that probes it first, such as the official Python SDK's `Client` in its
|
||||||
|
default mode, receives method-not-found and falls back to the handshake.
|
||||||
|
`tests/test_mcp_server.py` runs that client against the real subcommand.
|
||||||
|
|
||||||
|
## Registry listing
|
||||||
|
|
||||||
|
[`server.json`](../server.json) describes the PyPI package for the
|
||||||
|
[official MCP registry](https://registry.modelcontextprotocol.io/) under
|
||||||
|
`io.github.wiltodelta/remove-ai-watermarks`. The registry verifies ownership by
|
||||||
|
the `mcp-name:` marker in `README.md`, which becomes the PyPI description, so it
|
||||||
|
can only verify a release whose README carries that marker.
|
||||||
|
[`glama.json`](../glama.json) names the maintainer for Glama. Each release is
|
||||||
|
published automatically; the workflow is described in
|
||||||
|
[release and distribution](release-and-distribution.md#mcp-registry).
|
||||||
@@ -47,7 +47,35 @@ Important contracts:
|
|||||||
`EXIT_NO_VISIBLE_MARK`.
|
`EXIT_NO_VISIBLE_MARK`.
|
||||||
- `invisible` writes no output when no supported local signal is found, unless
|
- `invisible` writes no output when no supported local signal is found, unless
|
||||||
`--force` is supplied.
|
`--force` is supplied.
|
||||||
- The two no-signal conditions currently share exit code `2`.
|
It decodes the source before the scrub gate and exits `1` when it cannot,
|
||||||
|
because the gate reads undecodable bytes as "no target" and would otherwise
|
||||||
|
report the no-signal code for a file it never read (fixed 2026-10-17).
|
||||||
|
- The two no-signal conditions currently share exit code `2`, and so does a
|
||||||
|
`batch` or `video batch` directory with no supported file, which writes
|
||||||
|
nothing (decided 2026-10-18).
|
||||||
|
- An input that is not a readable container exits `1` from `identify`,
|
||||||
|
`metadata` and every `video` command, and fails its row in `batch` and `video
|
||||||
|
batch` (fixed 2026-10-18). The metadata scanners read raw bytes, so random
|
||||||
|
bytes, an empty file or a truncated PNG or MP4 used to print the same "No AI
|
||||||
|
metadata found" or "AI metadata stripped" a real file gets, with exit `0`.
|
||||||
|
`metadata.unreadable_reason` is the one probe: rasters must decode in Pillow or
|
||||||
|
OpenCV, ISOBMFF boxes and EBML elements must tile the file, and a RIFF file must
|
||||||
|
hold its declared size. `strip_and_verify` and `video._video_source` ask it, and
|
||||||
|
`identify` prints its report first, so an AI signal read from a truncated file
|
||||||
|
is still shown. Absent optional detectors are not unreadable input: without the
|
||||||
|
pixel stack `identify` keeps exit `0` and its caveat (decided 2026-10-18).
|
||||||
|
Regression coverage: `tests/test_fail_closed.py`.
|
||||||
|
- `visible` and `erase` strip and verify AI metadata on a staged sibling and exit
|
||||||
|
`1` with nothing published when a marker survives; they used to swallow a failed
|
||||||
|
strip unless `--verbose` was set. The library `remove_visible` does the same and
|
||||||
|
raises `MetadataStripIncomplete`, as `remove_all` already did. The diffusion
|
||||||
|
writer behind `invisible` stages the same way and raises with nothing published
|
||||||
|
(fixed 2026-10-19; it used to leave the marked file at the output path).
|
||||||
|
`_internal.utils.staged_sibling` is the shared stage: it never pre-creates the
|
||||||
|
name, so the published file keeps the default permissions. These writers strip
|
||||||
|
through `metadata.strip_written_output`, which skips the readability probe:
|
||||||
|
the file is the package's own encode, and the probe's full decode doubled the
|
||||||
|
cost of the strip (decided 2026-10-19).
|
||||||
- A quiet `metadata --check` and a successful `metadata --remove` (same for
|
- A quiet `metadata --check` and a successful `metadata --remove` (same for
|
||||||
`video metadata`) end by repeating the `identify` limit: the pixel channel is
|
`video metadata`) end by repeating the `identify` limit: the pixel channel is
|
||||||
untouched and a watermark such as SynthID has no local decoder once its
|
untouched and a watermark such as SynthID has no local decoder once its
|
||||||
@@ -236,10 +264,54 @@ payloads; TC260 removal changes the four-byte key to `free` and blanks only the
|
|||||||
validated JSON value with same-length spaces. Keyed generation metadata such
|
validated JSON value with same-length spaces. Keyed generation metadata such
|
||||||
as `workflow` and `prompt` uses the same structural walk; removal blanks the
|
as `workflow` and `prompt` uses the same structural walk; removal blanks the
|
||||||
complete key and value in place. This preserves every box size,
|
complete key and value in place. This preserves every box size,
|
||||||
`stco`/`co64` offset, encoded stream byte, and source-sized memory bound.
|
`stco`/`co64` offset, encoded stream byte apart from the Kling SEI tag below, and
|
||||||
Publication is atomic, and a malformed top-level walk is copied unchanged. A
|
source-sized memory bound.
|
||||||
|
Publication is atomic, and a malformed top-level walk is copied unchanged,
|
||||||
|
although `strip_and_verify` refuses such a source before the stripper runs. A
|
||||||
generic `AIGC` key whose value has no TC260 field is ignored.
|
generic `AIGC` key whose value has no TC260 field is ignored.
|
||||||
|
|
||||||
|
The in-memory image path (AVIF, HEIF, JPEG-XL and unknown suffixes, through
|
||||||
|
`strip_c2pa_boxes`) blanks the same top-level boxes to same-size `free` boxes.
|
||||||
|
It used to drop them, which moved `mdat` under an unchanged `iloc`: a phone HEIC
|
||||||
|
whose C2PA `uuid` box preceded `mdat` came out undecodable (2026-10-13).
|
||||||
|
`tests/test_metadata.py::TestIsobmffStripKeepsOffsets` decodes a cleaned HEIC to
|
||||||
|
guard it.
|
||||||
|
|
||||||
|
App-export JSON in a keyed or keyless metadata item is blanked the same way.
|
||||||
|
Dreamina's MP4 export writes it twice, under `LvMetaInfo` and
|
||||||
|
`com.apple.quicktime.artwork`. The file-level scan read both from the raw head
|
||||||
|
bytes while the stripper walked only TC260 and generation keys, so
|
||||||
|
`video metadata --remove` exited 1 with both markers still present on the
|
||||||
|
tracked Dreamina fixture (found 2026-10-17). Its `UserComment`
|
||||||
|
(`{"aigc_label_type":0,"source_info":"dreamina"}`) and `comment` (`vid:...`)
|
||||||
|
items carry no marker the scan counts, so they stay.
|
||||||
|
|
||||||
|
`test_no_flagged_metadata_value_survives_in_the_output_bytes` strips every
|
||||||
|
tracked MP4 fixture and searches the output byte for byte for each flagged
|
||||||
|
source value, after the pattern of byte-level leftover checks in public video
|
||||||
|
scrubbers. The walk reaches container metadata only. Strings inside the encoded
|
||||||
|
streams are out of its reach with one exception: x264 writes its version and
|
||||||
|
options as an H.264 SEI message, which is not AI provenance and survives the
|
||||||
|
stream copy; only a path that re-encodes the video stream (`video visible`,
|
||||||
|
`video invisible`) replaces it.
|
||||||
|
|
||||||
|
The exception is Kling's encoder tag. Every Kling video in the tracked fixtures
|
||||||
|
and local captures (Kling 2.6, 3.0, 3.0 Omni and O3, through Higgsfield, Runway
|
||||||
|
and Kling itself) carries one H.264 `user_data_unregistered` SEI message, payload
|
||||||
|
type 5, whose 25-byte payload is UUID `91ca6061-4aee-3854-8614-2d5f73f4ae2e`,
|
||||||
|
`kling-ai` and a NUL. `get_ai_metadata` reports it as `kling_sei` from the head
|
||||||
|
window, and `identify_video` names Kling from it when no TC260 label does, as on
|
||||||
|
`data/fixtures/visible/kling/provider-original.mp4`, which carries no other
|
||||||
|
signal. The streaming stripper finds every copy of the UUID plus `kling-ai` in
|
||||||
|
each top-level `mdat` during the copy pass it already makes, so the file is still
|
||||||
|
read once, and overwrites those 24 bytes with spaces. The NAL length,
|
||||||
|
SEI header and payload size stay, and a run of 0x20 cannot form an
|
||||||
|
emulation-prevention sequence, so the message stays well-formed and no decoder
|
||||||
|
reads it: `ffmpeg -f framemd5` hashes of every frame match before and after on
|
||||||
|
both tracked Kling fixtures (found and fixed 2026-10-17). The QuickTime `©cmt` item in
|
||||||
|
the Grok fixture (`Signature: <base64>`) also survives, because the xAI
|
||||||
|
signature rule requires the UUID half of the pair and that half is absent here.
|
||||||
|
|
||||||
[`_internal/ebml.py`](../src/remove_ai_watermarks/_internal/ebml.py) provides the
|
[`_internal/ebml.py`](../src/remove_ai_watermarks/_internal/ebml.py) provides the
|
||||||
corresponding bounded Matroska/WebM reader. It seeks over clusters and accepts
|
corresponding bounded Matroska/WebM reader. It seeks over clusters and accepts
|
||||||
only a `Segment.Tags.Tag.SimpleTag` pairing `TagName=AIGC` with a JSON
|
only a `Segment.Tags.Tag.SimpleTag` pairing `TagName=AIGC` with a JSON
|
||||||
@@ -637,6 +709,10 @@ the official C2PA registry. Refresh it with
|
|||||||
upstream schema invariants, records the exact source revision and writes a
|
upstream schema invariants, records the exact source revision and writes a
|
||||||
deterministic module. Runtime inspection remains offline, and registry membership
|
deterministic module. Runtime inspection remains offline, and registry membership
|
||||||
is name-only evidence rather than proof that a compatible decoder ships.
|
is name-only evidence rather than proof that a compatible decoder ships.
|
||||||
|
`--check`, run by `maintain.sh` and the CI lint job ahead of Ruff, compares the
|
||||||
|
snapshot with the live upstream revision, so an upstream registry commit fails
|
||||||
|
lint on `main` and on every pull request until the snapshot is refreshed, and the
|
||||||
|
Ruff step behind it does not run until then.
|
||||||
The GitHub API revision lookup sends `GITHUB_TOKEN` (or `GH_TOKEN`) when set,
|
The GitHub API revision lookup sends `GITHUB_TOKEN` (or `GH_TOKEN`) when set,
|
||||||
because the unauthenticated 60 requests/hour per IP limit is exhausted on
|
because the unauthenticated 60 requests/hour per IP limit is exhausted on
|
||||||
shared CI runners; `.env.example` documents the variable.
|
shared CI runners; `.env.example` documents the variable.
|
||||||
@@ -743,8 +819,11 @@ Key contracts:
|
|||||||
A.1 string (`"Label":"1","ContentProducer":"...","ProduceID":"..."`) under the
|
A.1 string (`"Label":"1","ContentProducer":"...","ProduceID":"..."`) under the
|
||||||
default SM2 user ID, and MiniMax's `LabelMataData` spelling is accepted beside
|
default SM2 user ID, and MiniMax's `LabelMataData` spelling is accepted beside
|
||||||
the guide's `Md`. Only the label signature is checked: MiniMax's binding
|
the guide's `Md`. Only the label signature is checked: MiniMax's binding
|
||||||
signature also verifies, but no verdict reads it, and its content hash is not
|
signature also verifies, but no verdict reads it. The guide's appendix A.2
|
||||||
recomputed because MiniMax names no `CntSel`. The key travels in the file, so
|
defines `DFT` for non-JPEG/PNG files as all bytes except the metadata label;
|
||||||
|
MiniMax names no `CntSel`, and measured whole-file, direct DFT byte-range,
|
||||||
|
metadata-box, and media-payload interpretations do not reproduce its content
|
||||||
|
hash. The key travels in the file, so
|
||||||
only a key pinned in `TC260_PRODUCERS` (the source of `TC260_SIGNING_KEYS`)
|
only a key pinned in `TC260_PRODUCERS` (the source of `TC260_SIGNING_KEYS`)
|
||||||
names a signer, and a pinned signer
|
names a signer, and a pinned signer
|
||||||
is checked with the pinned point whatever parity the file claims; an unpinned
|
is checked with the pinned point whatever parity the file claims; an unpinned
|
||||||
@@ -776,7 +855,7 @@ Key contracts:
|
|||||||
written output for surviving markers. If the metadata-preserving decoder
|
written output for surviving markers. If the metadata-preserving decoder
|
||||||
rejected the container but `image_io` can still decode its raster,
|
rejected the container but `image_io` can still decode its raster,
|
||||||
`strip_and_verify` normalizes that raster and scans again. A truly undecodable
|
`strip_and_verify` normalizes that raster and scans again. A truly undecodable
|
||||||
file keeps the surviving-marker result.
|
file raises `ValueError` before anything is written.
|
||||||
|
|
||||||
Detection and removal must stay in parity. A new marker is incomplete until the
|
Detection and removal must stay in parity. A new marker is incomplete until the
|
||||||
scanner can find it, the remover can reach every supported placement, and a
|
scanner can find it, the remover can reach every supported placement, and a
|
||||||
@@ -1498,6 +1577,15 @@ schemas 0-2 count as positives. Schema 3 is below the precision threshold: all
|
|||||||
same false payload after re-encoding. The official Adobe Variant P schema-1
|
same false payload after re-encoding. The official Adobe Variant P schema-1
|
||||||
fixture in `data/fixtures/provenance/` is the positive regression control.
|
fixture in `data/fixtures/provenance/` is the positive regression control.
|
||||||
|
|
||||||
|
An independent benchmark supports gating beyond a valid codeword. Kazempourradi
|
||||||
|
et al. ([arXiv:2610.04151](https://arxiv.org/abs/2610.04151), section 4.4)
|
||||||
|
found that the blind detectors of TrustMark B, C, P and Q in BCH_5 mode reported
|
||||||
|
a valid codeword on at least one of four decodes for 5.9% (P) to 15.3% (C) of 85
|
||||||
|
unmarked images, and on 3 of 8 images in a 16-bit HDR subset. Testing the same
|
||||||
|
decodes against the expected payload produced no false match. This detector is
|
||||||
|
blind and has no expected payload, so the re-encode stability and schema
|
||||||
|
restriction above stand in for that test.
|
||||||
|
|
||||||
#### Why the DWT-DCT decoder looks the way it does
|
#### Why the DWT-DCT decoder looks the way it does
|
||||||
|
|
||||||
Two things in `dwt_dct.py` are load-bearing and neither is obvious from the code.
|
Two things in `dwt_dct.py` are load-bearing and neither is obvious from the code.
|
||||||
@@ -1593,8 +1681,8 @@ their own: the Gemini sparkle has its own higher-confidence path, and the pill
|
|||||||
alone is too weak to attribute.
|
alone is too weak to attribute.
|
||||||
|
|
||||||
The set of marks that veto the Jimeng pill is derived from the registry rows:
|
The set of marks that veto the Jimeng pill is derived from the registry rows:
|
||||||
every other product from the same manufacturer. Doubao therefore vetoes the pill
|
every other product from the same manufacturer. Doubao and Dola therefore veto the pill
|
||||||
because both products are ByteDance, while Qwen, Kling, Yuanbao, RunningHub,
|
because all three products are ByteDance, while Qwen, Kling, Yuanbao, RunningHub,
|
||||||
Baidu, and LiblibAI do not become siblings merely because they use the same TC260
|
Baidu, and LiblibAI do not become siblings merely because they use the same TC260
|
||||||
standard. Marks from other manufacturers cannot enable or veto the
|
standard. Marks from other manufacturers cannot enable or veto the
|
||||||
ByteDance-specific arm.
|
ByteDance-specific arm.
|
||||||
@@ -1653,8 +1741,7 @@ assets and template ladder are immutable.
|
|||||||
|
|
||||||
A decorative sparkle/glint/star glyph in stylized typography -- a bright,
|
A decorative sparkle/glint/star glyph in stylized typography -- a bright,
|
||||||
near-white four-point accent, the same silhouette family the engine's own
|
near-white four-point accent, the same silhouette family the engine's own
|
||||||
template captures -- is a known, unresolved false-positive class. Incident
|
template captures -- is a known, unresolved false-positive class. A book-cover
|
||||||
2026-08-27 (raiw-app, Spaces uid 7e01ef8cb99949bba14ab4607a66ae06): a book-cover
|
|
||||||
title's decorative diamond accent inside a neon-glow letterform scored 0.64, just
|
title's decorative diamond accent inside a neon-glow letterform scored 0.64, just
|
||||||
under `_SPARKLE_FP_CONF`, and the removal step painted a permanent smudge over
|
under `_SPARKLE_FP_CONF`, and the removal step painted a permanent smudge over
|
||||||
the artwork. Investigation (margin, gradient, saturation, and a prototyped local
|
the artwork. Investigation (margin, gradient, saturation, and a prototyped local
|
||||||
@@ -1694,6 +1781,11 @@ Regression coverage:
|
|||||||
[`_text_mark_engine.py`](../src/remove_ai_watermarks/_text_mark_engine.py)
|
[`_text_mark_engine.py`](../src/remove_ai_watermarks/_text_mark_engine.py)
|
||||||
provides common localization, detection front ends, template caching, rival
|
provides common localization, detection front ends, template caching, rival
|
||||||
comparison, and footprint construction.
|
comparison, and footprint construction.
|
||||||
|
[`_text_mark_variants.py`](../src/remove_ai_watermarks/_text_mark_variants.py)
|
||||||
|
combines independently gated layouts under one registry key while retaining the
|
||||||
|
winning layout's template and mask geometry. Its bottom-right variant searches
|
||||||
|
only anchor-valid placements, so an unrelated stronger match elsewhere in the
|
||||||
|
corner ROI cannot hide the real edge-hugging mark.
|
||||||
|
|
||||||
Each vendor module supplies a `TextMarkConfig` and only the behavior that cannot
|
Each vendor module supplies a `TextMarkConfig` and only the behavior that cannot
|
||||||
be represented by the shared base:
|
be represented by the shared base:
|
||||||
@@ -1703,6 +1795,8 @@ be represented by the shared base:
|
|||||||
- [`qwen_engine.py`](../src/remove_ai_watermarks/qwen_engine.py)
|
- [`qwen_engine.py`](../src/remove_ai_watermarks/qwen_engine.py)
|
||||||
- [`kling_engine.py`](../src/remove_ai_watermarks/kling_engine.py)
|
- [`kling_engine.py`](../src/remove_ai_watermarks/kling_engine.py)
|
||||||
- [`yuanbao_engine.py`](../src/remove_ai_watermarks/yuanbao_engine.py)
|
- [`yuanbao_engine.py`](../src/remove_ai_watermarks/yuanbao_engine.py)
|
||||||
|
- [`dola_engine.py`](../src/remove_ai_watermarks/dola_engine.py)
|
||||||
|
- [`workbuddy_engine.py`](../src/remove_ai_watermarks/workbuddy_engine.py)
|
||||||
- [`samsung_engine.py`](../src/remove_ai_watermarks/samsung_engine.py)
|
- [`samsung_engine.py`](../src/remove_ai_watermarks/samsung_engine.py)
|
||||||
- [`runninghub_engine.py`](../src/remove_ai_watermarks/runninghub_engine.py)
|
- [`runninghub_engine.py`](../src/remove_ai_watermarks/runninghub_engine.py)
|
||||||
- [`baidu_engine.py`](../src/remove_ai_watermarks/baidu_engine.py)
|
- [`baidu_engine.py`](../src/remove_ai_watermarks/baidu_engine.py)
|
||||||
@@ -1747,7 +1841,10 @@ on 2026-09-04 uses `KlingAI 3.0`. The Latin template scored 0.429 on that 1024 x
|
|||||||
across the 94 available neighboring real and synthetic image controls, while an
|
across the 94 available neighboring real and synthetic image controls, while an
|
||||||
adversarial solid corner blob reached 0.379, so its strict gate is 0.40. This is
|
adversarial solid corner blob reached 0.379, so its strict gate is 0.40. This is
|
||||||
one-positive provisional calibration, intentionally separate from the older
|
one-positive provisional calibration, intentionally separate from the older
|
||||||
CJK detector rather than presented as broad recall evidence.
|
CJK detector rather than presented as broad recall evidence. Both layouts now
|
||||||
|
project row-wide and column-wide structure out of the winning patch before a
|
||||||
|
second correlation. This preserves glyph detail while rejecting ordinary panel
|
||||||
|
and image boundaries that previously matched the Latin template.
|
||||||
|
|
||||||
The measured Microsoft badge variant (2026-08-27 registration) is the first
|
The measured Microsoft badge variant (2026-08-27 registration) is the first
|
||||||
`tr`-corner mark and the first `long`-side scale basis: the pill tracks the
|
`tr`-corner mark and the first `long`-side scale basis: the pill tracks the
|
||||||
@@ -1758,7 +1855,8 @@ undersized the template by the aspect ratio (portrait carriers fell to
|
|||||||
its synthetic internal shapes knocked out - the holes are what separate it from
|
its synthetic internal shapes knocked out - the holes are what separate it from
|
||||||
any other bright rounded corner element (a plain white pill scores below the gate
|
any other bright rounded corner element (a plain white pill scores below the gate
|
||||||
in the tests). It does not claim coverage of Microsoft's other documented icon,
|
in the tests). It does not claim coverage of Microsoft's other documented icon,
|
||||||
wording, or position variants.
|
wording, or position variants beyond the separately gated beige `Made with AI`
|
||||||
|
text pill.
|
||||||
|
|
||||||
The 2026-08-27 rerun used the registered engine through
|
The 2026-08-27 rerun used the registered engine through
|
||||||
`scripts/registered_mark_calibrate.py`, rather than a copied detector
|
`scripts/registered_mark_calibrate.py`, rather than a copied detector
|
||||||
@@ -1795,8 +1893,9 @@ box grows), not the whole `footprint_mask`. Baidu extends right to the corner ta
|
|||||||
and LiblibAI extends left to the triangle logo; both inherit every guard around
|
and LiblibAI extends left to the triangle logo; both inherit every guard around
|
||||||
that arithmetic.
|
that arithmetic.
|
||||||
|
|
||||||
Doubao, Kling, and Samsung are deliberate sparse-mask exceptions because their alpha assets
|
Doubao, Kling, and Samsung retain deliberate sparse masks for their original
|
||||||
supply more information than a rectangle. Doubao's continuous top-hat response
|
layouts because those alpha assets supply more information than a rectangle.
|
||||||
|
Doubao's continuous top-hat response
|
||||||
locates the mark, then `DoubaoEngine.footprint_mask` resizes the alpha to that same winning
|
locates the mark, then `DoubaoEngine.footprint_mask` resizes the alpha to that same winning
|
||||||
`match_box` and masks only the glyphs. The canonical 2048-pixel fixture has bright
|
`match_box` and masks only the glyphs. The canonical 2048-pixel fixture has bright
|
||||||
branch texture behind the bottom-edge wordmark; bounding the thresholded response,
|
branch texture behind the bottom-edge wordmark; bounding the thresholded response,
|
||||||
@@ -1806,6 +1905,13 @@ with large triangular wedges. The aligned sparse mask keeps both frame edges
|
|||||||
untouched and still clears the detector. `force` cannot align a missing detection,
|
untouched and still clears the detector. `force` cannot align a missing detection,
|
||||||
so it retains the shared geometry-box fallback.
|
so it retains the shared geometry-box fallback.
|
||||||
|
|
||||||
|
The added outlined Doubao, stacked Yuanbao, Dola, WorkBuddy, Korean Samsung, and
|
||||||
|
beige Microsoft layouts use solid detector-aligned footprints where a sparse
|
||||||
|
synthetic glyph mask left visible outlines, sparkles, capsules, or shadows in
|
||||||
|
rendered removal checks. Each layout keeps its own detection threshold and mask
|
||||||
|
geometry; selecting one does not replace the calibrated behavior of the older
|
||||||
|
layout under the same registry key.
|
||||||
|
|
||||||
Doubao keeps the original low-saturation scan for pale corners. When enough of
|
Doubao keeps the original low-saturation scan for pale corners. When enough of
|
||||||
its corner is strongly colored and that scan falls below the strict gate, a
|
its corner is strongly colored and that scan falls below the strict gate, a
|
||||||
wider color filter searches for white glyphs tinted by the background. The
|
wider color filter searches for white glyphs tinted by the background. The
|
||||||
@@ -1933,14 +2039,12 @@ exists.
|
|||||||
|
|
||||||
#### OpenArt wordmark
|
#### OpenArt wordmark
|
||||||
|
|
||||||
Registered 2026-09-21 for a confirmed production miss reported through raiw-app:
|
Registered 2026-09-21 after a `visible`-mode run left a large, prominent OpenArt wordmark
|
||||||
a free `visible`-mode run (2026-09-11, 1528x2712 portrait, Spaces uid
|
|
||||||
`aefb1e07306f4a7280615a8daea01a39`) left a large, prominent OpenArt wordmark
|
|
||||||
(a bowtie/infinity icon followed by "OpenArt" in bold sans-serif, semi-
|
(a bowtie/infinity icon followed by "OpenArt" in bold sans-serif, semi-
|
||||||
transparent white) completely unremoved -- pixel-identical before and after.
|
transparent white) completely unremoved -- pixel-identical before and after.
|
||||||
`identify` found nothing at all for that case: no C2PA, EXIF AI tag, or China
|
`identify` found nothing at all for that case: no C2PA, EXIF AI tag, or China
|
||||||
AIGC/TC260 label, so the visible mark is the only known attribution route for
|
AIGC/TC260 label, so the visible mark is the only known attribution route for
|
||||||
a bare OpenArt export. The user rated the result 0 ("Didn't work").
|
a bare OpenArt export.
|
||||||
|
|
||||||
**This is the only registered mark placed at the FRAME CENTER rather than a
|
**This is the only registered mark placed at the FRAME CENTER rather than a
|
||||||
corner.** Every other engine in this family anchors to one of the four corners
|
corner.** Every other engine in this family anchors to one of the four corners
|
||||||
@@ -2004,10 +2108,14 @@ geometry, and until one exists the bare-arm question is settled by this
|
|||||||
measurement, not by gate tuning. The local calibration harness is not tracked.
|
measurement, not by gate tuning. The local calibration harness is not tracked.
|
||||||
(cohorts A/B/C over the no-signal pool).
|
(cohorts A/B/C over the no-signal pool).
|
||||||
|
|
||||||
The weak Jimeng pill detector lives in
|
The Jimeng pill detector lives in
|
||||||
[`pill_engine.py`](../src/remove_ai_watermarks/pill_engine.py). It uses a
|
[`pill_engine.py`](../src/remove_ai_watermarks/pill_engine.py). It combines the
|
||||||
synthetic silhouette for detection and a fixed top-left footprint. Its public,
|
existing weak synthetic-outline path with a high-confidence contrast template
|
||||||
published real-world regression carrier is
|
for the measured solid `AI生成` label. A Jimeng provenance match can corroborate
|
||||||
|
that verified label without the flat-background guard; the weak outline keeps
|
||||||
|
both the provenance and background gates described above. Removal follows the
|
||||||
|
matched label when verified and otherwise uses the fixed top-left footprint. Its
|
||||||
|
public, published real-world regression carrier is
|
||||||
`data/fixtures/visible/jimeng_pill/provider-published-example.jpg`; the direct
|
`data/fixtures/visible/jimeng_pill/provider-published-example.jpg`; the direct
|
||||||
detector scores the pill at 0.28 and the same image's Jimeng wordmark at 0.61.
|
detector scores the pill at 0.28 and the same image's Jimeng wordmark at 0.61.
|
||||||
|
|
||||||
@@ -2133,6 +2241,13 @@ image sizing, postprocessing, and the public engine
|
|||||||
interface. It delegates model execution to
|
interface. It delegates model execution to
|
||||||
[`_internal/watermark_remover.py`](../src/remove_ai_watermarks/_internal/watermark_remover.py).
|
[`_internal/watermark_remover.py`](../src/remove_ai_watermarks/_internal/watermark_remover.py).
|
||||||
|
|
||||||
|
The engine preserves source transparency for alpha-capable outputs. It captures
|
||||||
|
alpha after EXIF orientation is applied and before resizing, then rejoins the
|
||||||
|
unchanged alpha plane after restoring the RGB result to the original dimensions
|
||||||
|
and applying postprocessing. This also handles palette and colour-key PNG
|
||||||
|
transparency; diffusion itself still receives RGB. Regression coverage lives in
|
||||||
|
`test_invisible_engine.py::TestTransparency`.
|
||||||
|
|
||||||
`get_device` in that module answers only `cuda` or `cpu`. An `mps` or `xpu` answer
|
`get_device` in that module answers only `cuda` or `cpu`. An `mps` or `xpu` answer
|
||||||
would travel one frame to the same CUDA-only refusal while costing a device probe,
|
would travel one frame to the same CUDA-only refusal while costing a device probe,
|
||||||
and reporting it implied an Apple-silicon or Intel-GPU path that does not exist.
|
and reporting it implied an Apple-silicon or Intel-GPU path that does not exist.
|
||||||
@@ -2162,27 +2277,27 @@ Regression coverage:
|
|||||||
|
|
||||||
### CPU offload
|
### CPU offload
|
||||||
|
|
||||||
CPU offload is enabled only when requested. Nothing calls Diffusers'
|
Nothing calls Diffusers' `enable_model_cpu_offload` any more -- that belonged to
|
||||||
`enable_model_cpu_offload` any more -- that belonged to the deleted single-stage
|
the deleted single-stage profiles. `--cpu-offload` sets both residency fields, and
|
||||||
profiles. `--cpu-offload` sets both residency fields, and they do not travel the
|
they do not travel the same distance: the face field is read by the shared base and
|
||||||
same distance: the face field is read by the shared base and so applies to every
|
so applies to every profile, while the global field is read by `QwenZImagePipeline`
|
||||||
profile, while the global field is read by `QwenZImagePipeline` alone.
|
and `SdxlZImagePipeline`. `ChromaZImagePipeline` loads its global stack with a
|
||||||
`ChromaZImagePipeline` and `SdxlZImagePipeline` load their global stack with a
|
plain `.to(device)` and never consults it.
|
||||||
plain `.to(device)` and never consult it.
|
|
||||||
|
|
||||||
`GLOBAL_OFFLOAD_PROFILES` in `watermark_profiles.py` is the list, and
|
`GLOBAL_OFFLOAD_PROFILES` in `watermark_profiles.py` is the list, and
|
||||||
`global_offload_supported` the question; `WatermarkRemover._warn_if_global_offload_unsupported`
|
`global_offload_supported` the question; `WatermarkRemover._warn_if_global_offload_unsupported`
|
||||||
warns ahead of the model load rather than at construction, because `auto` picks
|
warns ahead of the model load rather than at construction, because `auto` picks
|
||||||
its engine per-image. An explicit `auto` preload uses the qwen-zimage fallback and
|
its engine per-image. An explicit `auto` preload uses the qwen-zimage fallback and
|
||||||
therefore keeps global offload active before that resolution. Below
|
therefore keeps global offload active before that resolution. Below
|
||||||
`RESIDENT_FACE_MODEL_MIN_VRAM_GIB` the flag is a no-op for the other two profiles --
|
`RESIDENT_FACE_MODEL_MIN_VRAM_GIB` the flag is a no-op for `chroma-zimage` -- the
|
||||||
the face stack was offloading anyway and the global one does not ask -- which is
|
face stack was offloading anyway and the global one does not ask -- which is what
|
||||||
what the warning exists to say before a download starts.
|
the warning exists to say before a download starts.
|
||||||
|
|
||||||
Residency is otherwise chosen from the card's total VRAM, once per stack:
|
Residency is otherwise chosen from the card's total VRAM, once per stack:
|
||||||
`resolve_global_model_residency` gates the mandatory Qwen stack at
|
`resolve_global_model_residency` gates the mandatory Qwen stack at
|
||||||
`RESIDENT_GLOBAL_MODEL_MIN_VRAM_GIB` and `resolve_face_model_residency` gates the
|
`RESIDENT_GLOBAL_MODEL_MIN_VRAM_GIB`, `resolve_sdxl_global_residency` gates the
|
||||||
optional Z-Image stack at `RESIDENT_FACE_MODEL_MIN_VRAM_GIB`.
|
SDXL stack at `RESIDENT_SDXL_GLOBAL_MIN_VRAM_GIB`, and `resolve_face_model_residency`
|
||||||
|
gates the optional Z-Image stack at `RESIDENT_FACE_MODEL_MIN_VRAM_GIB`.
|
||||||
|
|
||||||
Below the global floor, `_qwen_vram_config` streams the stack from disk, which is
|
Below the global floor, `_qwen_vram_config` streams the stack from disk, which is
|
||||||
what makes a 20B model runnable on a consumer card. At or above it, streaming is
|
what makes a 20B model runnable on a consumer card. At or above it, streaming is
|
||||||
@@ -2199,6 +2314,43 @@ The resident config deliberately passes no `"disk"` value anywhere. DiffSynth la
|
|||||||
`disk_offload` once, from `offload_dtype`, so leaving the sentinel in place while
|
`disk_offload` once, from `offload_dtype`, so leaving the sentinel in place while
|
||||||
pointing every device at CUDA would keep the meta-drop and re-read.
|
pointing every device at CUDA would keep the meta-drop and re-read.
|
||||||
|
|
||||||
|
Sequential SDXL offload receives the current CUDA device index explicitly, so
|
||||||
|
its hooks use the same GPU as the residency probes, input tensors, and seeded
|
||||||
|
generators. Omitting the index makes Diffusers select GPU 0 even after
|
||||||
|
`torch.cuda.set_device(1)`. `test_streaming_uses_the_current_cuda_device` in
|
||||||
|
`test_cpu_offload.py` covers automatic and forced streaming on nonzero devices.
|
||||||
|
|
||||||
|
The SDXL stack (fp16 UNet, Canny ControlNet, both text encoders, the fp16-fix VAE)
|
||||||
|
is ~8.8 GiB of weights, and the UNet and the ControlNet run together on every
|
||||||
|
denoising step. Below the 12 GiB floor `_load_global` fuses the Lightning LoRA on
|
||||||
|
the CPU and then calls `enable_sequential_cpu_offload`; the order matters, because
|
||||||
|
the offload hooks stream their own copy of each weight and a later fuse would never
|
||||||
|
reach it. Measured on an RTX 5060 Laptop (8 GiB, WSL2, 16 GiB system RAM) in
|
||||||
|
October 2026, one 1672x941 global pass at strength 0.15:
|
||||||
|
|
||||||
|
| Placement | Warm pass | Peak VRAM | Peak system RAM |
|
||||||
|
|---|---|---|---|
|
||||||
|
| resident `.to(device)` (previous behaviour) | did not finish in 10 min | 7.9 GiB used, spilling | -- |
|
||||||
|
| `enable_model_cpu_offload` | 237 s | 9.2 GiB reserved on an 8 GiB card | 12.4 GiB |
|
||||||
|
| group offload, block level, CUDA stream | 92 s, or hung | 12.4 GiB reserved | 15.6 GiB |
|
||||||
|
| group offload, leaf level, CUDA stream | 7.8 s, or hung | 5.6 GiB | 15.7 GiB |
|
||||||
|
| group offload, leaf level, `low_cpu_mem_usage` | 21.4 s | 5.6 GiB | 9.4 GiB |
|
||||||
|
| `enable_sequential_cpu_offload` | 12.8 s | 3-4 GiB | 8.5 GiB |
|
||||||
|
|
||||||
|
The resident and model-offload rows did not fail: under the Windows display driver
|
||||||
|
CUDA falls back to shared system memory, so PyTorch reserved more than the card
|
||||||
|
holds and the run crawled instead of raising. The streamed group-offload variants
|
||||||
|
pin every weight in host memory up front; on a 16 GiB machine that left the run
|
||||||
|
finishing or hanging depending on what else held RAM, which rules them out as a
|
||||||
|
default. All completed placements produced bit-identical images.
|
||||||
|
|
||||||
|
Fusing on the CPU instead of on CUDA rounds 788 of the UNet's 3256 fused tensors
|
||||||
|
differently, by at most one fp16 ulp (1.2e-4). On the pass above that moved the
|
||||||
|
output by a mean of 0.16 levels (PSNR 54.3 dB against a CUDA-fused, equally
|
||||||
|
streamed run). Cards at or above the floor keep the CUDA fuse and therefore the
|
||||||
|
previous pixels; only a streamed run, automatic or forced with `--cpu-offload`,
|
||||||
|
sees the difference.
|
||||||
|
|
||||||
Regression coverage:
|
Regression coverage:
|
||||||
|
|
||||||
- [`test_cpu_offload.py`](../tests/test_cpu_offload.py)
|
- [`test_cpu_offload.py`](../tests/test_cpu_offload.py)
|
||||||
|
|||||||
@@ -1,9 +1,9 @@
|
|||||||
# Provider oracle tooling
|
# Provider oracle tooling
|
||||||
|
|
||||||
`scripts/provider_oracles.py` is the development-only entry point for the four
|
`scripts/provider_oracles.py` is the development-only entry point for the four
|
||||||
remote watermark-oracle families used by this project. It drives Google through
|
remote watermark-oracle families used by this project. It drives Gemini through
|
||||||
the user's real authenticated Chrome and drives the other public Web surfaces
|
the user's real authenticated Chrome and drives every other public Web surface,
|
||||||
through isolated Playwright. It keeps provider scope, execution identity,
|
including SynthID Detector, through isolated Playwright. It keeps provider scope, execution identity,
|
||||||
immutable media, and verbatim results in one format without exposing any oracle
|
immutable media, and verbatim results in one format without exposing any oracle
|
||||||
through the installed CLI or public Python API.
|
through the installed CLI or public Python API.
|
||||||
|
|
||||||
@@ -13,7 +13,8 @@ through the installed CLI or public Python API.
|
|||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `openai-api` | OpenAI SynthID | Official Content Provenance API | Images |
|
| `openai-api` | OpenAI SynthID | Official Content Provenance API | Images |
|
||||||
| `microsoft-api` | Microsoft InvisMark | Azure Content Provenance Detection API | Images |
|
| `microsoft-api` | Microsoft InvisMark | Azure Content Provenance Detection API | Images |
|
||||||
| `gemini-web` | Google SynthID | Signed-in Gemini web account | Images and videos |
|
| `gemini-web` | Google SynthID | User's authenticated Chrome | Images and videos |
|
||||||
|
| `synthid-web` | Partner SynthID | Anonymous Playwright page at synthid.com | None recorded |
|
||||||
| `meta-web` | Meta Content Seal | Anonymous Meta identification page | Images |
|
| `meta-web` | Meta Content Seal | Anonymous Meta identification page | Images |
|
||||||
| `microsoft-web` | Microsoft InvisMark | Public Azure AI validation page | Images |
|
| `microsoft-web` | Microsoft InvisMark | Public Azure AI validation page | Images |
|
||||||
| `openai-web` | OpenAI SynthID | Public web verifier | Images |
|
| `openai-web` | OpenAI SynthID | Public web verifier | Images |
|
||||||
@@ -23,26 +24,53 @@ through the installed CLI or public Python API.
|
|||||||
The default selection order is API first, then Web. OpenAI therefore plans
|
The default selection order is API first, then Web. OpenAI therefore plans
|
||||||
`openai-api` before `openai-web`, and Microsoft plans `microsoft-api` before
|
`openai-api` before `openai-web`, and Microsoft plans `microsoft-api` before
|
||||||
`microsoft-web`. Google and Meta currently have no usable API adapter in this
|
`microsoft-web`. Google and Meta currently have no usable API adapter in this
|
||||||
tooling, so their Web surfaces are primary rather than fallback routes.
|
tooling, so each of their Web surfaces is primary. Google has two: `gemini-web`
|
||||||
|
and `synthid-web` at `https://synthid.com/`.
|
||||||
|
|
||||||
Inspect the machine-readable order before preparing a batch:
|
Inspect the machine-readable order before preparing a batch:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run python scripts/provider_oracles.py plan openai --json
|
uv run python scripts/provider_oracles.py plan openai --json
|
||||||
uv run python scripts/provider_oracles.py plan microsoft
|
uv run python scripts/provider_oracles.py plan microsoft
|
||||||
|
uv run python scripts/provider_oracles.py plan google
|
||||||
```
|
```
|
||||||
|
|
||||||
API-first is a selection policy, not automatic failover. Use Web only when the
|
API-first is the default order. Run the API surface when this tooling has a
|
||||||
provider, requested media, or local configuration has no usable API route, or
|
usable route for the provider, media, and local configuration. Google and Meta
|
||||||
when the operator explicitly requests a second surface. An API refusal, quota
|
have no API surface, so each of their Web oracles is primary. For Google that
|
||||||
response, transport failure, or indeterminate verdict is recorded as-is and
|
is `gemini-web`, then `synthid-web`. After a refusal, quota response, transport
|
||||||
never silently triggers a Web upload.
|
failure, or indeterminate verdict, record that result and continue with the next
|
||||||
|
surface as its own acknowledged command, then the remaining configured account
|
||||||
|
slots of those surfaces. One call does not upload the same file to the next
|
||||||
|
surface. Keep the verdicts separate. Do not ask for confirmation before the
|
||||||
|
switch. Anyone doing the task may run the configured checks.
|
||||||
|
|
||||||
The old Google SynthID Detector portal does not work. On 2026-09-07, the
|
The old Google SynthID Detector portal at `labs.google/synthid` does not work.
|
||||||
official `labs.google/synthid` entry point redirected a signed-in browser to a
|
On 2026-09-07 that entry point redirected a signed-in browser to a Google 404;
|
||||||
Google 404; on 2026-09-23 it redirected to
|
on 2026-09-23 it redirected to `deepmind.google.com/frontiers/synthid`, which
|
||||||
`deepmind.google.com/frontiers/synthid`, which also returned 404. Historical corpus rows may still name `synthid-portal`; that value
|
also returned 404. Historical corpus rows may still name `synthid-portal`; that
|
||||||
describes how an old result was obtained and is not an operational surface.
|
value describes how an old result was obtained and is not an operational surface.
|
||||||
|
|
||||||
|
The public SynthID Detector is a different site, [synthid.com](https://synthid.com/),
|
||||||
|
and it is the `synthid-web` surface. On 2026-10-17 the page title was
|
||||||
|
`SynthID Detector`. After the terms dialog, the uploader was present without a
|
||||||
|
completed sign-in in that session. The Account menu still offered Sign in, and
|
||||||
|
no file was submitted, so this check does not show whether a result requires an
|
||||||
|
account. The live FAQ named the partners as Google DeepMind, Nvidia, OpenAI,
|
||||||
|
and Kakao, and it did not list Apple. It defined two settled labels:
|
||||||
|
`Watermark detected` and `Watermark not detected`. The file input accepted
|
||||||
|
`.jpg`, `.jpeg`, `.png`, `.bmp`, `.webp`, `.avif`, `.heic`, `.heif`, `.tiff`,
|
||||||
|
`.tif`, `.gif`, `.wav`, `.mp3`, `.ogg`, `.flac`, `.aac`, `.m4a`, `.mp4`,
|
||||||
|
`.mov`, and `.webm`. This tooling still prepares only the project's existing
|
||||||
|
JPEG, PNG, WebP, MP4, MP3, and WAV suffixes. The portal's additional terms
|
||||||
|
forbid bulk verification and quota evasion. `run-web` may submit one prepared
|
||||||
|
upload through isolated Playwright, with a proxy when the slot names one, and
|
||||||
|
it does not start a sign-in. If the page says `Please sign in before detection.`
|
||||||
|
and returns no watermark label, keep that row and open the same prepared upload
|
||||||
|
in the user's existing real Chrome. That session stays on the direct route, with
|
||||||
|
no proxy and no new login. If that Chrome also asks for sign-in, leave its row
|
||||||
|
unsubmitted. A `Watermark detected` result is partner SynthID, not a Gemini-only
|
||||||
|
or OpenAI-only verdict.
|
||||||
Google's newer Search verification entry point did not provide an independent
|
Google's newer Search verification entry point did not provide an independent
|
||||||
oracle allowance in the session tested on 2026-09-13. Lens accepted the
|
oracle allowance in the session tested on 2026-09-13. Lens accepted the
|
||||||
metadata-stripped, previously Gemini-positive image
|
metadata-stripped, previously Gemini-positive image
|
||||||
@@ -161,9 +189,12 @@ Adobe Inspect and local C2PA, DWT-DCT, TrustMark, and research detectors are
|
|||||||
useful evidence readers, not interchangeable provider pixel-watermark oracles.
|
useful evidence readers, not interchangeable provider pixel-watermark oracles.
|
||||||
|
|
||||||
The surface capability and this project's evidence coverage are separate. The
|
The surface capability and this project's evidence coverage are separate. The
|
||||||
current Google, Microsoft, and Meta pages accept image, video, and audio inputs;
|
current Google, Microsoft, and Meta pages, including `https://synthid.com/`,
|
||||||
the OpenAI API accepts image and audio. A media type does not become certified
|
accept image, video, and audio inputs; the OpenAI API accepts image and audio.
|
||||||
merely because a surface accepts it. Run `list --json` to read both fields.
|
A media type does not become certified merely because a surface accepts it.
|
||||||
|
`synthid-web` therefore lists image, video, and audio as accepted media and
|
||||||
|
records no project evidence until a batch is verified. Run `list --json` to read
|
||||||
|
both fields.
|
||||||
|
|
||||||
## Execution slots
|
## Execution slots
|
||||||
|
|
||||||
@@ -194,36 +225,20 @@ default; `--slots PATH` explicitly overrides it.
|
|||||||
"network_label": "office"
|
"network_label": "office"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "openai-web-thordata-us",
|
"name": "synthid-proxy",
|
||||||
|
"surface": "synthid-web",
|
||||||
|
"account_label": null,
|
||||||
|
"browser_profile": null,
|
||||||
|
"network_label": "thordata",
|
||||||
|
"proxy_url_env": "THORDATA_PROXY"
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "openai-web-proxy",
|
||||||
"surface": "openai-web",
|
"surface": "openai-web",
|
||||||
"account_label": null,
|
"account_label": null,
|
||||||
"browser_profile": null,
|
"browser_profile": null,
|
||||||
"network_label": "thordata-us-residential",
|
"network_label": "thordata",
|
||||||
"proxy_url_env": "THORDATA_ROUTE_US"
|
"proxy_url_env": "THORDATA_PROXY"
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "microsoft-web-thordata-de",
|
|
||||||
"surface": "microsoft-web",
|
|
||||||
"account_label": null,
|
|
||||||
"browser_profile": null,
|
|
||||||
"network_label": "thordata-de-residential",
|
|
||||||
"proxy_url_env": "THORDATA_ROUTE_DE"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "meta-web-thordata-us",
|
|
||||||
"surface": "meta-web",
|
|
||||||
"account_label": null,
|
|
||||||
"browser_profile": null,
|
|
||||||
"network_label": "thordata-us-residential",
|
|
||||||
"proxy_url_env": "THORDATA_ROUTE_US"
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "openai-web-thordata-de",
|
|
||||||
"surface": "openai-web",
|
|
||||||
"account_label": null,
|
|
||||||
"browser_profile": null,
|
|
||||||
"network_label": "thordata-de-residential",
|
|
||||||
"proxy_url_env": "THORDATA_ROUTE_DE"
|
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "openai-api-1",
|
"name": "openai-api-1",
|
||||||
@@ -261,27 +276,34 @@ default; `--slots PATH` explicitly overrides it.
|
|||||||
```
|
```
|
||||||
|
|
||||||
A routed batch selects one slot explicitly. A Gemini slot must name the Google
|
A routed batch selects one slot explicitly. A Gemini slot must name the Google
|
||||||
account, its real Chrome profile, and its nonnegative `/u/N/` account index; an
|
account, its real Chrome profile, and its nonnegative `/u/N/` account index. An
|
||||||
API slot must name its account or project and cannot name a browser profile. The
|
API slot must name its account or project and cannot name a browser profile. A
|
||||||
|
Playwright slot for `openai-web`, `microsoft-web`, `meta-web`, or `synthid-web`
|
||||||
|
does not name an account or a browser profile. The
|
||||||
tool records the network route that the operator has already established.
|
tool records the network route that the operator has already established.
|
||||||
Non-Google Web slots may name `proxy_url_env`; Playwright reads the secret proxy
|
Those four pages may name `proxy_url_env`; Playwright reads the
|
||||||
URL from that environment variable without writing or logging it. The tool does
|
secret proxy URL from that environment variable without writing or logging it.
|
||||||
not merge sessions or automatically change network routes. Account continuation
|
The country and session are not part of that preparation. Each run replaces
|
||||||
is orchestrated by the skill, not by an individual adapter call. A request to
|
them with a new session in a country in Europe or the United States.
|
||||||
verify a bounded set of files with a provider covers its configured account slots
|
A signed-in account is required only for `gemini-web`. API calls stay on the
|
||||||
unless the operator restricts the account scope. Identify the finite set of
|
direct SDK transport and do not take `proxy_url_env`. Every other Web surface
|
||||||
available authenticated slots before starting and prepare one immutable batch
|
uses isolated Playwright and may use a proxy. The tool does
|
||||||
per slot. Record quota refusals as `refused` and service failures according to
|
not merge sessions inside one adapter call. Account continuation is orchestrated
|
||||||
their actual response, leave remaining rows in that slot unsubmitted, and
|
by the skill, not by an individual adapter call. A request to verify files with
|
||||||
continue automatically with the next authorized slot. Do not request permission
|
a provider covers its configured account slots, surfaces, and network routes.
|
||||||
again for files or accounts already covered by the task.
|
Anyone doing the task may run those checks. Identify the finite set of
|
||||||
|
configured slots before starting and prepare one immutable batch per slot.
|
||||||
|
Record quota refusals as `refused` and service failures according to their
|
||||||
|
actual response, leave remaining rows in that slot unsubmitted, and continue
|
||||||
|
with the next slot, surface, or configured route. Do not ask for confirmation
|
||||||
|
before that switch.
|
||||||
|
|
||||||
Submit each frozen artifact at most once per slot in that pass. Keep every
|
Submit each frozen artifact at most once per slot in one pass. Keep every
|
||||||
verbatim refusal, inconclusive reply, and disagreement between accounts. Stop
|
verbatim refusal, inconclusive reply, and disagreement between accounts. Stop
|
||||||
when the planned checks finish or no authorized usable slot remains, and report
|
when the plan is complete or no configured usable slot, surface, or route
|
||||||
unsubmitted rows. A new login, another provider, a new network route, or a new
|
remains, and report anything still open. Do not start a new login. Each
|
||||||
pass after the bounded plan is exhausted needs authorization unless already
|
submitted artifact stays frozen inside its own batch. Do not change candidate
|
||||||
requested. Candidate pixels remain frozen throughout account continuation.
|
pixels during the continuation.
|
||||||
|
|
||||||
Each `openai-api` slot must name exactly one `api_key_env`. Put the corresponding
|
Each `openai-api` slot must name exactly one `api_key_env`. Put the corresponding
|
||||||
values in the gitignored local `.env`, never in the slot JSON:
|
values in the gitignored local `.env`, never in the slot JSON:
|
||||||
@@ -290,29 +312,27 @@ values in the gitignored local `.env`, never in the slot JSON:
|
|||||||
OPENAI_API_KEY_1=replace-with-first-key
|
OPENAI_API_KEY_1=replace-with-first-key
|
||||||
OPENAI_API_KEY_2=replace-with-second-key
|
OPENAI_API_KEY_2=replace-with-second-key
|
||||||
OPENAI_API_KEY_3=replace-with-third-key
|
OPENAI_API_KEY_3=replace-with-third-key
|
||||||
THORDATA_ROUTE_US=http://td-customer-USERNAME-country-US:PASSWORD@HOST.pr.thordata.net:9999
|
THORDATA_PROXY=http://td-customer-USERNAME:PASSWORD@HOST.pr.thordata.net:9999
|
||||||
THORDATA_ROUTE_DE=http://td-customer-USERNAME-country-DE:PASSWORD@HOST.pr.thordata.net:9999
|
|
||||||
THORDATA_ROUTE_GB=http://td-customer-USERNAME-country-GB:PASSWORD@HOST.pr.thordata.net:9999
|
|
||||||
THORDATA_ROUTE_NL=http://td-customer-USERNAME-country-NL:PASSWORD@HOST.pr.thordata.net:9999
|
|
||||||
THORDATA_ROUTE_JP=http://td-customer-USERNAME-country-JP:PASSWORD@HOST.pr.thordata.net:9999
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The tool reads only the variable selected by the requested slot. A nonempty
|
The tool reads only the variable selected by the requested slot. A nonempty
|
||||||
process environment value overrides the matching `.env` entry. It never prints
|
process environment value overrides the matching `.env` entry. It never prints
|
||||||
the value or falls through to another token after authentication, entitlement,
|
the value. One call reads only its selected slot. After authentication,
|
||||||
quota, or transport failure. The no-slot compatibility path reads
|
entitlement, quota, or transport failure, continue with the next configured
|
||||||
|
slot in a new call. The no-slot compatibility path reads
|
||||||
`OPENAI_API_KEY`.
|
`OPENAI_API_KEY`.
|
||||||
|
|
||||||
Several Google accounts may share one real Chrome profile: give them distinct
|
Several Google accounts may share one real Chrome profile: give them distinct
|
||||||
slot names and `google_account_index` values while keeping `browser_profile` the
|
slot names and `google_account_index` values while keeping `browser_profile` the
|
||||||
same.
|
same.
|
||||||
|
|
||||||
The ThorData examples use its [residential proxy country parameter](https://doc.thordata.com/doc/proxies/residential-proxies/location-settings/country-region).
|
ThorData encodes the exit in the username with its [country parameter](https://doc.thordata.com/doc/proxies/residential-proxies/location-settings/country-region)
|
||||||
The proxy username encodes one explicit two-letter ISO country code. Route
|
and a [session id](https://doc.thordata.com/doc/proxies/residential-proxies/session-control).
|
||||||
selection remains explicit and never changes after a refusal. OpenAI,
|
A stored `country-XX` or `sessid` is not the exit this run will use. The runner
|
||||||
Microsoft, and Meta Web slots may reuse any route variable; the surface remains
|
discards both and appends one new session and one country from Europe or the
|
||||||
part of each distinct slot. OpenAI API slots never accept `proxy_url_env` and
|
United States. That pair stays for the browser context of the run. The next run
|
||||||
use the current direct network.
|
chooses again, including after a refusal. Gemini and API slots never accept
|
||||||
|
`proxy_url_env`. Anonymous Playwright slots may.
|
||||||
|
|
||||||
Validate and inspect the slots:
|
Validate and inspect the slots:
|
||||||
|
|
||||||
@@ -385,21 +405,28 @@ uv sync --extra dev
|
|||||||
uv run playwright install chromium
|
uv run playwright install chromium
|
||||||
```
|
```
|
||||||
|
|
||||||
Google is intentionally excluded from the standalone runner because its oracle
|
Gemini is the only surface excluded from the standalone runner, because it is
|
||||||
requires the user's existing authenticated Chrome session. Invoke the project
|
the only one that requires a signed-in account. Invoke the project
|
||||||
`provider-oracles` skill to prepare the batch, select the named real Chrome
|
`provider-oracles` skill to drive that existing Chrome profile at the named
|
||||||
profile and `/u/N/` Google account, upload each prepared artifact, capture the
|
`/u/N/` account. `synthid-web` is an anonymous page: `run-web` opens
|
||||||
settled SynthID tool outcome, and record it.
|
`https://synthid.com/` in isolated Playwright, attaches one prepared upload
|
||||||
|
through the page's file input, and reads the explicit `Watermark detected`,
|
||||||
|
`Watermark not detected`, or `SynthID was not detected` label. The portal has
|
||||||
|
no `/u/N/` index. If the page says `Please sign in before detection.` and
|
||||||
|
returns no watermark label, keep that row and continue the same prepared upload
|
||||||
|
in the user's existing real Chrome. That session is direct: no proxy, no new
|
||||||
|
login, and no cookie copy into the isolated browser. If that Chrome also asks
|
||||||
|
for sign-in, leave its row unsubmitted.
|
||||||
|
|
||||||
Ask the direct question stored in the batch manifest: `Does this
|
For Gemini, ask the direct question stored in the batch manifest: `Does this
|
||||||
image/video/audio contain a SynthID watermark? Use the SynthID verification
|
image/video/audio contain a SynthID watermark? Use the SynthID verification
|
||||||
tool only; do not infer from visible content, logos, text, or metadata.` A
|
tool only; do not infer from visible content, logos, text, or metadata.` A
|
||||||
general question about whether Google AI created or edited the media is not
|
general question about whether Google AI created or edited the media is not
|
||||||
equivalent: it may elicit origin or visual-content reasoning without a SynthID
|
equivalent: it may elicit origin or visual-content reasoning without a SynthID
|
||||||
verdict.
|
verdict. SynthID Detector has no prompt; its settled labels are the verdict.
|
||||||
|
|
||||||
OpenAI Web, Microsoft Web, and Meta run entirely through a fresh isolated
|
OpenAI Web, Microsoft Web, Meta, and SynthID Detector run through a fresh
|
||||||
Playwright browser. A direct-route example:
|
isolated Playwright browser. A direct-route example:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run python scripts/provider_oracles.py prepare meta-web input.png \
|
uv run python scripts/provider_oracles.py prepare meta-web input.png \
|
||||||
@@ -410,8 +437,9 @@ uv run python scripts/provider_oracles.py run-web \
|
|||||||
/tmp/meta-check-001/manifest.json --acknowledge-uploads
|
/tmp/meta-check-001/manifest.json --acknowledge-uploads
|
||||||
```
|
```
|
||||||
|
|
||||||
For a ThorData-routed OpenAI, Microsoft, or Meta Web slot, keep the generated
|
For an OpenAI, Microsoft, Meta, or SynthID Detector slot that names a ThorData
|
||||||
proxy URLs in the gitignored `.env`, then select one route explicitly:
|
credential, keep that URL in the gitignored `.env`. The run chooses the country
|
||||||
|
and the session:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv run python scripts/provider_oracles.py run-web \
|
uv run python scripts/provider_oracles.py run-web \
|
||||||
@@ -428,12 +456,10 @@ ThorData [user-and-password endpoints without a `sessid` rotate the exit on each
|
|||||||
request](https://doc.thordata.com/doc/proxies/residential-proxies/session-control).
|
request](https://doc.thordata.com/doc/proxies/residential-proxies/session-control).
|
||||||
That is unsuitable inside one browser context: page resources can
|
That is unsuitable inside one browser context: page resources can
|
||||||
arrive through different exits and trigger the provider's bot checks. For a
|
arrive through different exits and trigger the provider's bot checks. For a
|
||||||
ThorData hostname, the runner derives a non-secret sticky session id from the
|
ThorData hostname, each run therefore discards a prepared country and session,
|
||||||
immutable manifest hash and appends a 90-minute session window when the
|
then keeps one new session and one country in Europe or the United States for
|
||||||
configured username has no explicit session. The same batch therefore keeps
|
that browser context. The next invocation, including a retry after a refusal,
|
||||||
one route while the provider retains it, a new prepared batch gets a distinct
|
chooses again.
|
||||||
route, and an operator-supplied `sessid` remains unchanged. This pins a selected
|
|
||||||
route; it does not retry or fail over after a refusal.
|
|
||||||
For a selected proxy route it enables certificate handling in that isolated
|
For a selected proxy route it enables certificate handling in that isolated
|
||||||
Chromium launch and context; direct launches keep Playwright's normal TLS
|
Chromium launch and context; direct launches keep Playwright's normal TLS
|
||||||
verification. A 2026-09-07 preflight sent one read-only request through each
|
verification. A 2026-09-07 preflight sent one read-only request through each
|
||||||
@@ -450,7 +476,7 @@ quota refusal is `refused`, a transport failure is `unreachable`, and unknown or
|
|||||||
changed page wording is `indeterminate`.
|
changed page wording is `indeterminate`.
|
||||||
|
|
||||||
Each provider mounts its file input before the upload handler is necessarily
|
Each provider mounts its file input before the upload handler is necessarily
|
||||||
ready. Microsoft and Meta reach a usable handler at network idle. OpenAI keeps
|
ready. Microsoft, Meta, and SynthID Detector reach a usable handler at network idle. OpenAI keeps
|
||||||
background requests alive and never reaches network idle, so its observable
|
background requests alive and never reaches network idle, so its observable
|
||||||
readiness seam is the page `load` state instead. The adapter waits for the
|
readiness seam is the page `load` state instead. The adapter waits for the
|
||||||
surface-specific state before assigning the file. OpenAI's static FAQ contains
|
surface-specific state before assigning the file. OpenAI's static FAQ contains
|
||||||
|
|||||||
@@ -79,7 +79,8 @@ For a path input, `remove_visible` and `remove_visible_detailed`:
|
|||||||
- reads metadata provenance for the default `auto` sensitivity;
|
- reads metadata provenance for the default `auto` sensitivity;
|
||||||
- preserves a separate alpha channel;
|
- preserves a separate alpha channel;
|
||||||
- writes the output when an output path is supplied;
|
- writes the output when an output path is supplied;
|
||||||
- strips AI metadata from the written output by default;
|
- strips AI metadata from the written output by default, and raises
|
||||||
|
`MetadataStripIncomplete` without publishing it when a marker survives;
|
||||||
- preserves the original bytes for a same-format no-op copy.
|
- preserves the original bytes for a same-format no-op copy.
|
||||||
|
|
||||||
```python
|
```python
|
||||||
@@ -226,8 +227,9 @@ The complete field set, with the shipped defaults:
|
|||||||
and not to this table, because this seam had already drifted by eight fields
|
and not to this table, because this seam had already drifted by eight fields
|
||||||
before anything checked it.
|
before anything checked it.
|
||||||
|
|
||||||
If AI metadata survives the strip, `remove_all` raises `MetadataStripIncomplete`
|
If AI metadata survives the strip, `remove_all` and a path-input `remove_visible`
|
||||||
**before** writing anything: an AI-readable output is worse than no output.
|
raise `MetadataStripIncomplete` **before** publishing anything: an AI-readable
|
||||||
|
output is worse than no output.
|
||||||
|
|
||||||
`remove_batch` runs one mode over a directory and never lets a single bad file
|
`remove_batch` runs one mode over a directory and never lets a single bad file
|
||||||
end the run:
|
end the run:
|
||||||
@@ -525,7 +527,9 @@ It checks the written output and returns `(output_path, surviving_markers)`.
|
|||||||
When the first strip leaves markers in a malformed but raster-decodable image,
|
When the first strip leaves markers in a malformed but raster-decodable image,
|
||||||
it normalizes the container through `image_io` and checks again. That recovery
|
it normalizes the container through `image_io` and checks again. That recovery
|
||||||
path preserves the pixels but drops standard metadata. Treat a nonempty
|
path preserves the pixels but drops standard metadata. Treat a nonempty
|
||||||
`surviving_markers` mapping as a failure.
|
`surviving_markers` mapping as a failure. A source that is not a readable image,
|
||||||
|
video or audio container raises `ValueError` before anything is written;
|
||||||
|
`metadata.unreadable_reason(path)` returns the same reason without stripping.
|
||||||
|
|
||||||
`remove_ai_metadata` is the lower level fail-safe transformer. It may copy an
|
`remove_ai_metadata` is the lower level fail-safe transformer. It may copy an
|
||||||
undecodable input through unchanged, so its return alone must not be presented
|
undecodable input through unchanged, so its return alone must not be presented
|
||||||
|
|||||||
@@ -24,9 +24,10 @@ The package version appears in:
|
|||||||
|
|
||||||
- `pyproject.toml`;
|
- `pyproject.toml`;
|
||||||
- `src/remove_ai_watermarks/__init__.py`;
|
- `src/remove_ai_watermarks/__init__.py`;
|
||||||
|
- both `version` fields in `server.json`;
|
||||||
- the root package entry generated in `uv.lock`.
|
- the root package entry generated in `uv.lock`.
|
||||||
|
|
||||||
Update the first two, then refresh the lock file with uv. Do not edit a
|
Update the first three, then refresh the lock file with uv. Do not edit a
|
||||||
line-number-specific location in `uv.lock`; its package order changes.
|
line-number-specific location in `uv.lock`; its package order changes.
|
||||||
|
|
||||||
Before choosing a version, verify it is absent from git tags, GitHub Releases,
|
Before choosing a version, verify it is absent from git tags, GitHub Releases,
|
||||||
@@ -159,7 +160,10 @@ The Agent Skill and Claude plugin live in this repository:
|
|||||||
is the Claude plugin manifest, with source `./skills` so an install copies
|
is the Claude plugin manifest, with source `./skills` so an install copies
|
||||||
only the skill tree;
|
only the skill tree;
|
||||||
- [`.claude-plugin/marketplace.json`](../.claude-plugin/marketplace.json)
|
- [`.claude-plugin/marketplace.json`](../.claude-plugin/marketplace.json)
|
||||||
makes the repository a Claude Code marketplace.
|
makes the repository a Claude Code marketplace;
|
||||||
|
- [`.codex-plugin/plugin.json`](../.codex-plugin/plugin.json) is the Codex
|
||||||
|
plugin manifest, pointing at the same `./skills/` tree, with its icon in
|
||||||
|
[`assets/icon.svg`](../assets/icon.svg).
|
||||||
|
|
||||||
They are not a fifth required release surface. A CLI change that affects
|
They are not a fifth required release surface. A CLI change that affects
|
||||||
command routing, extras, exit codes, mark keys, or the intended-use boundary
|
command routing, extras, exit codes, mark keys, or the intended-use boundary
|
||||||
@@ -215,6 +219,25 @@ the catalog has not accepted the initial submission. The catalog-specific
|
|||||||
submission state and links live in
|
submission state and links live in
|
||||||
[`docs/agent-skill.md`](agent-skill.md#where-the-skill-can-actually-be-listed).
|
[`docs/agent-skill.md`](agent-skill.md#where-the-skill-can-actually-be-listed).
|
||||||
|
|
||||||
|
## MCP registry
|
||||||
|
|
||||||
|
[`server.json`](../server.json) lists the PyPI package in the
|
||||||
|
[official MCP registry](https://registry.modelcontextprotocol.io/) as
|
||||||
|
`io.github.wiltodelta/remove-ai-watermarks`, started as
|
||||||
|
`uvx remove-ai-watermarks mcp`. The registry accepts it only after PyPI serves a
|
||||||
|
release whose description, the README, carries the `mcp-name:` marker, and
|
||||||
|
`tests/test_mcp_server.py` pins `server.json` to the package version. The listing
|
||||||
|
is not a release surface that blocks a release.
|
||||||
|
|
||||||
|
[`publish-mcp-registry.yml`](../.github/workflows/publish-mcp-registry.yml)
|
||||||
|
publishes it on every published GitHub Release, once PyPI serves that version.
|
||||||
|
GitHub OIDC proves ownership of the `io.github.wiltodelta` namespace, so no login
|
||||||
|
or secret is stored. To list a version already on PyPI, dispatch the workflow
|
||||||
|
with that version; a version the registry already lists is skipped.
|
||||||
|
|
||||||
|
[`glama.json`](../glama.json) names the maintainer for Glama's claim flow.
|
||||||
|
Client setup and the tool contract are in [the MCP server guide](mcp-server.md).
|
||||||
|
|
||||||
## Release verification
|
## Release verification
|
||||||
|
|
||||||
Forensic transports are versioned independently from the package. Before publishing
|
Forensic transports are versioned independently from the package. Before publishing
|
||||||
|
|||||||
@@ -33,22 +33,28 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
| Dependencies | TrustMark extra on every supported Python (`trustmark>=0.9.2`) | [installation.md](installation.md) |
|
| Dependencies | TrustMark extra on every supported Python (`trustmark>=0.9.2`) | [installation.md](installation.md) |
|
||||||
| Live captures | Gemini Omni, Grok, Vidu, Wan, Apple, Google Photos, Nano Banana 2 Lite and Gemini Omni Flash through the API; Runway, Higgsfield and Dreamina | [watermarking-landscape.md](watermarking-landscape.md#live-captures-2026-09-23-and-2026-09-24) |
|
| Live captures | Gemini Omni, Grok, Vidu, Wan, Apple, Google Photos, Nano Banana 2 Lite and Gemini Omni Flash through the API; Runway, Higgsfield and Dreamina | [watermarking-landscape.md](watermarking-landscape.md#live-captures-2026-09-23-and-2026-09-24) |
|
||||||
|
|
||||||
## Open items
|
## Follow-up status
|
||||||
|
|
||||||
- **Google Photos Reimagine manifests.** Magic Editor, and so Reimagine, is not
|
The original follow-up list now mixes completed experiments, deliberately
|
||||||
offered in Google Photos on iOS, so no Reimagine file is captured. The
|
excluded claims, and acquisition gaps. Each row below names its current status
|
||||||
|
as of 2026-10-01 so this section can serve as the research backlog without
|
||||||
|
discarding the measured results.
|
||||||
|
|
||||||
|
- **Closed - Google Photos Reimagine manifests.** Magic Editor, and so
|
||||||
|
Reimagine, is not offered in Google Photos on iOS, so no Reimagine file is
|
||||||
|
captured. The
|
||||||
question it was meant to answer is settled without one: Google's checker found
|
question it was meant to answer is settled without one: Google's checker found
|
||||||
SynthID in all four Photos AI edits tested on 2026-09-25, none of which
|
SynthID in all four Photos AI edits tested on 2026-09-25, none of which
|
||||||
records SynthID in its manifest.
|
records SynthID in its manifest.
|
||||||
- **Vidu and Wan marks on more samples.** Both are registered from one real
|
- **Open - Vidu and Wan marks on more samples.** Both are registered from one
|
||||||
export each (`vidu` video, `wan` image); their gates are provisional until a
|
real export each (`vidu` video, `wan` image); their gates are provisional until a
|
||||||
wider cohort is measured, and a Wan video label is not yet captured. Public
|
wider cohort is measured, and a Wan video label is not yet captured. Public
|
||||||
Vidu feed videos (2026-09-26) carry the TC260 label naming ShengShu, and one
|
Vidu feed videos (2026-09-26) carry the TC260 label naming ShengShu, and one
|
||||||
names Tongyi Yunqi as producer with Vidu as propagator, but none shows the
|
names Tongyi Yunqi as producer with Vidu as propagator, but none shows the
|
||||||
visible `Vidu` mark; Wan's public demo videos show no mark either. Both marks
|
visible `Vidu` mark; Wan's public demo videos show no mark either. Both marks
|
||||||
appear only on account downloads.
|
appear only on account downloads.
|
||||||
- **TC260 practice guides.** The audio placements from TC260-PG-202510A are
|
- **Partially complete - TC260 practice guides.** The audio placements from
|
||||||
now read (the guide PDFs load in a browser from
|
TC260-PG-202510A are now read (the guide PDFs load in a browser from
|
||||||
`tc260.org.cn/tc260/sjzn/list.shtml`, not to scripted requests). The text-file
|
`tc260.org.cn/tc260/sjzn/list.shtml`, not to scripted requests). The text-file
|
||||||
guide (TC260-PG-20258A) covers documents this image and video tool does not
|
guide (TC260-PG-20258A) covers documents this image and video tool does not
|
||||||
process. The security-protection guide (TC260-PG-202511A) records a
|
process. The security-protection guide (TC260-PG-202511A) records a
|
||||||
@@ -66,15 +72,25 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
point; changing one `ProduceID` digit fails it. MiniMax's own spellings depart
|
point; changing one `ProduceID` digit fails it. MiniMax's own spellings depart
|
||||||
from the guide: `TBSData.Type` is `LabelMataData` (the guide's `Md`) and
|
from the guide: `TBSData.Type` is `LabelMataData` (the guide's `Md`) and
|
||||||
`Binding` with `BType` `0` (its `Bnd`); the second signature verifies over the
|
`Binding` with `BType` `0` (its `Bnd`); the second signature verifies over the
|
||||||
compact JSON of the `Bindings` entry, also on all four. That entry's SM3
|
compact JSON of the `Bindings` entry, also on all four. TC260-PG-202511A
|
||||||
content hash matched neither the whole file nor the file without its label,
|
appendix A.2 was re-read from the
|
||||||
and it names no `CntSel`, so the content binding is unverified. Other producers fill the same
|
[official PDF](https://www.tc260.org.cn/tc260/sjzn/202508/63ab55a3265f45c7865e9b8be15d4a29/files/dc379ec776f94abc8f1806affa1cda35.pdf)
|
||||||
|
on 2026-10-01. MiniMax omits its `CntSel`, and none of the guide-derived
|
||||||
|
byte selections reproduced its SM3 content hash on four measured videos, so
|
||||||
|
the content binding remains unverified
|
||||||
|
([module internals](module-internals.md)). Other producers fill the same
|
||||||
fields with opaque vendor strings (Qwen a base64 digest, Vidu 32 hex digits,
|
fields with opaque vendor strings (Qwen a base64 digest, Vidu 32 hex digits,
|
||||||
Wan and HappyHorse `L-`/`K-` prefixed values), not `SecurityData`. The key sits
|
Wan and HappyHorse `L-`/`K-` prefixed values), not `SecurityData`. The key sits
|
||||||
in the file it signs, so a check proves authenticity only against a pinned
|
in the file it signs, so a check proves authenticity only against a pinned
|
||||||
MiniMax key. `identify` now verifies it with a built-in SM3/SM2 and a pinned
|
MiniMax key. `identify` now verifies it with a built-in SM3/SM2 and a pinned
|
||||||
MiniMax key ([module internals](module-internals.md)).
|
MiniMax key. The remaining work is
|
||||||
- **Experiments.** Chroma restoration after regeneration (arXiv:2508.21072) was
|
MiniMax content-binding verification if its byte-selection rule becomes known,
|
||||||
|
and Honor's unsupported RSA/content form if an original export and signing
|
||||||
|
contract become available.
|
||||||
|
- **Partially complete - experiments.** Five of the six follow-ups are closed:
|
||||||
|
Z-Image-Turbo, Perth and PixelSeal, re-watermarking, VACE/ROSE, and newer
|
||||||
|
frozen-backbone probes. Chroma
|
||||||
|
restoration after regeneration (arXiv:2508.21072) was
|
||||||
measured on 2026-09-27 and is not shipped: transplanted source chroma drew one
|
measured on 2026-09-27 and is not shipped: transplanted source chroma drew one
|
||||||
explicit "inconclusive" and two unclear Gemini answers where the untouched
|
explicit "inconclusive" and two unclear Gemini answers where the untouched
|
||||||
outputs were all clean, and ghosts redrawn objects
|
outputs were all clean, and ghosts redrawn objects
|
||||||
@@ -126,12 +142,20 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
profile; retain its harness and local evidence for future comparison only.
|
profile; retain its harness and local evidence for future comparison only.
|
||||||
The existing source-conditioned Z-Image face-repair stage remains part of
|
The existing source-conditioned Z-Image face-repair stage remains part of
|
||||||
`qwen-zimage`, `sdxl-zimage`, and `chroma-zimage`.
|
`qwen-zimage`, `sdxl-zimage`, and `chroma-zimage`.
|
||||||
Still open: linear probes on newer frozen backbones for `classify` and RAVEN
|
The pre-registered modern-backbone campaign found that neither frozen
|
||||||
view-synthesis removal
|
SigLIP2 Base nor PE-Core B could match the production model's combined photo,
|
||||||
(arXiv:2601.08832, no code released as of 2026-09-24).
|
AI, and FLUX constraints, so the replacement hypothesis is closed
|
||||||
|
([classifier research](ai-generated-image-classifiers.md#modern-frozen-backbone-campaign-closed-2026-09-27)).
|
||||||
|
Only RAVEN view-synthesis removal remains blocked. RAVEN's
|
||||||
|
[official repository](https://github.com/fahadshamshad/raven-watermark-removal)
|
||||||
|
now exists, but its 2026-08-08 README is still a release placeholder: the
|
||||||
|
authors say they will publish code after their SynthID testing and Google
|
||||||
|
Security Bounty submission. There is no implementation to evaluate as of
|
||||||
|
2026-10-17, when the repository still held only that README (arXiv:2601.08832).
|
||||||
MiniMax-Remover (CC-BY-NC-4.0 weights) and ProPainter (S-Lab License,
|
MiniMax-Remover (CC-BY-NC-4.0 weights) and ProPainter (S-Lab License,
|
||||||
non-commercial) are excluded on license.
|
non-commercial) are deliberately excluded on license and are not backlog
|
||||||
- **New C2PA signers without a vendor row.** The
|
items.
|
||||||
|
- **Open - new C2PA signers without a vendor row.** The
|
||||||
[C2PA conformance list](https://github.com/c2pa-org/conformance-public/blob/main/conforming-products/conforming-products-list.json)
|
[C2PA conformance list](https://github.com/c2pa-org/conformance-public/blob/main/conforming-products/conforming-products-list.json)
|
||||||
names generator certificates `Amazon Bedrock`, `Getty Images
|
names generator certificates `Amazon Bedrock`, `Getty Images
|
||||||
AI-Generated Image` and `AI-Modified Image` (and the iStock pair), `Jasper`,
|
AI-Generated Image` and `AI-Modified Image` (and the iStock pair), `Jasper`,
|
||||||
@@ -148,7 +172,7 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
carrying C2PA: their marketing and model-page images are unsigned. A vivo
|
carrying C2PA: their marketing and model-page images are unsigned. A vivo
|
||||||
X300 camera capture carries a vivo device manifest (`digitalCapture`) that
|
X300 camera capture carries a vivo device manifest (`digitalCapture`) that
|
||||||
the bundled reader cannot open (`unknown algorithm`).
|
the bundled reader cannot open (`unknown algorithm`).
|
||||||
- **Chinese phone galleries.** Huawei's help pages say the Xiaoyi photo-edit AI
|
- **Open - Chinese phone galleries.** Huawei's help pages say the Xiaoyi photo-edit AI
|
||||||
watermark can be switched off; vivo's Album setting is already covered.
|
watermark can be switched off; vivo's Album setting is already covered.
|
||||||
The public-web follow-up below found Honor YOYO metadata and additional
|
The public-web follow-up below found Honor YOYO metadata and additional
|
||||||
primary-source labeling evidence, but not a controlled gallery export set. A live
|
primary-source labeling evidence, but not a controlled gallery export set. A live
|
||||||
@@ -159,11 +183,12 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
of an unedited control, an ordinary crop, and a generative edit are still needed
|
of an unedited control, an ordinary crop, and a generative edit are still needed
|
||||||
per app before selecting an AI attribution rule. Record the device, OS/app
|
per app before selecting an AI attribution rule. Record the device, OS/app
|
||||||
version, region, operation, and original download path with each sample.
|
version, region, operation, and original download path with each sample.
|
||||||
- **Claims not supported by a primary source.** Kling embedding SynthID
|
- **Closed - claims not supported by a primary source.** Kling embedding SynthID
|
||||||
(repeated by remover blogs; Kuaishou is not in Google's adopter list),
|
(repeated by remover blogs; Kuaishou is not in Google's adopter list),
|
||||||
Hailuo pixel watermarks, and platforms stripping C2PA on upload. None is used
|
Hailuo pixel watermarks, and platforms stripping C2PA on upload. None is used
|
||||||
by the library.
|
by the library.
|
||||||
- **Google Photos small edits (further work deferred, 2026-10-01).** The
|
- **Partially complete - Google Photos small edits (further work deferred,
|
||||||
|
2026-10-01).** The
|
||||||
maintainer deferred further photographic tiny Magic Eraser captures and
|
maintainer deferred further photographic tiny Magic Eraser captures and
|
||||||
verification retries for the inconclusive small edits. Resume only on an
|
verification retries for the inconclusive small edits. Resume only on an
|
||||||
explicit request; retain the existing evidence and uncertainty. On
|
explicit request; retain the existing evidence and uncertainty. On
|
||||||
@@ -193,8 +218,13 @@ This page is an index. Each finding lives in its canonical page.
|
|||||||
negative and one inconclusive reply. The quota refusal, synthetic capture,
|
negative and one inconclusive reply. The quota refusal, synthetic capture,
|
||||||
pixel comparison, mode caveat, and hash-bound outcomes are recorded in
|
pixel comparison, mode caveat, and hash-bound outcomes are recorded in
|
||||||
[synthid.md](synthid.md).
|
[synthid.md](synthid.md).
|
||||||
- **Captures blocked by region or device.** Doubao, Jimeng, Samsung. Higgsfield,
|
- **Partially complete - direct captures blocked by region or device.** Direct,
|
||||||
Runway and Dreamina were captured on paid plans on 2026-09-24.
|
controlled captures remain blocked for Doubao, Jimeng, and Samsung. The
|
||||||
|
public-web follow-up below later found one real marked Doubao video and
|
||||||
|
validated its visible mark and TC260 metadata, but the file remains local
|
||||||
|
because publication has not been cleared; it is not a controlled account
|
||||||
|
capture or a redistributable fixture. Higgsfield, Runway, and Dreamina were
|
||||||
|
captured on paid plans on 2026-09-24.
|
||||||
|
|
||||||
### Public-web sample search, 2026-10-01 UTC
|
### Public-web sample search, 2026-10-01 UTC
|
||||||
|
|
||||||
@@ -251,10 +281,16 @@ provider exports. No third-party photograph was added to tracked fixtures.
|
|||||||
[Getty's generator page](https://www.gettyimages.com/ai/generation),
|
[Getty's generator page](https://www.gettyimages.com/ai/generation),
|
||||||
[iStock's generator page](https://www.istockphoto.com/ai/generation), and
|
[iStock's generator page](https://www.istockphoto.com/ai/generation), and
|
||||||
[Reface's product site](https://reface.ai/) did not carry locally recognized
|
[Reface's product site](https://reface.ai/) did not carry locally recognized
|
||||||
C2PA. Jasper explicitly distinguishes input/mask/output examples, but an
|
C2PA. A 2026-10-01 recheck of Jasper's current official Image landing page
|
||||||
unsigned web derivative cannot establish its signed export's manifest shape.
|
reached its pre-rendered endpoint outputs directly and again found no C2PA.
|
||||||
|
Jasper explicitly distinguishes input/mask/output examples, but an unsigned
|
||||||
|
web derivative cannot establish its signed export's manifest shape.
|
||||||
Conformance certificates alone still do not justify fabricating that shape.
|
Conformance certificates alone still do not justify fabricating that shape.
|
||||||
- **Visible-image/video gaps remain.** The downloaded RunningHub workflow
|
- **Visible-image/video provider-original gaps remain.** Every registered mark
|
||||||
|
already has a publication-safe synthetic detector fixture in the
|
||||||
|
[visible-mark gallery](../data/fixtures/visible/README.md); the sample ledgers
|
||||||
|
now point to those fixtures and distinguish them from provider evidence. The
|
||||||
|
downloaded RunningHub workflow
|
||||||
illustrations are explanatory/branding assets, not verified output examples;
|
illustrations are explanatory/branding assets, not verified output examples;
|
||||||
Liblib API documentation's cat illustration has no visible historical wordmark.
|
Liblib API documentation's cat illustration has no visible historical wordmark.
|
||||||
A public [Inkpainting model page](https://www.liblib.art/modelinfo/2acd393c23924ff9b5363c9c1ec9129f)
|
A public [Inkpainting model page](https://www.liblib.art/modelinfo/2acd393c23924ff9b5363c9c1ec9129f)
|
||||||
|
|||||||
@@ -11,19 +11,21 @@ The `visible` command registers these mark keys:
|
|||||||
| Key | Mark | Expected area | Important limit |
|
| Key | Mark | Expected area | Important limit |
|
||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `gemini` | Google Gemini visible watermark (sparkle) | Usually bottom right | Detection includes a false positive gate. |
|
| `gemini` | Google Gemini visible watermark (sparkle) | Usually bottom right | Detection includes a false positive gate. |
|
||||||
| `doubao` | `豆包AI生成` | Bottom right | Vendor specific text detector. |
|
| `doubao` | `豆包AI生成` | Bottom right | Covers filled and outlined text layouts; the weak outlined layout requires matching Doubao provenance. |
|
||||||
| `jimeng` | `★ 即梦AI` | Bottom right | Vendor specific text detector. |
|
| `jimeng` | `★ 即梦AI` | Bottom right | Vendor specific text detector. |
|
||||||
| `qwen` | `千问AI生成` or the three-lobe Qwen symbol | Bottom right | Separate strict templates cover the text and Qwen Create symbol variants. |
|
| `qwen` | `千问AI生成` or the three-lobe Qwen symbol | Bottom right | Separate strict templates cover the text and Qwen Create symbol variants. |
|
||||||
| `wan` | Wan logo and `Wan` wordmark | Bottom right, hugging the corner | Provisional: one real Wan 2.7 Pro export. Requires NCC >= 0.55 and a match that ends within 0.015 of the short side from both edges. |
|
| `wan` | Wan logo and `Wan` wordmark | Bottom right, hugging the corner | Provisional: one real Wan 2.7 Pro export. Requires NCC >= 0.55 and a match that ends within 0.015 of the short side from both edges. |
|
||||||
| `kling` | `可灵AI 3.0` or `KlingAI 3.0` | Bottom right | Separate calibrated silhouettes cover the older CJK and current IMAGE 3.0 Latin variants. |
|
| `kling` | `可灵AI 3.0` or `KlingAI 3.0` | Bottom right | Separate calibrated silhouettes cover the older CJK and current IMAGE 3.0 Latin variants; a glyph-detail gate rejects plain UI boundaries. |
|
||||||
| `yuanbao` | `元宝` over `AI生成` | Bottom right | Standard two-line variant only. |
|
| `yuanbao` | `元宝` over `AI生成` | Bottom right | Separate strict and provenance-corroborated templates cover the measured two-line rasterizations. |
|
||||||
| `samsung` | `✦ Contenuti generati dall'AI` | Bottom left | Calibrated for the Italian text variant. |
|
| `dola` | `Dola AI` | Bottom right | Text detector and detector-aligned wordmark footprint. |
|
||||||
|
| `workbuddy` | `AI生成` over `WORKBUDDY>_` | Bottom right | Text detector and detector-aligned stacked-label footprint. |
|
||||||
|
| `samsung` | Italian or Korean Galaxy AI disclosure | Bottom left | Independently gated text layouts cover `Contenuti generati dall'AI` and `AI로 생성한 콘텐츠`. |
|
||||||
| `runninghub` | `RunningHub AI生成` | Top left | Strict visual and position gates. |
|
| `runninghub` | `RunningHub AI生成` | Top left | Strict visual and position gates. |
|
||||||
| `baidu` | `百度 AI生成` | Bottom right | Detector and extended removal footprint. |
|
| `baidu` | `百度 AI生成` | Bottom right | Detector and extended removal footprint. |
|
||||||
| `liblib` | `LiblibAI` | Bottom center | Historical wordmark variant; includes a minimum image size gate. |
|
| `liblib` | `LiblibAI` | Bottom center | Historical wordmark variant; includes a minimum image size gate. |
|
||||||
| `liblib_pill` | Compact `AI生成` pill | Top left | Auto removal requires LiblibAI metadata or the bottom-center wordmark, plus a flat background under the pill. |
|
| `liblib_pill` | Compact `AI生成` pill | Top left | Auto removal requires LiblibAI metadata or the bottom-center wordmark, plus a flat background under the pill. |
|
||||||
| `microsoft` | One Microsoft white AI-badge variant | Top right | Strict uses the visual gate; auto can use Microsoft provenance for the measured [relaxed gate](module-internals.md#visible-mark-removal). Other documented icon, text, and position variants are not covered. |
|
| `microsoft` | Microsoft white AI badge or beige `Made with AI` pill | Top right | Each layout has its own strict visual gate; auto can use Microsoft provenance for the original badge's measured [relaxed gate](module-internals.md#visible-mark-removal). |
|
||||||
| `jimeng_pill` | `AI生成` pill | Top left | Weak detector with additional product and background gates. |
|
| `jimeng_pill` | `AI生成` pill | Top left | A verified label layout can bypass the flat-background guard only with Jimeng provenance; the weak outline detector retains the product and background gates. |
|
||||||
| `generic_ai_label` | Bare `AI生成` text, no vendor wordmark | Bottom right | Brand-less TC260 compliance-stamp fallback (vivo and Xiaomi Gallery AI-edit output confirmed; see [module-internals.md](module-internals.md#generic-bare-ai生成-label)). Font-rendered synthetic asset; **calibrated on a synthetic corpus only**, not a real captured generator screenshot -- treat as weaker evidence than the vendor-tuned detectors above. |
|
| `generic_ai_label` | Bare `AI生成` text, no vendor wordmark | Bottom right | Brand-less TC260 compliance-stamp fallback (vivo and Xiaomi Gallery AI-edit output confirmed; see [module-internals.md](module-internals.md#generic-bare-ai生成-label)). Font-rendered synthetic asset; **calibrated on a synthetic corpus only**, not a real captured generator screenshot -- treat as weaker evidence than the vendor-tuned detectors above. |
|
||||||
| `openart` | `OpenArt` wordmark (bowtie/infinity icon + brand name) | Frame center, not a corner | Confirmed from one customer-reported production case; no known metadata signal, so the visible mark is the only attribution route. **UNCALIBRATED**: procedurally reconstructed asset, no captured corpus -- see [module-internals.md](module-internals.md#openart-wordmark). |
|
| `openart` | `OpenArt` wordmark (bowtie/infinity icon + brand name) | Frame center, not a corner | Confirmed from one customer-reported production case; no known metadata signal, so the visible mark is the only attribution route. **UNCALIBRATED**: procedurally reconstructed asset, no captured corpus -- see [module-internals.md](module-internals.md#openart-wordmark). |
|
||||||
|
|
||||||
@@ -93,6 +95,8 @@ The inspection and stripping code handles signals in these groups:
|
|||||||
- exact app-export provenance and AIGC disclosures from supported
|
- exact app-export provenance and AIGC disclosures from supported
|
||||||
ByteDance-family products, with product-only provenance excluded from the
|
ByteDance-family products, with product-only provenance excluded from the
|
||||||
generated-image verdict;
|
generated-image verdict;
|
||||||
|
- Kling's H.264 `user_data_unregistered` SEI tag (`kling-ai`) in MP4/MOV
|
||||||
|
video, blanked in place by `video metadata --remove` without re-encoding;
|
||||||
- IPTC AI disclosure fields;
|
- IPTC AI disclosure fields;
|
||||||
- PNG text chunks and embedded generation parameters, including keyed
|
- PNG text chunks and embedded generation parameters, including keyed
|
||||||
`workflow` and `prompt` tags in MP4/MOV metadata lists;
|
`workflow` and `prompt` tags in MP4/MOV metadata lists;
|
||||||
@@ -298,7 +302,7 @@ not a universal clean verdict.
|
|||||||
| Luma AI | None registered | No registered pixel decoder | PNG text generator tags (Uni-1) |
|
| Luma AI | None registered | No registered pixel decoder | PNG text generator tags (Uni-1) |
|
||||||
| ByteDance generators | Doubao and Jimeng marks | No registered pixel decoder | TC260 AIGC, supported C2PA, and exact app-export AIGC disclosures |
|
| ByteDance generators | Doubao and Jimeng marks | No registered pixel decoder | TC260 AIGC, supported C2PA, and exact app-export AIGC disclosures |
|
||||||
| Qwen | Qwen mark | No registered pixel decoder | TC260 AIGC |
|
| Qwen | Qwen mark | No registered pixel decoder | TC260 AIGC |
|
||||||
| Kling AI | Kling AI image and video marks | No registered pixel decoder | TC260 AIGC |
|
| Kling AI | Kling AI image and video marks | No registered pixel decoder | TC260 AIGC and the video SEI tag |
|
||||||
| Vidu (ShengShu) | Vidu video mark | No registered pixel decoder | TC260 AIGC in the MP4 `AIGC` key |
|
| Vidu (ShengShu) | Vidu video mark | No registered pixel decoder | TC260 AIGC in the MP4 `AIGC` key |
|
||||||
| Runway | None measured on paid-plan output | No registered pixel decoder | C2PA signed `RUNWAY AI, INC.` on its own models; third-party models keep their vendor's manifest |
|
| Runway | None measured on paid-plan output | No registered pixel decoder | C2PA signed `RUNWAY AI, INC.` on its own models; third-party models keep their vendor's manifest |
|
||||||
| Amazon Bedrock (Nova Canvas, Nova Reel) | None | Invisible watermark readable only through the Bedrock API | C2PA signed `Amazon Web Services, Inc.`, claim generator `Amazon Bedrock` |
|
| Amazon Bedrock (Nova Canvas, Nova Reel) | None | Invisible watermark readable only through the Bedrock API | C2PA signed `Amazon Web Services, Inc.`, claim generator `Amazon Bedrock` |
|
||||||
|
|||||||
@@ -200,7 +200,7 @@ route in this campaign. Mechanism detail stays in
|
|||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| Gowal et al., [arXiv:2510.09263](https://arxiv.org/abs/2510.09263) | Post-hoc encoder `f` / decoder `g`. Detection logit is not payload recovery. SynthID-O (partner variant) 136 bits at 512x512. TPR at 0.1% FPR 99.98% aggregated, 99.97% on the hardest spatial crop+resize. Trains against sampled semantics-preserving transforms, including weak VAE regeneration. Production decoder unpublished | Matches the architecture we treat as keyed `x' = x + g(x)`. Explains why a two-pixel crop kills `pipeline_lattice` but not the official oracle, and why a 22.3 dB foreign VAE still reads `detected` |
|
| Gowal et al., [arXiv:2510.09263](https://arxiv.org/abs/2510.09263) | Post-hoc encoder `f` / decoder `g`. Detection logit is not payload recovery. SynthID-O (partner variant) 136 bits at 512x512. TPR at 0.1% FPR 99.98% aggregated, 99.97% on the hardest spatial crop+resize. Trains against sampled semantics-preserving transforms, including weak VAE regeneration. Production decoder unpublished | Matches the architecture we treat as keyed `x' = x + g(x)`. Explains why a two-pixel crop kills `pipeline_lattice` but not the official oracle, and why a 22.3 dB foreign VAE still reads `detected` |
|
||||||
| DeepMind [US12094474B1](https://patents.google.com/patent/US12094474B1/en) and continuation [US20250149048A1](https://patents.google.com/patent/US20250149048A1/en) | Residual U-Net encoder, separate decoder, optional key, encoder/decoder ensembles that need not recognize each other | Constraint, not a recipe. Ensemble non-recognition is why one recovered Google tile cannot be reused as an OpenAI detector (S4) |
|
| DeepMind [US12094474B1](https://patents.google.com/patent/US12094474B1/en) and continuation [US20250149048A1](https://patents.google.com/patent/US20250149048A1/en) | Residual U-Net encoder, separate decoder, optional key, encoder/decoder ensembles that need not recognize each other | Constraint, not a recipe. Ensemble non-recognition is why one recovered Google tile cannot be reused as an OpenAI detector (S4) |
|
||||||
| Dathathri et al., [Nature 634:818-823 (2024)](https://www.nature.com/articles/s41586-024-08025-4) | SynthID-Text: tournament sampling of LLM tokens, open-source | Different system. Image/audio/video remain proprietary |
|
| Dathathri et al., [Nature 634:818-823 (2024)](https://www.nature.com/articles/s41586-024-08025-4) | SynthID-Text: tournament sampling of LLM tokens, open-source. Anthropic's [Claude text watermark](https://www.anthropic.com/news/claude-text-watermark) (announced 2026-08-14 for future Claude models, updated 2026-09-01) "is a version of the SynthID-Text approach", with its detection API in private preview | Different system. Image/audio/video remain proprietary |
|
||||||
| OpenAI, [advancing content provenance](https://openai.com/index/advancing-content-provenance/) (2026-05-19, audio 2026-07-31) and [content provenance API](https://developers.openai.com/api/docs/guides/content-provenance) | ChatGPT / API / Codex images carry C2PA plus SynthID. Audio from 2026-07-31. `POST /v1/content_provenance_checks`. `not_detected` does not rule out another vendor | This is the oracle. C2PA and SynthID are independent entries. Do not abuse the endpoint as an adaptive reverse-engineering loop |
|
| OpenAI, [advancing content provenance](https://openai.com/index/advancing-content-provenance/) (2026-05-19, audio 2026-07-31) and [content provenance API](https://developers.openai.com/api/docs/guides/content-provenance) | ChatGPT / API / Codex images carry C2PA plus SynthID. Audio from 2026-07-31. `POST /v1/content_provenance_checks`. `not_detected` does not rule out another vendor | This is the oracle. C2PA and SynthID are independent entries. Do not abuse the endpoint as an adaptive reverse-engineering loop |
|
||||||
|
|
||||||
### Keyless presence detectors in the literature
|
### Keyless presence detectors in the literature
|
||||||
|
|||||||
@@ -229,12 +229,20 @@ regeneration is already in [synthid.md](synthid.md#23-removal-attacks-and-forens
|
|||||||
| Goonatilake and Ateniese, [arXiv:2605.09203](https://arxiv.org/abs/2605.09203) | Six removers all leave a forensic residue a ResNet-50 sees at >98% TPR @ 1% FPR | Applies to UnMarker, Zhao's WatermarkAttacker, CtrlRegen+ | Defeating a provider oracle is not deniability. This is the product remainder |
|
| Goonatilake and Ateniese, [arXiv:2605.09203](https://arxiv.org/abs/2605.09203) | Six removers all leave a forensic residue a ResNet-50 sees at >98% TPR @ 1% FPR | Applies to UnMarker, Zhao's WatermarkAttacker, CtrlRegen+ | Defeating a provider oracle is not deniability. This is the product remainder |
|
||||||
| An et al., [arXiv:2401.08573](https://arxiv.org/abs/2401.08573) (WAVES, ICML 2024) | 26 attacks on StegaStamp, Stable Signature, Tree-Ring. Regeneration, not JPEG, is the attack that matters. StegaStamp TPR@1%FPR 1.00 to 0.01; Tree-Ring 0.99 to 0.12 | Open watermarks | Protocol. Our blur-sigma-7 and 16-32 scramble are closer to WAVES "distortion" than to regeneration |
|
| An et al., [arXiv:2401.08573](https://arxiv.org/abs/2401.08573) (WAVES, ICML 2024) | 26 attacks on StegaStamp, Stable Signature, Tree-Ring. Regeneration, not JPEG, is the attack that matters. StegaStamp TPR@1%FPR 1.00 to 0.01; Tree-Ring 0.99 to 0.12 | Open watermarks | Protocol. Our blur-sigma-7 and 16-32 scramble are closer to WAVES "distortion" than to regeneration |
|
||||||
| Bulychev et al., [arXiv:2605.16796](https://arxiv.org/abs/2605.16796) (Watermarks Attack Watermarks) | Apply a second watermark, usually with the victim's own post-processing encoder and a different message. On MS-COCO, its end-to-end pipeline lowers Video Seal victim bit accuracy from 0.998 to 0.520 (TPR@1%FPR 0.046) and Pixel Seal from 0.999 to 0.514 (TPR 0.029); the same-method attacker message recovers at 0.99 bit accuracy for both schemes. Quality is reported as an eight-metric normalized composite, not a raw PSNR operating point | Eight open image schemes, not production SynthID; Video Seal is tested on images | Directly motivated the [local open-encoder replication](#re-watermarking-on-open-encoders-2026-09-27). Its MIT [code](https://github.com/MariaBulychev/Watermarks-Attack-Watermarks/tree/0444f52ad972da9e009555f2dc1028d9abe0610e) calls external encoder packages and weights; the top-level license does not license those dependencies |
|
| Bulychev et al., [arXiv:2605.16796](https://arxiv.org/abs/2605.16796) (Watermarks Attack Watermarks) | Apply a second watermark, usually with the victim's own post-processing encoder and a different message. On MS-COCO, its end-to-end pipeline lowers Video Seal victim bit accuracy from 0.998 to 0.520 (TPR@1%FPR 0.046) and Pixel Seal from 0.999 to 0.514 (TPR 0.029); the same-method attacker message recovers at 0.99 bit accuracy for both schemes. Quality is reported as an eight-metric normalized composite, not a raw PSNR operating point | Eight open image schemes, not production SynthID; Video Seal is tested on images | Directly motivated the [local open-encoder replication](#re-watermarking-on-open-encoders-2026-09-27). Its MIT [code](https://github.com/MariaBulychev/Watermarks-Attack-Watermarks/tree/0444f52ad972da9e009555f2dc1028d9abe0610e) calls external encoder packages and weights; the top-level license does not license those dependencies |
|
||||||
|
| Yang et al., [arXiv:2609.01249](https://arxiv.org/abs/2609.01249) (One Prompt Is Enough) | One reconstruction prompt to a closed image editor (GPT Image 1, 1.5 and 2; Nano Banana, Pro and 2) launders the payload over 1,800 outputs; OpenAI models disrupt it most | DwtDct, DwtDctSvd and RivaGAN only. "SynthID was not tested" | Regeneration through a provider's own model. It breaks the scheme our DWT-DCT detector reads, and says nothing about production SynthID, which the same editors add to their output |
|
||||||
|
| Yao et al., [arXiv:2610.11290](https://arxiv.org/abs/2610.11290) (ProxyEraseAgent) | Ranks a pool of open decoders by calibrated response and attacks the strongest as proxy boundary estimators; 94.8% success over 11 systems against 79.5% for the best fixed Regen-VAE | Open schemes, TrustMark and VideoSeal among them. Not SynthID | Needs a decoder that responds to the mark. No public decoder responds to production SynthID, so it does not transfer to an oracle-only target |
|
||||||
|
| Bao et al., [arXiv:2609.08213](https://arxiv.org/abs/2609.08213) (DRIFT) | Deflects the diffusion trajectory. The adaptive variant searches a strength ladder per image for "the first verifier-rejected rung" from binary verifier feedback; 98-100% success on nine marks | In-generation diffusion marks (Tree-Ring family, Gaussian Shading and others). Not SynthID | The ladder stop is our [attack ladders](#attack-ladders-openai-oracle) run per image. Against a provider verifier that is the adaptive query loop its terms forbid, so it applies to local oracles only |
|
||||||
|
| Ye et al., [arXiv:2610.03166](https://arxiv.org/abs/2610.03166) (LiBRA) | White-box latent optimization with the key and decoder, steering decoding confidence toward chance from both sides so the mark is not left inverted but detectable; Stable Signature detection 1.00 to 0.03 | Released checkpoints of Stable Signature, the YU1 GAN fingerprint, HiDDeN and MBRS | Needs the decoder, so it is out of reach here. The inverted-mark point applies to any bit-flip attack |
|
||||||
|
| Aistov et al., [arXiv:2610.02010](https://arxiv.org/abs/2610.02010) (Latent Frequency Masking, ICDM 2026) | Replaces selected Fourier coefficients of the latent; [code](https://github.com/alvitrioliks/lfm) | Stable Signature, Tree-Ring, RingID, Gaussian Shading (resists), Maxsive, METR. Not SynthID | In-generation latent marks. Not the post-hoc encoder family Gowal describes |
|
||||||
|
| Li et al., [arXiv:2609.31693](https://arxiv.org/abs/2609.31693) (Disentangle and Drop) | Splits the image into a grayscale carrier and a residual branch, then suppresses the residual, with one shared operating setting | SSL, DwtDct, DwtDctSvd, RivaGAN, StegaStamp, Stable Signature, Tree-Ring | Open schemes only |
|
||||||
|
| Yang et al., [arXiv:2609.31722](https://arxiv.org/abs/2609.31722) (Push-Pull) | Key-free and decoder-free at deployment; average BER 0.3958 at 31.07 dB PSNR on a four-method sweep | Seven open methods. Not SynthID | A BER near 0.40 leaves much of the payload intact, as with SPFM-Net below |
|
||||||
|
| Ding et al., [arXiv:2609.34003](https://arxiv.org/abs/2609.34003) (DeMark, audio) | Query-free black box that suppresses the time-frequency artifacts the embedding leaves; average attack success 0.92 and 0.96 on two speech sets | AudioSeal, WavMark, Timbre, AudioMarkNet. Not audio SynthID | A candidate cohort for the pinned AudioSeal oracle in the [benchmark kernel](benchmark-kernel.md). No code located |
|
||||||
| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring) | In-generation Fourier pattern in initial noise | Not SynthID | Kill is DDIM inversion plus latent wipe, not a pixel scramble. Lin and Juarez [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) remove it from public knowledge |
|
| Wen et al., [arXiv:2305.20030](https://arxiv.org/abs/2305.20030) (Tree-Ring) | In-generation Fourier pattern in initial noise | Not SynthID | Kill is DDIM inversion plus latent wipe, not a pixel scramble. Lin and Juarez [arXiv:2506.10502](https://arxiv.org/abs/2506.10502) (USENIX 2025) remove it from public knowledge |
|
||||||
| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tuned VAE decoder | Not SynthID | Regeneration that replaces the decoder is in-family. Our foreign VAE at 22.3 dB did not kill OpenAI SynthID, which is the expected mismatch |
|
| Fernandez et al., Stable Signature, ICCV 2023 | Fine-tuned VAE decoder | Not SynthID | Regeneration that replaces the decoder is in-family. Our foreign VAE at 22.3 dB did not kill OpenAI SynthID, which is the expected mismatch |
|
||||||
| Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | reverse-SynthID does not systematically remove SynthID | Google spokesperson on the spectral bypass | Agrees with our V4 audit and with Bypass: reconstruction or decoder confusion, not a payload delete |
|
| Google to [The Verge, 2026-04-14](https://www.theverge.com/ai-artificial-intelligence/911579/google-synthid-ai-watermarking-system-reverse-engineered) | reverse-SynthID does not systematically remove SynthID | Google spokesperson on the spectral bypass | Agrees with our V4 audit and with Bypass: reconstruction or decoder confusion, not a payload delete |
|
||||||
|
|
||||||
Quiet pixel-only removal is not a published result on production
|
Quiet pixel-only removal is not a published result on production
|
||||||
OpenAI or Google SynthID (still true in the 2026-09-27 literature sweep).
|
OpenAI or Google SynthID (still true in the 2026-10-17 literature sweep).
|
||||||
Against open encoders the fidelity band has moved: WmForger
|
Against open encoders the fidelity band has moved: WmForger
|
||||||
([arXiv:2510.20468](https://arxiv.org/abs/2510.20468)) and Hide&Seek
|
([arXiv:2510.20468](https://arxiv.org/abs/2510.20468)) and Hide&Seek
|
||||||
([arXiv:2603.01067](https://arxiv.org/abs/2603.01067)) report roughly 30 dB,
|
([arXiv:2603.01067](https://arxiv.org/abs/2603.01067)) report roughly 30 dB,
|
||||||
|
|||||||
@@ -575,7 +575,13 @@ The SynthID decoder is proprietary and not released:
|
|||||||
> -- Gowal et al., arXiv:2510.09263
|
> -- Gowal et al., arXiv:2510.09263
|
||||||
|
|
||||||
There are no released payload-decoder weights or public algorithm. Google
|
There are no released payload-decoder weights or public algorithm. Google
|
||||||
provides verification in Gemini. The old limited SynthID Detector portal does
|
provides verification in Gemini and, since the public SynthID Detector opened,
|
||||||
|
at [synthid.com](https://synthid.com/). That site is the `synthid-web` oracle
|
||||||
|
surface in [`provider-oracles.md`](provider-oracles.md). On 2026-10-17 its FAQ
|
||||||
|
named Google DeepMind, Nvidia, OpenAI, and Kakao, and its settled labels were
|
||||||
|
`Watermark detected` and `Watermark not detected`. On 2026-10-19 UTC the
|
||||||
|
signed-in page also used the status line `SynthID was not detected`, which this
|
||||||
|
tooling records as `not_detected`. The old limited portal does
|
||||||
not work: on 2026-09-07, its official `labs.google/synthid` entry point
|
not work: on 2026-09-07, its official `labs.google/synthid` entry point
|
||||||
redirected a signed-in browser to a Google 404. On 2026-09-23 the same entry
|
redirected a signed-in browser to a Google 404. On 2026-09-23 the same entry
|
||||||
point redirected to `deepmind.google.com/frontiers/synthid`, which also returned
|
point redirected to `deepmind.google.com/frontiers/synthid`, which also returned
|
||||||
|
|||||||
@@ -92,8 +92,8 @@ the row count is host-dependent rather than the fixed 68 recorded here before. E
|
|||||||
(~500 images spanning format x provenance x aspect ratio), asserting exit-code semantics
|
(~500 images spanning format x provenance x aspect ratio), asserting exit-code semantics
|
||||||
rather than just absence of crash.
|
rather than just absence of crash.
|
||||||
|
|
||||||
Known trap to encode: **exit 2 is triply overloaded** (no-visible-mark, no-invisible-signal,
|
Known trap to encode: **exit 2 is overloaded four ways** (no-visible-mark, no-invisible-signal,
|
||||||
Click usage error). A wrapper cannot distinguish them without parsing stderr. Either the
|
empty batch directory, Click usage error). A wrapper cannot distinguish them without parsing stderr. Either the
|
||||||
sweep asserts on stderr, or the codes get split -- the latter is the better fix.
|
sweep asserts on stderr, or the codes get split -- the latter is the better fix.
|
||||||
|
|
||||||
#### Coverage method
|
#### Coverage method
|
||||||
@@ -262,8 +262,8 @@ and implemented by `scripts/provider_oracles.py`.
|
|||||||
|
|
||||||
Proprietary watermark removal cannot be verified locally by design -- no public decoder
|
Proprietary watermark removal cannot be verified locally by design -- no public decoder
|
||||||
exists. Each vendor has its own oracle and it covers only that vendor's content:
|
exists. Each vendor has its own oracle and it covers only that vendor's content:
|
||||||
the Content Provenance API first and `openai.com/verify` as an explicit fallback for
|
the Content Provenance API first and `openai.com/verify` for
|
||||||
OpenAI, the Gemini app for Google, and Microsoft's
|
OpenAI, the Gemini app and the SynthID Detector at `synthid.com` for Google, and Microsoft's
|
||||||
[Content Provenance Detection API](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/how-to/how-to-provenance-detection)
|
[Content Provenance Detection API](https://learn.microsoft.com/en-us/azure/ai-services/content-safety/how-to/how-to-provenance-detection)
|
||||||
for InvisMark. The Microsoft API reports `Watermark` and `C2PA` separately. It therefore
|
for InvisMark. The Microsoft API reports `Watermark` and `C2PA` separately. It therefore
|
||||||
needs a pixel-identical metadata-stripped control: the control must lose `C2PA` while
|
needs a pixel-identical metadata-stripped control: the control must lose `C2PA` while
|
||||||
@@ -280,6 +280,12 @@ protocol is in [`synthid-detector-removal-plan.md`](synthid-detector-removal-pla
|
|||||||
Scope honestly: this tier certifies strength floors on a handful of images per vendor, and
|
Scope honestly: this tier certifies strength floors on a handful of images per vendor, and
|
||||||
that is all it can do. See `docs/synthid.md`.
|
that is all it can do. See `docs/synthid.md`.
|
||||||
|
|
||||||
|
The project therefore publishes no aggregate pass rate per model or resolution: a large
|
||||||
|
enough sample needs bulk automated checks, which the OpenAI restriction above and the
|
||||||
|
SynthID Detector's [additional terms](https://synthid.com/terms) both forbid. Public
|
||||||
|
removal claims rest on this method and on individual hash-bound rows such as
|
||||||
|
`data/evaluations/video-synthid-oracle.csv`, never on an extrapolated rate.
|
||||||
|
|
||||||
### D1. Sampling frame
|
### D1. Sampling frame
|
||||||
|
|
||||||
Select a representative local sample without committing images, identifiers, dataset sizes, or oracle results. Stratify by signal family and preserve the sampling method outside the public repository.
|
Select a representative local sample without committing images, identifiers, dataset sizes, or oracle results. Stratify by signal family and preserve the sampling method outside the public repository.
|
||||||
|
|||||||
@@ -37,6 +37,12 @@ claim generator still falls back to the certificate issuer.
|
|||||||
Exact claim-generator mappings also cover `Grok Imagine` as xAI Grok Imagine
|
Exact claim-generator mappings also cover `Grok Imagine` as xAI Grok Imagine
|
||||||
and `Suno` as Suno. Both can carry a valid AI source claim without a separately
|
and `Suno` as Suno. Both can carry a valid AI source claim without a separately
|
||||||
readable issuer, so issuer-only attribution leaves the platform blank.
|
readable issuer, so issuer-only attribution leaves the platform blank.
|
||||||
|
Suno also [announced on 2026-08-06](https://suno.com/blog/building-the-future-of-music-responsibly)
|
||||||
|
that it would adopt "new audio watermarking and fingerprinting technology" in
|
||||||
|
the coming weeks. It did not name the technology
|
||||||
|
([TechCrunch](https://techcrunch.com/2026/08/06/amid-legal-battles-suno-says-it-will-start-watermarking-songs/)
|
||||||
|
asked whether it would be SynthID and got no answer), so there is no local
|
||||||
|
detector, and metadata stripping leaves any such mark in the audio.
|
||||||
|
|
||||||
Some signers expose the product in `signature_info` while retaining a generic
|
Some signers expose the product in `signature_info` while retaining a generic
|
||||||
claim generator. xAI Grok Imagine is recognized from that signed identity even
|
claim generator. xAI Grok Imagine is recognized from that signed identity even
|
||||||
@@ -128,6 +134,14 @@ against primary sources before adding jurisdiction-specific claims.
|
|||||||
|
|
||||||
**Google Gemini visible "sparkle" -- tier-dependent, and spec-undocumented by Google.** Google primary sources (the Nano Banana Pro blog and the gemini.google image-generation page, both WebFetch-verified) confirm Gemini images carry BOTH the invisible SynthID (on ALL Google-AI media) AND a visible sparkle, but the visible mark is **tier-gated**: applied for FREE and Google AI **Pro** users, and **REMOVED** for Google AI **Ultra** subscribers, inside **Google AI Studio**, and on **API / dev** output. Update 2026-09-23: since 2026-08-14 a Gemini "Media watermark" setting lets any consumer tier switch the visible mark off for images, Omni video and Lyria music, except work accounts and India, South Korea and Vietnam without Ultra (Gemini help 17405358; the setting was observed in the Gemini web app on 2026-09-23). The tier-gating above is therefore no longer the only reason a sparkle is missing. So a Google-C2PA image with NO visible sparkle is expected (setting off, Ultra, API), not evidence it is clean -- this reinforces the `identify` "no visible mark != clean" rule. The ONLY official verifier is the SynthID flow: upload to the Gemini app and ask directly whether the file contains a SynthID watermark, requiring the SynthID verification tool and prohibiting inference from visible content, logos, text, or metadata. A generic question about AI generation is not an oracle result. The tool reads the INVISIBLE mark; there is **no official visible-sparkle detector**, and Google publishes **no** glyph geometry / size / opacity / color / locale / placement spec. So our capture-based sparkle template is the only source of truth and cannot be validated against a vendor spec -- keep reverse-engineering from real captures (do not expect a published spec).
|
**Google Gemini visible "sparkle" -- tier-dependent, and spec-undocumented by Google.** Google primary sources (the Nano Banana Pro blog and the gemini.google image-generation page, both WebFetch-verified) confirm Gemini images carry BOTH the invisible SynthID (on ALL Google-AI media) AND a visible sparkle, but the visible mark is **tier-gated**: applied for FREE and Google AI **Pro** users, and **REMOVED** for Google AI **Ultra** subscribers, inside **Google AI Studio**, and on **API / dev** output. Update 2026-09-23: since 2026-08-14 a Gemini "Media watermark" setting lets any consumer tier switch the visible mark off for images, Omni video and Lyria music, except work accounts and India, South Korea and Vietnam without Ultra (Gemini help 17405358; the setting was observed in the Gemini web app on 2026-09-23). The tier-gating above is therefore no longer the only reason a sparkle is missing. So a Google-C2PA image with NO visible sparkle is expected (setting off, Ultra, API), not evidence it is clean -- this reinforces the `identify` "no visible mark != clean" rule. The ONLY official verifier is the SynthID flow: upload to the Gemini app and ask directly whether the file contains a SynthID watermark, requiring the SynthID verification tool and prohibiting inference from visible content, logos, text, or metadata. A generic question about AI generation is not an oracle result. The tool reads the INVISIBLE mark; there is **no official visible-sparkle detector**, and Google publishes **no** glyph geometry / size / opacity / color / locale / placement spec. So our capture-based sparkle template is the only source of truth and cannot be validated against a vendor spec -- keep reverse-engineering from real captures (do not expect a published spec).
|
||||||
|
|
||||||
|
**Gemini Notebook, formerly NotebookLM, adds its own visible watermark.** Google's
|
||||||
|
[plan help](https://support.google.com/gemininotebook/answer/16213268) says the mark
|
||||||
|
is added to generated images and videos "to further help you identify AI-generated
|
||||||
|
content", and AI Pro and Ultra subscribers can turn it off. That makes it an
|
||||||
|
AI-generation label under the [project scope](legal-and-safety.md#out-of-scope). No
|
||||||
|
template is planned (decided 2026-10-18), so the mark stays undetected; slide
|
||||||
|
decks export as PDF or PPTX, which the project does not process.
|
||||||
|
|
||||||
**Google Veo video marks use two incompatible visible designs.** Public raw
|
**Google Veo video marks use two incompatible visible designs.** Public raw
|
||||||
clips verify the current four-point diamond and the legacy bottom-right `Veo`
|
clips verify the current four-point diamond and the legacy bottom-right `Veo`
|
||||||
text. The independent
|
text. The independent
|
||||||
|
|||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://glama.ai/mcp/schemas/server.json",
|
||||||
|
"maintainers": ["wiltodelta"]
|
||||||
|
}
|
||||||
@@ -4,7 +4,7 @@
|
|||||||
# on it, including the ComfyUI node package. The console script below carries
|
# on it, including the ComfyUI node package. The console script below carries
|
||||||
# the same weight, since users have it on PATH.
|
# the same weight, since users have it on PATH.
|
||||||
name = "remove-ai-watermarks"
|
name = "remove-ai-watermarks"
|
||||||
version = "0.44.0"
|
version = "0.47.0"
|
||||||
description = "AI watermark remover for visible, invisible, and provenance marks in images and video"
|
description = "AI watermark remover for visible, invisible, and provenance marks in images and video"
|
||||||
readme = "README.md"
|
readme = "README.md"
|
||||||
requires-python = ">=3.11,<3.15"
|
requires-python = ">=3.11,<3.15"
|
||||||
@@ -205,6 +205,9 @@ dev = [
|
|||||||
"httpx>=0.28,<1",
|
"httpx>=0.28,<1",
|
||||||
# Development-only isolated browser automation for public provider oracles.
|
# Development-only isolated browser automation for public provider oracles.
|
||||||
"playwright>=1.62,<2",
|
"playwright>=1.62,<2",
|
||||||
|
# The official MCP client, used only to test `remove-ai-watermarks mcp`
|
||||||
|
# against the reference implementation. The server itself is stdlib-only.
|
||||||
|
"mcp>=2.3,<3",
|
||||||
"pytest>=8.0.0",
|
"pytest>=8.0.0",
|
||||||
"pytest-cov>=4.1.0",
|
"pytest-cov>=4.1.0",
|
||||||
"pytest-xdist>=3.5.0",
|
"pytest-xdist>=3.5.0",
|
||||||
|
|||||||
@@ -2,7 +2,7 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import datetime
|
from datetime import date
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
@@ -10,15 +10,15 @@ if TYPE_CHECKING:
|
|||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
def corpus_day(path: Path, corpus: Path) -> datetime.date | None:
|
def corpus_day(path: Path, corpus: Path) -> date | None:
|
||||||
"""Read the leading YYYY-MM-DD corpus segment, if present."""
|
"""Read the leading YYYY-MM-DD corpus segment, if present."""
|
||||||
try:
|
try:
|
||||||
return datetime.date.fromisoformat(path.relative_to(corpus).parts[0])
|
return date.fromisoformat(path.relative_to(corpus).parts[0])
|
||||||
except (ValueError, IndexError):
|
except (ValueError, IndexError):
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
def corpus_files(corpus: Path, since: datetime.date | None, *, suffixes: Collection[str] | None = None) -> list[Path]:
|
def corpus_files(corpus: Path, since: date | None, *, suffixes: Collection[str] | None = None) -> list[Path]:
|
||||||
"""Return a deterministic recursive walk, optionally date- and suffix-bounded."""
|
"""Return a deterministic recursive walk, optionally date- and suffix-bounded."""
|
||||||
if since is None:
|
if since is None:
|
||||||
candidates = corpus.rglob("*")
|
candidates = corpus.rglob("*")
|
||||||
|
|||||||
@@ -22,8 +22,7 @@ from the provider's canonical name, the provider is named separately.
|
|||||||
| --- | --- | --- | --- |
|
| --- | --- | --- | --- |
|
||||||
| `samsung_en_alpha.png` | Samsung Galaxy AI label, English locale ("AI-generated content", bottom-left) | 5 corpus files | POS 0.11-0.30 vs NEG max 0.40 (binary); tophat/gray tried, no separation. The registered Italian engine scores 0.18-0.31 on the same files -- same layout class, wrong glyph template. |
|
| `samsung_en_alpha.png` | Samsung Galaxy AI label, English locale ("AI-generated content", bottom-left) | 5 corpus files | POS 0.11-0.30 vs NEG max 0.40 (binary); tophat/gray tried, no separation. The registered Italian engine scores 0.18-0.31 on the same files -- same layout class, wrong glyph template. |
|
||||||
| `gemini_text_alpha.png` | "Generated with Gemini" text label (bottom-right; the registered gemini mark is the sparkle icon) | 3 corpus files | POS 0.07-0.22 vs NEG max 0.32; coverage gate finds the blob (0.29-0.35) but the Arial silhouette misses Google's letterforms. |
|
| `gemini_text_alpha.png` | "Generated with Gemini" text label (bottom-right; the registered gemini mark is the sparkle icon) | 3 corpus files | POS 0.07-0.22 vs NEG max 0.32; coverage gate finds the blob (0.29-0.35) but the Arial silhouette misses Google's letterforms. |
|
||||||
| `notebooklm_alpha.png` | NotebookLM wordmark (bottom-right) | 12 corpus files | locate geometry not yet fitted; POS max 0.12. |
|
| `notebooklm_alpha.png` | NotebookLM wordmark (bottom-right; the product is now Gemini Notebook) | 12 corpus files | parked: a template is not planned (decided 2026-10-18). Last measured POS max 0.12, locate geometry never fitted. |
|
||||||
| `dola_alpha.png` | DolaAI on images (the video mark is registered) | 12 corpus files | POS 0.11-0.21 vs NEG max 0.30. |
|
|
||||||
| `mindvideo_alpha.png` | MindVideo.AI (top-right) | 11 corpus files | POS 0.29-0.32 vs NEG max 0.30 -- borderline overlap, not shippable. |
|
| `mindvideo_alpha.png` | MindVideo.AI (top-right) | 11 corpus files | POS 0.29-0.32 vs NEG max 0.30 -- borderline overlap, not shippable. |
|
||||||
| `higgsfield_alpha.png` | `HIGGSFIELD AI` wordmark (Higgsfield AI, bottom-right; the boxed `AI` variant shares the cohort) | 5 wordmark files (16 in the boxed-AI OCR cluster) | POS max 0.26 vs NEG max 0.22 -- no separation; the mark may be two-part (wordmark + boxed AI) and needs a composed template. |
|
| `higgsfield_alpha.png` | `HIGGSFIELD AI` wordmark (Higgsfield AI, bottom-right; the boxed `AI` variant shares the cohort) | 5 wordmark files (16 in the boxed-AI OCR cluster) | POS max 0.26 vs NEG max 0.22 -- no separation; the mark may be two-part (wordmark + boxed AI) and needs a composed template. |
|
||||||
| `jianying_alpha.png` | `剪映AI` wordmark (Jianying / 剪映, the China product in the CapCut family; bottom-right) | 2 corpus files | POS 0.29 vs NEG max 0.35. |
|
| `jianying_alpha.png` | `剪映AI` wordmark (Jianying / 剪映, the China product in the CapCut family; bottom-right) | 2 corpus files | POS 0.29 vs NEG max 0.35. |
|
||||||
|
|||||||
|
Before Width: | Height: | Size: 3.4 KiB |
@@ -121,7 +121,7 @@ def load_pinned_models() -> tuple[object, object]:
|
|||||||
raw = convert_state_dict_for_scriptable_model(raw)
|
raw = convert_state_dict_for_scriptable_model(raw)
|
||||||
model = create(config, device="cpu") # type: ignore[call-arg]
|
model = create(config, device="cpu") # type: ignore[call-arg]
|
||||||
load_state_dict(model, state_dict=raw)
|
load_state_dict(model, state_dict=raw)
|
||||||
model.eval()
|
model.train(False)
|
||||||
return model
|
return model
|
||||||
|
|
||||||
generator = load(GENERATOR_CARD, "generator_base.pth", AudioSealWMConfig, create_generator)
|
generator = load(GENERATOR_CARD, "generator_base.pth", AudioSealWMConfig, create_generator)
|
||||||
|
|||||||
@@ -51,7 +51,7 @@ def export_onnx(checkpoint: Path, output: Path) -> None:
|
|||||||
if output.exists():
|
if output.exists():
|
||||||
raise SystemExit(f"refusing to overwrite existing output: {output}")
|
raise SystemExit(f"refusing to overwrite existing output: {output}")
|
||||||
output.parent.mkdir(parents=True, exist_ok=True)
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
model = VisionEmbedding(load_headed_clip(checkpoint, torch.device("cpu"))).eval()
|
model = VisionEmbedding(load_headed_clip(checkpoint, torch.device("cpu"))).train(False)
|
||||||
export_options: dict[str, Any] = {}
|
export_options: dict[str, Any] = {}
|
||||||
if "dynamo" in inspect.signature(torch.onnx.export).parameters:
|
if "dynamo" in inspect.signature(torch.onnx.export).parameters:
|
||||||
export_options["dynamo"] = False
|
export_options["dynamo"] = False
|
||||||
|
|||||||
@@ -252,7 +252,7 @@ def _lpips_model() -> tuple[Any, Any]:
|
|||||||
import torch
|
import torch
|
||||||
|
|
||||||
model = lpips.LPIPS(net="alex", verbose=False)
|
model = lpips.LPIPS(net="alex", verbose=False)
|
||||||
model.eval()
|
model.train(False)
|
||||||
return model, torch
|
return model, torch
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -173,7 +173,7 @@ def load_model() -> object:
|
|||||||
raise RuntimeError(
|
raise RuntimeError(
|
||||||
f"PixelSeal checkpoint/source mismatch: missing={loaded.missing_keys}, unexpected={loaded.unexpected_keys}"
|
f"PixelSeal checkpoint/source mismatch: missing={loaded.missing_keys}, unexpected={loaded.unexpected_keys}"
|
||||||
)
|
)
|
||||||
model.eval()
|
model.train(False)
|
||||||
return model
|
return model
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -5,7 +5,9 @@ from __future__ import annotations
|
|||||||
|
|
||||||
import logging
|
import logging
|
||||||
import os
|
import os
|
||||||
|
import random
|
||||||
import re
|
import re
|
||||||
|
import secrets
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from datetime import UTC, datetime
|
from datetime import UTC, datetime
|
||||||
from time import monotonic
|
from time import monotonic
|
||||||
@@ -22,7 +24,35 @@ if TYPE_CHECKING:
|
|||||||
|
|
||||||
log = logging.getLogger(__name__)
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
PLAYWRIGHT_SURFACES = frozenset({"openai-web", "microsoft-web", "meta-web"})
|
PLAYWRIGHT_SURFACES = frozenset({"openai-web", "microsoft-web", "meta-web", "synthid-web"})
|
||||||
|
# A proxy exit is chosen when the run starts. These are the countries that
|
||||||
|
# choice may use: the United States and European ISO codes.
|
||||||
|
EUROPE_OR_US_COUNTRIES = (
|
||||||
|
"AT",
|
||||||
|
"BE",
|
||||||
|
"CH",
|
||||||
|
"CZ",
|
||||||
|
"DE",
|
||||||
|
"DK",
|
||||||
|
"ES",
|
||||||
|
"FI",
|
||||||
|
"FR",
|
||||||
|
"GB",
|
||||||
|
"GR",
|
||||||
|
"HU",
|
||||||
|
"IE",
|
||||||
|
"IT",
|
||||||
|
"NL",
|
||||||
|
"NO",
|
||||||
|
"PL",
|
||||||
|
"PT",
|
||||||
|
"RO",
|
||||||
|
"SE",
|
||||||
|
"US",
|
||||||
|
)
|
||||||
|
_THORDATA_COUNTRY = re.compile(r"-country-[A-Za-z]{2}", re.IGNORECASE)
|
||||||
|
_THORDATA_SESSION = re.compile(r"-sessid-[A-Za-z0-9]+", re.IGNORECASE)
|
||||||
|
_THORDATA_SESSION_TIME = re.compile(r"-sesstime-\d+", re.IGNORECASE)
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
@dataclass(frozen=True)
|
||||||
@@ -51,6 +81,11 @@ _RESULT_MARKERS = {
|
|||||||
"No AI signatures from Meta were found",
|
"No AI signatures from Meta were found",
|
||||||
"No AI signatures were found",
|
"No AI signatures were found",
|
||||||
),
|
),
|
||||||
|
"synthid-web": (
|
||||||
|
"Watermark detected",
|
||||||
|
"Watermark not detected",
|
||||||
|
"SynthID was not detected",
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
_REFUSAL_MARKERS = (
|
_REFUSAL_MARKERS = (
|
||||||
@@ -63,7 +98,10 @@ _REFUSAL_MARKERS = (
|
|||||||
"access denied",
|
"access denied",
|
||||||
)
|
)
|
||||||
|
|
||||||
_INDETERMINATE_MARKERS = ("Something went wrong. Please try again. If the issue persists, contact support.",)
|
_INDETERMINATE_MARKERS = (
|
||||||
|
"Something went wrong. Please try again. If the issue persists, contact support.",
|
||||||
|
"Please sign in before detection.",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _contains_any(text: str, markers: tuple[str, ...]) -> bool:
|
def _contains_any(text: str, markers: tuple[str, ...]) -> bool:
|
||||||
@@ -111,6 +149,16 @@ def parse_provider_result(surface: str, page_text: str) -> WebVerdict:
|
|||||||
return WebVerdict("detected", provenance, response)
|
return WebVerdict("detected", provenance, response)
|
||||||
if "inconclusive" in lowered or "no provenance detected" in lowered:
|
if "inconclusive" in lowered or "no provenance detected" in lowered:
|
||||||
return WebVerdict("not_detected", provenance, response)
|
return WebVerdict("not_detected", provenance, response)
|
||||||
|
elif surface == "synthid-web":
|
||||||
|
# The exact labels are the verdict. A page that shows both, or that
|
||||||
|
# asks for a sign-in, has no single watermark result. The sign-in
|
||||||
|
# sentence is kept; the skill then continues in the user's existing Chrome.
|
||||||
|
detected = "Watermark detected" in response
|
||||||
|
not_detected = "Watermark not detected" in response or "SynthID was not detected" in response
|
||||||
|
if not_detected and not detected:
|
||||||
|
return WebVerdict("not_detected", "unavailable", response)
|
||||||
|
if detected and not not_detected:
|
||||||
|
return WebVerdict("detected", "unavailable", response)
|
||||||
else:
|
else:
|
||||||
if "no ai signatures from meta were found" in lowered or "no ai signatures were found" in lowered:
|
if "no ai signatures from meta were found" in lowered or "no ai signatures were found" in lowered:
|
||||||
return WebVerdict("not_detected", "unavailable", response)
|
return WebVerdict("not_detected", "unavailable", response)
|
||||||
@@ -123,13 +171,37 @@ def parse_provider_result(surface: str, page_text: str) -> WebVerdict:
|
|||||||
return WebVerdict("indeterminate", provenance, response or "Provider page returned no readable result")
|
return WebVerdict("indeterminate", provenance, response or "Provider page returned no readable result")
|
||||||
|
|
||||||
|
|
||||||
|
def choose_proxy_country(rng: random.Random | None = None) -> str:
|
||||||
|
"""Pick one exit country from Europe or the United States."""
|
||||||
|
picker = random.SystemRandom() if rng is None else rng
|
||||||
|
return picker.choice(EUROPE_OR_US_COUNTRIES)
|
||||||
|
|
||||||
|
|
||||||
|
def new_proxy_session() -> str:
|
||||||
|
"""Return a new ThorData session id. A later call does not reuse it."""
|
||||||
|
return "s" + secrets.token_hex(8)
|
||||||
|
|
||||||
|
|
||||||
|
def fresh_thordata_username(username: str, *, country: str, session_id: str) -> str:
|
||||||
|
"""Drop a prepared country and session, then attach the ones for this run."""
|
||||||
|
if country not in EUROPE_OR_US_COUNTRIES:
|
||||||
|
raise ValueError("proxy country must be in Europe or the US")
|
||||||
|
if re.fullmatch(r"[A-Za-z0-9]{1,32}", session_id) is None:
|
||||||
|
raise ValueError("proxy session id must be 1-32 ASCII letters or digits")
|
||||||
|
base = _THORDATA_SESSION_TIME.sub("", username)
|
||||||
|
base = _THORDATA_SESSION.sub("", base)
|
||||||
|
base = _THORDATA_COUNTRY.sub("", base).strip("-")
|
||||||
|
return f"{base}-country-{country}-sessid-{session_id}-sesstime-90"
|
||||||
|
|
||||||
|
|
||||||
def proxy_settings(
|
def proxy_settings(
|
||||||
slot: oracles.ExecutionSlot,
|
slot: oracles.ExecutionSlot,
|
||||||
environ: Mapping[str, str] = os.environ,
|
environ: Mapping[str, str] = os.environ,
|
||||||
*,
|
*,
|
||||||
sticky_session_id: str | None = None,
|
country: str | None = None,
|
||||||
|
session_id: str | None = None,
|
||||||
) -> ProxySettings | None:
|
) -> ProxySettings | None:
|
||||||
"""Resolve a secret proxy URL and pin rotating ThorData routes per batch."""
|
"""Resolve a secret proxy URL. A ThorData exit is chosen for this call."""
|
||||||
variable = slot.get("proxy_url_env")
|
variable = slot.get("proxy_url_env")
|
||||||
if variable is None:
|
if variable is None:
|
||||||
return None
|
return None
|
||||||
@@ -147,10 +219,11 @@ def proxy_settings(
|
|||||||
if parsed.username is not None:
|
if parsed.username is not None:
|
||||||
username = unquote(parsed.username)
|
username = unquote(parsed.username)
|
||||||
is_thordata = parsed.hostname.endswith((".thordata.net", ".thordata.online"))
|
is_thordata = parsed.hostname.endswith((".thordata.net", ".thordata.online"))
|
||||||
if is_thordata and sticky_session_id is not None and "-sessid-" not in username.casefold():
|
if is_thordata:
|
||||||
if re.fullmatch(r"[A-Za-z0-9]{1,32}", sticky_session_id) is None:
|
chosen_country = choose_proxy_country() if country is None else country
|
||||||
raise ValueError("sticky_session_id must be 1-32 ASCII letters or digits")
|
chosen_session = new_proxy_session() if session_id is None else session_id
|
||||||
username = f"{username}-sessid-{sticky_session_id}-sesstime-90"
|
username = fresh_thordata_username(username, country=chosen_country, session_id=chosen_session)
|
||||||
|
log.info("ThorData proxy exit country=%s session=%s", chosen_country, chosen_session)
|
||||||
settings["username"] = username
|
settings["username"] = username
|
||||||
if parsed.password is not None:
|
if parsed.password is not None:
|
||||||
settings["password"] = unquote(parsed.password)
|
settings["password"] = unquote(parsed.password)
|
||||||
@@ -198,7 +271,7 @@ def _set_upload_file(page: Page, surface: str, upload: Path, timeout_seconds: fl
|
|||||||
# OpenAI keeps background requests alive, so network-idle is
|
# OpenAI keeps background requests alive, so network-idle is
|
||||||
# unreachable. Its visible input becomes handler-ready at page load.
|
# unreachable. Its visible input becomes handler-ready at page load.
|
||||||
page.wait_for_load_state("load", timeout=timeout_ms)
|
page.wait_for_load_state("load", timeout=timeout_ms)
|
||||||
elif surface in {"meta-web", "microsoft-web"}:
|
elif surface in {"meta-web", "microsoft-web", "synthid-web"}:
|
||||||
# These pages mount hidden inputs before their selection handlers are
|
# These pages mount hidden inputs before their selection handlers are
|
||||||
# active. Network-idle is the observable handler-readiness seam.
|
# active. Network-idle is the observable handler-readiness seam.
|
||||||
page.wait_for_load_state("networkidle", timeout=timeout_ms)
|
page.wait_for_load_state("networkidle", timeout=timeout_ms)
|
||||||
@@ -288,19 +361,15 @@ def run_web_batch(
|
|||||||
oracles.verify_batch(manifest_path)
|
oracles.verify_batch(manifest_path)
|
||||||
manifest, results = oracles.load_batch(manifest_path)
|
manifest, results = oracles.load_batch(manifest_path)
|
||||||
surface = manifest["surface"]
|
surface = manifest["surface"]
|
||||||
if surface == "gemini-web":
|
if oracles.SURFACES[surface].automation == "real_browser":
|
||||||
raise ValueError("gemini-web requires the real Chrome workflow, not isolated Playwright")
|
raise ValueError(f"{surface} requires the real Chrome workflow, not isolated Playwright")
|
||||||
if surface not in PLAYWRIGHT_SURFACES:
|
if surface not in PLAYWRIGHT_SURFACES:
|
||||||
raise ValueError(f"surface {surface} is not a Playwright Web oracle")
|
raise ValueError(f"surface {surface} is not a Playwright Web oracle")
|
||||||
slot = manifest["slot"]
|
slot = manifest["slot"]
|
||||||
if slot is None:
|
if slot is None:
|
||||||
raise ValueError("Playwright batches require an explicit execution slot")
|
raise ValueError("Playwright batches require an explicit execution slot")
|
||||||
secret_values = dict(environ) if env_file is None else oracles.secret_environment(env_file, environ=environ)
|
secret_values = dict(environ) if env_file is None else oracles.secret_environment(env_file, environ=environ)
|
||||||
proxy = proxy_settings(
|
proxy = proxy_settings(slot, secret_values)
|
||||||
slot,
|
|
||||||
secret_values,
|
|
||||||
sticky_session_id=f"oracle{results['manifest_sha256'][:16]}",
|
|
||||||
)
|
|
||||||
pending = [
|
pending = [
|
||||||
(manifest_row, result_row)
|
(manifest_row, result_row)
|
||||||
for manifest_row, result_row in zip(manifest["rows"], results["rows"], strict=True)
|
for manifest_row, result_row in zip(manifest["rows"], results["rows"], strict=True)
|
||||||
|
|||||||
@@ -169,6 +169,26 @@ SURFACES: dict[str, OracleSurface] = {
|
|||||||
account_slots=True,
|
account_slots=True,
|
||||||
network_slots=True,
|
network_slots=True,
|
||||||
),
|
),
|
||||||
|
"synthid-web": OracleSurface(
|
||||||
|
key="synthid-web",
|
||||||
|
provider_key="google",
|
||||||
|
provider="Google",
|
||||||
|
signal_family="synthid",
|
||||||
|
surface="SynthID Detector",
|
||||||
|
automation="playwright",
|
||||||
|
url="https://synthid.com/",
|
||||||
|
media_types=("image", "video", "audio"),
|
||||||
|
project_evidence=(),
|
||||||
|
prompt=None,
|
||||||
|
clean_result_note=(
|
||||||
|
"Record detected only for the explicit label Watermark detected, and not_detected for "
|
||||||
|
"Watermark not detected or SynthID was not detected. The portal reports partner SynthID from "
|
||||||
|
"Google DeepMind, Nvidia, OpenAI, and Kakao without naming which partner. A not_detected "
|
||||||
|
"result is not proof of human authorship. Preserve quota, sign-in, and unclear wording."
|
||||||
|
),
|
||||||
|
account_slots=False,
|
||||||
|
network_slots=True,
|
||||||
|
),
|
||||||
"meta-web": OracleSurface(
|
"meta-web": OracleSurface(
|
||||||
key="meta-web",
|
key="meta-web",
|
||||||
provider_key="meta",
|
provider_key="meta",
|
||||||
@@ -278,6 +298,11 @@ def ordered_surfaces() -> list[OracleSurface]:
|
|||||||
return sorted(SURFACES.values(), key=_surface_order)
|
return sorted(SURFACES.values(), key=_surface_order)
|
||||||
|
|
||||||
|
|
||||||
|
def _automation_keys(automation: str) -> frozenset[str]:
|
||||||
|
"""Return the surface keys that use one automation kind."""
|
||||||
|
return frozenset(key for key, surface in SURFACES.items() if surface.automation == automation)
|
||||||
|
|
||||||
|
|
||||||
def _inside(path: Path, parent: Path) -> bool:
|
def _inside(path: Path, parent: Path) -> bool:
|
||||||
"""Return whether resolved PATH is inside resolved PARENT."""
|
"""Return whether resolved PATH is inside resolved PARENT."""
|
||||||
try:
|
try:
|
||||||
@@ -364,6 +389,8 @@ def load_slots(path: Path) -> dict[str, ExecutionSlot]:
|
|||||||
"api_key_env",
|
"api_key_env",
|
||||||
}
|
}
|
||||||
slots: dict[str, ExecutionSlot] = {}
|
slots: dict[str, ExecutionSlot] = {}
|
||||||
|
real_browser = _automation_keys("real_browser")
|
||||||
|
playwright = _automation_keys("playwright")
|
||||||
for index, raw_slot in enumerate(slot_items):
|
for index, raw_slot in enumerate(slot_items):
|
||||||
if not isinstance(raw_slot, dict):
|
if not isinstance(raw_slot, dict):
|
||||||
raise ValueError(f"slot {index} must be a JSON object")
|
raise ValueError(f"slot {index} must be a JSON object")
|
||||||
@@ -407,19 +434,18 @@ def load_slots(path: Path) -> dict[str, ExecutionSlot]:
|
|||||||
raise ValueError(f"surface {surface_key} requires an account_label")
|
raise ValueError(f"surface {surface_key} requires an account_label")
|
||||||
if account_label is not None and not SURFACES[surface_key].account_slots:
|
if account_label is not None and not SURFACES[surface_key].account_slots:
|
||||||
raise ValueError(f"surface {surface_key} does not use account slots")
|
raise ValueError(f"surface {surface_key} does not use account slots")
|
||||||
if surface_key == "gemini-web" and browser_profile is None:
|
if surface_key in real_browser and browser_profile is None:
|
||||||
raise ValueError("surface gemini-web requires a browser_profile")
|
raise ValueError(f"surface {surface_key} requires a browser_profile")
|
||||||
if surface_key == "gemini-web" and google_account_index is None:
|
if surface_key == "gemini-web" and google_account_index is None:
|
||||||
raise ValueError("surface gemini-web requires a google_account_index")
|
raise ValueError("surface gemini-web requires a google_account_index")
|
||||||
if surface_key != "gemini-web" and google_account_index is not None:
|
if surface_key != "gemini-web" and google_account_index is not None:
|
||||||
raise ValueError("only gemini-web uses google_account_index")
|
raise ValueError("only gemini-web uses google_account_index")
|
||||||
if SURFACES[surface_key].automation == "api" and browser_profile is not None:
|
if SURFACES[surface_key].automation == "api" and browser_profile is not None:
|
||||||
raise ValueError(f"API surface {surface_key} does not use a browser_profile")
|
raise ValueError(f"API surface {surface_key} does not use a browser_profile")
|
||||||
if surface_key != "gemini-web" and browser_profile is not None:
|
if surface_key not in real_browser and browser_profile is not None:
|
||||||
raise ValueError("only gemini-web uses a real browser_profile")
|
raise ValueError("only a real-browser surface uses a browser_profile")
|
||||||
if surface_key == "gemini-web" and proxy_url_env is not None:
|
# A proxy belongs to Playwright pages. Gemini and API calls stay direct.
|
||||||
raise ValueError("surface gemini-web does not use proxy_url_env")
|
if proxy_url_env is not None and surface_key not in playwright:
|
||||||
if proxy_url_env is not None and surface_key not in {"openai-web", "microsoft-web", "meta-web"}:
|
|
||||||
raise ValueError(f"surface {surface_key} does not use proxy_url_env")
|
raise ValueError(f"surface {surface_key} does not use proxy_url_env")
|
||||||
if surface_key == "openai-api" and api_key_env is None:
|
if surface_key == "openai-api" and api_key_env is None:
|
||||||
raise ValueError("surface openai-api requires an api_key_env")
|
raise ValueError("surface openai-api requires an api_key_env")
|
||||||
@@ -858,10 +884,11 @@ def list_oracles(as_json: bool) -> None:
|
|||||||
def plan(provider: str, as_json: bool) -> None:
|
def plan(provider: str, as_json: bool) -> None:
|
||||||
"""Show the API-first surface order for one provider."""
|
"""Show the API-first surface order for one provider."""
|
||||||
definitions = provider_plan(provider)
|
definitions = provider_plan(provider)
|
||||||
|
has_api = any(definition.automation == "api" for definition in definitions)
|
||||||
rows = []
|
rows = []
|
||||||
for index, definition in enumerate(definitions):
|
for definition in definitions:
|
||||||
row = definition.to_dict()
|
row = definition.to_dict()
|
||||||
row["preference"] = "primary" if index == 0 else "fallback"
|
row["preference"] = "fallback" if has_api and definition.automation != "api" else "primary"
|
||||||
rows.append(row)
|
rows.append(row)
|
||||||
if as_json:
|
if as_json:
|
||||||
click.echo(json.dumps(rows, indent=2, sort_keys=True))
|
click.echo(json.dumps(rows, indent=2, sort_keys=True))
|
||||||
@@ -869,7 +896,7 @@ def plan(provider: str, as_json: bool) -> None:
|
|||||||
for row in rows:
|
for row in rows:
|
||||||
click.echo(f"{row['preference']}\t{row['key']}\t{row['automation']}\t{row['surface']}")
|
click.echo(f"{row['preference']}\t{row['key']}\t{row['automation']}\t{row['surface']}")
|
||||||
if len(rows) > 1:
|
if len(rows) > 1:
|
||||||
click.echo("Web fallback requires a separate operator decision; failures do not trigger it automatically.")
|
click.echo("This command only prints the order. After a failure, run the next surface as its own command.")
|
||||||
|
|
||||||
|
|
||||||
@cli.command()
|
@cli.command()
|
||||||
@@ -1011,7 +1038,7 @@ def run_web(
|
|||||||
timeout_seconds: float,
|
timeout_seconds: float,
|
||||||
env_file: Path,
|
env_file: Path,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Submit a prepared non-Google Web batch through isolated Playwright."""
|
"""Submit a prepared anonymous Web batch through isolated Playwright."""
|
||||||
from provider_oracle_web import run_web_batch
|
from provider_oracle_web import run_web_batch
|
||||||
|
|
||||||
report = run_web_batch(
|
report = run_web_batch(
|
||||||
|
|||||||
@@ -0,0 +1,68 @@
|
|||||||
|
"""Rebuild synthetic silhouettes for additional AI disclosure layouts."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
sys.path.insert(0, str(ROOT))
|
||||||
|
from scripts.render_vendor_silhouettes import render # noqa: E402
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
ASSETS = ROOT / "src" / "remove_ai_watermarks" / "assets"
|
||||||
|
CJK = {"font": "/System/Library/Fonts/Hiragino Sans GB.ttc", "font_index": 2}
|
||||||
|
|
||||||
|
|
||||||
|
def centered_lines(
|
||||||
|
top: str, bottom: str, *, top_scale: float, bottom_height: float = 1.0, shear: float = 0.0
|
||||||
|
) -> np.ndarray:
|
||||||
|
upper, lower = render(top, opts=CJK), render(bottom, opts=CJK)
|
||||||
|
if bottom_height != 1.0:
|
||||||
|
lower = cv2.resize(lower, (upper.shape[1], round(upper.shape[0] * bottom_height)))
|
||||||
|
else:
|
||||||
|
upper = cv2.resize(upper, (round(lower.shape[1] * top_scale), round(lower.shape[0] * 0.95)))
|
||||||
|
lines = [upper, lower]
|
||||||
|
result = np.zeros((sum(line.shape[0] for line in lines) + 2, max(line.shape[1] for line in lines)), np.uint8)
|
||||||
|
y = 0
|
||||||
|
for line in lines:
|
||||||
|
x = (result.shape[1] - line.shape[1]) // 2
|
||||||
|
result[y : y + line.shape[0], x : x + line.shape[1]] = line
|
||||||
|
y += line.shape[0] + 2
|
||||||
|
if shear:
|
||||||
|
h, w = result.shape
|
||||||
|
extra = round(abs(shear) * h)
|
||||||
|
result = cv2.warpAffine(result, np.float32([[1, shear, extra], [0, 1, 0]]), (w + extra, h))
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def silhouettes() -> dict[str, np.ndarray]:
|
||||||
|
return {
|
||||||
|
"workbuddy_alpha.png": centered_lines("AI生成", "WORKBUDDY>_", top_scale=1.0, bottom_height=0.45),
|
||||||
|
"yuanbao_compact_alpha.png": centered_lines("元宝", "AI生成", top_scale=0.65, shear=-0.15),
|
||||||
|
"yuanbao_compact_gray_alpha.png": centered_lines("元宝", "AI生成", top_scale=0.65, shear=-0.15),
|
||||||
|
"doubao_outline_alpha.png": render("豆包AI生成", opts=CJK),
|
||||||
|
"samsung_ko_alpha.png": render(
|
||||||
|
"AI로 생성한 콘텐츠", opts={"font": "/System/Library/Fonts/AppleSDGothicNeo.ttc"}
|
||||||
|
),
|
||||||
|
"microsoft_text_alpha.png": render(
|
||||||
|
"Made with AI", opts={"font": "/System/Library/Fonts/Supplemental/Arial.ttf"}
|
||||||
|
),
|
||||||
|
"jimeng_label_alpha.png": render("AI生成", opts={"font": "/System/Library/Fonts/STHeiti Medium.ttc"}),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
logging.basicConfig(level=logging.INFO)
|
||||||
|
for name, alpha in silhouettes().items():
|
||||||
|
Image.fromarray(alpha).save(ASSETS / name)
|
||||||
|
log.info("Rendered %s, %s x %s", name, alpha.shape[1], alpha.shape[0])
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -73,7 +73,7 @@ MARKS = {
|
|||||||
"gemini_text_alpha.png": "Generated with Gemini",
|
"gemini_text_alpha.png": "Generated with Gemini",
|
||||||
# Candidate wordmarks measured on a local evaluation corpus (unregistered).
|
# Candidate wordmarks measured on a local evaluation corpus (unregistered).
|
||||||
"notebooklm_alpha.png": "NotebookLM",
|
"notebooklm_alpha.png": "NotebookLM",
|
||||||
"dola_alpha.png": "DolaAI",
|
"dola_alpha.png": "Dola AI",
|
||||||
"mindvideo_alpha.png": "MindVideo.AI",
|
"mindvideo_alpha.png": "MindVideo.AI",
|
||||||
"higgsfield_alpha.png": "HIGGSFIELD AI",
|
"higgsfield_alpha.png": "HIGGSFIELD AI",
|
||||||
"capcut_alpha.png": "CapCut AI",
|
"capcut_alpha.png": "CapCut AI",
|
||||||
@@ -92,6 +92,7 @@ _REGISTERED = {f"{key}_alpha.png" for key in mark_keys()} & MARKS.keys()
|
|||||||
|
|
||||||
# Per-mark post-processing for the multi-line / slanted stamps (see render()).
|
# Per-mark post-processing for the multi-line / slanted stamps (see render()).
|
||||||
MARK_OPTS: dict[str, dict[str, Any]] = {
|
MARK_OPTS: dict[str, dict[str, Any]] = {
|
||||||
|
"dola_alpha.png": {"font": "/System/Library/Fonts/Supplemental/Arial Bold.ttf"},
|
||||||
# Hiragino Sans GB W6, tight leading, dilation, and negative shear match the
|
# Hiragino Sans GB W6, tight leading, dilation, and negative shear match the
|
||||||
# standard Yuanbao stamp without clipping the lower line.
|
# standard Yuanbao stamp without clipping the lower line.
|
||||||
"yuanbao_alpha.png": {
|
"yuanbao_alpha.png": {
|
||||||
|
|||||||
@@ -98,7 +98,7 @@ def _train_detector(emb: dict[str, np.ndarray], catalog: dict) -> tuple[nn.Modul
|
|||||||
optimizer.zero_grad()
|
optimizer.zero_grad()
|
||||||
loss_fn(model(features).squeeze(-1), labels).backward()
|
loss_fn(model(features).squeeze(-1), labels).backward()
|
||||||
optimizer.step()
|
optimizer.step()
|
||||||
model.eval()
|
model.train(False)
|
||||||
dev = np.stack(
|
dev = np.stack(
|
||||||
[
|
[
|
||||||
emb[row["sha256"]]
|
emb[row["sha256"]]
|
||||||
@@ -161,7 +161,7 @@ def _train_provider(feat: dict[str, np.ndarray], catalog: dict) -> dict[str, nn.
|
|||||||
optimizer.zero_grad()
|
optimizer.zero_grad()
|
||||||
_focal_loss(model(features).squeeze(-1), labels).backward()
|
_focal_loss(model(features).squeeze(-1), labels).backward()
|
||||||
optimizer.step()
|
optimizer.step()
|
||||||
heads[name] = model.eval()
|
heads[name] = model.train(False)
|
||||||
log.info("provider %s pos=%d", name, len(positive))
|
log.info("provider %s pos=%d", name, len(positive))
|
||||||
return heads
|
return heads
|
||||||
|
|
||||||
|
|||||||
@@ -127,7 +127,7 @@ def load_model() -> object:
|
|||||||
import torch
|
import torch
|
||||||
|
|
||||||
model = torch.jit.load(str(checkpoint_path()), map_location="cpu")
|
model = torch.jit.load(str(checkpoint_path()), map_location="cpu")
|
||||||
model.eval()
|
model.train(False)
|
||||||
return model
|
return model
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
{
|
||||||
|
"$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
|
||||||
|
"name": "io.github.wiltodelta/remove-ai-watermarks",
|
||||||
|
"title": "Remove AI Watermarks",
|
||||||
|
"description": "Identify and remove AI watermarks and provenance metadata from your own images and video.",
|
||||||
|
"version": "0.47.0",
|
||||||
|
"repository": {
|
||||||
|
"url": "https://github.com/wiltodelta/remove-ai-watermarks",
|
||||||
|
"source": "github"
|
||||||
|
},
|
||||||
|
"websiteUrl": "https://github.com/wiltodelta/remove-ai-watermarks/blob/main/docs/mcp-server.md",
|
||||||
|
"packages": [
|
||||||
|
{
|
||||||
|
"registryType": "pypi",
|
||||||
|
"registryBaseUrl": "https://pypi.org",
|
||||||
|
"identifier": "remove-ai-watermarks",
|
||||||
|
"version": "0.47.0",
|
||||||
|
"runtimeHint": "uvx",
|
||||||
|
"transport": {
|
||||||
|
"type": "stdio"
|
||||||
|
},
|
||||||
|
"packageArguments": [
|
||||||
|
{
|
||||||
|
"type": "positional",
|
||||||
|
"value": "mcp"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -1,7 +1,7 @@
|
|||||||
{
|
{
|
||||||
"name": "remove-ai-watermarks",
|
"name": "remove-ai-watermarks",
|
||||||
"displayName": "Remove AI Watermarks",
|
"displayName": "Remove AI Watermarks",
|
||||||
"version": "1.0.10",
|
"version": "1.0.13",
|
||||||
"description": "Identify and remove AI provenance marks from images and video the user generated or edited, using the remove-ai-watermarks CLI.",
|
"description": "Identify and remove AI provenance marks from images and video the user generated or edited, using the remove-ai-watermarks CLI.",
|
||||||
"author": {
|
"author": {
|
||||||
"name": "wiltodelta",
|
"name": "wiltodelta",
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ license: Apache-2.0
|
|||||||
compatibility: Python 3.11-3.14. Invisible image removal needs NVIDIA CUDA. Some video writes need ffmpeg. Installer may be uv, pipx, or pip.
|
compatibility: Python 3.11-3.14. Invisible image removal needs NVIDIA CUDA. Some video writes need ffmpeg. Installer may be uv, pipx, or pip.
|
||||||
metadata:
|
metadata:
|
||||||
author: wiltodelta
|
author: wiltodelta
|
||||||
version: "1.0.10"
|
version: "1.0.13"
|
||||||
homepage: https://raiw.cc
|
homepage: https://raiw.cc
|
||||||
repository: https://github.com/wiltodelta/remove-ai-watermarks
|
repository: https://github.com/wiltodelta/remove-ai-watermarks
|
||||||
---
|
---
|
||||||
@@ -27,8 +27,12 @@ change.
|
|||||||
|
|
||||||
Refuse automatic removal of stock-agency, marketplace, classifieds,
|
Refuse automatic removal of stock-agency, marketplace, classifieds,
|
||||||
tiled-preview, or other marks that protect a third party's paid or copyrighted
|
tiled-preview, or other marks that protect a third party's paid or copyrighted
|
||||||
asset. `erase --region` is allowed only when the user names the file they own
|
asset, or product branding a paid plan removes, such as "Made with Gamma". A
|
||||||
and supplies the box themselves.
|
mark its vendor describes as identifying AI-generated content (the Gemini
|
||||||
|
sparkle, the Gemini Notebook watermark) stays in scope whatever plan hides it.
|
||||||
|
`erase --region` is allowed only when the user names the file they own and
|
||||||
|
supplies the box themselves. LLM text watermarks are out of scope: they live in
|
||||||
|
word choice, not characters, so do not offer to strip characters from text.
|
||||||
|
|
||||||
Removing a local signal does not prove the file is human-made, does not erase
|
Removing a local signal does not prove the file is human-made, does not erase
|
||||||
server-side generation history, and does not make a deceptive origin claim
|
server-side generation history, and does not make a deceptive origin claim
|
||||||
@@ -153,8 +157,8 @@ Read the process exit code and whether `-o` exists.
|
|||||||
| Exit | Meaning | What to tell the user |
|
| Exit | Meaning | What to tell the user |
|
||||||
| --- | --- | --- |
|
| --- | --- | --- |
|
||||||
| 0 | Requested work completed | The output path. Do not add "this is clean". |
|
| 0 | Requested work completed | The output path. Do not add "this is clean". |
|
||||||
| 2 | No targeted signal | Unknown, not clean. No new file from `visible`. |
|
| 2 | No targeted signal, or a `batch` directory with no supported file | Unknown, not clean. No new file from `visible`. |
|
||||||
| 1 | Error or partial `all` | Quote the error. Say which stage ran, if any. |
|
| 1 | Error, unreadable input, or partial `all` | Quote the error. Say which stage ran, if any. |
|
||||||
|
|
||||||
Exit 2 is ALSO what a bad invocation returns: a path that does not exist, an
|
Exit 2 is ALSO what a bad invocation returns: a path that does not exist, an
|
||||||
unknown flag, a directory where a file was expected. Read the output before
|
unknown flag, a directory where a file was expected. Read the output before
|
||||||
@@ -163,7 +167,9 @@ means the command never ran, so fix it and rerun; only a command that reached
|
|||||||
the image can say "no targeted signal".
|
the image can say "no targeted signal".
|
||||||
|
|
||||||
`identify` with no signal reports origin unknown. Never rephrase that as "no
|
`identify` with no signal reports origin unknown. Never rephrase that as "no
|
||||||
watermark" or "this is clean".
|
watermark" or "this is clean". When `identify` or `metadata` exits 1 with
|
||||||
|
`cannot read`, the file is damaged or not an image or video: any report above
|
||||||
|
that line is incomplete, so say the file could not be read.
|
||||||
|
|
||||||
`metadata --check` and `metadata --remove` answer a narrower question than
|
`metadata --check` and `metadata --remove` answer a narrower question than
|
||||||
`identify`: embedded AI metadata only. Both print that this is not a clean
|
`identify`: embedded AI metadata only. Both print that this is not a clean
|
||||||
|
|||||||
@@ -21,8 +21,8 @@ remove-ai-watermarks visible image.png --keep-metadata -o clean.png
|
|||||||
```
|
```
|
||||||
|
|
||||||
Image mark keys: `gemini`, `doubao`, `jimeng`, `qwen`, `wan`, `kling`, `yuanbao`,
|
Image mark keys: `gemini`, `doubao`, `jimeng`, `qwen`, `wan`, `kling`, `yuanbao`,
|
||||||
`samsung`, `runninghub`, `baidu`, `liblib`, `liblib_pill`, `microsoft`,
|
`dola`, `workbuddy`, `samsung`, `runninghub`, `baidu`, `liblib`, `liblib_pill`, `microsoft`,
|
||||||
`generic_ai_label`, `openart`, `jimeng_pill`. `wan` is new (since CLI 0.43.0). Default
|
`generic_ai_label`, `openart`, `jimeng_pill`. `dola` and `workbuddy` are new (since CLI 0.45.0). Default
|
||||||
`--mark auto` removes every selected match. `microsoft` is the top-right AI
|
`--mark auto` removes every selected match. `microsoft` is the top-right AI
|
||||||
badge, separate from the invisible InvisMark watermark on the same vendor's
|
badge, separate from the invisible InvisMark watermark on the same vendor's
|
||||||
images. `jimeng_pill` is a weak detector and needs corroboration. The compact
|
images. `jimeng_pill` is a weak detector and needs corroboration. The compact
|
||||||
@@ -138,13 +138,20 @@ user names a device, and never to claim a device the probe did not report.
|
|||||||
Without `-o`, video commands write `<source>_clean` and keep the original.
|
Without `-o`, video commands write `<source>_clean` and keep the original.
|
||||||
Output extension must match the input container.
|
Output extension must match the input container.
|
||||||
|
|
||||||
|
## MCP server
|
||||||
|
|
||||||
|
`remove-ai-watermarks mcp` (since CLI 0.46.0) serves `identify`, `remove`
|
||||||
|
and `batch` as MCP tools over stdin and stdout for MCP clients. It blocks
|
||||||
|
waiting for protocol messages, so do not run it from this skill; call the
|
||||||
|
commands above directly.
|
||||||
|
|
||||||
## Exit codes
|
## Exit codes
|
||||||
|
|
||||||
| Code | Meaning |
|
| Code | Meaning |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
| 0 | Requested work completed |
|
| 0 | Requested work completed |
|
||||||
| 2 | No targeted signal (`visible` / no-signal `invisible`) |
|
| 2 | No targeted signal (`visible` / no-signal `invisible`), or a `batch` directory with no supported file |
|
||||||
| 1 | Hard failure, or `all` skipped a required invisible stage |
|
| 1 | Hard failure, an input that is not a readable image or video (`identify` and `metadata` included), or `all` skipped a required invisible stage |
|
||||||
|
|
||||||
A written file plus exit `1` from `all` is a partial result.
|
A written file plus exit `1` from `all` is a partial result.
|
||||||
|
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ _GPU_MARKERS = ("GPU ", "gpu:")
|
|||||||
# told an agent its CLI was current right before it typed a value that build
|
# told an agent its CLI was current right before it typed a value that build
|
||||||
# rejects. tests/test_agent_skill.py pins it against the package version and
|
# rejects. tests/test_agent_skill.py pins it against the package version and
|
||||||
# against every "(since CLI X.Y.Z)" annotation in the skill.
|
# against every "(since CLI X.Y.Z)" annotation in the skill.
|
||||||
MIN_CLI_VERSION = (0, 43, 0)
|
MIN_CLI_VERSION = (0, 47, 0)
|
||||||
# Markers a build without the pixel stack prints. The first is the current CLI's own
|
# Markers a build without the pixel stack prints. The first is the current CLI's own
|
||||||
# install hint; the rest are what older builds emit before the guard existed, and the
|
# install hint; the rest are what older builds emit before the guard existed, and the
|
||||||
# Homebrew formula ships exactly such a build.
|
# Homebrew formula ships exactly such a build.
|
||||||
|
|||||||
@@ -35,7 +35,7 @@ _os.environ.setdefault("TRANSFORMERS_VERBOSITY", "error")
|
|||||||
_warnings.filterwarnings("ignore", message=r".*ImageProcessorFast.*")
|
_warnings.filterwarnings("ignore", message=r".*ImageProcessorFast.*")
|
||||||
|
|
||||||
|
|
||||||
__version__ = "0.44.0"
|
__version__ = "0.47.0"
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
"BatchItemResult",
|
"BatchItemResult",
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
"""Generated C2PA soft-binding registry; do not edit by hand.
|
"""Generated C2PA soft-binding registry; do not edit by hand.
|
||||||
|
|
||||||
Source: https://raw.githubusercontent.com/c2pa-org/softbinding-algorithm-list/main/softbinding-algorithm-list.json
|
Source: https://raw.githubusercontent.com/c2pa-org/softbinding-algorithm-list/main/softbinding-algorithm-list.json
|
||||||
Revision: d36d1c1742b1753ca064bb3307abfd95db912a28
|
Revision: d5960248ab69c17dc194906d2ef7d11ad164be70
|
||||||
Source license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
|
Source license: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
|
||||||
Changes: validated, ordered, and reduced to fields used at runtime; descriptions
|
Changes: validated, ordered, and reduced to fields used at runtime; descriptions
|
||||||
are shortened to their first sentence for display.
|
are shortened to their first sentence for display.
|
||||||
@@ -12,7 +12,7 @@ from __future__ import annotations
|
|||||||
C2PA_SOFT_BINDING_SOURCE_URL = (
|
C2PA_SOFT_BINDING_SOURCE_URL = (
|
||||||
"https://raw.githubusercontent.com/c2pa-org/softbinding-algorithm-list/main/softbinding-algorithm-list.json"
|
"https://raw.githubusercontent.com/c2pa-org/softbinding-algorithm-list/main/softbinding-algorithm-list.json"
|
||||||
)
|
)
|
||||||
C2PA_SOFT_BINDING_SOURCE_REVISION = "d36d1c1742b1753ca064bb3307abfd95db912a28"
|
C2PA_SOFT_BINDING_SOURCE_REVISION = "d5960248ab69c17dc194906d2ef7d11ad164be70"
|
||||||
C2PA_SOFT_BINDING_SOURCE_LICENSE = "CC BY 4.0"
|
C2PA_SOFT_BINDING_SOURCE_LICENSE = "CC BY 4.0"
|
||||||
|
|
||||||
C2PA_SOFT_BINDING_ROWS: tuple[
|
C2PA_SOFT_BINDING_ROWS: tuple[
|
||||||
@@ -632,4 +632,15 @@ C2PA_SOFT_BINDING_ROWS: tuple[
|
|||||||
(),
|
(),
|
||||||
False,
|
False,
|
||||||
),
|
),
|
||||||
|
(
|
||||||
|
54,
|
||||||
|
"be.immostory.videoseal.1",
|
||||||
|
"watermark",
|
||||||
|
("video",),
|
||||||
|
("video/mp4",),
|
||||||
|
"ImmoStory AI deployment of Meta FAIR's Video Seal 1.0 invisible video watermark",
|
||||||
|
"2026-10-12T00:00:00.000Z",
|
||||||
|
(),
|
||||||
|
False,
|
||||||
|
),
|
||||||
)
|
)
|
||||||
@@ -67,7 +67,7 @@ def load_headed_clip(checkpoint: Path, device: torch.device) -> HeadedCLIP:
|
|||||||
model = HeadedCLIP(clip)
|
model = HeadedCLIP(clip)
|
||||||
state = torch.load(checkpoint, map_location="cpu", weights_only=True)
|
state = torch.load(checkpoint, map_location="cpu", weights_only=True)
|
||||||
model.load_state_dict(state, strict=True)
|
model.load_state_dict(state, strict=True)
|
||||||
return model.to(device).eval()
|
return model.to(device).train(False)
|
||||||
|
|
||||||
|
|
||||||
def load_processor() -> Any:
|
def load_processor() -> Any:
|
||||||
|
|||||||
@@ -16,7 +16,7 @@ if TYPE_CHECKING:
|
|||||||
|
|
||||||
from remove_ai_watermarks.metadata import MAX_TC260_VALUE_BYTES, parse_tc260_aigc_json
|
from remove_ai_watermarks.metadata import MAX_TC260_VALUE_BYTES, parse_tc260_aigc_json
|
||||||
|
|
||||||
_EBML_MAGIC = b"\x1aE\xdf\xa3"
|
EBML_MAGIC = b"\x1aE\xdf\xa3"
|
||||||
_SEGMENT_ID = 0x18538067
|
_SEGMENT_ID = 0x18538067
|
||||||
_TAGS_ID = 0x1254C367
|
_TAGS_ID = 0x1254C367
|
||||||
_TAG_ID = 0x7373
|
_TAG_ID = 0x7373
|
||||||
@@ -95,6 +95,16 @@ def _iter_elements(
|
|||||||
pos = element_end
|
pos = element_end
|
||||||
|
|
||||||
|
|
||||||
|
def iter_top_level_ends(stream: BinaryIO, size: int) -> Iterator[int]:
|
||||||
|
"""End offset of each valid top-level element; stops early where a header fails.
|
||||||
|
|
||||||
|
An element whose declared size runs past ``size`` fails its header, so a truncated
|
||||||
|
file yields a last end short of ``size``. An unknown-size Segment runs to ``size``.
|
||||||
|
"""
|
||||||
|
for _element_id, _payload_start, element_end in _iter_elements(stream, 0, size):
|
||||||
|
yield element_end
|
||||||
|
|
||||||
|
|
||||||
def _read_bounded(
|
def _read_bounded(
|
||||||
stream: BinaryIO,
|
stream: BinaryIO,
|
||||||
start: int,
|
start: int,
|
||||||
@@ -133,7 +143,7 @@ def tc260_aigc_payloads(path: str | Path) -> tuple[bytes, ...]:
|
|||||||
found: list[bytes] = []
|
found: list[bytes] = []
|
||||||
try:
|
try:
|
||||||
with open(path, "rb") as stream:
|
with open(path, "rb") as stream:
|
||||||
if stream.read(4) != _EBML_MAGIC:
|
if stream.read(4) != EBML_MAGIC:
|
||||||
return ()
|
return ()
|
||||||
stream.seek(0, 2)
|
stream.seek(0, 2)
|
||||||
file_size = stream.tell()
|
file_size = stream.tell()
|
||||||
|
|||||||
@@ -7,7 +7,9 @@ without re-encoding, the image path drops matching boxes and emits the rest
|
|||||||
verbatim. The streaming MP4/MOV/M4A path instead preserves all offsets by
|
verbatim. The streaming MP4/MOV/M4A path instead preserves all offsets by
|
||||||
retyping matching boxes as ``free`` and blanking their payloads in place. The
|
retyping matching boxes as ``free`` and blanking their payloads in place. The
|
||||||
codestream (``mdat`` for ISOBMFF, ``jxlc`` / ``jxlp`` for JPEG-XL) is untouched,
|
codestream (``mdat`` for ISOBMFF, ``jxlc`` / ``jxlp`` for JPEG-XL) is untouched,
|
||||||
so pixel, video, and audio data is preserved bit-for-bit.
|
so pixel, video, and audio data is preserved bit-for-bit. The one exception is
|
||||||
|
Kling's SEI tag inside ``mdat``: its bytes are blanked in place, which no decoder
|
||||||
|
reads, so every decoded frame stays identical.
|
||||||
|
|
||||||
TC260-PG-20257A video metadata is nested instead:
|
TC260-PG-20257A video metadata is nested instead:
|
||||||
``moov.udta.meta.keys/ilst``. Its detector seeks through those boxes without
|
``moov.udta.meta.keys/ilst``. Its detector seeks through those boxes without
|
||||||
@@ -43,8 +45,11 @@ from remove_ai_watermarks.metadata import (
|
|||||||
C2PA_UUID,
|
C2PA_UUID,
|
||||||
IPTC_AI_FIELD_MARKERS,
|
IPTC_AI_FIELD_MARKERS,
|
||||||
IPTC_AI_MARKERS,
|
IPTC_AI_MARKERS,
|
||||||
|
KLING_SEI_TAG,
|
||||||
LOCAL_GENERATION_METADATA_KEYS,
|
LOCAL_GENERATION_METADATA_KEYS,
|
||||||
MAX_TC260_VALUE_BYTES,
|
MAX_TC260_VALUE_BYTES,
|
||||||
|
app_generator_from_metadata,
|
||||||
|
app_provenance_from_metadata,
|
||||||
parse_tc260_aigc_json,
|
parse_tc260_aigc_json,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -347,6 +352,55 @@ def _ai_metadata_regions(
|
|||||||
return regions
|
return regions
|
||||||
|
|
||||||
|
|
||||||
|
def _app_metadata_regions(
|
||||||
|
stream: BinaryIO,
|
||||||
|
file_size: int,
|
||||||
|
) -> list[tuple[tuple[int, int] | None, int, int]]:
|
||||||
|
"""Locate keyed or keyless values carrying an app-export provenance object.
|
||||||
|
|
||||||
|
Dreamina writes its export JSON under ordinary keys (``LvMetaInfo`` and a
|
||||||
|
second copy under ``com.apple.quicktime.artwork``). The file-level scan reads
|
||||||
|
it from the raw head bytes, so removal must blank every item it would match.
|
||||||
|
"""
|
||||||
|
return [
|
||||||
|
(key_span, value_start, value_end)
|
||||||
|
for _name, key_span, value_start, value_end, value in _metadata_items(stream, file_size)
|
||||||
|
if app_provenance_from_metadata(value) or app_generator_from_metadata(value)
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def _copy_finding_kling_sei(
|
||||||
|
source: BinaryIO,
|
||||||
|
target: BinaryIO,
|
||||||
|
mdat_ranges: list[tuple[int, int]],
|
||||||
|
) -> list[int]:
|
||||||
|
"""Copy ``source`` to ``target`` and return each Kling SEI tag offset in ``mdat``.
|
||||||
|
|
||||||
|
Detection reads the tag from the head window, but removal covers every copy in
|
||||||
|
the media data. The search rides on the copy, so the file is read once. Each
|
||||||
|
window carries one byte less than the tag from the previous chunk, so a copy
|
||||||
|
straddling a read boundary is found exactly once.
|
||||||
|
"""
|
||||||
|
keep = len(KLING_SEI_TAG) - 1
|
||||||
|
offsets: list[int] = []
|
||||||
|
tail = b""
|
||||||
|
pos = 0
|
||||||
|
while chunk := source.read(_STREAM_COPY_BYTES):
|
||||||
|
target.write(chunk)
|
||||||
|
if mdat_ranges:
|
||||||
|
window = tail + chunk
|
||||||
|
base = pos - len(tail)
|
||||||
|
found = window.find(KLING_SEI_TAG)
|
||||||
|
while found != -1:
|
||||||
|
start = base + found
|
||||||
|
if any(low <= start and start + len(KLING_SEI_TAG) <= high for low, high in mdat_ranges):
|
||||||
|
offsets.append(start)
|
||||||
|
found = window.find(KLING_SEI_TAG, found + 1)
|
||||||
|
tail = window[-keep:]
|
||||||
|
pos += len(chunk)
|
||||||
|
return offsets
|
||||||
|
|
||||||
|
|
||||||
def ai_metadata_tags(path: str | Path, *, strict: bool = False) -> dict[str, str]:
|
def ai_metadata_tags(path: str | Path, *, strict: bool = False) -> dict[str, str]:
|
||||||
"""Read keyed AI-generation metadata from an ISOBMFF container."""
|
"""Read keyed AI-generation metadata from an ISOBMFF container."""
|
||||||
try:
|
try:
|
||||||
@@ -601,15 +655,16 @@ def strip_isobmff_media_file(
|
|||||||
|
|
||||||
The output retains every box size and byte offset. A top-level C2PA/JUMBF or
|
The output retains every box size and byte offset. A top-level C2PA/JUMBF or
|
||||||
AI-label box is converted to a ``free`` box and its payload is zeroed; native
|
AI-label box is converted to a ``free`` box and its payload is zeroed; native
|
||||||
TC260 key/value spans are blanked in place. Keeping the original lengths is
|
TC260, generation-key and app-export key/value spans, and the Kling SEI tag in
|
||||||
required because removing a pre-``mdat`` box would invalidate absolute media
|
``mdat``, are blanked in place. Keeping the original lengths is required
|
||||||
offsets in an existing sample table.
|
because removing a pre-``mdat`` box would invalidate absolute media offsets in
|
||||||
|
an existing sample table.
|
||||||
|
|
||||||
The source is copied in bounded chunks to a sibling temporary file and
|
The source is copied in bounded chunks to a sibling temporary file and
|
||||||
atomically published only after all patches succeed. A malformed top-level
|
atomically published only after all patches succeed. A malformed top-level
|
||||||
walk is fail-safe: the input is copied unchanged.
|
walk is fail-safe: the input is copied unchanged.
|
||||||
|
|
||||||
Returns ``(provenance_boxes_blanked, native_tc260_keys_blanked)``.
|
Returns ``(provenance_boxes_blanked, native_tags_blanked)``.
|
||||||
"""
|
"""
|
||||||
from pathlib import Path as _Path
|
from pathlib import Path as _Path
|
||||||
|
|
||||||
@@ -627,12 +682,23 @@ def strip_isobmff_media_file(
|
|||||||
)
|
)
|
||||||
tc260_regions = _tc260_aigc_regions(stream, file_size) if targets is not None else []
|
tc260_regions = _tc260_aigc_regions(stream, file_size) if targets is not None else []
|
||||||
ai_metadata_regions = _ai_metadata_regions(stream, file_size) if targets is not None else []
|
ai_metadata_regions = _ai_metadata_regions(stream, file_size) if targets is not None else []
|
||||||
|
app_regions = _app_metadata_regions(stream, file_size) if targets is not None else []
|
||||||
|
mdat_ranges = (
|
||||||
|
[
|
||||||
|
(payload, end)
|
||||||
|
for _start, end, box_type, payload in iter_file_boxes(stream, 0, file_size)
|
||||||
|
if box_type == b"mdat"
|
||||||
|
]
|
||||||
|
if targets is not None
|
||||||
|
else []
|
||||||
|
)
|
||||||
tc260_key_spans = {region[0] for region in tc260_regions if region[0] is not None}
|
tc260_key_spans = {region[0] for region in tc260_regions if region[0] is not None}
|
||||||
ai_metadata_key_spans = {region[0] for region in ai_metadata_regions}
|
ai_metadata_key_spans = {region[0] for region in ai_metadata_regions}
|
||||||
|
app_key_spans = {region[0] for region in app_regions if region[0] is not None}
|
||||||
|
|
||||||
with atomic_video_output(output_path) as temporary_path:
|
with atomic_video_output(output_path) as temporary_path:
|
||||||
with source_path.open("rb") as source_stream, temporary_path.open("r+b") as temporary:
|
with source_path.open("rb") as source_stream, temporary_path.open("r+b") as temporary:
|
||||||
shutil.copyfileobj(source_stream, temporary, length=_STREAM_COPY_BYTES)
|
kling_sei_offsets = _copy_finding_kling_sei(source_stream, temporary, mdat_ranges)
|
||||||
if targets is not None:
|
if targets is not None:
|
||||||
for box_start, payload_start, box_end in targets:
|
for box_start, payload_start, box_end in targets:
|
||||||
temporary.seek(box_start + 4)
|
temporary.seek(box_start + 4)
|
||||||
@@ -647,6 +713,15 @@ def strip_isobmff_media_file(
|
|||||||
temporary.seek(key_span[0])
|
temporary.seek(key_span[0])
|
||||||
temporary.write(b"free" + b" " * (key_span[1] - key_span[0] - 4))
|
temporary.write(b"free" + b" " * (key_span[1] - key_span[0] - 4))
|
||||||
_overwrite_range(temporary, value_start, value_end, byte=b" ")
|
_overwrite_range(temporary, value_start, value_end, byte=b" ")
|
||||||
|
for key_span, value_start, value_end in app_regions:
|
||||||
|
if key_span is not None:
|
||||||
|
temporary.seek(key_span[0])
|
||||||
|
temporary.write(b"free" + b" " * (key_span[1] - key_span[0] - 4))
|
||||||
|
_overwrite_range(temporary, value_start, value_end, byte=b" ")
|
||||||
|
# Spaces keep the SEI message well-formed: its length fields stand,
|
||||||
|
# and a 0x20 run cannot form an emulation-prevention sequence.
|
||||||
|
for offset in kling_sei_offsets:
|
||||||
|
_overwrite_range(temporary, offset, offset + len(KLING_SEI_TAG), byte=b" ")
|
||||||
temporary.flush()
|
temporary.flush()
|
||||||
os.fsync(temporary.fileno())
|
os.fsync(temporary.fileno())
|
||||||
shutil.copymode(source_path, temporary_path)
|
shutil.copymode(source_path, temporary_path)
|
||||||
@@ -657,13 +732,21 @@ def strip_isobmff_media_file(
|
|||||||
source_path,
|
source_path,
|
||||||
)
|
)
|
||||||
return 0, 0
|
return 0, 0
|
||||||
return len(targets), len(tc260_key_spans | ai_metadata_key_spans)
|
# Every copy of the Kling tag is one tag, however many samples repeat it.
|
||||||
|
return len(targets), len(tc260_key_spans | ai_metadata_key_spans | app_key_spans) + bool(kling_sei_offsets)
|
||||||
|
|
||||||
|
|
||||||
|
def _blank_box(data: bytes, start: int, end: int, payload_off: int) -> bytes:
|
||||||
|
"""Return ``data[start:end]`` as a ``free`` box of the same length and size
|
||||||
|
encoding, with its payload zeroed, so no later file offset moves."""
|
||||||
|
return data[start : start + 4] + b"free" + data[start + 8 : payload_off] + bytes(end - payload_off)
|
||||||
|
|
||||||
|
|
||||||
def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
|
def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
|
||||||
"""Return ``(cleaned_bytes, stripped_count)`` with AI-provenance boxes removed.
|
"""Return ``(cleaned_bytes, stripped_count)`` with AI-provenance boxes blanked.
|
||||||
|
|
||||||
Walks top-level boxes and drops:
|
Walks top-level boxes and turns each of these into a same-length ``free`` box
|
||||||
|
with a zeroed payload:
|
||||||
- any ``uuid`` box whose UUID equals ``C2PA_UUID`` (a C2PA manifest);
|
- any ``uuid`` box whose UUID equals ``C2PA_UUID`` (a C2PA manifest);
|
||||||
- any ``uuid`` box whose payload carries an AI-label marker (an XMP packet
|
- any ``uuid`` box whose payload carries an AI-label marker (an XMP packet
|
||||||
with a TC260 / IPTC / IPTC-2025.1 AI field -- caught by content, not by the
|
with a TC260 / IPTC / IPTC-2025.1 AI field -- caught by content, not by the
|
||||||
@@ -671,10 +754,15 @@ def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
|
|||||||
non-AI XMP intact);
|
non-AI XMP intact);
|
||||||
- any ``jumb`` box (JPEG-XL JUMBF container).
|
- any ``jumb`` box (JPEG-XL JUMBF container).
|
||||||
|
|
||||||
All other boxes (incl. ``mdat`` / codestream) are emitted verbatim, so pixel
|
The box is blanked rather than dropped because HEIF ``iloc`` and MP4
|
||||||
and audio data is preserved bit-for-bit. Non-ISOBMFF input is returned
|
``stco``/``co64`` locate the coded data by absolute file offset: dropping a
|
||||||
|
box that precedes ``mdat`` moved the image without rewriting them, and a
|
||||||
|
phone HEIC cleaned that way no longer decoded (2026-10-13). This matches
|
||||||
|
:func:`strip_isobmff_media_file`. All other boxes (incl. ``mdat`` /
|
||||||
|
codestream) are emitted verbatim, so pixel and audio data is preserved
|
||||||
|
bit-for-bit. Non-ISOBMFF input is returned
|
||||||
unchanged. Despite the name this also covers MP4/MOV/M4A video and audio
|
unchanged. Despite the name this also covers MP4/MOV/M4A video and audio
|
||||||
(all ISOBMFF). NOTE: this drops only top-level boxes. AI metadata stored as an
|
(all ISOBMFF). NOTE: this blanks only top-level boxes. AI metadata stored as an
|
||||||
*item inside the ``meta`` box* (typical for AVIF/HEIF) is handled separately and
|
*item inside the ``meta`` box* (typical for AVIF/HEIF) is handled separately and
|
||||||
in place (same length, no offset rewrite): AI-label XMP by
|
in place (same length, no offset rewrite): AI-label XMP by
|
||||||
:func:`blank_ai_xmp_packets`, and AI-generator tokens in an ``Exif`` item by
|
:func:`blank_ai_xmp_packets`, and AI-generator tokens in an ``Exif`` item by
|
||||||
@@ -688,17 +776,16 @@ def strip_c2pa_boxes(data: bytes) -> tuple[bytes, int]:
|
|||||||
consumed = 0
|
consumed = 0
|
||||||
for start, end, box_type, payload_off in _iter_top_level_boxes(data):
|
for start, end, box_type, payload_off in _iter_top_level_boxes(data):
|
||||||
consumed = end
|
consumed = end
|
||||||
|
provenance = box_type == b"jumb"
|
||||||
if box_type == b"uuid":
|
if box_type == b"uuid":
|
||||||
# uuid boxes carry the 16-byte UUID immediately after the type.
|
# uuid boxes carry the 16-byte UUID immediately after the type.
|
||||||
is_c2pa = payload_off + 16 <= end and data[payload_off : payload_off + 16] == C2PA_UUID
|
is_c2pa = payload_off + 16 <= end and data[payload_off : payload_off + 16] == C2PA_UUID
|
||||||
has_ai_label = any(marker in data[payload_off:end] for marker in _AI_LABEL_MARKERS)
|
provenance = is_c2pa or any(marker in data[payload_off:end] for marker in _AI_LABEL_MARKERS)
|
||||||
if is_c2pa or has_ai_label:
|
if provenance:
|
||||||
stripped += 1
|
|
||||||
continue
|
|
||||||
elif box_type == b"jumb":
|
|
||||||
stripped += 1
|
stripped += 1
|
||||||
continue
|
out.extend(_blank_box(data, start, end, payload_off))
|
||||||
out.extend(data[start:end])
|
else:
|
||||||
|
out.extend(data[start:end])
|
||||||
|
|
||||||
# Fail-safe: the walker returns early on a malformed box (bad size, or a box
|
# Fail-safe: the walker returns early on a malformed box (bad size, or a box
|
||||||
# that runs past EOF), so anything after it was never visited. Emitting `out`
|
# that runs past EOF), so anything after it was never visited. Emitting `out`
|
||||||
|
|||||||
@@ -47,6 +47,12 @@ _LATENT_GRID = 8
|
|||||||
# The reference graph loads the Qwen LoRA at 0.8; carrying that number to a different
|
# The reference graph loads the Qwen LoRA at 0.8; carrying that number to a different
|
||||||
# LoRA on a different architecture would be imitation, not parity.
|
# LoRA on a different architecture would be imitation, not parity.
|
||||||
SDXL_STEPS = 4
|
SDXL_STEPS = 4
|
||||||
|
# Below this floor the global stack streams its weights from CPU layer by layer; at or
|
||||||
|
# above it the stack stays resident. The fp16 UNet, Canny ControlNet, both text
|
||||||
|
# encoders and the VAE are ~8.8 GiB of weights, and the UNet and the ControlNet run
|
||||||
|
# together on every step, so model-level offload cannot get under an 8 GiB card.
|
||||||
|
# Benchmark in docs/module-internals.md, "CPU offload".
|
||||||
|
RESIDENT_SDXL_GLOBAL_MIN_VRAM_GIB = 12.0
|
||||||
|
|
||||||
|
|
||||||
def sdxl_target_size(width: int, height: int) -> tuple[int, int]:
|
def sdxl_target_size(width: int, height: int) -> tuple[int, int]:
|
||||||
@@ -54,6 +60,17 @@ def sdxl_target_size(width: int, height: int) -> tuple[int, int]:
|
|||||||
return _target_size(width, height, _LATENT_GRID)
|
return _target_size(width, height, _LATENT_GRID)
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_sdxl_global_residency(
|
||||||
|
requested: bool | None,
|
||||||
|
*,
|
||||||
|
total_memory_gib: float,
|
||||||
|
) -> bool:
|
||||||
|
"""Keep the SDXL global stack resident when explicitly requested or safely sized."""
|
||||||
|
if requested is not None:
|
||||||
|
return requested
|
||||||
|
return total_memory_gib >= RESIDENT_SDXL_GLOBAL_MIN_VRAM_GIB
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SdxlZImagePipeline(TwoStageZImagePipeline):
|
class SdxlZImagePipeline(TwoStageZImagePipeline):
|
||||||
"""Lazy runtime for the SDXL global stage plus the inherited face stage."""
|
"""Lazy runtime for the SDXL global stage plus the inherited face stage."""
|
||||||
@@ -64,6 +81,12 @@ class SdxlZImagePipeline(TwoStageZImagePipeline):
|
|||||||
super().__post_init__()
|
super().__post_init__()
|
||||||
self._sdxl_pipe: Any = None
|
self._sdxl_pipe: Any = None
|
||||||
|
|
||||||
|
def _keep_global_models_resident(self) -> bool:
|
||||||
|
return resolve_sdxl_global_residency(
|
||||||
|
self.keep_global_models_on_device,
|
||||||
|
total_memory_gib=self._total_vram_gib(),
|
||||||
|
)
|
||||||
|
|
||||||
def _load_global(self) -> Any:
|
def _load_global(self) -> Any:
|
||||||
if self._sdxl_pipe is not None:
|
if self._sdxl_pipe is not None:
|
||||||
return self._sdxl_pipe
|
return self._sdxl_pipe
|
||||||
@@ -89,11 +112,19 @@ class SdxlZImagePipeline(TwoStageZImagePipeline):
|
|||||||
variant="fp16",
|
variant="fp16",
|
||||||
add_watermarker=False,
|
add_watermarker=False,
|
||||||
**token,
|
**token,
|
||||||
).to(self.device)
|
)
|
||||||
|
resident = self._keep_global_models_resident()
|
||||||
|
if resident:
|
||||||
|
pipe = pipe.to(self.device)
|
||||||
pipe.load_lora_weights(hf_hub_download(SDXL_LIGHTNING_MODEL_ID, SDXL_LIGHTNING_PATTERN, **token))
|
pipe.load_lora_weights(hf_hub_download(SDXL_LIGHTNING_MODEL_ID, SDXL_LIGHTNING_PATTERN, **token))
|
||||||
pipe.fuse_lora()
|
pipe.fuse_lora()
|
||||||
# SDXL-Lightning is distilled against trailing timestep spacing.
|
# SDXL-Lightning is distilled against trailing timestep spacing.
|
||||||
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
|
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config, timestep_spacing="trailing")
|
||||||
|
if not resident:
|
||||||
|
# After the fuse: the offload hooks keep their own CPU copy of each weight
|
||||||
|
# and stream that copy, so fusing later would never reach what runs.
|
||||||
|
self._progress("Streaming the SDXL stack from CPU (card below the residency floor)...")
|
||||||
|
pipe.enable_sequential_cpu_offload(device=self.device, gpu_id=torch.cuda.current_device())
|
||||||
self._sdxl_pipe = pipe
|
self._sdxl_pipe = pipe
|
||||||
return pipe
|
return pipe
|
||||||
|
|
||||||
|
|||||||
@@ -823,7 +823,7 @@ class TwoStageZImagePipeline:
|
|||||||
torch_dtype=self._face_stage_dtype(),
|
torch_dtype=self._face_stage_dtype(),
|
||||||
**kwargs,
|
**kwargs,
|
||||||
).to(self.device)
|
).to(self.device)
|
||||||
model.eval()
|
model.train(False)
|
||||||
self._sam_model = model
|
self._sam_model = model
|
||||||
self._sam_processor = processor
|
self._sam_processor = processor
|
||||||
return model, processor
|
return model, processor
|
||||||
|
|||||||
@@ -2,11 +2,15 @@
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import os
|
||||||
|
import secrets
|
||||||
from typing import TYPE_CHECKING
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
from remove_ai_watermarks._internal.constants import JPEG_SUFFIXES, SUPPORTED_FORMATS
|
from remove_ai_watermarks._internal.constants import JPEG_SUFFIXES, SUPPORTED_FORMATS
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Generator
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
@@ -22,3 +26,19 @@ def get_image_format(file_path: Path) -> str:
|
|||||||
inputs therefore use its PNG fallback, matching the established API contract.
|
inputs therefore use its PNG fallback, matching the established API contract.
|
||||||
"""
|
"""
|
||||||
return "JPEG" if file_path.suffix.casefold() in JPEG_SUFFIXES else "PNG"
|
return "JPEG" if file_path.suffix.casefold() in JPEG_SUFFIXES else "PNG"
|
||||||
|
|
||||||
|
|
||||||
|
@contextlib.contextmanager
|
||||||
|
def staged_sibling(output: Path) -> Generator[Path]:
|
||||||
|
"""Yield an unused sibling of ``output``, published over it only if the block succeeds.
|
||||||
|
|
||||||
|
A strip that leaves AI metadata behind raises inside the block, so nothing reaches
|
||||||
|
``output``. The name is not pre-created (``mkstemp`` would), so the writer gives the
|
||||||
|
published file the default permissions rather than ``mkstemp``'s 0600.
|
||||||
|
"""
|
||||||
|
staged = output.with_name(f".{output.stem}-{secrets.token_hex(8)}{output.suffix}")
|
||||||
|
try:
|
||||||
|
yield staged
|
||||||
|
os.replace(staged, output)
|
||||||
|
finally:
|
||||||
|
staged.unlink(missing_ok=True)
|
||||||
@@ -63,13 +63,13 @@ PROFILE_ADAPTIVE_POLISH = {
|
|||||||
|
|
||||||
# Which profiles let ``--cpu-offload`` reach their GLOBAL stack. The face stage
|
# Which profiles let ``--cpu-offload`` reach their GLOBAL stack. The face stage
|
||||||
# honours the flag everywhere, because residency lives on the shared base, but the
|
# honours the flag everywhere, because residency lives on the shared base, but the
|
||||||
# global stack is each profile's own: only qwen-zimage streams it, through
|
# global stack is each profile's own: qwen-zimage streams it through
|
||||||
# ``_qwen_vram_config``. chroma-zimage and sdxl-zimage load theirs with a plain
|
# ``_qwen_vram_config`` and sdxl-zimage through Diffusers sequential offload.
|
||||||
# ``.to(device)``, so on a card below the face-stage residency floor the flag has
|
# chroma-zimage loads its stack with a plain ``.to(device)``, so on a card below the
|
||||||
# nothing left to change and the run behaves as if it were absent. Data rather than
|
# face-stage residency floor the flag has nothing left to change there and the run
|
||||||
# a subclass attribute, because the caller that must warn cannot import a profile
|
# behaves as if it were absent. Data rather than a subclass attribute, because the
|
||||||
# module without pulling in Diffusers.
|
# caller that must warn cannot import a profile module without pulling in Diffusers.
|
||||||
GLOBAL_OFFLOAD_PROFILES = frozenset({QWEN_ZIMAGE_PROFILE})
|
GLOBAL_OFFLOAD_PROFILES = frozenset({QWEN_ZIMAGE_PROFILE, SDXL_ZIMAGE_PROFILE})
|
||||||
|
|
||||||
SDXL_LIGHTNING_MODEL_ID = "ByteDance/SDXL-Lightning"
|
SDXL_LIGHTNING_MODEL_ID = "ByteDance/SDXL-Lightning"
|
||||||
SDXL_LIGHTNING_PATTERN = "sdxl_lightning_4step_lora.safetensors"
|
SDXL_LIGHTNING_PATTERN = "sdxl_lightning_4step_lora.safetensors"
|
||||||
|
|||||||
@@ -175,9 +175,9 @@ class WatermarkRemover:
|
|||||||
return
|
return
|
||||||
logger.warning(
|
logger.warning(
|
||||||
"--cpu-offload does not reach the global stack of the '%s' pipeline: only "
|
"--cpu-offload does not reach the global stack of the '%s' pipeline: only "
|
||||||
"qwen-zimage streams its global model between CUDA calls. The face stage "
|
"qwen-zimage and sdxl-zimage stream their global model between CUDA calls. "
|
||||||
"still honours the flag, so below the face-stage residency floor this run "
|
"The face stage still honours the flag, so below the face-stage residency "
|
||||||
"behaves as if --cpu-offload were absent.",
|
"floor this run behaves as if --cpu-offload were absent.",
|
||||||
self.model_profile,
|
self.model_profile,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -212,14 +212,18 @@ class WatermarkRemover:
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from remove_ai_watermarks import image_io
|
from remove_ai_watermarks import image_io
|
||||||
|
from remove_ai_watermarks._internal.utils import staged_sibling
|
||||||
|
from remove_ai_watermarks.metadata import strip_written_output
|
||||||
|
|
||||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
bgr = np.ascontiguousarray(np.asarray(image.convert("RGB"))[:, :, ::-1])
|
bgr = np.ascontiguousarray(np.asarray(image.convert("RGB"))[:, :, ::-1])
|
||||||
if not image_io.imwrite(str(output_path), bgr, display_tags_from=display_tags_from):
|
# Staged and verified, so a marker that survives fails the run with nothing
|
||||||
image.save(output_path)
|
# published at output_path.
|
||||||
from remove_ai_watermarks.metadata import remove_ai_metadata
|
with staged_sibling(output_path) as staged:
|
||||||
|
if not image_io.imwrite(str(staged), bgr, display_tags_from=display_tags_from):
|
||||||
remove_ai_metadata(output_path, output_path, keep_standard=True)
|
image.save(staged)
|
||||||
|
if leftover := strip_written_output(staged):
|
||||||
|
raise RuntimeError(f"AI metadata survived the strip: {', '.join(sorted(leftover))}")
|
||||||
|
|
||||||
def remove_watermark(
|
def remove_watermark(
|
||||||
self,
|
self,
|
||||||
|
|||||||
@@ -181,6 +181,7 @@ class TextMarkDetection:
|
|||||||
# ``footprint_mask`` reuses a threaded detection only when it matches the STRICT
|
# ``footprint_mask`` reuses a threaded detection only when it matches the STRICT
|
||||||
# level its own re-detect would have used -- see TextMarkEngine._strict_detection.
|
# level its own re-detect would have used -- see TextMarkEngine._strict_detection.
|
||||||
provenance: bool = False
|
provenance: bool = False
|
||||||
|
template_asset: str | None = None
|
||||||
|
|
||||||
|
|
||||||
def best_detection(*detections: TextMarkDetection) -> TextMarkDetection:
|
def best_detection(*detections: TextMarkDetection) -> TextMarkDetection:
|
||||||
@@ -665,7 +666,7 @@ class TextMarkEngine:
|
|||||||
def _verdict(self, scan: TextMarkScan, *, provenance: bool) -> TextMarkDetection:
|
def _verdict(self, scan: TextMarkScan, *, provenance: bool) -> TextMarkDetection:
|
||||||
"""Apply the trust-level-dependent tail to a scan, as a fresh result object."""
|
"""Apply the trust-level-dependent tail to a scan, as a fresh result object."""
|
||||||
c = self.config
|
c = self.config
|
||||||
det = TextMarkDetection(provenance=provenance)
|
det = TextMarkDetection(provenance=provenance, template_asset=c.asset_name)
|
||||||
if scan.loc is None or scan.box is None:
|
if scan.loc is None or scan.box is None:
|
||||||
return det
|
return det
|
||||||
det.region = scan.loc.bbox
|
det.region = scan.loc.bbox
|
||||||
|
|||||||
@@ -0,0 +1,219 @@
|
|||||||
|
"""Combine independently gated text layouts and retain their mask geometry."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
import cv2
|
||||||
|
|
||||||
|
from remove_ai_watermarks._text_mark_engine import (
|
||||||
|
BottomRightAnchoredEngine,
|
||||||
|
TextMarkConfig,
|
||||||
|
TextMarkDetection,
|
||||||
|
TextMarkEngine,
|
||||||
|
TextMarkLocation,
|
||||||
|
best_detection,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from numpy.typing import NDArray
|
||||||
|
|
||||||
|
|
||||||
|
class TextMarkVariantsMixin(TextMarkEngine):
|
||||||
|
"""Add independently configured layouts to an existing text-mark engine."""
|
||||||
|
|
||||||
|
_variants: tuple[TextMarkEngine, ...] = ()
|
||||||
|
config: TextMarkConfig
|
||||||
|
|
||||||
|
def detect_both(self, image: NDArray[Any] | None) -> tuple[TextMarkDetection, TextMarkDetection]:
|
||||||
|
pairs = [super().detect_both(image), *(engine.detect_both(image) for engine in self._variants)]
|
||||||
|
return best_detection(*(pair[0] for pair in pairs)), best_detection(*(pair[1] for pair in pairs))
|
||||||
|
|
||||||
|
def detect(self, image: NDArray[Any], *, provenance: bool = False) -> TextMarkDetection:
|
||||||
|
return self.detect_both(image)[int(provenance)]
|
||||||
|
|
||||||
|
def variant_engine(self, detection: TextMarkDetection) -> TextMarkEngine:
|
||||||
|
for engine in self._variants:
|
||||||
|
if engine.config.asset_name == detection.template_asset:
|
||||||
|
return engine
|
||||||
|
return self
|
||||||
|
|
||||||
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
dilate: int | None = None,
|
||||||
|
detection: TextMarkDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
if image is None or image.size == 0 or force:
|
||||||
|
return super().footprint_mask(image, force=force, dilate=dilate, detection=detection)
|
||||||
|
det = detection if detection is not None else self.detect(image)
|
||||||
|
engine = self.variant_engine(det)
|
||||||
|
if engine is self:
|
||||||
|
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
|
return engine.footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
|
|
||||||
|
|
||||||
|
class GlyphTextMarkEngine(TextMarkEngine):
|
||||||
|
"""Use the accepted template's aligned strokes instead of a textured ROI."""
|
||||||
|
|
||||||
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
dilate: int | None = None,
|
||||||
|
detection: TextMarkDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
from remove_ai_watermarks._text_mark_engine import load_alpha_template
|
||||||
|
|
||||||
|
if force:
|
||||||
|
return super().footprint_mask(image, force=True, dilate=dilate, detection=detection)
|
||||||
|
if image is None or image.size == 0:
|
||||||
|
return None
|
||||||
|
det = detection if detection is not None else self.detect(image)
|
||||||
|
alpha = load_alpha_template(self.config.asset_name)
|
||||||
|
if alpha is None or not det.detected:
|
||||||
|
return None
|
||||||
|
return self._aligned_alpha_mask(
|
||||||
|
image,
|
||||||
|
det,
|
||||||
|
alpha,
|
||||||
|
alpha_floor=0.05,
|
||||||
|
dilate=2 if dilate is None else max(0, dilate),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class BottomRightGlyphTextMarkEngine(GlyphTextMarkEngine, BottomRightAnchoredEngine):
|
||||||
|
"""Glyph-mask variant whose accepted match must hug the bottom-right corner."""
|
||||||
|
|
||||||
|
_ANCHOR_MAX_RIGHT = 0.04
|
||||||
|
_ANCHOR_MAX_BOTTOM = 0.04
|
||||||
|
|
||||||
|
def _ladder_best(
|
||||||
|
self, image: NDArray[Any], loc: TextMarkLocation
|
||||||
|
) -> tuple[float, tuple[int, int, int, int] | None]:
|
||||||
|
"""Find the strongest match among placements that satisfy the anchor."""
|
||||||
|
response = self._detect_response(image, loc)
|
||||||
|
silhouette = self._glyph_silhouette()
|
||||||
|
if response is None or silhouette is None:
|
||||||
|
return 0.0, None
|
||||||
|
frame_h, frame_w = image.shape[:2]
|
||||||
|
anchor_base = min(frame_h, frame_w)
|
||||||
|
base = self.scale_base(image)
|
||||||
|
best_score = 0.0
|
||||||
|
best_box: tuple[int, int, int, int] | None = None
|
||||||
|
for scale in self.config.ladder:
|
||||||
|
glyph_w = max(self.config.min_gw, int(self.config.alpha_width_frac * base * scale))
|
||||||
|
glyph_h = max(4, int(self.config.alpha_height_frac * base * scale))
|
||||||
|
if glyph_w >= response.shape[1] or glyph_h >= response.shape[0]:
|
||||||
|
continue
|
||||||
|
template = cv2.resize(silhouette, (glyph_w, glyph_h), interpolation=cv2.INTER_AREA)
|
||||||
|
scores = cv2.matchTemplate(response, template, cv2.TM_CCOEFF_NORMED)
|
||||||
|
min_x = max(0, round(frame_w - loc.x - glyph_w - self._ANCHOR_MAX_RIGHT * anchor_base))
|
||||||
|
max_x = min(scores.shape[1] - 1, frame_w - loc.x - glyph_w)
|
||||||
|
min_y = max(0, round(frame_h - loc.y - glyph_h - self._ANCHOR_MAX_BOTTOM * anchor_base))
|
||||||
|
max_y = min(scores.shape[0] - 1, frame_h - loc.y - glyph_h)
|
||||||
|
if min_x > max_x or min_y > max_y:
|
||||||
|
continue
|
||||||
|
anchored = scores[min_y : max_y + 1, min_x : max_x + 1]
|
||||||
|
_, score, _, point = cv2.minMaxLoc(anchored)
|
||||||
|
if score > best_score:
|
||||||
|
x, y = min_x + int(point[0]), min_y + int(point[1])
|
||||||
|
best_score = float(score)
|
||||||
|
best_box = (x, y, x + glyph_w - 1, y + glyph_h - 1)
|
||||||
|
return best_score, best_box
|
||||||
|
|
||||||
|
|
||||||
|
def _box_footprint(
|
||||||
|
engine: GlyphTextMarkEngine,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool,
|
||||||
|
dilate: int | None,
|
||||||
|
detection: TextMarkDetection | None,
|
||||||
|
pad_left_height: float,
|
||||||
|
pad_right_height: float,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
"""Mask the full winning text rectangle for outlined or stacked layouts."""
|
||||||
|
if image is None or image.size == 0 or force:
|
||||||
|
return GlyphTextMarkEngine.footprint_mask(
|
||||||
|
engine,
|
||||||
|
image,
|
||||||
|
force=force,
|
||||||
|
dilate=dilate,
|
||||||
|
detection=detection,
|
||||||
|
)
|
||||||
|
det = detection if detection is not None else engine.detect(image)
|
||||||
|
if not det.detected or det.match_box is None:
|
||||||
|
return None
|
||||||
|
from remove_ai_watermarks import region_eraser
|
||||||
|
|
||||||
|
loc = engine.locate(image)
|
||||||
|
x0, y0, x1, y1 = det.match_box
|
||||||
|
extra_left = round((y1 - y0 + 1) * pad_left_height)
|
||||||
|
extra_right = round((y1 - y0 + 1) * pad_right_height)
|
||||||
|
pad = 2 if dilate is None else max(0, dilate)
|
||||||
|
return region_eraser.boxes_to_mask(
|
||||||
|
image.shape[:2],
|
||||||
|
[
|
||||||
|
(
|
||||||
|
loc.x + x0 - extra_left,
|
||||||
|
loc.y + y0,
|
||||||
|
x1 - x0 + 1 + extra_left + extra_right,
|
||||||
|
y1 - y0 + 1,
|
||||||
|
)
|
||||||
|
],
|
||||||
|
dilate=pad,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class BoxTextMarkEngine(GlyphTextMarkEngine):
|
||||||
|
"""Text detector with a solid detector-aligned removal footprint."""
|
||||||
|
|
||||||
|
_BOX_PAD_LEFT_HEIGHT = 0.0
|
||||||
|
_BOX_PAD_RIGHT_HEIGHT = 0.0
|
||||||
|
|
||||||
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
dilate: int | None = None,
|
||||||
|
detection: TextMarkDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
return _box_footprint(
|
||||||
|
self,
|
||||||
|
image,
|
||||||
|
force=force,
|
||||||
|
dilate=dilate,
|
||||||
|
detection=detection,
|
||||||
|
pad_left_height=self._BOX_PAD_LEFT_HEIGHT,
|
||||||
|
pad_right_height=self._BOX_PAD_RIGHT_HEIGHT,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class BottomRightBoxTextMarkEngine(BottomRightGlyphTextMarkEngine):
|
||||||
|
"""Anchored text detector with a solid detector-aligned footprint."""
|
||||||
|
|
||||||
|
_BOX_PAD_LEFT_HEIGHT = 0.0
|
||||||
|
_BOX_PAD_RIGHT_HEIGHT = 0.0
|
||||||
|
|
||||||
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
dilate: int | None = None,
|
||||||
|
detection: TextMarkDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
return _box_footprint(
|
||||||
|
self,
|
||||||
|
image,
|
||||||
|
force=force,
|
||||||
|
dilate=dilate,
|
||||||
|
detection=detection,
|
||||||
|
pad_left_height=self._BOX_PAD_LEFT_HEIGHT,
|
||||||
|
pad_right_height=self._BOX_PAD_RIGHT_HEIGHT,
|
||||||
|
)
|
||||||
@@ -132,33 +132,35 @@ def _write_visible_result(
|
|||||||
if not removed and not write_noop:
|
if not removed and not write_noop:
|
||||||
return
|
return
|
||||||
|
|
||||||
|
import shutil
|
||||||
|
|
||||||
from remove_ai_watermarks import image_io
|
from remove_ai_watermarks import image_io
|
||||||
|
from remove_ai_watermarks._internal.utils import staged_sibling
|
||||||
|
|
||||||
out_path = Path(output)
|
out_path = Path(output)
|
||||||
out_path.parent.mkdir(parents=True, exist_ok=True)
|
out_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
source_path = loaded.path
|
source_path = loaded.path
|
||||||
if not removed and source_path is not None and source_path.suffix.lower() == out_path.suffix.lower():
|
# Staged, so a strip that leaves AI metadata behind publishes nothing -- the
|
||||||
# Copy the ORIGINAL bytes instead of lossily re-encoding a no-op. An in-place
|
# MetadataStripIncomplete contract.
|
||||||
# call needs no copy and would otherwise raise shutil.SameFileError.
|
with staged_sibling(out_path) as staged:
|
||||||
if source_path.resolve() != out_path.resolve():
|
if not removed and source_path is not None and source_path.suffix.lower() == out_path.suffix.lower():
|
||||||
import shutil
|
# Copy the ORIGINAL bytes instead of lossily re-encoding a no-op.
|
||||||
|
shutil.copyfile(source_path, staged)
|
||||||
shutil.copyfile(source_path, out_path)
|
|
||||||
else:
|
|
||||||
# imwrite is contractually NON-RAISING, so this bool is the only signal the file
|
# imwrite is contractually NON-RAISING, so this bool is the only signal the file
|
||||||
# was not created. Unchecked, the metadata strip below ran on a nonexistent path
|
# was not created. Unchecked, the metadata strip below ran on a nonexistent path
|
||||||
# and surfaced as a confusing "cannot read image <INPUT>" naming the OUTPUT path
|
# and surfaced as a confusing "cannot read image <INPUT>" naming the OUTPUT path
|
||||||
# (Tier E, 2026-07-20). Raise here so a library caller and the CLI both get an
|
# (Tier E, 2026-07-20). Raise here so a library caller and the CLI both get an
|
||||||
# accurate message about the write.
|
# accurate message about the write.
|
||||||
if not image_io.write_bgr_with_alpha(
|
elif not image_io.write_bgr_with_alpha(
|
||||||
out_path, result, loaded.alpha, display_tags_from=source_path, orientation_applied=False
|
staged, result, loaded.alpha, display_tags_from=source_path, orientation_applied=False
|
||||||
):
|
):
|
||||||
raise OSError(f"failed to write output (is the destination writable?): {out_path}")
|
raise OSError(f"failed to write output (is the destination writable?): {out_path}")
|
||||||
|
|
||||||
if strip_metadata:
|
if strip_metadata:
|
||||||
from remove_ai_watermarks import metadata
|
from remove_ai_watermarks.metadata import strip_written_output
|
||||||
|
|
||||||
metadata.remove_ai_metadata(out_path, out_path)
|
if leftover := strip_written_output(staged):
|
||||||
|
raise MetadataStripIncomplete(set(leftover))
|
||||||
|
|
||||||
|
|
||||||
def remove_visible(
|
def remove_visible(
|
||||||
@@ -461,7 +463,7 @@ def remove_all(
|
|||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
from remove_ai_watermarks import image_io, watermark_registry
|
from remove_ai_watermarks import image_io, watermark_registry
|
||||||
from remove_ai_watermarks.metadata import strip_and_verify
|
from remove_ai_watermarks.metadata import strip_written_output
|
||||||
|
|
||||||
def say(stage: str, detail: str) -> None:
|
def say(stage: str, detail: str) -> None:
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
@@ -516,8 +518,7 @@ def remove_all(
|
|||||||
# ── 3. AI metadata ──
|
# ── 3. AI metadata ──
|
||||||
# Read the pristine ORIGINAL for provenance above and the STAGED file here:
|
# Read the pristine ORIGINAL for provenance above and the STAGED file here:
|
||||||
# the visible pass has already dropped this file's C2PA.
|
# the visible pass has already dropped this file's C2PA.
|
||||||
_, leftover = strip_and_verify(staged, staged)
|
if leftover := strip_written_output(staged):
|
||||||
if leftover:
|
|
||||||
# Before the write, on purpose -- see MetadataStripIncomplete.
|
# Before the write, on purpose -- see MetadataStripIncomplete.
|
||||||
raise MetadataStripIncomplete(set(leftover))
|
raise MetadataStripIncomplete(set(leftover))
|
||||||
say("metadata", "stripped")
|
say("metadata", "stripped")
|
||||||
@@ -676,7 +677,9 @@ def remove_batch(
|
|||||||
from remove_ai_watermarks.optional_deps import pixels_available
|
from remove_ai_watermarks.optional_deps import pixels_available
|
||||||
|
|
||||||
src_dir, out_dir = Path(directory), Path(output_dir)
|
src_dir, out_dir = Path(directory), Path(output_dir)
|
||||||
out_dir.mkdir(parents=True, exist_ok=True)
|
sources = sorted(p for p in src_dir.iterdir() if is_supported_format(p))
|
||||||
|
if sources: # an empty run leaves no empty output directory behind
|
||||||
|
out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
def say(path: Path, stage: str, detail: str) -> None:
|
def say(path: Path, stage: str, detail: str) -> None:
|
||||||
if progress is not None:
|
if progress is not None:
|
||||||
@@ -685,7 +688,7 @@ def remove_batch(
|
|||||||
unavailable: list[Path] = []
|
unavailable: list[Path] = []
|
||||||
errors: list[tuple[Path, str]] = []
|
errors: list[tuple[Path, str]] = []
|
||||||
items: list[BatchItemResult] = []
|
items: list[BatchItemResult] = []
|
||||||
for img_path in sorted(p for p in src_dir.iterdir() if is_supported_format(p)):
|
for img_path in sources:
|
||||||
out_path = out_dir / img_path.name
|
out_path = out_dir / img_path.name
|
||||||
try:
|
try:
|
||||||
result = _run_batch_one(img_path, out_path, mode, backend, sensitivity, invisible, force, engine, say)
|
result = _run_batch_one(img_path, out_path, mode, backend, sensitivity, invisible, force, engine, say)
|
||||||
|
|||||||
|
After Width: | Height: | Size: 2.8 KiB |
|
After Width: | Height: | Size: 3.7 KiB |
|
After Width: | Height: | Size: 4.1 KiB |
|
After Width: | Height: | Size: 2.9 KiB |
|
After Width: | Height: | Size: 2.8 KiB |
|
After Width: | Height: | Size: 9.5 KiB |
|
After Width: | Height: | Size: 8.3 KiB |
|
After Width: | Height: | Size: 8.3 KiB |
@@ -467,7 +467,7 @@ def _load_runtime(device: Any, backend: ClassifierBackend = "torch") -> _Runtime
|
|||||||
probe = np.load(_weight_path(folder, PROBE_FILE))
|
probe = np.load(_weight_path(folder, PROBE_FILE))
|
||||||
detector = _build_detector()
|
detector = _build_detector()
|
||||||
detector.load_state_dict(torch.load(_weight_path(folder, DETECTOR_FILE), map_location="cpu", weights_only=True))
|
detector.load_state_dict(torch.load(_weight_path(folder, DETECTOR_FILE), map_location="cpu", weights_only=True))
|
||||||
detector.to(device).eval()
|
detector.to(device).train(False)
|
||||||
packed = torch.load(_weight_path(folder, PROVIDER_FILE), map_location="cpu", weights_only=True)
|
packed = torch.load(_weight_path(folder, PROVIDER_FILE), map_location="cpu", weights_only=True)
|
||||||
heads = {}
|
heads = {}
|
||||||
for name in PROVIDER_LABELS:
|
for name in PROVIDER_LABELS:
|
||||||
@@ -476,7 +476,7 @@ def _load_runtime(device: Any, backend: ClassifierBackend = "torch") -> _Runtime
|
|||||||
continue
|
continue
|
||||||
head = _build_provider_head()
|
head = _build_provider_head()
|
||||||
head.load_state_dict(packed[name])
|
head.load_state_dict(packed[name])
|
||||||
heads[name] = head.to(device).eval()
|
heads[name] = head.to(device).train(False)
|
||||||
ridge_threshold = float(probe["thr_oi_1pct"])
|
ridge_threshold = float(probe["thr_oi_1pct"])
|
||||||
mlp_threshold, provider_margin = _operating_point(folder)
|
mlp_threshold, provider_margin = _operating_point(folder)
|
||||||
log.info("loaded photo-classify freeze from %s", folder)
|
log.info("loaded photo-classify freeze from %s", folder)
|
||||||
|
|||||||
@@ -125,6 +125,27 @@ def _validate_image(path: Path) -> Path:
|
|||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def _unreadable_input_exit(source: Path, *, after_report: bool, to_stderr: bool = False) -> None:
|
||||||
|
"""Exit 1 when ``source`` is not a readable container; return otherwise.
|
||||||
|
|
||||||
|
The metadata scanners read raw bytes, so on bytes that are not an image or video
|
||||||
|
they find nothing, and "nothing" used to print as the same no-signal answer a
|
||||||
|
real file gets, with exit 0. ``after_report`` says whether a partial report was
|
||||||
|
already printed above. ``to_stderr`` keeps a ``--json`` stdout parseable.
|
||||||
|
"""
|
||||||
|
from remove_ai_watermarks.metadata import unreadable_reason
|
||||||
|
|
||||||
|
if (reason := unreadable_reason(source)) is None:
|
||||||
|
return
|
||||||
|
scanned = "The scan above is incomplete" if after_report else "Nothing was scanned"
|
||||||
|
message = f"Error: cannot read {source.name}: {reason}. {scanned}, not a no-signal result."
|
||||||
|
if to_stderr:
|
||||||
|
click.echo(message, err=True)
|
||||||
|
else:
|
||||||
|
console.print(f" {message}")
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
def _resolved_strength_for_display(
|
def _resolved_strength_for_display(
|
||||||
source: Path,
|
source: Path,
|
||||||
strength: float | None,
|
strength: float | None,
|
||||||
@@ -308,8 +329,9 @@ _cpu_offload_option = click.option(
|
|||||||
"Offload model components to CPU between CUDA calls instead of keeping the "
|
"Offload model components to CPU between CUDA calls instead of keeping the "
|
||||||
"whole pipeline in VRAM, at the cost of speed. Forces the face stack to "
|
"whole pipeline in VRAM, at the cost of speed. Forces the face stack to "
|
||||||
"offload instead of using automatic residency, on every profile. It reaches "
|
"offload instead of using automatic residency, on every profile. It reaches "
|
||||||
"the global stack of qwen-zimage only: chroma-zimage and sdxl-zimage keep "
|
"the global stack of qwen-zimage and sdxl-zimage, which also stream theirs "
|
||||||
"theirs resident either way."
|
"automatically on a card too small to hold it; chroma-zimage keeps its "
|
||||||
|
"global stack resident either way."
|
||||||
),
|
),
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -426,6 +448,8 @@ def _write_output_or_exit(
|
|||||||
bgr: NDArray[Any],
|
bgr: NDArray[Any],
|
||||||
alpha: NDArray[Any] | None,
|
alpha: NDArray[Any] | None,
|
||||||
display_source: Path | None = None,
|
display_source: Path | None = None,
|
||||||
|
*,
|
||||||
|
strip_metadata: bool = False,
|
||||||
) -> None:
|
) -> None:
|
||||||
"""Write the final image, or fail with a readable error instead of a traceback.
|
"""Write the final image, or fail with a readable error instead of a traceback.
|
||||||
|
|
||||||
@@ -437,11 +461,29 @@ def _write_output_or_exit(
|
|||||||
Regression: `tests/test_cli_robustness.py::TestFailedWriteIsReported`.
|
Regression: `tests/test_cli_robustness.py::TestFailedWriteIsReported`.
|
||||||
|
|
||||||
`display_source` is the file whose decode produced `bgr`; its ICC profile and EXIF
|
`display_source` is the file whose decode produced `bgr`; its ICC profile and EXIF
|
||||||
orientation ride along into the re-encoded output (issue #98)."""
|
orientation ride along into the re-encoded output (issue #98).
|
||||||
|
|
||||||
|
``strip_metadata`` strips and verifies AI metadata on a staged copy before it is
|
||||||
|
published, so a marker that survives exits 1 with nothing written. Both commands
|
||||||
|
using this used to swallow a failed strip unless ``--verbose`` was set."""
|
||||||
|
from remove_ai_watermarks._internal.utils import staged_sibling
|
||||||
|
from remove_ai_watermarks.metadata import strip_written_output
|
||||||
|
|
||||||
output.parent.mkdir(parents=True, exist_ok=True)
|
output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
if not image_io.write_bgr_with_alpha(output, bgr, alpha, display_tags_from=display_source):
|
with staged_sibling(output) as staged:
|
||||||
console.print(f" Error: failed to write output (is the destination writable?): {output}")
|
if not image_io.write_bgr_with_alpha(staged, bgr, alpha, display_tags_from=display_source):
|
||||||
raise SystemExit(1)
|
console.print(f" Error: failed to write output (is the destination writable?): {output}")
|
||||||
|
raise SystemExit(1)
|
||||||
|
if not strip_metadata:
|
||||||
|
return
|
||||||
|
try:
|
||||||
|
leftover = strip_written_output(staged)
|
||||||
|
except (OSError, RuntimeError, ValueError) as e:
|
||||||
|
console.print(f" Error: failed to strip AI metadata from {output}: {e}")
|
||||||
|
raise SystemExit(1) from e
|
||||||
|
if leftover:
|
||||||
|
console.print(f" Error: AI metadata survived in {output}: {', '.join(sorted(leftover))}")
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
def _no_visible_mark_exit(source: Path) -> NoReturn:
|
def _no_visible_mark_exit(source: Path) -> NoReturn:
|
||||||
@@ -510,6 +552,10 @@ def _no_visible_mark_exit(source: Path) -> NoReturn:
|
|||||||
# instead of treating an unchanged image as a completed removal.
|
# instead of treating an unchanged image as a completed removal.
|
||||||
EXIT_NO_INVISIBLE_SIGNAL = 2
|
EXIT_NO_INVISIBLE_SIGNAL = 2
|
||||||
|
|
||||||
|
# Also 2: a batch directory with no supported file processed nothing, which is a
|
||||||
|
# no-signal outcome a wrapper must not read as a cleaned directory.
|
||||||
|
EXIT_NOTHING_PROCESSED = EXIT_NO_INVISIBLE_SIGNAL
|
||||||
|
|
||||||
|
|
||||||
def _no_invisible_signal_exit(source: Path) -> NoReturn:
|
def _no_invisible_signal_exit(source: Path) -> NoReturn:
|
||||||
"""Explain why the diffusion scrub was skipped, then exit non-zero.
|
"""Explain why the diffusion scrub was skipped, then exit non-zero.
|
||||||
@@ -635,7 +681,6 @@ def _run_visible_auto(
|
|||||||
|
|
||||||
|
|
||||||
def _run_visible_explicit(
|
def _run_visible_explicit(
|
||||||
ctx: click.Context,
|
|
||||||
source: Path,
|
source: Path,
|
||||||
output: Path,
|
output: Path,
|
||||||
*,
|
*,
|
||||||
@@ -684,15 +729,7 @@ def _run_visible_explicit(
|
|||||||
raise SystemExit(1) from e
|
raise SystemExit(1) from e
|
||||||
elapsed = time.monotonic() - t0
|
elapsed = time.monotonic() - t0
|
||||||
|
|
||||||
_write_output_or_exit(output, result, alpha, display_source=source)
|
_write_output_or_exit(output, result, alpha, display_source=source, strip_metadata=strip_metadata)
|
||||||
if strip_metadata:
|
|
||||||
try:
|
|
||||||
from remove_ai_watermarks.metadata import remove_ai_metadata
|
|
||||||
|
|
||||||
remove_ai_metadata(output, output)
|
|
||||||
except Exception as e:
|
|
||||||
if ctx.obj.get("verbose"):
|
|
||||||
console.print(f" Warning: Failed to strip metadata: {e}")
|
|
||||||
|
|
||||||
size_kb = output.stat().st_size / 1024
|
size_kb = output.stat().st_size / 1024
|
||||||
console.print(f" Saved: {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
console.print(f" Saved: {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
||||||
@@ -712,10 +749,8 @@ def _run_visible_explicit(
|
|||||||
@_visible_backend_option
|
@_visible_backend_option
|
||||||
@_visible_sensitivity_option
|
@_visible_sensitivity_option
|
||||||
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
|
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
|
||||||
@click.pass_context
|
|
||||||
@_pixels_required
|
@_pixels_required
|
||||||
def cmd_visible(
|
def cmd_visible(
|
||||||
ctx: click.Context,
|
|
||||||
source: Path,
|
source: Path,
|
||||||
output: Path | None,
|
output: Path | None,
|
||||||
detect: bool,
|
detect: bool,
|
||||||
@@ -753,7 +788,6 @@ def cmd_visible(
|
|||||||
return
|
return
|
||||||
|
|
||||||
_run_visible_explicit(
|
_run_visible_explicit(
|
||||||
ctx,
|
|
||||||
source,
|
source,
|
||||||
output,
|
output,
|
||||||
detect=detect,
|
detect=detect,
|
||||||
@@ -794,10 +828,8 @@ def _parse_region(spec: str) -> tuple[int, int, int, int]:
|
|||||||
@click.option("--inpaint-method", type=click.Choice(["telea", "ns"]), default="telea", help="cv2 inpaint method.")
|
@click.option("--inpaint-method", type=click.Choice(["telea", "ns"]), default="telea", help="cv2 inpaint method.")
|
||||||
@click.option("--dilate", type=int, default=3, help="Grow the box by this many px before inpainting.")
|
@click.option("--dilate", type=int, default=3, help="Grow the box by this many px before inpainting.")
|
||||||
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
|
@click.option("--strip-metadata/--keep-metadata", default=True, help="Strip AI metadata from output.")
|
||||||
@click.pass_context
|
|
||||||
@_pixels_required
|
@_pixels_required
|
||||||
def cmd_erase(
|
def cmd_erase(
|
||||||
ctx: click.Context,
|
|
||||||
source: Path,
|
source: Path,
|
||||||
regions: tuple[str, ...],
|
regions: tuple[str, ...],
|
||||||
output: Path | None,
|
output: Path | None,
|
||||||
@@ -838,16 +870,7 @@ def cmd_erase(
|
|||||||
raise SystemExit(1) from e
|
raise SystemExit(1) from e
|
||||||
elapsed = time.monotonic() - t0
|
elapsed = time.monotonic() - t0
|
||||||
|
|
||||||
_write_output_or_exit(output, result, alpha, display_source=source)
|
_write_output_or_exit(output, result, alpha, display_source=source, strip_metadata=strip_metadata)
|
||||||
|
|
||||||
if strip_metadata:
|
|
||||||
try:
|
|
||||||
from remove_ai_watermarks.metadata import remove_ai_metadata
|
|
||||||
|
|
||||||
remove_ai_metadata(output, output)
|
|
||||||
except Exception as e:
|
|
||||||
if ctx.obj.get("verbose"):
|
|
||||||
console.print(f" Warning: Failed to strip metadata: {e}")
|
|
||||||
|
|
||||||
size_kb = output.stat().st_size / 1024
|
size_kb = output.stat().st_size / 1024
|
||||||
console.print(f" Erased {len(boxes)} region(s) -> {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
console.print(f" Erased {len(boxes)} region(s) -> {output} ({size_kb:.0f} KB, {elapsed:.2f}s)")
|
||||||
@@ -914,6 +937,11 @@ def cmd_invisible(
|
|||||||
source = _validate_image(source)
|
source = _validate_image(source)
|
||||||
if output is None:
|
if output is None:
|
||||||
output = source.with_stem(source.stem + "_clean")
|
output = source.with_stem(source.stem + "_clean")
|
||||||
|
# The scrub gate reads an undecodable file as "no target", which would exit 2
|
||||||
|
# (nothing to remove) for a file this command never looked at.
|
||||||
|
if image_io.imread(source) is None:
|
||||||
|
console.print(f"Error: Failed to read image: {source}")
|
||||||
|
raise SystemExit(1)
|
||||||
|
|
||||||
# An explicit --vendor wins over detection (see the option help) and implies the
|
# An explicit --vendor wins over detection (see the option help) and implies the
|
||||||
# scrub runs: naming the cohort asserts the pixel watermark is present, so the
|
# scrub runs: naming the cohort asserts the pixel watermark is present, so the
|
||||||
@@ -931,13 +959,17 @@ def cmd_invisible(
|
|||||||
def progress_cb(msg: str) -> None:
|
def progress_cb(msg: str) -> None:
|
||||||
console.print(f" {msg}")
|
console.print(f" {msg}")
|
||||||
|
|
||||||
engine = InvisibleEngine(
|
try:
|
||||||
pipeline=pipeline,
|
engine = InvisibleEngine(
|
||||||
hf_token=hf_token,
|
pipeline=pipeline,
|
||||||
progress_callback=progress_cb,
|
hf_token=hf_token,
|
||||||
controlnet_conditioning_scale=controlnet_scale,
|
progress_callback=progress_cb,
|
||||||
cpu_offload=cpu_offload,
|
controlnet_conditioning_scale=controlnet_scale,
|
||||||
)
|
cpu_offload=cpu_offload,
|
||||||
|
)
|
||||||
|
except (OSError, RuntimeError, ValueError) as exc: # e.g. no CUDA device
|
||||||
|
console.print(f" Error: {exc}")
|
||||||
|
raise SystemExit(1) from exc
|
||||||
|
|
||||||
# Detect the SynthID vendor from the ORIGINAL (before processing strips C2PA) so the
|
# Detect the SynthID vendor from the ORIGINAL (before processing strips C2PA) so the
|
||||||
# displayed and executed strength agree on the vendor-adaptive default. An explicit
|
# displayed and executed strength agree on the vendor-adaptive default. An explicit
|
||||||
@@ -1046,6 +1078,7 @@ def cmd_metadata(
|
|||||||
# accepts video/audio containers, so the image-format warning would misfire.
|
# accepts video/audio containers, so the image-format warning would misfire.
|
||||||
# click's `exists=True` on the argument already enforces the file exists.
|
# click's `exists=True` on the argument already enforces the file exists.
|
||||||
_banner()
|
_banner()
|
||||||
|
_unreadable_input_exit(source, after_report=False)
|
||||||
|
|
||||||
if check or (not remove):
|
if check or (not remove):
|
||||||
has_ai = has_ai_metadata(source)
|
has_ai = has_ai_metadata(source)
|
||||||
@@ -1058,14 +1091,13 @@ def cmd_metadata(
|
|||||||
# Remove
|
# Remove
|
||||||
try:
|
try:
|
||||||
out, leftover = strip_and_verify(source, output, keep_standard=keep_standard)
|
out, leftover = strip_and_verify(source, output, keep_standard=keep_standard)
|
||||||
except (OSError, ValueError) as e: # unreadable / truncated / non-image (PIL raises OSError subclasses)
|
except (OSError, RuntimeError, ValueError) as e: # unreadable input, or ffmpeg absent or failing
|
||||||
console.print(f" Error: cannot process {source.name}: {e}")
|
console.print(f" Error: cannot process {source.name}: {e}")
|
||||||
raise SystemExit(1) from e
|
raise SystemExit(1) from e
|
||||||
|
|
||||||
if leftover:
|
if leftover:
|
||||||
console.print(f" FAILED: {len(leftover)} AI metadata marker(s) survived in {out}")
|
console.print(f" FAILED: {len(leftover)} AI metadata marker(s) survived in {out}")
|
||||||
console.print(f" still present: {', '.join(sorted(leftover))}")
|
console.print(f" still present: {', '.join(sorted(leftover))}")
|
||||||
console.print(" the file could not be decoded, so it was copied through unchanged")
|
|
||||||
raise SystemExit(1)
|
raise SystemExit(1)
|
||||||
console.print(f" AI metadata stripped -> {out}")
|
console.print(f" AI metadata stripped -> {out}")
|
||||||
_print_metadata_not_a_clean_verdict()
|
_print_metadata_not_a_clean_verdict()
|
||||||
@@ -1434,6 +1466,9 @@ def cmd_video_batch(
|
|||||||
except (OSError, RuntimeError, ValueError) as e:
|
except (OSError, RuntimeError, ValueError) as e:
|
||||||
raise click.ClickException(str(e)) from e
|
raise click.ClickException(str(e)) from e
|
||||||
|
|
||||||
|
if not result.items:
|
||||||
|
console.print(f" No supported videos found in {directory}; nothing was processed")
|
||||||
|
raise SystemExit(EXIT_NOTHING_PROCESSED)
|
||||||
for item in result.items:
|
for item in result.items:
|
||||||
if item.error is not None:
|
if item.error is not None:
|
||||||
console.print(f" FAILED {item.source.name}: {item.error}")
|
console.print(f" FAILED {item.source.name}: {item.error}")
|
||||||
@@ -1482,6 +1517,7 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
|||||||
|
|
||||||
if as_json:
|
if as_json:
|
||||||
click.echo(json.dumps(asdict(report), default=str, indent=2))
|
click.echo(json.dumps(asdict(report), default=str, indent=2))
|
||||||
|
_unreadable_input_exit(source, after_report=True, to_stderr=True)
|
||||||
return
|
return
|
||||||
|
|
||||||
_banner()
|
_banner()
|
||||||
@@ -1528,6 +1564,7 @@ def cmd_identify(ctx: click.Context, source: Path, no_visible: bool, as_json: bo
|
|||||||
console.print("\n Caveats:")
|
console.print("\n Caveats:")
|
||||||
for c in report.caveats:
|
for c in report.caveats:
|
||||||
console.print(f" - {c}")
|
console.print(f" - {c}")
|
||||||
|
_unreadable_input_exit(source, after_report=True)
|
||||||
|
|
||||||
|
|
||||||
# ── Metadata-free photo classification (opt-in, never a tail of identify) ──
|
# ── Metadata-free photo classification (opt-in, never a tail of identify) ──
|
||||||
@@ -1818,13 +1855,15 @@ def cmd_batch(
|
|||||||
|
|
||||||
if output_dir is None:
|
if output_dir is None:
|
||||||
output_dir = directory.parent / (directory.name + "_clean")
|
output_dir = directory.parent / (directory.name + "_clean")
|
||||||
output_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
images = sorted(p for p in directory.iterdir() if is_supported_format(p))
|
images = sorted(p for p in directory.iterdir() if is_supported_format(p))
|
||||||
|
|
||||||
if not images:
|
if not images:
|
||||||
console.print(f"No supported images found in {directory}")
|
# Nothing processed is the no-signal code, not success: a wrapper must not read
|
||||||
return
|
# an empty run as a cleaned directory.
|
||||||
|
console.print(f"No supported images found in {directory}; nothing was processed")
|
||||||
|
raise SystemExit(EXIT_NOTHING_PROCESSED)
|
||||||
|
output_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
console.print(f" Found {len(images)} images in {directory}")
|
console.print(f" Found {len(images)} images in {directory}")
|
||||||
console.print(f" Output -> {output_dir}")
|
console.print(f" Output -> {output_dir}")
|
||||||
@@ -1853,6 +1892,11 @@ def cmd_batch(
|
|||||||
if ctx.obj.get("verbose"):
|
if ctx.obj.get("verbose"):
|
||||||
console.print(f" {img.name}: {stage}{f' {detail}' if detail else ''}")
|
console.print(f" {img.name}: {stage}{f' {detail}' if detail else ''}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
engine = _batch_engine(mode, invisible_options)
|
||||||
|
except (OSError, RuntimeError, ValueError) as e: # e.g. no CUDA device
|
||||||
|
console.print(f" Error: {e}")
|
||||||
|
raise SystemExit(1) from e
|
||||||
summary = api_remove_batch(
|
summary = api_remove_batch(
|
||||||
directory,
|
directory,
|
||||||
output_dir,
|
output_dir,
|
||||||
@@ -1861,7 +1905,7 @@ def cmd_batch(
|
|||||||
sensitivity=_parse_sensitivity(sensitivity),
|
sensitivity=_parse_sensitivity(sensitivity),
|
||||||
invisible=invisible_options,
|
invisible=invisible_options,
|
||||||
force=force,
|
force=force,
|
||||||
engine=_batch_engine(mode, invisible_options),
|
engine=engine,
|
||||||
progress=on_progress,
|
progress=on_progress,
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -1905,5 +1949,19 @@ def cmd_batch(
|
|||||||
raise SystemExit(1)
|
raise SystemExit(1)
|
||||||
|
|
||||||
|
|
||||||
|
# ── MCP server ──
|
||||||
|
@main.command("mcp")
|
||||||
|
def cmd_mcp() -> None:
|
||||||
|
"""Run a Model Context Protocol server on stdin/stdout.
|
||||||
|
|
||||||
|
Exposes identify, remove and batch as MCP tools for agent clients. Each tool
|
||||||
|
runs one command of this CLI, so results and exit codes match it. Start it as
|
||||||
|
`uvx remove-ai-watermarks mcp` from the client's server configuration.
|
||||||
|
"""
|
||||||
|
from remove_ai_watermarks.mcp_server import serve
|
||||||
|
|
||||||
|
serve()
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
main()
|
main()
|
||||||
@@ -0,0 +1,35 @@
|
|||||||
|
"""Detect the bottom-right Dola AI image disclosure, independently of video marks."""
|
||||||
|
|
||||||
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig
|
||||||
|
from remove_ai_watermarks._text_mark_variants import BoxTextMarkEngine
|
||||||
|
|
||||||
|
_CONFIG = TextMarkConfig(
|
||||||
|
name="Dola AI image disclosure",
|
||||||
|
asset_name="dola_alpha.png",
|
||||||
|
corner="br",
|
||||||
|
margin_floor=4,
|
||||||
|
width_frac=0.28,
|
||||||
|
height_frac=0.09,
|
||||||
|
margin_x_frac=0.002,
|
||||||
|
margin_bottom_frac=0.002,
|
||||||
|
max_saturation=255,
|
||||||
|
logo_min_luma=110,
|
||||||
|
tophat_delta=8,
|
||||||
|
morph_open_size=3,
|
||||||
|
detect_min_coverage=0.01,
|
||||||
|
detect_ncc_threshold=0.50,
|
||||||
|
alpha_width_frac=0.10,
|
||||||
|
alpha_height_frac=0.10 * 86 / 347,
|
||||||
|
detect_frontend="tophat",
|
||||||
|
scale_basis="width",
|
||||||
|
ladder=(0.95, 1.0, 1.05, 1.20, 1.26),
|
||||||
|
provenance_ncc_factor=1.0,
|
||||||
|
min_gw=24,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class DolaEngine(BoxTextMarkEngine):
|
||||||
|
"""Detect and localize the Dola AI image wordmark."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__(_CONFIG)
|
||||||
@@ -25,10 +25,12 @@ from __future__ import annotations
|
|||||||
import dataclasses
|
import dataclasses
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from remove_ai_watermarks import _text_mark_engine, image_io
|
from remove_ai_watermarks import _text_mark_engine, image_io
|
||||||
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine, TextMarkScan
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine, TextMarkScan
|
||||||
|
from remove_ai_watermarks._text_mark_variants import BottomRightBoxTextMarkEngine, TextMarkVariantsMixin
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from numpy.typing import NDArray
|
from numpy.typing import NDArray
|
||||||
@@ -117,6 +119,41 @@ _CONFIG = TextMarkConfig(
|
|||||||
_TINTED_ENGINE = TextMarkEngine(dataclasses.replace(_CONFIG, max_saturation=_TINTED_MAX_SATURATION))
|
_TINTED_ENGINE = TextMarkEngine(dataclasses.replace(_CONFIG, max_saturation=_TINTED_MAX_SATURATION))
|
||||||
|
|
||||||
|
|
||||||
|
class _OutlinedDoubaoEngine(BottomRightBoxTextMarkEngine):
|
||||||
|
# The outlined layout's provenance-relaxed NCC floor overlaps ordinary corner
|
||||||
|
# texture. Require bright neutral strokes and edge detail in the winning box
|
||||||
|
# to distinguish the label from unmarked corners. Metadata confirms
|
||||||
|
# Doubao as the producer, not that the export retained a visible label.
|
||||||
|
_MIN_BRIGHT_NEUTRAL_FRACTION = 0.29
|
||||||
|
_MIN_MEAN_LAPLACIAN = 10.0
|
||||||
|
_BOX_PAD_LEFT_HEIGHT = 1.2
|
||||||
|
_BOX_PAD_RIGHT_HEIGHT = 0.7
|
||||||
|
|
||||||
|
@classmethod
|
||||||
|
def _outline_pixels_ok(cls, patch: NDArray[Any]) -> bool:
|
||||||
|
if patch.size == 0:
|
||||||
|
return False
|
||||||
|
patch = image_io.to_bgr(patch)
|
||||||
|
gray = cv2.cvtColor(patch, cv2.COLOR_BGR2GRAY)
|
||||||
|
saturation = cv2.cvtColor(patch, cv2.COLOR_BGR2HSV)[:, :, 1]
|
||||||
|
bright_neutral = float(np.mean((gray > 160) & (saturation < 60)))
|
||||||
|
edge_detail = float(np.mean(np.abs(cv2.Laplacian(gray, cv2.CV_32F))))
|
||||||
|
return bright_neutral >= cls._MIN_BRIGHT_NEUTRAL_FRACTION and edge_detail >= cls._MIN_MEAN_LAPLACIAN
|
||||||
|
|
||||||
|
def _scan(self, image: NDArray[Any] | None) -> TextMarkScan:
|
||||||
|
scan = super()._scan(image)
|
||||||
|
if image is None or scan.loc is None or scan.match_box is None or scan.score is None:
|
||||||
|
return scan
|
||||||
|
x0, y0, x1, y1 = scan.match_box
|
||||||
|
patch = image[
|
||||||
|
scan.loc.y + y0 : scan.loc.y + y1 + 1,
|
||||||
|
scan.loc.x + x0 : scan.loc.x + x1 + 1,
|
||||||
|
]
|
||||||
|
if not self._outline_pixels_ok(patch):
|
||||||
|
return dataclasses.replace(scan, score=0.0)
|
||||||
|
return scan
|
||||||
|
|
||||||
|
|
||||||
def _alpha_template() -> NDArray[Any] | None:
|
def _alpha_template() -> NDArray[Any] | None:
|
||||||
"""The bundled Doubao alpha template (float [0,1]), or None."""
|
"""The bundled Doubao alpha template (float [0,1]), or None."""
|
||||||
return _text_mark_engine.load_alpha_template(_CONFIG.asset_name)
|
return _text_mark_engine.load_alpha_template(_CONFIG.asset_name)
|
||||||
@@ -132,11 +169,29 @@ def _template_match_score(box_mask: NDArray[Any], scale_base: int) -> float:
|
|||||||
return _text_mark_engine.template_match_score(box_mask, scale_base, _CONFIG)
|
return _text_mark_engine.template_match_score(box_mask, scale_base, _CONFIG)
|
||||||
|
|
||||||
|
|
||||||
class DoubaoEngine(TextMarkEngine):
|
class DoubaoEngine(TextMarkVariantsMixin):
|
||||||
"""Detect/localize the visible Doubao "豆包AI生成" watermark (locate -> mask; mask feeds the fill)."""
|
"""Detect/localize the visible Doubao "豆包AI生成" watermark (locate -> mask; mask feeds the fill)."""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
super().__init__(_CONFIG)
|
super().__init__(_CONFIG)
|
||||||
|
self._variants = (
|
||||||
|
_OutlinedDoubaoEngine(
|
||||||
|
dataclasses.replace(
|
||||||
|
_CONFIG,
|
||||||
|
asset_name="doubao_outline_alpha.png",
|
||||||
|
width_frac=0.28,
|
||||||
|
height_frac=0.13,
|
||||||
|
scale_basis="width",
|
||||||
|
alpha_width_frac=0.14,
|
||||||
|
alpha_height_frac=0.14 * 0.75 * 77 / 346,
|
||||||
|
max_saturation=255,
|
||||||
|
detect_frontend="contrast",
|
||||||
|
ladder=(0.95, 1.0, 1.05),
|
||||||
|
detect_ncc_threshold=0.50,
|
||||||
|
provenance_ncc_factor=0.25,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
def _scan(self, image: NDArray[Any] | None) -> TextMarkScan:
|
def _scan(self, image: NDArray[Any] | None) -> TextMarkScan:
|
||||||
base = super()._scan(image)
|
base = super()._scan(image)
|
||||||
@@ -179,12 +234,15 @@ class DoubaoEngine(TextMarkEngine):
|
|||||||
det = detection if detection is not None else self.detect(image)
|
det = detection if detection is not None else self.detect(image)
|
||||||
if not det.detected or det.match_box is None:
|
if not det.detected or det.match_box is None:
|
||||||
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
alpha = _alpha_template()
|
engine = self.variant_engine(det)
|
||||||
|
if engine is not self:
|
||||||
|
return engine.footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
|
alpha = _text_mark_engine.load_alpha_template(engine.config.asset_name)
|
||||||
if alpha is None:
|
if alpha is None:
|
||||||
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
|
|
||||||
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
||||||
return self._aligned_alpha_mask(
|
return engine._aligned_alpha_mask(
|
||||||
image,
|
image,
|
||||||
det,
|
det,
|
||||||
alpha,
|
alpha,
|
||||||
|
|||||||
@@ -957,7 +957,8 @@ def _integrity_clashes(
|
|||||||
clashes.append(
|
clashes.append(
|
||||||
"Conflicting AI-origin attributions from independent signals: "
|
"Conflicting AI-origin attributions from independent signals: "
|
||||||
+ " vs ".join(parts)
|
+ " vs ".join(parts)
|
||||||
+ " -- one provenance set was likely spoofed, transplanted, or laundered."
|
+ " -- these fields may describe different stages of an edit chain; "
|
||||||
|
"the available evidence does not establish which attribution describes the final pixels."
|
||||||
)
|
)
|
||||||
|
|
||||||
# Rule 2: a camera-capture C2PA device next to an AI-generation marker. Only
|
# Rule 2: a camera-capture C2PA device next to an AI-generation marker. Only
|
||||||
@@ -967,13 +968,15 @@ def _integrity_clashes(
|
|||||||
# AI edit in ONE manifest, so the AI vendor is named only from that same
|
# AI edit in ONE manifest, so the AI vendor is named only from that same
|
||||||
# manifest (C2PA issuer + SynthID provenance) -- a legitimate edit chain, not a
|
# manifest (C2PA issuer + SynthID provenance) -- a legitimate edit chain, not a
|
||||||
# spoof. An EXIF/XMP generator, IPTC field, TC260 AIGC label, or second
|
# spoof. An EXIF/XMP generator, IPTC field, TC260 AIGC label, or second
|
||||||
# manifest naming AI on a camera capture is the real laundering tell.
|
# manifest naming AI on a camera capture records a second stage whose relation
|
||||||
|
# to the capture cannot be resolved from these fields alone.
|
||||||
independent_ai_marker = any(grp != _C2PA_MANIFEST_SOURCE for grp in source.values())
|
independent_ai_marker = any(grp != _C2PA_MANIFEST_SOURCE for grp in source.values())
|
||||||
if camera_label and camera_has_ai_marker and independent_ai_marker:
|
if camera_label and camera_has_ai_marker and independent_ai_marker:
|
||||||
vendors = ", ".join(sorted(set(ai_vendors.values()))) or "present"
|
vendors = ", ".join(sorted(set(ai_vendors.values()))) or "present"
|
||||||
clashes.append(
|
clashes.append(
|
||||||
f"Camera-capture C2PA credentials ({camera_label}) coexist with AI-generation markers "
|
f"Camera-capture C2PA credentials ({camera_label}) coexist with AI-generation markers "
|
||||||
f"({vendors}) -- a genuine camera capture is not AI-generated, so the provenance is inconsistent."
|
f"({vendors}) -- these fields may describe a captured image followed by an AI edit; "
|
||||||
|
"the available evidence does not establish which stage describes the final pixels."
|
||||||
)
|
)
|
||||||
|
|
||||||
return clashes
|
return clashes
|
||||||
|
|||||||
@@ -172,7 +172,8 @@ class InvisibleEngine:
|
|||||||
cpu_offload: Offload model components to CPU between CUDA calls instead
|
cpu_offload: Offload model components to CPU between CUDA calls instead
|
||||||
of keeping the whole pipeline in VRAM, at the cost of speed. Forces
|
of keeping the whole pipeline in VRAM, at the cost of speed. Forces
|
||||||
the face stack to offload instead of using automatic residency, on
|
the face stack to offload instead of using automatic residency, on
|
||||||
every profile; reaches the global stack of qwen-zimage only.
|
every profile; reaches the global stack of qwen-zimage and
|
||||||
|
sdxl-zimage, not chroma-zimage.
|
||||||
CUDA only.
|
CUDA only.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
@@ -310,6 +311,15 @@ class InvisibleEngine:
|
|||||||
with Image.open(image_path) as opened:
|
with Image.open(image_path) as opened:
|
||||||
image = ImageOps.exif_transpose(opened)
|
image = ImageOps.exif_transpose(opened)
|
||||||
orig_size = image.size # (width, height)
|
orig_size = image.size # (width, height)
|
||||||
|
# Diffusion only processes RGB. Keep alpha at the original display geometry
|
||||||
|
# (after EXIF, before any resize) and rejoin it after all RGB postprocessing.
|
||||||
|
# Palette and colour-key PNGs can carry transparency without an A band.
|
||||||
|
alpha = (
|
||||||
|
image.convert("RGBA").getchannel("A")
|
||||||
|
if output_path.suffix.lower() in image_io.ALPHA_FORMATS
|
||||||
|
and ("A" in image.getbands() or "transparency" in image.info)
|
||||||
|
else None
|
||||||
|
)
|
||||||
# Full-res original, kept for the adaptive-polish detail target (image is
|
# Full-res original, kept for the adaptive-polish detail target (image is
|
||||||
# reassigned to the resized copy below; PIL resize returns a new object).
|
# reassigned to the resized copy below; PIL resize returns a new object).
|
||||||
reference_pil = image
|
reference_pil = image
|
||||||
@@ -368,7 +378,7 @@ class InvisibleEngine:
|
|||||||
# Route those outputs through the same final resize so --no-adaptive-polish does
|
# Route those outputs through the same final resize so --no-adaptive-polish does
|
||||||
# not silently change e.g. 1448x1086 into 1448x1080.
|
# not silently change e.g. 1448x1086 into 1448x1080.
|
||||||
needs_restore = target is not None or any(dimension % 8 for dimension in orig_size)
|
needs_restore = target is not None or any(dimension % 8 for dimension in orig_size)
|
||||||
if humanize > 0.0 or unsharp > 0.0 or adaptive_polish or needs_restore:
|
if humanize > 0.0 or unsharp > 0.0 or adaptive_polish or needs_restore or alpha is not None:
|
||||||
import cv2
|
import cv2
|
||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
@@ -389,7 +399,13 @@ class InvisibleEngine:
|
|||||||
progress=self._progress_callback,
|
progress=self._progress_callback,
|
||||||
)
|
)
|
||||||
|
|
||||||
if not image_io.imwrite(out_path, out_cv, display_tags_from=image_path, orientation_applied=True):
|
if not image_io.write_bgr_with_alpha(
|
||||||
|
out_path,
|
||||||
|
out_cv,
|
||||||
|
np.asarray(alpha) if alpha is not None else None,
|
||||||
|
display_tags_from=image_path,
|
||||||
|
orientation_applied=True,
|
||||||
|
):
|
||||||
raise OSError(f"failed to write postprocessed output: {out_path}")
|
raise OSError(f"failed to write postprocessed output: {out_path}")
|
||||||
|
|
||||||
return out_path
|
return out_path
|
||||||
|
|||||||
@@ -56,6 +56,7 @@ from dataclasses import replace
|
|||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
import cv2
|
import cv2
|
||||||
|
import numpy as np
|
||||||
|
|
||||||
from remove_ai_watermarks import _text_mark_engine
|
from remove_ai_watermarks import _text_mark_engine
|
||||||
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine
|
||||||
@@ -136,8 +137,12 @@ _CONFIG = TextMarkConfig(
|
|||||||
_LATIN_CONFIG = replace(
|
_LATIN_CONFIG = replace(
|
||||||
_CONFIG,
|
_CONFIG,
|
||||||
asset_name="kling_latin_alpha.png",
|
asset_name="kling_latin_alpha.png",
|
||||||
alpha_width_frac=0.095,
|
width_frac=0.22,
|
||||||
alpha_height_frac=0.028,
|
height_frac=0.07,
|
||||||
|
margin_x_frac=0.002,
|
||||||
|
margin_bottom_frac=0.002,
|
||||||
|
alpha_width_frac=0.14,
|
||||||
|
alpha_height_frac=0.032,
|
||||||
ladder=(0.9, 1.0, 1.1),
|
ladder=(0.9, 1.0, 1.1),
|
||||||
detect_ncc_threshold=0.40,
|
detect_ncc_threshold=0.40,
|
||||||
)
|
)
|
||||||
@@ -153,12 +158,37 @@ def _glyph_silhouette() -> NDArray[Any] | None:
|
|||||||
return _text_mark_engine.glyph_silhouette(_CONFIG.asset_name)
|
return _text_mark_engine.glyph_silhouette(_CONFIG.asset_name)
|
||||||
|
|
||||||
|
|
||||||
class KlingEngine(TextMarkEngine):
|
class _KlingTextEngine(TextMarkEngine):
|
||||||
|
"""Require glyph structure beyond a horizontal or vertical background edge."""
|
||||||
|
|
||||||
|
def _ladder_best(
|
||||||
|
self, image: NDArray[Any], loc: _text_mark_engine.TextMarkLocation
|
||||||
|
) -> tuple[float, tuple[int, int, int, int] | None]:
|
||||||
|
score, box = super()._ladder_best(image, loc)
|
||||||
|
response = self._detect_response(image, loc)
|
||||||
|
silhouette = self._glyph_silhouette()
|
||||||
|
if box is None or response is None or silhouette is None:
|
||||||
|
return score, box
|
||||||
|
x, y, x1, y1 = box
|
||||||
|
patch = response[y : y1 + 1, x : x1 + 1].astype(np.float32)
|
||||||
|
template = cv2.resize(silhouette, (patch.shape[1], patch.shape[0]), interpolation=cv2.INTER_AREA)
|
||||||
|
|
||||||
|
def glyph_detail(array: NDArray[Any]) -> NDArray[Any]:
|
||||||
|
array = array.astype(np.float32)
|
||||||
|
return array - array.mean(axis=0, keepdims=True) - array.mean(axis=1, keepdims=True) + array.mean()
|
||||||
|
|
||||||
|
patch, template = glyph_detail(patch), glyph_detail(template)
|
||||||
|
denominator = float(np.linalg.norm(patch) * np.linalg.norm(template))
|
||||||
|
detail_score = float((patch * template).sum()) / denominator if denominator > 1e-6 else 0.0
|
||||||
|
return max(0.0, min(score, detail_score)), box
|
||||||
|
|
||||||
|
|
||||||
|
class KlingEngine(_KlingTextEngine):
|
||||||
"""Detect/localize Kling AI's CJK and Latin 3.0 marks (locate -> mask -> fill)."""
|
"""Detect/localize Kling AI's CJK and Latin 3.0 marks (locate -> mask -> fill)."""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
super().__init__(_CONFIG)
|
super().__init__(_CONFIG)
|
||||||
self._latin = TextMarkEngine(_LATIN_CONFIG)
|
self._latin = _KlingTextEngine(_LATIN_CONFIG)
|
||||||
|
|
||||||
def detect(self, image: NDArray[Any], *, provenance: bool = False) -> TextMarkDetection:
|
def detect(self, image: NDArray[Any], *, provenance: bool = False) -> TextMarkDetection:
|
||||||
"""Return the strongest CJK or Latin Kling wordmark verdict."""
|
"""Return the strongest CJK or Latin Kling wordmark verdict."""
|
||||||
@@ -195,14 +225,15 @@ class KlingEngine(TextMarkEngine):
|
|||||||
return super().footprint_mask(image, force=force, dilate=dilate, detection=detection)
|
return super().footprint_mask(image, force=force, dilate=dilate, detection=detection)
|
||||||
|
|
||||||
det = detection if detection is not None else self.detect(image)
|
det = detection if detection is not None else self.detect(image)
|
||||||
legacy = super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
if not det.detected:
|
||||||
|
return None
|
||||||
|
engine = self._latin if det.template_asset == _LATIN_CONFIG.asset_name else self
|
||||||
|
legacy = TextMarkEngine.footprint_mask(engine, image, force=False, dilate=dilate, detection=det)
|
||||||
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
||||||
result = legacy
|
result = legacy
|
||||||
for config in (_CONFIG, _LATIN_CONFIG):
|
alpha = _text_mark_engine.load_alpha_template(engine.config.asset_name)
|
||||||
alpha = _text_mark_engine.load_alpha_template(config.asset_name)
|
if alpha is not None:
|
||||||
if alpha is None:
|
core = engine._aligned_alpha_mask(
|
||||||
continue
|
|
||||||
core = self._aligned_alpha_mask(
|
|
||||||
image,
|
image,
|
||||||
det,
|
det,
|
||||||
alpha,
|
alpha,
|
||||||
|
|||||||
@@ -170,7 +170,7 @@ class LibLibPillEngine:
|
|||||||
"""Detect the compact top-left pill only when LiblibAI is corroborated."""
|
"""Detect the compact top-left pill only when LiblibAI is corroborated."""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
self._pill = PillEngine()
|
self._pill = PillEngine(include_label=False)
|
||||||
|
|
||||||
def _raw(self, image: NDArray[Any]) -> PillDetection:
|
def _raw(self, image: NDArray[Any]) -> PillDetection:
|
||||||
return self._pill.detect(image)
|
return self._pill.detect(image)
|
||||||
|
|||||||
@@ -0,0 +1,302 @@
|
|||||||
|
"""Model Context Protocol server over stdio, a thin wrapper around this package's CLI.
|
||||||
|
|
||||||
|
``remove-ai-watermarks mcp`` speaks newline-delimited JSON-RPC 2.0 on stdin and
|
||||||
|
stdout with the initialize handshake. Each tool call runs one CLI command in a
|
||||||
|
child process of the same interpreter, so a tool returns exactly what that command
|
||||||
|
would: its exit code, its output file and its text. Nothing but protocol messages
|
||||||
|
reaches stdout, and a crash inside a command cannot corrupt the stream.
|
||||||
|
|
||||||
|
The server needs no dependency beyond the standard library, so the bare package
|
||||||
|
runs it through ``uvx remove-ai-watermarks mcp``. Commands that need an optional
|
||||||
|
extra fail with the CLI's own install hint, returned as a tool error.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import contextlib
|
||||||
|
import json
|
||||||
|
import logging
|
||||||
|
import os
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from collections.abc import Callable
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import IO, Any, cast
|
||||||
|
|
||||||
|
from remove_ai_watermarks import __version__
|
||||||
|
from remove_ai_watermarks.video import VIDEO_EXTENSIONS
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
# Newest first. A client asking for an unlisted version gets the newest, and
|
||||||
|
# decides for itself whether it can continue.
|
||||||
|
PROTOCOL_VERSIONS = ("2025-11-25", "2025-06-18", "2025-03-26", "2024-11-05")
|
||||||
|
|
||||||
|
INSTRUCTIONS = (
|
||||||
|
"Identify and remove AI provenance marks from images and video the user generated "
|
||||||
|
"or edited themselves: C2PA, IPTC, EXIF and TC260 metadata, visible AI-generator "
|
||||||
|
"labels, and supported invisible watermarks. Do not use it on stock, agency, "
|
||||||
|
"marketplace or other third-party marks that protect someone else's asset. A "
|
||||||
|
"result of 'unknown' or exit code 2 means no supported signal was found, not "
|
||||||
|
"that the file is clean. Paths are read on the machine running the server."
|
||||||
|
)
|
||||||
|
|
||||||
|
_MODES = ("metadata", "visible", "all")
|
||||||
|
|
||||||
|
TOOLS: list[dict[str, Any]] = [
|
||||||
|
{
|
||||||
|
"name": "identify",
|
||||||
|
"title": "Identify AI provenance",
|
||||||
|
"description": (
|
||||||
|
"Report which AI provenance signals an image or video carries: metadata "
|
||||||
|
"(C2PA, IPTC, EXIF, TC260), visible AI-generator labels, and supported local "
|
||||||
|
"invisible-watermark decoders. Read-only. Video is detected by file suffix."
|
||||||
|
),
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"path": {"type": "string", "description": "Image or video file to inspect."},
|
||||||
|
"check_visible": {
|
||||||
|
"type": "boolean",
|
||||||
|
"default": True,
|
||||||
|
"description": "Run the pixel and visible-mark detectors. False reads metadata only.",
|
||||||
|
},
|
||||||
|
},
|
||||||
|
"required": ["path"],
|
||||||
|
"additionalProperties": False,
|
||||||
|
},
|
||||||
|
"annotations": {"readOnlyHint": True, "openWorldHint": False},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "remove",
|
||||||
|
"title": "Remove AI watermarks from one file",
|
||||||
|
"description": (
|
||||||
|
"Write a cleaned copy of an image or video the user generated or edited. "
|
||||||
|
"mode=metadata strips AI metadata without touching pixels or re-encoding video; "
|
||||||
|
"mode=visible also removes a known visible AI label; for an image, mode=all adds "
|
||||||
|
"the invisible-watermark scrub where the installed extras and hardware allow it. "
|
||||||
|
"The source is never overwritten; the default output is <name>_clean beside it."
|
||||||
|
),
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"path": {"type": "string", "description": "Image or video file to clean."},
|
||||||
|
"mode": {"type": "string", "enum": list(_MODES), "default": "metadata"},
|
||||||
|
"output": {"type": "string", "description": "Output file. Must differ from the source."},
|
||||||
|
},
|
||||||
|
"required": ["path"],
|
||||||
|
"additionalProperties": False,
|
||||||
|
},
|
||||||
|
"annotations": {"readOnlyHint": False, "destructiveHint": False, "openWorldHint": False},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "batch",
|
||||||
|
"title": "Remove AI watermarks from a directory",
|
||||||
|
"description": (
|
||||||
|
"Clean every supported image (media=image) or video (media=video) in a directory "
|
||||||
|
"the user owns, writing results to a separate directory (default <dir>_clean). "
|
||||||
|
"Modes as for the remove tool."
|
||||||
|
),
|
||||||
|
"inputSchema": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"directory": {"type": "string", "description": "Directory of input files."},
|
||||||
|
"mode": {"type": "string", "enum": list(_MODES), "default": "metadata"},
|
||||||
|
"media": {"type": "string", "enum": ["image", "video"], "default": "image"},
|
||||||
|
"output_dir": {"type": "string", "description": "Output directory. Must differ from the input."},
|
||||||
|
},
|
||||||
|
"required": ["directory"],
|
||||||
|
"additionalProperties": False,
|
||||||
|
},
|
||||||
|
"annotations": {"readOnlyHint": False, "destructiveHint": False, "openWorldHint": False},
|
||||||
|
},
|
||||||
|
]
|
||||||
|
|
||||||
|
_TOOL_SCHEMAS = {tool["name"]: tool["inputSchema"] for tool in TOOLS}
|
||||||
|
|
||||||
|
Runner = Callable[[list[str]], subprocess.CompletedProcess[str]]
|
||||||
|
|
||||||
|
|
||||||
|
class ToolInputError(ValueError):
|
||||||
|
"""A tool call whose arguments the server refuses before running anything."""
|
||||||
|
|
||||||
|
|
||||||
|
def run_cli(args: list[str]) -> subprocess.CompletedProcess[str]:
|
||||||
|
"""Run one CLI command in a child process of this interpreter."""
|
||||||
|
env = {**os.environ, "COLUMNS": "120", "NO_COLOR": "1", "TERM": "dumb"}
|
||||||
|
command = [sys.executable, "-m", "remove_ai_watermarks.cli", *args]
|
||||||
|
log.info("Running CLI command: %s", command)
|
||||||
|
completed = subprocess.run( # noqa: S603 -- fixed interpreter, argument list, no shell
|
||||||
|
command,
|
||||||
|
stdin=subprocess.DEVNULL,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
env=env,
|
||||||
|
check=False,
|
||||||
|
)
|
||||||
|
log.info("CLI command exited %d: %s", completed.returncode, command)
|
||||||
|
return completed
|
||||||
|
|
||||||
|
|
||||||
|
def _validated(tool: str, arguments: dict[str, Any]) -> dict[str, Any]:
|
||||||
|
"""Check arguments against the tool's own input schema and fill its defaults."""
|
||||||
|
schema = _TOOL_SCHEMAS[tool]
|
||||||
|
properties: dict[str, dict[str, Any]] = schema["properties"]
|
||||||
|
unknown = sorted(set(arguments) - set(properties))
|
||||||
|
if unknown:
|
||||||
|
raise ToolInputError(f"unknown argument(s): {', '.join(unknown)}")
|
||||||
|
values: dict[str, Any] = {}
|
||||||
|
for key, spec in properties.items():
|
||||||
|
if key not in arguments:
|
||||||
|
if key in schema["required"]:
|
||||||
|
raise ToolInputError(f"'{key}' is required")
|
||||||
|
if "default" in spec:
|
||||||
|
values[key] = spec["default"]
|
||||||
|
continue
|
||||||
|
value = arguments[key]
|
||||||
|
if spec["type"] == "boolean" and not isinstance(value, bool):
|
||||||
|
raise ToolInputError(f"'{key}' must be a boolean")
|
||||||
|
if spec["type"] == "string" and (not isinstance(value, str) or not value.strip()):
|
||||||
|
raise ToolInputError(f"'{key}' must be a non-empty string")
|
||||||
|
if "enum" in spec and value not in spec["enum"]:
|
||||||
|
raise ToolInputError(f"'{key}' must be one of {', '.join(spec['enum'])}")
|
||||||
|
values[key] = value
|
||||||
|
return values
|
||||||
|
|
||||||
|
|
||||||
|
def _is_video(path: str) -> bool:
|
||||||
|
return Path(path).suffix.lower() in VIDEO_EXTENSIONS
|
||||||
|
|
||||||
|
|
||||||
|
def _identify_command(values: dict[str, Any]) -> tuple[list[str], str | None]:
|
||||||
|
path = values["path"]
|
||||||
|
prefix = ["video", "identify"] if _is_video(path) else ["identify"]
|
||||||
|
return [*prefix, "--json", *([] if values["check_visible"] else ["--no-visible"]), "--", path], None
|
||||||
|
|
||||||
|
|
||||||
|
def _remove_command(values: dict[str, Any]) -> tuple[list[str], str | None]:
|
||||||
|
path, mode = values["path"], values["mode"]
|
||||||
|
source = Path(path)
|
||||||
|
output = values.get("output") or str(source.with_stem(source.stem + "_clean"))
|
||||||
|
if Path(output).resolve() == source.resolve():
|
||||||
|
raise ToolInputError("'output' must differ from the source; the server never overwrites an input")
|
||||||
|
command = ["metadata", "--remove"] if mode == "metadata" else [mode]
|
||||||
|
return [*(["video"] if _is_video(path) else []), *command, "-o", output, "--", path], output
|
||||||
|
|
||||||
|
|
||||||
|
def _batch_command(values: dict[str, Any]) -> tuple[list[str], str | None]:
|
||||||
|
directory = values["directory"]
|
||||||
|
resolved = Path(directory).resolve()
|
||||||
|
output_dir = values.get("output_dir") or str(resolved.with_name(resolved.name + "_clean"))
|
||||||
|
if Path(output_dir).resolve() == resolved:
|
||||||
|
raise ToolInputError("'output_dir' must differ from the input directory")
|
||||||
|
command = ["video", "batch"] if values["media"] == "video" else ["batch"]
|
||||||
|
return [*command, "--mode", values["mode"], "-o", output_dir, "--", directory], output_dir
|
||||||
|
|
||||||
|
|
||||||
|
_COMMANDS: dict[str, Callable[[dict[str, Any]], tuple[list[str], str | None]]] = {
|
||||||
|
"identify": _identify_command,
|
||||||
|
"remove": _remove_command,
|
||||||
|
"batch": _batch_command,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def call_tool(name: str, arguments: dict[str, Any], runner: Runner = run_cli) -> dict[str, Any]:
|
||||||
|
"""Run one tool and return an MCP ``tools/call`` result."""
|
||||||
|
try:
|
||||||
|
args, output = _COMMANDS[name](_validated(name, arguments))
|
||||||
|
except ToolInputError as exc:
|
||||||
|
return {"content": [{"type": "text", "text": f"Invalid arguments: {exc}"}], "isError": True}
|
||||||
|
try:
|
||||||
|
completed = runner(args)
|
||||||
|
except OSError as exc: # the child never started, so there is no exit code to relay
|
||||||
|
log.warning("CLI command could not start: %s: %s", args, exc)
|
||||||
|
return {"content": [{"type": "text", "text": f"The command could not start: {exc}"}], "isError": True}
|
||||||
|
code = completed.returncode
|
||||||
|
text = completed.stdout.strip()
|
||||||
|
stderr = completed.stderr.strip()[-4000:]
|
||||||
|
structured: dict[str, Any] = {"command": ["remove-ai-watermarks", *args], "exit_code": code, "output": output}
|
||||||
|
if "--json" in args and code == 0:
|
||||||
|
with contextlib.suppress(json.JSONDecodeError):
|
||||||
|
structured["report"] = json.loads(text)
|
||||||
|
if "report" not in structured:
|
||||||
|
structured["stdout"] = text
|
||||||
|
if stderr:
|
||||||
|
structured["stderr"] = stderr
|
||||||
|
# The CLI's own exit 2 means no supported signal, or a batch directory with nothing
|
||||||
|
# to process, both results. Click also exits 2 on
|
||||||
|
# a usage error, such as a missing file, and prints its usage line on stderr.
|
||||||
|
no_signal = code == 2 and "Usage:" not in stderr
|
||||||
|
is_error = code != 0 and not no_signal
|
||||||
|
summary = f"exit code {code}"
|
||||||
|
if no_signal:
|
||||||
|
summary += " (no supported signal found, or nothing to process; this is unknown, not proof of a clean file)"
|
||||||
|
body = "\n\n".join(part for part in (summary, text, stderr if is_error else "") if part)
|
||||||
|
return {"content": [{"type": "text", "text": body}], "structuredContent": structured, "isError": is_error}
|
||||||
|
|
||||||
|
|
||||||
|
def _error(request_id: object, code: int, message: str) -> dict[str, Any]:
|
||||||
|
return {"jsonrpc": "2.0", "id": request_id, "error": {"code": code, "message": message}}
|
||||||
|
|
||||||
|
|
||||||
|
def _object(value: object) -> dict[str, Any] | None:
|
||||||
|
"""A decoded JSON object, or ``None`` for any other JSON value."""
|
||||||
|
return cast("dict[str, Any]", value) if isinstance(value, dict) else None
|
||||||
|
|
||||||
|
|
||||||
|
def handle(raw: object, runner: Runner = run_cli) -> dict[str, Any] | None:
|
||||||
|
"""Answer one JSON-RPC message; ``None`` for notifications and client responses."""
|
||||||
|
message = _object(raw)
|
||||||
|
if message is None or message.get("jsonrpc") != "2.0":
|
||||||
|
return _error(None, -32600, "Invalid Request")
|
||||||
|
method = message.get("method")
|
||||||
|
if "id" not in message:
|
||||||
|
return None # a notification or a response to nothing this server sent
|
||||||
|
request_id: object = message["id"]
|
||||||
|
if not isinstance(method, str):
|
||||||
|
return _error(request_id, -32600, "Invalid Request")
|
||||||
|
params = _object(message.get("params") or {})
|
||||||
|
if params is None:
|
||||||
|
return _error(request_id, -32602, "params must be an object")
|
||||||
|
|
||||||
|
if method == "initialize":
|
||||||
|
requested = params.get("protocolVersion")
|
||||||
|
result: dict[str, Any] = {
|
||||||
|
"protocolVersion": requested if requested in PROTOCOL_VERSIONS else PROTOCOL_VERSIONS[0],
|
||||||
|
"capabilities": {"tools": {"listChanged": False}},
|
||||||
|
"serverInfo": {"name": "remove-ai-watermarks", "title": "Remove AI Watermarks", "version": __version__},
|
||||||
|
"instructions": INSTRUCTIONS,
|
||||||
|
}
|
||||||
|
elif method == "ping":
|
||||||
|
result = {}
|
||||||
|
elif method == "tools/list":
|
||||||
|
result = {"tools": TOOLS}
|
||||||
|
elif method == "tools/call":
|
||||||
|
name = params.get("name")
|
||||||
|
arguments = _object(params.get("arguments") or {})
|
||||||
|
if not isinstance(name, str) or name not in _COMMANDS:
|
||||||
|
return _error(request_id, -32602, f"Unknown tool: {name}")
|
||||||
|
if arguments is None:
|
||||||
|
return _error(request_id, -32602, "arguments must be an object")
|
||||||
|
result = call_tool(name, arguments, runner)
|
||||||
|
else:
|
||||||
|
return _error(request_id, -32601, f"Method not found: {method}")
|
||||||
|
return {"jsonrpc": "2.0", "id": request_id, "result": result}
|
||||||
|
|
||||||
|
|
||||||
|
def serve(stdin: IO[str] | None = None, stdout: IO[str] | None = None, runner: Runner = run_cli) -> None:
|
||||||
|
"""Serve newline-delimited JSON-RPC until stdin closes."""
|
||||||
|
source: IO[str] = sys.stdin if stdin is None else stdin
|
||||||
|
sink: IO[str] = sys.stdout if stdout is None else stdout
|
||||||
|
for line in source:
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
message = json.loads(line)
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
response: dict[str, Any] | None = _error(None, -32700, "Parse error")
|
||||||
|
else:
|
||||||
|
response = handle(message, runner)
|
||||||
|
if response is not None:
|
||||||
|
sink.write(json.dumps(response, ensure_ascii=False) + "\n")
|
||||||
|
sink.flush()
|
||||||
@@ -85,6 +85,11 @@ AI_KEYWORDS: tuple[str, ...] = (
|
|||||||
# Reference: https://spec.c2pa.org/specifications/specifications/2.1/specs/C2PA_Specification.html
|
# Reference: https://spec.c2pa.org/specifications/specifications/2.1/specs/C2PA_Specification.html
|
||||||
C2PA_UUID: bytes = bytes.fromhex("d8fec3d61b0e483c92975828877ec481")
|
C2PA_UUID: bytes = bytes.fromhex("d8fec3d61b0e483c92975828877ec481")
|
||||||
|
|
||||||
|
# Kling's encoder writes an H.264 ``user_data_unregistered`` SEI message (payload
|
||||||
|
# type 5) whose payload is this UUID followed by ``kling-ai``. It lives in the coded
|
||||||
|
# stream inside ``mdat``, where no metadata box walk reaches it.
|
||||||
|
KLING_SEI_TAG: bytes = bytes.fromhex("91ca60614aee385486142d5f73f4ae2e") + b"kling-ai"
|
||||||
|
|
||||||
|
|
||||||
def c2pa_marker_in(data: bytes) -> bool:
|
def c2pa_marker_in(data: bytes) -> bool:
|
||||||
"""True if ``data`` carries a real C2PA manifest marker, not just an
|
"""True if ``data`` carries a real C2PA manifest marker, not just an
|
||||||
@@ -1430,6 +1435,10 @@ def get_ai_metadata(image_path: Path) -> dict[str, str]:
|
|||||||
if app_generator:
|
if app_generator:
|
||||||
result.setdefault("app_aigc", f"App AIGC disclosure ({app_generator})")
|
result.setdefault("app_aigc", f"App AIGC disclosure ({app_generator})")
|
||||||
|
|
||||||
|
# Kling's encoder tag, an H.264 SEI message in the first video sample.
|
||||||
|
if KLING_SEI_TAG in app_scan:
|
||||||
|
result.setdefault("kling_sei", "Kling AI encoder tag (H.264 SEI kling-ai)")
|
||||||
|
|
||||||
# xAI / Grok signature pair in EXIF or an equivalent container-native field.
|
# xAI / Grok signature pair in EXIF or an equivalent container-native field.
|
||||||
if xai_signature(image_path):
|
if xai_signature(image_path):
|
||||||
result.setdefault("xai_signature", "xAI/Grok signature (UUID + Signature blob)")
|
result.setdefault("xai_signature", "xAI/Grok signature (UUID + Signature blob)")
|
||||||
@@ -1847,6 +1856,92 @@ def _sniff_image_format(head: bytes) -> str | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
_JXL_CONTAINER_MAGIC = b"\x00\x00\x00\x0cJXL \r\n\x87\n"
|
||||||
|
# Accepted on their magic alone: no local parser walks them, and the ffmpeg strip path
|
||||||
|
# fails on its own when one is unreadable. JPEG XL codestream, FLV, ID3, FLAC, Ogg.
|
||||||
|
_MAGIC_ONLY: tuple[bytes, ...] = (b"\xff\x0a", b"FLV", b"ID3", b"fLaC", b"OggS")
|
||||||
|
_PILLOW_RASTERS = frozenset({"png", "jpeg", "webp", "gif", "bmp", "tiff"})
|
||||||
|
|
||||||
|
|
||||||
|
def _tiling_reason(kind: str, ends: Iterable[int], size: int) -> str | None:
|
||||||
|
"""None when the last top-level element ends at EOF; a truncation leaves a gap."""
|
||||||
|
end = max(ends, default=0) # top-level end offsets only increase
|
||||||
|
if end == size:
|
||||||
|
return None
|
||||||
|
return f"the {kind} structure stops at byte {end} of {size} (truncated or corrupt)"
|
||||||
|
|
||||||
|
|
||||||
|
def _pixel_stack_decodes(path: Path) -> bool:
|
||||||
|
"""Whether the optional pixel stack decodes ``path``; False when it is absent."""
|
||||||
|
from remove_ai_watermarks.optional_deps import pixels_available
|
||||||
|
|
||||||
|
if not pixels_available():
|
||||||
|
return False
|
||||||
|
from remove_ai_watermarks import image_io
|
||||||
|
|
||||||
|
return image_io.imread(path) is not None
|
||||||
|
|
||||||
|
|
||||||
|
@functools.lru_cache(maxsize=8)
|
||||||
|
def _unreadable_reason_cached(path_str: str, _mtime_ns: int, size: int) -> str | None:
|
||||||
|
from pathlib import Path as _Path
|
||||||
|
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
from remove_ai_watermarks._internal.ebml import EBML_MAGIC, iter_top_level_ends
|
||||||
|
from remove_ai_watermarks._internal.isobmff import iter_file_boxes
|
||||||
|
from remove_ai_watermarks.forensic_metadata import sniff_format
|
||||||
|
|
||||||
|
path = _Path(path_str)
|
||||||
|
if size == 0:
|
||||||
|
return "the file is empty"
|
||||||
|
with path.open("rb") as stream:
|
||||||
|
head = stream.read(32)
|
||||||
|
kind = sniff_format(head)
|
||||||
|
if kind.startswith("isobmff:") or head.startswith(_JXL_CONTAINER_MAGIC):
|
||||||
|
return _tiling_reason("ISOBMFF box", (end for _s, end, _t, _p in iter_file_boxes(stream, 0, size)), size)
|
||||||
|
if head.startswith(EBML_MAGIC):
|
||||||
|
return _tiling_reason("EBML element", iter_top_level_ends(stream, size), size)
|
||||||
|
if kind in _PILLOW_RASTERS:
|
||||||
|
try:
|
||||||
|
with Image.open(path) as image:
|
||||||
|
image.load()
|
||||||
|
except Image.DecompressionBombError:
|
||||||
|
return None # readable, merely too large for a precautionary full decode
|
||||||
|
except Exception as exc:
|
||||||
|
# The pixel stack decodes some rasters Pillow rejects (a PNG with a bad eXIf
|
||||||
|
# CRC), and strip_and_verify recovers those through it, so they are readable.
|
||||||
|
return None if _pixel_stack_decodes(path) else f"the image cannot be decoded: {exc}"
|
||||||
|
return None
|
||||||
|
if head.startswith(b"RIFF") and len(head) >= 8:
|
||||||
|
declared = 8 + struct.unpack_from("<I", head, 4)[0]
|
||||||
|
return None if declared <= size else f"the RIFF header declares {declared} bytes but the file has {size}"
|
||||||
|
mpeg_audio_sync = len(head) > 1 and head[0] == 0xFF and head[1] & 0xE0 == 0xE0
|
||||||
|
if head.startswith(_MAGIC_ONLY) or mpeg_audio_sync:
|
||||||
|
return None
|
||||||
|
return "the content is not a recognized image, video or audio container"
|
||||||
|
|
||||||
|
|
||||||
|
def unreadable_reason(path: Path) -> str | None:
|
||||||
|
"""Why ``path`` is not a readable image, video or audio container, or None.
|
||||||
|
|
||||||
|
The metadata scanners read raw bytes, so on a file that is not a container at all
|
||||||
|
they find nothing and say so, which is the same answer a real file without metadata
|
||||||
|
gets. Callers that report "no metadata" or "stripped" ask this first, so an
|
||||||
|
unreadable input fails instead of reading as clean. Rasters must fully decode in
|
||||||
|
Pillow or, when it is installed, the pixel stack; ISOBMFF boxes and EBML elements
|
||||||
|
must tile the file, which is what a truncation breaks; a RIFF file must hold its
|
||||||
|
declared size. The formats in ``_MAGIC_ONLY`` and MPEG audio frames have no local
|
||||||
|
walker and pass on their magic. Memoized on (path, mtime, size): one command asks
|
||||||
|
several times about the same input.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
stat = path.stat()
|
||||||
|
return _unreadable_reason_cached(str(path), stat.st_mtime_ns, stat.st_size)
|
||||||
|
except OSError as exc:
|
||||||
|
return f"cannot be read: {exc}"
|
||||||
|
|
||||||
|
|
||||||
def strip_and_verify(
|
def strip_and_verify(
|
||||||
source_path: Path,
|
source_path: Path,
|
||||||
output_path: Path | None = None,
|
output_path: Path | None = None,
|
||||||
@@ -1857,10 +1952,12 @@ def strip_and_verify(
|
|||||||
|
|
||||||
:func:`remove_ai_metadata` is deliberately fail-safe: a file PIL cannot decode is
|
:func:`remove_ai_metadata` is deliberately fail-safe: a file PIL cannot decode is
|
||||||
copied through UNCHANGED rather than crashing a caller, and the path it returns is
|
copied through UNCHANGED rather than crashing a caller, and the path it returns is
|
||||||
indistinguishable from a real strip. Any caller that reports an outcome to a user
|
indistinguishable from a real strip. This wrapper is what a caller reporting an
|
||||||
therefore cannot tell a no-op from a success. A Samsung C2PA compatibility case
|
outcome uses instead: it refuses a source :func:`unreadable_reason` rejects with
|
||||||
exposed this when `metadata --remove` reported success while the output still read
|
``ValueError`` before writing anything, because a copy-through of bytes nothing can
|
||||||
as AI.
|
parse may carry no marker a re-scan sees, and then re-scans what it wrote. A
|
||||||
|
Samsung C2PA compatibility case exposed the original gap when `metadata --remove`
|
||||||
|
reported success while the output still read as AI.
|
||||||
|
|
||||||
If markers survive but OpenCV can still decode the raster, the verified path
|
If markers survive but OpenCV can still decode the raster, the verified path
|
||||||
normalizes the container through :mod:`image_io` and scans once more. This
|
normalizes the container through :mod:`image_io` and scans once more. This
|
||||||
@@ -1869,6 +1966,27 @@ def strip_and_verify(
|
|||||||
|
|
||||||
Returns ``(output_path, surviving_markers)``; an empty mapping means a real strip.
|
Returns ``(output_path, surviving_markers)``; an empty mapping means a real strip.
|
||||||
"""
|
"""
|
||||||
|
if (reason := unreadable_reason(source_path)) is not None:
|
||||||
|
raise ValueError(f"cannot read {source_path.name}: {reason}")
|
||||||
|
return _strip_and_rescan(source_path, output_path, keep_standard=keep_standard)
|
||||||
|
|
||||||
|
|
||||||
|
def strip_written_output(path: Path, *, keep_standard: bool = True) -> dict[str, str]:
|
||||||
|
"""Strip ``path`` in place and return the markers that survived.
|
||||||
|
|
||||||
|
For a file whose pixels this run has already decoded or written itself, so the
|
||||||
|
readability probe that :func:`strip_and_verify` runs first would only decode
|
||||||
|
them a second time.
|
||||||
|
"""
|
||||||
|
return _strip_and_rescan(path, path, keep_standard=keep_standard)[1]
|
||||||
|
|
||||||
|
|
||||||
|
def _strip_and_rescan(
|
||||||
|
source_path: Path,
|
||||||
|
output_path: Path | None,
|
||||||
|
*,
|
||||||
|
keep_standard: bool,
|
||||||
|
) -> tuple[Path, dict[str, str]]:
|
||||||
out = remove_ai_metadata(source_path, output_path, keep_standard=keep_standard)
|
out = remove_ai_metadata(source_path, output_path, keep_standard=keep_standard)
|
||||||
remaining = get_ai_metadata(out)
|
remaining = get_ai_metadata(out)
|
||||||
if not remaining:
|
if not remaining:
|
||||||
|
|||||||
@@ -38,10 +38,12 @@ status was not adjudicated, and 1200 non-overlapping no-signal controls:
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import replace
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from remove_ai_watermarks import _text_mark_engine
|
from remove_ai_watermarks import _text_mark_engine
|
||||||
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkEngine
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection
|
||||||
|
from remove_ai_watermarks._text_mark_variants import GlyphTextMarkEngine, TextMarkVariantsMixin
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from numpy.typing import NDArray
|
from numpy.typing import NDArray
|
||||||
@@ -101,8 +103,62 @@ def _alpha_template() -> NDArray[Any] | None:
|
|||||||
return _text_mark_engine.load_alpha_template(_CONFIG.asset_name)
|
return _text_mark_engine.load_alpha_template(_CONFIG.asset_name)
|
||||||
|
|
||||||
|
|
||||||
class MicrosoftEngine(TextMarkEngine):
|
class MicrosoftEngine(TextMarkVariantsMixin):
|
||||||
"""Detect/localize the measured Microsoft top-right AI badge."""
|
"""Detect/localize the measured Microsoft top-right AI badge."""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
super().__init__(_CONFIG)
|
super().__init__(_CONFIG)
|
||||||
|
self._variants = (
|
||||||
|
GlyphTextMarkEngine(
|
||||||
|
replace(
|
||||||
|
_CONFIG,
|
||||||
|
asset_name="microsoft_text_alpha.png",
|
||||||
|
width_frac=0.20,
|
||||||
|
height_frac=0.065,
|
||||||
|
margin_x_frac=0.002,
|
||||||
|
margin_bottom_frac=0.002,
|
||||||
|
scale_basis="width",
|
||||||
|
alpha_width_frac=0.14,
|
||||||
|
alpha_height_frac=0.14 * 49 / 339,
|
||||||
|
detect_frontend="contrast",
|
||||||
|
ladder=(0.95, 1.0, 1.05),
|
||||||
|
detect_ncc_threshold=0.60,
|
||||||
|
provenance_ncc_factor=1.0,
|
||||||
|
max_saturation=255,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any] | None,
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
dilate: int | None = None,
|
||||||
|
detection: TextMarkDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
|
"""Cover the enclosing beige pill for the text-only layout."""
|
||||||
|
if force or image is None or image.size == 0:
|
||||||
|
return super().footprint_mask(image, force=force, dilate=dilate, detection=detection)
|
||||||
|
det = detection if detection is not None else self.detect(image)
|
||||||
|
engine = self.variant_engine(det)
|
||||||
|
if engine is self or not det.detected or det.match_box is None:
|
||||||
|
return super().footprint_mask(image, force=False, dilate=dilate, detection=det)
|
||||||
|
|
||||||
|
from remove_ai_watermarks import region_eraser
|
||||||
|
|
||||||
|
loc = engine.locate(image)
|
||||||
|
x0, y0, x1, y1 = det.match_box
|
||||||
|
pad_x = int(image.shape[1] * 0.012)
|
||||||
|
pad_y = int(image.shape[1] * 0.014)
|
||||||
|
box = (
|
||||||
|
max(0, loc.x + x0 - pad_x),
|
||||||
|
max(0, loc.y + y0 - pad_y),
|
||||||
|
x1 - x0 + 1 + 2 * pad_x,
|
||||||
|
y1 - y0 + 1 + 2 * pad_y,
|
||||||
|
)
|
||||||
|
return region_eraser.boxes_to_mask(
|
||||||
|
image.shape[:2],
|
||||||
|
[box],
|
||||||
|
dilate=0 if dilate is None else max(0, dilate),
|
||||||
|
)
|
||||||
@@ -3,8 +3,7 @@
|
|||||||
OpenArt (openart.ai) stamps free-plan generations with a semi-transparent white
|
OpenArt (openart.ai) stamps free-plan generations with a semi-transparent white
|
||||||
"OpenArt" wordmark (a bowtie/infinity icon followed by the brand name) placed
|
"OpenArt" wordmark (a bowtie/infinity icon followed by the brand name) placed
|
||||||
**near the center of the frame**, not in a corner like every other text mark
|
**near the center of the frame**, not in a corner like every other text mark
|
||||||
this repository detects. Confirmed from one customer-reported production case
|
this repository detects. In the observed portrait layout, the mark sat roughly centered
|
||||||
(raiw-app, 2026-09-11, 1528x2712 portrait output): the mark sat roughly centered
|
|
||||||
over the subject's torso, and neither C2PA, EXIF AI tags, nor the China AIGC/TC260
|
over the subject's torso, and neither C2PA, EXIF AI tags, nor the China AIGC/TC260
|
||||||
label were present -- OpenArt apparently ships no machine-readable provenance
|
label were present -- OpenArt apparently ships no machine-readable provenance
|
||||||
signal alongside this visible mark, so a bare export was previously reported as
|
signal alongside this visible mark, so a bare export was previously reported as
|
||||||
|
|||||||
@@ -1,18 +1,20 @@
|
|||||||
"""Jimeng-basic 'AI生成' pill: a CAPTURE-LESS visible mark (issue #54).
|
"""Jimeng-basic ``AI生成`` pill visible mark (issue #54).
|
||||||
|
|
||||||
The Jimeng free-tier TC260 label is a rounded pill with 'AI生成' in the TOP-LEFT
|
The Jimeng free-tier TC260 label is a rounded pill with 'AI生成' in the TOP-LEFT
|
||||||
corner -- distinct from the ``jimeng`` "★ 即梦AI" wordmark (bottom-right). It has no
|
corner -- distinct from the ``jimeng`` "★ 即梦AI" wordmark (bottom-right). It has no
|
||||||
captured alpha map, so unlike the other marks it is detected purely by a synthetic
|
captured alpha map for its outline. Detection therefore has two paths before the
|
||||||
silhouette; like every mark it is then removed by the shared localize -> fill:
|
shared localize -> fill step:
|
||||||
|
|
||||||
* Detect: edge-NCC of a font-rendered SILHOUETTE (``assets/jimeng_pill.png``,
|
* A high-confidence contrast template for the measured solid ``AI生成`` label
|
||||||
|
(``assets/jimeng_label_alpha.png``). With Jimeng provenance this verified
|
||||||
|
label can bypass the flat-background safety gate.
|
||||||
|
* Edge-NCC of a font-rendered SILHOUETTE (``assets/jimeng_pill.png``,
|
||||||
synthetic, data-safe -- see ``scripts/render_pill_silhouette.py``) against the
|
synthetic, data-safe -- see ``scripts/render_pill_silhouette.py``) against the
|
||||||
top-left ROI, at the pill's known width fraction. The calibrated
|
top-left ROI, at the pill's known width fraction. Its calibrated
|
||||||
``_DETECT_THRESHOLD`` is 0.22.
|
``_DETECT_THRESHOLD`` is 0.22 and it retains the provenance and flat-background
|
||||||
* Remove: place the pill footprint at the matched location and inpaint it
|
product gates.
|
||||||
(MI-GAN / cv2 via the registry). Quality comes from the inpaint backend, so the
|
* Remove: mask the verified label match or the stable outline footprint and
|
||||||
silhouette need not be pixel-accurate -- which is why a synthetic render is
|
inpaint it (MI-GAN / cv2 via the registry).
|
||||||
sufficient and no source-derived asset is committed.
|
|
||||||
|
|
||||||
Geometry uses width ~0.161*W,
|
Geometry uses width ~0.161*W,
|
||||||
height ~0.091*W, top-left, margins ~0.02-0.05.
|
height ~0.091*W, top-left, margins ~0.02-0.05.
|
||||||
@@ -27,6 +29,7 @@ import cv2
|
|||||||
import numpy as np
|
import numpy as np
|
||||||
|
|
||||||
from remove_ai_watermarks import image_io
|
from remove_ai_watermarks import image_io
|
||||||
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from numpy.typing import NDArray
|
from numpy.typing import NDArray
|
||||||
@@ -74,6 +77,32 @@ class PillDetection(NamedTuple):
|
|||||||
detected: bool
|
detected: bool
|
||||||
confidence: float
|
confidence: float
|
||||||
region: tuple[int, int, int, int] # x, y, w, h of the matched pill
|
region: tuple[int, int, int, int] # x, y, w, h of the matched pill
|
||||||
|
verified_label: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
_LABEL_CONFIG = TextMarkConfig(
|
||||||
|
name="Jimeng AI生成 pill label",
|
||||||
|
asset_name="jimeng_label_alpha.png",
|
||||||
|
corner="tl",
|
||||||
|
margin_floor=2,
|
||||||
|
width_frac=0.34,
|
||||||
|
height_frac=0.14,
|
||||||
|
margin_x_frac=0.0,
|
||||||
|
margin_bottom_frac=0.0,
|
||||||
|
max_saturation=255,
|
||||||
|
logo_min_luma=100,
|
||||||
|
tophat_delta=8,
|
||||||
|
morph_open_size=3,
|
||||||
|
detect_min_coverage=0.01,
|
||||||
|
detect_ncc_threshold=0.55,
|
||||||
|
detect_frontend="contrast",
|
||||||
|
scale_basis="width",
|
||||||
|
ladder=(0.95, 1.0, 1.05),
|
||||||
|
alpha_width_frac=0.14,
|
||||||
|
alpha_height_frac=0.14 * 128 / 353,
|
||||||
|
min_gw=24,
|
||||||
|
provenance_ncc_factor=1.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _load_silhouette() -> NDArray[Any] | None:
|
def _load_silhouette() -> NDArray[Any] | None:
|
||||||
@@ -92,7 +121,18 @@ def _grad(gray: NDArray[Any]) -> NDArray[Any]:
|
|||||||
|
|
||||||
|
|
||||||
class PillEngine:
|
class PillEngine:
|
||||||
"""Detect + build the removal mask for the top-left 'AI生成' pill (edge-NCC of a synthetic silhouette)."""
|
"""Detect the top-left ``AI生成`` label or its synthetic pill outline."""
|
||||||
|
|
||||||
|
def __init__(self, *, include_label: bool = True) -> None:
|
||||||
|
self._label = TextMarkEngine(_LABEL_CONFIG) if include_label else None
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def _label_box(detection: TextMarkDetection) -> tuple[int, int, int, int] | None:
|
||||||
|
if not detection.detected or detection.match_box is None:
|
||||||
|
return None
|
||||||
|
x, y, _width, _height = detection.region
|
||||||
|
x0, y0, x1, y1 = detection.match_box
|
||||||
|
return x + x0, y + y0, x1 - x0 + 1, y1 - y0 + 1
|
||||||
|
|
||||||
def _match(self, image: NDArray[Any]) -> tuple[float, tuple[int, int, int, int]] | None:
|
def _match(self, image: NDArray[Any]) -> tuple[float, tuple[int, int, int, int]] | None:
|
||||||
sil = _load_silhouette()
|
sil = _load_silhouette()
|
||||||
@@ -113,12 +153,33 @@ class PillEngine:
|
|||||||
_, score, _, loc = cv2.minMaxLoc(res)
|
_, score, _, loc = cv2.minMaxLoc(res)
|
||||||
return float(score), (int(loc[0]), int(loc[1]), tw, th)
|
return float(score), (int(loc[0]), int(loc[1]), tw, th)
|
||||||
|
|
||||||
def detect(self, image: NDArray[Any]) -> PillDetection:
|
def _detection(
|
||||||
m = self._match(image)
|
self,
|
||||||
if m is None:
|
match: tuple[float, tuple[int, int, int, int]] | None,
|
||||||
return PillDetection(False, 0.0, (0, 0, 0, 0))
|
label: TextMarkDetection,
|
||||||
score, box = m
|
) -> PillDetection:
|
||||||
return PillDetection(score >= _DETECT_THRESHOLD, score, box)
|
label_box = self._label_box(label)
|
||||||
|
if match is None:
|
||||||
|
return PillDetection(label.detected, label.confidence, label_box or label.region, label.detected)
|
||||||
|
score, box = match
|
||||||
|
return PillDetection(
|
||||||
|
score >= _DETECT_THRESHOLD or label.detected,
|
||||||
|
max(score, label.confidence),
|
||||||
|
label_box if label.detected and label_box is not None else box,
|
||||||
|
label.detected,
|
||||||
|
)
|
||||||
|
|
||||||
|
def detect(self, image: NDArray[Any], *, provenance: bool = False) -> PillDetection:
|
||||||
|
"""Detect the outline, plus the solid label when Jimeng is corroborated."""
|
||||||
|
label_scan = self._label.detect(image) if self._label is not None else TextMarkDetection()
|
||||||
|
label = label_scan if provenance else TextMarkDetection()
|
||||||
|
return self._detection(self._match(image), label)
|
||||||
|
|
||||||
|
def detect_both(self, image: NDArray[Any]) -> tuple[PillDetection, PillDetection]:
|
||||||
|
"""Return strict outline and Jimeng-corroborated label verdicts from one scan."""
|
||||||
|
match = self._match(image)
|
||||||
|
label = self._label.detect(image) if self._label is not None else TextMarkDetection()
|
||||||
|
return self._detection(match, TextMarkDetection()), self._detection(match, label)
|
||||||
|
|
||||||
def _footprint_box(self, image: NDArray[Any]) -> tuple[int, int, int, int] | None:
|
def _footprint_box(self, image: NDArray[Any]) -> tuple[int, int, int, int] | None:
|
||||||
h, w = image.shape[:2]
|
h, w = image.shape[:2]
|
||||||
@@ -152,19 +213,34 @@ class PillEngine:
|
|||||||
"""True when the top-left footprint is flat enough for an invisible inpaint."""
|
"""True when the top-left footprint is flat enough for an invisible inpaint."""
|
||||||
return self.footprint_texture(image) <= thresh
|
return self.footprint_texture(image) <= thresh
|
||||||
|
|
||||||
def footprint_mask(self, image: NDArray[Any], *, force: bool = False) -> NDArray[Any] | None:
|
def footprint_mask(
|
||||||
|
self,
|
||||||
|
image: NDArray[Any],
|
||||||
|
*,
|
||||||
|
force: bool = False,
|
||||||
|
detection: PillDetection | None = None,
|
||||||
|
) -> NDArray[Any] | None:
|
||||||
"""Full-frame uint8 mask (255 = pill) over the pill's known top-left region.
|
"""Full-frame uint8 mask (255 = pill) over the pill's known top-left region.
|
||||||
|
|
||||||
Uses stable GEOMETRY (a generous fixed box), not the NCC match position: the
|
A verified label uses its detector-aligned box with measured padding. The
|
||||||
synthetic silhouette localizes only approximately, so a match-positioned mask
|
weaker synthetic outline uses stable geometry because its NCC position is
|
||||||
leaves outline residue, while the top-left corner is negative space, so a
|
only approximate and a match-positioned mask leaves rim residue. The caller
|
||||||
generous geometric box removes the pill cleanly and harmlessly. The caller
|
|
||||||
gates on :meth:`detect`, so a clean corner is never masked. ``force`` is
|
gates on :meth:`detect`, so a clean corner is never masked. ``force`` is
|
||||||
accepted for a uniform engine signature but ignored (the geometry box is
|
accepted for a uniform engine signature but ignored."""
|
||||||
fixed regardless)."""
|
|
||||||
if image is None or image.size == 0:
|
if image is None or image.size == 0:
|
||||||
return None
|
return None
|
||||||
box = self._footprint_box(image)
|
if detection is not None and detection.verified_label:
|
||||||
|
x, y, width, height = detection.region
|
||||||
|
pad_x = int(image.shape[1] * 0.025)
|
||||||
|
pad_y = int(image.shape[1] * 0.014)
|
||||||
|
box = (
|
||||||
|
max(0, x - pad_x),
|
||||||
|
max(0, y - pad_y),
|
||||||
|
min(image.shape[1], x + width + pad_x),
|
||||||
|
min(image.shape[0], y + height + pad_y),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
box = self._footprint_box(image)
|
||||||
if box is None:
|
if box is None:
|
||||||
return None
|
return None
|
||||||
# Same primitive the shared fill uses, rather than a private zeros/fill copy.
|
# Same primitive the shared fill uses, rather than a private zeros/fill copy.
|
||||||
|
|||||||
@@ -25,10 +25,12 @@ signal via the registry.
|
|||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import replace
|
||||||
from typing import TYPE_CHECKING, Any
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
from remove_ai_watermarks import _text_mark_engine, image_io
|
from remove_ai_watermarks import _text_mark_engine, image_io
|
||||||
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection, TextMarkEngine
|
from remove_ai_watermarks._text_mark_engine import TextMarkConfig, TextMarkDetection
|
||||||
|
from remove_ai_watermarks._text_mark_variants import GlyphTextMarkEngine, TextMarkVariantsMixin
|
||||||
|
|
||||||
if TYPE_CHECKING:
|
if TYPE_CHECKING:
|
||||||
from numpy.typing import NDArray
|
from numpy.typing import NDArray
|
||||||
@@ -107,11 +109,27 @@ def _template_match_score(box_mask: NDArray[Any], scale_base: int) -> float:
|
|||||||
return _text_mark_engine.template_match_score(box_mask, scale_base, _CONFIG)
|
return _text_mark_engine.template_match_score(box_mask, scale_base, _CONFIG)
|
||||||
|
|
||||||
|
|
||||||
class SamsungEngine(TextMarkEngine):
|
class SamsungEngine(TextMarkVariantsMixin):
|
||||||
"""Detect/localize the visible Samsung Galaxy AI text mark (locate -> mask; mask feeds the fill)."""
|
"""Detect/localize the visible Samsung Galaxy AI text mark (locate -> mask; mask feeds the fill)."""
|
||||||
|
|
||||||
def __init__(self) -> None:
|
def __init__(self) -> None:
|
||||||
super().__init__(_CONFIG)
|
super().__init__(_CONFIG)
|
||||||
|
self._variants = (
|
||||||
|
GlyphTextMarkEngine(
|
||||||
|
replace(
|
||||||
|
_CONFIG,
|
||||||
|
asset_name="samsung_ko_alpha.png",
|
||||||
|
width_frac=0.28,
|
||||||
|
height_frac=0.065,
|
||||||
|
alpha_width_frac=0.20,
|
||||||
|
alpha_height_frac=0.20 * 49 / 344,
|
||||||
|
ladder=(0.95, 1.0, 1.05),
|
||||||
|
max_saturation=255,
|
||||||
|
detect_frontend="contrast",
|
||||||
|
detect_ncc_threshold=0.46,
|
||||||
|
)
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
def footprint_mask(
|
def footprint_mask(
|
||||||
self,
|
self,
|
||||||
@@ -134,10 +152,29 @@ class SamsungEngine(TextMarkEngine):
|
|||||||
|
|
||||||
image = image_io.to_bgr(image)
|
image = image_io.to_bgr(image)
|
||||||
det = detection if detection is not None else self.detect(image)
|
det = detection if detection is not None else self.detect(image)
|
||||||
alpha = _alpha_template()
|
engine = self.variant_engine(det)
|
||||||
|
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
||||||
|
if engine is not self and det.detected and det.match_box is not None:
|
||||||
|
from remove_ai_watermarks import region_eraser
|
||||||
|
|
||||||
|
x0, y0, x1, y1 = det.match_box
|
||||||
|
loc = engine.locate(image)
|
||||||
|
glyph_height = y1 - y0 + 1
|
||||||
|
return region_eraser.boxes_to_mask(
|
||||||
|
image.shape[:2],
|
||||||
|
[
|
||||||
|
(
|
||||||
|
max(0, loc.x + x0 - round(1.3 * glyph_height)),
|
||||||
|
loc.y + y0,
|
||||||
|
x1 - x0 + 1 + round(1.3 * glyph_height),
|
||||||
|
glyph_height,
|
||||||
|
)
|
||||||
|
],
|
||||||
|
dilate=radius,
|
||||||
|
)
|
||||||
|
alpha = _text_mark_engine.load_alpha_template(engine.config.asset_name)
|
||||||
if alpha is not None:
|
if alpha is not None:
|
||||||
radius = _FOOTPRINT_DILATE if dilate is None else max(0, dilate)
|
aligned = engine._aligned_alpha_mask(
|
||||||
aligned = self._aligned_alpha_mask(
|
|
||||||
image,
|
image,
|
||||||
det,
|
det,
|
||||||
alpha,
|
alpha,
|
||||||
|
|||||||
@@ -32,7 +32,6 @@ _EBML_VIDEO_EXTENSIONS: frozenset[str] = frozenset({".webm", ".mkv"})
|
|||||||
_RIFF_VIDEO_EXTENSIONS: frozenset[str] = frozenset({".avi"})
|
_RIFF_VIDEO_EXTENSIONS: frozenset[str] = frozenset({".avi"})
|
||||||
_FLV_VIDEO_EXTENSIONS: frozenset[str] = frozenset({".flv"})
|
_FLV_VIDEO_EXTENSIONS: frozenset[str] = frozenset({".flv"})
|
||||||
_REGENERATED_VIDEO_EXTENSIONS: frozenset[str] = _ISOBMFF_VIDEO_EXTENSIONS
|
_REGENERATED_VIDEO_EXTENSIONS: frozenset[str] = _ISOBMFF_VIDEO_EXTENSIONS
|
||||||
_EBML_MAGIC = b"\x1aE\xdf\xa3"
|
|
||||||
|
|
||||||
|
|
||||||
def _require_video_runtime() -> None:
|
def _require_video_runtime() -> None:
|
||||||
@@ -237,17 +236,23 @@ def _video_source(source: str | Path) -> Path:
|
|||||||
if path.suffix.lower() not in VIDEO_EXTENSIONS:
|
if path.suffix.lower() not in VIDEO_EXTENSIONS:
|
||||||
supported = ", ".join(sorted(VIDEO_EXTENSIONS))
|
supported = ", ".join(sorted(VIDEO_EXTENSIONS))
|
||||||
raise ValueError(f"Unsupported video format {path.suffix or '<none>'}; expected one of: {supported}")
|
raise ValueError(f"Unsupported video format {path.suffix or '<none>'}; expected one of: {supported}")
|
||||||
|
from remove_ai_watermarks._internal.ebml import EBML_MAGIC
|
||||||
|
|
||||||
with path.open("rb") as stream:
|
with path.open("rb") as stream:
|
||||||
head = stream.read(12)
|
head = stream.read(12)
|
||||||
suffix = path.suffix.lower()
|
suffix = path.suffix.lower()
|
||||||
matches_container = (
|
matches_container = (
|
||||||
(suffix in _ISOBMFF_VIDEO_EXTENSIONS and len(head) >= 8 and head[4:8] == b"ftyp")
|
(suffix in _ISOBMFF_VIDEO_EXTENSIONS and len(head) >= 8 and head[4:8] == b"ftyp")
|
||||||
or (suffix in _EBML_VIDEO_EXTENSIONS and head.startswith(_EBML_MAGIC))
|
or (suffix in _EBML_VIDEO_EXTENSIONS and head.startswith(EBML_MAGIC))
|
||||||
or (suffix in _RIFF_VIDEO_EXTENSIONS and len(head) >= 12 and head[:4] == b"RIFF" and head[8:12] == b"AVI ")
|
or (suffix in _RIFF_VIDEO_EXTENSIONS and len(head) >= 12 and head[:4] == b"RIFF" and head[8:12] == b"AVI ")
|
||||||
or (suffix in _FLV_VIDEO_EXTENSIONS and head.startswith(b"FLV"))
|
or (suffix in _FLV_VIDEO_EXTENSIONS and head.startswith(b"FLV"))
|
||||||
)
|
)
|
||||||
if not matches_container:
|
if not matches_container:
|
||||||
raise ValueError(f"Video content does not match its {suffix} extension: {path}")
|
raise ValueError(f"Video content does not match its {suffix} extension: {path}")
|
||||||
|
from remove_ai_watermarks.metadata import unreadable_reason
|
||||||
|
|
||||||
|
if (reason := unreadable_reason(path)) is not None:
|
||||||
|
raise ValueError(f"Cannot read video {path}: {reason}")
|
||||||
return path
|
return path
|
||||||
|
|
||||||
|
|
||||||
@@ -377,6 +382,8 @@ def _platform_from_video_metadata(markers: dict[str, str]) -> str | None:
|
|||||||
return _tc260_video_platform(markers.get("aigc_producer", "")) or (
|
return _tc260_video_platform(markers.get("aigc_producer", "")) or (
|
||||||
"China AIGC-labeled content (TC260 standard)"
|
"China AIGC-labeled content (TC260 standard)"
|
||||||
)
|
)
|
||||||
|
if "kling_sei" in markers:
|
||||||
|
return _tc260_video_platform("kling")
|
||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
@@ -712,13 +719,13 @@ def remove_video_batch(
|
|||||||
)
|
)
|
||||||
if output_path.resolve() == directory_path.resolve():
|
if output_path.resolve() == directory_path.resolve():
|
||||||
raise ValueError("Video batch output directory must differ from the source directory")
|
raise ValueError("Video batch output directory must differ from the source directory")
|
||||||
output_path.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
sources = tuple(
|
sources = tuple(
|
||||||
path
|
path
|
||||||
for path in sorted(directory_path.iterdir(), key=lambda candidate: candidate.name.lower())
|
for path in sorted(directory_path.iterdir(), key=lambda candidate: candidate.name.lower())
|
||||||
if path.is_file() and path.suffix.lower() in VIDEO_EXTENSIONS
|
if path.is_file() and path.suffix.lower() in VIDEO_EXTENSIONS
|
||||||
)
|
)
|
||||||
|
if sources:
|
||||||
|
output_path.mkdir(parents=True, exist_ok=True)
|
||||||
items: list[VideoBatchItem] = []
|
items: list[VideoBatchItem] = []
|
||||||
invisible_runtime: VideoVaeRuntime | None = None
|
invisible_runtime: VideoVaeRuntime | None = None
|
||||||
invisible_runtime_error: str | None = None
|
invisible_runtime_error: str | None = None
|
||||||
|
|||||||
@@ -135,7 +135,7 @@ def load_video_vae_runtime(
|
|||||||
dtype = torch.float16 if resolved_device == "cuda" else torch.float32
|
dtype = torch.float16 if resolved_device == "cuda" else torch.float32
|
||||||
log.info("Loading %s on %s", model, resolved_device)
|
log.info("Loading %s on %s", model, resolved_device)
|
||||||
vae = AutoencoderKL.from_pretrained(model, torch_dtype=dtype).to(resolved_device)
|
vae = AutoencoderKL.from_pretrained(model, torch_dtype=dtype).to(resolved_device)
|
||||||
vae.eval()
|
vae.train(False)
|
||||||
vae.enable_slicing()
|
vae.enable_slicing()
|
||||||
scaling_factor = float(vae.config.scaling_factor)
|
scaling_factor = float(vae.config.scaling_factor)
|
||||||
log.info("Latent scaling factor %.5f", scaling_factor)
|
log.info("Latent scaling factor %.5f", scaling_factor)
|
||||||
|
|||||||
@@ -24,13 +24,15 @@ Entries:
|
|||||||
- ``qwen`` -- Alibaba Cloud Qwen "千问AI生成" text or three-lobe symbol, bottom-right.
|
- ``qwen`` -- Alibaba Cloud Qwen "千问AI生成" text or three-lobe symbol, bottom-right.
|
||||||
- ``kling`` -- Kuaishou Kling AI "可灵AI 3.0" / "KlingAI 3.0" strip, bottom-right.
|
- ``kling`` -- Kuaishou Kling AI "可灵AI 3.0" / "KlingAI 3.0" strip, bottom-right.
|
||||||
- ``yuanbao`` -- Tencent Yuanbao "元宝 / AI生成" two-line mark, bottom-right.
|
- ``yuanbao`` -- Tencent Yuanbao "元宝 / AI生成" two-line mark, bottom-right.
|
||||||
- ``samsung`` -- Samsung Galaxy AI "Contenuti generati dall'AI" strip, bottom-left.
|
- ``dola`` -- ByteDance Dola "Dola AI" image disclosure, bottom-right.
|
||||||
- ``jimeng_pill`` -- Jimeng-basic "AI生成" pill, top-left (capture-less).
|
- ``workbuddy`` -- Tencent WorkBuddy "AI生成 / WORKBUDDY" disclosure, bottom-right.
|
||||||
|
- ``samsung`` -- Samsung Galaxy AI Italian or Korean disclosure, bottom-left.
|
||||||
|
- ``jimeng_pill`` -- Jimeng-basic "AI生成" pill or solid label, top-left.
|
||||||
- ``runninghub`` -- RunningHub "RunningHub AI生成" text, top-left (gray front-end).
|
- ``runninghub`` -- RunningHub "RunningHub AI生成" text, top-left (gray front-end).
|
||||||
- ``baidu`` -- Baidu "百度 AI生成" text + white tag, bottom-right.
|
- ``baidu`` -- Baidu "百度 AI生成" text + white tag, bottom-right.
|
||||||
- ``liblib`` -- LiblibAI "LiblibAI" wordmark, bottom-center.
|
- ``liblib`` -- LiblibAI "LiblibAI" wordmark, bottom-center.
|
||||||
- ``liblib_pill`` -- LiblibAI compact "AI生成" pill, top-left.
|
- ``liblib_pill`` -- LiblibAI compact "AI生成" pill, top-left.
|
||||||
- ``microsoft`` -- one measured Microsoft white AI-badge variant, top-right.
|
- ``microsoft`` -- measured Microsoft white AI badge or beige text pill, top-right.
|
||||||
- ``generic_ai_label`` -- brand-less bare "AI生成" TC260 label (no vendor wordmark),
|
- ``generic_ai_label`` -- brand-less bare "AI生成" TC260 label (no vendor wordmark),
|
||||||
bottom-right; the fallback for a compliance stamp not tied to any tuned mark above.
|
bottom-right; the fallback for a compliance stamp not tied to any tuned mark above.
|
||||||
- ``openart`` -- OpenArt "OpenArt" wordmark (bowtie/infinity icon + brand name),
|
- ``openart`` -- OpenArt "OpenArt" wordmark (bowtie/infinity icon + brand name),
|
||||||
@@ -448,6 +450,8 @@ _ENGINE_CLASS: dict[str, tuple[str, str]] = {
|
|||||||
"jimeng": ("jimeng_engine", "JimengEngine"),
|
"jimeng": ("jimeng_engine", "JimengEngine"),
|
||||||
"qwen": ("qwen_engine", "QwenEngine"),
|
"qwen": ("qwen_engine", "QwenEngine"),
|
||||||
"kling": ("kling_engine", "KlingEngine"),
|
"kling": ("kling_engine", "KlingEngine"),
|
||||||
|
"dola": ("dola_engine", "DolaEngine"),
|
||||||
|
"workbuddy": ("workbuddy_engine", "WorkBuddyEngine"),
|
||||||
"yuanbao": ("yuanbao_engine", "YuanbaoEngine"),
|
"yuanbao": ("yuanbao_engine", "YuanbaoEngine"),
|
||||||
"samsung": ("samsung_engine", "SamsungEngine"),
|
"samsung": ("samsung_engine", "SamsungEngine"),
|
||||||
"jimeng_pill": ("pill_engine", "PillEngine"),
|
"jimeng_pill": ("pill_engine", "PillEngine"),
|
||||||
@@ -641,29 +645,44 @@ def _text_mark(
|
|||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# ── Capture-less mark: the Jimeng-basic "AI生成" pill (top-left) ──
|
# ── Jimeng-basic "AI生成" pill or solid label (top-left) ──
|
||||||
# Detection is edge-NCC of a synthetic silhouette; the mask is a fixed top-left
|
# The weak outline uses synthetic edge-NCC and fixed geometry. The verified solid
|
||||||
# geometry box (see pill_engine). Removal is the same localize -> fill as the rest.
|
# label uses its own contrast template and detector-aligned mask (see pill_engine).
|
||||||
def _pill_detect(image: NDArray[Any], *, provenance: bool = False) -> MarkDetection:
|
def _pill_detect(image: NDArray[Any], *, provenance: bool = False) -> MarkDetection:
|
||||||
del provenance # the pill detector is provenance-independent; its relaxation lives entirely in _keep_pill
|
d = _engine("jimeng_pill").detect(image, provenance=provenance)
|
||||||
d = _engine("jimeng_pill").detect(image)
|
return MarkDetection(
|
||||||
return MarkDetection("jimeng_pill", "Jimeng AI生成 pill", "top-left", d.detected, d.confidence, d.region)
|
"jimeng_pill",
|
||||||
|
"Jimeng AI生成 pill",
|
||||||
|
"top-left",
|
||||||
|
d.detected,
|
||||||
|
d.confidence,
|
||||||
|
d.region,
|
||||||
|
engine_detection=d,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def _pill_detect_both(image: NDArray[Any]) -> tuple[MarkDetection, MarkDetection]:
|
def _pill_detect_both(image: NDArray[Any]) -> tuple[MarkDetection, MarkDetection]:
|
||||||
# The pill detector is provenance-independent (`_pill_detect` discards the flag), so
|
strict, relaxed = _engine("jimeng_pill").detect_both(image)
|
||||||
# one call answers both levels. MarkDetection is frozen, so sharing it is safe.
|
|
||||||
d = _pill_detect(image)
|
def wrap(detection: Any) -> MarkDetection:
|
||||||
return d, d
|
return MarkDetection(
|
||||||
|
"jimeng_pill",
|
||||||
|
"Jimeng AI生成 pill",
|
||||||
|
"top-left",
|
||||||
|
detection.detected,
|
||||||
|
detection.confidence,
|
||||||
|
detection.region,
|
||||||
|
engine_detection=detection,
|
||||||
|
)
|
||||||
|
|
||||||
|
return wrap(strict), wrap(relaxed)
|
||||||
|
|
||||||
|
|
||||||
def _pill_mask(
|
def _pill_mask(
|
||||||
image: NDArray[Any], *, force: bool = False, detection: MarkDetection | None = None
|
image: NDArray[Any], *, force: bool = False, detection: MarkDetection | None = None
|
||||||
) -> NDArray[Any] | None:
|
) -> NDArray[Any] | None:
|
||||||
# The pill mask is a fixed top-left geometry box, independent of the detection;
|
engine_detection = detection.engine_detection if detection is not None else None
|
||||||
# accepted for the uniform _mask signature.
|
return _engine("jimeng_pill").footprint_mask(image, force=force, detection=engine_detection)
|
||||||
del detection
|
|
||||||
return _engine("jimeng_pill").footprint_mask(image, force=force)
|
|
||||||
|
|
||||||
|
|
||||||
def _pill_features(image: NDArray[Any]) -> dict[str, float]:
|
def _pill_features(image: NDArray[Any]) -> dict[str, float]:
|
||||||
@@ -732,6 +751,24 @@ _REGISTRY: tuple[KnownMark, ...] = (
|
|||||||
platform="Tencent Yuanbao (visible 元宝 / AI生成 mark detected)",
|
platform="Tencent Yuanbao (visible 元宝 / AI生成 mark detected)",
|
||||||
manufacturer="tencent",
|
manufacturer="tencent",
|
||||||
),
|
),
|
||||||
|
_text_mark(
|
||||||
|
"dola",
|
||||||
|
"Dola AI image wordmark",
|
||||||
|
"bottom-right",
|
||||||
|
platform="Dola AI (visible Dola AI disclosure)",
|
||||||
|
manufacturer="bytedance",
|
||||||
|
label_regime=None,
|
||||||
|
provenance_signals=(),
|
||||||
|
),
|
||||||
|
_text_mark(
|
||||||
|
"workbuddy",
|
||||||
|
"AI生成 / WORKBUDDY disclosure",
|
||||||
|
"bottom-right",
|
||||||
|
platform="WorkBuddy (visible AI content disclosure)",
|
||||||
|
manufacturer="tencent",
|
||||||
|
label_regime=None,
|
||||||
|
provenance_signals=(),
|
||||||
|
),
|
||||||
# Samsung Galaxy AI is a device editing marker (samsung_genai), not a TC260 label.
|
# Samsung Galaxy AI is a device editing marker (samsung_genai), not a TC260 label.
|
||||||
_text_mark(
|
_text_mark(
|
||||||
"samsung",
|
"samsung",
|
||||||
@@ -940,8 +977,14 @@ def _pill_suppressors() -> set[str]:
|
|||||||
return {m.key for m in _REGISTRY if m.manufacturer == pill.manufacturer and m.product != pill.product}
|
return {m.key for m in _REGISTRY if m.manufacturer == pill.manufacturer and m.product != pill.product}
|
||||||
|
|
||||||
|
|
||||||
def _keep_pill(keys: set[str], *, provenance: frozenset[str], footprint_flat: bool) -> bool:
|
def _keep_pill(
|
||||||
"""Whether to auto-remove the capture-less 'AI生成' pill given the fired marks.
|
keys: set[str],
|
||||||
|
*,
|
||||||
|
provenance: frozenset[str],
|
||||||
|
footprint_flat: bool,
|
||||||
|
verified_label: bool = False,
|
||||||
|
) -> bool:
|
||||||
|
"""Whether to auto-remove the ``AI生成`` pill or verified label.
|
||||||
|
|
||||||
Pure decision (the flatness feature is precomputed at perception time and passed
|
Pure decision (the flatness feature is precomputed at perception time and passed
|
||||||
in). The pill detector is weak and metadata/intent confirms the platform, not pill
|
in). The pill detector is weak and metadata/intent confirms the platform, not pill
|
||||||
@@ -950,23 +993,24 @@ def _keep_pill(keys: set[str], *, provenance: frozenset[str], footprint_flat: bo
|
|||||||
* bottom-right "★ 即梦AI" wordmark fired -> ~94% precise, and it survives
|
* bottom-right "★ 即梦AI" wordmark fired -> ~94% precise, and it survives
|
||||||
metadata-STRIPPED images: remove the pill unrestricted;
|
metadata-STRIPPED images: remove the pill unrestricted;
|
||||||
* TC260 metadata confirms Jimeng (``"jimeng" in provenance``, no wordmark) -> remove ONLY when the
|
* TC260 metadata confirms Jimeng (``"jimeng" in provenance``, no wordmark) -> remove ONLY when the
|
||||||
top-left footprint is flat enough for an invisible fill (``footprint_flat``),
|
solid label template verifies the text, or when the weaker outline's top-left
|
||||||
so real flat-scene pills (and harmless flat false fires) are cleaned while the
|
footprint is flat enough for an invisible fill (``footprint_flat``). Real
|
||||||
damaging textured false fires are left untouched.
|
flat-scene pills are cleaned while damaging textured false fires are left.
|
||||||
A Doubao image is TC260 too but is not Jimeng-basic, so its mark suppresses the
|
A Doubao image is TC260 too but is not Jimeng-basic, so its mark suppresses the
|
||||||
pill. Marks from other TC260 manufacturers are unrelated and do not participate in
|
pill. The independently attributed LiblibAI pill also wins over the generic
|
||||||
this ByteDance-family decision.
|
``AI生成`` label shape. Marks from other TC260 manufacturers are unrelated and do
|
||||||
|
not participate in this ByteDance-family decision.
|
||||||
No confirmation at all -> never remove (blocks false fires on non-Jimeng content).
|
No confirmation at all -> never remove (blocks false fires on non-Jimeng content).
|
||||||
|
|
||||||
The suppressor set is derived from the manufacturer and product fields rather than
|
The suppressor set is derived from the manufacturer and product fields rather than
|
||||||
from the shared TC260 standard. Marks from Google, Samsung, or another TC260
|
from the shared TC260 standard. Marks from Google, Samsung, or another TC260
|
||||||
manufacturer cannot enable or veto this ByteDance-specific arm."""
|
manufacturer cannot enable or veto this ByteDance-specific arm."""
|
||||||
if _pill_suppressors() & keys:
|
if "liblib_pill" in keys or _pill_suppressors() & keys:
|
||||||
return False
|
return False
|
||||||
if "jimeng" in keys:
|
if "jimeng" in keys:
|
||||||
return True
|
return True
|
||||||
if "jimeng" in provenance:
|
if "jimeng" in provenance:
|
||||||
return footprint_flat
|
return footprint_flat or verified_label
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
|
||||||
@@ -990,6 +1034,8 @@ def _build_candidates(image: NDArray[Any]) -> list[Candidate]:
|
|||||||
continue
|
continue
|
||||||
strict, relaxed = m.detect_both(image)
|
strict, relaxed = m.detect_both(image)
|
||||||
feats = m.features(image) if (strict.detected or relaxed.detected) else {}
|
feats = m.features(image) if (strict.detected or relaxed.detected) else {}
|
||||||
|
if relaxed.engine_detection is not None and getattr(relaxed.engine_detection, "verified_label", False):
|
||||||
|
feats["verified_label"] = 1.0
|
||||||
cands.append(
|
cands.append(
|
||||||
Candidate(
|
Candidate(
|
||||||
m.key,
|
m.key,
|
||||||
@@ -1010,7 +1056,7 @@ def decide(candidates: list[Candidate], context: Context) -> list[Decision]:
|
|||||||
|
|
||||||
All policy lives here, in one place: per-mark trust resolution (:func:`resolve_trust`,
|
All policy lives here, in one place: per-mark trust resolution (:func:`resolve_trust`,
|
||||||
which needs the strict-detected siblings for ``auto`` cross-mark corroboration) and
|
which needs the strict-detected siblings for ``auto`` cross-mark corroboration) and
|
||||||
the capture-less pill gate (:func:`_keep_pill`). No image, no I/O -- so it is unit-testable in isolation and
|
the Jimeng pill/label gate (:func:`_keep_pill`). No image, no I/O -- so it is unit-testable in isolation and
|
||||||
the same decision drives every caller."""
|
the same decision drives every caller."""
|
||||||
strict_keys = {c.key for c in candidates if c.detected_strict}
|
strict_keys = {c.key for c in candidates if c.detected_strict}
|
||||||
fired: list[Decision] = []
|
fired: list[Decision] = []
|
||||||
@@ -1026,7 +1072,13 @@ def decide(candidates: list[Candidate], context: Context) -> list[Decision]:
|
|||||||
if "jimeng_pill" in keys:
|
if "jimeng_pill" in keys:
|
||||||
pill = next(d for d in fired if d.candidate.key == "jimeng_pill")
|
pill = next(d for d in fired if d.candidate.key == "jimeng_pill")
|
||||||
flat = bool(pill.candidate.features.get("footprint_flat", 0.0))
|
flat = bool(pill.candidate.features.get("footprint_flat", 0.0))
|
||||||
if not _keep_pill(keys, provenance=context.provenance, footprint_flat=flat):
|
verified_label = bool(pill.candidate.features.get("verified_label", 0.0))
|
||||||
|
if not _keep_pill(
|
||||||
|
keys,
|
||||||
|
provenance=context.provenance,
|
||||||
|
footprint_flat=flat,
|
||||||
|
verified_label=verified_label,
|
||||||
|
):
|
||||||
fired = [d for d in fired if d.candidate.key != "jimeng_pill"]
|
fired = [d for d in fired if d.candidate.key != "jimeng_pill"]
|
||||||
return fired
|
return fired
|
||||||
|
|
||||||
|
|||||||