Collapse the duplicated detection path and lift the image pipeline into the library

The visible-mark path had grown three copies of one ladder sweep, four
near-identical `detect` arms, and four hand-rolled `footprint_mask` overrides;
mark knowledge sat in five hand-maintained tables across three modules; and the
flagship `all`/`batch` pipeline existed only in cli.py, written twice with
divergent behavior.

Detection is now one measurement. `_ladder_best` replaces the three sweeps,
`_scan`/`_verdict` replace the four arms, and the winning box travels to the
mask on `TextMarkDetection.match_box` instead of being swept a second time.
`detect_both` returns the strict and relaxed verdicts from one scan, which
halves the arbiter's perception cost (260 -> 130 matchTemplate calls on a 2048²
image, verdicts identical field for field). A per-mark demotion goes in the new
`_post_gate` hook, never in a `detect` override -- an override is invisible to
the single-pass path, which is how the RunningHub and Yuanbao anchor gates
briefly stopped applying.

Everything about a mark is now one registry row: product, label regime, the
platform sentence `identify` reports, the metadata signals that confirm it, and
its TC260 producer codes. `identify._VISIBLE_MARK_PLATFORM`, the signal mapping
in `api.visible_provenance`, `_PRODUCT_OF` and the pill veto are derived from
those rows.

`api.remove_all` / `api.remove_batch` are the library form of the `all` and
`batch` commands; the CLI is a wrapper that owns console text and exit codes.
Progress is a `(stage, detail)` pair of stable tokens, so the CLI keys its
wording off structure rather than parsing the library's prose back.

Two intentional behavior changes, both verified against a recorded 811-image
sample of detector verdicts, removal-mask hashes, arbiter decisions and
`identify` reports:

  * A TC260 label now relaxes the vendor its `ContentProducer` names rather than
    ByteDance's pair on every China-AIGC image. 333 of 811 samples move; on 185
    of them the previously relaxed pair was simply the wrong vendor, and the
    mark actually present never reached the relaxed gate its own
    `provenance_ncc_factor` was calibrated for.
  * A confident LibLibAI detection suppresses the Jimeng pill, like every other
    TC260 product's mark. It was registered alongside RunningHub and Baidu, both
    of which were added to the hand-written veto list, and it was not. 1 sample
    moves, and it is exactly the co-firing case.

Nothing else in that record changes: detector verdicts, mask hashes and
`identify` verdicts are byte-identical, and all 200 calibration constants are
untouched.

Also: `aigc_label` and friends plus `extract_c2pa_info` are memoized on
(path, mtime_ns, size) -- size because this package rewrites in place; the
native TC260 container readers route on magic bytes instead of the file
extension, so a mislabeled AVI or FLV is no longer invisible; `identify` shares
one pixel decode between the DWT-DCT and visible stages (TrustMark keeps its own
Pillow decode, which is not substitutable); and the six `stabilize_*` video
wrappers collapse into one policy table.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor Kuznetsov
2026-08-03 22:49:45 -07:00
co-authored by Claude Opus 5
parent 480f478484
commit 78d9e81d0f
48 changed files with 3177 additions and 1298 deletions
+5 -2
View File
@@ -1,7 +1,7 @@
"""How much recall is the coarse scale ladder costing, and what would a denser one cost?
THE FINDING THIS MEASURES
`_tophat_score` sweeps three rungs -- (0.8, 1.0, 1.25) -- and the `binary` front-end
`_ladder_best` sweeps three rungs -- (0.8, 1.0, 1.25) -- and the `binary` front-end
sweeps none at all. Measured on stamped marks over controlled backgrounds
(`scripts/detector_response.py`), the response is a COMB: doubao scores 0.99 exactly at
each rung and collapses to 0.37-0.48 between them, against a 0.50 gate. So a mark whose
@@ -93,7 +93,10 @@ def wilson(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
def score_at_scales(engine: Any, image: np.ndarray, scales: tuple[float, ...]) -> dict[float, float]:
"""`_tophat_score`'s inner loop, opened up so the ladder is a parameter.
"""`_ladder_best`'s inner loop, opened up so the ladder is a parameter.
A DELIBERATE second copy of the sweep: it exists to measure ladders the shipped
engine does not have, so it must not be folded back into `_ladder_best`.
Deliberately reaches into the engine (`tophat_response`, `_glyph_silhouette`): a
measurement script may, product code may not. Kept a faithful copy of the shipped
+4 -11
View File
@@ -62,17 +62,10 @@ OUT = REPO / ".local-eval" / "vendor-cohorts.jsonl"
FIRED = REPO / ".local-eval" / "visible-positives.jsonl"
SHEET_DIR = REPO / ".local-eval" / "vendor-cohort-sheets"
# A producer code is `001` + `1` + USCC(18) + a 5-digit app/product suffix, so two
# codes sharing the USCC are the same legal entity registering different products.
# Slicing is defensive: anything not matching the layout is grouped by its raw value.
_USCC_START, _USCC_END = 4, 22
def uscc_of(code: str) -> str:
"""The 18-char Unified Social Credit Code embedded in a TC260 producer code."""
if len(code) >= _USCC_END and code[:3] == "001":
return code[_USCC_START:_USCC_END]
return code
# `uscc_of` moved into the library (`metadata.uscc_of`) when the USCC -> vendor table
# started driving `api.visible_provenance`; this script must group by the same rule the
# product uses, so it imports rather than reimplements it.
from remove_ai_watermarks.metadata import uscc_of # noqa: E402
def _one(path_str: str) -> dict[str, Any] | None:
+3 -3
View File
@@ -97,7 +97,7 @@ def _score(args: ScoreArgs) -> dict[str, Any] | None:
eng = TextMarkEngine(build_config(asset, name, basis, overrides))
loc = eng.locate(img)
try:
score, box = eng._tophat_best(img, loc)
score, box = eng._ladder_best(img, loc)
except Exception:
return None
return {"path": path_str, "score": round(float(score), 4), "box": box}
@@ -225,7 +225,7 @@ def sheets(pos: list[dict[str, Any]], name: str, per_sheet: int = 24) -> None:
# genuinely unknown, which is the one case a dense ladder earns its cost -- but it also
# hands clean corners extra chances to match, so it must never set a gate.
_FIT_SCALES = tuple(round(0.4 * (1.03**i), 4) for i in range(80)) # 0.40 .. ~4.1
# The ladder the product actually ships (`_text_mark_engine._tophat_best`).
# The ladder the product actually ships (`_text_mark_engine._ladder_best`).
_SHIPPED_LADDER = (0.8, 1.0, 1.25)
@@ -460,7 +460,7 @@ def _cross_score(args: tuple[str, Any, Any]) -> dict[str, Any] | None:
eng = TextMarkEngine(cfg)
loc = eng.locate(img)
try:
score, _ = eng._tophat_best(img, loc)
score, _ = eng._ladder_best(img, loc)
except Exception:
return None
out[key] = round(float(score), 4)