Make the release verification actually verify the Space and the node

The HF Space step curled the root for HTTP 200 and then emitted a warning
and SUCCEEDED when it never got one. The documented contract was three other
checks entirely: the requirements pin, the runtime stage, and live identify
and visible calls. None was implemented, which is why the demo sat two
releases behind while every verification run stayed green. Implemented all
three; the pin check fails today, correctly, on 0.34.1 against 0.36.0.

The ComfyUI step read the latest run of the node repo's sync workflow. That
workflow also runs on a daily schedule, so a green nightly satisfied it
whether or not this release was ever synced. The registry does publish a JSON
API, contrary to the comment that justified the proxy, so read the node
directly and assert the active version declares this library release.

Both of those steps warned and continued; only Homebrew failed hard. Now all
of them exit non-zero, and the summary is assembled from lines each step
appends as it passes rather than an unconditional block that claimed five
verified surfaces regardless.

No step needs the GitHub token any more, so pin least-privilege permissions
here and in test.yml.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Victor KuznetsovandClaude Opus 5 committed 2026-09-01 15:33:11 -07:00
1 parent 29de21256b
commit 326840bab4
2 files changed
+122 -47

No files matched your search

+3
View File
@@ -5,6 +5,9 @@ on:
branches: [main]
pull_request:
permissions:
contents: read
concurrency:
group: tests-${{ github.ref }}
cancel-in-progress: true
+119 -47
View File
@@ -17,13 +17,24 @@ on:
workflows: ["Distribute on release"]
types: [completed]
# No step reads the GitHub token any more: the ComfyUI check moved to the
# registry's public API. Keep the job read-only so it cannot write on a repo
# whose default token permission is read/write.
permissions:
contents: read
jobs:
verify:
# Only run on a successful distribute (not on manual dispatch, where the
# version input or PyPI-latest provides the target).
if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.conclusion == 'success' }}
runs-on: ubuntu-latest
timeout-minutes: 10
timeout-minutes: 25
env:
# The Space id appears in the Hub API, the raw-file URL and the app host,
# spelled three different ways. Name it once so a move cannot half-land.
SPACE_REPO: wiltodelta/remove-ai-watermarks
SPACE_URL: https://wiltodelta-remove-ai-watermarks.hf.space
steps:
- name: Resolve target version
@@ -89,63 +100,124 @@ jobs:
echo " Homebrew: formula points to $VERSION"
# -- ComfyUI ------------------------------------------------------------------
- name: Verify ComfyUI sync succeeded
- name: Verify ComfyUI registry node requires this release
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
VERSION: ${{ steps.v.outputs.version }}
run: |
set -euo pipefail
echo "Checking ComfyUI node sync..."
# The registry itself has no public JSON API; the sync workflow in
# the node repo already tests compatibility and publishes only on
# success, so verifying its latest run succeeded is the proxy.
latest=$(gh run list \
--repo wiltodelta/ComfyUI-remove-ai-watermarks \
--workflow sync-library.yml \
--limit 1 \
--json conclusion \
| python3 -c "
import json, sys
runs = json.load(sys.stdin)
r = runs[0] if runs else {}
print(r.get('conclusion', 'unknown'))
")
echo " ComfyUI sync: $latest"
if [ "$latest" != "success" ]; then
echo "::warning::ComfyUI sync latest run is '$latest', not success."
echo " Check: https://github.com/wiltodelta/ComfyUI-remove-ai-watermarks/actions"
else
echo " ComfyUI: sync succeeded, node published"
fi
echo "Checking the ComfyUI Registry node..."
# The registry publishes the node as JSON, including the dependency the
# published version declares. Read that directly instead of proxying
# through the sync workflow's run status: the node repo runs
# sync-library.yml on a daily schedule too, so a green nightly run says
# nothing about whether THIS release was ever synced.
node=$(curl -fsSL "https://api.comfy.org/nodes/remove-ai-watermarks")
echo "$node" | VERSION="$VERSION" python3 -c "
import json, os, sys
version = os.environ['VERSION']
latest = json.load(sys.stdin).get('latest_version') or {}
deps = latest.get('dependencies') or []
floor = f'>={version}'
if latest.get('status') != 'NodeVersionStatusActive':
sys.exit(f\"::error::ComfyUI node {latest.get('version')} is {latest.get('status')}, not active\")
if not any(floor in d for d in deps):
sys.exit(
f'::error::ComfyUI node {latest.get(\"version\")} does not require '
f'{version}; it declares {deps}'
)
print(f' ComfyUI: node {latest[\"version\"]} requires {version}')
"
echo "- ComfyUI: registry node requires $VERSION" >> "$GITHUB_STEP_SUMMARY"
# -- HF Space -------------------------------------------------------------------
- name: Verify HF Space is alive
- name: Verify the HF Space serves this release
env:
VERSION: ${{ steps.v.outputs.version }}
run: |
set -euo pipefail
echo "Checking HF Space..."
space_url="https://wiltodelta-remove-ai-watermarks.hf.space"
# The Space may be sleeping after a factory rebuild; wake it.
for attempt in $(seq 1 12); do
status=$(curl -s -o /dev/null -w "%{http_code}" --max-time 30 "$space_url/" || echo "000")
if [ "$status" = "200" ]; then
echo " HF Space: alive (HTTP $status)"
exit 0
fi
echo " HF Space: HTTP $status (attempt $attempt), waiting for rebuild..."
echo "Checking the HF Space pin..."
# The Space pin bump is a MANUAL step (see docs/release-and-distribution.md):
# distribute.yml only factory-rebuilds whatever is already pinned. Failing
# here is the point -- a demo left on an older release is an incomplete
# release, and only this check can say so.
reqs=$(curl -fsSL "https://huggingface.co/spaces/$SPACE_REPO/raw/main/requirements.txt")
pinned=$(printf '%s\n' "$reqs" | sed -n 's/^remove-ai-watermarks==\([0-9][0-9a-zA-Z.]*\).*/\1/p' | head -1)
if [ -z "$pinned" ]; then
echo "::error::Space requirements.txt has no remove-ai-watermarks== pin"
exit 1
fi
if [ "$pinned" != "$VERSION" ]; then
echo "::error::HF Space is pinned to $pinned, not $VERSION."
echo " Bump remove-ai-watermarks in wiltodelta/raiw-hf-space (pyproject.toml,"
echo " uv lock, re-exported requirements.txt) and push; sync-to-hf.yml mirrors it."
exit 1
fi
echo " HF Space: requirements pin is $pinned"
- name: Verify the HF Space is running and answering
env:
VERSION: ${{ steps.v.outputs.version }}
run: |
set -euo pipefail
# A factory rebuild leaves the Space BUILDING for a while; wait for the
# runtime stage the API reports rather than for a bare HTTP 200, which a
# build page also returns.
for attempt in $(seq 1 20); do
stage=$(curl -fsSL "https://huggingface.co/api/spaces/$SPACE_REPO" \
| python3 -c "import json,sys;print((json.load(sys.stdin).get('runtime') or {}).get('stage',''))")
echo " attempt $attempt: runtime stage=$stage"
[ "$stage" = "RUNNING" ] && break
case "$stage" in
BUILD_ERROR|RUNTIME_ERROR|CONFIG_ERROR)
echo "::error::HF Space runtime stage is $stage"; exit 1 ;;
esac
sleep 15
done
echo "::warning::HF Space did not respond with HTTP 200 after 3 minutes."
echo " The Space may still be rebuilding after a factory restart."
echo " Verify manually at $space_url"
if [ "$stage" != "RUNNING" ]; then
echo "::error::HF Space never reached RUNNING (last stage: $stage)"
exit 1
fi
# Alive is not the same as working. Drive the two CPU surfaces the demo
# exists for, against the Space's own committed example, and require a
# real verdict rather than any HTTP 200.
export PATH="$HOME/.local/bin:$PATH"
curl -fsSL -o /tmp/example.png \
"https://huggingface.co/spaces/$SPACE_REPO/resolve/main/examples/doubao-aigc.png"
uv run --no-project --with 'gradio_client>=1,<2' python - <<'PYEOF'
import os
import sys
from gradio_client import Client, handle_file
# The committed example is a 5 MB PNG, and httpx's default 5 s write
# timeout aborts that upload on a slow link before the Space ever sees
# it -- which would read as a broken demo rather than a slow one.
client = Client(os.environ["SPACE_URL"], verbose=False, httpx_kwargs={"timeout": 300})
report = client.predict(path=handle_file("/tmp/example.png"), api_name="/run_identify")
if "Verdict:" not in report:
sys.exit(f"::error::/run_identify returned no verdict: {report[:200]!r}")
print(f" HF Space identify: {report.splitlines()[0]}")
_image, _file, info = client.predict(
path=handle_file("/tmp/example.png"),
sensitivity_label="auto (recommended)",
api_name="/run_visible",
)
if "Removed:" not in info:
sys.exit(f"::error::/run_visible removed nothing on the Doubao example: {info!r}")
print(f" HF Space visible: {info.splitlines()[0]}")
PYEOF
echo "- HF Space: pinned to $VERSION, RUNNING, identify + visible answered" >> "$GITHUB_STEP_SUMMARY"
- name: Summary
env:
VERSION: ${{ steps.v.outputs.version }}
run: |
cat <<EOF
Release $VERSION verified:
PyPI: wheel + sdist present
Clean install: --version returns $VERSION
Homebrew: formula points to $VERSION
ComfyUI: sync succeeded
HF Space: alive and responding
EOF
# Every check above appends its own line to the step summary as it passes,
# and every one of them fails the job instead of warning. So this heading
# can only be reached when all of them actually held -- an unconditional
# "verified" block printed here once claimed surfaces nobody had checked.
sed -i "1i ## Release $VERSION verified" "$GITHUB_STEP_SUMMARY"
cat "$GITHUB_STEP_SUMMARY"