mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-08-27 12:53:00 +02:00
External audit (@tg12, May 18) found that backend/services/updater.py silently skipped all SHA-256 integrity verification whenever the MESH_UPDATE_SHA256 env var was unset — which is the default. Nothing in any install doc tells operators to set it, so practically every deployment was running the auto-updater with zero integrity check. That made GitHub release pipeline compromise a single-step path to arbitrary code execution on every node that auto-updates. Investigation surfaced a deeper bug too: the updater downloads zipball_url (GitHub's auto-generated source archive) but the maintainer's release process publishes SHA256SUMS.txt for a separate named asset (ShadowBroker_v*.zip). So even if MESH_UPDATE_SHA256 WERE set, operators had no published digest to compare against — the file they were downloading wasn't the file the maintainer had signed. This PR fixes both issues with the same multi-source verification chain we shipped for the Tor bundle in PR #261: backend/services/updater.py _download_release() now prefers a maintainer-signed release asset matching ShadowBroker_v*.zip over zipball_url. Captures the SHA256SUMS.txt asset URL when present. _validate_zip_hash() rewritten as a four-source chain: 1. MESH_UPDATE_SHA256 env var (operator override, preserved) 2. SHA256SUMS.txt asset published with the release (primary — the maintainer's release process already publishes this) 3. Baked-in backend/data/release_digests.json (second line of defense for releases that lack the SHA256SUMS asset, or when the asset can't be fetched at update time) 4. HTTPS-only fallback with a loud warning (preserves the auto- update flow during transient outages) Mismatch from any source that DID respond is fatal — the update is refused and the existing install keeps running. Only the "no source reachable at all" case falls back to HTTPS-only. _fetch_sha256sums() new — fetches and parses a standard SHA256SUMS.txt asset. Handles both "<digest> <name>" and binary- marker "<digest> *<name>" formats. Tolerant to comments, blank lines, and malformed entries. backend/data/release_digests.json (new) Baked-in digest list keyed by release tag. Seeded with the v0.9.79 entries copied from the published SHA256SUMS.txt: ShadowBroker_v0.9.79.zip = f6877c1d6661... ShadowBroker_0.9.79_x64-setup.exe = f7b676ada45c... ShadowBroker_0.9.79_x64_en-US.msi = e0713c3cdda1... Whitelisted in .gitignore alongside the other static reference data files (kiwisdr_directory.json, tor_bundle_digests.json, aisstream_spki_pins.json). backend/tests/test_update_integrity_chain.py (new, 16 tests) - Each source matches → success, identifies which source verified - Each source mismatches → RuntimeError "mismatch" - No source reachable → https-only fallback with loud warning - Env override beats all other sources (preserved precedence) - SHA256SUMS.txt parser handles standard, binary-marker, comments, and network-failure cases Validation: pytest backend/tests/test_update_integrity_chain.py → 16 passed pytest (all 15 security test files together) → 105 passed UX impact: zero. Normal auto-update flow is unchanged for legitimate releases (path 2 catches everything because the release publishes SHA256SUMS.txt). Transient network failures during update gracefully fall through to path 3 then path 4 — no operator intervention needed. The only user-visible behavior change is in the compromised-release case, where the update is now refused instead of silently applied. Credit: @tg12 for the original bug report and the specific call-out that MESH_UPDATE_SHA256 was unreachable by default operators. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.7
parent
d0299fc0a0
commit
7f96151e56
@@ -101,6 +101,10 @@ backend/data/*
|
|||||||
# Issue #258: SPKI pins for stream.aisstream.io so we can survive upstream
|
# Issue #258: SPKI pins for stream.aisstream.io so we can survive upstream
|
||||||
# Let's Encrypt renewal failures without disabling TLS validation entirely.
|
# Let's Encrypt renewal failures without disabling TLS validation entirely.
|
||||||
!backend/data/aisstream_spki_pins.json
|
!backend/data/aisstream_spki_pins.json
|
||||||
|
# Issue #231: pinned SHA-256 digests for known release archives. Used by
|
||||||
|
# the self-updater as a second-line integrity check when the release's
|
||||||
|
# SHA256SUMS.txt asset can't be fetched.
|
||||||
|
!backend/data/release_digests.json
|
||||||
|
|
||||||
# OS generated files
|
# OS generated files
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|||||||
@@ -0,0 +1,40 @@
|
|||||||
|
{
|
||||||
|
"_comment": [
|
||||||
|
"Baked-in SHA-256 digests for known Shadowbroker release archives.",
|
||||||
|
"",
|
||||||
|
"Issue #231: the self-updater previously skipped integrity verification",
|
||||||
|
"entirely whenever the MESH_UPDATE_SHA256 env var was unset (which is the",
|
||||||
|
"default — nothing in the install docs tells operators to set it). That",
|
||||||
|
"made the auto-update a supply-chain RCE on any compromise of the GitHub",
|
||||||
|
"release pipeline.",
|
||||||
|
"",
|
||||||
|
"The fix uses a multi-source verification chain mirroring the Tor bundle",
|
||||||
|
"digest approach in #201:",
|
||||||
|
"",
|
||||||
|
" 1. MESH_UPDATE_SHA256 env var (operator override, preserved)",
|
||||||
|
" 2. SHA256SUMS.txt asset published alongside each release (primary —",
|
||||||
|
" the maintainer's release process already publishes this)",
|
||||||
|
" 3. This baked-in digest list (second line of defense for releases",
|
||||||
|
" missing a SHA256SUMS asset, or when the asset can't be fetched)",
|
||||||
|
" 4. HTTPS-only fallback with a loud warning (preserves auto-update",
|
||||||
|
" flow during transient outages so users don't get stuck)",
|
||||||
|
"",
|
||||||
|
"Mismatch from a source that DID respond is fatal — the update is",
|
||||||
|
"refused and the existing install keeps running. Only the 'no source",
|
||||||
|
"reachable at all' case falls back to HTTPS-only.",
|
||||||
|
"",
|
||||||
|
"Format: each entry is keyed by release tag and maps asset filenames",
|
||||||
|
"to their canonical SHA-256 digest (hex, lowercase). The updater",
|
||||||
|
"compares the locally-computed digest of the downloaded asset against",
|
||||||
|
"the value here.",
|
||||||
|
"",
|
||||||
|
"When the maintainer ships a new release, add its digests here BEFORE",
|
||||||
|
"removing the old ones so operators on the old code still validate",
|
||||||
|
"against the previous entries during the transition."
|
||||||
|
],
|
||||||
|
"v0.9.79": {
|
||||||
|
"ShadowBroker_v0.9.79.zip": "f6877c1d66614525315ea82636ce9f7b41178332c4dbf90d27431a1ea1d9cd47",
|
||||||
|
"ShadowBroker_0.9.79_x64-setup.exe": "f7b676ada45cac7da05868b0a353678c9ee700e3abcf456a7c0c038c36da446f",
|
||||||
|
"ShadowBroker_0.9.79_x64_en-US.msi": "e0713c3cdda184cfbea750bfac0d62a35678fec00847e6476f2cac8e7e42046e"
|
||||||
|
}
|
||||||
|
}
|
||||||
+232
-14
@@ -6,9 +6,11 @@ Public API:
|
|||||||
schedule_restart(project_root) (spawn detached start script, then exit)
|
schedule_restart(project_root) (spawn detached start script, then exit)
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
import json
|
||||||
import os
|
import os
|
||||||
import sys
|
import sys
|
||||||
import logging
|
import logging
|
||||||
|
import re
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import tempfile
|
import tempfile
|
||||||
@@ -29,6 +31,19 @@ DOCKER_UPDATE_COMMANDS = (
|
|||||||
"docker compose pull && docker compose up -d"
|
"docker compose pull && docker compose up -d"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Issue #231: baked-in release digests. Loaded lazily, used as a fallback
|
||||||
|
# verification source when the release's SHA256SUMS.txt asset can't be
|
||||||
|
# fetched (e.g. transient network failure during update).
|
||||||
|
_RELEASE_DIGESTS_FILE = (
|
||||||
|
Path(__file__).resolve().parent.parent / "data" / "release_digests.json"
|
||||||
|
)
|
||||||
|
# Pattern for the maintainer's signed source-archive release asset. This
|
||||||
|
# is the file we prefer over the auto-generated ``zipball_url`` because
|
||||||
|
# the maintainer's build process publishes it with a matching entry in
|
||||||
|
# SHA256SUMS.txt — the zipball does not have a signed digest.
|
||||||
|
_SOURCE_ASSET_PATTERN = re.compile(r"^ShadowBroker_v\d", re.IGNORECASE)
|
||||||
|
_SHA256SUMS_ASSET_NAME = "SHA256SUMS.txt"
|
||||||
|
|
||||||
|
|
||||||
def _is_docker() -> bool:
|
def _is_docker() -> bool:
|
||||||
"""Detect if we're running inside a Docker container."""
|
"""Detect if we're running inside a Docker container."""
|
||||||
@@ -40,7 +55,6 @@ def _is_docker() -> bool:
|
|||||||
except (FileNotFoundError, PermissionError):
|
except (FileNotFoundError, PermissionError):
|
||||||
pass
|
pass
|
||||||
return os.environ.get("container") == "docker"
|
return os.environ.get("container") == "docker"
|
||||||
_EXPECTED_SHA256 = os.environ.get("MESH_UPDATE_SHA256", "").strip().lower()
|
|
||||||
_ALLOWED_UPDATE_HOSTS = {
|
_ALLOWED_UPDATE_HOSTS = {
|
||||||
"api.github.com",
|
"api.github.com",
|
||||||
"codeload.github.com",
|
"codeload.github.com",
|
||||||
@@ -119,7 +133,16 @@ def _validate_update_url(url: str, *, allow_release_page: bool = False) -> str:
|
|||||||
# ---------------------------------------------------------------------------
|
# ---------------------------------------------------------------------------
|
||||||
def _download_release(temp_dir: str) -> tuple:
|
def _download_release(temp_dir: str) -> tuple:
|
||||||
"""Fetch latest release info and download the source zip archive.
|
"""Fetch latest release info and download the source zip archive.
|
||||||
Returns (zip_path, version_tag, download_url, release_url).
|
|
||||||
|
Issue #231: prefer the maintainer's signed release asset (matching
|
||||||
|
``ShadowBroker_v*.zip``) over the auto-generated ``zipball_url``,
|
||||||
|
because the maintainer's release process publishes a matching entry
|
||||||
|
in SHA256SUMS.txt for the named asset but NOT for the zipball.
|
||||||
|
|
||||||
|
Returns (zip_path, version_tag, download_url, release_url, asset_name,
|
||||||
|
sha256sums_url) — the last two are empty strings when the release
|
||||||
|
doesn't publish a signed asset, falling back to the legacy zipball
|
||||||
|
path.
|
||||||
"""
|
"""
|
||||||
logger.info("Fetching latest release info from GitHub...")
|
logger.info("Fetching latest release info from GitHub...")
|
||||||
_validate_update_url(GITHUB_RELEASES_URL)
|
_validate_update_url(GITHUB_RELEASES_URL)
|
||||||
@@ -131,9 +154,42 @@ def _download_release(temp_dir: str) -> tuple:
|
|||||||
tag = release.get("tag_name", "unknown")
|
tag = release.get("tag_name", "unknown")
|
||||||
release_url = str(release.get("html_url") or GITHUB_RELEASES_PAGE_URL).strip()
|
release_url = str(release.get("html_url") or GITHUB_RELEASES_PAGE_URL).strip()
|
||||||
_validate_update_url(release_url, allow_release_page=True)
|
_validate_update_url(release_url, allow_release_page=True)
|
||||||
zip_url = str(release.get("zipball_url") or "").strip()
|
|
||||||
if not zip_url:
|
# Prefer the maintainer-signed release asset. Fall back to the
|
||||||
raise RuntimeError("Latest release is missing a source archive URL")
|
# auto-generated zipball if the release doesn't publish one.
|
||||||
|
assets = release.get("assets") or []
|
||||||
|
asset_name = ""
|
||||||
|
asset_url = ""
|
||||||
|
sha256sums_url = ""
|
||||||
|
for a in assets:
|
||||||
|
name = str(a.get("name") or "").strip()
|
||||||
|
download = str(a.get("browser_download_url") or "").strip()
|
||||||
|
if not name or not download:
|
||||||
|
continue
|
||||||
|
if _SOURCE_ASSET_PATTERN.match(name) and name.lower().endswith(".zip"):
|
||||||
|
asset_name = name
|
||||||
|
asset_url = download
|
||||||
|
elif name == _SHA256SUMS_ASSET_NAME:
|
||||||
|
sha256sums_url = download
|
||||||
|
|
||||||
|
if asset_url:
|
||||||
|
zip_url = asset_url
|
||||||
|
logger.info(
|
||||||
|
"Using signed release asset %s (sha256sums=%s)",
|
||||||
|
asset_name,
|
||||||
|
"yes" if sha256sums_url else "no",
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
zip_url = str(release.get("zipball_url") or "").strip()
|
||||||
|
if not zip_url:
|
||||||
|
raise RuntimeError("Latest release is missing a source archive URL")
|
||||||
|
logger.warning(
|
||||||
|
"Release does not publish a signed ShadowBroker_v*.zip asset — "
|
||||||
|
"falling back to auto-generated zipball_url. Integrity will be "
|
||||||
|
"verified against the baked-in release_digests.json (if present) "
|
||||||
|
"or HTTPS-only otherwise."
|
||||||
|
)
|
||||||
|
|
||||||
_validate_update_url(zip_url)
|
_validate_update_url(zip_url)
|
||||||
|
|
||||||
logger.info(f"Downloading {zip_url} ...")
|
logger.info(f"Downloading {zip_url} ...")
|
||||||
@@ -150,19 +206,174 @@ def _download_release(temp_dir: str) -> tuple:
|
|||||||
|
|
||||||
size_mb = os.path.getsize(zip_path) / (1024 * 1024)
|
size_mb = os.path.getsize(zip_path) / (1024 * 1024)
|
||||||
logger.info(f"Downloaded {size_mb:.1f} MB — ZIP validated OK")
|
logger.info(f"Downloaded {size_mb:.1f} MB — ZIP validated OK")
|
||||||
return zip_path, tag, zip_url, release_url
|
return zip_path, tag, zip_url, release_url, asset_name, sha256sums_url
|
||||||
|
|
||||||
|
|
||||||
def _validate_zip_hash(zip_path: str) -> None:
|
def _compute_sha256(zip_path: str) -> str:
|
||||||
if not _EXPECTED_SHA256:
|
"""Return the hex SHA-256 of the file at ``zip_path`` (lowercase)."""
|
||||||
return
|
|
||||||
h = hashlib.sha256()
|
h = hashlib.sha256()
|
||||||
with open(zip_path, "rb") as f:
|
with open(zip_path, "rb") as f:
|
||||||
for chunk in iter(lambda: f.read(1024 * 128), b""):
|
for chunk in iter(lambda: f.read(1024 * 128), b""):
|
||||||
h.update(chunk)
|
h.update(chunk)
|
||||||
digest = h.hexdigest().lower()
|
return h.hexdigest().lower()
|
||||||
if digest != _EXPECTED_SHA256:
|
|
||||||
raise RuntimeError("Update SHA-256 mismatch")
|
|
||||||
|
def _load_baked_in_release_digests() -> dict:
|
||||||
|
"""Return the ``release_digests.json`` mapping, or an empty dict.
|
||||||
|
|
||||||
|
Schema (issue #231):
|
||||||
|
{
|
||||||
|
"<release_tag>": {
|
||||||
|
"<asset_filename>": "<sha256_hex>",
|
||||||
|
...
|
||||||
|
},
|
||||||
|
...
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
raw = _RELEASE_DIGESTS_FILE.read_text(encoding="utf-8")
|
||||||
|
parsed = json.loads(raw)
|
||||||
|
except (OSError, ValueError) as exc:
|
||||||
|
logger.debug("Release digest file unreadable: %s", exc)
|
||||||
|
return {}
|
||||||
|
if not isinstance(parsed, dict):
|
||||||
|
return {}
|
||||||
|
cleaned: dict[str, dict[str, str]] = {}
|
||||||
|
for k, v in parsed.items():
|
||||||
|
if not isinstance(k, str) or k.startswith("_"):
|
||||||
|
continue
|
||||||
|
if isinstance(v, dict):
|
||||||
|
entries = {
|
||||||
|
fname: digest.strip().lower()
|
||||||
|
for fname, digest in v.items()
|
||||||
|
if isinstance(fname, str) and isinstance(digest, str)
|
||||||
|
}
|
||||||
|
if entries:
|
||||||
|
cleaned[k] = entries
|
||||||
|
return cleaned
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_sha256sums(sha256sums_url: str) -> dict[str, str]:
|
||||||
|
"""Download a SHA256SUMS.txt and return {filename: digest_hex_lower}.
|
||||||
|
|
||||||
|
Standard ``sha256sum`` format: ``<digest> <filename>`` per line. The
|
||||||
|
leading ``*`` binary-mode marker (e.g. ``<digest> *<filename>``) is
|
||||||
|
handled.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
_validate_update_url(sha256sums_url)
|
||||||
|
except RuntimeError as exc:
|
||||||
|
logger.warning("SHA256SUMS URL rejected: %s", exc)
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
resp = requests.get(sha256sums_url, timeout=15)
|
||||||
|
resp.raise_for_status()
|
||||||
|
except requests.RequestException as exc:
|
||||||
|
logger.info("SHA256SUMS fetch failed: %s", exc)
|
||||||
|
return {}
|
||||||
|
out: dict[str, str] = {}
|
||||||
|
for line in resp.text.splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#"):
|
||||||
|
continue
|
||||||
|
# Tolerant split: handle both `<digest> <name>` and `<digest> *<name>`.
|
||||||
|
parts = line.split(None, 1)
|
||||||
|
if len(parts) != 2:
|
||||||
|
continue
|
||||||
|
digest, fname = parts
|
||||||
|
fname = fname.lstrip("*").strip()
|
||||||
|
digest = digest.strip().lower()
|
||||||
|
if len(digest) == 64 and all(c in "0123456789abcdef" for c in digest) and fname:
|
||||||
|
out[fname] = digest
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_zip_hash(
|
||||||
|
zip_path: str,
|
||||||
|
*,
|
||||||
|
asset_name: str = "",
|
||||||
|
sha256sums_url: str = "",
|
||||||
|
release_tag: str = "",
|
||||||
|
) -> str:
|
||||||
|
"""Verify the downloaded archive against trusted digest sources.
|
||||||
|
|
||||||
|
Issue #231: previously this returned silently when ``MESH_UPDATE_SHA256``
|
||||||
|
was unset, which made the auto-updater a supply-chain RCE vector on any
|
||||||
|
compromise of the GitHub release pipeline. The chain now is:
|
||||||
|
|
||||||
|
1. ``MESH_UPDATE_SHA256`` env var (operator override — preserved for
|
||||||
|
power-users who want to pin an exact digest manually)
|
||||||
|
2. ``SHA256SUMS.txt`` release asset (primary — the maintainer's
|
||||||
|
release process already publishes this)
|
||||||
|
3. Baked-in ``backend/data/release_digests.json`` (second line of
|
||||||
|
defense for releases that lack the SHA256SUMS asset, or when the
|
||||||
|
asset can't be fetched at update time)
|
||||||
|
4. HTTPS-only fallback with a loud warning (preserves the auto-update
|
||||||
|
flow during transient outages — but never silently)
|
||||||
|
|
||||||
|
A mismatch from a source that DID respond is fatal: the update is
|
||||||
|
refused and the existing install keeps running. Only the "no source
|
||||||
|
reachable at all" case falls back to HTTPS-only.
|
||||||
|
|
||||||
|
Returns a short human-readable description of which source verified
|
||||||
|
the archive (used in the update-success message).
|
||||||
|
"""
|
||||||
|
actual = _compute_sha256(zip_path)
|
||||||
|
|
||||||
|
# Source 1: explicit operator override.
|
||||||
|
override = os.environ.get("MESH_UPDATE_SHA256", "").strip().lower()
|
||||||
|
if override:
|
||||||
|
if actual == override:
|
||||||
|
return f"verified via MESH_UPDATE_SHA256 ({actual[:16]}...)"
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Update SHA-256 mismatch vs MESH_UPDATE_SHA256: archive={actual[:16]}..., "
|
||||||
|
f"expected={override[:16]}..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Source 2: SHA256SUMS.txt asset from the release.
|
||||||
|
sums_map: dict[str, str] = {}
|
||||||
|
if sha256sums_url and asset_name:
|
||||||
|
sums_map = _fetch_sha256sums(sha256sums_url)
|
||||||
|
|
||||||
|
sums_expected = sums_map.get(asset_name) if asset_name else None
|
||||||
|
if sums_expected:
|
||||||
|
if actual == sums_expected:
|
||||||
|
return f"verified via release SHA256SUMS.txt ({actual[:16]}...)"
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Update SHA-256 mismatch vs release SHA256SUMS.txt: "
|
||||||
|
f"archive={actual[:16]}..., expected={sums_expected[:16]}..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Source 3: baked-in digest list.
|
||||||
|
baked = _load_baked_in_release_digests()
|
||||||
|
baked_expected = ""
|
||||||
|
if release_tag and asset_name:
|
||||||
|
baked_expected = baked.get(release_tag, {}).get(asset_name, "")
|
||||||
|
if baked_expected:
|
||||||
|
if actual == baked_expected:
|
||||||
|
return f"verified via baked-in digest list ({actual[:16]}...)"
|
||||||
|
raise RuntimeError(
|
||||||
|
f"Update SHA-256 mismatch vs baked-in digest list: "
|
||||||
|
f"archive={actual[:16]}..., expected={baked_expected[:16]}..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Source 4: HTTPS-only fallback. We keep onboarding/auto-update working
|
||||||
|
# during transient outages (no SHA256SUMS reachable AND no baked-in
|
||||||
|
# entry for this release), but surface the degraded posture loudly so
|
||||||
|
# the operator can see it in logs and the maintainer can populate the
|
||||||
|
# digest list on the next release bump.
|
||||||
|
logger.warning(
|
||||||
|
"Update integrity check fell back to HTTPS-only trust "
|
||||||
|
"(no SHA256SUMS.txt response and no baked-in digest for "
|
||||||
|
"release=%s asset=%s). The archive SHA-256 is %s. Once the "
|
||||||
|
"release ships a SHA256SUMS.txt asset OR backend/data/"
|
||||||
|
"release_digests.json is updated with this release, the secure "
|
||||||
|
"path will activate automatically.",
|
||||||
|
release_tag or "unknown",
|
||||||
|
asset_name or "unknown",
|
||||||
|
actual,
|
||||||
|
)
|
||||||
|
return f"https-only (no digest source reachable, archive={actual[:16]}...)"
|
||||||
|
|
||||||
|
|
||||||
def _is_source_checkout(project_root: str) -> bool:
|
def _is_source_checkout(project_root: str) -> bool:
|
||||||
@@ -334,7 +545,7 @@ def perform_update(project_root: str) -> dict:
|
|||||||
temp_dir = tempfile.mkdtemp(prefix="sb_update_")
|
temp_dir = tempfile.mkdtemp(prefix="sb_update_")
|
||||||
manual_url = GITHUB_RELEASES_PAGE_URL
|
manual_url = GITHUB_RELEASES_PAGE_URL
|
||||||
try:
|
try:
|
||||||
zip_path, version, url, release_url = _download_release(temp_dir)
|
zip_path, version, url, release_url, asset_name, sha256sums_url = _download_release(temp_dir)
|
||||||
manual_url = release_url or manual_url
|
manual_url = release_url or manual_url
|
||||||
|
|
||||||
if in_docker:
|
if in_docker:
|
||||||
@@ -366,7 +577,13 @@ def perform_update(project_root: str) -> dict:
|
|||||||
),
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
_validate_zip_hash(zip_path)
|
verification_note = _validate_zip_hash(
|
||||||
|
zip_path,
|
||||||
|
asset_name=asset_name,
|
||||||
|
sha256sums_url=sha256sums_url,
|
||||||
|
release_tag=version,
|
||||||
|
)
|
||||||
|
logger.info("Update archive %s", verification_note)
|
||||||
backup_path = _backup_current(project_root, temp_dir)
|
backup_path = _backup_current(project_root, temp_dir)
|
||||||
copied = _extract_and_copy(zip_path, project_root, temp_dir)
|
copied = _extract_and_copy(zip_path, project_root, temp_dir)
|
||||||
|
|
||||||
@@ -378,6 +595,7 @@ def perform_update(project_root: str) -> dict:
|
|||||||
"manual_url": manual_url,
|
"manual_url": manual_url,
|
||||||
"release_url": release_url,
|
"release_url": release_url,
|
||||||
"download_url": url,
|
"download_url": url,
|
||||||
|
"integrity": verification_note,
|
||||||
"message": f"Updated to {version} — {copied} files replaced. Restarting...",
|
"message": f"Updated to {version} — {copied} files replaced. Restarting...",
|
||||||
}
|
}
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
|
|||||||
@@ -0,0 +1,338 @@
|
|||||||
|
"""Issue #231 — self-update SHA-256 verification.
|
||||||
|
|
||||||
|
Before this fix, ``_validate_zip_hash`` returned silently whenever the
|
||||||
|
``MESH_UPDATE_SHA256`` env var was unset (the default — nothing in the
|
||||||
|
install docs ever told operators to set it). That made the auto-updater
|
||||||
|
a supply-chain RCE on any compromise of the GitHub release pipeline.
|
||||||
|
|
||||||
|
The fix introduces a four-source verification chain:
|
||||||
|
|
||||||
|
1. ``MESH_UPDATE_SHA256`` env var (operator override, preserved)
|
||||||
|
2. ``SHA256SUMS.txt`` asset published alongside the release (primary)
|
||||||
|
3. Baked-in ``backend/data/release_digests.json`` (fallback)
|
||||||
|
4. HTTPS-only fallback with a loud warning (preserves auto-update during
|
||||||
|
transient outages so the user isn't stuck)
|
||||||
|
|
||||||
|
A mismatch from any source that DID respond is fatal. Only the "no
|
||||||
|
source reachable at all" case falls back to HTTPS-only.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services import updater
|
||||||
|
from services.updater import (
|
||||||
|
_compute_sha256,
|
||||||
|
_fetch_sha256sums,
|
||||||
|
_load_baked_in_release_digests,
|
||||||
|
_validate_zip_hash,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fake_archive(tmp_path):
|
||||||
|
"""A tiny synthetic zip-shaped file so we can compute a known digest."""
|
||||||
|
archive = tmp_path / "update.zip"
|
||||||
|
payload = b"this is not really a release archive"
|
||||||
|
archive.write_bytes(payload)
|
||||||
|
expected = hashlib.sha256(payload).hexdigest().lower()
|
||||||
|
return str(archive), expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_baked_in_release_digests_file_loads():
|
||||||
|
"""The shipped release_digests.json must parse and contain v0.9.79."""
|
||||||
|
digests = _load_baked_in_release_digests()
|
||||||
|
assert "v0.9.79" in digests
|
||||||
|
entry = digests["v0.9.79"]
|
||||||
|
assert "ShadowBroker_v0.9.79.zip" in entry
|
||||||
|
digest = entry["ShadowBroker_v0.9.79.zip"]
|
||||||
|
assert len(digest) == 64
|
||||||
|
assert all(c in "0123456789abcdef" for c in digest)
|
||||||
|
|
||||||
|
|
||||||
|
def test_baked_in_skips_comment_keys():
|
||||||
|
"""The _comment top-level key is ignored, not surfaced as a release."""
|
||||||
|
digests = _load_baked_in_release_digests()
|
||||||
|
assert "_comment" not in digests
|
||||||
|
|
||||||
|
|
||||||
|
def test_compute_sha256_matches_known_value(fake_archive):
|
||||||
|
archive, expected = fake_archive
|
||||||
|
assert _compute_sha256(archive) == expected
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# Source 1: MESH_UPDATE_SHA256 env override
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_matching_passes(fake_archive, monkeypatch):
|
||||||
|
"""Path 1: operator pinned the exact digest via env. Match = success."""
|
||||||
|
archive, expected = fake_archive
|
||||||
|
monkeypatch.setenv("MESH_UPDATE_SHA256", expected)
|
||||||
|
|
||||||
|
note = _validate_zip_hash(archive)
|
||||||
|
assert "MESH_UPDATE_SHA256" in note
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_mismatch_fails_loudly(fake_archive, monkeypatch):
|
||||||
|
"""Path 1: operator pinned a different digest. Mismatch = fatal."""
|
||||||
|
archive, _expected = fake_archive
|
||||||
|
monkeypatch.setenv("MESH_UPDATE_SHA256", "0" * 64)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError) as exc_info:
|
||||||
|
_validate_zip_hash(archive)
|
||||||
|
assert "mismatch" in str(exc_info.value).lower()
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# Source 2: SHA256SUMS.txt asset
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_sha256sums_matching_passes(fake_archive, monkeypatch):
|
||||||
|
"""Path 2: SHA256SUMS.txt has the correct digest for our asset."""
|
||||||
|
archive, expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
|
||||||
|
def fake_sums(url):
|
||||||
|
return {"ShadowBroker_v9.9.9.zip": expected}
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", fake_sums)
|
||||||
|
note = _validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v9.9.9.zip",
|
||||||
|
sha256sums_url="https://example.test/SHA256SUMS.txt",
|
||||||
|
release_tag="v9.9.9",
|
||||||
|
)
|
||||||
|
assert "SHA256SUMS.txt" in note
|
||||||
|
|
||||||
|
|
||||||
|
def test_sha256sums_mismatch_fails_loudly(fake_archive, monkeypatch):
|
||||||
|
"""Path 2: SHA256SUMS.txt has a different digest. Refuse."""
|
||||||
|
archive, _expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
|
||||||
|
def fake_sums(url):
|
||||||
|
return {"ShadowBroker_v9.9.9.zip": "0" * 64}
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", fake_sums)
|
||||||
|
with pytest.raises(RuntimeError) as exc_info:
|
||||||
|
_validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v9.9.9.zip",
|
||||||
|
sha256sums_url="https://example.test/SHA256SUMS.txt",
|
||||||
|
release_tag="v9.9.9",
|
||||||
|
)
|
||||||
|
assert "mismatch" in str(exc_info.value).lower()
|
||||||
|
assert "SHA256SUMS" in str(exc_info.value)
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# Source 3: baked-in digest list
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_baked_in_matching_passes(fake_archive, monkeypatch):
|
||||||
|
"""Path 3: SHA256SUMS unreachable, but the baked-in list has us."""
|
||||||
|
archive, expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", lambda url: {})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
updater,
|
||||||
|
"_load_baked_in_release_digests",
|
||||||
|
lambda: {"v9.9.9": {"ShadowBroker_v9.9.9.zip": expected}},
|
||||||
|
)
|
||||||
|
|
||||||
|
note = _validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v9.9.9.zip",
|
||||||
|
sha256sums_url="https://example.test/SHA256SUMS.txt",
|
||||||
|
release_tag="v9.9.9",
|
||||||
|
)
|
||||||
|
assert "baked-in" in note
|
||||||
|
|
||||||
|
|
||||||
|
def test_baked_in_mismatch_fails_loudly(fake_archive, monkeypatch):
|
||||||
|
"""Path 3: baked-in says something different. Refuse."""
|
||||||
|
archive, _expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", lambda url: {})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
updater,
|
||||||
|
"_load_baked_in_release_digests",
|
||||||
|
lambda: {"v9.9.9": {"ShadowBroker_v9.9.9.zip": "0" * 64}},
|
||||||
|
)
|
||||||
|
|
||||||
|
with pytest.raises(RuntimeError) as exc_info:
|
||||||
|
_validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v9.9.9.zip",
|
||||||
|
sha256sums_url="",
|
||||||
|
release_tag="v9.9.9",
|
||||||
|
)
|
||||||
|
assert "mismatch" in str(exc_info.value).lower()
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# Source 4: HTTPS-only fallback
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_https_only_fallback_when_no_source_available(fake_archive, monkeypatch, caplog):
|
||||||
|
"""Path 4: nothing matches — fall back to HTTPS-only with loud warning.
|
||||||
|
|
||||||
|
This preserves the auto-update flow during transient outages: an
|
||||||
|
operator on a flaky network during update doesn't get a hostile
|
||||||
|
error, they get a degraded-but-functional update with a clear log
|
||||||
|
message.
|
||||||
|
"""
|
||||||
|
import logging
|
||||||
|
|
||||||
|
archive, _expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", lambda url: {})
|
||||||
|
monkeypatch.setattr(updater, "_load_baked_in_release_digests", lambda: {})
|
||||||
|
|
||||||
|
with caplog.at_level(logging.WARNING):
|
||||||
|
note = _validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v99.99.zip",
|
||||||
|
sha256sums_url="",
|
||||||
|
release_tag="v99.99",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert "https-only" in note.lower()
|
||||||
|
assert any(
|
||||||
|
"fell back to HTTPS-only" in rec.getMessage() for rec in caplog.records
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_https_only_fallback_when_release_tag_unknown(fake_archive, monkeypatch):
|
||||||
|
"""Path 4 also kicks in when we have a baked-in list but it doesn't
|
||||||
|
contain THIS release tag — e.g. a brand-new release that the local
|
||||||
|
install hasn't seen a digest for yet."""
|
||||||
|
archive, _expected = fake_archive
|
||||||
|
monkeypatch.delenv("MESH_UPDATE_SHA256", raising=False)
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", lambda url: {})
|
||||||
|
monkeypatch.setattr(
|
||||||
|
updater,
|
||||||
|
"_load_baked_in_release_digests",
|
||||||
|
lambda: {"v0.0.1": {"old.zip": "0" * 64}}, # different tag, doesn't match
|
||||||
|
)
|
||||||
|
|
||||||
|
note = _validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="ShadowBroker_v99.99.zip",
|
||||||
|
sha256sums_url="",
|
||||||
|
release_tag="v99.99",
|
||||||
|
)
|
||||||
|
assert "https-only" in note.lower()
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# Precedence (env > SHA256SUMS > baked-in > https-only)
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_beats_all_other_sources(fake_archive, monkeypatch):
|
||||||
|
"""When MESH_UPDATE_SHA256 is set, it's the only source consulted.
|
||||||
|
|
||||||
|
The other sources may return false positives or negatives — they
|
||||||
|
shouldn't be queried at all when the operator pinned an exact value.
|
||||||
|
"""
|
||||||
|
archive, expected = fake_archive
|
||||||
|
monkeypatch.setenv("MESH_UPDATE_SHA256", expected)
|
||||||
|
|
||||||
|
def boom_sums(url):
|
||||||
|
raise AssertionError("SHA256SUMS source was queried despite env override")
|
||||||
|
|
||||||
|
def boom_baked():
|
||||||
|
raise AssertionError("Baked-in list was queried despite env override")
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater, "_fetch_sha256sums", boom_sums)
|
||||||
|
monkeypatch.setattr(updater, "_load_baked_in_release_digests", boom_baked)
|
||||||
|
|
||||||
|
note = _validate_zip_hash(
|
||||||
|
archive,
|
||||||
|
asset_name="any.zip",
|
||||||
|
sha256sums_url="https://example.test/SHA256SUMS.txt",
|
||||||
|
release_tag="any",
|
||||||
|
)
|
||||||
|
assert "MESH_UPDATE_SHA256" in note
|
||||||
|
|
||||||
|
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
# _fetch_sha256sums parser
|
||||||
|
# ──────────────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_sha256sums_parses_standard_format(monkeypatch):
|
||||||
|
"""Standard ``sha256sum`` output: ``<digest> <filename>``."""
|
||||||
|
class _Resp:
|
||||||
|
text = (
|
||||||
|
"f6877c1d66614525315ea82636ce9f7b41178332c4dbf90d27431a1ea1d9cd47 ShadowBroker_v0.9.79.zip\n"
|
||||||
|
"e0713c3cdda184cfbea750bfac0d62a35678fec00847e6476f2cac8e7e42046e ShadowBroker_0.9.79_x64_en-US.msi\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
def fake_get(url, timeout=15):
|
||||||
|
return _Resp()
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater.requests, "get", fake_get)
|
||||||
|
monkeypatch.setattr(updater, "_validate_update_url", lambda url, **kw: url)
|
||||||
|
sums = _fetch_sha256sums("https://example.test/SHA256SUMS.txt")
|
||||||
|
assert sums["ShadowBroker_v0.9.79.zip"].startswith("f6877c1d")
|
||||||
|
assert sums["ShadowBroker_0.9.79_x64_en-US.msi"].startswith("e0713c3c")
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_sha256sums_handles_binary_marker(monkeypatch):
|
||||||
|
"""sha256sum -b output: ``<digest> *<filename>``."""
|
||||||
|
class _Resp:
|
||||||
|
text = "f6877c1d66614525315ea82636ce9f7b41178332c4dbf90d27431a1ea1d9cd47 *ShadowBroker_v0.9.79.zip\n"
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater.requests, "get", lambda url, timeout=15: _Resp())
|
||||||
|
monkeypatch.setattr(updater, "_validate_update_url", lambda url, **kw: url)
|
||||||
|
sums = _fetch_sha256sums("https://example.test/SHA256SUMS.txt")
|
||||||
|
assert "ShadowBroker_v0.9.79.zip" in sums
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_sha256sums_skips_malformed_lines(monkeypatch):
|
||||||
|
"""Lines that don't parse cleanly are ignored, not aborted on."""
|
||||||
|
class _Resp:
|
||||||
|
text = (
|
||||||
|
"# comment line\n"
|
||||||
|
"\n"
|
||||||
|
"not-a-digest bogus.txt\n"
|
||||||
|
"f6877c1d66614525315ea82636ce9f7b41178332c4dbf90d27431a1ea1d9cd47 good.zip\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
def raise_for_status(self):
|
||||||
|
pass
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater.requests, "get", lambda url, timeout=15: _Resp())
|
||||||
|
monkeypatch.setattr(updater, "_validate_update_url", lambda url, **kw: url)
|
||||||
|
sums = _fetch_sha256sums("https://example.test/SHA256SUMS.txt")
|
||||||
|
assert "good.zip" in sums
|
||||||
|
assert "bogus.txt" not in sums
|
||||||
|
|
||||||
|
|
||||||
|
def test_fetch_sha256sums_handles_network_failure(monkeypatch):
|
||||||
|
"""If the SHA256SUMS asset can't be fetched, return empty (caller
|
||||||
|
falls through to baked-in / https-only)."""
|
||||||
|
import requests as _req
|
||||||
|
|
||||||
|
def fake_get(url, timeout=15):
|
||||||
|
raise _req.exceptions.ConnectionError("upstream down")
|
||||||
|
|
||||||
|
monkeypatch.setattr(updater.requests, "get", fake_get)
|
||||||
|
monkeypatch.setattr(updater, "_validate_update_url", lambda url, **kw: url)
|
||||||
|
sums = _fetch_sha256sums("https://example.test/SHA256SUMS.txt")
|
||||||
|
assert sums == {}
|
||||||
Reference in New Issue
Block a user