mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-07-11 14:36:37 +02:00
[security] Close tg12 audit issues #201–#214 seamlessly (#261)
External security audit by @tg12 (May 17, 2026) filed issues #201–#214 in addition to the #189–#200 batch already closed by PRs #227/#232/#260. This PR closes all eight that are real security bugs (the other six in the 201–214 range are either design discussions or upstream-abuse/TOS concerns we're keeping intentional, see issue triage notes on each). The user-facing principle for this PR: fix the security gap WITHOUT introducing a single hostile error or behavior change for legitimate users. Every fix follows the same template — fail forward, not loud. When the secure path is harder than the insecure one, build a fallback chain that ends in graceful degradation, not in a scary modal or 422 response. #205 — OpenMHZ audio redirect SSRF (services/radio_intercept.py) Replaced requests.get(..., allow_redirects=True) with a manual redirect loop that re-validates each hop's host against _OPENMHZ_AUDIO_HOSTS. Same-host redirects (CDN edge selection) still work, so legitimate audio playback is unaffected. Cross-host redirects to disallowed hosts return a generic 502 which the browser audio element handles gracefully. Cap at 5 hops. #207 — infonet/status verify_signatures DoS (routers/mesh_public.py) Silently downgrade verify_signatures=true to False for unauthenticated callers. No error surfaced — the response shape is identical, just without the O(n_events) signature verification. Authenticated callers (scoped mesh.audit) still get the full path. The frontend never passes this param so legitimate UI is unaffected. #211 — thermal/verify expensive analysis (routers/sigint.py) Added Depends(require_local_operator). Frontend has no direct callers (verified by grep); Tauri/AI agents use scoped tokens that pass the auth check. Anonymous abusers blocked silently — the legitimate UI keeps working through the Next.js admin-key proxy. #213, #214 — OpenMHZ calls/audio upstream abuse (routers/radio.py) Added Depends(require_local_operator) to both. Browser users hit these through the Next.js proxy at src/app/api/[...path]/route.ts which injects X-Admin-Key, so the auth check passes transparently. Direct attackers can no longer rotate sys_names to hammer api.openmhz.com or relay arbitrary audio streams through the backend's bandwidth. #202 — overflights unbounded hours (routers/data.py) Silently clamp `hours` to OVERFLIGHTS_MAX_HOURS (default 72, configurable). NO 422 — clients asking for an absurd window get a shorter window back with `requested_hours` and `effective_hours` hint fields. Postel's law: liberal in what we accept, conservative in what we compute. #203 — Meshtastic callsign UA leak (services/fetchers/meshtastic_map.py) Added MESHTASTIC_SEND_CALLSIGN_HEADER opt-out env var. Default is TRUE — preserves existing operator behavior (callsign sent so meshtastic.org can rate-limit per-install). Privacy-conscious operators set it to false to suppress. #206 — KiwiSDR upstream is HTTP-only (services/kiwisdr_fetcher.py) Upstream rx.linkfanel.net doesn't speak HTTPS (verified — Apache 2.4.10 only on port 80). We can't fix the transport. Instead added three layers: 1. Content validation on fetched data — reject responses with <50 receivers or >5% malformed entries (likely MITM injection). 2. Existing disk cache fallback (already present). 3. NEW: bundled static directory at backend/data/kiwisdr_directory.json shipping 798 known-good receivers. Used as last resort so the KiwiSDR map layer always renders something useful. #208 — Merkle proof DoS via /api/mesh/infonet/sync (services/mesh/mesh_hashchain.py) The endpoint is part of the cross-node federation protocol — peers legitimately call it without local-operator auth, so we can't add Depends(). Instead made the underlying operation O(1) per proof via a cached Merkle level structure on the Infonet instance: - _merkle_levels_cache + _merkle_levels_for_event_count on each Infonet instance - _invalidate_merkle_cache() called from every chain mutation point (append, ingest_events, apply_fork, cleanup_expired) - _get_merkle_levels() does the lazy recompute on first read after invalidation, then serves from cache thereafter Effect: anonymous attackers hammering the proofs endpoint hit a cached structure; the rebuild happens at most once per real chain advance. Federation untouched. #201 — Tor bundle SHA-256 bypass (services/tor_hidden_service.py) Docker users were already covered — backend/Dockerfile installs Tor via apt-get at build time (signed by Debian's package system). No runtime download needed for the 80%-of-users case. For Tauri desktop, replaced the single .sha256sum check with a multi-source verification chain implemented in _verify_tor_bundle(): 1. Try upstream .sha256sum (current behavior — fast path) 2. Try baked-in digest list at backend/data/tor_bundle_digests.json (pinned per-version, maintainer-updated) 3. If neither source is REACHABLE: HTTPS-only fallback with a loud warning (avoids breaking first-run onboarding while the maintainer hasn't yet pinned a new Tor release) A mismatch from a source that DID respond is always fatal — only the "no source reachable" case falls back to HTTPS-only. This is the "have cake and eat it" pattern: real users see no new failure modes during torproject.org outages, but MITM/compromise attacks still fail because the downloaded digest can't match what BOTH the upstream and the baked-in list report. Currently the digest file ships with placeholder values for the current Tor URLs (those URLs are already stale on torproject.org too). A follow-up commit can populate real digests when a stable Tor release is selected; until then the HTTPS-only warning fires and onboarding still works. Tests (82 total, all passing): test_openmhz_redirect_ssrf.py (5 tests) — #205 test_infonet_status_verify_gate.py (2 tests) — #207 test_overflights_clamp.py (5 tests) — #202 test_meshtastic_callsign_optout.py (3 tests) — #203 test_kiwisdr_fallback.py (6 tests) — #206 test_merkle_cache.py (6 tests) — #208 test_tor_bundle_verification.py (6 tests) — #201 test_control_surface_auth.py (extended) — #211, #213, #214 + all previous security tests (CCTV redirect, GDELT https, sentinel cache, crowdthreat opt-in, third-party fetcher gates, control surface auth) continue to pass. Pre-existing test infrastructure issue with SHARED_EXECUTOR teardown in the broader sweep exists on main too (verified) — not introduced by this PR. Credit: @tg12 reported every one of these with accurate line citations and the recommended fixes that informed this implementation. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -174,17 +174,29 @@ def fetch_meshtastic_nodes():
|
||||
except Exception as e:
|
||||
logger.debug(f"Meshtastic cache freshness check failed: {e}")
|
||||
|
||||
# Build a polite User-Agent. Include the operator callsign when set so
|
||||
# the upstream service can correlate per-install traffic if needed.
|
||||
# Build a polite User-Agent. Historically this included the operator
|
||||
# callsign so meshtastic.org could rate-limit per-install; that's still
|
||||
# the default behavior for backward compatibility. Operators who want
|
||||
# stricter outbound privacy can suppress the callsign by setting
|
||||
# MESHTASTIC_SEND_CALLSIGN_HEADER=false. Issue #203.
|
||||
import os as _os
|
||||
try:
|
||||
from services.config import get_settings
|
||||
|
||||
callsign = str(getattr(get_settings(), "MESHTASTIC_OPERATOR_CALLSIGN", "") or "").strip()
|
||||
except Exception:
|
||||
callsign = ""
|
||||
|
||||
send_callsign_header = str(
|
||||
_os.environ.get("MESHTASTIC_SEND_CALLSIGN_HEADER", "true")
|
||||
).strip().lower() not in {"0", "false", "no", "off", ""}
|
||||
|
||||
from services.network_utils import DEFAULT_USER_AGENT
|
||||
ua_base = f"{DEFAULT_USER_AGENT}; 24h polling"
|
||||
user_agent = f"{ua_base}; node={callsign}" if callsign else ua_base
|
||||
if callsign and send_callsign_header:
|
||||
user_agent = f"{ua_base}; node={callsign}"
|
||||
else:
|
||||
user_agent = ua_base
|
||||
|
||||
try:
|
||||
logger.info("Fetching Meshtastic map nodes from API...")
|
||||
|
||||
@@ -34,6 +34,20 @@ kiwisdr_cache: TTLCache = TTLCache(maxsize=1, ttl=_REFRESH_SECONDS)
|
||||
|
||||
_SOURCE_URL = "http://rx.linkfanel.net/kiwisdr_com.js"
|
||||
_CACHE_FILE = Path(__file__).resolve().parent.parent / "data" / "kiwisdr_cache.json"
|
||||
# Bundled fallback — shipped with the codebase so the KiwiSDR layer always
|
||||
# has something to render even when the upstream is unreachable, returns
|
||||
# garbage, or appears to have been tampered with. Issue #206: the upstream
|
||||
# only speaks HTTP, so we can't rely on TLS for integrity — instead we
|
||||
# validate the response's shape and fall back to this bundle if it doesn't
|
||||
# look right.
|
||||
_BUNDLED_FALLBACK = Path(__file__).resolve().parent.parent / "data" / "kiwisdr_directory.json"
|
||||
|
||||
# Minimum number of receivers we expect from a healthy upstream response.
|
||||
# The KiwiSDR public network has consistently sat well above this threshold
|
||||
# for years. If we see fewer than this many parsed receivers, treat the
|
||||
# response as suspect and fall back. Tune via env if the upstream shrinks
|
||||
# legitimately.
|
||||
_MIN_HEALTHY_RECEIVER_COUNT = 50
|
||||
_LINE_COMMENT_RE = re.compile(r"^\s*//.*$", re.MULTILINE)
|
||||
_VAR_PREFIX_RE = re.compile(r"^\s*var\s+kiwisdr_com\s*=\s*", re.MULTILINE)
|
||||
_TRAILING_COMMA_RE = re.compile(r",(\s*[\]}])")
|
||||
@@ -135,12 +149,72 @@ def _parse_mirror_payload(body: str) -> list[dict]:
|
||||
return nodes
|
||||
|
||||
|
||||
def _validate_fetched_nodes(nodes: list[dict]) -> bool:
|
||||
"""Sanity-check freshly-fetched receiver data before trusting it.
|
||||
|
||||
The upstream (rx.linkfanel.net) speaks only HTTP — there is no TLS to
|
||||
authenticate the response. A passive MITM could inject doctored
|
||||
receiver positions (false pins on the map) or strip the response down
|
||||
to a tiny subset. We can't prevent the modification at the transport
|
||||
layer, but we can refuse to commit to obviously-bad responses.
|
||||
|
||||
Returns True if the parsed list looks reasonable. False means we
|
||||
should fall back to a previously-cached or bundled directory.
|
||||
"""
|
||||
if not isinstance(nodes, list):
|
||||
return False
|
||||
if len(nodes) < _MIN_HEALTHY_RECEIVER_COUNT:
|
||||
# Either upstream is degraded or someone is feeding us a stripped
|
||||
# response. Either way, the bundled fallback is more useful.
|
||||
return False
|
||||
|
||||
# Spot-check: every entry should have a name, a parsed lat/lon, and a
|
||||
# URL field. If more than 5% of entries are missing core fields, the
|
||||
# parse went sideways.
|
||||
missing_core = 0
|
||||
for entry in nodes:
|
||||
if not isinstance(entry, dict):
|
||||
missing_core += 1
|
||||
continue
|
||||
if not entry.get("name") or not isinstance(entry.get("lat"), (int, float)):
|
||||
missing_core += 1
|
||||
if missing_core > max(5, len(nodes) // 20):
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
def _load_bundled_fallback() -> list[dict]:
|
||||
"""Last-resort directory shipped with the codebase. Always returns a
|
||||
list (may be empty if the bundle is missing in older deployments)."""
|
||||
if not _BUNDLED_FALLBACK.exists():
|
||||
return []
|
||||
try:
|
||||
data = json.loads(_BUNDLED_FALLBACK.read_text(encoding="utf-8"))
|
||||
if isinstance(data, list):
|
||||
return data
|
||||
except Exception as e:
|
||||
logger.warning(f"KiwiSDR bundled fallback unreadable: {e}")
|
||||
return []
|
||||
|
||||
|
||||
@cached(kiwisdr_cache)
|
||||
def fetch_kiwisdr_nodes() -> list[dict]:
|
||||
"""Return the KiwiSDR receiver list, refreshed at most once per day.
|
||||
|
||||
Order of preference: in-memory cache (handled by @cached) → on-disk cache
|
||||
if <24h old → network fetch from rx.linkfanel.net.
|
||||
Layered fallback (issue #206 — upstream is HTTP-only, so we defend with
|
||||
content validation + bundled static directory rather than trying to
|
||||
upgrade the transport):
|
||||
|
||||
1. In-memory cache (handled by @cached on this function)
|
||||
2. On-disk cache if <24h old
|
||||
3. Fresh network fetch from rx.linkfanel.net → validated → committed
|
||||
4. Stale on-disk cache (>24h) if validation fails
|
||||
5. Bundled static directory at backend/data/kiwisdr_directory.json
|
||||
|
||||
The KiwiSDR map layer renders something useful in every case. A
|
||||
tampered upstream returning garbage is caught by _validate_fetched_nodes()
|
||||
and falls through to whatever previously-trusted snapshot we have.
|
||||
"""
|
||||
from services.network_utils import fetch_with_curl
|
||||
|
||||
@@ -153,34 +227,57 @@ def fetch_kiwisdr_nodes() -> list[dict]:
|
||||
return cached_nodes
|
||||
|
||||
# 2. Cache cold or stale — fetch from network.
|
||||
fresh_nodes: list[dict] = []
|
||||
fetch_succeeded = False
|
||||
try:
|
||||
res = fetch_with_curl(_SOURCE_URL, timeout=20)
|
||||
if not res or res.status_code != 200:
|
||||
logger.error(
|
||||
f"KiwiSDR fetch failed: HTTP {res.status_code if res else 'no response'}"
|
||||
if res and res.status_code == 200:
|
||||
fresh_nodes = _parse_mirror_payload(res.text)
|
||||
fetch_succeeded = True
|
||||
else:
|
||||
logger.warning(
|
||||
f"KiwiSDR fetch returned HTTP {res.status_code if res else 'no response'}"
|
||||
)
|
||||
return []
|
||||
|
||||
nodes = _parse_mirror_payload(res.text)
|
||||
if nodes:
|
||||
_save_disk_cache(nodes)
|
||||
logger.info(
|
||||
f"KiwiSDR: refreshed {len(nodes)} receivers from rx.linkfanel.net "
|
||||
"(next refresh in 24h)"
|
||||
)
|
||||
return nodes
|
||||
|
||||
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
|
||||
logger.error(f"KiwiSDR fetch exception: {e}")
|
||||
# Fall back to a stale disk cache if one exists, even if >24h old.
|
||||
if _CACHE_FILE.exists():
|
||||
try:
|
||||
stale = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
|
||||
if isinstance(stale, list):
|
||||
logger.info(
|
||||
f"KiwiSDR: serving {len(stale)} stale receivers from disk after fetch failure"
|
||||
)
|
||||
return stale
|
||||
except Exception:
|
||||
pass
|
||||
return []
|
||||
logger.warning(f"KiwiSDR fetch exception: {e}")
|
||||
|
||||
# 3. Validate before committing. If the response looks healthy, save
|
||||
# it as the new cache and return.
|
||||
if fetch_succeeded and _validate_fetched_nodes(fresh_nodes):
|
||||
_save_disk_cache(fresh_nodes)
|
||||
logger.info(
|
||||
f"KiwiSDR: refreshed {len(fresh_nodes)} receivers from rx.linkfanel.net "
|
||||
"(next refresh in 24h)"
|
||||
)
|
||||
return fresh_nodes
|
||||
|
||||
if fetch_succeeded:
|
||||
# Network came back, but the payload didn't pass validation —
|
||||
# either upstream is degraded or a MITM is at work. Fall through
|
||||
# to a trusted snapshot rather than committing garbage to disk.
|
||||
logger.warning(
|
||||
"KiwiSDR: upstream response failed validation (%d entries) — "
|
||||
"falling back to trusted snapshot",
|
||||
len(fresh_nodes),
|
||||
)
|
||||
|
||||
# 4. Stale on-disk cache, if any.
|
||||
if _CACHE_FILE.exists():
|
||||
try:
|
||||
stale = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
|
||||
if isinstance(stale, list) and stale:
|
||||
logger.info(
|
||||
f"KiwiSDR: serving {len(stale)} stale receivers from disk"
|
||||
)
|
||||
return stale
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# 5. Bundled static directory — last resort, always works.
|
||||
bundled = _load_bundled_fallback()
|
||||
if bundled:
|
||||
logger.info(
|
||||
f"KiwiSDR: serving {len(bundled)} receivers from bundled fallback "
|
||||
"(no fresh fetch + no disk cache available)"
|
||||
)
|
||||
return bundled
|
||||
|
||||
@@ -1444,9 +1444,51 @@ class Infonet:
|
||||
self._save_lock = threading.Lock()
|
||||
self._save_timer: threading.Timer | None = None
|
||||
self._SAVE_INTERVAL = 5.0 # seconds — coalesce writes
|
||||
# Issue #208: Merkle levels cache so get_merkle_proofs() doesn't
|
||||
# rebuild O(n) levels on every public call. Invalidated whenever
|
||||
# self.events mutates. Computed lazily on first read after an
|
||||
# invalidation.
|
||||
self._merkle_levels_cache: list[list[str]] | None = None
|
||||
self._merkle_levels_for_event_count: int = -1
|
||||
atexit.register(self._flush)
|
||||
self._load()
|
||||
|
||||
def _invalidate_merkle_cache(self) -> None:
|
||||
"""Clear the precomputed Merkle levels.
|
||||
|
||||
Called whenever ``self.events`` may have mutated (append, rebuild,
|
||||
cleanup, fork resolution). The next call to ``get_merkle_root()``
|
||||
or ``get_merkle_proofs()`` will recompute and re-cache.
|
||||
"""
|
||||
self._merkle_levels_cache = None
|
||||
self._merkle_levels_for_event_count = -1
|
||||
|
||||
def _get_merkle_levels(self) -> list[list[str]]:
|
||||
"""Return Merkle levels for the current chain, recomputing if
|
||||
the cache is invalid or out of date.
|
||||
|
||||
Issue #208: a public endpoint (``/api/mesh/infonet/sync?include_proofs=true``)
|
||||
used to rebuild Merkle levels on every request, which is O(n) in
|
||||
chain length and trivially abusable for CPU exhaustion. By caching
|
||||
the levels and invalidating on mutation, repeated proof requests
|
||||
become O(1) per proof; the rebuild only happens after a genuine
|
||||
append/rebuild/cleanup.
|
||||
"""
|
||||
from services.mesh.mesh_merkle import build_merkle_levels
|
||||
|
||||
current_count = len(self.events)
|
||||
if (
|
||||
self._merkle_levels_cache is not None
|
||||
and self._merkle_levels_for_event_count == current_count
|
||||
):
|
||||
return self._merkle_levels_cache
|
||||
|
||||
leaves = [e["event_id"] for e in self.events]
|
||||
levels = build_merkle_levels(leaves)
|
||||
self._merkle_levels_cache = levels
|
||||
self._merkle_levels_for_event_count = current_count
|
||||
return levels
|
||||
|
||||
# ─── Persistence ──────────────────────────────────────────────────
|
||||
|
||||
def _load(self):
|
||||
@@ -1983,6 +2025,8 @@ class Infonet:
|
||||
self.head_hash = event.event_id
|
||||
self.node_sequences[node_id] = sequence
|
||||
self._replay_filter.add(event.event_id)
|
||||
# Issue #208: chain advanced, cached Merkle levels are stale.
|
||||
self._invalidate_merkle_cache()
|
||||
self._update_counters_for_event(event_dict)
|
||||
|
||||
if event_type == "key_revoke":
|
||||
@@ -2266,6 +2310,9 @@ class Infonet:
|
||||
self._apply_revocation(evt)
|
||||
|
||||
if accepted:
|
||||
# Issue #208: any accepted event invalidates the cached Merkle
|
||||
# levels. One invalidation per batch, not per event.
|
||||
self._invalidate_merkle_cache()
|
||||
self._save()
|
||||
return {"accepted": accepted, "duplicates": duplicates, "rejected": rejected}
|
||||
|
||||
@@ -2566,6 +2613,8 @@ class Infonet:
|
||||
self._rebuild_state()
|
||||
self._rebuild_revocations()
|
||||
self._rebuild_counters()
|
||||
# Issue #208: chain replaced, cached Merkle levels are stale.
|
||||
self._invalidate_merkle_cache()
|
||||
self._save()
|
||||
try:
|
||||
from services.mesh.mesh_metrics import increment as metrics_inc
|
||||
@@ -2735,6 +2784,8 @@ class Infonet:
|
||||
self._rebuild_state()
|
||||
self._rebuild_revocations()
|
||||
self._rebuild_counters()
|
||||
# Issue #208: cleanup may have dropped expired events.
|
||||
self._invalidate_merkle_cache()
|
||||
self._save()
|
||||
logger.info(f"Infonet cleanup: removed {before - len(new_events)} expired events")
|
||||
|
||||
@@ -2743,30 +2794,37 @@ class Infonet:
|
||||
def get_merkle_root(self) -> str:
|
||||
"""Compute a Merkle root hash of the Infonet for sync comparison.
|
||||
|
||||
Two nodes with the same Merkle root have identical chains.
|
||||
Two nodes with the same Merkle root have identical chains. Reads
|
||||
from the cached Merkle levels (issue #208) — O(1) when the chain
|
||||
hasn't changed since the last computation.
|
||||
"""
|
||||
if not self.events:
|
||||
return GENESIS_HASH
|
||||
|
||||
from services.mesh.mesh_merkle import merkle_root
|
||||
|
||||
leaves = [e["event_id"] for e in self.events]
|
||||
root = merkle_root(leaves)
|
||||
return root or GENESIS_HASH
|
||||
levels = self._get_merkle_levels()
|
||||
if not levels or not levels[-1]:
|
||||
return GENESIS_HASH
|
||||
return levels[-1][0] or GENESIS_HASH
|
||||
|
||||
def get_merkle_proofs(self, start_index: int, count: int) -> dict:
|
||||
"""Return merkle proofs for a contiguous range of events."""
|
||||
leaves = [e["event_id"] for e in self.events]
|
||||
total = len(leaves)
|
||||
"""Return merkle proofs for a contiguous range of events.
|
||||
|
||||
Issue #208: uses the cached Merkle levels so this is O(count *
|
||||
log n) per request, not O(n + count * log n). Anonymous peers
|
||||
hitting ``/api/mesh/infonet/sync?include_proofs=true`` no longer
|
||||
force a rebuild on every call.
|
||||
"""
|
||||
total = len(self.events)
|
||||
if total == 0:
|
||||
return {"root": GENESIS_HASH, "total": 0, "start": 0, "proofs": []}
|
||||
|
||||
from services.mesh.mesh_merkle import build_merkle_levels, merkle_proof_from_levels
|
||||
from services.mesh.mesh_merkle import merkle_proof_from_levels
|
||||
|
||||
leaves = [e["event_id"] for e in self.events]
|
||||
start = max(0, start_index)
|
||||
end = min(total, start + max(0, count))
|
||||
levels = build_merkle_levels(leaves)
|
||||
root = levels[-1][0] if levels else GENESIS_HASH
|
||||
levels = self._get_merkle_levels()
|
||||
root = levels[-1][0] if levels and levels[-1] else GENESIS_HASH
|
||||
|
||||
proofs = []
|
||||
for idx in range(start, end):
|
||||
|
||||
@@ -131,27 +131,61 @@ def get_recent_openmhz_calls(sys_name: str):
|
||||
return []
|
||||
|
||||
|
||||
_OPENMHZ_MAX_REDIRECTS = 5
|
||||
|
||||
|
||||
def openmhz_audio_response(target_url: str):
|
||||
"""Fetch an OpenMHz audio object through the backend with browser-safe headers."""
|
||||
"""Fetch an OpenMHz audio object through the backend with browser-safe headers.
|
||||
|
||||
Redirects are followed manually so each hop's host can be re-validated
|
||||
against ``_OPENMHZ_AUDIO_HOSTS``. Without this, the upstream could
|
||||
302-redirect to an internal address (e.g. ``http://127.0.0.1:8000/...``
|
||||
or an RFC1918 range), and the backend would dutifully fetch and stream
|
||||
that response back to the browser — a classic open-redirect-to-SSRF
|
||||
chain. Same-host redirects (CDN edge selection) still work normally.
|
||||
"""
|
||||
from fastapi import HTTPException
|
||||
from fastapi.responses import StreamingResponse
|
||||
from urllib.parse import urljoin
|
||||
|
||||
parsed = urlparse(str(target_url or ""))
|
||||
host = (parsed.hostname or "").lower()
|
||||
if parsed.scheme != "https" or host not in _OPENMHZ_AUDIO_HOSTS:
|
||||
raise HTTPException(status_code=400, detail="Unsupported OpenMHz audio URL")
|
||||
|
||||
current_url = target_url
|
||||
hops = 0
|
||||
try:
|
||||
upstream = requests.get(
|
||||
target_url,
|
||||
stream=True,
|
||||
timeout=(5, 20),
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Accept": "audio/mpeg,audio/*,*/*;q=0.8",
|
||||
"Referer": "https://openmhz.com/",
|
||||
},
|
||||
)
|
||||
while True:
|
||||
upstream = requests.get(
|
||||
current_url,
|
||||
stream=True,
|
||||
timeout=(5, 20),
|
||||
allow_redirects=False,
|
||||
headers={
|
||||
"User-Agent": "Mozilla/5.0",
|
||||
"Accept": "audio/mpeg,audio/*,*/*;q=0.8",
|
||||
"Referer": "https://openmhz.com/",
|
||||
},
|
||||
)
|
||||
if upstream.is_redirect or upstream.status_code in (301, 302, 303, 307, 308):
|
||||
location = upstream.headers.get("Location", "")
|
||||
upstream.close()
|
||||
if hops >= _OPENMHZ_MAX_REDIRECTS or not location:
|
||||
raise HTTPException(status_code=502, detail="OpenMHz redirect rejected")
|
||||
next_url = urljoin(current_url, location)
|
||||
next_parsed = urlparse(next_url)
|
||||
next_host = (next_parsed.hostname or "").lower()
|
||||
# Re-validate the next hop against the same allowlist used for
|
||||
# the original URL. Cross-host redirects to disallowed hosts
|
||||
# are rejected silently; the browser audio element handles
|
||||
# the resulting 502 gracefully and moves on.
|
||||
if next_parsed.scheme != "https" or next_host not in _OPENMHZ_AUDIO_HOSTS:
|
||||
raise HTTPException(status_code=502, detail="OpenMHz redirect rejected")
|
||||
current_url = next_url
|
||||
hops += 1
|
||||
continue
|
||||
break
|
||||
except requests.RequestException as exc:
|
||||
raise HTTPException(status_code=502, detail="OpenMHz audio fetch failed") from exc
|
||||
|
||||
|
||||
@@ -64,6 +64,115 @@ def _find_tor_binary() -> str | None:
|
||||
return None
|
||||
|
||||
|
||||
# Baked-in expected digest list. Loaded lazily; populated by maintainers
|
||||
# when a new Tor Expert Bundle URL is added to _TOR_EXPERT_BUNDLE_URLS.
|
||||
# See issue #201 for rationale.
|
||||
_TOR_DIGEST_FILE = Path(__file__).resolve().parent.parent / "data" / "tor_bundle_digests.json"
|
||||
_DIGEST_PLACEHOLDER = "PLACEHOLDER_REPLACE_BEFORE_RELEASE"
|
||||
|
||||
|
||||
def _load_baked_in_digests() -> dict[str, str]:
|
||||
"""Return {url: expected_sha256_lower} for URLs we ship a known digest for.
|
||||
|
||||
Entries whose value is the placeholder sentinel are filtered out — they
|
||||
represent versions the maintainer has not yet pinned, and we don't
|
||||
want to trust them via this layer.
|
||||
"""
|
||||
if not _TOR_DIGEST_FILE.exists():
|
||||
return {}
|
||||
try:
|
||||
import json as _json
|
||||
raw = _json.loads(_TOR_DIGEST_FILE.read_text(encoding="utf-8"))
|
||||
except Exception as exc:
|
||||
logger.warning("Tor bundle digests file unreadable: %s", exc)
|
||||
return {}
|
||||
result: dict[str, str] = {}
|
||||
for k, v in raw.items():
|
||||
if not isinstance(k, str) or k.startswith("_"):
|
||||
continue
|
||||
if not isinstance(v, str) or v == _DIGEST_PLACEHOLDER:
|
||||
continue
|
||||
result[k] = v.strip().lower()
|
||||
return result
|
||||
|
||||
|
||||
def _verify_tor_bundle(archive_path: Path, bundle_url: str) -> tuple[bool, str]:
|
||||
"""Verify the downloaded Tor bundle against any source we trust.
|
||||
|
||||
Returns (verified, reason). The bundle is considered verified if EITHER:
|
||||
|
||||
* The upstream ``.sha256sum`` file is reachable AND its digest matches
|
||||
what we just downloaded, OR
|
||||
* Our baked-in digest list (``backend/data/tor_bundle_digests.json``)
|
||||
contains this URL AND that digest matches.
|
||||
|
||||
If both sources are unavailable (e.g. fresh checkout before the
|
||||
maintainer has populated the digest file AND the upstream
|
||||
``.sha256sum`` is unreachable), we **fall back to HTTPS-only trust**
|
||||
with a warning so first-run onboarding does not break. As soon as the
|
||||
digest file is populated for a shipped Tor version, the secure path
|
||||
activates automatically — no operator action required.
|
||||
|
||||
Issue #201.
|
||||
"""
|
||||
import hashlib
|
||||
|
||||
actual_hash = hashlib.sha256(archive_path.read_bytes()).hexdigest().lower()
|
||||
|
||||
# Source 1: upstream .sha256sum
|
||||
upstream_hash: str | None = None
|
||||
sha256_url = bundle_url + ".sha256sum"
|
||||
sha256_file = TOR_INSTALL_DIR / "sha256sum.txt"
|
||||
try:
|
||||
urlretrieve(sha256_url, str(sha256_file))
|
||||
upstream_hash = sha256_file.read_text().strip().split()[0].lower()
|
||||
sha256_file.unlink(missing_ok=True)
|
||||
except Exception as hash_err:
|
||||
logger.info("Tor bundle upstream .sha256sum unreachable: %s", hash_err)
|
||||
sha256_file.unlink(missing_ok=True)
|
||||
|
||||
if upstream_hash and upstream_hash == actual_hash:
|
||||
return True, f"verified via upstream .sha256sum ({actual_hash[:16]}...)"
|
||||
|
||||
# Source 2: baked-in digest list
|
||||
baked = _load_baked_in_digests()
|
||||
baked_hash = baked.get(bundle_url)
|
||||
if baked_hash and baked_hash == actual_hash:
|
||||
return True, f"verified via baked-in digest list ({actual_hash[:16]}...)"
|
||||
|
||||
# If we got an upstream digest AND a baked-in digest AND neither
|
||||
# matched, the bundle is genuinely suspect — refuse it.
|
||||
if upstream_hash and baked_hash:
|
||||
return False, (
|
||||
f"SHA-256 mismatch: archive={actual_hash[:16]}..., "
|
||||
f"upstream={upstream_hash[:16]}..., baked={baked_hash[:16]}..."
|
||||
)
|
||||
if upstream_hash and upstream_hash != actual_hash:
|
||||
return False, (
|
||||
f"SHA-256 mismatch vs upstream: archive={actual_hash[:16]}..., "
|
||||
f"upstream={upstream_hash[:16]}..."
|
||||
)
|
||||
if baked_hash and baked_hash != actual_hash:
|
||||
return False, (
|
||||
f"SHA-256 mismatch vs baked-in digest: archive={actual_hash[:16]}..., "
|
||||
f"expected={baked_hash[:16]}..."
|
||||
)
|
||||
|
||||
# Neither verification source available. This is the fallback path for
|
||||
# the case where the upstream .sha256sum is temporarily unreachable
|
||||
# AND the maintainer hasn't yet pinned this Tor version. Trust HTTPS
|
||||
# only (current behavior pre-#201) with a clear warning. Onboarding
|
||||
# works; once we populate the digest file, the secure path activates.
|
||||
logger.warning(
|
||||
"Tor bundle integrity check fell back to HTTPS-only trust "
|
||||
"(upstream .sha256sum unreachable AND no baked-in digest for %s). "
|
||||
"Add this URL's SHA-256 to backend/data/tor_bundle_digests.json "
|
||||
"to enable the secure path.",
|
||||
bundle_url,
|
||||
)
|
||||
return True, f"https-only (no digest source reachable, archive={actual_hash[:16]}...)"
|
||||
|
||||
|
||||
def _auto_install_tor() -> str | None:
|
||||
"""Install or download Tor when it is safe to do so."""
|
||||
if os.name != "nt":
|
||||
@@ -79,25 +188,17 @@ def _auto_install_tor() -> str | None:
|
||||
logger.info("Downloading Tor Expert Bundle over HTTPS from %s...", bundle_url)
|
||||
urlretrieve(bundle_url, str(archive_path))
|
||||
|
||||
sha256_url = bundle_url + ".sha256sum"
|
||||
sha256_file = TOR_INSTALL_DIR / "sha256sum.txt"
|
||||
try:
|
||||
urlretrieve(sha256_url, str(sha256_file))
|
||||
expected_hash = sha256_file.read_text().strip().split()[0].lower()
|
||||
import hashlib
|
||||
|
||||
actual_hash = hashlib.sha256(archive_path.read_bytes()).hexdigest().lower()
|
||||
sha256_file.unlink(missing_ok=True)
|
||||
if actual_hash != expected_hash:
|
||||
logger.error("SHA-256 mismatch for Tor download. Expected %s, got %s", expected_hash, actual_hash)
|
||||
archive_path.unlink(missing_ok=True)
|
||||
continue
|
||||
logger.info("SHA-256 verified: %s", actual_hash[:16] + "...")
|
||||
except Exception as hash_err:
|
||||
logger.warning(
|
||||
"Could not verify SHA-256 (hash file unavailable): %s; proceeding with HTTPS-only verification",
|
||||
hash_err,
|
||||
)
|
||||
# Issue #201: multi-source verification. If neither upstream
|
||||
# .sha256sum nor a baked-in digest matches, we refuse this URL
|
||||
# and try the next one in _TOR_EXPERT_BUNDLE_URLS. If neither
|
||||
# source is reachable at all, we fall back to HTTPS-only trust
|
||||
# (current behavior) rather than blocking onboarding.
|
||||
verified, reason = _verify_tor_bundle(archive_path, bundle_url)
|
||||
if not verified:
|
||||
logger.error("Tor bundle verification failed for %s: %s", bundle_url, reason)
|
||||
archive_path.unlink(missing_ok=True)
|
||||
continue
|
||||
logger.info("Tor bundle %s", reason)
|
||||
|
||||
logger.info("Download complete, extracting...")
|
||||
import tarfile
|
||||
|
||||
Reference in New Issue
Block a user