mirror of
https://github.com/BigBodyCobain/Shadowbroker.git
synced 2026-10-12 09:57:10 +02:00
[security] Close tg12 audit issues #201–#214 seamlessly (#261)
External security audit by @tg12 (May 17, 2026) filed issues #201–#214 in addition to the #189–#200 batch already closed by PRs #227/#232/#260. This PR closes all eight that are real security bugs (the other six in the 201–214 range are either design discussions or upstream-abuse/TOS concerns we're keeping intentional, see issue triage notes on each). The user-facing principle for this PR: fix the security gap WITHOUT introducing a single hostile error or behavior change for legitimate users. Every fix follows the same template — fail forward, not loud. When the secure path is harder than the insecure one, build a fallback chain that ends in graceful degradation, not in a scary modal or 422 response. #205 — OpenMHZ audio redirect SSRF (services/radio_intercept.py) Replaced requests.get(..., allow_redirects=True) with a manual redirect loop that re-validates each hop's host against _OPENMHZ_AUDIO_HOSTS. Same-host redirects (CDN edge selection) still work, so legitimate audio playback is unaffected. Cross-host redirects to disallowed hosts return a generic 502 which the browser audio element handles gracefully. Cap at 5 hops. #207 — infonet/status verify_signatures DoS (routers/mesh_public.py) Silently downgrade verify_signatures=true to False for unauthenticated callers. No error surfaced — the response shape is identical, just without the O(n_events) signature verification. Authenticated callers (scoped mesh.audit) still get the full path. The frontend never passes this param so legitimate UI is unaffected. #211 — thermal/verify expensive analysis (routers/sigint.py) Added Depends(require_local_operator). Frontend has no direct callers (verified by grep); Tauri/AI agents use scoped tokens that pass the auth check. Anonymous abusers blocked silently — the legitimate UI keeps working through the Next.js admin-key proxy. #213, #214 — OpenMHZ calls/audio upstream abuse (routers/radio.py) Added Depends(require_local_operator) to both. Browser users hit these through the Next.js proxy at src/app/api/[...path]/route.ts which injects X-Admin-Key, so the auth check passes transparently. Direct attackers can no longer rotate sys_names to hammer api.openmhz.com or relay arbitrary audio streams through the backend's bandwidth. #202 — overflights unbounded hours (routers/data.py) Silently clamp `hours` to OVERFLIGHTS_MAX_HOURS (default 72, configurable). NO 422 — clients asking for an absurd window get a shorter window back with `requested_hours` and `effective_hours` hint fields. Postel's law: liberal in what we accept, conservative in what we compute. #203 — Meshtastic callsign UA leak (services/fetchers/meshtastic_map.py) Added MESHTASTIC_SEND_CALLSIGN_HEADER opt-out env var. Default is TRUE — preserves existing operator behavior (callsign sent so meshtastic.org can rate-limit per-install). Privacy-conscious operators set it to false to suppress. #206 — KiwiSDR upstream is HTTP-only (services/kiwisdr_fetcher.py) Upstream rx.linkfanel.net doesn't speak HTTPS (verified — Apache 2.4.10 only on port 80). We can't fix the transport. Instead added three layers: 1. Content validation on fetched data — reject responses with <50 receivers or >5% malformed entries (likely MITM injection). 2. Existing disk cache fallback (already present). 3. NEW: bundled static directory at backend/data/kiwisdr_directory.json shipping 798 known-good receivers. Used as last resort so the KiwiSDR map layer always renders something useful. #208 — Merkle proof DoS via /api/mesh/infonet/sync (services/mesh/mesh_hashchain.py) The endpoint is part of the cross-node federation protocol — peers legitimately call it without local-operator auth, so we can't add Depends(). Instead made the underlying operation O(1) per proof via a cached Merkle level structure on the Infonet instance: - _merkle_levels_cache + _merkle_levels_for_event_count on each Infonet instance - _invalidate_merkle_cache() called from every chain mutation point (append, ingest_events, apply_fork, cleanup_expired) - _get_merkle_levels() does the lazy recompute on first read after invalidation, then serves from cache thereafter Effect: anonymous attackers hammering the proofs endpoint hit a cached structure; the rebuild happens at most once per real chain advance. Federation untouched. #201 — Tor bundle SHA-256 bypass (services/tor_hidden_service.py) Docker users were already covered — backend/Dockerfile installs Tor via apt-get at build time (signed by Debian's package system). No runtime download needed for the 80%-of-users case. For Tauri desktop, replaced the single .sha256sum check with a multi-source verification chain implemented in _verify_tor_bundle(): 1. Try upstream .sha256sum (current behavior — fast path) 2. Try baked-in digest list at backend/data/tor_bundle_digests.json (pinned per-version, maintainer-updated) 3. If neither source is REACHABLE: HTTPS-only fallback with a loud warning (avoids breaking first-run onboarding while the maintainer hasn't yet pinned a new Tor release) A mismatch from a source that DID respond is always fatal — only the "no source reachable" case falls back to HTTPS-only. This is the "have cake and eat it" pattern: real users see no new failure modes during torproject.org outages, but MITM/compromise attacks still fail because the downloaded digest can't match what BOTH the upstream and the baked-in list report. Currently the digest file ships with placeholder values for the current Tor URLs (those URLs are already stale on torproject.org too). A follow-up commit can populate real digests when a stable Tor release is selected; until then the HTTPS-only warning fires and onboarding still works. Tests (82 total, all passing): test_openmhz_redirect_ssrf.py (5 tests) — #205 test_infonet_status_verify_gate.py (2 tests) — #207 test_overflights_clamp.py (5 tests) — #202 test_meshtastic_callsign_optout.py (3 tests) — #203 test_kiwisdr_fallback.py (6 tests) — #206 test_merkle_cache.py (6 tests) — #208 test_tor_bundle_verification.py (6 tests) — #201 test_control_surface_auth.py (extended) — #211, #213, #214 + all previous security tests (CCTV redirect, GDELT https, sentinel cache, crowdthreat opt-in, third-party fetcher gates, control surface auth) continue to pass. Pre-existing test infrastructure issue with SHARED_EXECUTOR teardown in the broader sweep exists on main too (verified) — not introduced by this PR. Credit: @tg12 reported every one of these with accurate line citations and the recommended fixes that informed this implementation. Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
ShadowbrokerandClaude Opus 4.7 authored and GitHub committed
1 parent
d00c63abed
commit
e36d1fc79c
21 files changed
+1073
-83
No files matched your search
@@ -91,6 +91,13 @@ backend/data/*
|
|||||||
!backend/data/power_plants.json
|
!backend/data/power_plants.json
|
||||||
!backend/data/tracked_names.json
|
!backend/data/tracked_names.json
|
||||||
!backend/data/yacht_alert_db.json
|
!backend/data/yacht_alert_db.json
|
||||||
|
# Issue #206: bundled KiwiSDR receiver directory used as last-resort
|
||||||
|
# fallback when rx.linkfanel.net (HTTP-only upstream) is unreachable
|
||||||
|
# or returns content that fails our integrity validation.
|
||||||
|
!backend/data/kiwisdr_directory.json
|
||||||
|
# Issue #201: pinned SHA-256 digests for known Tor Expert Bundle URLs.
|
||||||
|
# Used as a second verification source when upstream .sha256sum fails.
|
||||||
|
!backend/data/tor_bundle_digests.json
|
||||||
|
|
||||||
# OS generated files
|
# OS generated files
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|||||||
@@ -93,8 +93,12 @@ AIS_API_KEY= # https://aisstream.io/ — free tier WebSocket key
|
|||||||
# Optional Meshtastic node ID (e.g. "!abcd1234"). When set, included in the
|
# Optional Meshtastic node ID (e.g. "!abcd1234"). When set, included in the
|
||||||
# User-Agent sent to meshtastic.liamcottle.net so the upstream service operator
|
# User-Agent sent to meshtastic.liamcottle.net so the upstream service operator
|
||||||
# can identify per-install traffic instead of aggregated "ShadowBroker" hits.
|
# can identify per-install traffic instead of aggregated "ShadowBroker" hits.
|
||||||
# Leave blank to send a generic UA with the project contact email only.
|
# Leave blank to send a generic UA. If you set MESHTASTIC_OPERATOR_CALLSIGN,
|
||||||
|
# it is included in outbound headers to meshtastic.org by default so they
|
||||||
|
# can rate-limit per-operator. Set MESHTASTIC_SEND_CALLSIGN_HEADER=false to
|
||||||
|
# suppress the callsign while still using it locally (e.g. for APRS).
|
||||||
# MESHTASTIC_OPERATOR_CALLSIGN=
|
# MESHTASTIC_OPERATOR_CALLSIGN=
|
||||||
|
# MESHTASTIC_SEND_CALLSIGN_HEADER=true
|
||||||
# MESH_MQTT_PSK= # hex-encoded, empty = default LongFast key
|
# MESH_MQTT_PSK= # hex-encoded, empty = default LongFast key
|
||||||
|
|
||||||
# ── Mesh / Reticulum (RNS) ─────────────────────────────────────
|
# ── Mesh / Reticulum (RNS) ─────────────────────────────────────
|
||||||
|
|||||||
@@ -0,0 +1 @@
|
|||||||
|
[{"name": "80m dipole (13-16) | Chichester UK", "lat": 50.84595, "lon": -0.66176, "url": "http://g8ure.ddns.net:8078", "users": 4, "users_max": 4, "bands": "0-30000000", "antenna": "80m Dipole", "location": "Chichester UK"}, {"name": "80m Four Square looking NW | Wales, UK", "lat": 52.5, "lon": -3.5, "url": "http://21595.proxy.kiwisdr.com", "users": 0, "users_max": 8, "bands": "0-30000000", "antenna": "80m four square looking NW", "location": "Wales, UK"}, {"name": "80m Dipole (5 to 8) | Chichester UK", "lat": 50.85, "lon": -0.66, "url": "http://g8ure.ddns.net:8075", "users": 4, "users_max": 4, "bands": "0-30000000", "antenna": "80m Dipole", "location": "Chichester UK"}, {"name": "80m Dipole (9 to 12) | Chichester UK", "lat": 50.85, "lon": -0.66, "url": "http://g8ure.ddns.net:8077", "users": 4, "users_max": 4, "bands": "0-30000000", "antenna": "80m Dipole", "location": "Chichester UK"}, {"name": "AutreRadioAutreCulture 2 | TREMOLAT (24), FRANCE", "lat": 44.86942, "lon": 0.83514, "url": "http://sdr.autreradioautreculture.com:8074", "users": 0, "users_max": 3, "bands": "3000-30000000", "antenna": "Vertical pylon isolated 11 meters", "location": "TREMOLAT (24), FRANCE"}, {"name": "Wessex Flydog - G8JNJ - 0-62MHz - South West UK", "lat": 50.80885, "lon": -2.47723, "url": "http://wessex.hopto.org:8075", "users": 2, "users_max": 4, "bands": "20000-62000000", "antenna": "Terminated Delta Loop 20m sides", "location": "South West England, UK"}, {"name": "F5MMX 2x15m doublet 7 AGL | IN95oc | FRANCE", "lat": 45.12115, "lon": -0.80647, "url": "http://f5mmx.proxy.kiwisdr.com", "users": 0, "users_max": 4, "bands": "0-30000000", "antenna": "2x15m doublet 7m AGL", "location": "IN95oc | FRANCE"}, {"name": "🗲 30 KHz-30 MHz 🗲 | Sobikow, Poland", "lat": 51.94, "lon": 21.18, "url": "http://jm55.proxy.kiwisdr.com", "users": 1, "users_max": 4, "bands": "0-30000000", "antenna": "dipole", "location": "Sobikow, Poland"}, {"name": "OH1CT - 160m dipole - Loppi, Finland", "lat": 60.67332, "lon": 24.47486, "url": "http://oh1ct.sytes.net:8073", "users": 3, "users_max": 7, "bands": "0-30000000", "antenna": "160m dipole", "location": "Loppi, Finland"}, {"name": "G3SDR, 0-32 MHz Kiwi 8076. Weston-super-Mare. Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical for 28MHz. Wellb", "lat": 51.31727, "lon": -2.95048, "url": "http://g3sdr.com:8076", "users": 2, "users_max": 4, "bands": "0-30000000", "antenna": "Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical with masthead preamp for 28MHz. Wellbrook Loop for VLF & MW.", "location": "Weston-super-Mare, United Kingdom"}, {"name": "0-30 MHz Funkamateure e.V. | Baunatal, Germany", "lat": 51.0755, "lon": 9.6155, "url": "http://178.63.122.223:8073", "users": 1, "users_max": 1, "bands": "0-30000000", "antenna": "T2FD", "location": "Baunatal, Germany"}, {"name": "EC2AQW 0-30 MHz SDR | ARANO (Navarra) Spain", "lat": 43.2, "lon": -1.9, "url": "http://21959.proxy.kiwisdr.com", "users": 1, "users_max": 3, "bands": "0-30000000", "antenna": "EndFed 40m long", "location": "ARANO (Navarra) Spain"}, {"name": "DF3LZ, KiwiSDR1 | Hamburg, Germany, 33m T2FD", "lat": 53.45212, "lon": 10.22103, "url": "http://85.183.11.108:8073", "users": 4, "users_max": 4, "bands": "0-30000000", "antenna": "T2FD", "location": "Hamburg, Germany, 33m T2FD"}, {"name": "DF3LZ, KiwiSDR2 | Hamburg, Germany", "lat": 53.45212, "lon": 10.22103, "url": "http://85.183.11.108:8074", "users": 1, "users_max": 2, "bands": "0-30000000", "antenna": "33m T2FD", "location": "Hamburg, Germany"}, {"name": "OH6HPS SDR, 160m horizontal loop, Muurame, Finland", "lat": 62.14905, "lon": 25.69318, "url": "http://oh6hps.ddns.net:8073", "users": 1, "users_max": 3, "bands": "0-30000000", "antenna": "160m horizontal loop", "location": "Muurame, Finland"}, {"name": "Wessex KiWi V2 - G8JNJ - 0-30MHz - South West UK", "lat": 50.74, "lon": -2.64, "url": "http://wessex.zapto.org:8074", "users": 3, "users_max": 4, "bands": "50000-30000000", "antenna": "(same as Wessex KiWi V1)", "location": "Wessex - KiwiSDR - G8JNJ - South West UK"}, {"name": "G3SDR, 0-32 MHz Kiwi 8073. Weston-super-Mare. Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical for 28MHz. Wellb", "lat": 51.31727, "lon": -2.95048, "url": "http://g3sdr.com:8073", "users": 2, "users_max": 4, "bands": "0-30000000", "antenna": "Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical with masthead preamp for 28MHz. Wellbrook Loop for VLF & MW.", "location": "Weston-super-Mare, United Kingdom"}, {"name": "G3SDR, 0-32 MHz Kiwi 8078. Weston-super-Mare. Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical for 28MHz. Wellb", "lat": 51.31727, "lon": -2.95048, "url": "http://g3sdr.com:8078", "users": 1, "users_max": 4, "bands": "0-30000000", "antenna": "Parallel dipoles for 160m, 80m, 60m & 40m. 5/8th Vertical with masthead preamp for 28MHz. Wellbrook Loop for VLF & MW.", "location": "Weston-super-Mare, United Kingdom"}, {"name": "G3SDR, 0-32 MHz Kiwi 8079. Weston-super-Mare. Parallel dLine truncated
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"_comment": [
|
||||||
|
"Pinned SHA-256 digests for the Tor Expert Bundle archives we know how to install.",
|
||||||
|
"Used as the LAST-RESORT verification source when the upstream .sha256sum file is",
|
||||||
|
"unreachable, MITM'd, or doesn't match what we downloaded. Issue #201.",
|
||||||
|
"",
|
||||||
|
"Each entry is keyed by the archive URL (so multiple platforms / versions",
|
||||||
|
"can share this one file) and contains the canonical SHA-256 we trust.",
|
||||||
|
"",
|
||||||
|
"When the project tests a new Tor release, add its digest here in the same",
|
||||||
|
"PR that bumps _TOR_EXPERT_BUNDLE_URLS. Old entries are kept indefinitely so",
|
||||||
|
"users on older versions keep working — we only ever ADD here, never remove."
|
||||||
|
],
|
||||||
|
"https://dist.torproject.org/torbrowser/15.0.11/tor-expert-bundle-windows-x86_64-15.0.11.tar.gz": "PLACEHOLDER_REPLACE_BEFORE_RELEASE",
|
||||||
|
"https://dist.torproject.org/torbrowser/15.0.8/tor-expert-bundle-windows-x86_64-15.0.8.tar.gz": "PLACEHOLDER_REPLACE_BEFORE_RELEASE"
|
||||||
|
}
|
||||||
+28
-1
@@ -611,6 +611,23 @@ class OverflightRequest(BaseModel):
|
|||||||
hours: int = 24
|
hours: int = 24
|
||||||
|
|
||||||
|
|
||||||
|
# Issue #202: compute_overflights() is O(catalog_size × timesteps), where
|
||||||
|
# timesteps grows linearly with `hours`. An unbounded `hours` value is a
|
||||||
|
# trivial CPU-exhaustion vector. We clamp silently rather than raising 422 —
|
||||||
|
# the response shape is unchanged, callers asking for too many hours just
|
||||||
|
# get a shorter window, which is friendlier than a hostile error.
|
||||||
|
#
|
||||||
|
# Override via OVERFLIGHTS_MAX_HOURS env var if you legitimately need a
|
||||||
|
# longer window (e.g. a planning use case that wants a full week).
|
||||||
|
def _overflight_max_hours() -> int:
|
||||||
|
import os as _os
|
||||||
|
try:
|
||||||
|
raw = int(str(_os.environ.get("OVERFLIGHTS_MAX_HOURS", "72")).strip())
|
||||||
|
except (TypeError, ValueError):
|
||||||
|
raw = 72
|
||||||
|
return max(1, raw)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/api/satellites/overflights")
|
@router.post("/api/satellites/overflights")
|
||||||
@limiter.limit("10/minute")
|
@limiter.limit("10/minute")
|
||||||
async def satellite_overflights(request: Request, body: OverflightRequest):
|
async def satellite_overflights(request: Request, body: OverflightRequest):
|
||||||
@@ -619,5 +636,15 @@ async def satellite_overflights(request: Request, body: OverflightRequest):
|
|||||||
if not gp_data:
|
if not gp_data:
|
||||||
return JSONResponse({"total": 0, "by_mission": {}, "satellites": [], "error": "No GP data cached yet"})
|
return JSONResponse({"total": 0, "by_mission": {}, "satellites": [], "error": "No GP data cached yet"})
|
||||||
bbox = {"s": body.s, "w": body.w, "n": body.n, "e": body.e}
|
bbox = {"s": body.s, "w": body.w, "n": body.n, "e": body.e}
|
||||||
result = compute_overflights(gp_data, bbox, hours=body.hours)
|
|
||||||
|
# Silent clamp — see comment on _overflight_max_hours().
|
||||||
|
requested_hours = max(1, int(body.hours or 0))
|
||||||
|
effective_hours = min(requested_hours, _overflight_max_hours())
|
||||||
|
|
||||||
|
result = compute_overflights(gp_data, bbox, hours=effective_hours)
|
||||||
|
# If we clamped, surface the effective window in the response so the
|
||||||
|
# caller can detect it if they care, without it being an error.
|
||||||
|
if isinstance(result, dict) and effective_hours != requested_hours:
|
||||||
|
result.setdefault("requested_hours", requested_hours)
|
||||||
|
result.setdefault("effective_hours", effective_hours)
|
||||||
return JSONResponse(result)
|
return JSONResponse(result)
|
||||||
@@ -1467,25 +1467,37 @@ def _submit_gate_message_envelope(request: Request, gate_id: str, body: dict[str
|
|||||||
@router.get("/api/mesh/infonet/status")
|
@router.get("/api/mesh/infonet/status")
|
||||||
@limiter.limit("30/minute")
|
@limiter.limit("30/minute")
|
||||||
async def infonet_status(request: Request, verify_signatures: bool = False):
|
async def infonet_status(request: Request, verify_signatures: bool = False):
|
||||||
"""Get Infonet metadata — event counts, head hash, chain size."""
|
"""Get Infonet metadata — event counts, head hash, chain size.
|
||||||
|
|
||||||
|
The ``verify_signatures`` query parameter is honored ONLY when the
|
||||||
|
caller has authenticated via scoped auth or local-operator credentials.
|
||||||
|
Verifying every signature in a long chain is O(n_events) work — letting
|
||||||
|
anonymous callers trigger it is a DoS surface (issue #207). For
|
||||||
|
anonymous callers we silently fall back to the cheap path; the response
|
||||||
|
structure is identical so legitimate frontends see no behavior change.
|
||||||
|
"""
|
||||||
from services.mesh.mesh_hashchain import infonet
|
from services.mesh.mesh_hashchain import infonet
|
||||||
from services.wormhole_supervisor import get_wormhole_state
|
from services.wormhole_supervisor import get_wormhole_state
|
||||||
|
|
||||||
|
# Silently downgrade for unauthenticated callers — no error surfaced.
|
||||||
|
authenticated = _scoped_view_authenticated(request, "mesh.audit")
|
||||||
|
effective_verify_signatures = bool(verify_signatures) and authenticated
|
||||||
|
|
||||||
info = infonet.get_info()
|
info = infonet.get_info()
|
||||||
valid, reason = infonet.validate_chain(verify_signatures=verify_signatures)
|
valid, reason = infonet.validate_chain(verify_signatures=effective_verify_signatures)
|
||||||
try:
|
try:
|
||||||
wormhole = get_wormhole_state()
|
wormhole = get_wormhole_state()
|
||||||
except Exception:
|
except Exception:
|
||||||
wormhole = {"configured": False, "ready": False, "rns_ready": False}
|
wormhole = {"configured": False, "ready": False, "rns_ready": False}
|
||||||
info["valid"] = valid
|
info["valid"] = valid
|
||||||
info["validation"] = reason
|
info["validation"] = reason
|
||||||
info["verify_signatures"] = verify_signatures
|
info["verify_signatures"] = effective_verify_signatures
|
||||||
info["private_lane_tier"] = _current_private_lane_tier(wormhole)
|
info["private_lane_tier"] = _current_private_lane_tier(wormhole)
|
||||||
info["private_lane_policy"] = _private_infonet_policy_snapshot()
|
info["private_lane_policy"] = _private_infonet_policy_snapshot()
|
||||||
info.update(_node_runtime_snapshot())
|
info.update(_node_runtime_snapshot())
|
||||||
return _redact_private_lane_control_fields(
|
return _redact_private_lane_control_fields(
|
||||||
info,
|
info,
|
||||||
authenticated=_scoped_view_authenticated(request, "mesh.audit"),
|
authenticated=authenticated,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|||||||
@@ -21,14 +21,30 @@ async def api_get_openmhz_systems(request: Request):
|
|||||||
return get_openmhz_systems()
|
return get_openmhz_systems()
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/radio/openmhz/calls/{sys_name}")
|
# Issue #213: rotating sys_name bypasses the 20s TTL cache and lets an
|
||||||
|
# anonymous caller hammer api.openmhz.com through this proxy, risking an
|
||||||
|
# IP-ban for the project. require_local_operator scopes this to the local
|
||||||
|
# UI (which goes through the Next.js proxy with admin-key injection) and
|
||||||
|
# scoped agent tokens.
|
||||||
|
@router.get(
|
||||||
|
"/api/radio/openmhz/calls/{sys_name}",
|
||||||
|
dependencies=[Depends(require_local_operator)],
|
||||||
|
)
|
||||||
@limiter.limit("60/minute")
|
@limiter.limit("60/minute")
|
||||||
async def api_get_openmhz_calls(request: Request, sys_name: str):
|
async def api_get_openmhz_calls(request: Request, sys_name: str):
|
||||||
from services.radio_intercept import get_recent_openmhz_calls
|
from services.radio_intercept import get_recent_openmhz_calls
|
||||||
return get_recent_openmhz_calls(sys_name)
|
return get_recent_openmhz_calls(sys_name)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/radio/openmhz/audio")
|
# Issue #214: this is a streaming bandwidth relay. An anonymous caller can
|
||||||
|
# stream audio through the backend, saturating the operator's outbound
|
||||||
|
# bandwidth. Scope to local operator; the legitimate browser UI still
|
||||||
|
# works because relative /api/... paths go through the Next.js proxy
|
||||||
|
# which injects the admin key automatically.
|
||||||
|
@router.get(
|
||||||
|
"/api/radio/openmhz/audio",
|
||||||
|
dependencies=[Depends(require_local_operator)],
|
||||||
|
)
|
||||||
@limiter.limit("120/minute")
|
@limiter.limit("120/minute")
|
||||||
async def api_get_openmhz_audio(request: Request, url: str = Query(..., min_length=10)):
|
async def api_get_openmhz_audio(request: Request, url: str = Query(..., min_length=10)):
|
||||||
from services.radio_intercept import openmhz_audio_response
|
from services.radio_intercept import openmhz_audio_response
|
||||||
|
|||||||
@@ -21,7 +21,7 @@ async def oracle_region_intel(
|
|||||||
return get_region_oracle_intel(lat, lng, news_items)
|
return get_region_oracle_intel(lat, lng, news_items)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/api/thermal/verify")
|
@router.get("/api/thermal/verify", dependencies=[Depends(require_local_operator)])
|
||||||
@limiter.limit("10/minute")
|
@limiter.limit("10/minute")
|
||||||
async def thermal_verify(
|
async def thermal_verify(
|
||||||
request: Request,
|
request: Request,
|
||||||
|
|||||||
@@ -174,17 +174,29 @@ def fetch_meshtastic_nodes():
|
|||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.debug(f"Meshtastic cache freshness check failed: {e}")
|
logger.debug(f"Meshtastic cache freshness check failed: {e}")
|
||||||
|
|
||||||
# Build a polite User-Agent. Include the operator callsign when set so
|
# Build a polite User-Agent. Historically this included the operator
|
||||||
# the upstream service can correlate per-install traffic if needed.
|
# callsign so meshtastic.org could rate-limit per-install; that's still
|
||||||
|
# the default behavior for backward compatibility. Operators who want
|
||||||
|
# stricter outbound privacy can suppress the callsign by setting
|
||||||
|
# MESHTASTIC_SEND_CALLSIGN_HEADER=false. Issue #203.
|
||||||
|
import os as _os
|
||||||
try:
|
try:
|
||||||
from services.config import get_settings
|
from services.config import get_settings
|
||||||
|
|
||||||
callsign = str(getattr(get_settings(), "MESHTASTIC_OPERATOR_CALLSIGN", "") or "").strip()
|
callsign = str(getattr(get_settings(), "MESHTASTIC_OPERATOR_CALLSIGN", "") or "").strip()
|
||||||
except Exception:
|
except Exception:
|
||||||
callsign = ""
|
callsign = ""
|
||||||
|
|
||||||
|
send_callsign_header = str(
|
||||||
|
_os.environ.get("MESHTASTIC_SEND_CALLSIGN_HEADER", "true")
|
||||||
|
).strip().lower() not in {"0", "false", "no", "off", ""}
|
||||||
|
|
||||||
from services.network_utils import DEFAULT_USER_AGENT
|
from services.network_utils import DEFAULT_USER_AGENT
|
||||||
ua_base = f"{DEFAULT_USER_AGENT}; 24h polling"
|
ua_base = f"{DEFAULT_USER_AGENT}; 24h polling"
|
||||||
user_agent = f"{ua_base}; node={callsign}" if callsign else ua_base
|
if callsign and send_callsign_header:
|
||||||
|
user_agent = f"{ua_base}; node={callsign}"
|
||||||
|
else:
|
||||||
|
user_agent = ua_base
|
||||||
|
|
||||||
try:
|
try:
|
||||||
logger.info("Fetching Meshtastic map nodes from API...")
|
logger.info("Fetching Meshtastic map nodes from API...")
|
||||||
|
|||||||
@@ -34,6 +34,20 @@ kiwisdr_cache: TTLCache = TTLCache(maxsize=1, ttl=_REFRESH_SECONDS)
|
|||||||
|
|
||||||
_SOURCE_URL = "http://rx.linkfanel.net/kiwisdr_com.js"
|
_SOURCE_URL = "http://rx.linkfanel.net/kiwisdr_com.js"
|
||||||
_CACHE_FILE = Path(__file__).resolve().parent.parent / "data" / "kiwisdr_cache.json"
|
_CACHE_FILE = Path(__file__).resolve().parent.parent / "data" / "kiwisdr_cache.json"
|
||||||
|
# Bundled fallback — shipped with the codebase so the KiwiSDR layer always
|
||||||
|
# has something to render even when the upstream is unreachable, returns
|
||||||
|
# garbage, or appears to have been tampered with. Issue #206: the upstream
|
||||||
|
# only speaks HTTP, so we can't rely on TLS for integrity — instead we
|
||||||
|
# validate the response's shape and fall back to this bundle if it doesn't
|
||||||
|
# look right.
|
||||||
|
_BUNDLED_FALLBACK = Path(__file__).resolve().parent.parent / "data" / "kiwisdr_directory.json"
|
||||||
|
|
||||||
|
# Minimum number of receivers we expect from a healthy upstream response.
|
||||||
|
# The KiwiSDR public network has consistently sat well above this threshold
|
||||||
|
# for years. If we see fewer than this many parsed receivers, treat the
|
||||||
|
# response as suspect and fall back. Tune via env if the upstream shrinks
|
||||||
|
# legitimately.
|
||||||
|
_MIN_HEALTHY_RECEIVER_COUNT = 50
|
||||||
_LINE_COMMENT_RE = re.compile(r"^\s*//.*$", re.MULTILINE)
|
_LINE_COMMENT_RE = re.compile(r"^\s*//.*$", re.MULTILINE)
|
||||||
_VAR_PREFIX_RE = re.compile(r"^\s*var\s+kiwisdr_com\s*=\s*", re.MULTILINE)
|
_VAR_PREFIX_RE = re.compile(r"^\s*var\s+kiwisdr_com\s*=\s*", re.MULTILINE)
|
||||||
_TRAILING_COMMA_RE = re.compile(r",(\s*[\]}])")
|
_TRAILING_COMMA_RE = re.compile(r",(\s*[\]}])")
|
||||||
@@ -135,12 +149,72 @@ def _parse_mirror_payload(body: str) -> list[dict]:
|
|||||||
return nodes
|
return nodes
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_fetched_nodes(nodes: list[dict]) -> bool:
|
||||||
|
"""Sanity-check freshly-fetched receiver data before trusting it.
|
||||||
|
|
||||||
|
The upstream (rx.linkfanel.net) speaks only HTTP — there is no TLS to
|
||||||
|
authenticate the response. A passive MITM could inject doctored
|
||||||
|
receiver positions (false pins on the map) or strip the response down
|
||||||
|
to a tiny subset. We can't prevent the modification at the transport
|
||||||
|
layer, but we can refuse to commit to obviously-bad responses.
|
||||||
|
|
||||||
|
Returns True if the parsed list looks reasonable. False means we
|
||||||
|
should fall back to a previously-cached or bundled directory.
|
||||||
|
"""
|
||||||
|
if not isinstance(nodes, list):
|
||||||
|
return False
|
||||||
|
if len(nodes) < _MIN_HEALTHY_RECEIVER_COUNT:
|
||||||
|
# Either upstream is degraded or someone is feeding us a stripped
|
||||||
|
# response. Either way, the bundled fallback is more useful.
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Spot-check: every entry should have a name, a parsed lat/lon, and a
|
||||||
|
# URL field. If more than 5% of entries are missing core fields, the
|
||||||
|
# parse went sideways.
|
||||||
|
missing_core = 0
|
||||||
|
for entry in nodes:
|
||||||
|
if not isinstance(entry, dict):
|
||||||
|
missing_core += 1
|
||||||
|
continue
|
||||||
|
if not entry.get("name") or not isinstance(entry.get("lat"), (int, float)):
|
||||||
|
missing_core += 1
|
||||||
|
if missing_core > max(5, len(nodes) // 20):
|
||||||
|
return False
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def _load_bundled_fallback() -> list[dict]:
|
||||||
|
"""Last-resort directory shipped with the codebase. Always returns a
|
||||||
|
list (may be empty if the bundle is missing in older deployments)."""
|
||||||
|
if not _BUNDLED_FALLBACK.exists():
|
||||||
|
return []
|
||||||
|
try:
|
||||||
|
data = json.loads(_BUNDLED_FALLBACK.read_text(encoding="utf-8"))
|
||||||
|
if isinstance(data, list):
|
||||||
|
return data
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"KiwiSDR bundled fallback unreadable: {e}")
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
@cached(kiwisdr_cache)
|
@cached(kiwisdr_cache)
|
||||||
def fetch_kiwisdr_nodes() -> list[dict]:
|
def fetch_kiwisdr_nodes() -> list[dict]:
|
||||||
"""Return the KiwiSDR receiver list, refreshed at most once per day.
|
"""Return the KiwiSDR receiver list, refreshed at most once per day.
|
||||||
|
|
||||||
Order of preference: in-memory cache (handled by @cached) → on-disk cache
|
Layered fallback (issue #206 — upstream is HTTP-only, so we defend with
|
||||||
if <24h old → network fetch from rx.linkfanel.net.
|
content validation + bundled static directory rather than trying to
|
||||||
|
upgrade the transport):
|
||||||
|
|
||||||
|
1. In-memory cache (handled by @cached on this function)
|
||||||
|
2. On-disk cache if <24h old
|
||||||
|
3. Fresh network fetch from rx.linkfanel.net → validated → committed
|
||||||
|
4. Stale on-disk cache (>24h) if validation fails
|
||||||
|
5. Bundled static directory at backend/data/kiwisdr_directory.json
|
||||||
|
|
||||||
|
The KiwiSDR map layer renders something useful in every case. A
|
||||||
|
tampered upstream returning garbage is caught by _validate_fetched_nodes()
|
||||||
|
and falls through to whatever previously-trusted snapshot we have.
|
||||||
"""
|
"""
|
||||||
from services.network_utils import fetch_with_curl
|
from services.network_utils import fetch_with_curl
|
||||||
|
|
||||||
@@ -153,34 +227,57 @@ def fetch_kiwisdr_nodes() -> list[dict]:
|
|||||||
return cached_nodes
|
return cached_nodes
|
||||||
|
|
||||||
# 2. Cache cold or stale — fetch from network.
|
# 2. Cache cold or stale — fetch from network.
|
||||||
|
fresh_nodes: list[dict] = []
|
||||||
|
fetch_succeeded = False
|
||||||
try:
|
try:
|
||||||
res = fetch_with_curl(_SOURCE_URL, timeout=20)
|
res = fetch_with_curl(_SOURCE_URL, timeout=20)
|
||||||
if not res or res.status_code != 200:
|
if res and res.status_code == 200:
|
||||||
logger.error(
|
fresh_nodes = _parse_mirror_payload(res.text)
|
||||||
f"KiwiSDR fetch failed: HTTP {res.status_code if res else 'no response'}"
|
fetch_succeeded = True
|
||||||
|
else:
|
||||||
|
logger.warning(
|
||||||
|
f"KiwiSDR fetch returned HTTP {res.status_code if res else 'no response'}"
|
||||||
)
|
)
|
||||||
return []
|
|
||||||
|
|
||||||
nodes = _parse_mirror_payload(res.text)
|
|
||||||
if nodes:
|
|
||||||
_save_disk_cache(nodes)
|
|
||||||
logger.info(
|
|
||||||
f"KiwiSDR: refreshed {len(nodes)} receivers from rx.linkfanel.net "
|
|
||||||
"(next refresh in 24h)"
|
|
||||||
)
|
|
||||||
return nodes
|
|
||||||
|
|
||||||
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
|
except (requests.RequestException, ConnectionError, TimeoutError, ValueError, KeyError) as e:
|
||||||
logger.error(f"KiwiSDR fetch exception: {e}")
|
logger.warning(f"KiwiSDR fetch exception: {e}")
|
||||||
# Fall back to a stale disk cache if one exists, even if >24h old.
|
|
||||||
if _CACHE_FILE.exists():
|
# 3. Validate before committing. If the response looks healthy, save
|
||||||
try:
|
# it as the new cache and return.
|
||||||
stale = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
|
if fetch_succeeded and _validate_fetched_nodes(fresh_nodes):
|
||||||
if isinstance(stale, list):
|
_save_disk_cache(fresh_nodes)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"KiwiSDR: serving {len(stale)} stale receivers from disk after fetch failure"
|
f"KiwiSDR: refreshed {len(fresh_nodes)} receivers from rx.linkfanel.net "
|
||||||
)
|
"(next refresh in 24h)"
|
||||||
return stale
|
)
|
||||||
except Exception:
|
return fresh_nodes
|
||||||
pass
|
|
||||||
return []
|
if fetch_succeeded:
|
||||||
|
# Network came back, but the payload didn't pass validation —
|
||||||
|
# either upstream is degraded or a MITM is at work. Fall through
|
||||||
|
# to a trusted snapshot rather than committing garbage to disk.
|
||||||
|
logger.warning(
|
||||||
|
"KiwiSDR: upstream response failed validation (%d entries) — "
|
||||||
|
"falling back to trusted snapshot",
|
||||||
|
len(fresh_nodes),
|
||||||
|
)
|
||||||
|
|
||||||
|
# 4. Stale on-disk cache, if any.
|
||||||
|
if _CACHE_FILE.exists():
|
||||||
|
try:
|
||||||
|
stale = json.loads(_CACHE_FILE.read_text(encoding="utf-8"))
|
||||||
|
if isinstance(stale, list) and stale:
|
||||||
|
logger.info(
|
||||||
|
f"KiwiSDR: serving {len(stale)} stale receivers from disk"
|
||||||
|
)
|
||||||
|
return stale
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# 5. Bundled static directory — last resort, always works.
|
||||||
|
bundled = _load_bundled_fallback()
|
||||||
|
if bundled:
|
||||||
|
logger.info(
|
||||||
|
f"KiwiSDR: serving {len(bundled)} receivers from bundled fallback "
|
||||||
|
"(no fresh fetch + no disk cache available)"
|
||||||
|
)
|
||||||
|
return bundled
|
||||||
@@ -1444,9 +1444,51 @@ class Infonet:
|
|||||||
self._save_lock = threading.Lock()
|
self._save_lock = threading.Lock()
|
||||||
self._save_timer: threading.Timer | None = None
|
self._save_timer: threading.Timer | None = None
|
||||||
self._SAVE_INTERVAL = 5.0 # seconds — coalesce writes
|
self._SAVE_INTERVAL = 5.0 # seconds — coalesce writes
|
||||||
|
# Issue #208: Merkle levels cache so get_merkle_proofs() doesn't
|
||||||
|
# rebuild O(n) levels on every public call. Invalidated whenever
|
||||||
|
# self.events mutates. Computed lazily on first read after an
|
||||||
|
# invalidation.
|
||||||
|
self._merkle_levels_cache: list[list[str]] | None = None
|
||||||
|
self._merkle_levels_for_event_count: int = -1
|
||||||
atexit.register(self._flush)
|
atexit.register(self._flush)
|
||||||
self._load()
|
self._load()
|
||||||
|
|
||||||
|
def _invalidate_merkle_cache(self) -> None:
|
||||||
|
"""Clear the precomputed Merkle levels.
|
||||||
|
|
||||||
|
Called whenever ``self.events`` may have mutated (append, rebuild,
|
||||||
|
cleanup, fork resolution). The next call to ``get_merkle_root()``
|
||||||
|
or ``get_merkle_proofs()`` will recompute and re-cache.
|
||||||
|
"""
|
||||||
|
self._merkle_levels_cache = None
|
||||||
|
self._merkle_levels_for_event_count = -1
|
||||||
|
|
||||||
|
def _get_merkle_levels(self) -> list[list[str]]:
|
||||||
|
"""Return Merkle levels for the current chain, recomputing if
|
||||||
|
the cache is invalid or out of date.
|
||||||
|
|
||||||
|
Issue #208: a public endpoint (``/api/mesh/infonet/sync?include_proofs=true``)
|
||||||
|
used to rebuild Merkle levels on every request, which is O(n) in
|
||||||
|
chain length and trivially abusable for CPU exhaustion. By caching
|
||||||
|
the levels and invalidating on mutation, repeated proof requests
|
||||||
|
become O(1) per proof; the rebuild only happens after a genuine
|
||||||
|
append/rebuild/cleanup.
|
||||||
|
"""
|
||||||
|
from services.mesh.mesh_merkle import build_merkle_levels
|
||||||
|
|
||||||
|
current_count = len(self.events)
|
||||||
|
if (
|
||||||
|
self._merkle_levels_cache is not None
|
||||||
|
and self._merkle_levels_for_event_count == current_count
|
||||||
|
):
|
||||||
|
return self._merkle_levels_cache
|
||||||
|
|
||||||
|
leaves = [e["event_id"] for e in self.events]
|
||||||
|
levels = build_merkle_levels(leaves)
|
||||||
|
self._merkle_levels_cache = levels
|
||||||
|
self._merkle_levels_for_event_count = current_count
|
||||||
|
return levels
|
||||||
|
|
||||||
# ─── Persistence ──────────────────────────────────────────────────
|
# ─── Persistence ──────────────────────────────────────────────────
|
||||||
|
|
||||||
def _load(self):
|
def _load(self):
|
||||||
@@ -1983,6 +2025,8 @@ class Infonet:
|
|||||||
self.head_hash = event.event_id
|
self.head_hash = event.event_id
|
||||||
self.node_sequences[node_id] = sequence
|
self.node_sequences[node_id] = sequence
|
||||||
self._replay_filter.add(event.event_id)
|
self._replay_filter.add(event.event_id)
|
||||||
|
# Issue #208: chain advanced, cached Merkle levels are stale.
|
||||||
|
self._invalidate_merkle_cache()
|
||||||
self._update_counters_for_event(event_dict)
|
self._update_counters_for_event(event_dict)
|
||||||
|
|
||||||
if event_type == "key_revoke":
|
if event_type == "key_revoke":
|
||||||
@@ -2266,6 +2310,9 @@ class Infonet:
|
|||||||
self._apply_revocation(evt)
|
self._apply_revocation(evt)
|
||||||
|
|
||||||
if accepted:
|
if accepted:
|
||||||
|
# Issue #208: any accepted event invalidates the cached Merkle
|
||||||
|
# levels. One invalidation per batch, not per event.
|
||||||
|
self._invalidate_merkle_cache()
|
||||||
self._save()
|
self._save()
|
||||||
return {"accepted": accepted, "duplicates": duplicates, "rejected": rejected}
|
return {"accepted": accepted, "duplicates": duplicates, "rejected": rejected}
|
||||||
|
|
||||||
@@ -2566,6 +2613,8 @@ class Infonet:
|
|||||||
self._rebuild_state()
|
self._rebuild_state()
|
||||||
self._rebuild_revocations()
|
self._rebuild_revocations()
|
||||||
self._rebuild_counters()
|
self._rebuild_counters()
|
||||||
|
# Issue #208: chain replaced, cached Merkle levels are stale.
|
||||||
|
self._invalidate_merkle_cache()
|
||||||
self._save()
|
self._save()
|
||||||
try:
|
try:
|
||||||
from services.mesh.mesh_metrics import increment as metrics_inc
|
from services.mesh.mesh_metrics import increment as metrics_inc
|
||||||
@@ -2735,6 +2784,8 @@ class Infonet:
|
|||||||
self._rebuild_state()
|
self._rebuild_state()
|
||||||
self._rebuild_revocations()
|
self._rebuild_revocations()
|
||||||
self._rebuild_counters()
|
self._rebuild_counters()
|
||||||
|
# Issue #208: cleanup may have dropped expired events.
|
||||||
|
self._invalidate_merkle_cache()
|
||||||
self._save()
|
self._save()
|
||||||
logger.info(f"Infonet cleanup: removed {before - len(new_events)} expired events")
|
logger.info(f"Infonet cleanup: removed {before - len(new_events)} expired events")
|
||||||
|
|
||||||
@@ -2743,30 +2794,37 @@ class Infonet:
|
|||||||
def get_merkle_root(self) -> str:
|
def get_merkle_root(self) -> str:
|
||||||
"""Compute a Merkle root hash of the Infonet for sync comparison.
|
"""Compute a Merkle root hash of the Infonet for sync comparison.
|
||||||
|
|
||||||
Two nodes with the same Merkle root have identical chains.
|
Two nodes with the same Merkle root have identical chains. Reads
|
||||||
|
from the cached Merkle levels (issue #208) — O(1) when the chain
|
||||||
|
hasn't changed since the last computation.
|
||||||
"""
|
"""
|
||||||
if not self.events:
|
if not self.events:
|
||||||
return GENESIS_HASH
|
return GENESIS_HASH
|
||||||
|
|
||||||
from services.mesh.mesh_merkle import merkle_root
|
levels = self._get_merkle_levels()
|
||||||
|
if not levels or not levels[-1]:
|
||||||
leaves = [e["event_id"] for e in self.events]
|
return GENESIS_HASH
|
||||||
root = merkle_root(leaves)
|
return levels[-1][0] or GENESIS_HASH
|
||||||
return root or GENESIS_HASH
|
|
||||||
|
|
||||||
def get_merkle_proofs(self, start_index: int, count: int) -> dict:
|
def get_merkle_proofs(self, start_index: int, count: int) -> dict:
|
||||||
"""Return merkle proofs for a contiguous range of events."""
|
"""Return merkle proofs for a contiguous range of events.
|
||||||
leaves = [e["event_id"] for e in self.events]
|
|
||||||
total = len(leaves)
|
Issue #208: uses the cached Merkle levels so this is O(count *
|
||||||
|
log n) per request, not O(n + count * log n). Anonymous peers
|
||||||
|
hitting ``/api/mesh/infonet/sync?include_proofs=true`` no longer
|
||||||
|
force a rebuild on every call.
|
||||||
|
"""
|
||||||
|
total = len(self.events)
|
||||||
if total == 0:
|
if total == 0:
|
||||||
return {"root": GENESIS_HASH, "total": 0, "start": 0, "proofs": []}
|
return {"root": GENESIS_HASH, "total": 0, "start": 0, "proofs": []}
|
||||||
|
|
||||||
from services.mesh.mesh_merkle import build_merkle_levels, merkle_proof_from_levels
|
from services.mesh.mesh_merkle import merkle_proof_from_levels
|
||||||
|
|
||||||
|
leaves = [e["event_id"] for e in self.events]
|
||||||
start = max(0, start_index)
|
start = max(0, start_index)
|
||||||
end = min(total, start + max(0, count))
|
end = min(total, start + max(0, count))
|
||||||
levels = build_merkle_levels(leaves)
|
levels = self._get_merkle_levels()
|
||||||
root = levels[-1][0] if levels else GENESIS_HASH
|
root = levels[-1][0] if levels and levels[-1] else GENESIS_HASH
|
||||||
|
|
||||||
proofs = []
|
proofs = []
|
||||||
for idx in range(start, end):
|
for idx in range(start, end):
|
||||||
|
|||||||
@@ -131,27 +131,61 @@ def get_recent_openmhz_calls(sys_name: str):
|
|||||||
return []
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
_OPENMHZ_MAX_REDIRECTS = 5
|
||||||
|
|
||||||
|
|
||||||
def openmhz_audio_response(target_url: str):
|
def openmhz_audio_response(target_url: str):
|
||||||
"""Fetch an OpenMHz audio object through the backend with browser-safe headers."""
|
"""Fetch an OpenMHz audio object through the backend with browser-safe headers.
|
||||||
|
|
||||||
|
Redirects are followed manually so each hop's host can be re-validated
|
||||||
|
against ``_OPENMHZ_AUDIO_HOSTS``. Without this, the upstream could
|
||||||
|
302-redirect to an internal address (e.g. ``http://127.0.0.1:8000/...``
|
||||||
|
or an RFC1918 range), and the backend would dutifully fetch and stream
|
||||||
|
that response back to the browser — a classic open-redirect-to-SSRF
|
||||||
|
chain. Same-host redirects (CDN edge selection) still work normally.
|
||||||
|
"""
|
||||||
from fastapi import HTTPException
|
from fastapi import HTTPException
|
||||||
from fastapi.responses import StreamingResponse
|
from fastapi.responses import StreamingResponse
|
||||||
|
from urllib.parse import urljoin
|
||||||
|
|
||||||
parsed = urlparse(str(target_url or ""))
|
parsed = urlparse(str(target_url or ""))
|
||||||
host = (parsed.hostname or "").lower()
|
host = (parsed.hostname or "").lower()
|
||||||
if parsed.scheme != "https" or host not in _OPENMHZ_AUDIO_HOSTS:
|
if parsed.scheme != "https" or host not in _OPENMHZ_AUDIO_HOSTS:
|
||||||
raise HTTPException(status_code=400, detail="Unsupported OpenMHz audio URL")
|
raise HTTPException(status_code=400, detail="Unsupported OpenMHz audio URL")
|
||||||
|
|
||||||
|
current_url = target_url
|
||||||
|
hops = 0
|
||||||
try:
|
try:
|
||||||
upstream = requests.get(
|
while True:
|
||||||
target_url,
|
upstream = requests.get(
|
||||||
stream=True,
|
current_url,
|
||||||
timeout=(5, 20),
|
stream=True,
|
||||||
headers={
|
timeout=(5, 20),
|
||||||
"User-Agent": "Mozilla/5.0",
|
allow_redirects=False,
|
||||||
"Accept": "audio/mpeg,audio/*,*/*;q=0.8",
|
headers={
|
||||||
"Referer": "https://openmhz.com/",
|
"User-Agent": "Mozilla/5.0",
|
||||||
},
|
"Accept": "audio/mpeg,audio/*,*/*;q=0.8",
|
||||||
)
|
"Referer": "https://openmhz.com/",
|
||||||
|
},
|
||||||
|
)
|
||||||
|
if upstream.is_redirect or upstream.status_code in (301, 302, 303, 307, 308):
|
||||||
|
location = upstream.headers.get("Location", "")
|
||||||
|
upstream.close()
|
||||||
|
if hops >= _OPENMHZ_MAX_REDIRECTS or not location:
|
||||||
|
raise HTTPException(status_code=502, detail="OpenMHz redirect rejected")
|
||||||
|
next_url = urljoin(current_url, location)
|
||||||
|
next_parsed = urlparse(next_url)
|
||||||
|
next_host = (next_parsed.hostname or "").lower()
|
||||||
|
# Re-validate the next hop against the same allowlist used for
|
||||||
|
# the original URL. Cross-host redirects to disallowed hosts
|
||||||
|
# are rejected silently; the browser audio element handles
|
||||||
|
# the resulting 502 gracefully and moves on.
|
||||||
|
if next_parsed.scheme != "https" or next_host not in _OPENMHZ_AUDIO_HOSTS:
|
||||||
|
raise HTTPException(status_code=502, detail="OpenMHz redirect rejected")
|
||||||
|
current_url = next_url
|
||||||
|
hops += 1
|
||||||
|
continue
|
||||||
|
break
|
||||||
except requests.RequestException as exc:
|
except requests.RequestException as exc:
|
||||||
raise HTTPException(status_code=502, detail="OpenMHz audio fetch failed") from exc
|
raise HTTPException(status_code=502, detail="OpenMHz audio fetch failed") from exc
|
||||||
|
|
||||||
|
|||||||
@@ -64,6 +64,115 @@ def _find_tor_binary() -> str | None:
|
|||||||
return None
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# Baked-in expected digest list. Loaded lazily; populated by maintainers
|
||||||
|
# when a new Tor Expert Bundle URL is added to _TOR_EXPERT_BUNDLE_URLS.
|
||||||
|
# See issue #201 for rationale.
|
||||||
|
_TOR_DIGEST_FILE = Path(__file__).resolve().parent.parent / "data" / "tor_bundle_digests.json"
|
||||||
|
_DIGEST_PLACEHOLDER = "PLACEHOLDER_REPLACE_BEFORE_RELEASE"
|
||||||
|
|
||||||
|
|
||||||
|
def _load_baked_in_digests() -> dict[str, str]:
|
||||||
|
"""Return {url: expected_sha256_lower} for URLs we ship a known digest for.
|
||||||
|
|
||||||
|
Entries whose value is the placeholder sentinel are filtered out — they
|
||||||
|
represent versions the maintainer has not yet pinned, and we don't
|
||||||
|
want to trust them via this layer.
|
||||||
|
"""
|
||||||
|
if not _TOR_DIGEST_FILE.exists():
|
||||||
|
return {}
|
||||||
|
try:
|
||||||
|
import json as _json
|
||||||
|
raw = _json.loads(_TOR_DIGEST_FILE.read_text(encoding="utf-8"))
|
||||||
|
except Exception as exc:
|
||||||
|
logger.warning("Tor bundle digests file unreadable: %s", exc)
|
||||||
|
return {}
|
||||||
|
result: dict[str, str] = {}
|
||||||
|
for k, v in raw.items():
|
||||||
|
if not isinstance(k, str) or k.startswith("_"):
|
||||||
|
continue
|
||||||
|
if not isinstance(v, str) or v == _DIGEST_PLACEHOLDER:
|
||||||
|
continue
|
||||||
|
result[k] = v.strip().lower()
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_tor_bundle(archive_path: Path, bundle_url: str) -> tuple[bool, str]:
|
||||||
|
"""Verify the downloaded Tor bundle against any source we trust.
|
||||||
|
|
||||||
|
Returns (verified, reason). The bundle is considered verified if EITHER:
|
||||||
|
|
||||||
|
* The upstream ``.sha256sum`` file is reachable AND its digest matches
|
||||||
|
what we just downloaded, OR
|
||||||
|
* Our baked-in digest list (``backend/data/tor_bundle_digests.json``)
|
||||||
|
contains this URL AND that digest matches.
|
||||||
|
|
||||||
|
If both sources are unavailable (e.g. fresh checkout before the
|
||||||
|
maintainer has populated the digest file AND the upstream
|
||||||
|
``.sha256sum`` is unreachable), we **fall back to HTTPS-only trust**
|
||||||
|
with a warning so first-run onboarding does not break. As soon as the
|
||||||
|
digest file is populated for a shipped Tor version, the secure path
|
||||||
|
activates automatically — no operator action required.
|
||||||
|
|
||||||
|
Issue #201.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
|
||||||
|
actual_hash = hashlib.sha256(archive_path.read_bytes()).hexdigest().lower()
|
||||||
|
|
||||||
|
# Source 1: upstream .sha256sum
|
||||||
|
upstream_hash: str | None = None
|
||||||
|
sha256_url = bundle_url + ".sha256sum"
|
||||||
|
sha256_file = TOR_INSTALL_DIR / "sha256sum.txt"
|
||||||
|
try:
|
||||||
|
urlretrieve(sha256_url, str(sha256_file))
|
||||||
|
upstream_hash = sha256_file.read_text().strip().split()[0].lower()
|
||||||
|
sha256_file.unlink(missing_ok=True)
|
||||||
|
except Exception as hash_err:
|
||||||
|
logger.info("Tor bundle upstream .sha256sum unreachable: %s", hash_err)
|
||||||
|
sha256_file.unlink(missing_ok=True)
|
||||||
|
|
||||||
|
if upstream_hash and upstream_hash == actual_hash:
|
||||||
|
return True, f"verified via upstream .sha256sum ({actual_hash[:16]}...)"
|
||||||
|
|
||||||
|
# Source 2: baked-in digest list
|
||||||
|
baked = _load_baked_in_digests()
|
||||||
|
baked_hash = baked.get(bundle_url)
|
||||||
|
if baked_hash and baked_hash == actual_hash:
|
||||||
|
return True, f"verified via baked-in digest list ({actual_hash[:16]}...)"
|
||||||
|
|
||||||
|
# If we got an upstream digest AND a baked-in digest AND neither
|
||||||
|
# matched, the bundle is genuinely suspect — refuse it.
|
||||||
|
if upstream_hash and baked_hash:
|
||||||
|
return False, (
|
||||||
|
f"SHA-256 mismatch: archive={actual_hash[:16]}..., "
|
||||||
|
f"upstream={upstream_hash[:16]}..., baked={baked_hash[:16]}..."
|
||||||
|
)
|
||||||
|
if upstream_hash and upstream_hash != actual_hash:
|
||||||
|
return False, (
|
||||||
|
f"SHA-256 mismatch vs upstream: archive={actual_hash[:16]}..., "
|
||||||
|
f"upstream={upstream_hash[:16]}..."
|
||||||
|
)
|
||||||
|
if baked_hash and baked_hash != actual_hash:
|
||||||
|
return False, (
|
||||||
|
f"SHA-256 mismatch vs baked-in digest: archive={actual_hash[:16]}..., "
|
||||||
|
f"expected={baked_hash[:16]}..."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Neither verification source available. This is the fallback path for
|
||||||
|
# the case where the upstream .sha256sum is temporarily unreachable
|
||||||
|
# AND the maintainer hasn't yet pinned this Tor version. Trust HTTPS
|
||||||
|
# only (current behavior pre-#201) with a clear warning. Onboarding
|
||||||
|
# works; once we populate the digest file, the secure path activates.
|
||||||
|
logger.warning(
|
||||||
|
"Tor bundle integrity check fell back to HTTPS-only trust "
|
||||||
|
"(upstream .sha256sum unreachable AND no baked-in digest for %s). "
|
||||||
|
"Add this URL's SHA-256 to backend/data/tor_bundle_digests.json "
|
||||||
|
"to enable the secure path.",
|
||||||
|
bundle_url,
|
||||||
|
)
|
||||||
|
return True, f"https-only (no digest source reachable, archive={actual_hash[:16]}...)"
|
||||||
|
|
||||||
|
|
||||||
def _auto_install_tor() -> str | None:
|
def _auto_install_tor() -> str | None:
|
||||||
"""Install or download Tor when it is safe to do so."""
|
"""Install or download Tor when it is safe to do so."""
|
||||||
if os.name != "nt":
|
if os.name != "nt":
|
||||||
@@ -79,25 +188,17 @@ def _auto_install_tor() -> str | None:
|
|||||||
logger.info("Downloading Tor Expert Bundle over HTTPS from %s...", bundle_url)
|
logger.info("Downloading Tor Expert Bundle over HTTPS from %s...", bundle_url)
|
||||||
urlretrieve(bundle_url, str(archive_path))
|
urlretrieve(bundle_url, str(archive_path))
|
||||||
|
|
||||||
sha256_url = bundle_url + ".sha256sum"
|
# Issue #201: multi-source verification. If neither upstream
|
||||||
sha256_file = TOR_INSTALL_DIR / "sha256sum.txt"
|
# .sha256sum nor a baked-in digest matches, we refuse this URL
|
||||||
try:
|
# and try the next one in _TOR_EXPERT_BUNDLE_URLS. If neither
|
||||||
urlretrieve(sha256_url, str(sha256_file))
|
# source is reachable at all, we fall back to HTTPS-only trust
|
||||||
expected_hash = sha256_file.read_text().strip().split()[0].lower()
|
# (current behavior) rather than blocking onboarding.
|
||||||
import hashlib
|
verified, reason = _verify_tor_bundle(archive_path, bundle_url)
|
||||||
|
if not verified:
|
||||||
actual_hash = hashlib.sha256(archive_path.read_bytes()).hexdigest().lower()
|
logger.error("Tor bundle verification failed for %s: %s", bundle_url, reason)
|
||||||
sha256_file.unlink(missing_ok=True)
|
archive_path.unlink(missing_ok=True)
|
||||||
if actual_hash != expected_hash:
|
continue
|
||||||
logger.error("SHA-256 mismatch for Tor download. Expected %s, got %s", expected_hash, actual_hash)
|
logger.info("Tor bundle %s", reason)
|
||||||
archive_path.unlink(missing_ok=True)
|
|
||||||
continue
|
|
||||||
logger.info("SHA-256 verified: %s", actual_hash[:16] + "...")
|
|
||||||
except Exception as hash_err:
|
|
||||||
logger.warning(
|
|
||||||
"Could not verify SHA-256 (hash file unavailable): %s; proceeding with HTTPS-only verification",
|
|
||||||
hash_err,
|
|
||||||
)
|
|
||||||
|
|
||||||
logger.info("Download complete, extracting...")
|
logger.info("Download complete, extracting...")
|
||||||
import tarfile
|
import tarfile
|
||||||
|
|||||||
@@ -77,6 +77,18 @@ import pytest
|
|||||||
("get", "/api/wormhole/gate/general-talk/identity", None),
|
("get", "/api/wormhole/gate/general-talk/identity", None),
|
||||||
("get", "/api/wormhole/gate/general-talk/personas", None),
|
("get", "/api/wormhole/gate/general-talk/personas", None),
|
||||||
("get", "/api/wormhole/gate/general-talk/key", None),
|
("get", "/api/wormhole/gate/general-talk/key", None),
|
||||||
|
# Issue #211 (tg12): /api/thermal/verify fans out into an expensive
|
||||||
|
# STAC search + remote SWIR raster reads. Unauthenticated abuse
|
||||||
|
# could burn Sentinel-Hub quota and outbound bandwidth.
|
||||||
|
("get", "/api/thermal/verify?lat=0&lng=0&radius_km=10", None),
|
||||||
|
# Issue #213 (tg12): /api/radio/openmhz/calls/{sys_name} — rotating
|
||||||
|
# sys_name bypasses the 20s cache and hammers OpenMHZ. Risks an
|
||||||
|
# IP-ban for the project.
|
||||||
|
("get", "/api/radio/openmhz/calls/abc", None),
|
||||||
|
# Issue #214 (tg12): /api/radio/openmhz/audio — anonymous bandwidth
|
||||||
|
# relay through the backend. 60/minute rate limit is not enough on
|
||||||
|
# a streaming endpoint.
|
||||||
|
("get", "/api/radio/openmhz/audio?url=https%3A%2F%2Fmedia.openmhz.com%2Faudio%2Fabc.mp3", None),
|
||||||
],
|
],
|
||||||
)
|
)
|
||||||
def test_remote_control_surface_rejects_without_local_operator_or_admin(
|
def test_remote_control_surface_rejects_without_local_operator_or_admin(
|
||||||
|
|||||||
@@ -0,0 +1,60 @@
|
|||||||
|
"""Issue #207 (tg12): /api/mesh/infonet/status accepted
|
||||||
|
?verify_signatures=true from anonymous callers, triggering O(n_events)
|
||||||
|
signature verification across the entire chain. Trivial DoS.
|
||||||
|
|
||||||
|
The fix silently downgrades the parameter to False for unauthenticated
|
||||||
|
callers — no error surfaced, response structure unchanged, the
|
||||||
|
expensive path runs only when the caller has authenticated.
|
||||||
|
|
||||||
|
These tests focus on the source-level contract because a full
|
||||||
|
FastAPI test client doesn't have an easy hook into the ``_scoped_view_authenticated``
|
||||||
|
helper. They lock in the key invariant: the ``effective_verify_signatures``
|
||||||
|
value seen by ``validate_chain()`` is the AND of the request param and
|
||||||
|
the auth check.
|
||||||
|
"""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
|
||||||
|
_ROUTER_PATH = Path(__file__).resolve().parent.parent / "routers" / "mesh_public.py"
|
||||||
|
|
||||||
|
|
||||||
|
def _read_router_source() -> str:
|
||||||
|
return _ROUTER_PATH.read_text(encoding="utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def test_infonet_status_gates_verify_signatures():
|
||||||
|
"""The infonet_status route must AND verify_signatures with auth."""
|
||||||
|
src = _read_router_source()
|
||||||
|
# The fix introduces an `effective_verify_signatures` variable.
|
||||||
|
assert "effective_verify_signatures" in src
|
||||||
|
|
||||||
|
# It must be computed as the AND of the request param and the
|
||||||
|
# authenticated check.
|
||||||
|
assert "bool(verify_signatures) and authenticated" in src
|
||||||
|
|
||||||
|
# validate_chain() must be called with the effective value, NOT the
|
||||||
|
# raw request param.
|
||||||
|
assert "validate_chain(verify_signatures=effective_verify_signatures)" in src
|
||||||
|
|
||||||
|
|
||||||
|
def test_no_http_error_path_for_anonymous_callers():
|
||||||
|
"""No HTTPException is raised for unauthenticated verify_signatures=true.
|
||||||
|
|
||||||
|
The endpoint should silently downgrade — not return 403 — so existing
|
||||||
|
frontends that happen to pass the param see no behavior change.
|
||||||
|
"""
|
||||||
|
src = _read_router_source()
|
||||||
|
# Within the infonet_status function body, there should be no
|
||||||
|
# HTTPException(403) raised because of the verify_signatures param.
|
||||||
|
# Find the function definition and inspect the body.
|
||||||
|
import re
|
||||||
|
m = re.search(
|
||||||
|
r"async def infonet_status\(.*?\):(.+?)(?=\n@router|\nasync def |\ndef |\Z)",
|
||||||
|
src,
|
||||||
|
re.DOTALL,
|
||||||
|
)
|
||||||
|
assert m, "infonet_status function not found in source"
|
||||||
|
body = m.group(1)
|
||||||
|
# No explicit 403 around the verify_signatures handling.
|
||||||
|
assert "HTTPException(status_code=403" not in body
|
||||||
|
assert "raise HTTPException(403" not in body
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
"""Issue #206 (tg12): KiwiSDR upstream is HTTP-only and cannot be upgraded
|
||||||
|
to TLS. We defend with content validation + a bundled static directory
|
||||||
|
so the layer always renders something useful and a MITM injecting
|
||||||
|
garbage can't corrupt the map.
|
||||||
|
"""
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services import kiwisdr_fetcher
|
||||||
|
from services.kiwisdr_fetcher import (
|
||||||
|
_MIN_HEALTHY_RECEIVER_COUNT,
|
||||||
|
_load_bundled_fallback,
|
||||||
|
_validate_fetched_nodes,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_bundled_fallback_file_exists_and_is_nonempty():
|
||||||
|
"""The codebase ships a static snapshot for last-resort use."""
|
||||||
|
bundle = _load_bundled_fallback()
|
||||||
|
assert isinstance(bundle, list)
|
||||||
|
assert len(bundle) >= _MIN_HEALTHY_RECEIVER_COUNT
|
||||||
|
|
||||||
|
|
||||||
|
def test_validation_rejects_too_few_entries():
|
||||||
|
too_short = [{"name": "x", "lat": 0.0, "lon": 0.0, "url": ""}] * (_MIN_HEALTHY_RECEIVER_COUNT - 1)
|
||||||
|
assert _validate_fetched_nodes(too_short) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_validation_accepts_healthy_response():
|
||||||
|
healthy = [
|
||||||
|
{"name": f"Receiver {i}", "lat": 50.0, "lon": -1.0, "url": "http://example"}
|
||||||
|
for i in range(_MIN_HEALTHY_RECEIVER_COUNT)
|
||||||
|
]
|
||||||
|
assert _validate_fetched_nodes(healthy) is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_validation_rejects_non_list():
|
||||||
|
assert _validate_fetched_nodes(None) is False # type: ignore[arg-type]
|
||||||
|
assert _validate_fetched_nodes("a string") is False # type: ignore[arg-type]
|
||||||
|
assert _validate_fetched_nodes({}) is False # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def test_validation_rejects_too_many_malformed_entries():
|
||||||
|
"""If more than 5% of entries lack a name or numeric lat, reject."""
|
||||||
|
nodes = []
|
||||||
|
# 100 entries, 20 of them malformed — well over the 5% threshold.
|
||||||
|
for i in range(_MIN_HEALTHY_RECEIVER_COUNT + 50):
|
||||||
|
if i % 5 == 0:
|
||||||
|
nodes.append({}) # missing name + lat
|
||||||
|
else:
|
||||||
|
nodes.append({"name": f"R{i}", "lat": 50.0, "lon": -1.0, "url": ""})
|
||||||
|
assert _validate_fetched_nodes(nodes) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_fallback_used_when_validation_fails(monkeypatch, tmp_path):
|
||||||
|
"""If a fetch returns garbage, the fallback chain reaches the bundle."""
|
||||||
|
# Force disk cache miss
|
||||||
|
fake_cache = tmp_path / "kiwisdr_cache.json"
|
||||||
|
monkeypatch.setattr(kiwisdr_fetcher, "_CACHE_FILE", fake_cache)
|
||||||
|
|
||||||
|
# Make fetch_with_curl return a parseable but UNHEALTHY response
|
||||||
|
# (only 3 entries — well below the validation threshold).
|
||||||
|
class _GarbageResp:
|
||||||
|
status_code = 200
|
||||||
|
text = "var kiwisdr_com = [{\"name\":\"x\",\"gps\":\"(0,0)\"}];"
|
||||||
|
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"services.network_utils.fetch_with_curl", lambda *a, **kw: _GarbageResp()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Bypass the @cached decorator
|
||||||
|
kiwisdr_fetcher.kiwisdr_cache.clear()
|
||||||
|
|
||||||
|
result = kiwisdr_fetcher.fetch_kiwisdr_nodes()
|
||||||
|
# Should be the bundled fallback (798 entries), not the garbage (1 entry)
|
||||||
|
assert isinstance(result, list)
|
||||||
|
assert len(result) >= _MIN_HEALTHY_RECEIVER_COUNT
|
||||||
@@ -0,0 +1,114 @@
|
|||||||
|
"""Issue #208 (tg12): Merkle proofs were rebuilt from scratch on every
|
||||||
|
public ``/api/mesh/infonet/sync?include_proofs=true`` request. The
|
||||||
|
endpoint is part of the federation protocol so we can't add auth — the
|
||||||
|
fix is to cache the levels at append time so retrieval is O(1) per
|
||||||
|
proof, eliminating the DoS surface without breaking peer sync.
|
||||||
|
|
||||||
|
These tests verify:
|
||||||
|
|
||||||
|
* A fresh Infonet has no cache (lazy state).
|
||||||
|
* After ``append()``, the cache is invalidated.
|
||||||
|
* Two consecutive ``get_merkle_proofs()`` calls without an append return
|
||||||
|
identical results and don't rebuild — we assert this by reaching into
|
||||||
|
the cache attributes directly.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services.mesh.mesh_hashchain import Infonet
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fresh_infonet(monkeypatch, tmp_path):
|
||||||
|
"""Build a clean Infonet rooted at a temp directory."""
|
||||||
|
# Redirect persistence to the temp dir so we don't pollute real state.
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"services.mesh.mesh_hashchain.CHAIN_FILE",
|
||||||
|
tmp_path / "infonet_chain.json",
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
"services.mesh.mesh_hashchain.WAL_PATH",
|
||||||
|
tmp_path / "infonet_chain.wal",
|
||||||
|
raising=False,
|
||||||
|
)
|
||||||
|
inst = Infonet()
|
||||||
|
inst.events = [] # ensure empty
|
||||||
|
inst._invalidate_merkle_cache()
|
||||||
|
return inst
|
||||||
|
|
||||||
|
|
||||||
|
def test_cache_starts_empty(fresh_infonet):
|
||||||
|
"""The cache fields exist and start in their lazy state."""
|
||||||
|
assert hasattr(fresh_infonet, "_merkle_levels_cache")
|
||||||
|
assert fresh_infonet._merkle_levels_cache is None
|
||||||
|
assert fresh_infonet._merkle_levels_for_event_count == -1
|
||||||
|
|
||||||
|
|
||||||
|
def test_get_merkle_root_populates_cache(fresh_infonet):
|
||||||
|
"""First call computes and caches the levels."""
|
||||||
|
# Add a synthetic event so there's something to hash
|
||||||
|
fresh_infonet.events = [{"event_id": "a" * 64}, {"event_id": "b" * 64}]
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
assert fresh_infonet._merkle_levels_cache is not None
|
||||||
|
assert fresh_infonet._merkle_levels_for_event_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_repeated_root_calls_reuse_cache(fresh_infonet):
|
||||||
|
"""The cache survives multiple reads when no events were appended."""
|
||||||
|
fresh_infonet.events = [{"event_id": "a" * 64}, {"event_id": "b" * 64}]
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
cached_levels = fresh_infonet._merkle_levels_cache
|
||||||
|
cached_count = fresh_infonet._merkle_levels_for_event_count
|
||||||
|
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
# Same object — no rebuild.
|
||||||
|
assert fresh_infonet._merkle_levels_cache is cached_levels
|
||||||
|
assert fresh_infonet._merkle_levels_for_event_count == cached_count
|
||||||
|
|
||||||
|
|
||||||
|
def test_append_invalidates_cache(fresh_infonet):
|
||||||
|
"""After events change, the cache_for_count diverges from len(events).
|
||||||
|
|
||||||
|
The next read recomputes; that's the architectural point.
|
||||||
|
"""
|
||||||
|
fresh_infonet.events = [{"event_id": "a" * 64}]
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
assert fresh_infonet._merkle_levels_for_event_count == 1
|
||||||
|
|
||||||
|
# Simulate an append's side effect (the real append() also calls
|
||||||
|
# _invalidate_merkle_cache() — we test that integration in the
|
||||||
|
# in-tree append-flow test, not here).
|
||||||
|
fresh_infonet.events.append({"event_id": "b" * 64})
|
||||||
|
fresh_infonet._invalidate_merkle_cache()
|
||||||
|
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
assert fresh_infonet._merkle_levels_for_event_count == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_proofs_use_cache(fresh_infonet):
|
||||||
|
"""get_merkle_proofs() reads from the same cache get_merkle_root() does."""
|
||||||
|
fresh_infonet.events = [
|
||||||
|
{"event_id": (str(i) * 64)[:64]} for i in range(8)
|
||||||
|
]
|
||||||
|
_ = fresh_infonet.get_merkle_root()
|
||||||
|
cached_levels = fresh_infonet._merkle_levels_cache
|
||||||
|
|
||||||
|
proofs = fresh_infonet.get_merkle_proofs(0, 8)
|
||||||
|
assert proofs["total"] == 8
|
||||||
|
assert len(proofs["proofs"]) == 8
|
||||||
|
# Cache wasn't rebuilt — same object as before the proof call.
|
||||||
|
assert fresh_infonet._merkle_levels_cache is cached_levels
|
||||||
|
|
||||||
|
|
||||||
|
def test_empty_chain_returns_genesis(fresh_infonet):
|
||||||
|
"""An empty chain should serve GENESIS_HASH without computing levels."""
|
||||||
|
from services.mesh.mesh_hashchain import GENESIS_HASH
|
||||||
|
|
||||||
|
root = fresh_infonet.get_merkle_root()
|
||||||
|
assert root == GENESIS_HASH
|
||||||
|
|
||||||
|
proofs = fresh_infonet.get_merkle_proofs(0, 0)
|
||||||
|
assert proofs["total"] == 0
|
||||||
|
assert proofs["root"] == GENESIS_HASH
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
"""Issue #203 (tg12): meshtastic_map.py was unconditionally including
|
||||||
|
``MESHTASTIC_OPERATOR_CALLSIGN`` in the outbound User-Agent header,
|
||||||
|
which contradicted the README's "no user data transmitted" claim.
|
||||||
|
|
||||||
|
The fix preserves the existing default behavior (callsign sent — that's
|
||||||
|
what operators who configured the variable expected) but adds an
|
||||||
|
opt-out env var ``MESHTASTIC_SEND_CALLSIGN_HEADER=false`` for
|
||||||
|
privacy-conscious operators.
|
||||||
|
"""
|
||||||
|
import importlib
|
||||||
|
import sys
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
|
||||||
|
def _reload_meshtastic_module():
|
||||||
|
"""Reload meshtastic_map so settings are re-read on demand."""
|
||||||
|
if "services.fetchers.meshtastic_map" in sys.modules:
|
||||||
|
del sys.modules["services.fetchers.meshtastic_map"]
|
||||||
|
return importlib.import_module("services.fetchers.meshtastic_map")
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_behavior_includes_callsign(monkeypatch):
|
||||||
|
"""Operators who set the callsign and don't change anything else
|
||||||
|
keep their existing behavior (callsign sent in UA)."""
|
||||||
|
# We test the UA construction logic by exercising the same branches
|
||||||
|
# the fetcher uses. Direct fetch isn't run because it makes a real
|
||||||
|
# network call — we just verify the env-var-driven decision.
|
||||||
|
import os
|
||||||
|
monkeypatch.setenv("MESHTASTIC_OPERATOR_CALLSIGN", "N0CALL")
|
||||||
|
monkeypatch.delenv("MESHTASTIC_SEND_CALLSIGN_HEADER", raising=False)
|
||||||
|
|
||||||
|
raw = str(os.environ.get("MESHTASTIC_SEND_CALLSIGN_HEADER", "true")).strip().lower()
|
||||||
|
send_callsign_header = raw not in {"0", "false", "no", "off", ""}
|
||||||
|
assert send_callsign_header is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_opt_out_suppresses_callsign(monkeypatch):
|
||||||
|
"""Setting MESHTASTIC_SEND_CALLSIGN_HEADER=false suppresses the header."""
|
||||||
|
import os
|
||||||
|
monkeypatch.setenv("MESHTASTIC_OPERATOR_CALLSIGN", "N0CALL")
|
||||||
|
monkeypatch.setenv("MESHTASTIC_SEND_CALLSIGN_HEADER", "false")
|
||||||
|
|
||||||
|
raw = str(os.environ.get("MESHTASTIC_SEND_CALLSIGN_HEADER", "true")).strip().lower()
|
||||||
|
send_callsign_header = raw not in {"0", "false", "no", "off", ""}
|
||||||
|
assert send_callsign_header is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_various_falsy_values_all_opt_out(monkeypatch):
|
||||||
|
"""Common falsy strings should all suppress the callsign header."""
|
||||||
|
import os
|
||||||
|
for falsy in ("0", "false", "FALSE", "no", "off"):
|
||||||
|
monkeypatch.setenv("MESHTASTIC_SEND_CALLSIGN_HEADER", falsy)
|
||||||
|
raw = str(os.environ.get("MESHTASTIC_SEND_CALLSIGN_HEADER", "true")).strip().lower()
|
||||||
|
send_callsign_header = raw not in {"0", "false", "no", "off", ""}
|
||||||
|
assert send_callsign_header is False, f"value {falsy!r} did not opt out"
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
"""Issue #205 (tg12): the OpenMHZ audio proxy must re-validate the host on
|
||||||
|
every redirect hop, not just the first one.
|
||||||
|
|
||||||
|
Before this fix, ``openmhz_audio_response()`` called
|
||||||
|
``requests.get(..., stream=True, timeout=...)`` with the default
|
||||||
|
``allow_redirects=True``. The initial URL host was validated against
|
||||||
|
``_OPENMHZ_AUDIO_HOSTS``, but any subsequent redirect was silently
|
||||||
|
followed — even to ``http://127.0.0.1:8000`` or RFC1918 internal ranges.
|
||||||
|
Classic open-redirect-to-SSRF.
|
||||||
|
|
||||||
|
After the fix, redirects are followed manually with per-hop host
|
||||||
|
re-validation. Same-host redirects (CDN edge selection) still work,
|
||||||
|
so legitimate audio playback is unaffected.
|
||||||
|
"""
|
||||||
|
import pytest
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
from fastapi import HTTPException
|
||||||
|
|
||||||
|
from services.radio_intercept import _OPENMHZ_MAX_REDIRECTS, openmhz_audio_response
|
||||||
|
|
||||||
|
|
||||||
|
class _Resp:
|
||||||
|
"""Minimal mock for requests.Response."""
|
||||||
|
|
||||||
|
def __init__(self, status_code=200, headers=None, is_redirect=False):
|
||||||
|
self.status_code = status_code
|
||||||
|
self.headers = headers or {}
|
||||||
|
self.is_redirect = is_redirect
|
||||||
|
self.closed = False
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
self.closed = True
|
||||||
|
|
||||||
|
def iter_content(self, chunk_size=64 * 1024):
|
||||||
|
return iter([])
|
||||||
|
|
||||||
|
|
||||||
|
@patch("services.radio_intercept.requests.get")
|
||||||
|
def test_redirect_to_internal_address_rejected(mock_get):
|
||||||
|
"""A 302 from media.openmhz.com -> 127.0.0.1 must be rejected."""
|
||||||
|
mock_get.side_effect = [
|
||||||
|
_Resp(status_code=302, headers={"Location": "http://127.0.0.1:8000/api/secret"}, is_redirect=True),
|
||||||
|
]
|
||||||
|
with pytest.raises(HTTPException) as exc_info:
|
||||||
|
openmhz_audio_response("https://media.openmhz.com/audio/abc.mp3")
|
||||||
|
assert exc_info.value.status_code == 502
|
||||||
|
|
||||||
|
|
||||||
|
@patch("services.radio_intercept.requests.get")
|
||||||
|
def test_redirect_to_arbitrary_domain_rejected(mock_get):
|
||||||
|
"""A 302 to an attacker-controlled domain must be rejected."""
|
||||||
|
mock_get.side_effect = [
|
||||||
|
_Resp(status_code=302, headers={"Location": "https://evil.example/exfil"}, is_redirect=True),
|
||||||
|
]
|
||||||
|
with pytest.raises(HTTPException) as exc_info:
|
||||||
|
openmhz_audio_response("https://media.openmhz.com/audio/abc.mp3")
|
||||||
|
assert exc_info.value.status_code == 502
|
||||||
|
|
||||||
|
|
||||||
|
@patch("services.radio_intercept.requests.get")
|
||||||
|
def test_redirect_to_another_openmhz_cdn_followed(mock_get):
|
||||||
|
"""A 302 from media.openmhz.com -> media2.openmhz.com (same allowlist) is OK."""
|
||||||
|
mock_get.side_effect = [
|
||||||
|
_Resp(status_code=302, headers={"Location": "https://media2.openmhz.com/audio/abc.mp3"}, is_redirect=True),
|
||||||
|
_Resp(status_code=200, headers={"Content-Type": "audio/mpeg"}),
|
||||||
|
]
|
||||||
|
resp = openmhz_audio_response("https://media.openmhz.com/audio/abc.mp3")
|
||||||
|
# StreamingResponse-shaped object — we just check it was constructed.
|
||||||
|
assert resp is not None
|
||||||
|
|
||||||
|
|
||||||
|
@patch("services.radio_intercept.requests.get")
|
||||||
|
def test_redirect_chain_length_bounded(mock_get):
|
||||||
|
"""A redirect loop must terminate within _OPENMHZ_MAX_REDIRECTS."""
|
||||||
|
mock_get.side_effect = [
|
||||||
|
_Resp(status_code=302, headers={"Location": "https://media.openmhz.com/loop"}, is_redirect=True)
|
||||||
|
for _ in range(_OPENMHZ_MAX_REDIRECTS + 2)
|
||||||
|
]
|
||||||
|
with pytest.raises(HTTPException) as exc_info:
|
||||||
|
openmhz_audio_response("https://media.openmhz.com/audio/abc.mp3")
|
||||||
|
assert exc_info.value.status_code == 502
|
||||||
|
|
||||||
|
|
||||||
|
@patch("services.radio_intercept.requests.get")
|
||||||
|
def test_redirect_to_http_scheme_rejected(mock_get):
|
||||||
|
"""A 302 to http:// (instead of https://) must be rejected even on same host."""
|
||||||
|
mock_get.side_effect = [
|
||||||
|
_Resp(status_code=302, headers={"Location": "http://media.openmhz.com/audio/abc.mp3"}, is_redirect=True),
|
||||||
|
]
|
||||||
|
with pytest.raises(HTTPException) as exc_info:
|
||||||
|
openmhz_audio_response("https://media.openmhz.com/audio/abc.mp3")
|
||||||
|
assert exc_info.value.status_code == 502
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
"""Issue #202 (tg12): the satellite overflights endpoint accepted an
|
||||||
|
unbounded ``hours`` parameter, letting an anonymous caller trigger
|
||||||
|
``O(catalog_size × timesteps)`` work by asking for an absurd window.
|
||||||
|
|
||||||
|
The fix clamps ``hours`` silently rather than raising a 422. The
|
||||||
|
response shape is identical, just covering a shorter window — this
|
||||||
|
keeps the API liberal in what it accepts (Postel) while removing the
|
||||||
|
DoS surface.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
|
||||||
|
from routers.data import _overflight_max_hours
|
||||||
|
|
||||||
|
|
||||||
|
def test_default_max_hours_is_72(monkeypatch):
|
||||||
|
monkeypatch.delenv("OVERFLIGHTS_MAX_HOURS", raising=False)
|
||||||
|
assert _overflight_max_hours() == 72
|
||||||
|
|
||||||
|
|
||||||
|
def test_env_override_accepted(monkeypatch):
|
||||||
|
monkeypatch.setenv("OVERFLIGHTS_MAX_HOURS", "168")
|
||||||
|
assert _overflight_max_hours() == 168
|
||||||
|
|
||||||
|
|
||||||
|
def test_invalid_env_value_falls_back_to_default(monkeypatch):
|
||||||
|
monkeypatch.setenv("OVERFLIGHTS_MAX_HOURS", "not-a-number")
|
||||||
|
assert _overflight_max_hours() == 72
|
||||||
|
|
||||||
|
|
||||||
|
def test_negative_env_value_clamped_to_minimum(monkeypatch):
|
||||||
|
monkeypatch.setenv("OVERFLIGHTS_MAX_HOURS", "-5")
|
||||||
|
assert _overflight_max_hours() == 1
|
||||||
|
|
||||||
|
|
||||||
|
def test_clamp_arithmetic_silent():
|
||||||
|
"""The endpoint should clamp huge requests without erroring.
|
||||||
|
|
||||||
|
We don't exercise the full FastAPI route (compute_overflights needs
|
||||||
|
cached GP data), but we do verify the clamping math used by the
|
||||||
|
route: min(requested, cap).
|
||||||
|
"""
|
||||||
|
requested = 1_000_000
|
||||||
|
cap = _overflight_max_hours()
|
||||||
|
effective = min(max(1, requested), cap)
|
||||||
|
assert effective == cap
|
||||||
|
assert effective < requested
|
||||||
@@ -0,0 +1,145 @@
|
|||||||
|
"""Issue #201 (tg12): Tor bundle integrity must come from at least one
|
||||||
|
trusted source. Previously, if the upstream ``.sha256sum`` was
|
||||||
|
unreachable, the bundle was extracted and executed anyway with only
|
||||||
|
HTTPS-level transport trust.
|
||||||
|
|
||||||
|
The fix introduces a multi-source verification chain:
|
||||||
|
|
||||||
|
1. Upstream ``.sha256sum`` (current behavior)
|
||||||
|
2. Baked-in digest list at ``backend/data/tor_bundle_digests.json``
|
||||||
|
3. If neither source is reachable AT ALL: HTTPS-only fallback with a
|
||||||
|
loud warning (avoids breaking first-run onboarding while the
|
||||||
|
maintainer hasn't yet pinned a new Tor release)
|
||||||
|
|
||||||
|
A mismatch from a source that DID respond is always fatal — only the
|
||||||
|
"no source reachable" case falls back to HTTPS-only.
|
||||||
|
"""
|
||||||
|
import hashlib
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from services import tor_hidden_service as tor_svc
|
||||||
|
from services.tor_hidden_service import (
|
||||||
|
_DIGEST_PLACEHOLDER,
|
||||||
|
_load_baked_in_digests,
|
||||||
|
_verify_tor_bundle,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def fake_bundle(tmp_path):
|
||||||
|
"""A tiny synthetic 'bundle' so we can compute its digest deterministically."""
|
||||||
|
archive = tmp_path / "fake-tor.tar.gz"
|
||||||
|
payload = b"this is not really a tar archive"
|
||||||
|
archive.write_bytes(payload)
|
||||||
|
expected = hashlib.sha256(payload).hexdigest().lower()
|
||||||
|
return archive, expected
|
||||||
|
|
||||||
|
|
||||||
|
def test_baked_in_digests_skips_placeholders(tmp_path, monkeypatch):
|
||||||
|
"""Entries with the placeholder value are filtered out."""
|
||||||
|
digest_file = tmp_path / "digests.json"
|
||||||
|
digest_file.write_text(
|
||||||
|
'{"https://example.com/a.tar.gz": "PLACEHOLDER_REPLACE_BEFORE_RELEASE", '
|
||||||
|
'"https://example.com/b.tar.gz": "deadbeef"}',
|
||||||
|
encoding="utf-8",
|
||||||
|
)
|
||||||
|
monkeypatch.setattr(tor_svc, "_TOR_DIGEST_FILE", digest_file)
|
||||||
|
|
||||||
|
digests = _load_baked_in_digests()
|
||||||
|
assert "https://example.com/a.tar.gz" not in digests
|
||||||
|
assert digests.get("https://example.com/b.tar.gz") == "deadbeef"
|
||||||
|
|
||||||
|
|
||||||
|
def test_verification_succeeds_when_upstream_matches(fake_bundle, monkeypatch):
|
||||||
|
"""Path A: upstream .sha256sum returns the matching digest."""
|
||||||
|
archive, expected = fake_bundle
|
||||||
|
|
||||||
|
def fake_urlretrieve(url, dest):
|
||||||
|
dest_path = Path(dest)
|
||||||
|
dest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
dest_path.write_text(f"{expected} bundle.tar.gz\n", encoding="utf-8")
|
||||||
|
|
||||||
|
monkeypatch.setattr(tor_svc, "urlretrieve", fake_urlretrieve)
|
||||||
|
monkeypatch.setattr(tor_svc, "_load_baked_in_digests", lambda: {})
|
||||||
|
|
||||||
|
verified, reason = _verify_tor_bundle(archive, "https://example.com/bundle.tar.gz")
|
||||||
|
assert verified is True
|
||||||
|
assert "upstream" in reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_verification_succeeds_via_baked_in_when_upstream_unreachable(fake_bundle, monkeypatch):
|
||||||
|
"""Path B: upstream .sha256sum fails; baked-in digest matches."""
|
||||||
|
archive, expected = fake_bundle
|
||||||
|
|
||||||
|
def fake_urlretrieve(url, dest):
|
||||||
|
raise RuntimeError("upstream unreachable")
|
||||||
|
|
||||||
|
monkeypatch.setattr(tor_svc, "urlretrieve", fake_urlretrieve)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
tor_svc, "_load_baked_in_digests",
|
||||||
|
lambda: {"https://example.com/bundle.tar.gz": expected},
|
||||||
|
)
|
||||||
|
|
||||||
|
verified, reason = _verify_tor_bundle(archive, "https://example.com/bundle.tar.gz")
|
||||||
|
assert verified is True
|
||||||
|
assert "baked-in" in reason
|
||||||
|
|
||||||
|
|
||||||
|
def test_verification_fails_when_upstream_disagrees(fake_bundle, monkeypatch):
|
||||||
|
"""Mismatch from a source that DID respond is always fatal."""
|
||||||
|
archive, _expected = fake_bundle
|
||||||
|
|
||||||
|
def fake_urlretrieve(url, dest):
|
||||||
|
dest_path = Path(dest)
|
||||||
|
dest_path.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
dest_path.write_text("0" * 64 + " bundle.tar.gz\n", encoding="utf-8")
|
||||||
|
|
||||||
|
monkeypatch.setattr(tor_svc, "urlretrieve", fake_urlretrieve)
|
||||||
|
monkeypatch.setattr(tor_svc, "_load_baked_in_digests", lambda: {})
|
||||||
|
|
||||||
|
verified, reason = _verify_tor_bundle(archive, "https://example.com/bundle.tar.gz")
|
||||||
|
assert verified is False
|
||||||
|
assert "mismatch" in reason.lower()
|
||||||
|
|
||||||
|
|
||||||
|
def test_verification_fails_when_baked_in_disagrees(fake_bundle, monkeypatch):
|
||||||
|
"""Even with no upstream, a baked-in mismatch is fatal."""
|
||||||
|
archive, _expected = fake_bundle
|
||||||
|
|
||||||
|
def fake_urlretrieve(url, dest):
|
||||||
|
raise RuntimeError("upstream unreachable")
|
||||||
|
|
||||||
|
monkeypatch.setattr(tor_svc, "urlretrieve", fake_urlretrieve)
|
||||||
|
monkeypatch.setattr(
|
||||||
|
tor_svc, "_load_baked_in_digests",
|
||||||
|
lambda: {"https://example.com/bundle.tar.gz": "0" * 64},
|
||||||
|
)
|
||||||
|
|
||||||
|
verified, reason = _verify_tor_bundle(archive, "https://example.com/bundle.tar.gz")
|
||||||
|
assert verified is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_verification_falls_back_to_https_when_no_source_reachable(fake_bundle, monkeypatch, caplog):
|
||||||
|
"""No source available → HTTPS-only fallback with a loud warning.
|
||||||
|
|
||||||
|
This preserves first-run onboarding while the maintainer hasn't
|
||||||
|
yet pinned a particular Tor release in the digest file.
|
||||||
|
"""
|
||||||
|
archive, _expected = fake_bundle
|
||||||
|
|
||||||
|
def fake_urlretrieve(url, dest):
|
||||||
|
raise RuntimeError("upstream unreachable")
|
||||||
|
|
||||||
|
monkeypatch.setattr(tor_svc, "urlretrieve", fake_urlretrieve)
|
||||||
|
monkeypatch.setattr(tor_svc, "_load_baked_in_digests", lambda: {})
|
||||||
|
|
||||||
|
import logging
|
||||||
|
with caplog.at_level(logging.WARNING):
|
||||||
|
verified, reason = _verify_tor_bundle(archive, "https://example.com/bundle.tar.gz")
|
||||||
|
assert verified is True
|
||||||
|
assert "https-only" in reason.lower()
|
||||||
|
assert any(
|
||||||
|
"fell back to HTTPS-only" in record.getMessage() for record in caplog.records
|
||||||
|
)
|
||||||
Reference in new issue
Block a user